SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Point removal experiment Figure 4 shows test accuracy curves for the point removal experiment. Overall, all data valuation algorithms except Data-OOB perform better than the random baseline. Data-OOB often shows worse performance than the random baseline, indicating its effectiveness for finding low-quality data points does not imply finding helpful data points. AME, DataShapley, BetaShapley, and KNNShapley generally show superior performance. In particular, on the 'fried' dataset, these methods achieve at most  $75\%$  test accuracy after removing  $20\%$  of the most influential data points. DataBanzhaf and InfluenceFunction also show solid performance, while DVRL shows large performance differences across different datasets. LOO shows a slightly better performance than the random baseline.
Point addition experiment Figure 5 shows test accuracy curves for the point addition experiment. All data valuation algorithms generally outperform the random baseline, showing their efficacy in identifying low-quality data points. Data-OOB uniformly shows the lowest test accuracy compared
Figure 4: Point removal experiment. Test accuracy curves as a function of the most valuable data points removed. Lower curve indicates better data valuation algorithm. The error bar indicates a  $95\%$  confidence interval based on 50 independent experiments. AME and Shapley-based methods exhibit superior performance.
Figure 5: Point addition experiment. Test accuracy curves as a function of the least valuable data points added. Lower curve indicates better data valuation algorithm. The error bar indicates a  $95\%$  confidence interval based on 50 independent experiments. Data-OOB, DVRL, and Shapley-based methods exhibit superior performance.
to other algorithms, demonstrating the best performance in finding low-quality samples. Following Data-OOB, DVRL, DataShapley, and BetaShapley show promising performance in this task. Volume-based Shapley, LAVA, LOO, and InfluenceFunction show similar test accuracy, but InfluenceFunction exhibits a slightly better performance than other methods across all datasets. Volume-based Shapley is generally worse or only better than the random baseline, which has also been observed in the literature [13]. This is because the Volume-based Shapley is independent of the label information, and thus it incompletely reflects the quality of data points.
4.3 Runtime comparison
We also conduct a comparison of the runtime required to compute 1000 data values for the noisy label data detection task. In this task, we use the three tabular datasets (2dplanes, electricity, and
Figure 6: Runtime comparison. The runtime to compute 1000 data values is measured for the noisy label data detection task. The  $x$ -axis represents the F1-score used in Section 4.1 and the  $y$ -axis represents the runtime in the log scale. KNNShapley shows the best performance in terms of speed, while Data-OOB achieves a better F1-score in a reasonable time frame. Both DataShapley and BetaShapley are computationally expensive because they are based on marginal contribution estimation algorithms. They perform similarly in BBC, IMDB, and CIFAR10 datasets, and their points are visually overlapped.
MiniBooNE), two text datasets (BBC and IMDB), and one image dataset (CIFAR10) in order to show that findings are overall consistent across different modalities. We find a similar pattern with other tabular datasets. We utilize a noise dataset containing label noise with a noise ratio of  $p_{\mathrm{noise}} = 20\%$ , but we highlight that the runtime is independent of the specific type of noise or its intensity. Figure 6 illustrates the runtime of the data valuation algorithms, along with the corresponding F1-scores for the noisy label data detection. Across most datasets, LAVA and KNNShapley demonstrate the shortest computation time because LAVA does not require model training and KNNShapley has a closed-form expression. In contrast, DataShapley and BetaShapley are shown to be computationally costly because they need to estimate the marginal contribution, which requires a large number of model training. Aside from these three algorithms, most of the other data valuation algorithms exhibit similar computational times as their key hyperparameter, the number of models to train, is set to the same value of 1000. Among them, Data-OOB demonstrates the best F1-score in noisy label data detection.
In this paper, we propose OpenDataVal, an easy-to-use benchmark environment, and provide a convenient way to compute and compare data valuation algorithms with a few lines of Python codes. To ensure transparency and reproducibility, we have made our framework publicly available.
Our benchmarking analysis demonstrates no single method is uniformly superior across all evaluation metrics. Data-OOB achieves a powerful performance in the noisy label data detection tasks, but it performs less well in the point removal experiment. That is, it is very effective in identifying low-quality data points, but it fails to find the most positively influential data points. In contrast, AME shows the opposite results. By leveraging the LASSO model, it shows competitive performance in identifying the most beneficial data points in the point removal experiment, while it shows weak performance in finding low-quality data points in the noisy data detection task. We also find that Shapley-based methods (DataShapley, BetaShapley, and KNNShapley) have comparable performances. KNNShapley is faster but is limited to using KNN predictors, while DataShapley and BetaShapley can flexibly work with other predictors. Many theoretically rigorous and empirically
comprehensive analyses are critical, and we believe OpenDataVal opens the first step and facilitates fair and reproducible analyses in various experimental settings.
There are several intriguing future directions. In many real-world applications data can be duplicated and a fraction of them can be hard-encoded with a certain value (e.g., -9999 for missing data) [14]. Furthermore, data can contain incomplete information in survival analysis, semi-supervised learning, and weakly-supervised learning settings, raising questions about how to evaluate the influence of such data points. Developing data valuation for sequentially observed data is also an important direction. For instance, in time-series forecasting or multi-armed bandit problems, previously observed data can substantially affect the subsequent data points, which makes it difficult to deploy techniques in typical i.i.d. learning settings. Developing a notion of data values in these scenarios will be an interesting and potentially influential problem.
In a broader context, the practical implementation of data valuation can give rise to several economic and societal questions. In data marketplaces, for instance, data providers might duplicate, transform, or even maliciously modify data to maximize their business profits. Most of the existing data valuation algorithms are not designed to handle these types of attacks, and as a result, erroneously evaluated data values can negatively affect the establishment of reliable and transparent data marketplaces. Developing incentive-compatible data valuation is an interesting new direction [32, 28, 34]. Another potential concern is data security in distributed and federated learning scenarios. Data owners may hesitate to share their data with the main server due to privacy issues, especially if data owners are competing with each other [34]. Evaluating data points based on aggregated summaries (e.g., gradients) will be an interesting topic as existing data valuation algorithms require direct data access.
The authors would like to thank all anonymous reviewers for their comments. We would like to thank Eunsil Seok and Ian Covert for their constructive feedback. We also acknowledge computing resources from Columbia University's Shared Research Computing Facility project, which is supported by NIH Research Facility Improvement Grant 1G20RR030893-01, and associated funds from the New York State Empire State Development, Division of Science Technology and Innovation (NYSTAR) Contract C090171, both awarded April 15, 2010.
[1] Anish Agarwal, Munther Dahleh, and Tuhin Sarkar. A marketplace for data: An algorithmic solution. In Proceedings of the 2019 ACM Conference on Economics and Computation, pages 701-726, 2019.
[2] David Arthur and Sergei Vassilvitskii. k-means++ the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 1027-1035, 2007.
[3] Laurent Candillier and Vincent Lemaire. Design and analysis of the nomao challenge active learning in the real-world. In Proceedings of the ALRA: active learning in real-world applications, workshop ECML-PKDD, pages 1-15. Citeseer, 2012.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
[5] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 2881-2891, 2020.
[6] Joao Gama, Pedro Medas, Gladys Castillo, and Pedro Rodrigues. Learning with drift detection. In Advances in Artificial Intelligence-SBIA 2004: 17th Brazilian Symposium on Artificial Intelligence, Sao Luis, Maranhao, Brazil, September 29-October 1, 2004. Proceedings 17, pages 286-295. Springer, 2004.
[7] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242-2251, 2019.
The appendix provides technical details on data valuation algorithms (Section A) and the four downstream tasks (Section B) available in OpenDataVal. Additional benchmarking analysis results are also presented (Section C) and we observe consistent findings on different datasets. Lastly, we provide detailed descriptions on OpenDataVal APIs in (Section D).
A Data valuation algorithms
This section elaborates on different data valuation algorithms available in OpenDataVal. While some notations have already been defined in Section 2, we present a comprehensive set of notations here. For  $d \in \mathbb{N}$ , we denote an input space and an output space by  $\mathcal{X} \subseteq \mathbb{R}^d$  and  $\mathcal{Y} \subseteq \mathbb{R}$ , respectively. We denote a training dataset by  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$  where  $x_i \in \mathcal{X}$  and  $y_i \in \mathcal{Y}$  are an input and its label for the  $i$ -th datum. For a set  $S$ , we denote its power set by  $2^S$  and its cardinality by  $|S|$ . We set  $[j] := \{1, \ldots, j\}$  for  $j \in \mathbb{N}$ , and  $\mathbb{1}(A)$  is an indicator whose value is one if a statement  $A$  is true and zero otherwise.
A.1 Marginal Contribution-based methods
The marginal contribution-based methods are formally introduced, which include LOO, DataShapley [7], KNNShapley [12], Volume-based Shapely [41], BetaShapley [17], DataBanzhaf [39], and AME [22]. We first define the marginal contribution below.
Definition A.1 (Marginal contribution). For a set function  $U: 2^{\mathcal{D}} \to \mathbb{R}$  and  $j \in [n]$ , the marginal contribution of  $(x_i, y_i) \in \mathcal{D}$  with respect to  $j$  samples is defined as follows.
$$
\Delta_{j}(x_{i},y_{i};U):= \frac{1}{\binom{n - 1}{j - 1}}\sum_{S\subseteq \mathcal{D}_{j}^{\setminus (x_{i},y_{i})}}U(S\cup \{(x_{i},y_{i})\}) - U(S),
$$
where  $\mathcal{D}_j^{\backslash (x_i,y_i)}\coloneqq \{S:S\subseteq \mathcal{D}\backslash \{(x_i,y_i)\} ,|S| = j - 1\}$
The marginal contribution  $\Delta_j(x_i, y_i; U)$  measures the average change in  $U$  when a particular data point  $(x_i, y_i)$  is removed. Here, a common choice for a set function  $U$  is the performance of a model trained on a subset of the training dataset  $\mathcal{D}$ . We provide two examples below.
Example A.1 (Classification). A default choice for a set function  $U$  in OpenDataVal is the validation classification accuracy of an empirical risk minimizer trained on a subset of  $\mathcal{D}$ . To be more specific, we denote the validation dataset by  $\{(\tilde{x}_i,\tilde{y}_i)\}_{i = 1}^{n_{\mathrm{val}}}$ . Then,
$$
U (S) := \frac {1}{n _ {\mathrm {v a l}}} \sum_ {i = 1} ^ {n _ {\mathrm {v a l}}} \mathbb {1} (\tilde {y} _ {i} = \hat {f} _ {S} (\tilde {x} _ {i})),
$$
where  $\hat{f}_S\coloneqq \operatorname *{argmin}_{f\in \mathcal{F}}\sum_{j\in S}\mathbb{1}(y_j\neq f(x_j))$  for some class of classification models  $\mathcal{F} = \{f|f:$ $\mathcal{X}\to \mathcal{Y}\}$ . When  $S = \{\}$  is the empty set,  $U(S)$  is set to be the performance of the best constant model by convention.
Example A.2 (Regression). A default choice for a set function  $U$  in OpenDataVal is the validation negative mean squared error of an empirical risk minimizer trained on a subset of  $\mathcal{D}$ . Using the same notation defined in Example A.1,
$$
U (S) := - \frac {1}{n _ {\mathrm {v a l}}} \sum_ {i = 1} ^ {n _ {\mathrm {v a l}}} (\tilde {y} _ {i} - \hat {f} _ {S} (\tilde {x} _ {i})) ^ {2},
$$
where  $\hat{f}_S\coloneqq \operatorname *{argmin}_{f\in \mathcal{F}}\sum_{j\in S}(y_j - f(x_j))^2$  for some class of regression models  $\mathcal{F} = \{f|f:\mathcal{X}\to \mathcal{Y}\}$ .
It is noteworthy that a set function  $U$  may depend on the choice of learning algorithms (e.g., empirical risk minimization) and a model class  $\mathcal{F}$ . For notational simplicity, we suppress a learning algorithm and a model class.
Estimation of the marginal contributions. Following the existing literature [7], OpenDataVal uses a truncated Monte Carlo algorithm. Specifically, let  $\pi = (\pi_1,\dots ,\pi_n)\in \mathbb{N}^n$  be a random permutation of  $[n]$ . Then, for  $j\in [n]$ , we compute the utility  $U(\{(x_{\pi_k},y_{\pi_k})\}_{k = 1}^j)$ . To gradually increase the cardinality, we iteratively increment  $j$  from 1 to  $n$ . We stop increasing the cardinality if a relative change is converged. Specifically, for  $l\in [n]$ , we define the relative change as follows.
$$
V _ {l} := \frac {\left| U \left(\left\{\left(x _ {\pi_ {k}} , y _ {\pi_ {k}}\right) \right\} _ {k = 1} ^ {l} \cup \left\{x _ {\pi_ {l + 1}} , y _ {\pi_ {l + 1}} \right\}\right) - U \left(\left\{\left(x _ {\pi_ {k}} , y _ {\pi_ {k}}\right) \right\} _ {k = 1} ^ {l}\right) \right|}{U \left(\left\{\left(x _ {\pi_ {k}} , y _ {\pi_ {k}}\right) \right\} _ {k = 1} ^ {l}\right)}.
$$
We stop increasing the cardinality at  $j = \operatorname{argmin}\{j' \in [n] : |\{l \leq j' : V_l \leq 10^{-8}\}| \geq 10\}$ , i.e., it is the smallest cardinality  $j$  that there are 10 cardinalities less than  $j$  with a small relative change  $V_l$ .
While repeatedly generating random permutations, we use the utility gap to estimate the marginal contribution. In other words, the multiple estimates of  $U(\{\pi_1,\dots ,\pi_j,\pi_{j + 1}\}) - U(\{\pi_1,\dots ,\pi_j\})$  are used to estimate  $\Delta_{j + 1}(x_{\pi_{j + 1}},y_{\pi_{j + 1}})$ . We stop generating random permutations when a simple average of marginal contributions is converged. We used the Gelman-Rubin statistics [38] by constructing ten independent Monte Carlo chains for each training data point. The maximum of these statistics across samples is considered, and we stop drawing random permutations if the maximum value of Gelman-Rubin statistics is less than the threshold value of 1.05 or the number of random permutations exceeds  $10^{4}$ . In OpenDataVa1, to obtain reliable estimates, we set the default value for the minimum number of random permutations as 300.
LOO. The LOO measures the change in  $U$  when one data point of interest is removed from the entire dataset, and it is given as follows.
$$
\phi_ {\mathrm {l o o}} (x _ {i}, y _ {i}) := \Delta_ {n} (x _ {i}, y _ {i}).
$$
Note that  $\phi_{\mathrm{loo}}(x_i,y_i) = U(\mathcal{D}) - U(\mathcal{D}\backslash \{(x_i,y_i)\})$  . In OpenDataVal, the computation of LOO is done by naively computing  $U(\mathcal{D})$  and  $U(\mathcal{D}\backslash \{(x_i,y_i)\})$  for all  $i\in [n]$
DataShapley. DataShapley takes a simple average of all the marginal contributions as follows.
$$
\phi_ {\mathrm {s h a p}} \left(x _ {i}, y _ {i}\right) := \frac {1}{n} \sum_ {j = 1} ^ {n} \Delta_ {j} \left(x _ {i}, y _ {i}\right).
$$
In OpenDataVal, as suggested in Ghorbani and Zou [7], we estimate the marginal contributions using the truncated Monte Carlo algorithm and compute its simple average.
KNNShapley. KNNShapley, like DataShapley, is based on the Shapley value but is characterized by the use of a specific utility function. In classification settings, for instance, KNNShapley uses the following utility function.
$$
U (S) = \frac {1}{n _ {\mathrm {v a l}} k} \sum_ {i = 1} ^ {n _ {\mathrm {v a l}}} \sum_ {(x _ {j}, y _ {j}) \in \mathcal {N} (\tilde {x} _ {i}, S)} \mathbb {1} (\tilde {y} _ {i} = y _ {j}),
$$
where  $k \in [n]$  is a preset hyperparameter and represents  $k$  nearest neighbors in KNN predictors and  $\mathcal{N}(\tilde{x}_i, S) \subseteq \mathcal{D}$  is a set of  $\min(k, |S|)$  nearest neighbors of  $\tilde{x}_i$ . In regression settings,
$$
U (S) = - \frac {1}{n _ {\mathrm {v a l}} k} \sum_ {i = 1} ^ {n _ {\mathrm {v a l}}} \sum_ {(x _ {j}, y _ {j}) \in \mathcal {N} (\tilde {x} _ {i}, S)} (\tilde {y} _ {i} - y _ {j}) ^ {2}.
$$
With these utility functions, [12] showed that KNNShapley yields a closed-form expression, and they are implemented in OpenDataVal.
Volume-based Shapley The idea of the volume-based Shapley by Xu et al. [41] is to use the same Shapley value function as DataShapley, but it is characterized by using the volume of input data for a utility function.
$$
U (S) = \operatorname {V o l} \left(X _ {S}\right) := \sqrt {\left| \sum_ {i \in S} x _ {i} x _ {i} ^ {T} \right|}, \tag {1}
$$
where  $|G|$  is a determinant of a matrix  $G \in \mathbb{R}^{d \times d}$ . It is noteworthy that Equation (1) does not require a validation dataset, leading to a validation-free data valuation method. However, it does not use label information, and thus it is not appropriate to describe the quality of labels. The previous work [13] observed that it is often the main reason for poor performance in many downstream tasks.
BetaShapley. BetaShapley has a form of a weighted mean of the marginal contributions.
$$
\phi_ {\mathrm {b e t a}} (x _ {i}, y _ {i}) := \sum_ {j = 1} ^ {n} w _ {\mathrm {b e t a}, j} \Delta_ {j} (x _ {i}, y _ {i}),
$$
where  $w_{\mathrm{beta},j} = \binom{n-1}{j-1}\frac{\mathrm{Beta}(j+\beta-1,n-j+\alpha)}{\mathrm{Beta}(\alpha,\beta)}$ ,  $\mathrm{Beta}(\alpha,\beta) = \Gamma(\alpha)\Gamma(\beta)/\Gamma(\alpha+\beta)$  is the Beta function and  $\Gamma$  is the Gamma function. In OpenDataVal,  $(\alpha,\beta)=(4,1)$  is set to the default parameter<sup>4</sup>, and similar to DataShapley, BetaShapley values are obtained after estimating the marginal contributions.
DataBanzhaf. DataBanzhaf is based on the Banzhaf value, which is defined as follows.
$$
\phi_ {\mathrm {b a n z}} (x _ {i}, y _ {i}) := \sum_ {j = 1} ^ {n} w _ {\mathrm {b a n z}, j} \Delta_ {j} (x _ {i}, y _ {i}),
$$
where  $w_{\mathrm{banz},j} = 2^{-n}\binom{n-1}{j-1}$ . In OpenDataVal, instead of using the marginal contribution estimates, we follow the original algorithms proposed by Wang and Jia [39].
AME. Lin et al. [22] proposed the average marginal effect, which is also known as AME.
$$
\psi_ {\mathrm {A M E}} \left(x _ {i}, y _ {i}\right) := \mathbb {E} _ {S} \left[ U \left(S \cup \left\{\left(x _ {i}, y _ {i}\right) \right\}\right) - U (S) \right],
$$
where the expectation  $\mathbb{E}_S$  is taken over a random set  $S$  with a user-defined distribution defined on the discrete space  $\cup_{j=1}^{n}\mathcal{D}_{j}^{\backslash (x_{i},y_{i})}$ . Lin et al. [22] showed that AME can be efficiently estimated by predicting a model's prediction. Specifically, for a regularization parameter  $\lambda > 0$ , a predefined transformation function  $g:\{0,1\}^{n}\to \mathbb{R}^{n}$ , and a set  $S = \{S:S\subseteq \mathcal{D}\}$ , AME can be estimated as a solution of a LASSO regression model.
$$
\operatorname * {a r g m i n} _ {\gamma \in \mathbb {R} ^ {n}} \frac {1}{| \mathcal {S} |} \sum_ {S \in \mathcal {S}} \left(U (S) - g (\mathbb {1} _ {S}) ^ {T} \gamma\right) ^ {2} + \lambda \sum_ {i = 1} ^ {n} | \gamma_ {i} |,
$$
where  $\mathbb{1}_S\in \{0,1\} ^n$  is  $n$  -dimensional vector whose element is one if its index is an element of  $S$  , zero otherwise. In OpenDataVal, following Lin et al. [22], the same transformation function  $g$  and the  $\mathcal{S}$  that follows the same uniform distribution are used. We choose the optimal regularization parameter with the cross-validation using LassoCV' in 'scikit-learn' with its default parameter values.
A.2 Gradient-based methods
Influence function. The influence function is originally introduced in robust statistics to measure the effect of a data point on a statistical estimator by its Gateaux derivative [9]. In machine learning, it has been often approximated by LOO, or more generally, the difference between two average model performances: one containing a data point of interest in the training procedure and the other not  $[15]^5$ . To be more specific, for a pre-defined hyperparameter  $M \in [n]$ , we denote a set of subsets with the same cardinality by  $\mathcal{S}^{(M)} := \{S : S \subseteq \mathcal{D}, |S| = M\}$ . For  $i \in [n]$ , we set  $\mathcal{S}_{i,\mathrm{in}}^{(M)} := \{S \in \mathcal{S}^{(M)} : (x_i, y_i) \in S\}$  and  $\mathcal{S}_{i,\mathrm{out}}^{(M)} := \{S \in \mathcal{S}^{(M)} : (x_i, y_i) \notin S\}$ . Note that  $\mathcal{S}_{i,\mathrm{in}}^{(M)} \cup \mathcal{S}_{i,\mathrm{out}}^{(M)} = \mathcal{S}$  and  $\mathcal{S}_{i,\mathrm{in}}^{(M)} \cap \mathcal{S}_{i,\mathrm{out}}^{(M)} = \{\}$  for all  $i \in [n]$ . Then, the influence function used in Feldman and Zhang [5] is defined as follows.
$$
\phi_ {\mathrm {I F}} (x _ {i}, y _ {i}) := \frac {1}{| \mathcal {S} _ {i , \mathrm {i n}} ^ {(M)} |} \sum_ {S \in \mathcal {S} _ {i, \mathrm {i n}} ^ {(M)}} U (S) - \frac {1}{| \mathcal {S} _ {i , \mathrm {o u t}} ^ {(M)} |} \sum_ {S \in \mathcal {S} _ {i, \mathrm {o u t}} ^ {(M)}} U (S).
$$
In OpenDataVal, as suggested by Feldman and Zhang [5],  $M$  is chosen as  $\lfloor 0.7 \times n \rfloor$  where  $\lfloor x \rfloor$  is the biggest integer that is less than or equal to  $x$ .
4The authors in [17] show that  $(\alpha ,\beta) = (16,1)$  yields the best performance, but we empirically find  $(\alpha ,\beta) = (4,1)$  also yields competitive results in various datasets. In our settings,  $(\alpha ,\beta) = (4,1)$  often shows better results, and thus it is chosen as the default value.
5The authors in [15] proposed an efficient way to compute the influence function using Hessian-vector products, but it has been approximated with LOO.
LAVA. LAVA [13] proposed to measure how fast the optimal transport cost between a training dataset and a validation dataset changes when a training data point of interest is more weighted. This idea is formalized via the gradient of the optimal transport cost with respect to the probability mass, and in their paper, they showed that this gradient is expressed as follows.
$$
\phi_ {\mathrm {L A V A}} (x _ {i}, y _ {i}) := h _ {i} ^ {*} - \frac {1}{n - 1} \sum_ {j \in [ n ] \backslash \{i \}} h _ {j} ^ {*},
$$
where  $(h_1^*,\ldots,h_n^*)$  is a part of the optimal solution of the dual problem: for a validation dataset  $\{(\tilde{x}_i,\tilde{y}_i)\}_{i=1}^{n_{\mathrm{val}}}$ ,
$$
h ^ {*}, g ^ {*} := \operatorname {a r g m a x} _ {h, g \in C ^ {0} (\mathcal {X} \times \mathcal {Y})} \langle h, \frac {1}{n} \delta_ {(x _ {i}, y _ {i})} \rangle + \langle g, \frac {1}{n _ {\mathrm {v a l}}} \delta_ {(\tilde {x} _ {i}, \tilde {y} _ {i})} \rangle ,
$$
where  $C^0 (\mathcal{X}\times \mathcal{Y})$  is the set of all continuous function defined on  $\mathcal{X}\times \mathcal{Y}$ , and  $\delta_{(x,y)}$  is the Delta measure at  $(x,y)\in \mathcal{X}\times \mathcal{Y}$ .
A.3 Importance weight-based methods