Validation Free and Replication Robust Volume-based Data Valuation
Authors: Xinyi Xu
Zhaoxuan Wu
Chuan-Sheng Foo
Bryan Kian Hsiang Low
Venue: NeurIPS 2021 Poster (2021)
Keywords: Data Valuation
Replication Robustness
Linear Regression
Model-Agnostic
Task-Agnostic

Abstract: Data valuation arises as a non-trivial challenge in real-world use cases such as collaborative machine learning, federated learning, trusted data sharing, data marketplaces. The value of data is often associated with the learning performance (e.g., validation accuracy) of a model trained on the data, which introduces a close coupling between data valuation and validation.  However, a validation set may notbe available in practice and it can be challenging for the data providers to reach an agreement on the choice of the validation set. Another practical issue is that of data replication: Given the value of some data points, a dishonest data provider may replicate these data points to exploit the valuation for a larger reward/payment. We observe that the diversity of the data points is an inherent property of a dataset that is independent of validation. We formalize diversity via the volume of the data matrix (i.e., determinant of its left Gram), which allows us to establish a formal connection between the diversity of data and learning performance without requiring validation. Furthermore, we propose a robust volume measure with a theoretical guarantee on the replication robustness by following the intuition that copying the same data points does not increase the diversity of data.  We perform extensive experiments to demonstrate its consistency in valuation and practical advantages over existing baselines and show that our method is model- and task-agnostic and can be flexibly adapted to handle various neural networks.

Validation Free and Replication Robust Volume-based Data Valuation
Xinyi Xu Zhaoxuan Wu Chuan Sheng Foo Bryan Kian Hsiang Low Dept. of Computer Science, National University of Singapore, Republic of Singapore! Institute of Data Science, National University of Singapore, Republic of Singapore Integrative Sciences and Engineering Programme, NUSGS, Republic of Singapore 1 Institute for Infocomm Research, A * Republic of Singapore nus, edu.sg
wu.zhaoxuancu..om nus .us. Coo_hhunn_shenghe a-starr edu. sg"
Abstract
Data valuation arises as non-trivial challenge in real-world use cases such collaborative machine learning, federated learning, trusted data sharing, data mar- ketplaces. The value of data is often associated with the learning performance (e.g.. validation accuracy) of a model trained on the data, which introduces a close cou- pling between data valuation and validation. However, a validation set may not be available in practice and it can be challenging for the data providers to reach agreement the choice of the validation set. Another practical issue is that of data replication: Given the value some data points, dishonest data provider may replicate these data points to exploit the valuation for larger twwad//aymmet... We observe that the diversity of the data points is an inherent property that is independent of validation. We formalize diversity via the volume of the data matrix (i.e., determinant its left Gram), which allows us to establish a formal con- nection between the diversity of data and learning performance without requiring validation. Furthermore, we propose robust volume measure with a theoretical guarantee the replication robustness by following the intuition that copying the same data points does not increase the diversity of data. We perform extensive experiments to demonstrate its consistency in valuation and practical advantages over existing baselines and show that our method is model- and task task-agnostic and can be flexibly adapted to handle various neural networks.
dataset
Introduction
Data is increasingly recognized valuable resource [19].
need principled measure of its worth. A suitable data valuation has wide-ranging applications such as fairly compensating clinical trial researchers for their collected data [12 [6 25). fostering collaborative machine learning and
federated learning among industrial organizations
35 36, 39]. encouraging trusted data sharing and
building data marketplaces 7 30 32 37). among others.
popular viewpoint is that the value of data should correlate with the learning performance a model trained on the data [14] 18), which enforces close coupling between data valuation and validation. However, a validation set may not always be available in practice [35] Also, as different choices of the validation can lead to different data valuations, it is challenging for the data providers to agree on the choice of such a validation set 35 Since valuation is coupled with validation, if the
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

validation set is not sufficiently representative of the distribution of test queries in a learning task, the resulting valuation may not be as accurate/useful [40]. We adopt different perspective: The value of data should be related to its intrinsic properties and valuation can be decoupled from validation by considering the inherent diversity of the data. Intuitively, a more diverse collection of data points corresponds to a higher-quality dataset and thus yields a larger value. This perspective circumvents the above practical limitations and allows our valuation method to be model- and task-agnostic. We formalize diversity via the volume of the data matrix (i.e., determinant of its left Gram).
Data replication is another practical issue in data valuation due to the digital nature and anonymous setting of data marketplaces [[5]. Supposing a dataset has some value and data provider instead offers one containing two copies of every data point in this dataset, is this "new" dataset twice as valuable as the original one? Intuitively, the answer should be no as replication adds no new data and so does not increase diversity. We formalize this intuition by constructing a compressed version of the original data to assign little value to replicated data and still preserve its inherent diversity, hence guaranteeing replication robustness.
We provide theoretical justifications for formalizing diversity via volume: Firstly, diversity should be non-negative and monotonic [4 35 38 and volume satisfies both properties. Secondly, a greater diversity should lead to a better learning performance [23]: We formally show that a larger volume generally leads to a better performance using the ordinary least squares (OLS) framework and our method can be flexibly adapted to handle more complex machine learning models (i.e.. various neural networks) in our experiments. Specifically, data with a larger volume can lead more accurate pseudo-inverse (i.e., key component of the least squares solution) and smaller mean squared error.
To ensure replication robustness, we find that the marginal increase in value from replication must diminish to zero. Otherwise, a data provider can exploit this valuation by making infinite copies of the data to achieve infinite value. We thus formalize the notion of replication robustness via the asymptotic value attainable through replication. Unfortunately, the conventional definition of volume does not have this property. So, we propose robust volume (RV) measure by constructing compressed version of the original data that groups similar data via discretized cubes of the input feature space and represents those in each cube via a statistic. The RV measure offers practitioners the flexibility to trade off between diversity representation and replication robustness via the cube's s width. We perform extensive experiments on synthetic and real-world datasets to demonstrate that our method produces consistent valuations with existing methods while making fewer assumptions.
The specific contributions of our work here include:
Formalizing a measure of data diversity via the volume of data (Sec. 2 and justifying the suitability of volume for data valuation both theoretically (Sec. 3 and empirically (Sec.55):
Formalizing the notion of replication robustness and designing a data valuation method based on the robust volume (RV) measure with a theoretical guarantee on replication robustness (Sec. 4 Performing extensive empirical comparisons with baselines to demonstrate that our method is consistent in valuation without validation, replication robust, and can be flexibly adapted to handle complex machine learning models such as various neural networks (Sec.
2 Problem Setting and Notations
Consider two data submatrices Xs and X to be valued that contain s and s' rows of d-dimensional input feature vectors, respectively. Let P. [ 0 E Raxd be the zero-padded version of Xs Rex We concatenate the data submatrices along the rows form the full data matrix X E Ba i.c., X= = X X and n = s + s'. We denote the corresponding observed labels/responses as y == 1y y E R The least squares solution from OLS is w := =x+
argming lly - X where X = (X (x T is the pseudo-inverse of X. Similarly, we denote X as the pseudo-inverse of Xs and w = x x s. To ease notations, let V = Vol(X) and V/g Vol(Xs) where Vol() is defined below. Let |A| denote the determinant of a square matrix A. The
left Gram matrix of X is G = x E dRd so for data submatrix Xss G5 == X X E Rdddd
Definition 1 (Volume). For full- X gaxd with d. Vol(X) =
VIG.
We adopt the above definition of volume for several reasons: (a) Often, the input feature space of the data is pre-determined and fixed due to the data collection process. But, new data can stream in and

---

so, n can grow indefinitely while d remains fixed 10]. (b) By leveraging the formal connection between volume and learning performance (Sec. B we can design a validation free volume-based data valuation to assign larger value to data leading to a better learning performance. (c) This affords an intuitive interpretation between volume and diversity: Adding a data point to a dataset can increase the diversity/volume depending on the data points already in the dataset (Lemma D.
We restrict our discussion full-rank matrices X, Xs, and X5r since otherwise we can adopt the Gram-Schmide process to remove the linearly dependent columns ()] [O]. In practice, we perform pre pre-processing such as principal component analysis to reduce the dimension of the input feature space to ensure that this assumption is satisfied. This assumption is to ensure that there are no redundant features, namely, features that can be exactly reconstructed using other features. For instance, if a dataset already contains monthly salaries, then an annual salary would be redundant.
3 Larger Volume Entails Better Learning Performance
The value of data (sub)matrix depends on the learning performance trained on it [14] 18] which, will show, depends on its volume. Simply put, the larger the volume, the better the learning performance. In this section, we will formalize this claim through the ordinary least squares (OLS) framework. In particular, we will investigate two metries for learning performance: (a) the quality
of the pseudo-inverse represented by biass == | X+ because estimating X accurately is
important to achieving small mean squared error (MSE) 9 and where P = (X x3) and (b) the MSE denoted as L(Ws) ly Xw
3.1 Larger Volume Entails Smaller Bias
In regression problems, the closed-form optimal solution is constructed via X+ computed using X. So, the bias of 2 from X indirectly determines the value of Xs 9 i.e., smaller bias means larger value. We show in Proposition beloo that 'a larger volume means a smaller bias' always holds for d For d 1, requires additional assumptions which are mostly satisfied via empirical verification (Fig. [
Proposition 1 (Volume vs. Bias for d = 1). For non-zero Xs, X of X E 1ux1 V8 2 Vs biass- biasg < O.
The above result can be generalized to M > non-zero data submatrices: Let X [X], X X s., and w.l.o.g.. suppose that Vs. 2 Vs, > Vss Then, biass; < biass, < < biassa. For d> > 1, counterexamples exist (see Fig. so we instead compare bias
and bias in the next result:
Proposition 2 (Volume Bias in General). For full-rank Xs, Xs of E ERX
bias.- - -bias
1 V
|
1
Q +2
1 12
QX 1
Ps 1 VS
Q
V
IT= M1I,, (el denotes the k unique eigen- values of the left Gram matrix G of X, Qs are similarly defined w.r.t Gs,Gs, Ps and Ps are, respectively, zero-padded versions of Xs and Xs. and ai (-1+(- A (Thee An
where Q = k (ho)
The proof of Proposition T (Appendix A.I relies on a key observation that for d = 1, the left Gram matrix is number and the rest of the proof follows. However, it cannot be generalized to that for d > 1, so we resort to a different proof technique. The proof of 2 Appendix A.I which establishes formal connection between volume and G using the Sylvester's formula. To obtain Vs 2 Vs = biass < biass,, there are two cases requiring different additional assumptions: (A) Vs > Vs. and (B) Q
Lemma i
X S and V > max(Vs, Vs:). Case is intuitive: Vs > Vs: means X is much "larger" in volume than Xsr, so bias is smaller. Case when X5 and Xs are similar (e.g., when they are sampled from the same data distribution). The intuition is that the first difference term will be relatively large in magnitude (so, its sign will dominate the overall expression), while the second inner product term will be relatively small in magnitude. This is because the first difference term involves 1/V3 and 1/V3, v. but the second inner
3

---

product term involves 1/(V2 x V and 1/(V² x V and we show V > max(Vs, Vs.)) (Lemma 3
in Appendix A.1 Subsequently, Q ~ x and Vs 2 Vs. suggest that the first
difference term (and thus the overall expression) likely negative. We empirically verify in Fig.
that Vs 2 Vsa =
biass I b holds for more than 80% of times.
3.2 Larger Volume Entails Smaller MSE
In Proposition 3 (see proof in Appendix [A.2) below, we will show a similar result (to Proposition I theoretically analyzing the connection between volume and MSE when d 1, which may be surprising since Vol() (Definition D does not consider y at all and can yet determine which data submatrix offers better predictions on the rest of the (unobserved) data. Unfortunately, such result
does not directly generalize to d > 1 or beyond two submatrices. Nevertheless, we will analyze the effect of volume on the learning performance (i.e., MSE) in general.
Proposition 3 (Volume MSE for d = 1). For non-zero Xs, X of X E Rax! V3 2 V3 L(ws) - L(Ws)) I 0.
Unfortunately, the above result does not generalize to d > 1. For full-rank Xs, X of X Raxd have derived in Appendix A.2 that
L(ws) - L( (wws) = (ws - W (X X + X + ) 2X y)
and also shown in Appendix A.2 that since L(ws) - L(W ) explicitly depends on y [ and Vol() does not include y all, it is possible to adversarially construct y s.t. L(ws) - L(ws:) 0 or L(ws) - L(ws) < 0 for some fixed Xs, Xsr.
The adversarial cases notwithstanding. volume is regarded as a good surrogate measure of the quality of data applied to active learning and matrix subsampling with theoretical performance guarantees [T (28). Similarly, we can adopt the perspective that Vol() is measure of the diversity in the input features 23 which provides an intuitive interpretation for Proposition B more diverse dataset with a larger volume gives a better learning performance (i.e., smaller MSE). We will show in Sec. (52 not requiring labels/responses can be an advantage in practice if the labels/responses are noisy/corrupted or there is distributional difference between the validation and test sets.
We conclude Sec. 3 by empirically verifying whether the additional assumptions described in the last paragraph of Sec. 3.1 are satisfied by checking the percentage of times that Vs 2 Vs! = biasg biasg I holds. To elaborate, we randomly and identically sample equal-sized Xg, Xss over 500 independent trials and compute the percentage of times that larger volume leads to better learning performance (vertical axis) against the size of Xs, Xs. (horizontal axis). We consider sampling X X from either uniform or normal distribution of varying dimensions: In Fig. [ for example, *N d
denotes Xs; Xsr being sampled from 1-dimensional standard normal distribution. For MSE, the response y of a data point x is calculated from y = sin((w x)) where the true parameters w* are randomly sampled from (0, 2)d. Fig. (left) shows that a larger volume leads to a smaller bias for more than 80% of times, thus verifying that the additional assumptions in Sec. [.re aree satisfied. Fig. (right) shows that larger volume leads to a smaller MSE for more than 50% times for d < 10, which consistent with the above implications from
Pertentage of ttimes
Volume vs. Bias
1.0
Volume vs. MSE
0.90
0.8
0.60
0.85
Percentage
0.6
0.75
.d-1
Nd-2
Wd-5
W d-10
05
W0-1
W0-2
X'd55
N d-10
0.70
Udd1
Ud-2
Udd5
U d-10
0.3
Ud-1
U
U'de5
U--10
0.65
200
300
400
300
100
230
300
500
Size of Xs.
X5
Size of X5. X5:
Figure 1: Volume vs. bias (left) and volume vs. MSE (right) for both identically sampled, equal- datasets Xs, Xs from either a uniform (0, 1)d or normal (0, 1)d distribution. The vertical axis shows the percentage of times over 500 independent trials that the dataset with a larger volume leads to better learning performance (i.c., smaller bias or MSE).

---

Robustifying Volume-based Data Valuation
As larger volume can entail a better learning performance (Sec. 3.. we consider a volume- based data valuation method. Unfortunately, volume (Definition I is not robust to replication via direct data copying. Hence, we will introduce a modified volume measure that can trade off a more refined representation of diversity for greater robustness replication.
4.1 First Attempt of Volume-based Data Valuation
Directly using Vol(X) a valuation of X satisfies both non -egativvt and monotonicity which follow directly from Definition lnn the matrix determinant lemma, respectively:
Proposition 4 (Non-negativity and Monotonicity of Vol(). For full-rank X E RRax,, Vol(X) 2 0 and Vol([ x x]]) 2 Vol(X)
where x E 11xd is new data point.
The properties of Vol( in Proposition 4 iimply that bigger-sized X (i.e., more data) should yield
larger value [14 18 35 However, Vol() is unbounded and has multiplicative scaling factor w.r.t. replication. The implication is that a data provider can arbitrarily "inflate" the volume or value of data by replicating the data infinitely, as shown in the following result (see proof in Appendix (A.3)
Lemma (Unbounded Multiplicative Scaling of Vol(X) from Replication). For full-rank X Raxd, let Xx E 1xxd be data point replicated for 2 1 times and Xxep = [X" xg x 1T
R((+m)xx xd Then, Vol(X.ep) = Vol(X) (1 + m x x, X x) J
Replication robustness defined via inflation. We define measure of inflation as the ratio v(replicate X, (X) where v() is a data valuation function (e.g., Vol()) mapping data matrix to real value, the function replicate X; c) directly copies the data in and appends them back to X to output Xee E (nc)xxd, and the replication factor e denotes the amount of replication. One way of replication is to copy the entire X for e times. Another way is to copy some data submatrix for a certain number of times s.t. X E R(n))xd, We consider the second way because replicating different data increases the value differently (Lemma I.. We define below a measure of replication robustness to formalize the intuition that greater robustness should guarantee smaller inflation:
Definition 2 (Replication Robustness of Data Valuation v()). Define replication robustness of v() as % == (X) /SuPp22 v(Xxe)) where X = replicate (X c) E [R(
The theoretically optimal robustness is % = 1, which implies no additional gain from replicating data, hence discouraging replication completely. In contrast, the worst-case robustness % which is the case for any v() that strictly monotonically increases with replication and, in particular, NVal 0 by applying Lemmal] [ As result, replication robust data valuation function must have diminishing marginal value from replication: The additional gain from having more copies of the same data converges asymptotically to w.n.t. c. This aligns with what we observe in practice: Repeatedly adding the same data to training set does not improve the learning performance indefinitely.
4.2 Replication Robust Volume (RV)-Based Data Valuation
We will propose an RV measure by constructing compressed version of original data matrix X that groups similar data points via discretized cubes of the input feature space and represents those in each cube via statistic. The RV measure offers practitioners the flexibility to trade off a more refined diversity representation for greater replication robustness by increasing the cube's s width.
Definition (Replication Robust Volume (RV)). Let the d-dimensional input feature space/domain for X be discretized into set Te of d-cubes of width//isccettizatein coefficient w, &; denote the number of data points in d-cube E L μ4 E d-cube i, and X = [n TTe #0 be compressed version of X s.I. each row of X is statistic p.; of the data points in non-empty d-cube i. The replication robust volume is
xd be a statistic (e.g., mean vector) of the data points in
RV(X; w) = Vol(X) x II
(2)
where Pi := - a with hyperparameter [0, 1] controlling the degree of robustness.
5

---

In contrast to the unbounded Vol(), we ensure that (V is bounded by setting Ih Pi to be bounded and convergent w.r.t. the size of the replicated data. Note that o; = 0 = P: = (i.e.. an empty d-cube) and de > 0
P; >> 1. Before considering any robustness guarantee, we will first show in Proposition (see proof in Appendix AA.) below that RV (Definition 3 preserves the
original volume in a relative sense, i.e., the ratio V s srr is preserved. The implication is a similar effect of RV on the learning performance (Sec. 3). as empirically demonstrated in Sec. 5.1
Proposition 5 (Bounded Distortion of RV (V(X::w) (V(Xs;:))) Define distortion 8(w)
[RV(X s: w)/ RV(X sy: w))[(Vol(Xs)/ Vol( xx)) ]] Then, (exp(8- S 8(w) S exp(3- for any w > 0 where 3 (/(o)). For example, 3 10 bounds 8(w) E [0.905, 1.105] approximately.
Near-optimal robustness by upper-bounding inflation. We have previously defined robust- ness (Definition 2 as the maximum attainable inflation via replication. Since Pi and inflation are monotonic in Oi we consider the asymptotic inflation: &i - oC. In Definition [3] even when the data in d-cube is replicated infinitely many times, the inflation from this d-cube is still upper- beunddboudded by a constant. This can be generalized to all the d-cubes as each can be considered independently and there is constant number of d-cubes for fixed X and w.
Proposition 6 (Robustness 'RR)). For a E [0, TRV 2 (1- a)ll where, with slight abuse of notation, V denotes the set of non-empty d-cubes. For a = 1, TnV O.
Its proof is in Appendix A.3 Recall from Definition 2 that TRV = 1 is optimal robustness. From Proposition 6 reducing achieves a smaller upper bound on inflation and greater robustness. However, if a is too small, then it may have an undesirable effect: RV(X;w) w) < Vol(X) for some X (with similar data points) from an honest provider without replication. In this case, RV has an over-correcting effect: RV is designed to avoid exploitation of Vol() due to replication but mistakenly leads to a decrease in the value of an honest dataset. Therefore, a should be set to achieve a certain upper bound on inflation but should not be unnecessarily small; more details are given in Proposition g in Appendix A.3 In particular, setting a = 15(nn) guarantees a constant upper bound cxp(8 on the inflation, as proven in Lemma
5 in Appendix A.33 For instance, setting 3 10 and a = 1/(3n) guarantees RV ireelicate(X,e)) c); w) S 110% x RV WV):: However, it requires us to know the true n without any replication. In practice, as we can only observe the data with replication (if any) [15] we
estimate n with the number | of rows in X.
Trading off diversity representation for replication robustness via w. A smaller w means that the d-cubes are more refined and RV can better represent the original data instead of crudely grouping many data points together and representing them via a statistic. On the other hand, larger w means a less refined diversity representation but greater replication robustness. In the extreme case, sufficiently large w results in grouping all data points together and representing them all using a single statistic, hence foregoing the diversity in data. So, practitioner should determine the trade-off between diversity representation vs. replication robustness based on the requirements of the world use case. The following result (see proof in Appendix A.3) formalizes both extremes of the trade-off:
Proposition 7 (Reduction to Vol() vs. Optimal Robustness). Set w to be s.I. each d-cube only contains completely identical data points, and
1. set pt to some constant Kx for i V based on a recursive application of Lemma ! Then, RV(::w) = Vol();
2. set = 0. So. P = 1(p. + 0) and name formulation RV RV1(=;). Then, TRV, = 1.
RV, (:sw) can be seen as reducing all potential replications to one data point. It achieves robustness but loses the density information of each d-cube due to the indicator function. Specifically, the true distribution may have different densities at different d-cubes, which is reflected via Φ, 's. But, this information is completely lost in RV (+w). In contrast, Vol() represents all the data indiscriminately, hence sacrificing robustness. Furthermore, while we restrict our consideration of replication to direct copying, it is natural to additionally consider noisy replication (i.e., adding small random perturbations to copies [5]). Intuitively, RV
RV1(;; (+:w) is not robust to noisy replication as the replicated data are perturbed. Our preliminary empirical study in Appendices B.2 B.2 and B.3 shows that RV is robust to noisy replication if the noise magnitude is small relative to w. So, a future work is to devise a way to optimize the trade-off between diversity representation and replication robustness via w. In our work here, we empirically find w 0.1 suitable for the case of standardized input features.

---

Efrect Train Loss
Effec: on Test Loss
Effect on Train Less
Effect on Test Loss
Cledit Card
0.0
High RV
0.8
Lops
Random
0.4
0.50
Uber lyft
Loss
0.45
0.4
0.40
0.35
0.35
terrovdd
Figure 2: Effect of removing/adding dataset with highest/lowest RV on train/test loss for real-world credit card and Uber Lyft datasets. Plots show the average and standard errors over 50 random trials.
In using standardized input features, we implicitly assume that the input features follow normal distribution. This makes the data further away from the mean (i.e., statistically rarer) more valuable in learning [[]. We also observe this in Sec. 5.2 where data closer to the mean are valued to be smaller across all baselines and our method. Our work here excludes considerations of outliers as they are not truly representative of the true data distribution.
Experiments and Discussion
In this section, we will first verify our claim in Sec. 3 that larger volume leads to a better learning performance and reveal some interesting practical perspectives in Sec. 5.1 Then, in Sec. we will show that RV produces results consistent with existing baseline methods and also demonstrate the limitations these baselines. In particular, RV model- and task-agnostic while another baseline with an explicit dependence on the validation set is shown have some deviation in data valuation as the validation set changes. Lastly, in Sec. (.3] we will verify our robustness guarantee by analyzing its asymptotic behavior under replication. Importantly.
empirical study has gone beyond the OLS framework used for the theoretical analysis in Sec. 3 as our method can be flexibly adapted to handle various neural network architectures different machine learning tasks including both image classification and natural language processing. All experiments have been run on a server with Int) Xeon(R)@ @ 2. 2.70GHz processor and 256GB RAM. Our code is publicly available at: https:// thub. com /ttp///aawxaa//uuaaaaaaa wwtpl////oolleeeeee SalmeeBasdd-Dtttaalllnn
5.1 Effect of Robust Volume (RV) on Learning Performance
In this subsection, use RV and volume interchangeably replication is not considered here and Proposition 5 guarantees that RV preserves the original volume. We consider the setting of sequentially addin//removing the dataset with highest/loeest RV to analyze the effect of RV on the learning performance [14]. We include random selection as a baseline. We simulate 8 data providers to make the results more generalizable. In this experiment,
two wellworld datasets: credit card fraud detection (2 (i.e., transaction amount prediction) and Uber & Lyft [5] (i.e., carpool ride price prediction) which pre-processed to contain 8 and 12 standardized input features, respectively. Fig. 2 shows the results. Additional results on two other real-world datasets are in Appendix B.4
It can be observed that adding (resp., removing) a dataset with larger RV leads to a smaller (resp., larger) train loss, thus verifying Proposition]] 2 that larger volume leads to a more accurate pseudo- inverse and smaller train loss in terms of mean squared error. This observation is also consistent with the results on the test loss, albeit with larger standard errors. This confirms (I) that in a higher dimensional input feature space, larger volume does not immediately guarantee a smaller test loss.
Interesting practical perspectives. The results on adding datasets provide justification for data buyer with a limited budget to spend on datasets with larger RVs first to achieve the best learning performance, thereby resonating with the active learning paradigm 229). On the other hand, the results on removing datasets sheds light on the following question: If training on all collected datasets too costly due to memory or time constraints, then which dataset should removed first without compromising the learning performance much (i.c., the dataset with smallest RV)?
7

---

5.2 Empirical Comparison of Robust Volume (RV) Shapley Value with Baselines
We will demonstrate that RV without validation gives results consistent with existing baseline methods which may require validation. Then, we will empirically show the limitations of these baselines.
To design principled, fair payments to the data providers, we use (robust) volume as the characteristic function in the commonly used Shapley value to measure the expected marginal contributions of their Our robust volume Shapley value (RVSV) is defined follows 33:
datasets [14 18 35
RVSV,, m (1/M!) EccMi(s..) [[C]] (M- - -CC| 1)!] x [RV (Xcu(s...i) RV RV(X::)) (3)
where M = ( (S.....: Sas) denotes set of M data providers/datasets and X denotes data matrix constructed from concatenating the data matrix Xs... of every data provider Sm E C C M. Our
volume Shapley value (VSV) is computed by replacing RV (::w in (3) with Vol(). We compare
VSV and RVSV with the following baselines: validation loss leave-one-oul (L.OO) value
validation loss Shapley value (VL.SV [14 18] and information gain Shapley value (IGSV) 335 We
consider the contributions of M = 3 data roviides//matrccss/aaaaae Xs.. Xs and Xs, (35] The
input features are standardized and we set w 0.1. LOO and VLSV use MSE on a validation set.
Synthetic data from baseline distributions. We first consider simpler experimental settings on synthetic data drawn from the 6D Hartmann function 22 defined over [0, 1]6 with four baseline data distributions for Xs. xs, and Xs,; : (A) independent and identical distribution (i.id.) where Xs,, Xs., and Xs, contain 200 i.i.d. samples each; (B) ascending dataset size where Xs, Xs2. and Xs, contain 20, 50, and 200 i.i.d. samples, respectively: (C) disjoint input domains where Xs, Xs,, and Xs, are sampled from the input domains of [0, 1/3 [1/3, 2/3 and [2/3, 1]6, respectively; and (D) supersets Xs, C xx,, Xs, with the respective sizes 200, 400, and 600 where Xs, (resp., Xs,) has 200 i.idd data samples in addition Xs, (resp.. Xs.).
The results in Fig. 3 show that both VSV and RVSV are generally consistent with IGSV. For (B) ascending dataset size, VSV, RVSV, and IGSV increase from Xs, to Xss, while VLSV surprisingly values the contributions of Xs,, Xs,, and Xss, to be nearly equal; the latter may be due to VLSV's sensitivity to the definition of the value v(0) of empty dataset/matrix 0 when calculating the Shapley value. Fig. 4 illustrates that for i.i.d., VLSV is sensitive to the definition of v(0:: For example, setting v00 to O 188 1.06 (by initializing parameters zeros), and 8.75 (by initializing parameters randomly using (O. 1) 14) yield different VISVVs of 0.346, 0.183, and 0.330 for Xs respectively. These conflicting choices of (0) add to the difficulties of applying VLSV in practice.
Interestingly, under (C) disjoint input domains, all methods unanimously value the contribution of Xss to be the lowest despite their input domains to be of the same size, which is due to the standardization of the input features and so offers the following interpretation: The data in the "center" is the most common if we assume the true data distribution follows a normal one. Therefore, the most common data are valued less while the statistically "rarer" data at the two tails of the distribution are valued more. Additional experimental results with this distribution are reported in Appendix [B.5] It is counter-intuitive to see that for i.i.d,, LOO values the contribution of Xs, to be O, which may be due to instability from the calculation of their contributions {
Real-world datasets with different preferences of validation sets. We use two real-world datasets: UK used car dataset ①1] (i.e., car price prediction) and credit card fraud detection dataset [2] (i.e., transaction amount prediction) where there are different preferences of validation sets [35]. For instance, car dealers for different manufacturers such as Audi, Ford, and Toyota may have different preferences over data. So, we construct two different validation sets comprising cars from different manufacturers. Similarly, different financial institutions may differ in their interests of the transaction amounts. For example, smaller banks typically manage and focus on smaller transaction amounts, so we construct two different validation sets comprising large (i.e., $1000) vs. small transaction amounts. The results in 5 show that the effect of different preferences of validation sets on LOO is pronounced, as expected. The effect on VLSVV is less due to the averaging of marginal contributions. On the other hand, there is no effect on IGSV, VSV, and RVSV as they do not require a validation set.
5.3 Replication Robustness
We first perform a simpler experiment to demonstrate the effect of replication and then perform more extensive experiments under more complex settings to show the asymptotic behavior of RVSV and existing baseline methods under replication.
8

---

1.00
Size
Dijeint
Supersets
3.00
Contibubog
0.75
0.75
0.50
0.58
0.25
0.25
0.00
0.00
Dataset index
Calaset incee
Daliset index
Dataset index
Figure 3:
Contributions of X X and Xs from Hartmann function with baseline data distribu- tions: (A) ii.d., (B) ascending dataset size, (C) disjoint input domains, and (D) supersets.
Xs:
Xs.
Used Car
Credit Card
0.4
1.00
VISV
Contribuzion
0.75
X-
X
0.50
0.00
u(D) of empty datasat
VLSV
IGSV
VSV
RVSV
VSV RVSV
Figure Sensitivity of VLSV to varying Figure 5: Contributions of Xs, S and Xs, for vali- v(0 (e.g., 3 red dotted lines).
dation distinguished by darker lighter shades.
Contributions of Xs, X and Xs, under i..d setting. We perform this experiment on the Trip Advisor hotel reviews dataset 4 (i.e., numerical rating prediction) which contains text reviews data. We utilize the GloVe (31] word embeddings and bidirectional long short-term memory model with a fully-connected layer of hidden units. Regression is performed over the 8-dimensional latent features from this model. Data matrices Xs, S and Sg follow an i.i.d. partition of the processed data and subsequently, Xs, and Xs, are replicated for and 10 times, respectively. The results in Fig. [6] show noticeable increases the contribution X.s, for IGSV and VSV, which implies that they are not replication robust. On the other hand, both VLSV and RVSV appear robust.
Contributions of Xs, Xs2, and Xs, under non-i.... settings. As includes sup, (Definition 2). we investigate large replication factors c of up to 100. Since the previous experiment shows that VLSV is robust, use it as the baseline for comparison. We additionally consider two non-i.i.d. data distributions extended from the previous setting: supersets and disjoint input domains for 4 real-world datasets: California housing price prediction (CaliH) 20 Kings county housing sales prediction (KingH) (3), US census income prediction (USCensus) [6] and age estimation from facial images (FaceA) [II]. We use 60% of data construct Xss, Xs., and X.s, and the remaining 40% as the validation set for LOO and VLSV. For i.i.d. and supersets, we Xs, Xs, s.t. Xs, simulates an honest data provider and we examine the effect of replicating Xs, For supersets, we vary the proportion of data from Xs, that is contained in Xs, If the ratio 0.1, then Xss, contains 10% data from Xs, if the ratio 1, then Xs, C Xs.- For disjoint input domains, we vary how disjoint they are for Xs. Xs., and Xs, via a ratio: 0 (resp., 1) means that Xs., Xs,, and Xs, have completely disjoint (resp., overlapped) input domains. In other words, with ratio 0, they do not contain any similar data, while with ratio 1, they may contain some similar data. Fig. shows results for two datasets with i.i.d. data distribution. For CaliH, we use the latent features from the last layer of neural network with 2 fully connected layers of 64 and 10 hidden units and
replication robustness
1.00
Hotel Reviews
Contribution of Xs.
Velluaion vs. Replication
Valuation Replication
you
Contribution
0.75
Xs,
2.7
2.8
Contribution of e B
0.50
Xs
Xs.
0.00
VLSV
Replication factor factor
Replication factor
Figure 6: Effect of replication on contribu- tions of Xs,, Xs,. Xs,- Darker (lighter) shade denotes before (after) replication.
Figure 7: Contribution of the replicated Xs, with varying replication factors for CaliH (left) and
FaceA (right) datasets.

---

the rectified linear wnit (ReLU) as the activation function. Additional details on data distributions, datasets, and models are in Appendix [B.6]
Next, we compare the similarity of RVSV and baseline methods to VLSV using similarity measures such as the Pearson correlation coefficient (rp) [18, cosine similarity (cos), and the reciprocal of the le norm of the difference [36]. For RVSV, we set w 0.05 and 0.1, which are respectively denoted by RVSV-005 and RVSV-01. Table [ shows results averaged over varying replication factors e for CaliH; the other results are reported in Appendix [.6 VSV and IGSV are not robust and may be exploited as both increase relatively quickly with replication for < 20 (Fig. 7. Furthermore, our additional experiments on varying hyperparameter choices (Appendix B.7 show that IGSV is sensitive to the choice of hyperparameter whereas RVSV consistent, even with varying w. From Fig. 7 RVSV is replication robust. RVSV can also achieve a high degree of similarity to VLSV without requiring validation, as seen in Table T
Table 1: Effect of replication on similarity of RVSV and existing baseline methods to VLSV for CaliH dataset. Values in bold indicate the best results.
Method
k.d.
1/11
disjoint 0
cos
1/4
0g
disjoint Cos
1,12
Tp
supersets 0.1
Cos
1/12
7p
supersets
COS
1/12
LO0
-0.991
0.730
1.894
-0.459
0.816
2,457
-0.488
0.406
0.770
-0.339
018001
2.362
-0.590
0.771
2,100
IGSV
-0.9003
0.637
1.591
0.640
0.639
1.583
-0.763
0.666
1,589
-0.893
0.636
1,580
-0.716
0.653
1.687
0.787
2.493
0.614
-0.780
2.335
-0.892
2.389
0.813
2.696
RVSV-005
0.767
0.959
5.857
0.700
1.000
77.714
-0.784
0,998
28.479
0.810
0.983
9.314
0.918
0.946
5.051
RVSV-01
0.767
0.920
4.055
0.351
0.999
47.066
-0.939
0.997
20.845
0.808
0,976
7,839
0.917
0.914
3,901
6 Related Work
Data valuation methods assign larger value to data that leads to better learning performance [4.
18
Existing methods such as leave-one-out approaches [18. the Shapley value-based methods 14 17], and a reinforcement learning framework [40] require validation. Due to the tight coupling between valuation and validation, these methods may face practical limitations arising from using a validation set (Sec. D. The work of [35] has proposed an information -ehoorrttcttchccceee approach to valuing data based on the information gain (IG) the model parameters avoid the need for validation. However, it has not proven that a larger IG (value) leads to a better learning performance. Our method has this desirable theoretical property without needing validation (Sec.B) While existing methods demonstrate some effectiveness against replication using carefully selected validation sets [4 18) our method achieves such a guarantee without needing validation. The work of [15] has considered replication from a different perspective and is thus not directly comparable to our method.
35
40
7 Conclusion and Future Work
This paper describes a model- and task-agnostic replication robust data valuation method that requires no validation. In particular, we value data based on its inherent diversity formalized as the volume of the data matrix because we have shown in Sec. 3 that larger volume entails a better learning performance. We have identified that volume is not robust to replication, so we design data valuation method based on the novel robust volume (RV) measure with a theoretical guarantee on replication robustness (Sec. 4). In our experiments (Sec. [5), we have used RV as a characteristic function in the Shapley value and empirical comparison with existing baseline methods verifies its effectivenesss in data valuation and its robustness guarantee. Importantly, we have tested various real-world datasets and our robust volume data valuation method can be flexibly adapted to handle machine learning models more complex than OLS (i.e., various neural networks) to demonstrate its practical applicability. Current works on data pricing may build on our perspective to ease the dependence on the validation set. For future work, we plan to consider more sophisticated replication techniques and investigate how to optimize the trade-off between diversity representation vs. replication robustness.
Acknowledgments and Disclosure of Funding
This research supported by the National Research Foundation, Singapore under its Al Singapore Programme (Award No: LISG2-RP-2020-0 Any opinions, findings and conclusions recom-
10

---

mendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. Xinyi Xu is supported by the Institute for Infocomm Research of Agency for Science, Technology and Research (A*STAR). The authors thank Fusheng Liu for many interesting discussions.
References
[1] 100,000 UK used car dataset. URL used car -ar-datastt-foor and and-mercedes
http::/www..leggl....c///ddaaaaa
[2] Credit card fraud detection. URL
[3] House sales King County, USA. housesalesppedictiion
URL
https///www.kggll..cc/////a/al/
[4] Trip Advisor hotel reviews. hhap--adviaor-hhtee-reieeee
URL
attp:://ww..kagglle.ccm/adaddeee
[5] Uber and Lyft dataset Boston, MA.
URL
http:://ww..eegglee.c/////l////
uber-an and- lyft -add-lyf-datast--bbss haataset-bsston-te
[6] US census demographic data.
URL
us www.cosuddddoogrpiiiiccooooooooo
[7] A. Agarwal, M. Dahleh, and T. Sarkar. A marketplace for data: An algorithmic solution. In Proc. ACM EC, pages 701-726, 2019.
[8] S. Basu, P. Pope, and S. Feizi. Influence functions in deep learning are fragile. In Proc. ICLR, 2021.
(9] M. Dereziáski and M. K. Warmuth. Unbiased estimates for linear regression via volume sampling. In Proc. NeurlPS, pages 3087-3096, 2017.
[10] M. Dereziáski and K. Warmuth. Reverse iterative volume sampling for linear regression. Journal of Machine Learning Research, 19(23):1-39, 2018.
[11] M. Dereziáski. M. K. Warmuth, and D. Hsu. Leveraged volume sampling for linear regression. In Proc. NeurlPS, pages 25 2510-2519, 2018.
[12] P. Devereaux, G. Guyatt, H. Gerstein. S. Connolly, and S. Yusuf. Toward faimess in data sharing. New England Journal of Medicine, 375(5):405-40) 2016.
[13] J. H. Friedman. Multivariate adaptive regression splines. Annals of Statistics. 19(1): 1991.
[14] A. Ghorbani and J. Zou. Data Shapley: Equitable valuation of data for machine learning. In Proc. ICML, pages 2242-2251, 2019.
[15] D. Han, M. Wooldridge, A. Rogers, S. Tople, O. Ohrimenko, and S. Tschiatschek. Replication- robust payoff-allocation for machine learning data markets. arXiv:2006. 01.220066.15588 14583, 2021.
[16] C. A. Jackevicius, J. An, D. T. Ko, J. S. Ross, S. Angraal, J. D. Wallach, M. Koh, J. Song, and H. M. Krumholz. Submissions from the sprint data analysis challenge on clinical risk prediction:
A cross-sectional evaluation. BMJ Open, 9(3), 2019.
[17] R. Jia, Dao, B. Wang. F. A. Hubis, M. Gurel, B. Li, C. Zhang, Spanos, and D. Song. Ef- ficient task-specific data valuation for nearest neighbor algorithms. In Proc. VLDB Endowment, pages 1610-1623, 2019.
[18] R. Jia, D. Dao, B. Wang, F. A. Hubis, N. Hynes, N. M. Gurel, B. Li, C. Zhang, D. Song, and C. Spanos. Towards efficient data valuation based the Shapley value. In Proc. AISTATS, pages 1167-1176, 2019.
[19] S. Jossen. The world's most valuable resource is longer oil, but data. The Economist, 2017.

---

[20] R. Kelley Pace and R. Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3):291-297, 1997.
[21] P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In Proc. ICML, pages 1885-1894, 2017.
[22] A. Krause, A. Singh, and C. Guestrin. Near-optimal sensor placements in Gaussian processes. Theory, efficient algorithms and empirical studies. Journal of Machine Learning Research, 988):235-28 284, 2008.
[23] A. Kulesza and B. Taskar. Determinantal point processes for machine learning. Foundations and Trends in Machine Learning, 5(2-3):123-286, 2012.
[24] D. J. Lizotte. Practical Bayesian optimization. PhD thesis, University of Alberta, Canada, 2008.
[25] B. Lo and D. L. DeMets. Incentives for clinical trialists share data, New England Journal of Medicine, (1:5122:1112-1111 2016.
[26] L. Lovász. Submodular functions and convexity. In A. Bachem, B. Korte, and M. Grötschel, editors, Mathematical Programming: The State of the Art Bonn 1982, pages 235-257. Springer, Berlin, Heidelberg, 1983.
[27] L. Lyu, X. Xu, Q. Wang, and H. Yu. Collaborative fairness in federated learning. In Q. Yang, L. Fan, and H. Yu, editors, Federated Learning: Privacy and Incentive, Lecture Notes in Computer Science, pages 189-204. Springer International Publishing. Cham, 2020.
[28] A. Mikhalev and V. Oseledets. Rectangular maximum-volume submatrices and their applica- tions. Linear Algebra and its Applications, 538:187-211, 2018.
[29] C. Orhan and Ö. Taştan. ALEVS: Active learning by statistical leverage sampling. In Proc. ICML Workshop on Active Learning, 2015.
[30] J. Pei. A survey on data pricing: From economics to data science. IEEE TKDE, 2020.
[31]
Pennington, R. Socher, and C. D. Manning. GloVe: Global vectors for word representation. In Proc. EMNLP, pages 1532-1543, 2014.
[32] R. Raskar, P. Vepakomma, T. Swedish, and A. Sharan. Data markets to support Al for all: Pricing, valuation and governance. arXiv:1905.0668 2019.
[33] L. S. Shapley, A value for n-person games. In H. W. Kuhn and A. W. Tucker, editors, Contributions to the Theory of Games, volume 2, pages 307-317. Princeton Univ. Press, 1953.
[34] M. C. Shewry and H. P. Wynn. Maximum entropy sampling. Journal of Applied Statistics, 14(2):165-170, 1987.
[35] R. H. L. Sim, Y. Zhang, M. C. Chan, and B. K. H. Low. Collaborative machine learning with incentive-aware model rewards. In Proc. ICML, pages 8927-8936, 2020.
[36] T. Song. Y. Tong, and S. Wei. Profit allocation for federated learning. In Proc. IEFE Big Data, pages 2577-2586, 2019.
[37] S. Tay, X. Xu, C. Foo, and B. K. H. Low. Incentivizing collaboration in machine learning via synthetic data rewards. In Proc. AAAI, 2022.
[38] T. Wang, J. Rausch, C. Zhang, R. Jia, and D. Song. A principled approach to data valuation for federated learning. In Q. Yang. I.. Fan, and H. Yu, editors, Federated Learning: Privacy and Incentive, Lecture Notes in Computer Science, pages 153-167. Springer International Publishing, Cham, 2020.
[39] X. Xu, L. Lyu, X. Ma, C. Miao, C. S. Foo, and H. Low. Gradient driven rewards to guarantee fairness in collaborative machine learning. In Proc. NeurlPS, 2021.
[40] Yoon, S.O. Arik, and T. Pfister. Data valuation using reinforcement learning. In Proc. ICMI., pages 10842-10851, 2020.
[41] Z. Zhang, Y. Song, and H. Qi. Age progression/regressinn by conditional adversarial autoen- coder. In Proc. CVPR, pages 4352-4360, 2017.
12


Reviews:
[{"review_id": "eTCy0Z6i5op", "reviewer": "Reviewer_deSj", "summary": "The paper proposes to use the volume of the data matrix as a data value metric. The contributions of the paper include (1) presenting theoretical and empirical evidence about the connection between volume and learning performance (2) designing a robust volume and (3) experimental validation", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- the paper is well-written\n- the theoretical analysis of the paper is solid\n\nCons:\n- While the reviewer appreciates the solid theoretical analysis around volume and replication-robust volume, the motivations for this paper are questionable. One of the motivations is to have a validation set-free data value. In practice, the data's value is about  not only performance but other ethical concerns like fairness. With a validation set, it's flexible to encode fairness by maybe using an unbiased validation set. Another example is when the data buyer wants to build a model for a targeted population; then the validation set could include only data from a certain population. With a validation set, it's convenient and flexible to encode data buyer's expectations about data, which may vary from application to application. However, the volume does not seems to offer this flexibility. The second motivation is to enable replication robustness. Although this is interesting theoretically, there are very mature existing deduplication techniques in the database literature. In practice, data replicates would not be much of a concern because one can just run these duplication techniques and then perform data valuation.\n- The use case of volume is limited to linear regression thus far; it's not clear how to extend it to other widely-used models.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the paper is not sufficiently discussed. For the feedbacks, please refer to the previous section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "SDl1DGT3Ug", "reviewer": "Reviewer_s1Fn", "summary": "This paper intends to design a method to evaluate the value of a dataset. Such methods have been previously developed by calculating the Shapley value of the dataset based on its effect on model accuracy. The authors argued that since the model accuracy depends on a validation data set, selection of the validation dataset highly affects the Shapley value of each party's dataset, and hence will be a source of disagreement. To overcome this issue the authors set the value of a dataset based on the determinant (volume) of the left Gram matrix of its input features. The value of duplicated data is removed by constructing a compressed version of the dataset that preserves its diversity. The authors refer to this as robustness, since the values are robust to duplications. The authors provide some propositions connecting volume with bias and MSE. This has been validated experimentally as well. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper intends to design a method to evaluate the value of a dataset. Such methods have been previously developed by calculating the Shapley value of the dataset based on its effect on model accuracy. The authors argued that since the model accuracy depends on a validation data set, selection of the validation dataset highly affects the Shapley value of each party's dataset, and hence will be a source of disagreement. To overcome this issue the authors set the value of a dataset based on the determinant (volume) of the left Gram matrix of its input features. The value of duplicated data is removed by constructing a compressed version of the dataset that preserves its diversity. The authors refer to this as robustness, since the values are robust to duplications. The authors provide some propositions connecting volume with bias and MSE. This has been validated experimentally as well. \n\nIt is nice that the authors decoupled valuation and validation. However, their approach comes with its own issues. For example, in reality, there is a higher value to a point that appears multiple times in a dataset. (To understand this think of an extreme case where the points are uniform samples of the true distribution and 99% of them are the same point.) But this approach set the value of the duplicates to 0. If removing the value of the duplicates is because we don't trust that the parties are going to share their actual data point, they may as well generate completely fake datapoints. \n\n20: ... such as the gold standard for oil. Does not read well?\n44:  to better performance ->  to a better performance\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not discuss any societal impact. I don't think there is any major negative ones. ", "review_text": "", "extra_scores": {"ethical_concerns": "There is no ethical concern ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "4PipfRETIqb", "reviewer": "Reviewer_WiPz", "summary": "The paper proposes a measure of datasets’ values using the volume of the feature matrix (defined to be the determinant of its left Gram), which captures the diversity in features. They show the effectiveness of this measure both theoretically and empirically. The paper proves that for regression problems,  a larger volume usually leads to a smaller bias and a lower mean squared error. They also verify this claim using randomly generated datasets. They also proposed a replication robust measure by compressing similar data. Finally, they use experiments on both real-world datasets and synthetic data to verify their claims and demonstrate that their measures without validation give results consistent with other methods which may require validation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a novel data valuation method that does not require validation using test data, which can also be robust to data replication. The paper is strong in the sense that it provides both solid theoretical guarantees and extensive empirical evaluations of their measures. I find no obvious weakness of the paper.\n\nThe paper is well written and easy to follow. I didn’t find any glaring mistakes in the results and the proofs.\n\nSome other relevant papers:\n“A Distributional Framework For Data Valuation”  Amirata Ghorbani, Michael Kim, James Zou\n“Truthful Data Acquisition via Peer Prediction” Yiling Chen, Yiheng Shen, Shuran Zheng\n“Replication-Robust Payoff-Allocation for Machine Learning Data Markets” Dongge Han, Michael Wooldridge, Alex Rogers, Shruti Tople, Olga Ohrimenko, Sebastian Tschiatschek\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies an important area. The reviewers had some concern about the motivation for validation-free data value and replication robustness, and limitations of the model studies in this paper. Still, the general agreement was that validation-free data value would be important when there are not enough test data, and while the method has its limitations, it's still a worthwhile contributions. Overall, while a borderline accept, the votes lean toward accepting the paper.
Decision: Accept (Poster)