Data Acquisition via Experimental Design for Data Markets
Authors: Charles Lu
Baihe Huang
Sai Praneeth Karimireddy
Praneeth Vepakomma
Michael Jordan
Ramesh Raskar
Venue: NeurIPS 2024 poster (2024)
Keywords: Data valuation
experimental design
data markets
data acquisition
federated learning
Primary area: other
TL;DR: We identify weaknesses in current data valuation methods for data markets and propose a federated data acquisition method based on experimental design that achieves state of the art performance on several real world medical datasets.

Abstract: The acquisition of training data is crucial for machine learning applications. Data markets can increase the supply of data, particularly in data-scarce domains such as healthcare, by incentivizing potential data providers to join the market. A major challenge for a data buyer in such a market is choosing the most valuable data points from a data seller.  Unlike prior work in data valuation, which assumes centralized data access, we propose a federated approach to the data acquisition problem that is inspired by linear experimental design. Our proposed data acquisition method achieves lower prediction error without requiring labeled validation data and can be optimized in a fast and federated procedure. The key insight of our work is that a method that directly estimates the benefit of acquiring data for test set prediction is particularly compatible with a decentralized market setting.

Data Acquisition via Experimental Design for Data Markets
Charles Lu" MIT
Baihe Huang UC Berkeley
Sai Praneeth Karimireddy USC, UC Berkeley
Praneeth Vepakomma MBZUAI, MIT
Michael I. Jordan
Ramesh Raskar
UC Berkeley
MIT
Abstract
The acquisition of training data is crucial for machine learning applications. Data markets increase the supply of data, particularly in data-scarce domains such as healthcare, by incentivizing potential data providers to join the market. major challenge for data buyer in such market is choosing the most valuable data points from a data seller. Unlike prior work in data valuation, which assumes centralized data access, we propose federated approach the data acquisition problem that is inspired by linear experimental design. Our proposed data acquisition method achieves lower prediction error without requiring labeled validation data and be optimized in fast and federated procedure. The key insight of our work is that method that directly estimates the benefit of acquiring data for test prediction is particularly compatible with a decentralized market setting.
Introduction
While massive training datasets enable major machine learning breakthroughs, they remain largely inaccessible outside of large companies, motivating mechanisms for broader data access. A related point is that many data owners have become resistant to having their data collected indiscriminately without their consent or without their participation in the fruits of predietive modeling, resulting in legal challenges against prominent Al companies [16] 33). These trends motivate the study data marketplaces, which aim to incentivize data sharing between sellers, that provide access to data, and buyers, that pay compensation for data access [1] 54].
For practical data acquisition scenarios, a data buyer has a specific goal in mind and, in particular, wants training data to predict their test data in specified context. Accessing different datapoints may require varying prices associated with each datapoint, which may reflect heterogeneous cost, quality, or privacy levels for each datapoint
40)
For example, consider a hospital that wants to make a prediction for a specific patient's X-ray. The hospital can submit this X-ray as an unlabeled test query the marketplace, along with a budget to access relevant training data. The marketplace selects useful training data to build model for this specific prediction task, sharing only the final prediction rather than the raw data to protect privacy. This process can be repeated for each new patient query (see Figure T}. However, not all X-ray images will be equally relevant. Thus, we want to select only those seller datapoints that are most useful for answering the buyer's s query and fit the buyer's budget.
This goal of data acquisition has motivated the development of many data-valuation techniques
(e.g., (23 29 38
60
52
39
57
46 30). However, we argue that current data valuation techniques
"luchar@mit.ee edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Buyer wants diagnosis on their X-ray for $100
Data market platform selects data from sellers using buyer's X-ray
Platform chooses most. cost-effective datapoints
Predicted: 90% pneumonia
Split $100 budget between data sellers
Perform inference buyer's test. data
model using selected data
Figure Overview of Data Marketplace Approach. buyer brings their test query (c.g., patient's chest X-ray needing diagnosis) and budget to the marketplace. DAVED selects the most relevant subset of seller training data to minimize prediction error on the buyer's specified test query while respecting budget constraints. Unlike prior methods that require labeled validation data, DAVED directly optimizes for test performance. This enables targeted, cost-effective data acquisition compared to purchasing entire datasets.
are misaligned with the data acquisition problem, particularly the context data marketplaces. They all face at least one of the following limitations:
The selection process may not adaptive to the buyer's (unlaheled) test queries, potentially failing to identify the most relevant data. In data marketplace, buyers typically need to purchase only small subset of datapoints most relevant to their test data, which may follow significantly different distribution than the overall seller data.
When adaptive selection implemented, these techniques rely on labeled validation data, which is often impractical. Further, when a small quantity of such data is available, the selection may overfit the validation data and result in poor performance on the test queries.
The algorithms are not scalable and typically require retraining the ML model numerous times. Hence, they are unable select from realistic seller corpora (> 100K+ datapoints).
Instead, we propose data acquisition via experimental design (DAVED) method that overcomes all of these limitations. Unlike most previous work in data valuation, our approach does not require labeled validation dataset and instead directly optimizes data selection for the buyer's 's unlabeled test queries.
Additionally, our approach accounts for budget constraints and able to weigh the price of each seller's datapoint against its potential benefit, simultaneously solving the budget and revenue al- location problems [64]. Moreover, it can be implemented a federated manner, achieving lower prediction error even compared to centralized baselines.
Our contributions are the following:
1. Formulate the data acquisition problem for data marketplaces and demonstrate that data valuation methods make fundamental theoretical mistake of "inference after selection" (Theorem I..
2. Design novel, highly scalable, and distributed data selection procedure that eliminates the need for validation data by directly selecting the most cost-effective seller data to answer the buyer's test queries (Algorithm DD.
3. Demonstrate state-of-the-art performance on synthetic data and medical data various modalities (X-rays, images, and text).

---

Dimension = 30 / Validation 100
Dimension = 10 / Validation 10
2.0
Test
1.5
eπror
1.0
1.0
0.5
0.0
10
15
20
25
Budget
Budget
DAVED (ours)
Data Shapley
Influence
LAVA
Data OOB
DVRL
KNN Shapley
Random
Figure 2: Failures of Current Validation-based Data Valuation Methods. Current data valuation methods overfit when data dimensionality is high or validation sets are small. Using 1,000 seller datapoints (each with cost I) of Gaussian distributed data, we compare test error across methods as buyers acquire data under different budgets. (Left) Validation-basee data valuation methods overfit when the data is too high dimensional (d 30). (Right) Even with low-dimensional data (d 10), overfitting occurs when the validation set is too small (n 10), resulting in worse performance than random selection. Our validation-Fi free method (DAVED) method maintains low error both scenarios.
2 Data Acquisition versus Data Valuation
In a decentralized data marketplace, data acquisition must be performed before full data access is granted to the buyer (36). This relates Arrow's Information Paradox 7 sellers unwilling share data before payment, while buyers need to evaluate utility before purchasing. This distinction between data valuation and data acquisition for data marketplaces is also discussed in recent data acquisition benchmark, where data value must be estimated without requiring white-box access to the seller's data (i.e., full, unrestricted access the data) [12].
more fundamental issue with validation-based data valuation approaches is exemplified by the Data Shapley value approach 23 29 38 etc.], which measures the marginal contribution of each training datapoint's 's improvement to a validation metric. They are great for after-the- attributing the relative influence of the training data. However, they cannot be used to make decisions about which datapoints should be included in the training. This is because of, as noted earlier, the "inference after selection" issue. Using validation data to select training data leads substantial over-fitting the validation data.
Illustrative example: Suppose that we only have single validation datapoint. Then, it is clear that will select training data similar to this singular datapoint, and our selection has no hope working on the test dataset. While this clearly demonstrates overfitting in an extreme scenario, Figure 2 that increasing the validation set size does not circumvent this issue. We see that other data valuation techniques have poor test prediction errors some techniques random selection baseline! This clearly demonstrates overfitting. Our proposed method maintains low test error more seller training data is selected. In Section phenomenon and prove a very strong theoretical lower bound. We show that any approach that relies upon validation data for data selection can perform badly throwing away all the training data and simply training on the validation data alone! This is especially true when our budget is small compared the dimensionality of the problem, is likely data market setting the data is typically high dimensional, and we can only select a very small fraction of the total available data.
show in
underperform even
will dig in deeper into this
3 Setup and Limitations of Prior Methods
Description of Data Acquisition Setting, As shown in Figure 1 in our setting of data acquisition, a buyer has a budget and test data. The platform the buyer's 's data to select training datapoints from the seller that optimize buyer test eиror. The interaction proceeds as follows:
The buyer brings their test data Xtest [xtest ..... test and budget to the market
platform. This test data is associated with unknown target labels test [y]est ..... yest
Test

---

2. The platform also has access to datapoints from data sellers Ztrain = {(zz, by) and their associated costs {cy}j=m,.....
3. Given only the covariates X test, the platform assigns a selection weight wy to each datapoint (z)) This weight wj E {0, } represents the discrete action of selecting datapoint j.
4. The platform selects datapoints from the sellers according to w = (w1,.... ((......wp)). trains a model Jatw>> makes the predictions Sū(w) (Xtest), and distributes Cj wy payment for each datapoint used in training.
In general, we not make i.i.d assumptions between the train and test we expect the test queries will not be similar to the total available train data. The goal then is to pick the weights which minimizes the prediction error for the buyer while adhering to their budget constraint Σラ= my S B. This gives rise to the following problem:
min ((w) w
= 17 Σ
(a) (attent) (Fantt y/ test
s.t. Σ j=1
Hy S B,
(I)
where is squared loss. Here, the expectation is over the conditional label distribution of y/ |ztess and the potential randomness the algorithm. Note that this problem can not be solved because we do not know the targets ytest, Instead, we need to rely surrogate objective function (proxy) [(w). One approach to constructing such proxy is by using validation data.
Folly of Relying on Validation Data. In most data valuation methods, e.g., Data Shapley 23 the
value of data is evaluated using a labeled validation set Zual = {{(zjal y)² } Implicitly, these methods assume that the known Z is drawn from the same distribution as the unknown Ztest Then using this validation data, scores ($1,..., , 8n) are assigned to the seller training datapoints Ztrain For two datapoints j train, the score 8; > 8j if the datapoint is more valuable than j [46] 30 More concretely, 8; 8j implies that training with i would lead to a smaller validation loss than if was used instead. Thus, these scores can used select the most valuable datapoints.
However, note that we used the validation dataset to compute the scores. Thus, selecting the top-k scores results in implicitly minimizing the validation loss i.e., all validation-baset data valuation schemes implicitly optimize the following proxy loss
min WC (0,1)
Eval
Thval Σ
(Jecus (1rg))) v.
s.t. Σ
a
wjys S B.
This approach heavily relies on the quantity and quality of the validation dataset in order to generalize the actual test dataset. In fact, we have the following minimax lower bound even when restricting ourselves to simple linear models.
Theorem 1 (Informal version of Theorem I.Ipp Let w" denote the solution of our original problem ① and w solve Q Suppose that all our data Zrrain Zral Zrest are drawn i.i.d. from some distribution Dx where Dx is supported on Bá (zero-centered ball with radius in TRd), and Y = A X + & where e is independent zero-meaned noise with variance o²_ For any training algorithm, when the number of training data is sufficiently large, with high probability,
odd
inf sup Ex [C(W) -(w")[( Drix
Tvall
This result implies that the expected test error for any validation-based approach can in the worst-case scale as d/n /aal with high probability. This dependence on the dimension d and the number of validation points neaa highlights that this method may be suboptimal in high-dimensional settings or when the validation dataset is small. In fact, we would get the same error scaling if we threw
away the training data and trained a model on the nyal validation datapoints alone. This explains the striking overfitting we observed earlier in Figure Furthermore, obtaining a large amount of ground-truth labeled data may be challenging in many real-world applications. Instead, we propose validation-free approach to data acquisition based on experimental design.

---

4 Our Methods and Implementations
We propose an alternative approach based on proxy objective. Our key assumption is that the conditional distribution yy y = fe_ (x) + e is identical across Ztrain and z This is a natural assumption in many domains for instance, if an X-ray exhibits indicators of specific discase, should receive the same diagnosis regardless of whether it appears in the training test set. Without this assumption, our problem becomes intractable since the same z, could map to arbitrarily different labels across train and test sets. Under this framework, we can reformulate our problem using V-optimal experiment design
Step 1: Linearizing the problem. Our goal is to design a proxy loss function [(w) which approximates the true test loss C(w). To do this, we have to reason about how different choices of training data S C Ztrain could impact the prediction on a particular test datapoint Xtest This is notoriously challenging problem for general deep learning models [9]. Instead, we use linear approximation and model the complicated training dynamics with kernelized linear regression. We suppose we have a known data is generated as
extractor ø X' + Id and an unknown 0 E Rdo such that the
y = 0" ((x) +
where is independent noise with mean zero and do is the embedding dimensionality. The function 6(.) can be the empirical Neural Tangent Kernel (eNTK) 127 embeddings extracted from deep neural network such as CLIP 49
58) of the model, or even the While this may be bad approximation in general [61, recent line of work has shown that such eNTK representation very closely approximates the fine-tuning dynamics of pre-trained models both theoretically [58] 42 as well as emperically [22] 63). In fact, such linear approximations have also been used to speed up validation-based data attribution computations [46).
Step Experimental design proxy. Given the assumption on our data from Eqn. (3)
we can use the V-optimal experiment design framework [5I 26] to define proxy objective. First, suppose that S C (ø(xtrain) traiin), ytrain) is the subset selected by w and then we performed least-squares regression.
The resulting estimate ((w) can be computed in closed form as
((w)
(E w,)(z))
(
Henceforth, we will drop the e when obvious from context and simply use r. We can further use Eqn. 3 to compute the expected error on an arbitrary test query æo, yo as follows:
[((((w) zo y) train zo
JTx æ. ((((w) ()) )²) TT)T B||e/|
E [[((Ō(w) - ) xo + +
= () (w) 100 + E||=|1
_ a0 " (E wyJ
'an B||e|12
II(w)
Here ag uses the unbiasedness of the ordinary least squares (OLS) estimator and a4 plugs in the closed form of (ww and simplifies. With this, we end up with very clean expression for the expected test error on an arbitrary point zo. and the matrix T(w) is known the Fisher information matrix. While regression suffices for our use case, the procedure can be extended to general linear models. Dropping the fixed EB|s|| (2, we can use this to build our proxy function [ED (WD(w) and arrive at the following optimization problem
min (E(0.1)
CDD (BD(Ww)= = 1/m Em test T I(w) ((
s.t. Σj= wye S B.
This optimization objective directly measures how useful each training point would be for predicting the specified test query. The matrix Z(w) captures how much information each selected datapoint provides about the test point in the embedded feature space.
Note that our proxy function (w) can be computed using just Xtrsin Xtest and does not even need access to training labels. Unfortunately, the objective 4 is NP-hard to optimize We next see how to derive fast and provably good approximation algorithms for (4)
5

---

Step 3: Fast approximation. To make Eq. amenable to gradient- based optimization, we drop the constraint that W {0. and allow to be a continuous positive vector i.e., w > O and Σj=1 Wjj; I B. With this relaxation, the proxy objective (BD(w) is continuous and convex in w [10]. We then run the "herding" variant of the Frank- Wolfe algorithm [7] 53 65 this, define (wr == wr/c) for any t. We start from Wo ey and iteratively update as
To do
We+1 - (1 Ote) W, orej
where jr arg max(
m., (w:))/e))
(5)
jE(m]
Note that if we use the size at = x+1 in 5). We satisfies special property at any iteration
W. E A and further 1) E {0,
Run the procedure until the last iteration t = to for which we still have ||w:, 11 S B. We can adapt the theory from 8 to analyze the above procedure and show the following.
Theorem 2 (Informal). Let us run Frank- Frank-Wolfe herding update ( for to steps such that it is selection vector
last step which satisfies [wi,l1 S B. We Wt, = ((tu + 1) /c) and we would have selected to datapoints. Then, under some assumptions, we have
log to
LED ((t + 1) S
min wE((..)" M
LDD (BD(W)= +0
40
The above theorem shows that continuous relaxation does not significantly affect the optimality of our result- we get O fo close to the optimal solution to the original NP-hard 4 If all datapoints have equal cost c, then to LB/c], and so our approximation quality improves as we increase the budget. While better approximation guarantees are attainable [3). their procedure is significantly more involved and not easily amenable efficient federated implementations as ours
to
is.
Step 4: Efficient federated implementation. Our practical implementation directly restricts E A instead of W in the theoretical implementation above i.e., we run
W < (1 - w; + o4e5 where je = arg max V (ww)(//)/
(6)
jE(m]
This way w can be directly interpreted to be the sampling probability for different seller training datapoints. The bottleneck to efficiently implementing 66) is computing the gradient. At step t, the negative gradient be shown to be
9j = Vw, ((w)) = 1/m Σm TI(::) (x)rrin))
Thus, if we have the inverse information matrix I(w:)f pre-computed, 9j well the update (6)
can be trivially computed by seller j using only their data 2r) trarain (and the test data). Next, we show how to efficiently maintain the inverse information matrix. Note that the update (6) has special structure: all coordinates are shrunk, and then only a single coordinate of w, is increased. We can relate the resulting I matrices with rank-one update as:
I(w++1) = - (1-az) ((w.) + a.
Define P; I(..)t. We can use the Sherman-Morrisse formula 50 compute Pr+1 I(W++1)t
as
axP Pr
P+1
P. 0t
(8)
1 04 + a1 PeN
For each round I, this update only involves the current matrix P. I(w;) and the single datapoint Ty selected for the round. Thus, the seller can also locally compute this update as well as the updated cost L(w) as in Eq. (4) for any az:
((w) -I--a.7 Em (x: P. x.)
Em (xr T Pe (..
I+aix ja
PLas,
Here, ej is the standard basis vector along axis j. and in w w/c, the division is performed element wise.

---

Algorithm DAVED: Iterative Optimization Procedure
1: Input: buyer test datapoint X E Rmxd seller training data X E RRxd seller weights
E A", iteration steps T, regularization parameter AReg [0, 1|, and seller datapoint costs
c E R
2: Wo 1/n
# Initialize weight vector uniform distribution # Initialize P (Bq. 10
3: Po + ((1- -Reeg) X diag(wo) X + AReg xx xxlnx))
4:
for E (1,2..... .......... T do
5:
9 + -V((w;:
# Compute negative gradients (Eq. 7
6:
Jr + arg max, (01/es)
# Select coordinate based costs
7:
< LINE TREE_SEARH([4 ())
# Find optimal step size (Eq.
9)
8:
W < (1 04 we) + a,C,, Shrink weights and upweight the chosen coordinate (Eq.
9:
Pi+1* +- SHERMAN (MO.RMOOROOO....... (j. 04) # Update inverse information matrix (Eq.
10: end for
11: Output: Sample seller data according to E A without replacement until budget B runs out.
Thus, a line search can be performed determine the optimal step size a E [0, 1] to minimize the proxy loss as. This differs from (5) where we used a specific choice of ay Frank-Wi Wolfe is known to be more stable with the line search [53 655 The seller can communicate this oa and Tj. to the platform to compute the updated Pr+1 using only ((d) communication.
An additional practical consideration is that by initializing wo c1e1, have ill-conditioned inverse information matrix Po. We instead use an initialization of wo = L./n E A and further add feature-wist regularization term. This makes the initial Po
Po ((1 AReg) T diag(wo) + ARey diag(a))
(10)
where 8; V + -y [X -2, is the empirical standard deviation of feature i. The complete details are summarized in Algorithm
Single-step variant. We can also forgo the iterative process and instead linearly approximate the cost function (Eq 4 with single step that selects the top l datapoints under the budget B,
si single_step(r x, B) top_k( (( Σ [( Pox,] 1=1)
(11)
This simplified version is extremely fast while still maintaining relatively good performance.
Experiments
We evaluate our proposed method for data acquisition (DAVED) against common data valuation methods on both synthetic data and four real-world medical:
1. Fitzpatrick17K (24], skin lesion dataset, where the task is to predict Fitzpatrick skin tone on a 6-point scale from dermatology images.
2. RSNA Pediatric Bone Age dataset (25). where the task is assess bone age (in months) from X-ray images of an infant's hand.
3. Medical Information Mart for Intensive Care (MIMIC-II) 31, where the task is to predict the length of hospital stay from 48 attributes such as demographics, insurance, and medical conditions.
4. DrugLib reviews text reviews of drugs where the task is to predict ratings (1-10).
For validation-based methods, we use a validation set of 100 datapoints. We re- port test errors over 100 buyers. For more details the experimental setup, see Appendix C data-acqui Our code available at this repo: httpp:://itthbt www.tthhu..ommmiiic. www.ccoocccaoooooo www.raaaweeeeppaaaeeeee -deeennnnllddiiggn
Comparing Performance on Data with Homogeneous Costs. Figure we evaluate our method and several other data valuation methods on varying amounts of Gaussian data with homogencous
7

---

Gaussian 1K
Gaussian 5K
Gaussian 100K
1:25
1.25-
Test
error Lo-
0.78
0.75
0.5
0.5
0.5
0.25
0.25
0.0
15
Budget
Budget
Budget
DAVED (multi step)
Data 0OB
DVRL
Shapley
Leave One Out
DAVED (singla step)
Data Shapley
Influence
Random
Figure 3:
Data Acquisition Performance across different Market Sizes Synthetic Data. We
compare test prediction error as seller training data is selected under varying budgets and amount of data for sale, with total available seller data of 1K (left), 5K (middle), and 100K (right) points. Our data selection method (DAVED) consistently achieves lower MSE with fewer purchased datapoints, i.e., better data acquisition efficiency, than other data valuation methods. Both multi-step and single-step variants of DAVED achieve lower test MSE with fewer training points compared to validation-based methods. The performance gap especially pronounced with small budgets (5-10 points). Unless otherwise specified, all results are averaged over 100 random test points.
fixed costs. Compared to other methods, both multi- and single-step versions of DAVED have lower test errors across budgets on synthetic data. This performance gap is especially large when the buyer has small budget (around 5-10 seller training datapoints). In Figure 4 we evaluate our method on real image and text data embedded through CLIP and GPT-2 feature representations. We observe that DAVED has better performance compared to most other baselines on all three datasets, highlighting that the proposed method is practical for embeddings of high-dimensional data. Table summarizes our results on all datasets. For the Gaussian data and MIMIC datasets, we report the mean error of budgets from I to 10, while for the embedded datasets (RSNA, Fitzpatrick1 1K, DrugLib), we report the mean error of budgets from to 100 in intervals of five.
Comparing Performance on Data with Heterogencous Costs. Next, we compare methods on seller data with non-homogeneous costs. We uniformly sample costs E (1, 4,5) } for each seller datapoint and consider two cost functions, Cj = /c and с₁ = c², which downweights gradient of that datapoint 2j (see Equation 7 To simulate heterogeneous utility across datapoints, we introduce cost-dependent label noise, ~ (ū, o²), to each datapoint V/ == y/a + BBē/c,, where ÿ is the mean target value and 3 is the overall noise level, which we fix at 30% throughout our experiments. For these experiments, we did not evaluate Data Shapley 23 LOO 13). and Influence (2 that had very long runtimes. In Table 2 we report additional mean test error across budgets 1-30 for both cost functions. We find that our DAVID method is more budget-efficient in choosing cost-effective noisy datapoints than other methods across datasets. We provide additional plots for heterogeneous costs in Appendix D.I
Comparing Runtime. In Figure we compare the optimization runtime of our data selection method on 1,000 datapoints while increasing the dimensionality of the data well when the dimensionality is fixed to 30, and the number of seller datapoints is increased to 100,000. Data Shapley (23 and LOO [3] took too long to run for large amounts of datapoints or high dimensional data and not reported. In both experiments, our multi-step compares favorably to efficiency-optimizze such KNN Shapley 29 while our single-step method had the fastest runtime. This demonstrates that our method can scale to marketplaces with millions of datapoints.
techniques
Regularization Strength. In Appendix D.2 we vary the amount of regularization applied on the MIMIC, DrugLib, and RSNA datasets. We find that applying a moderate amount of regularization between 0.2 and 0.6 can lead to improved performance. Even when the information matrix is set to identity, i.e., x = 1, performance on the DrugLib datasets is still reasonable. Note that for all other experiments, we do not apply any regularization.
Amount of Buyer Data. Appendix [D.3] we vary how many buyer test datapoints are simultaneously optimized over on sampled from the same distribution, the number of buyer datapoints still affects the optimization procedure. In general, we find that increasing the number of datapoints in the "test batch" increases
MIMIC, and RSNA datasets. While all buyer and seller data
8

---

Filzpatrick17K
Druglib
20
2000
Test
15
1200
error
1000
20
40
80
100
120
40
80
100
120
1400
20
60
100
120
140
Budget
Budget
Budget
DAVED (multi step)
Data ODB
Influence
LAVA
DAVED (single step)
DVRL
KNN Shapley
Random
Figure Data Acquisition Performance on Real Medical Datasets. DAVED demonstrates strong performance real-world medical imaging and drug review datasets. (Left to right) Results Fitzpatrick17K (skin lesions), RSNA Bone Age (X-rays), and DrugLib (drug reviews) where high- dimensional raw data is embedded via CLIP (images) GPT-2 (text). Each method selects training points under budget constraints to train a regression model on the embedded data. DAVED achieves lower test prediction error using fewer training points compared to validation-base approaches, demonstrating effectiveness on high-dimensional data.
Table 1: Test Error of Data Valuation Methods. We compared the test mean squared error on the buyer test point synthetic Gaussian-distributed data and four medical datasets: MIMIC, RSNA, Fitzpatrick17K, and DrugLib. The subheading denotes the number of seller training data available for that experiment, and "N/A" denotes that the method exceeded runtime constraints for the experiment. We optimize separate random sample of training and validation data for each buyer and average over 100 buyers. Bolded values indicate the best rest-performing method and underlined values denote the bestnddbee---rffrmigg method.
Gaussian
MIMIC
RSNA
Fitzpatrick 15K
DrugLib
Method
1K
100K
1K
35K
12K
3.5K
Random baseline
1.38
1.01
301.0
283.7
1309.1
1.49
21.4
Data Shapley 23
0.87
N/A
294.9
N/A
N/A
N/A
N/A
Leave One Out [13
1.31
N/A
1125.0
N/A
N/A
N/A
N/A
Influence 21
1.47
0.97
189.4
876.4
1614.5
1.93
12.8
DVRL 62
1.33
1.26
229.7
285.5
3528.8
3.00
12.6
LAVA 32
1.47
1.10
190.9
417.3
1867.5
1.45
17.4
KNN Shapley 29
1.55
1.18
175.7
229.6
1387.0
1.82
19.0
Data OOB
39
1.24
0.98
169.7
215.6
1020.3
1.35
10.0
DAVED (single step)
0.58
0.27
277.4
659.9
900.2
0.73
9.0
DAVED (multi-step)
0.37
0.16
206.7
171.4
785.2
0.67
9.2
Table 2: Test Error with Heterogeneous Costs. Comparing data selection methods for two different
cost functions, ve and c². For each budget constraint, we select seller datapoints until the budget exceeded and calculate test prediction error on the buyer data. We average over 100 buyers and report the mean test еттог across budgets from to 30.
Gaussian
MIMIC
RSNA
Fitzpatrick
DrugLib
COST FUNCTION
Ve
2
Ve
d²
Ve
2
Vc
2
Ve
2
Random baseline
2.36
77.7
288
285
2254
2065
2.14
2.11
16.5
18.4
DVRL
1.67
2.1
214
215
24003
5588
1.46
8.90
22.5
20.7
LAVA
2.13
3.3
482
475
1667
1587
2.09
2.21
35.7
34.8
KNN Shapley
2.13
69.0
217
956
2754
2506
1.85
2.15
13.6
13.0
Data OOB
2.19
3.3
243
246
1695
1205
2.08
2.52
10.8
10.8
DAVED (single)
1.54
251.5
598
585
1734
1550
0.75
0.71
9.4
10.1
DAVED (multi)
0.04
0.2
169
168
1076
942
0.76
0.75
12.6
11.4

---

Total datapoints 1,,00
Dimension
10a
10a
Runtime (seconds) 10:
10
10
1000
102
Dimensions
Number of seller data
DAVED (single step)
Data OOB
DVRL
KNN Shapley
Leave One Out
DAVED (ours)
Data Shapley
Influence
LAVA
Figure 5: Computational efficiency comparison. DAVED has significantly lower computational overhead compared to model-based data valuation methods. (Left) Runtime scaling with data dimensionality (fixed 1,000 datapoints). (Right) Runtime scaling with the amount of seller data (fixed 30 dimensions). Our single-step variant faster than even optimized methods like KNN Shapley, while the multi-step variant remains efficient while achieving better performance. Our optimization procedure only requires O(d) communication per round, which makes particularly suited for decentralized data market settings. For Data Shapley and Leave-One-Out, some experiments were omitted due to prohibitively long runtimes.
test errors. Therefore, we recommend keeping the number of test datapoints the buyer's "query" between 1-8 for each data acquisition.
Number of Steps. In Appendix D.4 we vary the number of optimization steps in our method on the Gaussianddistrbuuee and RSNA datasets. We find that more iterations generally improve prediction performance. Intuitively, one expects that selecting points requires at least steps of iterative optimization. We recommend setting the number of steps to be 2-5 times the desired budget for homogeneous costs.
Convex versus Iterative Optimization In Appendix D.6 we compare the iterative optimization procedure against convex optimization solver [8]] We find that our iterative approach results in several orders magnitude speedup while maintaining similar levels of test error.
evaluate fine-tuning versus linear probing for datapoints selected using DAVID and random selection. We find that using DAVID for fine-tuning performs similarly to linear probing results on DrugLib with BERT [17].
Finetuning versus Linear Probe. In Figure 14
Discussion
While other validation-free methods exist [60 5), our method uniquely combines test-adaptivity, theoretical grounding, and superior empirical performance. Moreover, major advantage of our method is that it is amenable to federated optimization requiring (dd) communication per round, making it well-suited for decentralized data marketplaces, unlike other methods that require seller data to be centralized in order to repeatedly train models to estimate data value. Additionally, our method does not require labeled data, whereas other data valuation methods assume that all datapoints come with corresponding ground-truth labels. As discussed in Section [ and Section 3 the existing paradigm valuing data with a validation set is suboptimal. Incidentally, the second-best performing method, Data OOB [39], is the only other method that does not use validation set.
Limitations. However, our algorithm comes with some limitations that form exciting directions for future work. Our approach currently communicates every step. Instead, integrating local steps like in FedAvg 43 or Scaffold (35) would decrease communication costs. Further, integrating differential privacy techniques would provide formal privacy guarantees to the buyers and sellers [9]]. While DAVED shows strong performance across datasets, its effectiveness depends on having a good feature extractor that captures relevant aspects of the data. We recommend using pre-trained foundation models (e.g., CLIP, GPT-2) as they can extract general-purper features. Future work could explore adapting the feature extraction to specific domains or handling cases where key features missing
10

---

Acknowledgments
C.I., is funded by the National Science Foundation Graduate Research Fellowship Program. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. 2141064. Any opinions, findings, and conclusions recom- mendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. S.P.K. and M.J. were funded by the European Union P.V. was partially supported by the ADIA Lab Fellowship.
DBCC-202-SYG-OCANN-OO
References
[1] Anish Agarwal, Munther Dahleh, and Tuhin Sarkar. A marketplace for data: An algorithmic solution. In Proceedings of the 2019 ACM Conference on Economics and Computation, pages 701-726, 2019.
[2] S Damla Ahipaşaoğlu and Michael Todd. modified frank-wolfe algorithm for comput- ing minimum-area enclosing ellipsoidal cylinders: Theory and algorithms. Computational Geometry, 46(5):494 519, 2013.
[3] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via prrmttrrattti.i.aionn.. In International conference on machine learning, pages 242-252 PMLR, 2019.
[4] Zeyuan Allen-Zhu, Yuanzhi Li, Aarti Singh, and Yining Wang. Near-optimal discrete optimiza- tion for experimental design: A regret minimization approach. Mathematical Programming, 1 186:439-478, 2021.
[5] Mohammad Mohammadi Amiri, Frederic Berdoz, and Ramesh Raskar. Fundamentals of task- agnostic data valuation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9226-9234- 2023.
[6] Muhammad Jehangir Amjad, Christophe Diot, Dimitris Konomis, Branislav Kveton, Augustin Soule, and Xiaolong Yang. Optimal probing with statistical guarantees for network monitoring at scale. arXiv preprint vrXiv:2109.07743. 2021.
[7] Kenneth Joseph Arrow. Economic welfare and the allocation of resources for invention. Springer, 1972.
[8] Francis Bach, Simon Lacoste-Julien, and Guillaume Obozinski. On the equivalence between herding and conditional gradient algorithms. In International Conference on Machine Learning, 2012.
[9] Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. arXiv preprint arXiv:2006. 14651, 2020.
[10] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[11) Raul Castro Fernandez. Data-sharing markets: Model, protocol, and algorithms to incentivize the formation of data-sharing consortia. Proceedings of the ACM on Management of Data, 1(2):
1-25, 2023.
[12] Lingjiao Chen, Bilge Acun, Newsha Ardalani, Yifan Sun, Feiyang Kang, Hanrui Lyu, Yongchan Kwon, Ruoxi Jia, Carole-Jean Wu. Matei Zaharia, al. Data acquisition: A new frontier in data-centric AI. arXiv preprint arXiv:2311.13711 2023.
[13] R Dennis Cook. Detection of influential observation in linear regression. Technometrics, 19(1):
15-18, 1977.
[14] Thomas M Cover. Elements of Information Theory. John Wiley & Sons, 1999.
[15] S Damla Ahipasaoglu, Peng Sun, and Michael J Todd. Linear convergence of a modified frank-wolfe algorithm for computing minimun-volume enclosing ellipsoids. Optimisation Methods and Software, 23

---

[16] Deven R Desai and Mark Riedl. Between copyright and computer science: The law and ethics of generative arXiv preprint arXiv:2403.1465. 2024.
[17] Jacob Devlin, Ming-Wer Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1a1.:008880040 2018.
[18] Steven Diamond and Stephen Boyd. CVXPY: A Python-embeddded modeling language for convex optimization. The Journal of Machine Learning Research, 17(1):2009-2113, 2016.
[19] Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming, pages 1-12. Springer, 2006.
[20] Valerii Vadimovich Fedorov. Theory of optimal experiments. Elsevier, 2013.
[21] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 2881-2891, 2020.
[22] Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-588 2020.
[23] Amirata Ghorbani and James Zou. Data Shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242 2251. PMLR, 2019.
[24] Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images in dermatology with the Fitzpatrick 17k dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1820-1822 2021.
(25] Safwan S Halabi, Luciano M Prevedello, Jayashree Kalpathy-Cramee Artem B Mamonov, Alexander Bilbily, Mark Cicero, Ian Pan, Lucas Araújo Pereira, Rafael Teixeira Sousa, Nitamar Abdala, et al. The rsna pediatric bone age machine learning challenge. Radiology, 290(2): 498-503, 2019.
[26] Baihe Huang, Sai Praneeth Karimireddy, and Michael I Jordan. Evaluating and incentivizing diverse data contributions collaborative learning. arXiv preprint arXiv:2306.05550 2023.
[27] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[28] Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In Interna- lional conference on machine learning, pages 427-435. PMLR, 2013.
[29] Ruoxi Jia, David Dao, Boxin Wang. Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J Spanos, and Dawn Song. Efficient task-specing data valuation for nearest neighbor algorithms. arXiv preprint 190i: 08619, 2019.
[30] Kevin Fu Jiang, Weixin Liang, James Zou, and Yongchan Kwon, Opendataval: a unified benchmark for data valuation. arXiv preprint arXiv:2306. 10577, 2023.
[31] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng. Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. MIMIC-III a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[32] Hoang Anh Just, Feiyang Kang, Jiachen T Wang, Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jia. Lava: Data valuation without pre-specified learning algorithms. arXiv preprint arXiv:2305.000 00054, 2023.
[33] Zeynep Ülkū Kahvesi. Attribution problem of generative ai: a view from us copyright law. Journal of Intellectual Property Law and Practice, 18(11):796-807 2023,
12

---

[34] Surya Kallumadi and Felix Grer. Drug Review Dataset (Druglib.com). UCI Machine Learning Repository, 2018. DOI: https://bb.orr/1122444//5556
[35] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning, In International conference on machine learning, pages 5132-5143. PMLR, 2020.
[36] Javen Kennedy, Pranav Subramaniam, Sainyam Galhotra, and Raul Castro Feraandez. Revisiting online data markets in 2022: A seller and buyer perspective. ACM SIGMOD Record, 51(3): 30-37, 2022.
[37] Lconid G Khachiyan. Rounding of polytopes in the real number model of computation. Mathematics of Operations Research, 21(2):307-320, 1996.
[38] Yongchan Kwon and James Zou. Beta Shapley: unified and noise-reduced data valuation framework for machine learning. arXiv preprint arXiv:2 10. 14049, 2021.
[39] Yongchan Kwon and James Zou. Data-oob: Out-of-bag estimate as simple and efficient data value. arXiv preprint arXiv:2304.07718, 2023.
[40] Chao Li, Daniel Yang Li, Gerome Miklau, and Dan Suciu. A theory of pricing private data. ACM Transactions on Database Systems (TODS), 39(4):128, 2014.
[41]
Philip M Long. Properties of the after kernel. arXiv preprint arXiv:2105. 2021.
[42] Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pages PMLR, 2023.
[43] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication -effciittttttneeeeiiettt learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282. PMLR, 2017.
[44] Aileen Nielsen. Whose data, whose value? simple exercises in data and modeling evaluation with implications for technology law and policy. New York University Law Review, 99:63, 2023.
[45] Raymond EAC Paley and Antoni Zygmund. note on analytic functions in the unit circle. In Mathematical Proceedings of the Cambridge Philosophical Society, volume 28, pages 266-272. Cambridge University Press, 1932.
[46] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXivv2303.144136 2023.
[47] Friedrich Pukelsheim. Optimal design of experiments. SIAM, 2006.
[48] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 188)9, 2019.
[49] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8763. PMLR, 2021.
[50] Jack Sherman and Winifred Morrison. Adjustment an inverse matrix corresponding to change in one element of given matrix. The Annals of Mathematical Statistics, 21(():124-127, 1950.
[51] SD Silvey. Optimal design measures with singular information matrices. Biometrika, 65(3): 353-559,10 1978.
[52] Rachael Hwee Ling Sim, Xinyi Xu, and Bryan Kian Hsiang Low. Data valuation in machine "ingrdig"" strategies, and open challenges. In Proc. IJCAI, pages 5607-5614, 2022.
[53] Peng Sun and Robert Freund. Computation of minimum-volume covering ellipsoids. Opera- tions Research, 52(5):690-706, 2004.
13

---

[54] Matias Travizano. Carlos Sarraute, Mateusz Dolata, Aaron M French, and Horst Treiblmaier. Wibson: A case study of decentralized, distributed ledger technology use cases: Applications and lessons learned, pages 149-170, 2020.
data marketplace. Blockchain and
[55] Joel Tropp.
User-friendly tail bounds for sums of random matrices. Foundations of computa- tional mathematics, 12:389-434 2012.
[56] Martin J Wainwright. High- Dige-Dimnnsional Statistics: A Non-Asymptotic Viewpoint, volume 48. Cambridge university press, 2019.
[57] Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pages 6388-6421. PMLR, 2023.
(58) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.016 01620,165,, 2021.
[59] Henry P Wynn. The sequential generation of d-optimum experimental designs. The Annals of Mathematical Statistics, 1655,1665,, 1970.
[60] Xinyi Xu, Zhaoxuan Wu, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Validation free and replication robust volume-based based data valuation. Advances in Neural Information Processing Systems, 34: 10837-10888, 2021.
[61] Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522 2020.
[62] Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference Machine Learning, pages 10842-10851. PMLR, 2020.
[63] Yaodong Yu, Alexander Wei, Sai Praneeth Karimireddy, Yi Ma. and Michael Jordan. Tet: Convexifying federated learning using bootstrapped neural tangent kernels. Advances in Neural Information Processing Systems, 11:30882-30897, 2022.
[64] Boxin Zhao, Boxiang Lyu, Raul Castro Fernandez, and Mladen Kolar. Addressing budget allocation and revenue allocation in data market environments using an adaptive sampling algorithm. arXiv preprint arXiv:2306.0255 2023.
[65] Renbo Zhao and Robert M Freund. Analysis of the wolnk-wolfe method for convex composite optimization involving agarittmicalll--omogeeeen barrier. Mathematical programming, 199(1):123-163, 2023.
14

---

A Proof of Theorem 1
Theorem A.l. Let w denote the solution of Problem T and let & denote the solution of Problem
Let the data Z ZUS are drawn i.i.d. from the distribution Dx,y where 'Dx is supported on B (zero-cented centoeee ball with radius R in R2), and Y = f X + where 1) is independent zero-meaned noise with variance o². Suppose Dx x and the training data Xtrain is supported on B (eero-centered
ball with radius R in I2.)) and is square loss, then there exist numerical constants, C12 c2 C3 such that:
1. With probability at least exp (-C1nreal R²):
C20²d
inf 0 X
sup
xx [C(w)) [[w")] 2
nval
2. If there exists >> O such that A(Br<Dx [x S K. (EEnDy 1 (12) (here X denotes the largest eigenvalue and denotes the outer product), then for any training algorithm used by the platform, with probability at least 0.99 exp (-c1 R²2
Csk we have
c202d
sup
((W) - [(w")(()))))))))))>)))
Dy x x Xieain
Thval
Proof. Let fo(x) = 0Tx a and & = NNO,00 0²). Define the parameter space resulting from the training algorithm:
θ ((w) a((ml))
When is sufficiently large, Lemma A.2 implies that there exists {01 02, C θ such that
00, x & Vi E [K]
(1,0--0)
0,) x val ll x Trval vi E [K].
Let Fo, denote the conditional distribution Dylz of the target when the underlying model parameter Og it then follows that
noa
(L(Pe,IIPo,) S
o2
Applying Lemma
A.3
we obtain that
c
inf
sup
E
(0(w) - 8r) x
(12)
Dy 1x.xxxXte Xtrnin
Oval
nval
Now define Σ = Br [xx J. by Lemma
A.4 we have that with probability at least 1
exp(=S2(Benl/ RR)))
- Σ x x val (Xva)) = 2E.
1
(13)
2
Notice that
x [C(W) L(w" (d(w) (() T
Therefore, under the event of Eq.
12
Eq.
13
implics that
(0(0)) 0²) x 2
inf sup Xtrat
Ex [C(W)
L(
2 inf sup á Dyix.
E
Dvval
2d
nval
This establishes the first inequality.
15

---

For the second inequality, we have that under the condition A(EznDx (z
K
DRx (202]02). the following holds
Va ((0(0)) - 0r 00.z) E (0((0) 0", x0 E
(0((8)) - -0", 272
E [((((8) 0r) 100000 2000 }
(B(+)) - -00)00 2002 27
((0(()) - 00)00 , 21
- [200 02
S (N 1). E (8(()) 0-, 0,,²)
By Lemma A.5 for any and 0", we have that with probability at least 0.99 0 0
k+m
((w) - L(w")=
[6 0*) X X 1|
m
1
0 0" test 2
(0( - 0²)
IV
0.0001
5
C
2V
Vl
Combining this and the first inequality by union bound, we establish the second inequality.
A.1 Supporting Lemma
Lemma A.2 (Metric entropy, Wainwright (56)). Let || || denote the Euclidean norm on Rd and let B be the unit balls (i.e., B {0 |||||||| < 1}). Then the 8-covering number of in the || ||-norm obeys the bounds
dlog (3)
S log (o; II) s d 1+ 11
Lemma A.3 (Fano's s inequality, Cover [4] then for any estimator such that + +
When uniformly distributed over the index set [M].
PlO(Z) # N I(Z:0) 0) log
log M
Lemma A.4 (Matrix-Chernnn bound, Tropp (55
Consider an independent sequence { of
random, self-adjoint matrices in M,, satisfying X; 2 0 and Xmax \Ma) S. R. almost surely, for each i k}. Define
4 ΣEX EX,
Panin
Aminn
k Σ EX,
Mmar = Amnix
1=1
Then,
σ (1 8)1+8
P
Amax
Σx
X₁
2 (1 + 5)
for 5 2 0;
i=1
&
- (1 - 8)1-8
Pamm
P
Amnia
Σ Xi
(1 8) 8)
n
for E [0,
i=1
16

---

Lemma A.5 (Paley Zygmund inequality, Paley and Zygmund 45)) If Z is a random variable with finite variance and Z 2 0 almost surely, then
E|Z]²
P(Z R
40 E (0, 1). x 2 (1 - 0) E|
B Convergence Rate of Frank-Wolfe (Proof of Theorem
Setup. We mostly follow the proof technique in Bach al. and Jaggi
28
Recall the optimization
problem for the optimal design loss Eq. (4)
min L(w).
WED
Here, D the scaled simplex defined by the constraints Rg and و Cjw S B. The Trank-Wolff update on this function is then: For ....... repeatedly perform the following steps
Compute 8; arg max ( (V..,,), W. u).
update W2+1 (1 04 w. + a;s;
Note that this update procedure identical to the updates in Eq. [ gap as
We can then define the duality
g(w) = sup(w - s, VL(()).
SED
We also define the curvature constant
C sup 8, w.
(C(u) - -(w) - -(u -.,, VZ(w))).
72
7E(0.1)
u= u=(1->)w+-8
We assume that the curvature constant is finite, i.e., G cc. This is true for L(w) long as both the algorithm and the true optimum are bounded away from the boundary of D - see detailed discussions this in Ahipaşaoglu and Todd 22. better analysis might be able to avoid this assumption, c.g., Zhao and Freund [65] use certain homogeneity properties of [(w) to derive better assumption-fri free convergence rates for the FW method on D-optimal experiment design. We leave the question of adapting these results to our setting (V-optimal experiment design) for a challenging future work.
Lemma B.1 (Lemma 5, (28 ). For any E (0,1). 1),
[(w(+1) s [(w.) arg( w.) + a
Ct.
2
Theorem B.2. the Frank- Frank-Wolfe algorithm, our update algorithm in Eq.
uses a₁ = 141.
1. For this,
we have
[(w)) S min L(w) + Ci(1 + log
WED
2t
Proof.
Define h(w) = L(w) - min D L(w). Using Lemma B.1
a Ci
h((w)) S h(w -4- +
a- Ci
< <h(We-1) - - - +
2
- (1--a-1)((w1-1)
Ci
2
Here used the convexity of L follows:
g(W.-1) = sup(w w
*ED
> (w,-1 w", VL(w,-1) ))
> L(w,,1) min L(w). wED
17

---

Continuing our derivation, recall we have
a C₁
h(w,)) S (1 00- -4.-1) t - h(W,-
2
1
- 2
Ci
h (w.,-)..
28
Thus we have
h.w,))) S -1- (1-1)/((,,)) + Ci
28
Unrolling this recursion, we get
[-1
Ci(1 + logg)
((w,,,) 1
2k
2
k=1
This yields the theorem claim.
Finally, to finish the proof of Theorem 2 note that adding additional constraints only increases the loss, i.e.
min L(w) S WED
min WED and WE (0.1) L(w).
Hence, we showed O(log t/t) approximation to the otherwise intractable combinatorial problem. With additional assumptions on the structure of the loss function, one can even show improved
quadratic or even exponential approximations [15] []. Finally, we note that Frank-Wolfe is well-
known method to efficiently approximate the optimal experiment design objective [59 20
and this
also motivates using this approach in practice 2 6
C Experimental Setup
For each buyer test point, we optimize each selection algorithm the 1,000 seller datapoints and select the highest value data based on the validation set of 100 datapoints (our method and Data OOB do use the validation set). For each point, we train linear regression model on the selected seller points and report test mean squared error (MSE) on the buyer's data and average test error over 100 buyers.
For reproducibility, our full implementation is available at: https://github.c cta.thubb.om//luuu clu5/ data- sition-viiiiii wwweeiie-eeeeeeeeee -ees-dsin gn
C.1 Implementation Details
We conduct all experiments GPU, For implementation of bascline data valuation methods, we use the OpenData Val package [30] version 1. We the default hyperparameter settings for all methods except for Data Shapley (changed 100 Monte-Carlo epochs with 10 models per iteration), Influence Subsample (from 1000 to 500 models), and Data OOB (from 0000 to 500 models) to reduce computational runtime.
Intel Xeon E5-2620 CPU with 40 cores and Nvidia GTX 1080 Ti
In our experiments, we use the following setting of hyperparameters for DAVED:
500 iterations for multi-step variant, 1 iteration for single-step variance
Line search for step size a E (0, 0.9)
Regularization = (unless otherwise specified)
No early stopping
For experiments with heterogeneous costs, we uniformly sample costs e 21,2,3, 4,5] for each seller datapoint and apply either l(()) Ve l(()) c² as the cost function. Noisy labels are generated by adding Gaussian noise scaled inversely proportional the cost, with overall noise level set to 30%.
18

---

C.2 Dataset Details and Processing
For Fitzpatrick and RSNA Bone Age datasets, each image was embedded through CLIP ViT-B/32 model [4)] while for the DrugLib dataset, each text review was embedded through GPT-2 model 48 with max context length of 4096.
For the Gaussian dataset, we generate regression dataset according to the following Python code:
import numpy as np
def get iutssgausiiann data (num samples=100, dim=10, noise=0.1, costs=None)
np. randenn waandn..oormal.. normal (size= (num samples, dim))
X /- np. mp.linalg.nomi
axis=1, keepdims-True)
if costs is not None:
X costs
coef nn.random. exponential (scale-1, size-dim)
coef x= np. sign (np. random. uniform (1ow=-1, high-1, size=dim))
10
= X @ ccef + noise * np.randem, randn (num (aamm__smlles return dict (X-X, y=y, coef=coef, ncise=noise, dim=din, costs=costs)
The RSNA Pediatric Bone Age Challenge (2017) dataset 25 may be down- loaded
here
https:/ www. rsna. org/
rsna selaa-pediaee atria wwwiaaatrii--e-eeeeeeee ae-e---oonneeeeeheeee cha Secchalleegge lenge-2nee 2017 We use the training set for our ex- periments, resulting 12,611 images in total. Using following function, each image was
embedded through a pre-trained CLIP ViT-B/32 model.
import clip import torch from PIL import Image
def mmbed_imaged (img__ppeee paths, device="cuda"):
model, preprocess clip.load ("VIT-B/3"", device=device)
inferencc_func bodel.encode_imeee
embecdings = [] with for img_path in tqdm (img_paths)
():
img Image.open (img_path)
12
embedding inference_iune (preprocess (img) [None] to (device)
13
embeddings append (embedding. ())
return torch. cat (embedaings)
The Fitzpatrick 17K 24 can be downloaded from here ht / com wtt..//woom/mo.o.o mattgroh/
fitzpatrick17k Missing or corrupted images were excluded, resulting 16,536 total images. Some images were annotated with two separate annotation platforms. We averaged the two skin type ratings for these images, resulting 12 possible labels (0-6 in 0.5 increments). Each image was embedded in fashion similar to the RSNA Bone Age dataset.
woget.org//ce cong/ntnttt//// mimiciii// 47) The task is to predict the length of stay (LOS) in the number of days a pa- tient stays in the Intensive Care Unit. The dataset contains 51,036 rows with both real-valued and one-hot-encoded attributes with the following names:
The MIMIC dataset [3I]
be accessed here https: hhttps://hhys:
"LOS", "blood" "circulatory"" "congenital"" "digestive" "endocrine" "gen- itourinary". "infectious" "injury" "mental" "misc" "mmucullar" "neo- plasms" "nervous". "pregnancy" "prenatal" "respiratory" "skin" "GEN- DER" "ICU" "NICU" "ADM ELECTIVE". "ADM EMERGENCY"
"ADM NEWBORN". -URGETT "INS Government" "INS Medicaid" "INS Medicare" "INS Private" "INS Self Pay" "REL NOT SPECI- FIED" "REL RELIGIOUS" "REL UNOBTAINABLE" "ETH ASIAN"
"ETH BLACK/AFRICAN AMERICAN" "ETH HISPANIC/LATINOO" AAGEmmiddle OTHEER/NKKOWN"" "ETH_WHITE" "AGE middle miiddle_dult""
youun_aduut", adult", "MAR DIVORCED" "MAR LIFE PARTNER" "MAR MARRIED* "MAR "MAR_SEPARAA SEPARATED"
AGG__nebbrn" "AGE_senior" LLGE__yung__S
"MAR SINGLE", UAKKNWN (DEFAULT)". "MAR WIDOWED"
19

---

Each attribute was min-max scaled to lie in the range [0. 1].
The DrugLib dataset
be downloaded here https: //rchhiciee ics uci eduu ib+com 'The task is to predict overall pa- tient satisfaction with drug's side effects and effectiveness ten-point scale. We format each review using the following prompt template to feed GPT-2:
34
dataset/461/drn /e0/11/drrt/eee Sarugrevvewwte www.eeeewttttttteeeeeeeee.ee
Benefits: $BENEF IISIIITS_IIII
Side effects: $SIDE_ET EFF BETS_REVIIEE Comments: $COMMENTS_RENT REEVIEW
where SONNEFITSRREVEE is the corresponding portion of the drug review.
import terch
from transformers import GPT2Tokenizer, GPT2Mode1
def embed_text feext_inpput [itst device=*cuda")
tokenizer GPT2Tokeni zer. frem prem_p ned (model_name)
model GP12Mode1 wrwoooprerraineeee (model _aadd___mmm)))) to (device) ce)
embeddings []
for x in Eqdm(text inputs):
inputs tokenizer (x, return "ituun_tensoos="p0 truncation=True) to
device)
10
with torch. norchan ()
I1
outputs - model (*+inputs)
12
embeddings appeddd
outputs. hoist_hidden_sttee mean (dim=1) .cpu
mean
15
return torch. cat (embeddingg))
C.3
Evaluation Protocol
For all experiments (unless otherwise specified), we use the following evaluation settings:
Training split: Variable number of seller points (1K-100K)
Validation split: 100 points for baseline data valuation methods
Test split: 100 buyer points evaluated independently
Metrics: Mean squared error averaged over all buyer points
Budget ranges: 1-30 for cost experiments, 1-150 otherwise Number random seeds: All results averaged over 3 seeds
For a fair comparison, we use the same train/valin trann/validation/eestt splits across all methods for each random seed. The validation set is only used by data valuation baseline methods that require it our method DAVED operates directly on the test queries without requiring validation data.
20

---

D Additional Experiments
D.1 Budget Results with Heterogeneous Costs
Ours (multi-step)
DVRL
LAVA Random
Ours (single step)
KNN Shapley
Data OOB
Gaussian h(c) 2²
Gaussian h(c) vcc
Buyer eгrог
Buyer 3
error
85
30
10
15
25.
36
Max budget
Max budget
Figure 6: DAVED maintains performance advantage with heterogeneous costs synthetic data. We evaluate data selection methods on 10K synthetic Gaussian datapoints with costs randomly sampled from {1, 2, 3, 5,5) and transformed using cost functions: /e (left) and c² (right). Each datapoint's 's label noise is scaled inversely with cost (s/c;) to simulate quality differences. DAVED achieves lower test MSE across budgets 1-30 while respecting cost constraints, demonstrating robust performance even with varying data quality and costs. Results are averaged over 100 random buyer test points.
Ours (multi-step)
DVRL
LAVA Random
Ours (single step)
KNN Shapley
Data 0OB
MIMIC h(c) c2
MIMIC h(c) vT
700
Buyer error
Buyer error
300
310
200
100
10
20
10 16
36
Max budget
Max budget
Figure 7: Comparing performance MIMIC healthcare data with heterogeneous costs. Experiments on MIMIC-III dataset (35K patients with 48 normalized clinical features) compare performance under cost functions: /e (left) and c² (right). Costs uniformly sampled from 1, 2, 3, 4, 5 per datapoint. Label noise s ~ (O, a²) is scaled by T/h(c,) where ğ is the mean label value and h((c) is the cost function, modeling how higher-cost data has lower noise. DAVED achieves lower prediction error for length-of-stay prediction across budgets 1-30, Results averaged over 100 random test patients.
21

---

(multi-strp Ours (single step)
KNN Shapley
Random
Data OOB
Fitzpatrick h(c)=
Fitzpatrick =VE
Buyer
Buyer error
error
Max budget
Max budget
Figure Comparing performance on Fitzpatrick skin lesion dataset with varying costs. Evaluation FitzpatrickI7K 17K dataset (15K images embedded through CLIP) under two cost functions: ve (left) and c² (right). Costs sampled from 1, 3, 4, with noise & ~ .N (0, o²) scaled by ÿ/ j/h(c,), where ÿ is the mean Fitzpatrick score and h(c) is the cost function. This scaling ensures higher- cost images have proportionally less label noise. DAVED achieves consistent performance while respecting heterogeneous constraints. Results averaged over 100 random test images.
(mulli-step)
Ours (single step)
KNN Shapley
Random
Data 0OB
Bone age h(c)=
Bone age h(c) = v'c
Buyer error
Buyer
2300
erfor
3000
2007
2500
2003
2000
1507
1300
1002
1000
503
Max budget
Max budget
Figure 9: Comparing performance on RSNA bone age prediction with cost variations. Analysis on RSNA Bone Age dataset (12K X-ray images embedded via CLIP) using cost functions vc (left) and 2² (right). With costs sampled from 2, 4, 5 and label noise & ~ N'(0, o²) scaled by ū/h(c,). where ÿ is the mean age and h(c) is the cost function. This models how higher-cost X-rays have more accurate age labels. DAVED maintains superior prediction accuracy compared to other data valuation baselines. Results show mean error over 100 random test X-rays.
22

---

DAVED
OOB
LAVA Random
(single step)
Influence
DVRL
KNN Shapley
Druglib =
Drughib =
32
Buyer eтror
Buyer error
27
23
13
10
Max budget
Max budget
Comparing performance on drug review text data with heterogeneous costs. Evalu- ation DrugLib reviews (3.5K reviews embedded using GPT-2) under cost functions /e and c². Costs sampled from 2, 4, 5 with noise & ~ NNO,, o²) scaled by T/h(c,). where is the mean rating and h(c) is the cost function, reflecting how higher-cost reviews tend to have more reliable ratings. DAVED achieves robust rating prediction performance despite varying data quality. Results averaged over 100 random test reviews.
Figure 10:
D.2 Regularization
MIMIC
DrugLib Reviews
RSNA Bone Age
200
12
800
11
10
Test
180
810
error
120
700
80
Budget
Budget
Budget
Amoung of Regularization
0.2
0.4
0.6
0.8
1.0
Regularization strength significantly impacts DAVED performance across domains. We evaluate prediction error versus budget as À varies from 0 (no regularization) to (identity infor- mation matrix) on: MIMIC dataset (length-of-stay prediction). DrugLib reviews (rating prediction with GPT-2 embeddings), and RSNA Bone Age (age prediction from X-rays using CLIP embed- dings). Moderate regularization (A between 0.2-0.6) improves stability and performance by better conditioning the information matrix, though DAVED remains effective even without regularization. Results averaged over 100 random test points per budget level.
Figure 11:
23

---

D.3 Amount of Buyer Data
Caussian
MIMIC
RSNA Bone Age
1500
2.0
Test
0.5
10
15
20
25
0
25
50
75 Budget
100
125
25
50
75 Budget
100
125
Budget
Number of Buyer Datapoints
Figure 12: Buyer test size affects prediction performance of DAVED. We vary the number of simultameous test points (1 to 32) being optimized over for: Gaussian synthetic data (30 dimensions), MIMIC clinical data (48 dimensions), and RSNA Bone Age X-rays (512 dimensions). While all buyer and seller data follow the same distribution within each dataset, increasing test batch size leads higher prediction errors as the optimization problem becomes more challenging. Results suggest keeping buyer queries to 1-8 test points for optimal performance. Each curve shows mean error over 100 random trials with different sets.
D.4 Number of Iteration Steps
Caussian
Bone Age
1250
Test error
0.5
0.0
500
40
60
80
100
120
140
100
120
140
Budget
Budget
Number of Optimization Steps
5
10
50
100
500
1000
Figure 13: Number of optimization steps impacts DAVED performance. We evaluate prediction versus budget while varying optimization steps from 1 to 1, 000 on Gaussian data (30 dimen- sions) and RSNA Bone Age dataset (embedded through CLIP). More iterations generally improve performance as the algorithm better approximates the optimal selection weights. For homogeneous costs (c, = 1), we recommend using 2-5 times more optimization steps than the desired budget to ensure convergence. Results show mean error over 100 random test points per configuration.

---

D.5 Linear Feature Space Approximation
16-
Lisear probe (DAVED)
Fanetniin (DAVED)
Fanetaning (Random))
Test 20-
Error
Budget
Figure 14: Linear models in feature space approximate full model fine-tuning, We compare BERT performance on DrugLib reviews using either full fine-tuning (which updates all model parameters) linear probing (which only trains linear layer on frozen embeddings) on data selected by DAVED versus random selection. The similar performance patterns between these approaches empirically validates use of kernelized linear regression in the feature space proxy for the full training dynamics. This aligns with recent theoretical work showing that fine-tuning of pre-trained models is well-approximated by linear models in the empirical Neural Tangent Kernel (cNTK) regime. Since our method relies on this linear approximation for efficient data selection, these results support choice of feature extractor & and linear modeling approach. Results show test error averaged over 100 random trials using different test reviews.
D.6 Iterative versus Convex optimization
3.0
Data selection algorithm
2.5
Frank-Wolfe (multi-step)
Frank-Wolfe (single step)
Test error
Convex opl.
Data OOB
1.5
DVRL
1.0
LAVA
KNN Shapley
0.0
5
10
20
25
Training data points
Figure 15: DAVED's iterative optimization matches convex solver accuracy. We compare the prediction error of Frank-Woil optimization with convex optimization solver on 1, 0000 datapoints sampled from 30-dimensional Gaussian distribution. Both single-step and multi-step variants of iterative approach achieve comparable accuracy to the optimal convex solution while being significantly faster to optimize. Results averaged over 100 random trials with homogeneous costs (c, 1).
25

---

CVXPY
Frank-Wolfe (multi-step)
Frank-Wolfe
(single step)
10-3 3
10
10
10"
101
10a
10a
Runtime (seconds)
Figure 16: DAVED provides orders of magnitude runtime speedup over convex optimization. Runtime comparison between Frank- Wolfe iterative approach and convex solver for the data selection problem 1000 datapoints from 30-din 30-dimensional Gaussian distribution. Our iterative method achieves several orders of magnitude speedup while maintaining similar levels of prediction error (see Figure 15). The single-step variant provides additional acceleration with minimal performance loss
E Broader Impacts
We believe that AI developers face important ethical questions when acquiring data for Al develop- ment, as highlighted by recent class-action lawsuits against Al companies regarding data consent and compensation. Our work presents scalable and decentralized approach data acquisition that can help address these concerns. By enabling targeted selection of the most valuable datapoints, our method reduces both costs and potential privacy risks compared to broad, indiscriminate data access 44 The decentralized nature of our approach enhances transparency and gives individual data owners more control over how their data shared and used, while avoiding the privacy and security risks of centralizing data with brokers. However, while our method enables more efficient and privacy-pres privacy-preserving data transactions that could democratize access to high-quality training data, especially in domains like healthcare, several societal and technical challenges remain. These include ensuring fair compensation for data owners, preventing misuse of acquired data, and developing appropriate governance frameworks for decentralized data markets. Addressing these challenges will be crucial for the responsible development of data marketplaces that can sustainably increase the supply of training data while protecting individual privacy and data rights.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state our main contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We address the limitations of our methods in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
We provide proof' of all main theorems in Appendix 1 and Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide complete implementation details, hyperparameters, and experi- mental setup Appendix C Our code publicly available at at/ttps://github. com/
c clu5/data acquis s tion vi experr-eeeeriimmeee wwweeeettalddeisiooeee design
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: In the supplemental material, we provide a simplified code example and will release the full codebase public code repository upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines
cc/
pub Subblic/on CodeeSSu ss wwsssonoollii ollyy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. cc/ wttppp/ppppccccccccccccccccccclp i c/ des CodeSubri ss i onPol icy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide details of our experimental setup in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
All experimental results are averaged over 100 random test trials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Hardware specs and runtime analysis provided Appendix C and Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research complies with NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix E discusse both the benefits and risks of our work in the context of data marketplaces.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We believe this paper does not pose such risks as do not release any models or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite and provide links to all datasets used in Appendix
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaperswi paperswithcooes com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce any new datasets or models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subjects research was conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "hHNNyELl4O", "reviewer": "Reviewer_wuU5", "summary": "The paper introduces a new data valuation algorithm based on linear experimental design. The technique does not require a validation set and can be used in a federated setting.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper studies a well-motivated trending topic. Data valuation and data selection are important subfields in data-centric ML. I also like the motivation of developing validation-free data valuation algorithms a lot.", "weaknesses": "I have reviewed this paper at ICML. I think the new revision has addressed my previous concerns. The following is my ICML review:\n- The writing can be improved a bit. For example, in Section 3.3, the proof or citation for the test loss of LS estimator should be provided. In Section 3.4, I don't understand why the constraint of costs can be removed. Such derivation details are better to be explicitly written, it's okey to be in Appendix. I also don't see the benefit of using Frank-Wolfe algorithm.\n- The proposed approach requires a feature extractor, and I believe the performance depends greatly on the choice of feature extractor. Afterall, the theoretical justification of this appraoch is based on the assumption that the groundtruth is a noisy linear combination of the feature, that's why the requirement of test labels can be removed. It's good to explicitly acknowledge this point.\n- In Figure 3, why does the error significantly increases with full data for many curves? Are there a small portion of the data is being corrupted?\n\nIn this new revision, the author has clarified the linear model setup and the derivation is super easy to follow. The idea of using linear model as a proxy is straightforward but I am not aware of other works doing it. My remaining concern is that while this approach does not require a validation set, it requires a feature extractor. My major question is how this feature extraction can be picked in practice. I understand this is a hard question and the use of a public feature embedding is quite common in data selection techniques, therefore I will not reject the paper based on this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and the positive evaluation of our work. We agree that the choice of feature extractor is very important. In short, how well our linear proxy model works depends on the quality of our feature extractor. If there are features that are relevant to the task but are not present in the embedding output of the extractor, then the linearity assumption would not hold. Thus, we recommend general-purpose pre-trained foundation models that can extract very broad features from the input. We will add more discussion on the sensitivity of our results to the choice of feature extractors in the final version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYY2HE0agm", "reviewer": "Reviewer_BRC7", "summary": "This paper introduces a novel method for acquiring training data in a decentralized data market without requiring labeled validation data. Based on detailed and reliable theoretical support, the authors comes to a promising conclusion that the selection methods based on validation set has the performance as bad as the models trained by validation itself. Thus, it theoretically exceeds a series of methods represented by Shapley. This approach is amendable to federated optimization and reduce prediction error, making it more suitable for decentralized markets compared to traditional centralized data valuation methods.", "questions": "Q1: Is it reasonable to assume that the training data in theorem1 is infinitely large?\nQ2: If the assumptions about data distribution in Line 127 of Section 4(w2) cannot hold true, how does this proof work through V-optimal experiment? Or can you explain why this assumption always holds true even with data quality problems?", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "S1: This paper combines actual scenarios in the data market to summarize challenges that may exist for sellers and buyers, providing detailed and precise definitions for the data acquisition task. The overall topic is very interesting and practical.\nS2: The writing level is very good. Most of the presented viewpoints are well motivated, and the proofs, derivations, and theoretical implications of the formulas mentioned are all well explained and easy to understand.\nS3: For data acquisition methods relied on validation data, this paper theoretically deduces the lower bound of its error and proves that validation data(or labels) itself is unnecessary. This is crucial for the field of data acquisition where labels are very insufficient.\nS4: The method has good scalability, with sufficient experimental validation.", "weaknesses": "W1: The input for data acquisition task is too idealistic. In the actual data market, existing data acquisition tasks are often combined with data discovery, emphasizing the retrieval of data from environments such as data lakes that have different distributions and poor data quality. This paper simplifies the input data as datapoints, lacking consideration for issues related to data quality and failing to explain why datapoints are directly considered without integrating them into the context of the data market.\n\nW2: Although this article provides detailed and solid theoretical deductions, some assumptions are too strong. For example, in Line 127 of Section 4: \" First, we assume that the conditional distribution D_(y|x) is identical across Z^Train and Z^Test\". In fact, due to the presence of a large number of missing values in the train set, assumptions about data distribution may not hold true. If these assumptions are not valid, the explanation provided in the article regarding V-optimal experiment is insufficient, which makes me confused.\n\nW3: The proof of theorem 1 in the article is interesting, but some details need to be discussed if we want to use this conclusion to overperform methods based on Shapley Value. The existence of parameter σ makes it difficult for me to clearly understand how small this lower bound is. If users have some high-quality validation data (and in practical scenarios of data markets, users should provide high-quality data as much as possible), would this lower bound be acceptable?\n\nW4: The assumption of infinite training data in theorem 1 is not very reasonable. From my experience, the data available in the data market should be a subset of the results from the data discovery task and joinable subsets of datasets in the data lake, with its scale yet to be examined.\n\nW5: The application of kernelized linear regression is not well motivated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank the reviewer for their close reading of our work, their detailed feedback, and their enthusiastic support. We address the comments raised below.\n\n> W1: The input for data acquisition task is too idealistic ...\n\nWe absolutely agree that a truly practical implementation of our method for data markets will involve a lot of very exciting large-scale database research questions including data discovery. As we show, our arguably simplistic model is still very useful to discover important data valuation methods. We hope that our work inspires the database community to integrate data valuation methods such as DAVED to develop real-world data markets. \n\n> W2: ... missing values in the train set, assumptions about data distribution may not hold true...\n\nFirst, note that some assumptions on the data distribution are necessary. If the train data may be arbitrarily unrelated to the test data distribution, then the only meaningful training data selection strategy is random sampling. Further, our experiments show that when using a high-quality feature extractor, even if there are data quality issues such as missing values (as in the MIMIC dataset) or slight violations of the assumption, our method still has excellent performance.\n\nHaving said this, in practice, we imagine that there are instances where our covariate shift assumption is strongly violated - either due to features missing not at random (MNAR) as noted by the reviewer or due to other heterogeneous effects. We believe this represents excellent future research directions which we hope the data valuation community will build upon.\n\n\n> W3: The existence of parameter σ makes it difficult for me to clearly understand how small this lower bound is. If users have some high-quality validation data (and in practical scenarios of data markets, users should provide high-quality data as much as possible), would this lower bound be acceptable?\n\nLet us compare two different strategies users might take. In strategy 1, they will use the validation dataset of $n_{val}$ datapoints to use Shapely value-like methods to select the training data and enjoy some performance worse than $\\frac{\\sigma^2 d}{n_{val}}$ according to our lower-bound. \n\nIn strategy 2, they will completely ignore the training data and simply use the $n_{val}$ validation data to directly train a linear regression model. The error achieved by this strategy is $\\frac{\\sigma^2 d}{n_{val}}$ (see Proposition 3.5 in [Bach 2024](https://www.di.ens.fr/~fbach/ltfp_book.pdf)). Comparing this to the lower bound above tells us that the user may be better off directly using their validation data to train a model, instead of submitting it to the platform which uses Shapley-value based data selection. We hope this conveys the strength of our lower bound and will include this discussion in the final version of our paper.\n\n\n> W4: The assumption of infinite training data in theorem 1 is not very reasonable.\n\nThe assumption that there is infinite training data is only to make the math simpler. We only assume this to guarantee that there is training data of sufficient diversity to select from with high probability. If we had very few training data points, just by random chance, we might end up in a setting where all of them are very similar, making the data selection problem easier. As shown empirically in Fig. 1, 1000 data points are sufficient to satisfy this condition in practice and for prior methods to work comparably to random selection.\n\n\n> W5: The application of kernelized linear regression is not well motivated.\n\nWe study the kernelized linear regression setting mostly because it is the only case that is theoretically tractable to analyze. Further, numerous past works have shown that when using a high-quality feature extractor, this kernelized linear proxy model is a good approximation of the full training process (lines 138--143). We also show this empirically in our setting by the performance of our method, and also in Appendix D.5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "plmeRX6ofb", "reviewer": "Reviewer_MmPt", "summary": "This paper uses the linear experimental design to tackle the data acquisition problem.", "questions": "1. Could authors explain how to get the eq.(7) derived from the eq.(4) ?  \n\n2. $\\textbf{confusion part:}$ How $g_i$ in eq.(7)  can be only calculated by the seller $j$ using training data $x_j^{\\text{train}}$, given $x_i^{\\text{test}}$ is needed( or $\\mathbb{E}(x^{\\text{test}})$ appears in eq.(7) ) ? \n\nLine 176:  \"$g_i$ as well as the update (6) can be trivially computed by seller $j$ using only their data $x_j^{\\text{train}}$ (and the test data). \" \n\nLine 516:  \"our method and Data OOB do not use the validation set\" \n\n$\\textbf{Question:}$  Whether optimization in eq.(7) involves the test dataset? If it involves, I guess some baselines could also do the same process for data selection : for example,  LAVA could also only use  feature space $\\mathcal{X}$ to compare $x^{\\text{train}}$ and  $x^{\\text{test}}$, and select data points?  If not, back to the question of the confusion part.\n\n It will be better to provide more details about your analysis or related works for references in the paper for better understanding. ]\n\n3. In the experiment setting, the validation set has only 100 datapoints, could authors conduct the experiments when the validation size increases ?  \n\n4. Authors consider low-budgets situations ( 1~10 datapoints ) , it will be better to provide experiments when budgets are larger ?\n\n\n————————\nAll my concerns have been addressed. I will raise the score.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper tackles a very challenging but important problem in the data marketplace : evaluating data quality before data transactions as well as minimizing the test error.  The main contribution is the Federated approach for the experimental design.  \n\nMotivation is clear and paper is well-organized. The experimental results are convincing.", "weaknesses": "1. Overall I like this paper and take a few hours trying to understand the technical part and experimental results. But some necessary analysis and equations are simplified, not so friendly to readers who are not very familiar with the V-optimal experiment design framework and some optimization parts. Please provide some preliminaries in your work. \n\n2. line 136 .. suppose we have a known feature-extractor ...  how to get such good extractor $\\phi$ and whether your evaluation results are sensitive to a poor $\\phi$? \n\n3. The authors consider the low-budgets cases ( lower 50 training datapoints ) and the validation size for baselines is small. More experiment results are needed. \n\n4. Some parts are confusing for me, please see the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We agree that our exposition is dense and some derivations may be inaccessible to reader unfamiliar with experiment design. We will add a more detailed derivation in the Appendix where we will go through all the steps in more detail. Further, we will aim to rewrite in a manner such that despite some of the details being inaccessible, the high-level intuitions and resulting algorithm is accessible to all readers. Having said this, we adress the main concerns raised by the reviewer below.\n\nAlso, we would like to stress that the **most useful and challenging setting** for data valuation methods is when the budget is low, and the validation data size is small. This is why we focus on this setting. We believe this is a strength of our work, not a weakness - the other settings are strictly easier. With a large enough budget, even random selection can work. Similarly, if we can collect a large clean validation dataset, the need for a data market is less justified.\n\n> [...] how to get such good extractor $\\phi$ and whether your evaluation results are sensitive to a poor $\\phi$? \n\nWe note that our experiments on Gaussian and MIMIC data are conducted without any feature extractor. In general, it is common to use a pretrained model as a feature extractor but we will provide additional discussion of the choice of feature extractor in the final paper.\n\n> Could authors explain how to get the eq.(7) derived from the eq.(4) ?\n\nEq 7 is the gradient update for a single datapoint $x_j$. The seller that owns the datapoint $x_j$ can calculate its gradient using the buyer data $x_i^\\text{ test }$ and the information matrix $\\mathcal{I} := \\sum_{j=1}^{n} w_j x_j x_j^\\top$.\n\n> \"our method and Data OOB do not use the validation set\"\n\nWe meant that while other methods require a validation set to estimate data value, our approach (and OOB) does not use a held-out validation set to perform data selection.\n\n> Whether optimization in eq.(7) involves the test dataset?\n\nYes, optimization uses the unlabelled test datapoint $x^\\text{ test }_i$. That is, our method is adaptive to the test data point - the train data most relevant to the specific test data is selected. While our focus was to develop a theoretically justified algorithm with provable guarantees, we hope our work will inspire future empirical work that similarly explores selection based on the feature space (including ones based on LAVA). In our current work, we compare against LAVA as described in the paper which requires class labels.\n\n>  In the experiment setting, the validation set has only 100 datapoints, could authors conduct the experiments when the validation size increases ? \n\nOn our datasets, we did not find much difference with larger validation sets but we will include more experiments in the revised paper. Also note that as the size of the validation dataset increases, the motivation for trying to select additional training data decreases. Data valuation methods are most useful when the validation dataset is substantially smaller than the training dataset.\n\n> Authors consider low-budgets situations ( 1~10 datapoints ) , it will be better to provide experiments when budgets are larger ?\n\nOur real-world experiments (Fig. 4) use a budget up to 150. We found that past this, the error did not decrease anymore - our method was already filtering out the irrelevant data and selecting the most useful ones. Note that given a high enough budget, even random selection will perform well since eventually all relevant data will get selected. Hence, we decided to focus on the low-budget case which is both the most useful and most challenging. We will add the experiments with higher budgets as well to the appendix.\n\n\nWe believe all the concerns raised by the reviewer are addressed above. If so, we again thank the reviewer for their detailed feedback and request them to kindly re-evaluate our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ggm99VTR0T", "reviewer": "Reviewer_yTne", "summary": "The paper focuses on the problem of data acquisition in decentralized data marketplaces. The paper claims that acquiring training data using validation-based data valuation methods can be overfitting. Thus, the paper proposes DAVED, which directly selects training data by optimizing test loss on a given test set rather than relying on the validation set. DAVED approximates the test loss trained on a subset of the training data using a linear experimental design framework and employs gradient-based optimization to compute the sampling probability of each data point, considering budget constraints and decentralized settings. Experiments on synthetic and real-world datasets are conducted to validate the advantages of DAVED.", "questions": "(D1) The paper claims that existing data valuation methods are misaligned with the data acquisition problem and introduces DAVED to select data points based on the test set rather than the validation set. In the experiments, the paper compares DAVED with a series of data valuation methods including Influence, Data Shapley, and Data OOB, showing the advantages of DAVED. However, the paper does not discuss or compare its performance with methods that focus on data acquisition problems with or without a validation set. In addition, given that DAVED is designed for decentralized data marketplaces, the paper does not discuss or verify how it compares with existing methods in federated learning, such as [62].\n\n\n(D2) The paper does not conduct experiments to analyze the impact of different test data distributions and the training data distribution. The paper should include experiments to measure the performance of DAVED under the scenario where there is a distribution shift between the training and test sets.\n\n(D3) The paper uses an example to illustrate their setting: \"the buyer may be a patient who cares about diagnosing their own chest X-ray and is willing to pay a fixed budget to access other X-ray images to train a model to have low error on their 'test set' (see Figure 1).\" However, this example may be impractical as it is unlikely that a patient would train a diagnostic model themselves rather than relying on a hospital or healthcare institution. If the data acquisition is indeed for a hospital, the test data for the hospital cannot be fixed because patients are continually coming in. From my understanding, Algorithm 1 is designed for a fixed size of test data. This creates a gap between the paper's setting and real-world scenarios. The authors should either clarify this gap or explain if I have misunderstood the motivation or algorithm.", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "(S1) The paper studies an interesting and important problem.\n\n(S2) The paper proposes a novel approach for data acquisition in decentralized data marketplaces.\n\n(S3) Experiments show the effectiveness of the proposed approach.", "weaknesses": "(W1) Lack of comparison with data acquisition approaches.\n\n(W2) Lack of experimental evaluation on the test data distribution.\n\n(W3) The motivation could be better clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank the reviewer for their close reading of our work, their detailed feedback, and their enthusiastic support. We address the comments raised below.\n\n> [...] compare its performance with methods that focus on data acquisition problems\n\nFirst, we want to emphasize that our method uses an *unlabelled* test dataset whereas the previous methods need a labeled validation dataset. In this paper, we tried to show the limitations of such data valuation approaches but we agree that comparing against other federated client selection methods will be important for more realistic decentralized market scenarios. As we note in our limitations section, adapting our methods to a realistic communication-constrained federated learning setting is an important future direction we hope to explore. \n\n> [...] distribution shift between the training and test sets\n\nNote that we make no distributional assumptions on our data. In fact, this is one of the biggest strengths of our method. Both our derivation in Step 2 (line 149) as well as our guarantee in Theorem 2 holds for an arbitrary test dataset - it may be completely different from the training dataset. The only assumption we make is on that the *conditional distribution* $Y | X$ is the same  i.e. the same image will have always the same label (an image with a cat will not be labeled as a cat in the test and as Felis genus in another). Our experiments show that even if there are data quality issues (as in the MIMIC dataset) our method based on this assumption has excellent performance.\n\nHaving said this, there are certainly scenarios where our assumption may be violated e.g. when we have features missing not at random (MNAR), or other heterogenous effects. Formalizing and extending our method to such scenarios is a very promising direction for future work.\n\n\n\n> [...] test data for the hospital cannot be fixed because patients are continually coming in.\n\nIn this example, the test data consists of only an individual incoming patient's X-ray which requires a prediction. The data-market platform selects the data and trains a model to predict that patient's data and only shares the final model/prediction to the patient. The selected data points are not directly shared. As correctly identified by the reviewer, this means that the procedure needs to be *re-run* for each new test patient. \n\nWe stress that only the model/prediction is shared and not the data selected. This is because of data privacy and IP issues. Since data is infinitely and easily copyable, it is not feasible to run a marketplace where raw data is ever shared - the buyer can easily make a copy and resell for a lower price. Thus, in our model, only the trained model/prediction is shared.\n\nWe once again thank the reviewer for their great feedback of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of selecting valuable data points in a decentralized data marketplace, particularly focusing on domains where data is scarce. The authors propose a federated data acquisition method inspired by linear experimental design, which does not require labeled validation data and instead optimizes data acquisition directly based on test set performance. The method is designed to function efficiently in a decentralized market setting, showing state-of-the-art performance on several real-world medical datasets.

Overall, all reviewers are positive about the paper, appreciating the timeliness of the research question, the novel approach, and the clear writing. There are some reservations, such as the requirement for a high-quality feature extractor, assumptions about data distribution that may not always hold true, and a limited scope regarding budget constraints in practical marketplaces. However, the strengths of the paper clearly outweigh the weaknesses. We recommend accepting the paper and encourage the authors to incorporate the reviewers' comments in their final camera-ready version.
Decision: Accept (poster)