Title: Data Acquisition via Experimental Design for Data Markets
Venue: NeurIPS 2024
Accepted: yes
Keywords: Data valuation

experimental design

data markets

data acquisition

federated learning
Abstract: The acquisition of training data is crucial for machine learning applications. Data markets can increase the supply of data, particularly in data-scarce domains such as healthcare, by incentivizing potential data providers to join the market. A major challenge for a data buyer in such a market is choosing the most valuable data points from a data seller.  Unlike prior work in data valuation, which assumes centralized data access, we propose a federated approach to the data acquisition problem that is inspired by linear experimental design. Our proposed data acquisition method achieves lower prediction error without requiring labeled validation data and can be optimized in a fast and federated procedure. The key insight of our work is that a method that directly estimates the benefit of acquiring data for test set prediction is particularly compatible with a decentralized market setting.

Data Acquisition via Experimental Design for Data Markets

Abstract

The acquisition of training data is crucial for machine learning applications. Data markets can increase the supply of data, particularly in data-scarce domains such as healthcare, by incentivizing potential data providers to join the market. A major challenge for a data buyer in such a market is choosing the most valuable data points from a data seller. Unlike prior work in data valuation, which assumes centralized data access, we propose a federated approach to the data acquisition problem that is inspired by linear experimental design. Our proposed data acquisition method achieves lower prediction error without requiring labeled validation data and can be optimized in a fast and federated procedure. The key insight of our work is that a method that directly estimates the benefit of acquiring data for test set prediction is particularly compatible with a decentralized market setting.

1 Introduction

While massive training datasets enable major machine learning breakthroughs, they remain largely inaccessible outside of large companies, motivating mechanisms for broader data access. A related point is that many data owners have become resistant to having their data collected indiscriminately without their consent or without their participation in the fruits of predictive modeling, resulting in legal challenges against prominent AI companies [16, 33]. These trends motivate the study of data marketplaces , which aim to incentivize data sharing between sellers, that provide access to data, and buyers, that pay compensation for data access [11, 1, 54].

For practical data acquisition scenarios, a data buyer has a specific goal in mind and, in particular, wants training data to predict their test data in a specified context. Accessing different datapoints may require varying prices associated with each datapoint, which may reflect heterogeneous cost, quality, or privacy levels for each datapoint [44, 40].

For example, consider a hospital that wants to make a prediction for a specific patient's X-ray. The hospital can submit this X-ray as an unlabeled test query to the marketplace, along with a budget to access relevant training data. The marketplace selects useful training data to build a model for this specific prediction task, sharing only the final prediction rather than the raw data to protect privacy. This process can be repeated for each new patient query (see Figure 1). However, not all X-ray images will be equally relevant. Thus, we want to select only those seller datapoints that are most useful for answering the buyer's query and fit the buyer's budget.

This goal of data acquisition has motivated the development of many data-valuation techniques (e.g., [23, 29, 38, 60, 52, 39, 57, 46, 30]). However, we argue that current data valuation techniques

Buyer wants diagnosis

Data market platform selects datafrom

Platform choosesmost

ontheirX-rayfor$100

sellersusingbuyer'sX-ray

cost-effectivedatapoints

Predicted:

90%

6pneumonia

Split $100 budget

Performinference

Train model using

betweendatasellers

onbuyer'stest data

selecteddata

are misaligned with the data acquisition problem, particularly in the context of data marketplaces. They all face at least one of the following limitations:

The selection process may not be adaptive to the buyer's (unlabeled) test queries, potentially failing to identify the most relevant data. In a data marketplace, buyers typically need to purchase only a small subset of datapoints most relevant to their test data, which may follow a significantly different distribution than the overall seller data.

When adaptive selection is implemented, these techniques rely on labeled validation data, which is often impractical. Further, when a small quantity of such data is available, the selection may overfit the validation data and result in poor performance on the test queries.

The algorithms are not scalable and typically require retraining the ML model numerous times. Hence, they are unable to select from realistic seller corpora ( > 100K+ datapoints).

Instead, we propose data acquisition via experimental design (DAVED) method that overcomes all of these limitations. Unlike most previous work in data valuation, our approach does not require a labeled validation dataset and instead directly optimizes data selection for the buyer's unlabeled test queries.

Additionally, our approach accounts for budget constraints and is able to weigh the price of each seller's datapoint against its potential benefit, simultaneously solving the budget and revenue allocation problems [64]. Moreover, it can be implemented in a federated manner, achieving lower prediction error even compared to centralized baselines.

Our contributions are the following:

Formulate the data acquisition problem for data marketplaces and demonstrate that data valuation methods make a fundamental theoretical mistake of 'inference after selection' (Theorem 1).

Design a novel, highly scalable, and distributed data selection procedure that eliminates the need for validation data by directly selecting the most cost-effective seller data to answer the buyer's test queries (Algorithm 1).

Demonstrate state-of-the-art performance on synthetic data and medical data in various modalities (X-rays, images, and text).

1

5

10

15

20

25

Budget

0.0

0.5

1.0

1.5

2.0

Test

error

Dimension = 30   /   Validation = 100

1

2

3

4

5

6

7

8

Budget

0.0

0.5

1.0

1.5

Dimension = 10   /   Validation = 10

DAVED (ours)

Data OOB

Data Shapley

DVRL

Influence

KNN Shapley

LAVA

Random

2 Data Acquisition versus Data Valuation

In a decentralized data marketplace, data acquisition must be performed before full data access is granted to the buyer [36]. This relates to Arrow's Information Paradox [7] - sellers are unwilling to share data before payment, while buyers need to evaluate utility before purchasing. This distinction between data valuation and data acquisition for data marketplaces is also discussed in a recent data acquisition benchmark, where data value must be estimated without requiring white-box access to the seller's data (i.e., full, unrestricted access to the data) [12].

A more fundamental issue with validation-based data valuation approaches is exemplified by the Data Shapley value approach [23, 29, 38, etc.], which measures the marginal contribution of each training datapoint's improvement to a validation metric. They are great for after-the-fact attributing the relative influence of the training data. However, they cannot be used to make decisions about which datapoints should be included in the training. This is because of, as noted earlier, the 'inference after selection' issue. Using validation data to select training data leads to substantial over-fitting to the validation data.

Illustrative example: Suppose that we only have a single validation datapoint. Then, it is clear that we will select training data similar to this singular datapoint, and our selection has no hope of working on the test dataset. While this clearly demonstrates overfitting in an extreme scenario, we show in Figure 2 that increasing the validation set size does not circumvent this issue. We see that other data valuation techniques have poor test prediction errors-some techniques even underperform even a random selection baseline! This clearly demonstrates overfitting. Our proposed method maintains low test error as more seller training data is selected. In Section 3, we will dig in deeper into this phenomenon and prove a very strong theoretical lower bound. We show that any approach that relies upon validation data for data selection can perform as badly as throwing away all the training data and simply training on the validation data alone! This is especially true when our budget is small compared to the dimensionality of the problem, as is likely in a data market setting - the data is typically high dimensional, and we can only select a very small fraction of the total available data.

3 Setup and Limitations of Prior Methods

Description of Data Acquisition Setting. As shown in Figure 1, in our setting of data acquisition, a buyer has a budget and test data. The platform uses the buyer's data to select training datapoints from the seller that optimize buyer test error. The interaction proceeds as follows:

The buyer brings their test data X test = [ x test 1 , . . . , x test m ] and a budget B to the market platform. This test data is associated with unknown target labels Y test = [ y test 1 , . . . , y test m ] .

The platform also has access to n datapoints from data sellers Z train = { ( x j , y j ) } j =1 ,...,n and their associated costs { c j } j =1 ,...,n .

Given only the covariates X test , the platform assigns a selection weight w j to each datapoint ( x j , y j ) . This weight w j ∈ { 0 , 1 } represents the discrete action of selecting datapoint j .

The platform selects datapoints from the sellers according to w = ( w 1 , . . . , w n ) , trains a model f ˆ θ ( w ) , makes the predictions f ˆ θ ( w ) ( X test ) , and distributes c j · w j payment for each datapoint used in training.

In general, we do not make i.i.d assumptions between the train and test - we expect the test queries will not be similar to the total available train data. The goal then is to pick the weights w , which minimizes the prediction error for the buyer while adhering to their budget constraint ∑ n j =1 w j c j ≤ B . This gives rise to the following problem:

where l is squared loss. Here, the expectation is over the conditional label distribution of y test i | x test i , and the potential randomness of the algorithm. Note that this problem can not be solved because we do not know the targets Y test . Instead, we need to rely on a surrogate objective function (proxy) ˆ L ( w ) . One approach to constructing such a proxy is by using validation data.

Folly of Relying on Validation Data. In most data valuation methods, e.g., Data Shapley [23], the value of data is evaluated using a labeled validation set Z val = { ( x val j , y val j ) } n val j =1 . Implicitly, these methods assume that the known Z val is drawn from the same distribution as the unknown Z test . Then using this validation data, scores ( s 1 , . . . , s n ) are assigned to the seller training datapoints Z train . For two datapoints i, j ∈ train , the score s i > s j if the datapoint i is more valuable than j [46, 30]. More concretely, s i > s j implies that training with i would lead to a smaller validation loss than if j was used instead. Thus, these scores can used to select the most valuable datapoints.

However, note that we used the validation dataset to compute the scores. Thus, selecting the top-k scores results in implicitly minimizing the validation loss i.e., all validation-based data valuation schemes implicitly optimize the following proxy loss

This approach heavily relies on the quantity and quality of the validation dataset in order to generalize to the actual test dataset. In fact, we have the following minimax lower bound even when restricting ourselves to simple linear models.

Theorem 1 (Informal version of Theorem A.1) . Let w ∗ denote the solution of our original problem (1) and ˆ w solve (2) . Suppose that all our data Z train , Z val , Z test are drawn i.i.d. from some distribution D X,Y where D X is supported on B d R (zero-centered ball with radius R in R d ), and Y = θ ⊤ X + ε where ε is independent zero-meaned noise with variance σ 2 . For any training algorithm, when the number of training data is sufficiently large, with high probability,

This result implies that the expected test error for any validation-based approach can in the worst-case scale as d/n val with high probability. This dependence on the dimension d and the number of validation points n val highlights that this method may be suboptimal in high-dimensional settings or when the validation dataset is small. In fact, we would get the same error scaling if we threw away the training data and trained a model ˆ θ on the n val validation datapoints alone. This explains the striking overfitting we observed earlier in Figure 2. Furthermore, obtaining a large amount of ground-truth labeled data may be challenging in many real-world applications. Instead, we propose a validation-free approach to data acquisition based on experimental design.

4 Our Methods and Implementations

We propose an alternative approach based on a proxy objective. Our key assumption is that the conditional distribution D y | x : y = f θ ∗ ( x ) + ϵ is identical across Z train and Z test . This is a natural assumption in many domains - for instance, if an X-ray exhibits indicators of a specific disease, it should receive the same diagnosis regardless of whether it appears in the training or test set. Without this assumption, our problem becomes intractable since the same x j could map to arbitrarily different labels across train and test sets. Under this framework, we can reformulate our problem using V-optimal experiment design [47].

Step 1: Linearizing the problem. Our goal is to design a proxy loss function ˆ L ( w ) which approximates the true test loss L ( w ) . To do this, we have to reason about how different choices of training data S ⊂ Z train could impact the prediction on a particular test datapoint in X test . This is a notoriously challenging problem for general deep learning models [9]. Instead, we use a linear approximation and model the complicated training dynamics with kernelized linear regression. We suppose we have a known feature-extractor ϕ : X → R d 0 and an unknown θ ∗ ∈ R d 0 such that the data is generated as

where ε is independent noise with mean zero and d 0 is the embedding dimensionality. The function ϕ ( · ) can be the empirical Neural Tangent Kernel (eNTK) [27, 41, 58] of the model, or even the embeddings extracted from a deep neural network such as CLIP [49]. While this may be a bad approximation in general [61], a recent line of work has shown that such eNTK representation very closely approximates the fine-tuning dynamics of pre-trained models both theoretically [58, 42] as well as emperically [22, 63]. In fact, such linear approximations have also been used to speed up validation-based data attribution computations [46].

Step 2: Experimental design proxy. Given the assumption on our data from Eqn. (3), we can use the V-optimal experiment design framework [51, 47, 26] to define a proxy objective. First, suppose that S ⊆ ( ϕ ( X train ) , Y train ) is the subset selected by w and then we performed least-squares regression. The resulting estimate ˆ θ ( w ) can be computed in closed form as

Henceforth, we will drop the ϕ when obvious from context and simply use x . We can further use Eqn. (3) to compute the expected error on an arbitrary test query x 0 , y 0 as follows:

Here a 3 uses the unbiasedness of the ordinary least squares (OLS) estimator and a 4 plugs in the closed form of ˆ θ ( w ) and simplifies. With this, we end up with a very clean expression for the expected test error on an arbitrary point x 0 , and the matrix I ( w ) is known as the Fisher information matrix . While regression suffices for our use case, the procedure can be extended to general linear models. Dropping the fixed E ∥ ε ∥ 2 , we can use this to build our proxy function ˆ L ED ( w ) and arrive at the following optimization problem

This optimization objective directly measures how useful each training point would be for predicting the specified test query. The matrix I ( w ) captures how much information each selected datapoint provides about the test point in the embedded feature space.

Note that our proxy function ˆ L ED ( w ) can be computed using just X train , X test and does not even need access to training labels. Unfortunately, the objective in (4) is NP-hard to optimize [4]. We next see how to derive fast and provably good approximation algorithms for (4).

Step 3: Fast approximation. To make Eq. 4 amenable to gradient-based optimization, we drop the constraint that w j ∈ { 0 , 1 } and allow it to be a continuous positive vector i.e., w ≥ 0 and ∑ n j =1 w j c j ≤ B . With this relaxation, the proxy objective ˆ L ED ( w ) is continuous and convex in w [10]. We then run the 'herding' variant of the Frank-Wolfe algorithm [59, 37, 53, 8, 65]. To do this, define ( ˜ w t := w t / c ) for any t . We start from a ˜ w 0 = e 0 and iteratively update as 2

Note that if we use the step-size α t = 1 t +1 in (5), w t satisfies a special property at any iteration t :

Run the procedure until the last iteration t = t o for which we still have ∥ w t 0 ∥ 1 ≤ B . We can adapt the theory from [8, 28] to analyze the above procedure and show the following.

Theorem 2 (Informal) . Let us run Frank-Wolfe herding update (5) for t 0 steps such that it is last step which satisfies ∥ w t 0 ∥ 1 ≤ B . We use ˜ w t 0 = (( t 0 +1) w t 0 / c ) as our selection vector and we would have selected t 0 datapoints. Then, under some assumptions, we have

The above theorem shows that our continuous relaxation does not significantly affect the optimality of our result - we get O ( log t 0 t 0 ) close to the optimal solution to the original NP-hard (4). If all datapoints have equal cost c , then t 0 = ⌊ B/c ⌋ , and so our approximation quality improves as we increase the budget. While better approximation guarantees are attainable [3], their procedure is significantly more involved and is not easily amenable to efficient federated implementations as ours is.

Step 4: Efficient federated implementation. Our practical implementation directly restricts w ∈ ∆ n instead of ˜ w in the theoretical implementation above i.e., we run

This way w can be directly interpreted to be the sampling probability for different seller training datapoints. The bottleneck to efficiently implementing (6) is computing the gradient. At step t , the negative gradient can be shown to be

Thus, if we have the inverse information matrix I ( w t ) † pre-computed, g j as well as the update (6)

can be trivially computed by seller j using only their data x train j (and the test data). Next, we show how to efficiently maintain the inverse information matrix. Note that the update (6) has a special structure: all coordinates are shrunk, and then only a single coordinate of w t is increased. We can relate the resulting I matrices with a rank-one update as:

Define P t := I ( w t ) † . Wecan use the Sherman-Morrison formula [50] to compute P t +1 = I ( w t +1 ) † as

For each round t , this update only involves the current matrix P t = I ( w t ) † and the single datapoint x j t selected for the round. Thus, the seller can also locally compute this update as well as the updated cost ˆ L ( w ) as in Eq. (4) for any α t :

Algorithm 1 DAVED: Iterative Optimization Procedure

1: Input: buyer test datapoint X test ∈ R m × d , seller training data X ∈ R n × d , seller weights w ∈ ∆ n , iteration steps T , regularization parameter λ Reg ∈ [0 , 1] , and seller datapoint costs c ∈ R n +

2: w 0 ← ✶ /n # Initialize weight vector to uniform distribution 3: P 0 ← ( (1 - λ Reg ) X ⊤ diag ( w 0 ) X + λ Reg · σ X I n × n ) - 1 # Initialize P (Eq. 10) 4: for t ∈ { 1 , 2 , . . . , T } do

5: g ←-∇ ˆ L ( w t ) # Compute negative gradients (Eq. 7)

6: j t ← arg max j ( g j / c j ) # Select coordinate based on costs

7: α t ← LINE _ SEARCH L ) # Find optimal step size (Eq. 9)

( ˆ

8: w t +1 ← (1 - α t ) w t + α t e j t # Shrink weights and upweight the chosen coordinate (Eq. 6)

9: P t +1 ← SHERMAN _ MORRISON ( P t , x j t , α t ) # Update inverse information matrix (Eq. 8)

10: end for

11: Output: Sample seller data according to w T ∈ ∆ n without replacement until budget B runs out.

Thus, a line search can be performed to determine the optimal step size α t ∈ [0 , 1] to minimize the proxy loss as. This differs from (5) where we used a specific choice of α t . Frank-Wolfe is known to be more stable with the line search [53, 65]. The seller can communicate this α t and x j t to the platform to compute the updated P t +1 using only O ( d ) communication.

An additional practical consideration is that by initializing w 0 = c 1 e 1 , we have an ill-conditioned inverse information matrix P 0 . We instead use an initialization of w 0 = ✶ n /n ∈ ∆ n and further add a feature-wise regularization term. This makes the initial P 0

where ˆ σ i = √ 1 n ∑ n j =1 ( X ji -¯ X i ) 2 is the empirical standard deviation of feature i . The complete details are summarized in Algorithm 1.

Single-step variant. We can also forgo the iterative process and instead linearly approximate the cost function (Eq 4) with a single step that selects the top k datapoints under the budget B ,

This simplified version is extremely fast while still maintaining relatively good performance.

5 Experiments

We evaluate our proposed method for data acquisition (DAVED) against common data valuation methods on both synthetic data and four real-world medical:

Fitzpatrick17K [24], a skin lesion dataset, where the task is to predict Fitzpatrick skin tone on a 6-point scale from dermatology images.

RSNA Pediatric Bone Age dataset [25], where the task is to assess bone age (in months) from X-ray images of an infant's hand.

Medical Information Mart for Intensive Care (MIMIC-III) [31], where the task is to predict the length of hospital stay from 48 attributes such as demographics, insurance, and medical conditions.

DrugLib reviews [34], text reviews of drugs where the task is to predict ratings (1-10).

For validation-based methods, we use a validation set of 100 datapoints. We report mean test errors over 100 buyers. For more details on the experimental setup, see Appendix C. Our code is available at this repo: https://github.com/clu5/ data-acquisition-via-experimental-design .

Comparing Performance on Data with Homogeneous Costs. In Figure 3, we evaluate our method and several other data valuation methods on varying amounts of Gaussian data with homogeneous

0

5

10

15

20

25

Budget

0.0

0.5

1.0

1.5

Test

error

0

5

10

15

20

25

Budget

0.0

0.25

0.5

0.75

1.0

1.25

0

5

10

15

20

25

Budget

0.0

0.25

0.5

0.75

1.0

1.25

DAVED (multi step)

DAVED (single step)

Data OOB

Data Shapley

DVRL

Influence

KNN Shapley

LAVA

Leave One Out

Random

fixed costs. Compared to other methods, both multi- and single-step versions of DA VED have lower test errors across budgets on synthetic data. This performance gap is especially large when the buyer has a small budget (around 5-10 seller training datapoints). In Figure 4, we evaluate our method on real image and text data embedded through CLIP and GPT-2 feature representations. We observe that DAVED has better performance compared to most other baselines on all three datasets, highlighting that the proposed method is practical for embeddings of high-dimensional data. Table 1 summarizes our results on all datasets. For the Gaussian data and MIMIC datasets, we report the mean error of budgets from 1 to 10, while for the embedded datasets (RSNA, Fitzpatrick17K, DrugLib), we report the mean error of budgets from 1 to 100 in intervals of five.

Comparing Performance on Data with Heterogeneous Costs. Next, we compare methods on seller data with non-homogeneous costs. We uniformly sample costs c ∈ { 1 , 2 , 3 , 4 , 5 } for each seller datapoint and consider two cost functions, c j = √ c and c j = c 2 , which downweights gradient of that datapoint x j (see Equation 7). To simulate heterogeneous utility across datapoints, we introduce cost-dependent label noise, ϵ ∼ N (¯ y, σ 2 ) , to each datapoint ˜ y i := y i + β ˜ ϵ/c j , where ¯ y is the mean target value and β is the overall noise level, which we fix at 30% throughout our experiments. For these experiments, we did not evaluate Data Shapley [23], LOO [13], and Influence [21] that had very long runtimes. In Table 2, we report additional mean test error across budgets 1-30 for both cost functions. We find that our DAVID method is more budget-efficient in choosing cost-effective noisy datapoints than other methods across datasets. We provide additional plots for heterogeneous costs in Appendix D.1.

Comparing Runtime. In Figure 5, we compare the optimization runtime of our data selection method on 1,000 datapoints while increasing the dimensionality of the data as well as when the dimensionality is fixed to 30, and the number of seller datapoints is increased to 100,000. Data Shapley [23] and LOO [13] took too long to run for large amounts of datapoints or high dimensional data and are not reported. In both experiments, our multi-step compares favorably to efficiency-optimized techniques such as KNN Shapley [29] while our single-step method had the fastest runtime. This demonstrates that our method can scale to marketplaces with millions of datapoints.

Regularization Strength. In Appendix D.2, we vary the amount of regularization applied on the MIMIC, DrugLib, and RSNA datasets. We find that applying a moderate amount of regularization between 0.2 and 0.6 can lead to improved performance. Even when the information matrix is set to identity, i.e., λ = 1 , performance on the DrugLib datasets is still reasonable. Note that for all other experiments, we do not apply any regularization.

Amount of Buyer Data. In Appendix D.3, we vary how many buyer test datapoints are simultaneously optimized over on Gaussian-distributed, MIMIC, and RSNA datasets. While all buyer and seller data is sampled from the same distribution, the number of buyer datapoints still affects the optimization procedure. In general, we find that increasing the number of datapoints in the 'test batch' increases

0

20

40

60

80

100

120

140

Budget

0.5

1.0

1.5

2.0

Test

error

Fitzpatrick17K

0

20

40

60

80

100

120

140

Budget

500

1000

1500

2000

RSNA Pediatric Bone Age

0

20

40

60

80

100

120

140

Budget

5

10

15

20

Druglib Reviews

DAVED (multi step)

DAVED (single step)

Data OOB

DVRL

Influence

KNN Shapley

LAVA

Random

0

200

400

600

800

1000

Dimensions

10 -1

10 1

10 3

Runtime

(seconds)

Total data(oints = 1,000

10 3

10 4

10 5

N,mber o  seller data

10 -1

10 1

10 3

Dimension = 30

DAVED (single ste()

DAVED (ours)

Data OOB

Data Shapley

DVRL

Influence

KNN Shapley

LAVA

Leave One Out

test errors. Therefore, we recommend keeping the number of test datapoints in the buyer's 'query' between 1-8 for each data acquisition.

Number of Steps. In Appendix D.4, we vary the number of optimization steps in our method on the Gaussian-distributed and RSNA datasets. We find that more iterations generally improve prediction performance. Intuitively, one expects that selecting T points requires at least T steps of iterative optimization. We recommend setting the number of steps to be 2-5 times the desired budget for homogeneous costs.

Convex versus Iterative Optimization In Appendix D.6, we compare the iterative optimization procedure against a convex optimization solver [18]. We find that our iterative approach results in several orders of magnitude speedup while maintaining similar levels of test error.

Finetuning versus Linear Probe. In Figure 14, we evaluate fine-tuning versus linear probing for datapoints selected using DAVID and random selection. We find that using DAVID for fine-tuning performs similarly to linear probing results on DrugLib with BERT [17].

6 Discussion

While other validation-free methods exist [60, 5], our method uniquely combines test-adaptivity, theoretical grounding, and superior empirical performance. Moreover, a major advantage of our method is that it is amenable to federated optimization requiring O ( d ) communication per round, making it well-suited for decentralized data marketplaces, unlike other methods that require seller data to be centralized in order to repeatedly train models to estimate data value. Additionally, our method does not require labeled data, whereas other data valuation methods assume that all datapoints come with corresponding ground-truth labels. As discussed in Section 2 and Section 3, the existing paradigm of valuing data with a validation set is suboptimal. Incidentally, the second-best performing method, Data OOB [39], is the only other method that does not use a validation set.

Limitations. However, our algorithm comes with some limitations that form exciting directions for future work. Our approach currently communicates every step. Instead, integrating local steps like in FedAvg [43] or Scaffold [35] would decrease communication costs. Further, integrating differential privacy techniques would provide formal privacy guarantees to the buyers and sellers [19]. While DAVED shows strong performance across datasets, its effectiveness depends on having a good feature extractor that captures relevant aspects of the data. We recommend using pre-trained foundation models (e.g., CLIP, GPT-2) as they can extract general-purpose features. Future work could explore adapting the feature extraction to specific domains or handling cases where key features are missing

Acknowledgments

C.L. is funded by the National Science Foundation Graduate Research Fellowship Program. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. 2141064. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. S.P.K. and M.J. were funded by the European Union (ERC-2022-SYG-OCEAN-101071601). P.V. was partially supported by the ADIA Lab Fellowship.

References

Anish Agarwal, Munther Dahleh, and Tuhin Sarkar. A marketplace for data: An algorithmic solution. In Proceedings of the 2019 ACM Conference on Economics and Computation , pages 701-726, 2019.

S Damla Ahipa¸ sao˘ glu and Michael J Todd. A modified frank-wolfe algorithm for computing minimum-area enclosing ellipsoidal cylinders: Theory and algorithms. Computational Geometry , 46(5):494-519, 2013.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International conference on machine learning , pages 242-252. PMLR, 2019.

Zeyuan Allen-Zhu, Yuanzhi Li, Aarti Singh, and Yining Wang. Near-optimal discrete optimization for experimental design: A regret minimization approach. Mathematical Programming , 186:439-478, 2021.

Mohammad Mohammadi Amiri, Frederic Berdoz, and Ramesh Raskar. Fundamentals of taskagnostic data valuation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 9226-9234, 2023.

Muhammad Jehangir Amjad, Christophe Diot, Dimitris Konomis, Branislav Kveton, Augustin Soule, and Xiaolong Yang. Optimal probing with statistical guarantees for network monitoring at scale. arXiv preprint arXiv:2109.07743 , 2021.

Kenneth Joseph Arrow. Economic welfare and the allocation of resources for invention . Springer, 1972.

Francis Bach, Simon Lacoste-Julien, and Guillaume Obozinski. On the equivalence between herding and conditional gradient algorithms. In International Conference on Machine Learning , 2012.

Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. arXiv preprint arXiv:2006.14651 , 2020.

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Raul Castro Fernandez. Data-sharing markets: Model, protocol, and algorithms to incentivize the formation of data-sharing consortia. Proceedings of the ACM on Management of Data , 1(2): 1-25, 2023.

Lingjiao Chen, Bilge Acun, Newsha Ardalani, Yifan Sun, Feiyang Kang, Hanrui Lyu, Yongchan Kwon, Ruoxi Jia, Carole-Jean Wu, Matei Zaharia, et al. Data acquisition: A new frontier in data-centric AI. arXiv preprint arXiv:2311.13712 , 2023.

R Dennis Cook. Detection of influential observation in linear regression. Technometrics , 19(1): 15-18, 1977.

Thomas M Cover. Elements of Information Theory . John Wiley & Sons, 1999.

S Damla Ahipasaoglu, Peng Sun, and Michael J Todd. Linear convergence of a modified frank-wolfe algorithm for computing minimum-volume enclosing ellipsoids. Optimisation Methods and Software , 23(1):5-19, 2008.

Deven R Desai and Mark Riedl. Between copyright and computer science: The law and ethics of generative ai. arXiv preprint arXiv:2403.14653 , 2024.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. The Journal of Machine Learning Research , 17(1):2909-2913, 2016.

Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming , pages 1-12. Springer, 2006.

Valerii Vadimovich Fedorov. Theory of optimal experiments . Elsevier, 2013.

Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems , 33: 2881-2891, 2020.

Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems , 33:5850-5861, 2020.

Amirata Ghorbani and James Zou. Data Shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning , pages 2242-2251. PMLR, 2019.

Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images in dermatology with the Fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1820-1828, 2021.

Safwan S Halabi, Luciano M Prevedello, Jayashree Kalpathy-Cramer, Artem B Mamonov, Alexander Bilbily, Mark Cicero, Ian Pan, Lucas Araújo Pereira, Rafael Teixeira Sousa, Nitamar Abdala, et al. The rsna pediatric bone age machine learning challenge. Radiology , 290(2): 498-503, 2019.

Baihe Huang, Sai Praneeth Karimireddy, and Michael I Jordan. Evaluating and incentivizing diverse data contributions in collaborative learning. arXiv preprint arXiv:2306.05592 , 2023.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In International conference on machine learning , pages 427-435. PMLR, 2013.

Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms. arXiv preprint arXiv:1908.08619 , 2019.

Kevin Fu Jiang, Weixin Liang, James Zou, and Yongchan Kwon. Opendataval: a unified benchmark for data valuation. arXiv preprint arXiv:2306.10577 , 2023.

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. MIMIC-III, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016.

Hoang Anh Just, Feiyang Kang, Jiachen T Wang, Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jia. Lava: Data valuation without pre-specified learning algorithms. arXiv preprint arXiv:2305.00054 , 2023.

Zeynep Ülkü Kahveci. Attribution problem of generative ai: a view from us copyright law. Journal of Intellectual Property Law and Practice , 18(11):796-807, 2023.

Surya Kallumadi and Felix Grer. Drug Review Dataset (Druglib.com). UCI Machine Learning Repository, 2018. DOI: https://doi.org/10.24432/C55G6J.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning , pages 5132-5143. PMLR, 2020.

Javen Kennedy, Pranav Subramaniam, Sainyam Galhotra, and Raul Castro Fernandez. Revisiting online data markets in 2022: A seller and buyer perspective. ACM SIGMOD Record , 51(3): 30-37, 2022.

Leonid G Khachiyan. Rounding of polytopes in the real number model of computation. Mathematics of Operations Research , 21(2):307-320, 1996.

Yongchan Kwon and James Zou. Beta Shapley: a unified and noise-reduced data valuation framework for machine learning. arXiv preprint arXiv:2110.14049 , 2021.

Yongchan Kwon and James Zou. Data-oob: Out-of-bag estimate as a simple and efficient data value. arXiv preprint arXiv:2304.07718 , 2023.

Chao Li, Daniel Yang Li, Gerome Miklau, and Dan Suciu. A theory of pricing private data. ACM Transactions on Database Systems (TODS) , 39(4):1-28, 2014.

Philip M Long. Properties of the after kernel. arXiv preprint arXiv:2105.10585 , 2021.

Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning , pages 23610-23641. PMLR, 2023.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics , pages 1273-1282. PMLR, 2017.

Aileen Nielsen. Whose data, whose value? simple exercises in data and modeling evaluation with implications for technology law and policy. New York University Law Review , 99:63, 2023.

Raymond EAC Paley and Antoni Zygmund. A note on analytic functions in the unit circle. In Mathematical Proceedings of the Cambridge Philosophical Society , volume 28, pages 266-272. Cambridge University Press, 1932.

Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186 , 2023.

Friedrich Pukelsheim. Optimal design of experiments . SIAM, 2006.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Jack Sherman and Winifred J Morrison. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics , 21(1):124-127, 1950.

SD Silvey. Optimal design measures with singular information matrices. Biometrika , 65(3): 553-559, 1978.

Rachael Hwee Ling Sim, Xinyi Xu, and Bryan Kian Hsiang Low. Data valuation in machine learning:'ingredients', strategies, and open challenges. In Proc. IJCAI , pages 5607-5614, 2022.

Peng Sun and Robert M Freund. Computation of minimum-volume covering ellipsoids. Operations Research , 52(5):690-706, 2004.

Matias Travizano, Carlos Sarraute, Mateusz Dolata, Aaron M French, and Horst Treiblmaier. Wibson: A case study of a decentralized, privacy-preserving data marketplace. Blockchain and distributed ledger technology use cases: Applications and lessons learned , pages 149-170, 2020.

Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics , 12:389-434, 2012.

Martin J Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint , volume 48. Cambridge university press, 2019.

Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics , pages 6388-6421. PMLR, 2023.

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 , 2021.

Henry P Wynn. The sequential generation of d -optimum experimental designs. The Annals of Mathematical Statistics , 41(5):1655-1664, 1970.

Xinyi Xu, Zhaoxuan Wu, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Validation free and replication robust volume-based data valuation. Advances in Neural Information Processing Systems , 34:10837-10848, 2021.

Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522 , 2020.

Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning , pages 10842-10851. PMLR, 2020.

Yaodong Yu, Alexander Wei, Sai Praneeth Karimireddy, Yi Ma, and Michael Jordan. Tct: Convexifying federated learning using bootstrapped neural tangent kernels. Advances in Neural Information Processing Systems , 35:30882-30897, 2022.

Boxin Zhao, Boxiang Lyu, Raul Castro Fernandez, and Mladen Kolar. Addressing budget allocation and revenue allocation in data market environments using an adaptive sampling algorithm. arXiv preprint arXiv:2306.02543 , 2023.

Renbo Zhao and Robert M Freund. Analysis of the frank-wolfe method for convex composite optimization involving a logarithmically-homogeneous barrier. Mathematical programming , 199(1):123-163, 2023.

A Proof of Theorem 1

Theorem A.1. Let w ∗ denote the solution of Problem (1) and let ˆ w denote the solution of Problem (2) . Let the data Z val , Z test are drawn i.i.d. from the distribution D X,Y where D X is supported on B d R (zero-centered ball with radius R in R d ), and Y = θ ⊤ X + η where η is independent zero-meaned noise with variance σ 2 . Suppose D X and the training data X train is supported on B d R (zero-centered ball with radius R in R d ) and l is square loss, then there exist numerical constants, c 1 , c 2 , c 3 , such that:

With probability at least 1 -exp ( -c 1 n val /R 2 ) ,

If there exists κ > 0 such that λ ( E x ∼D X [ x ⊗ 4 ]) ≤ κ · λ ( E x ∼D X [ x ⊗ 2 ] ⊗ 2 ) (here λ denotes the largest eigenvalue and ⊗ denotes the outer product), then for any training algorithm used by the platform, with probability at least 0 . 99 -exp ( -c 1 n val /R 2 ) -c 3 κ κ + m , we have

Proof. Let f θ ( x ) = θ ⊤ x and E = N (0 , σ 2 ) . Define the parameter space resulting from the training algorithm:

When n is sufficiently large, Lemma A.2 implies that there exists { θ 1 , θ 2 , . . . , θ K } ⊂ Θ such that

Let P θ i denote the conditional distribution D y | x of the target when the underlying model parameter is θ i , it then follows that

Applying Lemma A.3, we obtain that

Now define Σ = E x ∼D X [ xx ⊤ ] , by Lemma A.4, we have that with probability at least 1 -exp( -Ω( n val /R 2 )) ,

Therefore, under the event of Eq. (12), Eq. (13) implies that

This establishes the first inequality.

Notice that

For the second inequality, we have that under the condition λ ( E x ∼D X [ x ⊗ 4 ]) ≤ κ · λ ( E x ∼D X [ x ⊗ 2 ] ⊗ 2 ) , the following holds

By Lemma A.5, for any θ and θ ∗ , we have that with probability at least 0 . 99 -O ( κ κ + m ) ,

Combining this and the first inequality by union bound, we establish the second inequality.

A.1 Supporting Lemma

Lemma A.2 (Metric entropy, Wainwright [56]) . Let ∥ · ∥ denote the Euclidean norm on R d and let B be the unit balls (i.e., B = { θ ∈ R d |∥ θ ∥ ≤ 1 } ). Then the δ -covering number of B in the ∥ · ∥ -norm obeys the bounds

Lemma A.3 (Fano's inequality, Cover [14]) . When θ is uniformly distributed over the index set [ M ] , then for any estimator ˆ θ such that θ → Z → ˆ θ

̸

Lemma A.4 (Matrix-Chernoff bound, Tropp [55]) . Consider an independent sequence { X i } k i =1 of random, self-adjoint matrices in M n satisfying X i ≥ 0 and λ max ( X i ) ≤ R almost surely, for each i ∈ { 1 , . . . , k } . Define

Then,

Lemma A.5 (Paley-Zygmund inequality, Paley and Zygmund [45]) . If Z is a random variable with finite variance and Z ≥ 0 almost surely, then

B Convergence Rate of Frank-Wolfe (Proof of Theorem 2

Setup. We mostly follow the proof technique in Bach et al. [8] and Jaggi [28]. Recall the optimization problem for the optimal design loss Eq. (4):

Here, D is the scaled simplex defined by the constraints w ∈ R n ≥ 0 and ∑ n j =1 c j w j ≤ B . The Frank-Wolfe update on this function is then: For t = 1 , 2 , . . . , repeatedly perform the following steps

Compute s t = arg max u ∈D ⟨∇L ( w t ) , w t -u ⟩ .

update w t +1 = (1 -α t ) w t + α t s t

Note that this update procedure is identical to the updates in Eq. 5. We can then define the duality gap as

We also define the curvature constant

We assume that the curvature constant is finite, i.e., C l < ∞ . This is true for L ( w ) as long as both the algorithm and the true optimum are bounded away from the boundary of D -see detailed discussions on this in Ahipa¸ sao˘ glu and Todd [2]. A better analysis might be able to avoid this assumption, e.g., Zhao and Freund [65] use certain homogeneity properties of L ( w ) to derive better assumption-free convergence rates for the FW method on D-optimal experiment design. We leave the question of adapting these results to our setting (V-optimal experiment design) for a challenging future work.

Lemma B.1 (Lemma 5, [28]) . For any α ∈ (0 , 1) ,

Theorem B.2. In the Frank-Wolfe algorithm, our update algorithm in Eq. 5 uses α t = 1 t +1 . For this, we have

Proof. Define h ( w ) = L ( w ) -min w ∈D L ( w ) . Using Lemma B.1,

Here we used the convexity of L as follows:

Continuing our derivation, recall we have

Thus we have

Unrolling this recursion, we get

This yields the theorem claim.

Finally, to finish the proof of Theorem 2, note that adding additional constraints only increases the loss, i.e.,

Hence, we showed a O (log t/t ) approximation to the otherwise intractable combinatorial problem. With additional assumptions on the structure of the loss function, one can even show improved quadratic or even exponential approximations [15, 8]. Finally, we note that Frank-Wolfe is a wellknown method to efficiently approximate the optimal experiment design objective [59, 20, 2] and this also motivates using this approach in practice [2, 6].

C Experimental Setup

For each buyer test point, we optimize each selection algorithm over the 1,000 seller datapoints and select the highest value data based on the validation set of 100 datapoints (our method and Data OOB do not use the validation set). For each test point, we train a linear regression model on the selected seller points and report test mean squared error (MSE) on the buyer's data and average test error over 100 buyers.

For reproducibility, our full implementation is available at: https://github.com/clu5/ data-acquisition-via-experimental-design .

C.1 Implementation Details

We conduct all experiments on an Intel Xeon E5-2620 CPU with 40 cores and a Nvidia GTX 1080 Ti GPU. For implementation of baseline data valuation methods, we use the OpenDataVal package [30] version 1.2.1. We use the default hyperparameter settings for all methods except for Data Shapley (changed 100 Monte-Carlo epochs with 10 models per iteration), Influence Subsample (from 1000 to 500 models), and Data OOB (from 1 , 000 to 500 models) to reduce computational runtime.

In our experiments, we use the following setting of hyperparameters for DAVED:

500 iterations for multi-step variant, 1 iteration for single-step variance

Line search for step size α ∈ (0 , 0 . 9)

Regularization λ = 0 (unless otherwise specified)

No early stopping

For experiments with heterogeneous costs, we uniformly sample costs c ∈ { 1 , 2 , 3 , 4 , 5 } for each seller datapoint and apply either h ( c ) = √ c or h ( c ) = c 2 as the cost function. Noisy labels are generated by adding Gaussian noise scaled inversely proportional to the cost, with overall noise level set to 30% .

C.2 Dataset Details and Processing

For Fitzpatrick17K and RSNA Bone Age datasets, each image was embedded through a CLIP ViT-B/32 model [49], while for the DrugLib dataset, each text review was embedded through GPT-2 model [48] with a max context length of 4096.

For the Gaussian dataset, we generate a regression dataset according to the following Python code:

The RSNA Pediatric Bone Age Challenge (2017) dataset [25] may be downloaded here https://www.rsna.org/rsnai/ai-image-challenge/ rsna-pediatric-bone-age-challenge-2017 . We use the training set for our experiments, resulting in 12,611 images in total. Using the following function, each image was embedded through a pre-trained CLIP ViT-B/32 model.

The Fitzpatrick17K [24] can be downloaded from here https://github.com/mattgroh/ fitzpatrick17k . Missing or corrupted images were excluded, resulting in 16,536 total images. Some images were annotated with two separate annotation platforms. We averaged the two skin type ratings for these images, resulting in 12 possible labels (0-6 in 0.5 increments). Each image was embedded in a fashion similar to the RSNA Bone Age dataset.

The MIMIC dataset [31] can be accessed here https://physionet.org/content/ mimiciii/1.4/ . The task is to predict the length of stay (LOS) in the number of days a patient stays in the Intensive Care Unit. The dataset contains 51,036 rows with both real-valued and one-hot-encoded attributes with the following names:

'LOS' , 'blood' , 'circulatory' , 'congenital' , 'digestive' , 'endocrine' , 'genitourinary' , 'infectious' , 'injury' , 'mental' , 'misc' , 'muscular' , 'neoplasms' , 'nervous' , 'pregnancy' , 'prenatal' , 'respiratory' , 'skin' , 'GENDER' , 'ICU' , 'NICU' , 'ADM_ELECTIVE' , 'ADM_EMERGENCY' , 'ADM_NEWBORN' , 'ADM_URGENT' , 'INS_Government' , 'INS_Medicaid' , 'INS_Medicare' , 'INS_Private' , 'INS_Self Pay' , 'REL_NOT SPECIFIED' , 'REL_RELIGIOUS' , 'REL_UNOBTAINABLE' , 'ETH_ASIAN' , 'ETH_BLACK/AFRICAN AMERICAN' , 'ETH_HISPANIC/LATINO' , 'ETH_OTHER/UNKNOWN' , 'ETH_WHITE' , 'AGE_middle_adult' , 'AGE_newborn' , 'AGE_senior' , 'AGE_young_adult' , 'MAR_DIVORCED' , 'MAR_LIFE PARTNER' , 'MAR_MARRIED' , 'MAR_SEPARATED' , 'MAR_SINGLE' , 'MAR_UNKNOWN (DEFAULT)' , 'MAR_WIDOWED'

Each attribute was min-max scaled to lie in the range [0, 1].

The DrugLib dataset [34] can be downloaded here https://archive.ics.uci.edu/ dataset/461/drug+review+dataset+druglib+com . The task is to predict overall patient satisfaction with a drug's side effects and effectiveness on a ten-point scale. We format each review using the following prompt template to feed GPT-2:

where $BENEFITS_REVIEW is the corresponding portion of the drug review.

C.3 Evaluation Protocol

For all experiments (unless otherwise specified), we use the following evaluation settings:

Training split: Variable number of seller points ( 1 K100 K)

Validation split: 100 points for baseline data valuation methods

Test split: 100 buyer points evaluated independently

Metrics: Mean squared error averaged over all buyer points

Budget ranges: 1 -30 for cost experiments, 1 -150 otherwise Number of random seeds: All results averaged over 3 seeds

For a fair comparison, we use the same train/validation/test splits across all methods for each random seed. The validation set is only used by data valuation baseline methods that require it - our method DAVED operates directly on the test queries without requiring validation data.

D Additional Experiments

D.1 Budget Results with Heterogeneous Costs

5

10

15

20

25

30

Max budget

0

1

2

3

4

5

Buyer

error

5

10

15

20

25

30

Max budget

0

1

2

3

4

5

Buyer

error

Ours (multi-step)

Ours (single step)

Data OOB

DVRL

KNN Shapley

LAVA

Random

5

10

15

20

25

30

Max budget

100

200

300

400

500

600

700

Buyer

error

MIMIC h(c)= c 2

5

10

15

20

25

30

Max budget

100

200

300

400

500

600

700

Buyer

error

MIMIC h(c)=√c

Ours (multi-step)

Ours (single step)

Data OOB

DVRL

KNN Shapley

LAVA

Random

5

10

15

20

25

30

Max budget

0

1

2

3

4

5

Buyer

error

5

10

15

20

25

30

Max budget

0

1

2

3

4

5

Buyer

error

Ours (multi-step)

Ours (single step)

Data OOB

DVRL

KNN Shapley

LAVA

Random

5

10

15

20

25

30

Max budget

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Buyer

error

Bone age h(c) = c 2

5

10

15

20

25

30

Max budget

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Buyer

error

Bone age h(c) = √ c

Ours (multi-step)

Ours (single step)

Data OOB

DVRL

KNN Shapley

LAVA

Random

0

20

40

60

80

100

Max budget

5

10

15

20

25

30

35

40

Buyer

error

Druglib h(c) = c 2

0

20

40

60

80

100

Max budget

5

10

15

20

25

30

35

40

Buyer

error

Druglib h(c) = √ c

DAVED (ours)

DAVED (single step)

DVRL

Data OOB

Influence

KNN Shapley

LAVA

Random

D.2 Regularization

0

25

50

75

100

125

150

Budget

80

100

120

140

160

180

200

Test

error

MIMIC

0

25

50

75

100

125

150

Budget

4

5

6

7

8

9

10

11

12

DrugLib Reviews

0

25

50

75

100

125

150

Budget

600

650

700

750

800

850

900

Amoung of Regularization

0

0.2

0.4

0.6

0.8

1.0

D.3 Amount of Buyer Data

0

5

10

15

20

25

Budget

0.0

0.5

1.0

1.5

2.0

2.5

Test

error

0

25

50

75

100

125

Budget

0

250

500

750

1000

MIMIC

0

25

50

75

100

125

Budget

500

1000

1500

Number of Buyer Datapoints

32

16

8

4

2

1

D.4 Number of Iteration Steps

0

20

40

60

80

100

120

140

Budget

0.0

0.5

1.0

Test

error

0

20

40

60

80

100

120

140

Budget

500

750

1000

1250

Number of Optimization Steps

1

5

10

50

100

500

1000

D.5 Linear Feature Space Approximation

0

5

10

15

20

25

Budget

0

10

20

30

40

50

Test

Error

Linear probe (DAVED)

Finetuning (DAVED)

Finetuning (Random)

D.6 Iterative versus Convex optimization

1

5

10

15

20

25

# Training data points

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Test

error

Data selection algorithm

Data OOB

DVRL

LAVA

KNN Shapley

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 3

Runtime (seconds)

(single step)

(multi-step)

CVXPY

E Broader Impacts

We believe that AI developers face important ethical questions when acquiring data for AI development, as highlighted by recent class-action lawsuits against AI companies regarding data consent and compensation. Our work presents a scalable and decentralized approach to data acquisition that can help address these concerns. By enabling targeted selection of the most valuable datapoints, our method reduces both costs and potential privacy risks compared to broad, indiscriminate data access [44]. The decentralized nature of our approach enhances transparency and gives individual data owners more control over how their data is shared and used, while avoiding the privacy and security risks of centralizing data with brokers. However, while our method enables more efficient and privacy-preserving data transactions that could democratize access to high-quality training data, especially in domains like healthcare, several societal and technical challenges remain. These include ensuring fair compensation for data owners, preventing misuse of acquired data, and developing appropriate governance frameworks for decentralized data markets. Addressing these challenges will be crucial for the responsible development of data marketplaces that can sustainably increase the supply of training data while protecting individual privacy and data rights.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state our main contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We address the limitations of our methods in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide proofs of all main theorems in Appendix 1 and Appendix B. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide complete implementation details, hyperparameters, and experimental setup in Appendix C. Our code is publicly available at https://github.com/ clu5/data-acquisition-via-experimental-design .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In the supplemental material, we provide a simplified code example and will release the full codebase in a public code repository upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide details of our experimental setup in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experimental results are averaged over 100 random test trials.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Hardware specs and runtime analysis provided in Appendix C and Figure 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research complies with NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Appendix E discusses both the benefits and risks of our work in the context of data marketplaces.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We believe this paper does not pose such risks as we do not release any new models or datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite and provide links to all datasets used in Appendix C.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not introduce any new datasets or models.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our research does not involve crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects research was conducted.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the problem of data acquisition in decentralized data marketplaces. The paper claims that acquiring training data using validation-based data valuation methods can be overfitting. Thus, the paper proposes DAVED, which directly selects training data by optimizing test loss on a given test set rather than relying on the validation set. DAVED approximates the test loss trained on a subset of the training data using a linear experimental design framework and employs gradient-based optimization to compute the sampling probability of each data point, considering budget constraints and decentralized settings. Experiments on synthetic and real-world datasets are conducted to validate the advantages of DAVED.\n\nSTRENGTHS:\n(S1) The paper studies an interesting and important problem.\n\n(S2) The paper proposes a novel approach for data acquisition in decentralized data marketplaces.\n\n(S3) Experiments show the effectiveness of the proposed approach.\n\nWEAKNESSES:\n(W1) Lack of comparison with data acquisition approaches.\n\n(W2) Lack of experimental evaluation on the test data distribution.\n\n(W3) The motivation could be better clarified.\n\nQUESTIONS:\n(D1) The paper claims that existing data valuation methods are misaligned with the data acquisition problem and introduces DAVED to select data points based on the test set rather than the validation set. In the experiments, the paper compares DAVED with a series of data valuation methods including Influence, Data Shapley, and Data OOB, showing the advantages of DAVED. However, the paper does not discuss or compare its performance with methods that focus on data acquisition problems with or without a validation set. In addition, given that DAVED is designed for decentralized data marketplaces, the paper does not discuss or verify how it compares with existing methods in federated learning, such as [62].\n\n\n(D2) The paper does not conduct experiments to analyze the impact of different test data distributions and the training data distribution. The paper should include experiments to measure the performance of DAVED under the scenario where there is a distribution shift between the training and test sets.\n\n(D3) The paper uses an example to illustrate their setting: \"the buyer may be a patient who cares about diagnosing their own chest X-ray and is willing to pay a fixed budget to access other X-ray images to train a model to have low error on their 'test set' (see Figure 1).\" However, this example may be impractical as it is unlikely that a patient would train a diagnostic model themselves rather than relying on a hospital or healthcare institution. If the data acquisition is indeed for a hospital, the test data for the hospital cannot be fixed because patients are continually coming in. From my understanding, Algorithm 1 is designed for a fixed size of test data. This creates a gap between the paper's setting and real-world scenarios. The authors should either clarify this gap or explain if I have misunderstood the motivation or algorithm.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses the linear experimental design to tackle the data acquisition problem.\n\nSTRENGTHS:\nThis paper tackles a very challenging but important problem in the data marketplace : evaluating data quality before data transactions as well as minimizing the test error.  The main contribution is the Federated approach for the experimental design.  \n\nMotivation is clear and paper is well-organized. The experimental results are convincing.\n\nWEAKNESSES:\n1. Overall I like this paper and take a few hours trying to understand the technical part and experimental results. But some necessary analysis and equations are simplified, not so friendly to readers who are not very familiar with the V-optimal experiment design framework and some optimization parts. Please provide some preliminaries in your work. \n\n2. line 136 .. suppose we have a known feature-extractor ...  how to get such good extractor $\\phi$ and whether your evaluation results are sensitive to a poor $\\phi$? \n\n3. The authors consider the low-budgets cases ( lower 50 training datapoints ) and the validation size for baselines is small. More experiment results are needed. \n\n4. Some parts are confusing for me, please see the questions below.\n\nQUESTIONS:\n1. Could authors explain how to get the eq.(7) derived from the eq.(4) ?  \n\n2. $\\textbf{confusion part:}$ How $g_i$ in eq.(7)  can be only calculated by the seller $j$ using training data $x_j^{\\text{train}}$, given $x_i^{\\text{test}}$ is needed( or $\\mathbb{E}(x^{\\text{test}})$ appears in eq.(7) ) ? \n\nLine 176:  \"$g_i$ as well as the update (6) can be trivially computed by seller $j$ using only their data $x_j^{\\text{train}}$ (and the test data). \" \n\nLine 516:  \"our method and Data OOB do not use the validation set\" \n\n$\\textbf{Question:}$  Whether optimization in eq.(7) involves the test dataset? If it involves, I guess some baselines could also do the same process for data selection : for example,  LAVA could also only use  feature space $\\mathcal{X}$ to compare $x^{\\text{train}}$ and  $x^{\\text{test}}$, and select data points?  If not, back to the question of the confusion part.\n\n It will be better to provide more details about your analysis or related works for references in the paper for better understanding. ]\n\n3. In the experiment setting, the validation set has only 100 datapoints, could authors conduct the experiments when the validation size increases ?  \n\n4. Authors consider low-budgets situations ( 1~10 datapoints ) , it will be better to provide experiments when budgets are larger ?\n\n\n————————\nAll my concerns have been addressed. I will raise the score.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method for acquiring training data in a decentralized data market without requiring labeled validation data. Based on detailed and reliable theoretical support, the authors comes to a promising conclusion that the selection methods based on validation set has the performance as bad as the models trained by validation itself. Thus, it theoretically exceeds a series of methods represented by Shapley. This approach is amendable to federated optimization and reduce prediction error, making it more suitable for decentralized markets compared to traditional centralized data valuation methods.\n\nSTRENGTHS:\nS1: This paper combines actual scenarios in the data market to summarize challenges that may exist for sellers and buyers, providing detailed and precise definitions for the data acquisition task. The overall topic is very interesting and practical.\nS2: The writing level is very good. Most of the presented viewpoints are well motivated, and the proofs, derivations, and theoretical implications of the formulas mentioned are all well explained and easy to understand.\nS3: For data acquisition methods relied on validation data, this paper theoretically deduces the lower bound of its error and proves that validation data(or labels) itself is unnecessary. This is crucial for the field of data acquisition where labels are very insufficient.\nS4: The method has good scalability, with sufficient experimental validation.\n\nWEAKNESSES:\nW1: The input for data acquisition task is too idealistic. In the actual data market, existing data acquisition tasks are often combined with data discovery, emphasizing the retrieval of data from environments such as data lakes that have different distributions and poor data quality. This paper simplifies the input data as datapoints, lacking consideration for issues related to data quality and failing to explain why datapoints are directly considered without integrating them into the context of the data market.\n\nW2: Although this article provides detailed and solid theoretical deductions, some assumptions are too strong. For example, in Line 127 of Section 4: \" First, we assume that the conditional distribution D_(y|x) is identical across Z^Train and Z^Test\". In fact, due to the presence of a large number of missing values in the train set, assumptions about data distribution may not hold true. If these assumptions are not valid, the explanation provided in the article regarding V-optimal experiment is insufficient, which makes me confused.\n\nW3: The proof of theorem 1 in the article is interesting, but some details need to be discussed if we want to use this conclusion to overperform methods based on Shapley Value. The existence of parameter σ makes it difficult for me to clearly understand how small this lower bound is. If users have some high-quality validation data (and in practical scenarios of data markets, users should provide high-quality data as much as possible), would this lower bound be acceptable?\n\nW4: The assumption of infinite training data in theorem 1 is not very reasonable. From my experience, the data available in the data market should be a subset of the results from the data discovery task and joinable subsets of datasets in the data lake, with its scale yet to be examined.\n\nW5: The application of kernelized linear regression is not well motivated.\n\nQUESTIONS:\nQ1: Is it reasonable to assume that the training data in theorem1 is infinitely large?\nQ2: If the assumptions about data distribution in Line 127 of Section 4(w2) cannot hold true, how does this proof work through V-optimal experiment? Or can you explain why this assumption always holds true even with data quality problems?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new data valuation algorithm based on linear experimental design. The technique does not require a validation set and can be used in a federated setting.\n\nSTRENGTHS:\nThe paper studies a well-motivated trending topic. Data valuation and data selection are important subfields in data-centric ML. I also like the motivation of developing validation-free data valuation algorithms a lot.\n\nWEAKNESSES:\nI have reviewed this paper at ICML. I think the new revision has addressed my previous concerns. The following is my ICML review:\n- The writing can be improved a bit. For example, in Section 3.3, the proof or citation for the test loss of LS estimator should be provided. In Section 3.4, I don't understand why the constraint of costs can be removed. Such derivation details are better to be explicitly written, it's okey to be in Appendix. I also don't see the benefit of using Frank-Wolfe algorithm.\n- The proposed approach requires a feature extractor, and I believe the performance depends greatly on the choice of feature extractor. Afterall, the theoretical justification of this appraoch is based on the assumption that the groundtruth is a noisy linear combination of the feature, that's why the requirement of test labels can be removed. It's good to explicitly acknowledge this point.\n- In Figure 3, why does the error significantly increases with full data for many curves? Are there a small portion of the data is being corrupted?\n\nIn this new revision, the author has clarified the linear model setup and the derivation is super easy to follow. The idea of using linear model as a proxy is straightforward but I am not aware of other works doing it. My remaining concern is that while this approach does not require a validation set, it requires a feature extractor. My major question is how this feature extraction can be picked in practice. I understand this is a hard question and the use of a public feature embedding is quite common in data selection techniques, therefore I will not reject the paper based on this point.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-25"}