Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sources
Authors: Feiyang Kang
Hoang Anh Just
Anit Kumar Sahu
Ruoxi Jia
Venue: NeurIPS 2023 poster (2023)
Keywords: data-centric AI
data acquisition
data valuation
performance prediction
data markets
optimal transport
scaling laws
TL;DR: A new framework to predict model performance on training data from multiple sources using only a few pilot data via Optimal Transport, and also finds optimal data source composition for any data scale through efficient gradient-based optimization.

Abstract: Traditionally, data selection has been studied in settings where all samples from prospective sources are fully revealed to a machine learning developer. However, in practical data exchange scenarios, data providers often reveal only a limited subset of samples before an acquisition decision is made. Recently, there have been efforts to fit scaling functions that predict model performance at any *size and data source composition* using the limited available samples. However, these scaling functions are usually black-box, computationally expensive to fit, highly susceptible to overfitting, or/and difficult to optimize for data selection. This paper proposes a framework called *<projektor>*, which predicts model performance and supports data selection decisions based on partial samples of prospective data sources. Our approach distinguishes itself from existing work by introducing a novel *two-stage* performance inference process. In the first stage, we leverage the Optimal Transport distance to predict the model's performance for any data mixture ratio within the range of disclosed data sizes. In the second stage, we extrapolate the performance to larger undisclosed data sizes based on a novel parameter-free mapping technique inspired by neural scaling laws. We further derive an efficient gradient-based method to select data sources based on the projected model performance. Evaluation over a diverse range of applications (e.g., vision, text, fine-tuning, noisy data sources, etc.) demonstrates that *<projektor>* significantly improves existing performance scaling approaches in terms of both the accuracy of performance inference and the computation costs associated with constructing the performance predictor. Also, *<projektor>* outperforms by a wide margin in data selection effectiveness compared to a range of other off-the-shelf solutions. We provide *<projektor>* an open-source toolkit.

Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sources
Feiyang Kang" Bradley Department of ECE Virginia Tech Blacksburg, VA 24061 fyk@vt. edu
Hoang Anh Just" Bradley Department of ECE
Virginia Tech Blacksburg, VA 24061 justêvt. edu
Anit Kumar Sahu
Ruoxi Jia Bradley Department of ECE
Alexa AI Amazon
Virginia Tech Blacksburg, VA 24061 zuoxijia@vt.co edu
Seattle, WA 98121
anit. sahu@gmail.......mmmaamaamaa com
Abstract
Traditionally, data selection has been studied in settings where all samples from prospective sources are fully revealed to a machine learning developer. However, in practical data exchange scenarios, data providers often reveal only limited subset of samples before an acquisition decision is made. Recently, there have been efforts to fit scaling functions that predict model performance at any size and data source composition using the limited available samples. However, these scaling functions are usually black-box, computationally expensive to fit, highly susceptible to overfitting, or/and difficult to optimize for data selection. This paper proposes framework called projeks and supports data selection decisions based on partial samples of prospective data sources. Our approach distinguishes itself from existing work by introducing a novel two-stage performance inference process. In the first stage, we leverage the Optimal Transport distance to predict the model's performance for any data mixture ratio within the range of disclosed data sizes. In the second stage, extrapolate the performance to larger undisclosed data sizes based on a novel parameter- free mapping technique inspired by neural scaling laws. We further derive an efficient gradient- based method to select data sources based on the projected model performance. Evaluation over diverse range of applications (e.g., vision, text, fine-tuning, noisy data sources, etc.) demonstrates that projekts or significantly improves existing performance scaling approaches in terms of both the accuracy of performance inference and the computation costs associated with constructing the performance predictor. Also, pro projekk jojeeto or outperforms by wide margin in data selection effectiveness compared to a range other off-the-shelf solutions. We provide projektor as an lopen-source toolkit
which predicts model performance
1 Introduction
The choice of training data is one of the most crucial components when it comes to extracting the best performance out of model. Since data is typically acquired from various sources, such as different organizations or vendors, machine learning practitioners often encounter central question: how to select and combine samples from these data sources?
"Equal contribution. This work also appears at Data-centric Machine Learning Research (DMLR) Work- shop@ICIa ICML. 2023. Code Repository:
httpa////ooooooiaaaoooooaaaaaaaa 37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Data Sources
Publicty Available Pilot Data
Input Budget
(a) (b)
BDD100K
0. of
15
D
PROJEKTOR
Optimal Composition
p.= 0.22 of D
MAP (%)
Prediction
Selection
Data Sources
D2
(i pert
Ni Minmaa part
P. 2.2.
Actual mAP Predicted mAP Random Selection
Forecasted Performance
90% Acc
Target
Validation Set D
2K
3K
Model
Figure 1: (a) Overview of projektor which take as inputs the public pilot data from each source, selection budget, target model, validation set representing the test distribution, and return the optimal combination of data sources as well as the prediction of the resulting model performance. (b) Optimal data source composition and performance projection in practical autonomous driving data acquisition scenario [5]. pro projekt jektor achieves accurate model performance projection from IK pilot samples and effective selection. Please see Evaluation Metrics (Section for details mAP.
Although data selection has been extensively studied in the literature related to active learning M. coreset selection [2], and data valuation (38) most techniques are designed for a fully-observable setting where all data sources are fully revealed to the model developer. The core ideas behind these techniques are to compare the relative importance of different data points or enumerate possible combinations of data points, all of which require complete knowledge of the entire collection of data points. While these methods have shown promising results, their practical applications in real- worldworld scenarios are limited due to a significant gap: the acquisition decision-making processes require knowledge of the entire data sets, while data owners may only reveal limited samples before an 10 provide the examples in real-world data markets).
acquisition decision is made (e.g.,
To bridge the gap, this paper explores strategic data selection in partially observable setlings, where only limited samples of data sources (referred to pilot datasets) are accessible. The goal is to determine an optimal allocation of the selection budget to each source, only based on the pilot datasets, such that the model trained on the mixture of collected data achieves the best performance at some given objectives.
Technical challenges. In the fully-observable setting, the evaluation and eventually ranking the candidate data selection decisions, including the number samples to be selected and the ratio of samples from each source ("mixing ratios"), be determined directly the available datasets However, the partially observable setting presents considerable challenges for evaluating selection decision one can no longer directly evaluate model performance the entire data. With limited samples from each data source, the best possible evaluation is the resulting model performance for any combination of the pilot datasets. Then, make informed selection decision, it is necessary to understand the model's performance when trained on potentially larger datasets (target scales) at various mixing ratios. In other words, there is a need for prediction and projection of model performance onto larger data scales at different mixing ratios.
A recent study 13 proposes performance scaling law that takes into account the data size and mixing ratio to predict model performance. Providing a preliminary exploration of this problem, though, this approach faces two major limitations: (1) The numerical instability of its high-order form for scaling functions causes significant difficulty in fitting its parameters, rendering the fitted function susceptible to overfitting and often fails extrapolate model performance unseen data mixtures. (2) It hypothesizes on the separability of model performance scaling with data composition and with data size, which is generally untrue evidenced by latest research [14) and leads to unsatisfactory performance prediction results in empirical observations. Besides, this method requires parameters that grow quadratically with the number of data sources, demanding many (mixing ratio, resulting performance) pairs to fit the function, resulting in substantial computational overhead. Thus, there remains a considerable lack of effective and practical approaches to this problem.
Contributions. The paper investigates two fundamental building blocks for strategic data selection in the partially observable setting:
(Q1) How to provide an accurate projection of model performance trained on any combination of data sources based limited samples? and (Q2) How to determine the optimal selection strategy?

---

Towards that end, the paper makes the following contributions.
Parameter-Fiftcien Performance Prediction based on Optimal Transport (Addressing Q1). In contrast to existing model performance scaling methods that feature a one-shot fitring of a non- informative parametric model ("surrogate"), our approach is novel two-stage performance inference process, where the first stage addresses the dependence of the model performance on the mixing ratio by fitting& a parameter-efficien model between model performance and Optimal Transport [T6] distance between the mixtures of training data and the validation data (Section 4.2). Then, for stage two, we propose a parameter-
mapping that directly projects model performance onto larger data scales, achieving remarkable accuracy as it fully preserves this the dependency of model performance scaling with data sizes and data distributions (Section 4.3).
Determining optimal data selection strategies (Addressing Q2). We consider the typical data selection goal: maximizing the resulting model performance with fixed data acquisition budgets (data quantity). With model performance predicted by the proposed tools, these problems translate into convex losses that are optimized effectively via gradient-basee methods (Section 4.4p. We also provide in Appendix B how it similarly applies to alternative objectives such as minimizing data acquisition costs for the resulting model performance to reach a given level.
Experiments. We experiment on a variety of applications (vision, natural language processing (NLP) etc., with simple to complex models) with rich diversity of tasks and real- world scenarios (e.g., autonomous driving, fine-tuning, noisy data sources, unlabeled data). The proposed approach is highly effective in performance prediction, demonstrating superior prediction precision to many baselines while being much more efficient to be constructed (Section 5) We test the performance of data selection by optimizing the performance predictor and show that it improves over existing methods by 3% on ImageNet-100.
2 Related Work
The recent line of research on Data valuation aims assess the relative importance of each data source ("value") "") to machine learning applications [ While originally designed for data pricing. these values are frequently used to inform data selection [7] : in more detail, one can rank the data sources based on their values and select the data points with the highest values. While value- based selection shows some promising results, data values are not directly related to model performance and hence cannot inform the prediction of model performance resulting from the selected data. Besides, values for different data sources typically cannot be combined to measure the value of their compositions [ Notably, distributional distances including Optimal Transport have seen major presence in data valuation as an implicit proxy for model performance (8] 19), but no connection has been made to directly relate data distance to model performance. Our work bridges this gap and directly addresses this long-standing problem. On another line, Coreset selection attempts to find a representative subset of data points to speed up learning 22. Coreset selection methods have been studied for different learning algorithms [20] 21). For example, a popular coresct selection approach for neural networks is to cast data selection as bilevel optimization problem that selects the optimal subset to maximize the performance evaluated on a validation dataset [22]. However, coreset selection techniques rely on access all the data samples to be chosen from, which limits their use in the partially observable setting. Besides, Predicting the resulting model performance associated with dataset without performing actual training on it has attracted a lot of attention in different use -which leverages surrogate functions to model the black-box relationships between model performance and training data, or predicting performance under the distributional shift (25). Our work resembles the idea of predicting model performance from data but differs in the technique of leveraging the data distance in the performance predictor. Scaling laws, predicting how the model performance changes with the scale of training data, model parameters, and computation budget (26). have seen increasingly successful in a variety of tasks pertaining to vision and text processing (27] The performance of machine-learning models generally adheres to a power law relationship with the scale of these variables, which allows for predicting the model performance on larger scales with high precision [26] and provides a viable approach to predicting the potential usefulness of target data from only a small proportion of the set. [14 shows that data from different distributions generally scale at different rates. Our work provides a novel approach that materializes this dependency of scaling relationships with data distributions and achieves remarkable empirical results.
cases, such as interpreting the model learning process 23 24
3

---

3 Problem Formulation
Data provider. Suppose that there are m prospective data providers. Datasets (data sources) held by these providers are denoted by Di ...., D respectively. We focus on the case that only partial data (samples) from these sources are made available to the public, replicating practical data exchange scenarios [9. We refer to the public subset of each data source as pilot dataset and denote it by Df' where DT S Di and |D, n, < N₁ |Dell for all i. Each provider i, upon accepting the purchasing order for acquiring n, samples (n; < No. will randomly sample subset S; of size na from Dell and return the subset to the requester
Data collector (or requester, machine learning practitioner). Now, consider a data collector who would like acquire samples from the providers to train a model. Notably, the collector's acquisition decisions must be made based only on the pilot datasets. We assume the collector has a validation set D representing the desired target data distribution. For ease of exposition, we assume the collector has a target learning algorithm that is going to be applied to the collected data as well as a target performance metric L which takes the input of a trained model and a validation set and returns performance score. The model performance resulting from training on any dataset S can be thus expressed as L(A(S), D'
Given a selection budget of N samples, a mixing ratio of data sources p = {pr..... Pm } such that Vr, p; and > p; 1, and m datasets Di,.... D., to be mixed, we denote the selected dataset by D(N, Si U ... U Sm, where each S is random subset of De and |S. N. Using these notations, we now describe the typical acquisition goals that can be accommodated by approach:
(Primary) Fixed-budget selection for maximal performance: The collector seeks to maximize the resulting model performance by strategically choosing the mixing ratio p of data sources at a pre-specified selection budget N, S Ein N;. The objective can be formalized as maxp ((A(D(N, P)), )
(Alternative) Flexible-budgee selection for reaching performance threshold with minimal costs: The collector seeks attain target model performance u!" by choosing both the mixing ratio as well as the selection budget N. More formally, the objective can be expressed as minN, p- N s.t. maxp ((A(D(N) p), D'a) 2 tdT
The alternative objective can be treated as a direct extension of the primary, where one solves the "performance maximization" problem for different data quantities N and performs line search for minimal data quantity that meets the performance requirement. We defer to Appendix B for its complete solution procedure due the similarity.
Design challenge and key idea. The primary challenge is that the collector cannot access Dell ..... Dall for decision making and hence cannot directly evaluate the two optimization ob- jectives for every N. Yet, as the pilot datasets are public, the collector can evaluate and observe the model performance associated with various mixtures of the pilot datasets for N (N : Np, I |.P I,..., m) project the evaluations onto larger data scales. Our high-level idea tackle the challenge is to first predict the model performance associated with any mixture of prospective unrevealed data sources based on observations on pilot datasets and project the predictions onto different data scales using scaling laws, then determine the data selection strategy by optimizing the predicted performance the target scales.
4 Methodology of projektor: prediction, projection, and selection
4.1 Preliminaries on Optimal Transport
Optimal Transport (OT) a metric for measuring the discrepancy between probability distribu- tions 16.. Compared to other measures such the Kullback-Leibler Divergence 29 Maxi-
Thii paper will assume each provider homestly provides requested samples and leave in-depth study of potential security risks, such as malicious data manipulation 28 future work. 'The proposed data selection approach can support multiple learning algorithms by simply applying it to different choices of algorithms/metrics and picking the best one.

---

mum Mean Discrepancies [30, OT enjoys advantageous analytical properties (is a valid metrie; compatible with Saars--upport distributions; stable with respect deformations of the distri- butions' supports [3 32 ). Given probability measures 14, M4, over the space 2, the OT dis- tance defined as (T (b)) == el J22 C(2, (2) z'), where [T(44,)4))) == { E P22 x 2) 12 2222"))22 pe, J2 a (2,22'ddz' = -Me denotes a collection of couplings be- tween two distributions 14, and Mv. C: 2 x - IR+ is a symmetric positive-defing cost function (with C(2, = 0). respectively. A popular choice of C is given in 3 by considering
the feature- label pair (at, y)- The computation of the OT distance usually relies on the Sinkhorn algorithm [5), which attains almost linear time complexity and memory overhead with the state-of-the-an imple- mentations and applies to large scales with parallel computing [31 (2]. Given D, = {( y/ b of size N, and D. = {a,, 1 of size T. one can construct discrete probability measures (1, v) = N Ei ò ((,,.)) and (1, v) = + Ei ((o),a()> where d is the Dirac delta func- tion. With slight abuse of notation, we use OT( D.) to denote the OT distance between their corresponding discrete measures (T y), pu (x. y))-
I
Extensive theoretical studies show that the OT distance between two distributions provides an upper bound on the difference of a model's performance when evaluated on the two distributions [36
37 g
Largely built upon Kantorovich-Rubinseer Duality [38. existing theoretical results require assumptions on the Lipschitz constant of the model with respect to the input space. However, the
constant is rarely known in practice, nor can it be bounded tightly for complex models such as deep
neural networks. As result, despite its widespread popularity as a performance proxy [8] 36 [3]] one cannot directly apply the existing theoretical results to directly estimate the model performance based on the OT distance, posing an important gap.
4.2 Aligning data distance with performance predictions
Inspired by the theoretical results that the upper bound on the difference between training loss and validation loss can be tightly bounded by an affine transformation of the OT distance [33 g
our first
proposed approach is to directly estimate this transformaie by empirically fitting data distances to model performance and then the estimated transformation can be used for predicting the model performance for different data mixtures. Formally, we consider the following performance estimator:
È (A(D(NN, p)), D= a1 OT (P(N, p). D + ag:
(I)
where scaling parameter @n and centering parameter og define the affine transformation. These two parameters can be estimated through least-square fitting. In particular, consider collecting the "training data" by forming the set of tuples {(NV,, Pj, L(A(D(N), D))))) where N, is randomly sampled from (1...., E.= [ and Py is sampled from probability simplex. Then, these parameters can be estimated as
(dg, do) arg min Σ
-
We refer to this method as center-scaling (projektor/CS). a₁ can be considered an empirical estimate of the Lipschitz constant, and this treatment has been informally adopted in various works under different names 42). projekter has only two parameters that need to be estimated, which brings an important benefit of efficiency: we only need to have a few training iterations to get the "training data" for the above least-square fitting.
While proposed pro fekttr/////eee / CS is sufficient to provide reliable performance predictions in most circumstances, we found it possible to further improve the prediction accuracy by making the scaling parameter and centering parameter function of the mixing ratio. The intuition is that for samples from different data sources (i.e., data lying in different manifolds of the input space), the Lipschitz constant of the model along the combined manifold may vary with the mixing ratio. Hence, we supplement projektoz or/Cs with simple nonlinear terms to characterize the dependence on each data source, leading to the pseudo- psendo-quadratic quadratic jprojekton or (2) method, which is given as
T
(LA(D(N.P)),
(b 1
+
bb) OT(D(N, p), D I (cs
+ + eo), (2)
where (b2, b1, bo, C2 C1, Co) are pseudo-quadratic parameters where the fitting process similar to Eq. (I) projektor/PQ has ((m) parameters (m is the number of data sources). So its fitting
5

---

process will require more re-training than pro jektor/CS. However, as we will show in Section S
still significantly improves the efficiency over the existing baselines [T.. This pseudo form is chosen for it contains the simplest nonlinear terms and we want to preserve the convexity for numerical stability. We trimmed off the cross terms the quadratic function as they often do not contribute much, resulting in the number of parameters growing linearly rather than quadratically with the number of data sources as in [3]. greatly easing the computation burden in parameter fitting.
4.3 Parameter-free performance projection onto larger data scales
Once the parameters are learned, the performance predictors (1 and (2 can be used to predict the validation performance associated with a training set by calculating the OT distance between the training and the validation set and plugging as input the predictors. Then, we need to project these predictions onto the target data scales. Neural scaling laws showcase the predictability of empirical performance with respect to the size of the training dataset, where it typically follows a log-linear scaling relationship as
Ev Ev|C(A(D(NN p)); D ~ -a log(N) + C
where a and are some constants 26 Recent work [4] shows that when data from different sources differ in "quality" (e.g., noise level), which is the most likely scenario in practice, the scaling parameters are often vastly different, rendering parameters a and C' functions of data composition p and model performance for different data mixtures p scales with different rates. [13] assumes the same constant a for all data mixtures, leading to unsatisfactory scaling results. The difficulty underlying this dependence is that these scaling parameters differ for every data mixture and there is no closed-form expression available for this functional relationship. With the performance prediction tools proposed above. it is possible to directly predict model performance of any data mixture at the scale the tool has been fitted. That is, for any given data scale No. as long as one completes the one-off fitting process of the performance predictor, model performance for any data composition at data size No can be inferred directly using Eq. ① or Eq. (2) Thus, by performing the fitting process different small scales for once, for any desired data mixture, we can directly the neural scaling laws for this particular distribution and project it onto larger data scales, without needing to train any additional model or make any further approximations. We formalize it into the following theorem.
Theorem 1 (Data Composition Dependent Performance Projection). Consider log- log-linear perfor- mance scaling relationship depending on both data size N and data composition p given as Ev Ev[C[A(D(M,, p)); D = -a((p) log(N) + C(p). Assume one hus completed the filting of the performance predictor on two different scales No < N1, which gives L(A((D(o, p)); De and L(A(((N,, p)); D for all data mixtures p. Then, the model performance ((A(((N. D for any data mixture p at any data scale can be predicted as
N1 No
N É(A(D(N) P)): D' log N1 log No
(): log
LAA(D p); D
without requiring fitting any additional parameters. The proof and derivations are given in Appendix B We refer to this method as parameter-free projection for model performance. As this procedure does
not rely on any additional assumption or parameterized surrogate, it requires minimal computational overhead while achieving considerably higher prediction accuracy compared to existing approaches such as [3].. Not exclusive to the performance predictor proposed in this work, this method can be plugged into other predictors seamlessly and provides benefits at large, marking a novel contribution to performance projection in data acquisition problems.
4.4 Performance-guided data selection
The intention of creating the proposed tools is not limited to providing predictions for model performance, rather, we expect the predictions to support determining the optimal data acquisition strategy. We show that these problems are convex and differentiable (Appendix (B) and thus can be solved effectively via gradient- selection for maximal performance, with the proposed performance predictors with projection, we solve for p" arg maxp L(A(D(N,, p)), D We solve it iteratively with the following procedure. First, initialize the algorithm with p pº where pd can be chosen arbitrarily provided that
methods. Specifically, for our primary objective fixe-bbudger

---

Σ, = Then, at each step, we perform the gradient update as
p + + p' + d' AÉ(A(D(N,, p)), Dval)
0p
p=p
where d' is the step size at iteration i and we obtain p p at convergence as the desired solution. Optimal Transport naturally provides its gradient w.r.t. the probability mass of data points in its dual solutions [8], which directly gives the gradients w.r.t. data mixtures p. This easy availability of gradients renders the optimization highly efficient in computation, resulting in remarkably fast solutions. We use the calibrated gradient of OT from [8] which ensures the updated mixture remains within the simplex E; Pi 1 at each step. We provide technical details of the gradient computation in Appendix B The alternative objective can be treated direct extension of the primary and we defer to Appendix B for its solution procedure. The pseudo-code for projektor is provided in Appendix
TRAINING
INFERENCE
Traininng Data Preparation Each K
Function Fitting For M
III. Data Distance Performance Projection
Optimal Source Selection
Prediction Mixture Ratio
Training Sample
Training
Gradient Update
Projection Onto
Optimal Ratio
ratlo P/
N
N
T
N
Get performance of model trained on the composed pilot datimet
90% Acc
aL pt
p*
dp
Compute transport
No
No
N
p
Equation 1/2
Equation
Equation 4,4
Figure 2: Workflow chart for the two-stage performance inference process of projektor.
Evaluation
In this section, cover main applications for projektor. 1) Performance projection, where for any mixing ratio of data sources and any data scale, we want to predict the performance of the model trained on a given composed dataset. We also demonstrate efficiency and efficacy of our method for different scenarios of data sources, such as mislabeled or unlabeled data. 2) Optimal data source acquisition strategy, where for a given data budget, find a mixing ratio of data sources that can maximize the performance of model. We present solution to select optimal data source composition for two learning paradigms scenarios: training from scratch and model fine-tumin fine-tuning.
We compare with six existing baseline methods, where the first four: (1) Linear 13, which assumes linear relationship with the data compositions; (2) Pseudo-Quadrate assumes simple non-linear relationship; (3) Quadratic 13] assumes a fully quadratic relationship; (4) Rational [3] models a relationship through the of set of rational functions. For m data sources, each function has m parameters, and there are m such functions, totaling m² parameters; (5) LOO [5] measures the importance of data source by computing the performance difference after removing that source; (6) Shapley [1]] is a game-theoretic method which computes the average marginal contribution of data source to different subsets of other sources. Baselines (5) and (6) are suitable for informing the selection of data sources but are unable to predict model performance,
we only include them in data source selection experiments. Details on implemented baselines are described in Appendix [ and further explained in 13] For all experiments, we set the problem with three data sources, where each source consists of different classes, and we refer the reader for additional information on the experimental setup, algorithm, datasets, models, implementations, code repository, and ablation studies on the number of data sources to Appendix [ We also showcase the runtime vs performance prediction trade-off and comparison with bascline methods.
Evaluation Metrics. We use mean absolute error (MAF) to measure the performance of our method by calculating the absolute difference between the predicted accuracy and the actual accuracy. For
7

---

Traiinng Eseudo-Quadratte
Linear
Projokter/CS (Ours)
Linear
Projektor/CS (Ours)
Psaudo-quadratic
Rational
Predicted Accuracy
MAE
20
Quadratic
2
2 Rational
D
ProjektoriPQ (Ours)
2:
31
4
2
40 # Model Trainings
10 20 30
50
3
2
15
22
25
30
Actual (ccuracy(%)
Figure Predicted model performance vs. actual model performance for extrapolation on CIFAR-10 with total of 1000 data points for data sources. Comparison between projektor and baselines.
Figure 4: MAE vs. number of model trainings used for fitting the method. Efficiency comparison between projektor and baselines.
the object detection task, we adopt a commonly used metric, mean average precision (mAP), which measures the average precision of model across multiple classes or categories, providing single value that represents the overall accuracy. The average precision represents the area under the precision-recall curve for single class.
Hyperparameters. For practical reasons, we set the data scale Ni in Eq. 5 to be the size of the smallest pilot dataset, i.e. Nj min; [ Upon selecting N1, we empirically choose No to be N1 For further investigations on selecting No, we provide sensitivity analysis No in Appendix D
5.1 Performance Prediction
Method
MNIST
CIFAR-
ImageNet100
IMDB
Prujekteu/PO (Ours)
7.277 3.01)
0.84 1.26/ 0.00)
0.53 0,38( 0.00)
0.25 1.66A 0.00)
Projektor/CS (Ours)
3.30 4. 0.00)
0.97 1.95(A 0.69)
0.54 1,03/ 0.65)
0.90 1.87(A 0.71)
Linear
5.27 28.03(2 23.77)
4.44 6.04(A 4.78)
1.47 4,59(A 4.21)
1.88 7.48(A 6.32)
Pseudo Quadratic
1.63 3.01)
22902 A 1.64)
0.77 2,72 2.34)
0.43 2,74(A 1.58)
Quadratic
1 3.01)
3.96)
2.34AA 1.96)
0.43
1.58)
Rational
$.28 27.29 23.03)
0.83 2,95(A 1.69)
0.44 2,27(A 1.89)
1.91 7.11(A 5.95)
Table 1: Training and test MAE values for extrapolation of data source compositions for each method. For each cell the left/rihht column value denotes the MAE value for predicting the training/test data, respectively. Underlined green/bold red denotes lowest training/test MAE value, respectively.
A. Predicting Performance for Unseen Data Mixtures p
In this experiment, we fit the parameters on limited compositions and extrapolate the prediction to unseen compositions. Specifically, we choose one data source and limit its maximum composition to < 55% of contribution in the training set of the performance predictor, then predict accuracy on the compositions consisting 2 55% of contribution. As we observe in Figure 3 Linear and heudo-Quadratic methods cannot fit well the training data, which indicates that these methods do not have a strong representation power. While Quadratic and Rational baselines can fit the training data, they suffer from overfitting and do not generalize to unseen compositions. On the other hand, seen Table I. our method projektor /PQ achieves the best training and extrapolation performance. pro or CS achieves second best extrapolation performance. Furthermore, efficiency of our method compared to other baselines. As observed in Figure 4 pro jektor not only achieves the lowest MAE score but also converges with around 15 training data for pro jektor / CS and 25 for pro azojektor or or /PQ, which demonstrates low computational requirement of our method. With shown strong predictive power of pro jektor, now proceed to practical applications in performance projections onto larger data scales.
analyze the
B. Performance Projection to Larger Data Scales
Mislabeled Data Sources. In this experiment, we project performance onto larger data scales and also assume more practical setting where data sources might not be of high quality and contain

---

Model Perforaacc
Bandom
Accuracy
Actual
ProjektorPO
34
AL1
K1.4
10.3
Projektor/CS
35
60%
Linear
100
46%
45.6
Pscudo Quadratic
Quadratic
Rational
Model
LOO
Shapley
100
47%
PQ (Ours) CS (Durs)
Quedratic
Stepley
Random
52%
Figure 7: Performance projection of selected optimul mixture ratios in (Fig. 6 onto 50K ImageNet- 100 from 10K samples. Comparison with the actual model performance.
Figure 6: Optimal data source composition selection for 50K ImageNet-100 from 10K samples and the
actual model performance.
then critical to factor such irregularity for the performance prediction into method. Given three mislabeled data sources formed by sampling CIFAR-I0 10, each of which releases pilot dataset of size 1K, we project performance for various mixing ratios onto larger data sizes, i.e. 2K, 5K, 7K, and 10K. We then measure the MAE value across all data scales. We observe in Fig. Fig.5]ther that projektor achieves the best projection performance compared to all baseline methods. pro jrojerrrr/ /PQ achieves the lowest MAE score below 2% and pro jektor/ /CS is slightly above 2%. The improved performance of our method can attributed to the incorporation of actual data distance computation. This inclusion allows for more accurate representation of mislabeling information in performance projection, unlike baseline methods that neglect this crucial information.
noisy labels [33]
The promising results demonstrate the potential of our method to project performance of any composition any data scale, which is important in the case mislabeled data sources in the partially- reveeee setting.
2.70
2.00
MAE(N0
2.14
Unlabeled Data Sources. As mentioned earlier, data sources often contain noisy labels, and the process of labeling data can be costly. On the other hand, it not un- common to encounter unlabeled data sources. Therefore, we would like to extend our method to accommodate the setting of data sources without labeled data, and aim to project performance unlabeled data compositions from pilot data source mixtures of 1K samples from CIFAR- 10. The three data sources contain unlabeled data from different classes of CIFAR-10. In this case, we compute the optimal transport distance on the feature space only, and we assume access to the labels of the pilot datasets, which enables us to train a model and obtain performance values. Consequently, we project per- formance across various mixing ratios onto larger data sizes (2K, 5K, 7K, and 10K). To visualize the performance of our method, we plot the difference between the projected performance and the actual performance. The closer the value approaches zero, the more optimal the projection becomes. Surprisingly, as we illustrate in Fig. g projektor can consistently maintain good projection performance for larger data sizes. Even at 10K, the largest error is below 10%. These outcomes show that pro je jekt or can also be extended to unlabeled data sources, which demonstrates the flexibility and practicality of our method.
Protektor
Pretektor
Liner
Pacuto:
Qustrutic
Rational
Figure 5: Performance projections from 1K CIFAR- 10 samples across various mixing ra- tios and larger data scales: 2K, 5K, 7K, 10K. Comparison between projekto and base- lines.
5.2 Optimal Data Source Selection
Training a model requires a tremendous amount of resources to tune hyperparameters to achieve the highest performance. We demonstrate that by choosing data strategically, can also improve model performance. We consider a setting, where we are facing the problem of choosing only 50K to train ResNet-50 on ImageNet100 and would like maximize the model's performance.

---

2K
However, we are provided with only a pilot dataset of size 10K from each data source. As we observe in Fig-
Performance Difference (%6
0.2
6.4
ure G our optimized mixing ratio based on (4.4) achieves the highest model performance compared to all baselines. Further, using the functions from the previous step, we
5K
project the performance of our selected mixing ratio, and
0.6
we observe in Figure that pro projektrr not only most
7
closely predicts the accuracy to the actual accuracy but
also attains the highest actual accuracy out of all methods.
The improved selection of mixture ratios in our method can be attributed to our proposed selection approach (4.4)
0o
0.
Unlike baseline methods that assume the same optimal composition for all data scales,
method finds optimal compositions specific to each data scale. For more exper- iments CIFAR-10, we refer the reader to Appendix [
9:0 2.0
0.0
0.6
Figure 8: A direct visualization for the landscape of prediction errors in model performance for all data compositions as the scale of data to be predicted grows. We consider the setting of 3 data sources with unlabeled samples from CIFAR- 10. We construct the proposed performance predictors within Ik samples that are considered accessible to the practitioner. We then predict model performance for all data compositions and larger data scales (2-10h 10k samples). X-Y axes rep- resent the proportion of data from data sources #1 and #2 (consequentially, the proportion of data from data source #3 will be 100%-X-V X) and the prediction errors are visualized in Z-axis (horizon- tal plane represents zero error, red spikes blue dips represent deviations from over- /under-prediction). As the data scale gets larger, the performance predicted from the initial IK samples becomes slightly less accurate, but prediction error re- mains mostly within 5% even at the scale of 10k data (10 times larger than the pi- lot data), retaining its effectiveness and functionality.
5.3 Application to Fine-tuning
As powerful architectures has been introduced and com- putation power has improved, larger models and datasets has become increasingly prevalent in training visual and natural language tasks. However, retraining these large pre- trained foundation models can be cost- cost-prohibitive which leads widespread adoption for fine-tuning these mod- els. However, these large pre-trained foundation models are expensive to retrain but are popular for fine-tuning for more customized tasks. In our case, we adopt a pre- trained Faster R-CNN model trained on COCO dataset. Our task is to fine-tune the model on the autonomous driving dataset for object detection, BDD100K [T5]. We assume each data source to specialize in taking pictures at a specific time of day, i.e. daytime, night, or dawn/dusk pictures. Similarly to the previous task, we select optimal data source composition and project the mean average pre- cision (mAP) onto larger data scales. In Figure [b). we observe fine-tuning accuracy projection onto eight larger data scales from 1000 samples and observe that our predic- tions do not deviate from the actual accuracy by more than 0.4, which indicates pro jektor's 's extended capability of performance projection for fine-tuning.
Discussion and Outlook
This paper presents a novel framework conduct data selection from partially revealed sources, addressing the practical challenge in emerging data market scenarios. In particular, in contrast to existing work that tries to directly fit non-informative parametric surrogates on the limited available samples to predict model performance at different data sizes and compositions of data sources, which suffers from pronounced computational burdens and often unsatisfactory results, our key technical contribution is an OT-based performance scaling method. The take-away from our empirical study is that despite being extensively adopted in the past, fitting con-informative parametric surrogates for predicting performance scaling is actually and less accurate; utilizing data distance in the performance prediction provides immediate benefits and presents a better pathway to construct the predictors.
computationlllll inefficient, often impractical,
Contributing a new perspective with performance and efficiency improvements, this work still has some Limitations and opens up many new investigation venues, such as lifting the requirement on val- idation data, accounting for malicious data owners, and extending to data sources that are misaligned in feature space. Additional discussions and Broader Impacts are provided in Appendix
10

---

Acknowledgments and Disclosure of Funding
RJ and the ReDS lab acknowledge support through grants from the Amazon- Amazon-Virginia Tech Initiative for Efficient and Robust Machine Learning, the National Science Foundation under Grant No. IIS-2312794, NSF 233-1333,30, NSF OAC-2239622, and the Commonwealth Cyber Initiative.
References
[1] Burr Settles. Active learning: Synthesis lectures artificial intelligence and machine learning. Long Island, NY: Morgan & Clay Pool, 10:S00429ED1 V01 Y201207/IM018, 2012. 2
[2] Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepeore: A comprehensive library for coreset 33rd International Conference, DEXA 2022, Vienna, Austria, August 22-24, 2022, Proceedings, Part I, pages 181-195. Springer, 2022.
selection in deep learning. In Database and Expert Systems Applications:
[3] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017. 33
[4] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1167-1176. PMLR, 2019. 3
[5] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242-2251, PMLR, 2019.
[6] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-19930 19930, 2020.
[7] Tom Yan and Ariel D Procaccia. If you like shapley then you't '1l love the core. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5751-5759, 2021.
[8] Hoang Anh Just, Feiyang Kang. Tianhao Wang, Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jia. Lava: Data valuation without pre-specified learning algorithms. In I 1Ith International Conference on Learning Representations, ICLR, page appear, 2023.
[9] Datarade, httts:://ataraaee. ai- htt.://iilddattaaatteeeeeeee
[10] Innodata,
[11] Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint arXiv:173..0644 2017. 2
[12] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. Be- yond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206. 2022. 2
[13] Tatsunori Hashimoto. Model performance scaling with multiple data sources. In International Conference on Machine Learning, pages 4107-4116 16. PMLR, 2021.
18
[14] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06700 2021.
6
[15] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous mul- titask learning. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pages 2636-2644 2020.
20
[16] Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009,
[17] Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning, arXiv preprint arXiv:2 2021.
11

---

[18] Ruoxi Jia, David Dao, Boxin Wang. Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang. Costas Spanos, and Dawn Song. Efficient specific data valuation for nearest neighbor algorithms. arXiv preprint arXiv: arXiiv:/98.000 1908.0869, 2019.
7 21
[19] Zhaoxuan Wu, Yao Shu, and Bryan Kian Hsiang Low. Davinz: Data valuation using deep neural networks at initialization. International Conference on Machine Leurning, pages 24150-24176. PMLR, 2022. B
[20] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pages 6950-6960.
PMLR, 2020. 3
[21] Hanlin Lu, Ming-Ju Li, Ting He, Shiqiang Wang, Vijaykrishnan Narayanan, and Kevin S Chan. Robust coreset construction for distributed machine learning. IEEE Journal on Selected Areas
in Communications, 38( 38(10):2400-4417, 2020. 3
[22] Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for contin- ual learning and streaming. Advances in Neural Information Processing Systems, 33:14879-
14890, 2020. 3
[23] Andrew Ilys, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Predicting predictions from training data. arXiv preprint arXiv:2202.00622, 2022.
[24] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry, Trak: Attributing model behavior at scale. arXiv preprini arXiv:2303. /400266633.//6, 2023. 3
[25] Christina Baek, Yiding Jiang, Aditi Raghunathan, and Zico Kolter. corremeent-onthh--r Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35:19274-19289, 2022. 3
[26] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wxXi:2001..08661 2020. 3
[27] Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300- 22312, 2022.
[28] Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Mądry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1563-1580, 2022.
23
[29] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1): 1951.
[30] Gabor Szekely, Maria L Rizzo, et al. Hierarchical clustering via joint between- distances: Extending ward's 's minimum variance method. Journal of classification, 22(2):151-184, 2005.
[31] Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn divergences. In International Conference on Artificial Intelligence and Statistics, pages 1608- 1617. PMLR, 2018. 5
[32] Jean Feydy, Thibault Séjourné, François- Fraviees-Xavier Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2681-2690. PMLR, 2019, 5
[33] Leonid V Kantorovich. On the translocation of masses. In Dokl. Akad. Nauk. USSR (NS). volume 37, pages 199-201, 1942. 5
[34] David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33:21428-211 22439, 2020. 5
12

---

[35] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems. 26, 2013.
[36] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution
optimal transportation for domain adaptation. Advances in neural information processing systems, 30, 2017. 5
[37] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. [
[38] David A Edwards. On the kantorovich-rubinttiin theorem.
Expositiones Mathematicae,
29(4):387-398, 2011. 5 18
[39] Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving gans using optimal transport. arXiv preprint arXiv:/803.05573 2018. 5
[40] Jeremy EJ Cohen, Todd Huster, and Ra Cohen. Universal lipschitz approximation in bounded
depth neural networks. arXiv preprint arXiv:1904.0486/ 2019. [
[41] Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation. In International Conference on Machine Learning, pages 291-301. PMLR, 2019.
[42] Zhouxing Shi, Yihan Wang, Huan Zhang, Zico Kolter, and Cho-Jui Hsich. Efficiently computing local lipschitz constants of neural networks via bound propagation. Advances in Neural Information Processing Systems, 35:2350- 2264,, 2022. 5
[43] Davood Karimi, Haoran Dou, Simon K Warfield, and Ali Gholipour. Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis. Medical Image Analysis, 65:101759, 2020.
9
[44] Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3):334-334, 1997. 20
[45] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150,
2011. 20
[46] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdbmmisst/
1998.
20
[47] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.
20
[48] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 11::211-252, 2015. 20
[49] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation.
988) 1997. 20
[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision-EECC 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 630-645. Springer, 2016. 20
[51] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition,
pages 770-778, 2016. 20
[52] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances neural information processing systems, 28, 2015. 20
13

---

[53] Tsung- Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollär, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer VisVo 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 740-755. Springer, 2014. 20
[54] Rafid Mahmood, James Lucas, David Acuna, Daiqing Li, Jonah Philion, Jose M Alvarez, Zhiding Yu, Sanja Fidler, and Mare T Law. How much more data do need? estimating requirements for downstream tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 275-284, 2022. 23
14

---

Appendices
projektor: : Framework and Algorithms
16
A.I
projektor Pipeline
16
Proofs and Optimization Details
17
Proof for Theorem
17
B.2
Objectives for Data Selection
18
B.
Optimization and Convexity
18
Gradients Calculation, Stepsize Selection, and Convergence
19
Sampling Stochasticity and Market Practices
20
Experiment Details and Additional Results
20
D.I
Datasets and Models
20
D.2
Details on Baseline Methods
21
D.3
Performance Prediction for Unseen Data Mixtures p: Ablation Study
D.4
Extended Application to Multiple Number of Data Sources
D.5
Optimal Data Source Composition
22
Additional Discussion and Broader Impacts
15

---

Appendix A projektor: Framework and Algorithms
A.1 projektor Pipeline
Training Data Preparation. For data distance-based performance prediction, the first step of the pipeline prepare "training data" (introduced in Section 4.2) fit parameters of Equations [ 2 The data consists different data source compositions for some given data scales No. Ni. where for each composition p. compute the OT distance of the composed dataset D(N1, p) to
the validation data and then train model on 1(N,, p) to get the actual model performance. For simplicity, select compositions through grid search. This step is represented in Algorithm from lines 4-9 and is the first (I) step in the pipeline Figure 2
II Fitting Predictor Function. Once the training data is prepared, proceed fit our function in Eqs. T and 2 This step is shown in lines 10-11 1 in Algorithm and is the second (II) step of the pipeline Figure 2 Then, with the performance predictors fitted for data scales No and Ni, move on to the inference stage, where we can perform 2 tasks: performance projection and data source selection.
III. Two-Stage Performance Projection. For performance projection, we project the performance to any data size N for any mixing ratio in two stages. (1) We predict the performance given the mixing ratio p at data scales No and N2. (2) We use 5g project performance prediction to any data size N. This process is represented in line 12 of Algorithm [ and is the third (III) step of the pipeline Figure 2
IV. Optimal Data Source Selection. For optimal data source selection, we solve optimization problem through gradient descent, which is provided in Eq. [B3.1] The gradient computation uses parameters of the fitted functions from step II and the process terminates when the mixing ratio converges. This process is represented in line 12 of Algorithm and the fourth (IV) step of the pipeline Figure
Algorithm 1: projektor performance predictor.
In : Pilot Datasets DTT D ....., Dri. Query Data Budget N: Query Mixing Ratio p; O--Data Scale Size No: 1-Data Scale Size N₁; Learning Algorithm A; Performance Metric Function C(., Deel) ); OT Distance Function OT", DuR),
Out Projected Model Performance + [0, 1].
P +- Generate mixing ratios
DTo, DT1 + Initialize empty lists store OT distances
Lo, L1 t Initialize empty lists to store performance values
for Mixing Ratio Pt in do
So. S = D(No. Pı), D(N1, Pi) newly composed datasets of size No. N1
DTo + append OT(S,, Dval)
Optimal Transport distance between So and Deal
DT1 uppend OT(S1. D
Optimal Transport distance between S₁ and Deat
Lo append L(A(S)), Deal) Performance of model trained on So
L1 + append ((A(S)), D Performance of model trained on S
10 L(A(D(NO.))) v + Fit the function from Eq. with ((P,DTo), Lo)
L(A(D(N) 1val)
Fit the function from Eq. with ((P, DT,), In)
12 L(A(D(N, p)); Dval) t Project performance by substituting ((A( p)), D and
É(A(D(N1,,)) p)), D into Eq.
return L(A(D(N, D
16

---

Algorithm 2: Optimal data source composition p" Update.
In
Pilot Datasets D D ...... D Query Data Budget N; Query Mixing Ratio p:
O-Data Scale Size No; I-Data Scale Size N₁; Trained projektor Models with No and
N1 Data Budgets: fu. f1; Enquired Data Budget: N; OT Distance Function "T..:D"m
Validation Set: Dval
Out: data source composition p".
<- Initialize Random Data Source Composition
2 while p not converged do
So. S ( pi), 1(N1, Pi) newly composed datasets of size No, Ni
gradient + Compute gradient wrt. p using Eqs.
p Update composition p with the gradient update according to Eq. B.3.1
return p
Appendix B
Proofs and Optimization Details
B.1
Proof for Theorem
Theorem (Data Composition Dependent Performance Projection (restated)). Consider log-linear performance scaling relationship depending on both data size N and data composition p given as Ev [V(CC(D p)); Drd)]] aa()) log(N) + C(p)
(4)
Assume one has completed the fitting of the performance predictor two different scales No < N1. which gives L(A(D(No, p)); D and ((LA(D(N,P))) D for all data mixtures p. Then, the model performance L(A(D(N. Dral) for any data mixture p any data scale can be predicted as
N1 No
N LLA(D(N) p)); D)": log LLA(D(N D): No
N N
P)): D'h
log
log
(5)
Proof. From Eq. for any data mixture Ps, we have
Ev Ev]C(A(D(N, Pz)); ) ap log(N) + C'(2))
Then, for data scales No and N₁ where one has completed fitting the performance predictors, we have
Ev Ev[C(A(D(No, Pa)); ) -a (Px) log( No) C(pz)
Ey Ev|C(A(D(NI, p2)); -a (P=) log( (N1) C(p2)
which gives
6(p2) L(A(D(No, Px)); Dval L(A(D(N, P2)); D
log(N,) (og)
where L(A(D(No, Px)); ) and L(A(D(N) Pz)); D are given by the fitted predictors.
For any data scale NN. we have
Ev [C(((((, Pz)); )r (p())) log(NN) C(px)
Plugging in the above equations, the performance prediction can be given by [(A(D(N, Pa)); ) å(pz) (log(N.) log( ((A((((N, p)); m
= - (Â(A(D(No P²)): D L(A(D(N, Px)); D log (N.) - log(
log(NN) (N1) - log(No)
+ L(A(D(N), p)): D
N1
N₂
No L(A(D( Pw): 1 ((A(D(AV) DT llog Ni
log
log No
No.
which completes the proof.
Q.E.D.
17

---

B.2 Objectives for Data Selection
Problem formulation is provided in Section where the following two objectives are introduced.
Given a selection budget of N samples, mixing ratio of data sources p {Pi...., Pm J such that Ve, S P: S and Σ P: = 1, and m datasets DI..... Dm to be mixed, we denote the selected dataset by D(N, p) S U... U Sm. where each S, is random subset of D and |St| = p.N. Using these notations, we now describe the typical acquisition goals that can be accommodated by our approach:
(Primary) Fixed-budget selection for maximal performance: The collector seeks to maximize the resulting model performance by strategically choosing the mixing ratio p of data sources at a pre-specified selection budget N, S Em N₁. The objective can be formalized as maxp ((A(((N,, p)).DT!)
(Alternative) Flexible-budee selection for reaching performance threshold with minimal costs: The collector seeks to attain a target model performance ut by choosing both the mixing ratio p as well the selection budget N. More formally, the objective can be expressed as minN,p N s.t. maxp L(A(D(N, p)a) udt
The primary objective, "fixed-budget selection for maximal performance", is formulated as a convex optimization and we solve it via gradient-basee based methods. The alternative objective can be treated as a direct extension of the primary. where solves the ffixed-budge selection for maximal performance" problem for different data quantities N and performs a line search for minimal data quantity N that meets the performance requirement.
B.3
Optimization and Convexity
B.3.1
Primary
For our primary objective fixed-budger selection for maximal performance, with the proposed perfor- mance predictors with projection, we solve for
pd arg max ((A(D p)),
P
We show this objective function is convex in data composition p. where the proposed gradient-based method will guarantee find its optimal solution p" efficiently.
Empirically, model performance L always appears convex in data composition p. which is also reported in [13]. This means the model trained on data combined from multiple sources always achieves no worse performance than the average performance of models trained separately on data from each source. Theoretically, as given in (38), the gap between training and validation performance can be tightly bounded by the OT distance between training and validation data, whereas the OT distance is always convex in data composition p. We now show our proposed performance predictors as well as the optimization problem based on them are also convex in data composition
For projektor/CS in Eq. D consider data compositions po * P1 and convex combination D2 = apo + (1 - a) for some constant a (0, 1). Then, we have
A (A(D(N, P2)), D a& (A(D(N, Po)), D (1 a (A(D(N,, pI)), a [OT (D(N, Dral aOT (D(N, po), Dval) - (1 ajoT (1). 20
where the inequality holds because a1 O always holds and OT distance is always convex in data composition p by definition [16]. Thus, projektor/CS convex in data composition
constructed as pro projertor/CS with pseudo quadratic terms and its convexity in p can be shown similarly.
projektor/Po
We solve the above optimization problem iteratively with the following procedure. First, we initialize the algorithm with p = pd where pd can be chosen arbitrarily provided that Σ. P. = 1. Then, at each step, we perform the gradient update
t+1 p* d: aL(A(D(N,, p)), D
0p
18

---

where d' is the step size at iteration t and we obtain p = p" at convergence as the desired optimal solution.
B.3.2 Alternative
Then, for flexible-budget selection for reaching performance threshold with minimal budget:
min N s.t. max L(A(((N.))) 2 tar N
We solve it through bi-level optimization, where the lower level is the same the primary objective and the upper level is line search for optimal data quantity N Note that the model performance C(A(D(N, p), D is monotonically non non-decree non-decreasing and concave in N. Thus, the optimal data quantity N can be found via a straightforward line search. Initialize No and p = p where pd can be chosen arbitrarily provided that E; Pi = 1. Then, at each step, we perform the gradient update as
N++1 + N + dd. @L(A(D(N, p)))), )'u)
ON
N=N
with
arg max L(A(D(N, p))
p)),
where d' is the step size at iteration i and p is the optimal data mixture at N₁, respectively. Continue until L(A(D(N,, ))) Dval)= 2 u eer is achieved and then the data acquisition strategy D(N,, p**) accepted. Note that at each step, p² is initialized from p (t-1)+ and the optimization for p is completed fairly easily within few steps.
B.4 Gradients Calculation, Stepsize Selection, and Convergence
For the primary problem optimizing over p(N) where is the target data quantity, the gradients can be calculated as
(C(A(D(N. D 0p
N1 log
N aÉ(A(D(N), p)); D
N aL(A(D(No, p): D
log
- log N₂
No.
No
Op
Öp
(6)
Optimal Transport naturally provides the gradient information the OT distance w.r.t. the probability mass of datapoints on which it is computed in its dual solutions. B provides an approach that directly constructs the calibrated gradient from the output of the OT solver that informs how the OT distance changes as the probability mass of the datapoints changes, while ensuring the updated mixture remains within the simplex E, P: = 1 at each step.
Specifically, for projektor / CS, recalling Eq. (T. we have
aC(A(D(N.p): D B0T(AAD(N, p): D
aOT(A(D(N, p); D aOT(A(D(N, p): D
0p
0p
8p1
8pm
where P1 P2 P. 1. Let (rt, rf...." ...r. ...r""" } be the samples consisting R(N, p) where r; represents samples from data source i. Then, the calibrated gradient can be given as
aOT(A(D(N, p): Dral)
1 n4
Σ P
Σ
T2 y
Öp:
(1..
where fl the dual solution of OT that corresponds r.. The calibrated gradient ensures the updated mixture p remains within the simplex E.P= = at each step. Similarly, for projektoo jertor/P in Eq.
the calibrated gradient given as
aL(A(D(N, p); )
(b² p. + bj p. + 6o) (OD(A(D(N, p): Dall
0p:
Op:
(8)
+ |b (2pz) + b3] OT(A(D(N, + Job (2p:) + cij.
19

---

Then, to perform gradient-basse based optimization, first, we initialize the algorithm with p = where p" can be chosen arbitrarily provided Σ, Ps 1. Then, each step, we perform the gradient update as
11 p + + d' (E(A(D(N).P)). )
0p
p=p'
where d' > 0 the step size at iteration 1. In practice, we choose diminishing step sizes that satisfy Brobiins-Monre conditions such that d' < dd+1, E, d' =x, and (d'd) < oc. then the series p is guaranteed to converge to the optimal solution p 44 given that objective function is convex and bounded. Gradients for pro jektor /QQ can be obtained similarly and the solution procedure is the same. The proposed method is shown to achieve remarkable performance and fast convergence, yielding satisfactory results in swift manner.
Appendix C
Sampling Stochasticity and Market Practices
In the data selection problem formulated in this work, we aim to optimize predicted performance with objectives given in terms of finite samples from the pilot dataset. We note that this pilot dataset is considered a random sample from the whole dataset of each data provider and is inevitably affected by the stochasticity of the sampling process. Our performance predictions C are empirical estimates of the expectation for the variable based the samples. Thus, it substantially depends the sampling process for the estimates to be unbiased and precise. Data providers should adhere to certain guidelines when selecting the pilot datasets. The sampling process should be unbiased where each sample is selected with an equal chance. Examples include sampling with a Bernoulli process where each sample is selected with fixed probability p: or permutation sampling where one selects the first N samples from the random permutation.
There might be strategic providers that do not adhere the guidelines. Multiple mechanisms available to incentivize providers to provide true samples. Since the full dataset will be revealed after purchase, the data buyer examine the posterior probability for the pilot dataset being sampled from the whole dataset according to the prescribed sampling protocol. The chance of the distribution of the pilot dataset having large deviation from that of the whole dataset should be small. A threshold for hypothesis testing can be set to determine whether to accept or reject the pilot dataset as an unbiased sample from the whole dataset. If there are external supervisions (e.g., market regulators), they can conduct sequential hypothesis testing to check whether the samples provided by each seller converge to the whole dataset or comply the prescribed sampling procedure.
Appendix D Experiment Details and Additional Results
D.1
Datasets and Models
For our experiments, we use the following vision and language datasets:
Dataset
Total Training Data
Total Test Data
Number of Classes
IMDB
45
25,000
25,000
2
MNIST 46
50,000
10,000
10
CIFAR-10 10 [4
50,000
10,000
10
ImageNet100 48
130,000
5,000
100
BDD100K [S
70,000
10,000
Table 2: Details on datasets used in experiments.
For IMDB dataset, we trained Long Short-Term Memory (LSTM) network model 49 for 20 epochs; for MNIST dataset, we use Support Vector Machines (SVMs) with RBF kernel. For CIFAR- 10 dataset, we trained on the pre-activation Resnet with identity mappings (PreActResNNet-18 (50) for 100 epochs. We trained ImageNet-100 on ResNet-50 (51) backbone for 200 epochs with cosine annealing as the learning rate scheduler. For the autonomous driving object detection dataset, BDD100K, we fine-tune an improved pre-trained faster region proposal-CNe CNN network (Faster-RCNN ResNe150 (52) on COCO dataset (53) for 30 epochs.
20

---

D.2 Details on Baseline Methods
For samples (data quantity) from m data sources with mixing ratio p {pI, ................................ Pm:: we consider the following baselines
Linear: ((A(D(N, p); D == a'p + blog(N) + c, where a = {ao. Q1 ... am b, and c are coefficients be fitted.
Leave-one-oul (LO0) and Shapley can considered special cases for Linear, where the coefficients calculated as the marginal contribution of the data source (LO0) or its averaged contribution to different combinations of other data sources (Shapley) [18, as opposed to the least-square fitting as in Linear.
Pseudo-quadratic: [(A(D(N, p); D == E
P + of pr + co) + blog((N)
Quadratic: C(A(D(N, p); D == -m (c2 p2 +c ps +co) cc) + (c p.P,)) +blo((N)
Rational: ((A(D(N, p); D EM EE c² pp)
blog(N)
We fit the Rational baseline according to the setup detailed in 13 and to our best effort. Originally, the method is intended for predicting log loss, whereas our case, we aim to predict model accuracy. Thus, we replaced the log loss with log (1 accuracy) for the prediction target.
D.3
Performance Prediction for Unseen Data Mixtures Ablation Study
Data Source
Case I
Case 2
Case 3
Case
Case 5
(0,3,6.7 7) (4,5
(1,7)
(1,5, 8,9,
{4. 6,9)
[0, 1,
6,,,,688.9)
(2,7)
(8)
(2
(,)
5,
(0,3, 4]
2,7,,
[3,5, 5,55,7. 7,
Table 3: Class distributions of data sources for five different cases.
In the previous experiments, we presented results over a single data source class distribution. Here, we present more comprehensive view of the projektor's performance by running multiple times over random class distributions of data sources, according to Table As shown in Table 4 for all cases, either projekt oz/ CS pro or achieves the highest performance, while baseline methods struggle to get close performance. Although, in many cases (1,2.3,5), Quadratic or Rational baseline obtains the lowest training data MAE, but in test performance, these methods have poor generalization scores, which indicate high overfitting to the training data. These results indicate that predicting performance from data source composition is insufficient for fitting and OT distance plays an important role in better alignment data sources to the model accuracy.
Method
Case
Case 2
Case 3
Case 4
Case
projektor/PQ
1.41 (A 0.00)
1.11 1,35
(A 0.00)
1.24 1.22 (A 0.15)
0.88 1.39 (A 0.00)
0.84 1,26 (A 0.00)
projektar/CS
7.05 (A 4.44)
1.12 1.49
0.14)
1.24
1,07 (A 0.00)
0.91 -231
0.92)
097 1.95 0.69)
Linear
3.31 10.67
8.00)
5,51 (A
4.35
(A
624)
5.38 8.8
(A
7.53)
4.44 6.04 4.78)
Pscudo-Quadratic
1.35 10.18 7.55)
1.04
3,86)
1.82
3,87)
1.34
2.15)
L88 2.90 1.64)
Qundratic
1.05 -942 6.79)
1.02 4.94 (A 3.59)
0.97
3.48 (A 3.41)
0.93 3328 1.89)
0.88 5.22 (A 3.96)
Rational
1.06 15.99 (A 13,36)
1.09
(2 (A.49) 4.89)
1.04 4.89 (A 3.82)
1.04 -300 1.63)
0.83 2.95 1.69)
Table 4: Multiple runs over random data source class distributions in CIFARIO. For each cell the left-column value denotes the fitting MAE value for fitting the training data and we marked in underlined green, whereas the right-column value denotes the MAE value for testing data and we marked in bold red.
D.4 Extended Application Multiple Number of Data Sources
So far, our experiments are focused solely on the cases with three data sources. To show the practicality of our method, we extend projektor more challenging cases, where we have more than three data sources. Specifically, we explore the setting with 4, 5, and 6 data sources and illustrate
21

---

the results with baseline comparisons in Table As we observe, our methods, both pro jektor/cs and pro jektorr///eee /Po, achieve the best testing MAE scores as well as one of the lowest training MAE scores. While both Linear and Pseudo- Quauud-t baselines are under-performing and receive poor testing MAE values. observed previous experiments, Quadratic baseline presents strong training data fitting but weak testing predictions. For the Rational baseline, we tried our best to fit this method, but we demonstrate that with the larger number of data sources, it is even harder to properly train this function which results in increasing errors, respectively. This experiment demonstrates the capability of our method to extend to more practical settings with multiple data sources.
Method
4 Sources
5 Sources
6 Sources
pro /ro
1.15 2.83(A 0.00)
0.99 3,40( 0.00)
1.14 2.53(A 0.00)
projekttor/Co
1.28 3.51 (A 0.68)
4.13 (A 0.74)
1.29 2.91 (A 0.38)
Linear
2.12 4.58 (A 1.75)
2.38 4.74 (A 1.34)
1.99 4.00 (A 1.50)
Psuedo-
Psuedo-Quadratic Quadratic
1.32 3,97 (A 1.14)
(A 1.84)
1.34 3.91 (A 1.38)
Quadratic
1.06 (A 1,10)
0.97 0.97-8.70 (A 5.30)
1.02 4.67 (A 2.14)
Rational
52.72 57.93 (A 55.1)
611.5 501 (A 497.6)
1034 1155 (A 1152)
Table 5: Performance on various number of data sources (4,5,6) in CIFAR-IO. 10. For each cell the left-column value denotes the fitting MAE value for fitting the training data and we marked in underlined green, whereas the right-column value denotes the MAE value for testing data and we marked in bold red.
D.5 Optimal Data Source Composition
Here, we showcase the optimal data source selection on CIFAR-10 10 dataset. Similarly as in Section 5.2 given pilot dataset of size 1.5K from each of the three data sources, we would like to find the mixing ratio that can maximize the model performance when trained on 10K dataset. As illustrated in Fig. 9 pro jektor/Pppp and pro projektor
select mixing ratios that achieve the highest model performance, gaining 4% and 2% in performance improvement over the best baseline method. Our methods select nontrivial mixing ratios which also outperform the uniform mixing ratio by 5% and 3%, respectively. Furthermore, we observe in Fig. 10 that our methods also perform well in performance prediction into larger data scales. Specifically pro jekotr/et//ee /ooorr//gg and pro projekkor /Cs predictions are within 2.2% discrepancy from the actual model performance for 10K dataset, while the best baseline method prediction has over 5% error. To demonstrate projeks jektor capability, we additionally present results on the case where data sources contain some mislabeled data.
Source
Model Performance/5)
Projektor/PQ [Ours]
40
72%
Model werfooraacaaaa
6L.1
6o
62.0
$8.2
Prejekter/CS (Ours)
38
30
32
70%
45.5
100
37%
Pseudo Quadratie
Quadratic
28
68%
Rational
44
19
37
67%
LOO
100
31%
30
Shapley
100
37%
(0uns) CS
(Dung
Guadratic
Shapley
Rase
67%
Figure 10: Performance projection of selected op- timal mixture ratios (Fig. onto 10K CIFAR-10 from 1.5K samples. Comparison with the actual model performance.
Figure 9: Optimal source composition selection for 10K CIFAR- 10 from 1.5K samples and the actual
model performance.
In this case, we assume data sources have noisy labels. In particular, data sources have 20%, 15%, 25% of mislabeled data, respectively. In Fig. 11 surprisingly, we observe that even though the first data source has 20% of mislabeled data, choosing more of that source improves the model performance. A possible explanation is that the first source dataset contains classes that are important for learning
22

---

and the second source has less important for model performance or requires less to be learned (especially since it has lower mislabeled rate). Moreover, we notice that projektor ratio can improve the model performance by over 3% from the performance of the best baseline. Results from Fig. 12 indicate that projektor 's performance projection onto 10K dataset has prediction error within 1.7%, while the best baseline has an error of over 2.2%. To sum up, we have shown possibilities of projekt baseline methods have the same mixing ratio over any data scales, our method chooses different optimal mixing ratios for different data scales, which has shown some advantages in improving model performance.
mixing
in mixing ratio selection and performance prediction. While
Source1
Data Source
Data Source
Hodel Perforaancx(1
Random Sclecion
Prist ctod Accuricy
Actial Accuroc
60
38.5
Projektar/PQ [Duns
38
31
31
58%
Model Performances
56.1
Projektor/CS (Ourd)
37
31
32
57%
40
40.1
Linear
30
Pesudo-Quadratic
33
40
20
Rational
38
29
33
54%
LOO
100
26%
Shapley
Protettor
Projektor
Linear
Poeudo-
Rational
LDO &
33
33
34
54%
Figure 12: Performance projection of selected opri- mal mixture ratios in (Fig. [ onto 10K CIFARO- 10 from 1.5K samples. Comparison with the actual model performance.
25 50
75
100
Figure 11: Optimal data source composition selec- tion for 10K CIFAR-10 10 from 1.5K samples with misla- beled data sources, and the actual model performance.
Appendix E
Additional Discussion and Broader Impacts
Limitations. Despite contributing new perspective with performance and efficiency improvements, this work still has some limitations and opens up many new investigation venues: (1) How to quantify the influence or further lift the dependence on validation data? While a validation set representative of the downstream lcarning task is a common assumption in the ML literature, it may or may not be available during data exchange and its quality may vary. (2) Our design could be vulnerable to estimation errors in the scaling law for data sizes, which could lead to magnified prediction errors on larger scales and affect data acquisition decisions [54]. Especially, noises inevitable due to the performance stochasticity of ML models as well as the sampling process to generate the pilot dataset. (3) Our current framework does not take into consideration of broader tasks that aim for goals beyond accuracy, e.g., fairness, variable data costs, as well as broader acquisition scenarios where data sources have misaligned feature space. Incorporating other objectives and extending to heterogeneous data sources is an exciting direction. (4) Our setup considers honest data providers and the requested samples are faithfully sampled from the actual data sources, leaving an in-depth study of potential security risks, such as malicious data manipulation [28 to future work.
Broader Impacts. This work will have significant impacts beyond advancing the research data selection and data markets. The techniques developed in this work can be applied to a variety of other subfields of ML related to data acquisition, data valuation, interpretability. robustness, etc. The results of this paper will facilitate the automation of data selection and quality management in machine learning, which in turn, accelerates research and improves services based on ML. Data exchanges and data markets are also at the heart of the global data economy. The advancements in practical data exchanges in this work will substantially benefit the development of data markets and promote data sharing, contributing to the business and economy as well as society as a whole.
23


Reviews:
[{"review_id": "QEZ1CmPQF4", "reviewer": "Reviewer_Paf5", "summary": "The paper considers the data acquisition setting where benefit to model performance from acquiring new streams of training data may be supported by the inspection of limited segments of a candidate corpus, such that one may wish to evaluate the benefit to model performance to support selection from a set of candidate corpi. Such forms of data acquisition evaluation for estimating model performance impact has much prior work, the first claimed novelty in this paper is associated with utilizing forms of optimal transport metrics to improve selecting ratios of data to be served from multiple candidate corpi. This estimate is benchmarked to suggest a resulting improvement to estimates of model performance in comparison to prior work, and then by extending the use of the derived estimates of an ideal ratio to another component of evaluation associated with data scaling laws there is another improvement to estimates of model performance. ", "questions": "In figure 5 do you have any theories as to why with increasing sample scales the method appears to increasingly underestimate model performance?", "limitations": " ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The abstract claims of significant improvements to computational costs of the application with use of this approach was suggestive of a material contribution but I had difficulty evaluating that further from the rest of the writeup. It is intuitive that for real world application in industry the prioritization of data streams is impactful to bottom line, such that scalable solutions to more easily allocate training compute would be impactful. ", "weaknesses": "The contributions of the paper associated with incorporation of an optimal transport metric on its own strikes me as much more of an iterative contribution rather than a significant one. I recognize this is common to the field, but I had a hard time getting comfort on the statistical significance of the benchmark findings - even a few more details on experimental setup could have possibly helped. I agree that the figures appear to demonstrate improvement but I don’t think the paper convinced me that it was anything more than suggestive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *“...claims of significant improvements to computational costs of the application with use of this approach was suggestive of a material contribution but I had difficulty evaluating…”* (*Strengths*)\n\n***TL;DR: Scalability improvements of the proposed framework are on magnitudes.** Methods proposed in this work only require fitting as few as a fixed number of 2 parameters, orders less than existing methods. Figure 3 shows **empirically that the proposed method outperforms the strongest baselines with less than 1/5 computational overhead** of the latters'.*\n\n**Re**: Thanks for pointing out a potential source of confusion in the presentation of this work. **Previous approaches rely on fitting parametric functions as surrogates** (e.g., rational functions [1], or Datamodels [2]), which treat the model and data pipelines as a black box. The accuracy of such methods fundamentally relies on the expressive power of the surrogate functions. **The surrogates are non-informative**–i.e., they predict the performance solely based on the size of data or its composition ratios (how much from each data source) while neglecting the information of the content of data. **The improvement in their accuracy primarily relies on adding more parameters, which require more repetitions of the model training to fit and mitigate the elevated consequence from overfitting.** Under practical limitations on computing resources, the accuracy often needs to be traded off with robustness/reliability (using higher-order nonlinear surrogates may cause large deviations due to overfitting) and is generally unsatisfactory.\n\n**Our methods leverage data distance (measured by the Optimal Transport distance) to incorporate additional information into the prediction of model performance. Our simplest model, OTPP/CS only uses a fixed number of 2 parameters that can be well-fitted through a handful of model training and outperforms large parametric models with the number of parameters quadratic to the number of data sources, demonstrating a substantial advantage over the previous approaches. The sharp contrast highlights the remarkable scalability improvements from the proposed approach.**\n\nEmpirical validations are depicted in Figure 3. OTPP/CS converges to a low prediction error (i.e., high accuracy) after 10 times of model training and outperforms non-informative parametric surrogates with up to a quadratic number of parameters even after 50 times of model training.  The scalability improvements are on magnitudes.\n\nThe authors acknowledge the responsibility for the presentation of this work and appreciate having the chance to revise the manuscript accordingly for better readability.\n\n> *[1] Tatsunori Hashimoto. Model performance scaling with multiple data sources. In International Conference on Machine Learning, pages 4107–4116. PMLR, 2021.*\n\n> *[2] Ilyas, Andrew, et al. \"Datamodels: Predicting predictions from training data.\", Proceedings of the 39th International Conference on Machine Learning, 2022.*\n\n***\n\n> *\"...contributions of the paper associated with incorporation of an optimal transport metric ...more of an iterative contribution rather than a significant one.\"* (*Weaknesses*)\n\n**Re**: We thank the reviewer for the effort to help assess the contribution of this work and for the responsibility held to the community. We would like to take this opportunity to share our understanding of the importance of this work and how it positions in the field. ***[We provide a full response to this question in a separate comment to all reviewers. (R3)]***\n\n***\n>  *\"...significance of the benchmark findings - even a few more details on experimental setup could have possibly helped.\"* (*Weaknesses*)\n\n**Re**: As part of the effort to improve the presentation of this paper, ***[We provide a full response to this question in the summarization rebuttal to all reviewers. (R1)]***\n\nWe apologize for the lack of clarity in experimental details. We are revising our manuscript in light of your valuable feedback.\n\n***\n\n> *“In figure 5 do you have any theories as to why with increasing sample scales the method appears to increasingly underestimate model performance?”* (*Questions*)\n\n**Re**:  Figure 5 is a qualitative figure that is provided to facilitate the reader’s interpretation of how this framework works. We apologize if it ends up the opposite and adds to the confusion.\n***[We provide a full response to this question in a separate comment to all reviewers. (R4)]***\n\nWe will add more explanations to avoid possible misunderstandings. \n"}, {"review_id": "72Uz3mbYNg", "reviewer": "Reviewer_WNMz", "summary": "This paper considers the problem of predicting model performance (and subsequent data selection) under a partially revealed setting. The two challenges are estimating the right proportions as well as extrapolating to dataset scales beyond the observed scales. The paper proposes a two-stage approach called projektor: in the first stage, it predicts model performance as a simple function \n(either an affine transformation of a quadratic) of the optimal transport distance between target distribution D_val and the input distribution. In the second stage, the performance is extrapolated using ideas and functional forms from neural scaling laws. \nThe paper also proposes a gradient-based method for data selection using the model performance estimator.", "questions": "- In stage two, is the intuition that by fitting the two different scales N0 and N1 in stage 1, you eliminate the need to explicitly estimate the scaling parameters (alpha and C) in the scalings laws?\n- How are OT distances computed for images? Is it done in some feature space? which one?", "limitations": "Fine.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Novel technical solution to a well motivated problem. The proposed approach is well-motivated, interesting, and well presented.\n- Well written overall: motivates problem well (section 1), contextualize well within related work (section 2), clear set up (section 3), \n", "weaknesses": "- The experiments are a bit disappointing in terms of their practicality. \nAs far as I can tell, the data sources are just different subsets of distributions of fixed datasets (potentially unlabeled or mislabeled). Given the recent developments in the field (training language/diffusion models on large unfiltered data sources), I would have liked to see more experiments on more realistic/noisy data sources. Even for ImageNet, a better data source would be raw images from different online sources (e.g., Flickr),. Hence, it's a bit hard to judge the practical utility of the proposed approach (beyond the settings considered in the paper, which I am not sure are settings in practice where people need much better data selection strategies).\n- Section 5 (evaluation) is a bit hard to follow in terms of setup: what exactly are the datasets used, etc.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *\"...experiments are a bit disappointing in terms of their practicality ...would have liked to see more experiments on more realistic/noisy data sources. \"* (*Weaknesses: 1*)\n\n**Re:** We appreciate the review for the crisp understanding of the conceptual narrative of this work, and would like to take this chance to discuss a bit more on the considerations in designing empirical studies for this work.\n\n**In short, precisely as the reviewer pointed out, this data selection pipeline developed in this work could potentially apply to a variety of downstream tasks in the real-world.** ***[We provide a clear and complete presentation of experiments conducted in this work in the summarization rebuttal to all reviewers (R1).]***\n\nBesides, we are in active collaboration with industrial researchers on a variety of problems. **Besides, in collaboration with industrial partners, we succeeded in deploying the proposed data selection pipeline in state-of-the-art automatic speech recognition (ASR) use cases.** ***[We are adding this as a set of NEW experiments with additional results provided in the summarization rebuttal to all reviewers (R2).]*** In ASR tasks, a number of datasets are available consisting of many audio sources (e.g., accents, recording quality, environmental conditions), the practitioners are able to curate a validation dataset representing their target user groups and scenarios (e.g., North American market, home environment) that they hope their applications to perform best. Due to the large scale of samples, training the model each time is resource-intensive. **Finding an ideal combination of data for training on the target tasks can be prohibitively expensive if conducted with manual searches.** The tools proposed in this work provide a complete landscape of target model performance for any data composition on any scale (data quantity), helping inform practitioners in deciding what type of data to acquire and by how much so that the objectives are best met.\n\n**Together with the experiments presented in the manuscript, we demonstrate a diversity of tasks with real-world instances, practical considerations, and realistic settings, showcasing the versatile capabilities of the proposed framework as well as the significance of the potential impact on both industrial applications and academic research.**\n\n***\n\n> *\"Section 5 (evaluation) is a bit hard to follow in terms of setup: what exactly are the datasets used, etc.\"* (*Weaknesses: 2*)\n\n**Re:**  We apologize for lack of clarity in experimental details and will address them properly here. ***[We provide a full response to this question in the summarization rebuttal to all reviewers (R1).]***\n\n***\n\n> *\"In stage two, is the intuition that by fitting the two different scales N0 and N1 in stage 1, you eliminate the need to explicitly estimate the scaling parameters (alpha and C) in the scalings laws?\"* (*Questions: 1*)\n\n**Re: Exactly!** Function fitting is replaced with a direct mapping. Simple, natural, and quite effective in solving the practical challenge.\n\n***\n\n> *\"How are OT distances computed for images? Is it done in some feature space? which one?\"* (*Questions: 2*)\n\n**Re:** We followed the standard treatment for calculating distributional discrepancy measures including OT distances used in this work. For vision tasks other than MNIST (on which we directly compute OT distances on the pixel space), **we trained a smaller model (ResNet-18) from scratch on the validation data (owned by the data buyer) and then used its output of the penultimate layer (before the final output layer) as the feature space (on which the OT distance is then computed).**\n\nThe train-from-scratch procedure can also be replaced by fine-tuning off-the-shelf pre-trained feature embedders (such as the models pre-trained on ImageNet). Such procedures are more common for language model tasks and less important for vision tasks. For tabular data, generally, no feature embedding is needed and we only normalize the features of different dimensions to the same scale. If the dimension of data is very high (e.g., hundreds), sometimes dimension reduction (e.g., PCA) or feature selection (e.g., RFE, recursive feature elimination) can be applied.\n"}, {"review_id": "pXJoiiYV9s", "reviewer": "Reviewer_d24K", "summary": "Developing ML systems typically requires collecting data from multiple sources. A natural question is how much data to collect from each source. This paper proposes a two-stage estimator that (1) estimates the relationship between data set proportions vs validation loss using Optimal Transport and then (2) optimizes the proportion of each source to emphasize in a dataset given a fixed budget. They validate aspects of the framework on several settings to demonstrate improved performance from good data selection decisions.", "questions": "Please see the above weaknesses. Most importantly:\n\n1. How does the log learning curve estimation framework in the proposed setup compare vs power law-based estimation strategies? Can we use a power law estimator rather than a log one? How would that affect the empirical performance?\n2. Can you give better interpretation of the OT-based estimator? For example, it would be nice to read out an OT-based bound on validation loss and then show how the proposed estimator’s components capture the different elements of the bound.\n3. How does the methodology compare against the related prior literature on estimating and optimizing data set sizes? \n\n\n", "limitations": "Limitations are reasonably discussed in the Appendix.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The topic is timely and of growing importance. The optimal transport perspective to data selection is novel and interesting.", "weaknesses": "1.\tThe paper makes some confusing and potentially incorrect statements regarding the related literature, particularly with power laws. For example, \n   - Abstract: “these scaling functions are black-box …”. Neural scaling laws are very interpretive and give clear explanations on how data set size relates with model performance. More so, I do not see how the proposed work is less black-box than a neural scaling law.\n   - The general learning curve assumed in this work is a logarithmic model of “performance $= -\\alpha \\log N + C$ where $\\alpha, C$ are parameters to learn. While logarithmic learning curves have been used to model performance in the past, the current literature typically considers power laws. Moreover, the cited work in this part of the paper, [1], specifically discusses power laws and how they may be more effective than log learning curves (e.g., Fig 23 in [1]). While this paper is free to use a logarithmic model for ease of implementation or other design choices, the surrounding discussion is unclear about the motivation of the log learning curve as well as the implications of using log curves vs power laws in this setup.\n2.\tThe problem of understanding how much data to collect given a fixed budget or given a performance threshold from multiple sources has been studied in several recent prior works, for example [2, 3]. It would be important to differentiate this work from this prior literature (e.g., the OT framework). \n3.\tThe baselines considered in the numerical experiments are mostly simplified versions of the proposed method. Pertaining to the above 2 points, it would be important to validate the design decisions against more relevant related methods, for example, the data selection methods in [2, 3], or at the very least, a meaningful comparison against power law-based performance estimation strategies. \n4.\tWhile the OT-based estimator in eq. (1) is interesting, it is not well-motivated in this paper. I appreciated the interpretation of $a_1$ and hope to read an interpretation of $a_0$ as well. Furthermore, while bounds on validation loss w.r.t. OT have been studied, how tight are these bounds? This is particularly important because OT is used as an estimator rather than a bound, suggesting either (i) that OT can very tightly bound validation loss or (ii) that the constant term $a_0$ is carrying a lot of the water in this bound. This is particularly interesting because $a_0$ is assumed to be independent of data set size.\n5.\tIf I understand correctly, in the numerical experiments, the initial pilot training data set is fixed to 55% of the full data set size. This generally makes the learning curve estimation problem significantly easier as we are well in the power law stage of the learning curve at this point. Moreover, this experiment setup weakens the motivation of having access to only a small pilot data subset. It would more interesting to consider meaningfully smaller data fractions.\n\n\n[1] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\n[2] Tejero, Javier Gamazo, Martin S. Zinkernagel, Sebastian Wolf, Raphael Sznitman, and Pablo Márquez-Neila. \"Full or Weak annotations? An adaptive strategy for budget-constrained annotation campaigns.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11381-11391. 2023.\n\n[3] Mahmood, Rafid, James Lucas, Jose M. Alvarez, Sanja Fidler, and Marc Law. \"Optimizing data collection for machine learning.\" Advances in Neural Information Processing Systems 35 (2022): 29915-29928.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *\"This paper proposes a two-stage estimator that (1) estimates the relationship between data set proportions vs validation loss using Optimal Transport and then (2) optimizes the proportion...\"* (*Summary*)\n\n**Re:** The authors would like to thank the reviewer for providing detailed and in-depth feedback and comments on the manuscript. **We would like to first clarify a potential misperception and the concerning confusion it may have caused.**\n\nThis work proposes a two-stage scheme for **performance prediction (Stage I) and projection onto large scales (Stage II)**, referred to as **\"performance scaling with data composition\"** and **\"performance scaling with data quantity\"** in the manuscript, **respectively. The use of term \"performance scaling\" may be somewhat abused that causes unnecessary ambiguities in different contexts.** ***[Due to its critical importance and the space limit here, we are providing the full response in a separate comment below (C1).]***\n\n***\n\n> *\"...some confusing and potentially incorrect statements regarding the related literature, particularly with power laws. For example, Abstract: “these scaling functions are black-box …” \"* (*Weaknesses: 1.1*)\n\n**Re:** The authors acknowledge the responsibility for the choice of phrases that may have caused unnecessary ambiguities under different contexts and apologize for the confusion it may have caused. **As discussed above, we may have abused the term \"performance scaling\" and used it for different meanings in contexts outside of scaling laws (w.r.t. data quantity/data scales).**\n\n**We agree with the reviewer that these scaling laws are intuitive and interpretable, and \"black-box functions\" are references to the use of non-informative surrogates in representations of the relationship between data composition from different sources and the resulting model performance (\"performance scaling with data composition\").** ***[Due to its critical importance and the space limit here, we are providing the full response in a separate comment below (C2).]***\n\n***\n\n> *\"... the current literature typically considers power laws... [1]... specifically discusses power laws and how they may be more effective than log learning curves...\"* (*Weaknesses: 1.2*)\n\n**Re:** Thanks for the excellent comment. We are appreciative of your insights and pointing out our negligence in elaboration of equations. **In short, we are using power laws rather than log functions. It looks like log-linear relationships because we brought everything to log space for better numerical properties**–we took log operations on both sides of the equation and the power law in the original space became log-linear in the log space. ***[Due to its critical importance and the space limit here, we are providing the full response in a separate comment below (C3).]***\n\n***\n\n> *\"...it would be important to validate the design decisions against more relevant related methods, for example, the data selection methods in [2, 3], or … against power law-based performance estimation strategies.\"* (*Weaknesses: 2, 3*)\n\n**Re:** We appreciate the reviewer for providing additional references and we would be glad to discuss them. At the time of submission, we were aware of [3] and cited a related paper [4] while [2] was published after our submission. **These papers are indeed interesting and provide inspiration for the line of research on data selection/data acquisition, though, we would like to point out that their scope and target problems are different from ours and the conceptual and technical contributions are non-overlapping.** ***[Due to its critical importance and the space limit here, we are providing the full response in a separate comment below (C4).]***\n\n\n***\n\n> *\"... while bounds on validation loss w.r.t. OT have been studied, how tight are these bounds? ... suggesting either (i) that OT can very tightly bound validation loss or (ii) that the constant term is carrying a lot of the water in this bound.\"* (*Weaknesses: 4*)\n\n**Re:** This is also an excellent question. We would appreciate the chance to thoroughly explain about it. ***[Due to its critical importance and the space limit here, we are providing the full response in a separate comment below (C5).]***\n\n\n***\n\n> *“...the initial pilot training data set is fixed to 55% of the full data set size… makes the learning curve estimation problem significantly easier …weakens the motivation of having access to only a small pilot data subset… consider meaningfully smaller data fractions.”* (*Weaknesses: 5*)\n\n**Re: We apologize for the misunderstanding. 55% is for the part of work on predicting model performance on data composited from multiple data sources with arbitrary combinations (i.e., the proportion of data from each source), which is irrelevant to data quantity.** When fitting this relationship, we limit the proportion of data from each data source to no more than 55% such that we can test the **extrapolation performance** of the fitted predictors–to predict the performance of data compositions where the proportion of data from a data source exceeds 55%. Since the actual performance of these data compositions is not covered in fitting the predictors, we will be able to examine whether these fitted relationships suffer from overfitting that leads to excessive deviations in extrapolation tasks. **This is the first part of the work where we construct predictors on the pilot data with limited samples. We set the size of pilot data to 10%-20% of the larger data scales (data quantity).**\n\n**In the later part of the work, we extend the predictions to project them onto larger data scales (data quantity) where the target data scales are 5- to 10-fold of the pilot data.** E.g, for CIFAR-10, we assume access to 1k samples and project the performance to larger scales of 2k-10k. For ImageNet100, given 10k samples, we project performance to up to 50k samples. For the BDD100K dataset, we project from 1K samples to up to 5K samples."}, {"review_id": "ax2ZnWyonx", "reviewer": "Reviewer_WiXX", "summary": "In this paper, the authors address the problem of data selection, acknowledging that complete data availability is often not possible, and data can only be obtained from specific providers. They recognize that different data sources may have varying impacts on the performance of the model, emphasizing the importance of optimizing the data selection strategy to enhance model training. To achieve this goal, the authors leverage optimal transport, validation data, the target model, and scaling law to allocate the selection budget effectively across different data sources. The experimental results presented in the paper validate the effectiveness of the proposed method.\n\nPros:\n\nThis paper is well-written, presenting a clear and logical flow of ideas, along with a comprehensive description of the method employed. The setting of the study is intriguing, considering the current landscape where data providers offer limited options for data access. Exploring the optimal combination within a constrained budget is a valuable contribution to both academia and industry research.\n\nCons:\n\n1. While the chosen setting is interesting, it may be somewhat impractical since scenarios where different data providers offer data for the same task with identical distribution are uncommon. The authors implicitly assume that data from all providers share the same distribution, which might weaken the persuasiveness of the results. It would be beneficial if the authors could present a real-world scenario to support their assumptions.\n\n2. The assumption that the selection model has access to validation data with the same distribution as the testing data might not be practical in many cases. Often, the real testing data and its distribution are unknown, making this assumption less realistic.\n\n3. Some of the figures presented in the paper are difficult to comprehend. For instance, Figure 5 lacks clarity in terms of color interpretation, as well as the meaning of the x and y axes.\n\nOverall, this paper offers a solid contribution to the field of data selection. Despite the presence of certain strong assumptions, the insights provided are valuable for enhancing the data selection process.", "questions": "None", "limitations": "The assumptions are a bit strong and may degenerate the usefulness in practical setting.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written, presenting a clear and logical flow of ideas, along with a comprehensive description of the method employed. The setting of the study is intriguing, considering the current landscape where data providers offer limited options for data access. Exploring the optimal combination within a constrained budget is a valuable contribution to both academia and industry research.", "weaknesses": "1. While the chosen setting is interesting, it may be somewhat impractical since scenarios where different data providers offer data for the same task with identical distribution are uncommon. The authors implicitly assume that data from all providers share the same distribution, which might weaken the persuasiveness of the results. It would be beneficial if the authors could present a real-world scenario to support their assumptions.\n\n2. The assumption that the selection model has access to validation data with the same distribution as the testing data might not be practical in many cases. Often, the real testing data and its distribution are unknown, making this assumption less realistic.\n\n3. Some of the figures presented in the paper are difficult to comprehend. For instance, Figure 5 lacks clarity in terms of color interpretation, as well as the meaning of the x and y axes.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *\"impractical since scenarios where different data providers offer data for the same task with identical distribution are uncommon... implicitly assume that data from all providers share the same distribution...\"* (*Weaknesses: 1*)\n\n**Re:** We appreciate the valuable feedback on the presentation of this manuscript. **We do not make assumptions about data being identically distributed. **\n\n**We rely on the fact that the distributions are different so that the discrepancy measures (such as Optimal Transport) can be leveraged to provide information from data in the construction of performance predictors.** This is also under the technical challenge of scaling laws, where the scaling ratios vary for data of different distributions. We then tackle it with the organic integration of the OT-based performance predictor to construct a self-adaptive pipeline, automatically fitting scaling parameters for the data distribution based on the prediction tools.\n\nIn this work, we refined our scope to data of the same type (uni-modal)–for example, images are of the same format (i.e., resolution or the number of color channels), though, we note this framework is capable of extending to multi-modal data (e.g., multi-modal data by aligning the distribution on the joint embedding space of image and text). Our work provides the groundwork for future papers that might explore modality-misaligned data sources.\n\nWe apologize for the potential unclarity and confusion it may have caused. **We would thoroughly proofread our manuscript to improve readability and prevent potential misperceptions.**\n\n***\n\n> *\"The assumption that the selection model has access to validation data with the same distribution as the testing data might not be practical in many cases.... the real testing data and its distribution are unknown, making this assumption less realistic.\"* (*Weaknesses: 2*)\n\n**Re:** Thanks for the interesting comment and we would be happy to discuss more about it.\n\nIn general practice, data acquisition pipelines typically randomly partition the validation data into non-overlapping parts and use them for testing and validation purposes, respectively. In such cases, validation data and test data are i.i.d. **We also followed this scheme in this work.**\n\n**To implement the tools proposed in this method, the practitioner would need to first curate a validation dataset that represents the target tasks they expect the collected data to perform well on. This dataset does not need to be large, but is often considered balanced that can well represent all the target objectives.** This is generally perceived as plausible for industrial practitioners.\n\nFor example, in state-of-the-art automatic speech recognition (ASR) tasks, a number of datasets are available consisting of many audio sources (e.g., accents, recording quality, environmental conditions), **the practitioners are able to curate a validation dataset representing their target user groups and scenarios** (e.g., North American market, home environment) that they hope their applications to perform best. Due to the large scale of samples, training the model each time is resource-intensive. Finding an ideal combination of data for training on the target tasks can be prohibitively expensive if conducted with manual searches. **The tools proposed in this work would provide a complete landscape of target model performance for any data composition on any scale (data quantity), helping inform practitioners in deciding what type of data to acquire and by how much so that the objectives are best met.** ***[We are adding this as a set of NEW experiments with additional results provided in the summarization rebuttal to all reviewers.]***\n\nBesides, in relevant research of data valuation *(ref [3-8] in the manuscript)* or coreset selection *(ref [2][20][21] in the manuscript)* (cited and discussed in this work), the availability of such validation datasets is also assumed. **The assumptions made in this work do not exceed those of existing research and well align with practical needs in industrial applications.**\n\n***\n\n> *\"Some of the figures presented in the paper are difficult to comprehend. For instance, Figure 5 lacks clarity in terms of color interpretation, as well as the meaning of the x and y axes.\"* (*Weaknesses: 3*)\n\n**Re**:  Figure 5 is a qualitative figure that is provided to facilitate the reader’s interpretation of how this framework works. We apologize if it ends up the opposite and adds to the confusion.\n***[We provide a full response to this question in a separate comment to all reviewers. (R4)]***\n\nWe will add more explanations to avoid possible misunderstandings. \n"}]

Author rebuttal:
### Summary:
**All reviewers recognize the importance of the problem and the conceptual novelty of the proposed framework and original methods.**

Reviewers WNMz and WiXX confirm the **solid development** of this work and its **multi-faceted technical contributions**. Reviewers WNMz and WiXX appreciate the **overall presentation** of this work and acknowledge it as well-written, well-motivated, and well-contextualized with a clear and logical flow of ideas and comprehensive descriptions. Reviewer Paf5 and WiXX acknowledge the **significance of potential impacts** on industrial applications.

**Reviewers share the questions for experiment settings and their practicality.** Reviewer d24K did not include the part of work on predicting model performance for data composed from different combinations of multiple data sources. **Both point to the need for improving the current presentation of the manuscript and including additional discussions.**

***

### In our response, we would like to

- **R1.** Clearly present the experiments conducted in this work, **showcasing their diversity and practicality** and clarifying their settings and considerations.

- **R2.** **Provide additional results of a NEW set of experiments on data selection for automatic speech recognition (ASR)**, a highly practical application in collaboration with industrial practitioners.

- **R3.** Further discuss the **multi-facet contributions** of this work and their **potential impacts** on both industrial and academic research.

- **R4.** Clarify **Figure 5** with additional explanations on its settings and visualization scheme.

***
### R1. Experiments conducted in this work

**a. We validate the proposed pipeline in stylized tasks with standard datasets. On MNIST (simple patterns), IMDB (text data), CIFAR-10 and ImageNet100 (vision tasks)**, we divide the samples into different data sources representing specific data categories, where each source contains only certain classes that are not necessarily exclusive. Then, based on a small amount of pilot data that is considered accessible to the practitioners (i.e., <20% of the target scale), we construct the "projektor" performance prediction tools that visualize the entire performance landscape for any data composition at any data size. We examine the accuracy of these predictions against the actual model performance and compare the error with baselines. Also, we perform data selection based on the predicted performance. We then train the model on the selected data and demonstrate advantageous performance for data selected by "projektor" compared to data selected by baseline methods.

**b. We then examine the practical performance of the proposed methods in real-world instances and extended scenarios.**

Beyond training from scratch, we apply our framework in **data selection tasks for fine-tuning**, which is of high relevance for pre-trained large models that are attracting growing attention. In particular, we implemented “projektor” to select fine-tuning data from the **autonomous driving dataset BDD100K**, the largest open driving video dataset for multi-object tracking (MOT) and segmentation (MOTS) challenges where we use the image frames, for a **Faster-RCNN model pre-trained on COCO**, a large-scale object detection, segmentation, and captioning dataset. For autonomous driving tasks, trained models are often sensitive to diverse weather conditions which limit the visibility of the road. **Thus, we divide the BDD100K data sources into three different challenging visibility categories: daytime, night, or dawn/dusk.**

Similar to previous tasks, we construct our predictors with as few as 1k samples and predict model performance for up to 5k samples.  Visualized in Figure 1(b), **selecting data based on our predictions achieves significantly higher model performance than random selections, and the predictions are highly accurate**–not deviating from the actual accuracy for more than 0.4%, indicating projektor’s practicality in realistic instances and its extended capability for fine-tuning tasks.

Additionally, to model the **effects of data sources with varying data quality**, we included experiments with **added label noise** to a portion of sources (10%-20%). We projected performance from 1K samples onto larger data scales 2-10K and observed (Fig. 4) that the proposed methods **achieve the lowest errors** (MAE scores)–the predicted performance well aligns with the actual performance even in the case of noisy labels, showcasing that the proposed data-distance-based approach **well models the effects of varying data quality and accommodates such tasks with high practical relevance.**

***
### R2. Additional Results: automatic speech recognition (ASR)

For ASR tasks, there are often a number of datasets available, consisting of many audio sources (e.g., accents, recording quality, environmental conditions). **Due to the large scale of samples, training the model each time is considerably resource-intensive.** Finding an ideal combination of data for training on the target tasks can be prohibitively expensive if conducted with manual searches. We consider the case of **fine-tuning a pre-trained Emformer RNN-T model on recordings from LibriSpeech and TEDLIUM-3**, datasets with contrasting speech recording styles–**the former contains clear and stable read speech (scenario 1) and the latter contains spontaneous speech in non-studio environments (scenario 2)**. Our target was to find the best composition of these data sources to improve the fine-tuned performance on both scenarios **with access to limited data (1% of total recordings, ~1hr).**

***(continuing in the comment below)***


Meta-review: In the reviewers' words: this paper studies a "well-motivated" and "timely" problem with an approach that is "novel", "interesting" and "well-motivated". It is "well-written" with a "clear set up" and "contextualizes" the work within the literature. Given the unanimous recommendation for acceptance from the four reviewers, I recommend accepting this "valuable contribution".
Decision: Accept (poster)