Predicting the impact of dataset composition on model performance
Authors: Tatsunori Hashimoto
Venue:  (2021)
Keywords: Experimental design
generalization
data collection

Abstract:   Real-world machine learning systems are often are trained using a mix of data sources with varying cost and quality. Understanding how the size and composition of a training dataset affect model performance is critical for advancing our understanding of generalization, as well as designing more effective data collection policies. We show that there is a simple, accurate way to predict the loss incurred by a model based on data size and composition. Our work expands recent observations of log-linear generalization error and uses this to cast model performance prediction as a learning problem. Using the theory of optimal experimental design, we derive a simple rational function approximation to generalization error that can be fitted using a few model training runs. Our approach achieves nearly exact ($r^2>.93$) predictions of model performance under substantial extrapolation in two different standard supervised learning tasks and is accurate ($r^2 > .83$) on more challenging machine translation and question answering tasks where baselines achieve worse-than-random performance.

Under review as a conference paper at ICLR 2021
PREDICTING THE IMPACT OF DATASET COMPOSITION ON MODEL PERFORMANCE
A Anonymous authors Paper under double-blind review
ABSTRACT
Real-world machine learning systems are often are trained using mix of data sources with varying cost and quality. Understanding how the size and compo- sition of training dataset affect model performance is critical for advancing our understanding of generalization, as well as designing more effective data collec- tion policies. We show that there is simple, accurate way predict the loss in- curred by a model based on data size and composition. Our work expands recent observations of log-lineat generalization error and uses this to cast model perfor- mance prediction as a learning problem. Using the theory of optimal experimental design, we derive simple rational function approximation generalization error that can be fitted using a few model training runs. Our approach achieves nearly exact (r² -93) predictions of model performance under substantial extrapolation in two different standard supervised learning tasks and is accurate (r >> .83) on more challenging machine translation and question answering tasks where base- lines achieve morse-than-randooe performance.
INTRODUCTION
The success of large scale machine learning systems depends critically on the quantity and quality of data used during training, and we cannot expect these systems to succeed there is not enough training data or if that data does not cover all the phenomena contained in the test distribution (Ben- David et al. 2010).. Knowing this, the designer a machine learning system might create multiple sources of data, with each targeting different feature domain that the model ought to do well on (Crammer et al., 2007 Wang et al. 2019a). This data-driven design strategy provides powerful tools to improve and evaluate model behavior, but also poses an additional challenge: what is the right way to combine these various data sources? What is the optimal data collection policy for given budget?
Our goal to answer these questions by quantifying the relationship between data sources and model performance how well will our model do we were train on samples using data mixture (41 allow us to both identify the optimal data collection policy and quantify cost-performance tradeoffs.
over our K data sources. A precise model for predicting model performance will
The starting point of our work is the recent observation across speech, vision and text (Hestness) al.) 2017 Kaplan et al.) 2020 Rosenfeld et al.) (2200) that the empirical performance of model remarkably predictable, and follows the log-linear formula
log(error) ~ -a log(n) + C.
(1)
In this work, we expand this observation to the -ulti- setting and discover the surprising fact that the slope of the log-linear relationship (a:) does not vary with data composition and that the data composition only affects the intercept (C).
The simple dependence of log-error on data size allows us to reduce the problem estimating model error into learning problem. Our approach is straightforward: we hypothesize that model error follows (nn, q) := exp( -a (n) (o))(( for a simple parametric functional form C(q), and fit this to observed pairs of (n, q, error) that obtain by subsampling the dataset and re-training a model. We show that there is a natural and simple choice C'q)) as a rational function that we derive from optimal experimental design for linear regression, M-estimation, and nonparametric

---

Under review as a conference paper at ICLR 2021
smoothing. The simple and parametric dependence of ((n,9) n allows estimates to predict model performance under substantial extrapolation in data size.
our resulting
Empirically, the resulting predictions extremely accurate and hold under substantial extrapo- lation. On the Amazon review prediction dataset (Mansour et al., 2009). we can learn to predict model performance nearly perfectly (r² 0.96) from small dataset of 1200 examples across sources and extrapolate to predict the model error on datasets of up to 4000 examples. We show this high accuracy continues hold on real-world task oriented dialogue system (r² 0.93), multi-domain machine translation system (r² 0.83), and boolean question answering with weak supervision (r² 0.86). In each of the cases, our proposed approach substantially outperforms the best baseline, with the baselines performing worse-than teaat-aannnnnnnnn in both the machine translation and question answering tasks.
Related work Quantifying the effect of data composition model performance is closely lated to the classical ideas of optimal experimental design, as well as more recent machine learning methods such as active learning and data valuation.
Our work will draw inspiration from the classical V-optimal experimental design (John & Draper 1975] as a way understand how model performance will change with the data collection policies. However, our approach differs substantially beyond this. Instead of making strong linearity assump- tions and identifying closed form formulas for model performance, treat identifying the impact of data sources on errors as itself a prediction problem, which allows us to quantify these effects for neural networks and non-separable objectives.
Active learning provides methods for incrementally selecting new points to rapidly reduce loss (Hanneke, 2007). These approaches only consider the problem of optimal data collection and do not seek to predict model performance under all data collection strategies (including suboptimal ones), which critical when making cost-performance tradeoffs across data sources. The model performance predictions produced in our work complements existing work on active learning by providing accurate forecasts of model performance under different data collection strategies.
Finally, data valuation methods such the Shapley value attempt to assign estimate the impact of data source on model performance (Ghorbani & Zou| 2019 Jia et al., 2019 Ghorbani et al.] 2020 Yoon et al., 2019). These approaches are natural when pricing data sources part of market mechanism Ohrimenko et al., 2019, Agarwal et al., 2019) due to the axiomatic properties of the Shapley value. Our approach differs that we seek simply to estimate the performance of model rather than to assign single price to examples from a data source. This difference means that axioms such as additivity that critical for the Shapley value are not relevant for our goal. We show that for the purpose of predicting errors, a rational function (rather than a linear cost) follows naturally from optimal experimental design. Our experiments also show that our rational function approximation provides better model performance predictions than a linear, additive model.
2 PROBLEM STATEMENT AND EMPIRICAL OBSERVATIONS
Our goal is to predict the performance of model as a function of the number of training samples as well as the dataset composition q. where qk represents the fraction of the training data drawn from data source k. We will now define this goal more formally in terms of the training data distribution, model fitting, and test loss.
The training data consists of an n-sample training set Pr.q that is created by sampling from the mixture p ; ΣkE[K] 4&Pk where pk are data generating distributions for each of the data sources and 9A are mixture weights with Qh 2 0 and Σ Os = 1. Using this dataset, we learn a prediction model Ô that incurs loss E(O; xx y) for a training example (æ,, y)- The fitted model is the empirical loss minimizer, which we define as
0(png) == arg min Pp.. [e(0; x. y)] 8E0
The performance of this classifier is evaluated training distribution by a covariate shift (i.e. p④y | x) = (y x)). We are interested in model performance as a function of the data size and composition (and not fixed empirical distribution
test distribution which may differ from the
2

---

Under review as a conference paper at ICLR 2021
Pn.g) and thus goal is to predict the model's expected excess loss over draws in both the training and test distributions.
L(n, g) :- EE ((0(pm inf E [e(0; x.
Estimating requires that we hypothesize relationship between (n, q) and the expected model loss. Following earlier observations by Hestness al. (2017), we expect a log-lineat relationship between [nn. q) and log(n) for any fixed which implics a possible approximation as
log(L(n, q)) x log(V (n, g)) = a(g) log(n) + C(q).
(2)
We now examine this hypothesis in simple toy example.
Linear toy data: We will start with the simplest nontrivial example of linear least-squares regression to study L(n, q) In this example, there two data sources over RR The first data source has substantial variability on the first coordinate Ty but not z1 and vice versa for the second data source. The overall generative process
x
[0.5,
1] +
Z Bern(g)
E~ (0,
x
= 0
N
0,
0.001
0
))
x =|z=l~N
0,
00.001
9))
Let [n,, q) be the excess squared loss of linear least squares model trained with n samples from mixture I and evaluated on a test distribution with q = 0.5. What will L(n, q) look like? Figure Ta shows a clear linear relationship between log dataset size (log(n) and log(L(n, q)) The intercept of the linear relationship seems to vary with the data mixture but the slope seems constant.
Expected error intercept
error
0.00
10-2
0.78
2
1.00
102
10a
0.0
0.2
0.4
0.6
0.8
1.0
Training data
Data source proportian
(a) Log excess loss (y axis) is linear with log- dataset size (x axis). Changing the data distribu- tion by varying (line color) changes the intercept but not the slope.
(b) Intercept (G(g)) of the loss-dataset log linear relationship. The loss is lowest when the dataset is a mix of both data sources (q & 0.5) and rapidly increases when exclusively using one data source.
Figure The log-linear effects data composition and size on linear toy dataset.
Examining Figure la more closely, we find that the extremes of using either data source exclusively (blue/ purple lines) performs worse than mix suggesting that log(I.(n, 4)) is unlikely to be linear in Intuitively, we can think of each data distribution as having a different strength (i.c. more variance in either zo x1) and combining the two results in a better data distribution than either alone. We can see this more clearly when we estimate the intercept for each of these lines (Figure Ib). The estimated intercepts show U-shaped curve that rapidly increases q O 1 and is generally flat from 0.2 to 0.8.
3 METHOD AND THEORY
We have observed that in the case of a simple linear regression, the log-error not only follows the relationship outlined in equation but also that the slope is constant we vary the data com- position (and we will further validate this claim on more complex tasks and models in subsequent sections). This observation shows we may be able to further simplify the log- linear approximation as
log( Log(L(n, q)) z (n, (n, -a log(n) log(C(4)).
Now note that this functional form decouples the data size n and mixture proportions C(q) into two terms. This is the key observation of our work: log (VoVVVn (n, q)) has very simple dependence on n,
3

---

Under review as a conference paper at ICLR 2021
and the more complex term C(g) has dependence on Therefore problem, where we learn a and a parametric function Cx(q) based on the model's 's error over a range of q and small n, and extrapolate this for large using the log- log-linear dependence of log V
cast this learning
Concretely, given dataset with {n1... ...) we can generate a subsampled dataset with ftę Uni (0k))))))))))))))) samples from each source. This results in training set with data size ń Σx fi and composition Ộk 2
We fit model this subsampled data and compute its loss L q). Given the triple (ñ, q, (n. 4))
we can now simply fit the hypothesized functional form,
min Eg.a A,a
(Log(L(A. 9)) log(n) log(Cx(())))
The experimental data does not specify the functional form of (x except that should handle convex functions like those seen in Figure lb that a natural choice is the rational function
We will now study V V(n, q) theoretically and argue
Cx(9)
Σ
Σ Puat
In the subsequent sections, will study three settings: ordinary linear regression, M-estimation, and nonparametric regression and show that our hypothesized log-linear approximation arises natu- rally all three cases.
3.1 LINEAR REGRESSION
We begin by characterizing L(n, q) in the linear regression case, where we can derive closed form expressions for the expected loss as a function sample linear regression, defined as y x 3 + with i.i.d. N(0, 1). Our training data follows p ΣnE[K] 9kPk where each data source has full-rank second moments Σ3 Epwpx [
training data. Our setting is d-dimensional, n-
Define the ordinary least squares estimator 3 (XT x)- X y in terms of the features X E Rax and Y R.". The excess test loss of this estimator over any x² ~ p" and y" = x= B + is defined
as
L(n, q) = [| 3)
The theory of V-optimal experimental design Pukelsheim 2006) allows to characterize this excess loss.
Proposition 3.1. The excess expected loss for ordinary least squares trained mixture q with data size n and subgaussian x follows
Vlog(1/8) Vn
log(L(n, q)) = -loggn)) + log
Σ
+0
RE(A]
C(a)
with probability at least where Σ"
Ezupp
[xx and Σg = Egcop [x0
We will defer all proofs to the appendix due to space constraints. Clearly C(g) is not linear even in this simple case, and the terms for qe appear within an inverse. Naively, we might hypothesize that behaves much more closely to a linear rational function (i.e. (E, -1g and this intuition will turn out to be correct whenever E* and Σ are approximately diagonalizable.
Corollary 3.1. Let P be an orthogonal matrix which approximately simultaneously diagonalizes P-IE- = D", P-1E, Dr. R& for diagonal some matrices D. Then for full-rank E. and sufficiently small Rg-
Σ Ee Da.se to (12 2h Rielee)
D
Tr Σ¹
Σ It KE]K]
dEld
4

---

Under review as a conference paper at ICLR 2021
The first order term exactly matches the hypothesized C(q) rational function with d terms and validates this choice for linear regression. To interpret this corollary, the approximate diagonaliz- abilitty condition states that the cigenvectors for Σ and Σg coincide, and that D and Da. are these eigenvalues. The ratio Σe 9. D.. measures the ratio of variance in the test distribution to that of the training distribution for the i-th eigenvector.
The key observation is that the variance (i.e. the information each data source contributes to partic- ular coordinate i) is linear, but the dependence of model enror to training variance is inverse and that there are different coordinates making the overall dependence of errors on data composition non- linear. There are clear qualitative differences between linear and rational function approximation to C(q), with the rational function being strongly convex with diminishing returns in
3.2 GENERAL M ESTIMATORS
We might rightfully ask whether this kind of approximation continues to hold for nonlinear models and losses like neural networks. The same analysis as above can be extended to the asymptotic be- havior of substantially more general class of models known as M Mcestimators, which are empirical loss minimizers of differentiable loss.
For the regression case, we relied on a closed-fome characterization of 3. For M-estimators we will asymptotic normality under the sampling distribution,
Theorem 3.1 (van der Vaart (1998)). Consider a twice differentiable loss & whose gradients are bounded and Donsker. Let O,, be an estimator which fulfills the approximate first-order optimality condition with minimizer Osc.
Epn VV(y,
o(n 11/3)
and
VV(y,
If 0n 3s 2x and both 1 = Ep Eyl 000) and Dg == oz exist,
Vn(0, - 0x) - N
NO, (0. Eo. To.)
Now that we have the asymptotic distribution of the -estimattor, we can quantify the (asymptotic) form of C(q) with respect to test distribution p" simply by taking the Taylor expansion of the loss at Oxe-
Corollary 3.2. Under the conditions of Theorem 3.1 let xy,,;;0)
loggooy |
x)) and there
exists some 8" = 0xx such that pa- (y x) = p(y x) then
Io -') + o(n +o(n-1))
log(L(n, q)) = - log(n) ) + log
Tr E'
for Σg = (H;(,,,;;0)) and E- := Ep- [H((y,
This result relies on two additional assumptions: the loss is a log loss, and the model is well- specified. The first assumption is weak, many models today use log softmax type losses. The well-specified assumption stronger but may be reasonable for nearly nonparametric functions such neural networks. For less simple but more general result, Corollary A. in the appendix.
This has the same functional form as before: C(q) is the trace of a test distribution dependent matrix E* and the inverse of data source matrices Σg. The difference now is that instead of covariances, we are looking at the Hessian of the parameters with respect to the unknown optimal model 0*.. Applying the simultaneous diagonalization argument from earlier once again results in rational function that is captured by C)))).
3.3 NONPARAMETRI MODELS
Finally, we show that the same relationship holds for nonparametric models such as kernel smooth- ing or binning. Our goal will be estimate some ground truth map y f(z) + for e i..d (O, and f a differentiable L-Lipschitz function. The quality of an estimate will be measured by some wwie-ddifferentablee loss ((y.x) x) with bounded first two derivatives.
5

---

Under review as a conference paper at ICLR 2021
Given n samples (21, (1) ... (Zn, Un) [0, 1] x R drawn i..d from some density p Ex 9kPk: one natural estimator for this problem is the nonparametrie binning estimator f which we define in terms of axis-aligned hypereubes (s(x, S) {a' E E [=//0] [z/0]}. Let Xn, == {211...0 then we can define our estimator,
fa(x) = [T, xn)
Σ
yi
2,E (m,
Assuming we choose and n sufficiently large that each bin concentrates to its expected value, we have the following error estimate
Proposition 3.2. Let Bs(x, = Ex'~pp [Bs(a, {x')i] be the probability of drawing some of Pk in the same bin as I, and assume Bs Pk) is bounded away from zero. Then
[(n,0 q) = )) [x)
e" (()),
vlog(y=1) + (l)g(o)
= E
+0
+OLLVV++ (2
Σx 9h (x.
V2nn
holds with probability at least
7 where the expectation is taken with respect to draws of y.
Once again, we see rational function in g, with no further approximation needed. Each bin is term in the rational function approximation with weight (=))).
EXPERIMENTS
We have seen that rational function reasonable approximation C(g) across 3 different set- tings, We will now show that this is the case in practice, and additionally that C()) can be accurately estimated using few models trained on small datasets. The resulting estimates of model perfor- mance are accurate for models with an order of magnitude more data.
Baselines and implementation Our evaluations focus on our ability to predict the loss incurred by model L(n, q). To do
will compare the rational function approximation procedure against several natural baselines for predicting the loss of a model. Each of the baselines correspond to a different assumption about the functional form of log(V(n) (n,q)) that we use to approximate log( (og(L(n,))) 9)).
Datasize: Assume functional form of log(Von,,nnn) 9)) a (n) + ignoring the data composition and dependence on q.
Linear: Assume functional form of log(V(n, q)) a log(n) 3T + c. This is the natural approach if we treat log(V(nn, q)) as linear in 9 and log-linear in n.
Ablation and Shapley: further constrain the linear baseline by setting 3 to either the log- Shapley value obtained as the marginal contribution of a data source (for the Shapley baselines) or the log- ratio of losses obtained after removing data source (ablation). We this approach to dominate the usual assumption of treating V(n, as being linear in the Shapley value.
found
As the baselines all linear in log(n) and g solve the optimal parameters for these models in closed form with least squares regression. We will refer to our approach as Rational, and we fit this using the Adagrad (Duchi al. 2010) optimizer with 20000 steps and learning rate set via cross validation over the interval [0.005, 0.5]. The number of terms in the rational function sum is set to be one more than the number of data sources in all experiments.
4.1 FOCUSED EVALUATION: AMAZON SENTIMENT
We now consider the Amazon sentiment prediction dataset in (Mansour al. 2009) where the goal to predict Amazon ratings for books (from to 5 stars) using bag-of-words features from the reviews. The training data comes from 3 domains that differ from the test data: kitchen, DVD, and electronics reviews. The model is a standard ridge regularized regression model. Our experimental setup for estimating model loss is the following: uniformly randomly sample the dataset size for

---

Under review as a conference paper at ICLR 2021
each source (resulting in between 0 and 1200 examples for each source), and train a model on this dataset. We measure the test error via average squared loss on the books domain.
We fit (nn. g) with 4 terms for C(q) by minimizing the squared loss with respect to log-error on models containing 0- D-1200 examples total. We then use (nn, q) to predict log-error on the models trained on 1200-3600 examples from each domain. The results of this extrapolation task are shown in Table 2 Our V(n,9) 9) estimate is nearly perfect (r² = 0.96) and extrapolate from the low data to high data regime without issue. This correlation is substantially higher than either using data set size (r² -0.65) a linear model (r² = 0.76) and even better the training error of the best additive
model (r² 0.87).
Metric
Datasize
Ablation
Shapley
Linear
Rational
Train
0.20
0.77
0.80
0.87
0.95
Extrapolation
-0.65
0.43
0.51
0.76
0.96
Bootstrap interval
(-1.86, -0.08)
(0.018, 0.60)
(0.15, 0.66)
(0.58, 0.83)
(0.94, 0.97)
Table 1. Accuracy of L(n, q) estimates the Amazon review sentiment prediction task. Bold in- dicates the best performing model under extrapolation, identified by bootstrapped paired difference test.
The data size predictor has a negative r² on the extrapolation setting which may seem surprising. However, this happen whenever a predictor fails to perform better than predicting the mean of the test set. It is nontrivial to predict the mean of the test set in an extrapolation setting, and in this case, data size estimates are generally uninformative as data from the kitchen domain not useful for predicting book review scores.
4.2 BROAD EVALUATION: SEMANTIC PARSING, TRANSLATION, AND QUESTION ANSWERING
We now perform broader evaluation of the 3 methods (linear, rational, and datasize) on 3 tasks that violate our assumptions about model performance prediction. We excluded the two ablation based methods they are special cases of the linear model, and generally performed worse.
Fask-oriented dialogue
Machine Translation
Multitask QA
method
Train
Extrapolation
Train
Extrapolation
Train
Extrapolation
Datasize
0.54
0.69 (0.52, 0.81)
0.05
-0.80 (-3.37, -0.03)
0.78
-1.39 (-6.4, -0.88)
Linear
0.99
0.91 (0.78, 0.96)
0.27
-0.69 (-3.10, -0.03)
0.98
-0.92 (-6.63, -0.17)
Rational 0.99
0.93 (0.83, 0.97)
0.98
0.83 (0.59, 0.92)
0.97
0.86 (0.66, 0.89)
Table 2. Accuracy of error estimates 3 real-world tasks that pose challenges for performance predic- tion due to their use of deep neural networks, non-separable losses such as BLEU, and weak supervi- sion. In all cases, the rational function approximation provides good estimates of model performance.
Task-oriented dialogue We perform this analysis on a real world task-oriented dialogue system based upon the SMCalFlow dataset and model (Andreas al. (020).. The task differs from the Amazon setting in two ways: the model is a nonlinear neural model for which there is no closed form optimal experimental design and the task is semantic parsing which is a more complex structured prediction problem. There are 105727 total dialogues across 4 data sources consisting of a wizard- of-oz style crowdsourced dialogues, paraphrases of existing dialogues, on-policy dialogues between the system and crowdworkers, and hand-crafted dialogues by expert data scientists. We sample the number of dialogues for each source with a uniform distribution to determine q and then further subsample each data source by 10.1, 0.3, 0.7, 1.0) to vary n. Test errors are measured by whether the execution of the model matches human references.
We fit ((n.g) with 5 terms for C(q) on 10 models containing less than 16,000 examples, and testing on 19 models containing between 16,000 and 100,000 examples. The results in Table 2 show our approach is accurate (r = 0.93) and outperforms baselines including data size (r²2 0.69) and the additive model (x-2 = 0.91). While the bootstrap intervals for the 72 for the rational function approximation is sufficiently wide that it contains the mean r² estimate of the linear model, more powerful paired difference test between the linear and rational approximations shows that this gap statistically significant at a 5% level for the bootstrap distribution.
7

---

Under review as a conference paper at ICLR 2021
Log excess loss
3.75
3.75
3.50
3,50
3.5
3.25
3.25
3.0
3.0
3.2
3.4
3.6
3.8
3.0
3.2
3.4
3.6
3.8
3.0
3.2
3.4 3.6
3.8
log error
log errar
log error
(a) Data size
(b) Lincar
(c) Rational function
Figure 2. Performance prediction multi-domain machine translation task with BLEU the performance measure. There little correlation between dataset size and loss (left panel) while the rational function approximation provides reasonable predictions (right).
Machine translation Thus far, we have evaluated separable losses such as squared error, or model accuracy. We now show that our approach to predicting model performance continues to work for non-separable losses such BLEU for machine translation. Our task is the standard multi-domain machine translation dataset from Koehn & Knowles 2017). We use the preprocessed data, model, and hyperparameters from Hu et al. (2019) for a baseline sequence to sequence machine translation model. The model is trained on 4 data sources: Acquis (legal text), EMEA (parliamentary proceedings), IT (IT assistance), and Koran (translations of the Quran). Evaluation is performed the Acquis test set using sacrebleu to compute BLUE (Post 2018).
To estimate the performance of models under varying data composition, we subsample up to 300,000 sentences from each data source, fit the estimators on 19 datasets of size less than 600,000 total sentences, and evaluate on 11 datasets of size 600,000 1, 1,200,000. Since BLEU is similarity measure and is penalized by reference ambiguity, we consider 50-BLEU to be the excess error. The rational function approximation baselines. The difference in prediction accuracies is apparent when plotting predicted and observed log-loss (Figure 2). The linear model even has low truining set r², suggesting that the relationship between data composition and performance is fundamentally nonlinear.
the only procedure to achieve a positive ,2 (0.83) among the
Multitask question answering Finally, we consider a multitask learning problem where some of the data sources are auxiliary tasks that may not directly be useful for the test time task. This breaks the covariate shift assumption that has been implicit throughout this paper. The target task is the BoolQ question answering dataset, and we train this model using a combination of 4 data sources: the MNLI entailment task (Williams et al. (2018), 50,000 examples subsampled), STS sentence similarity judgment task (Cer al. (2017). 5749 examples), MRPC paraphrasing task (Dolan et al.) (2044) 3668 examples), and the BoolQ training set (Clark et al. (2019) 9427 examples). We use the GLUE data with the Jiant package to train a baseline BERT based model for this task (Wang et al.) 2019b).
The challenge with this task is that only the BoolQ training set provides direct supervision for the test-time task, and the other data sources provide weak supervision that may may not be helpful in the downstream problem. The model performance estimates are fitted on 6 datasets with up to 25,000 total examples and evaluated on 17 datasets with more than 25,000 examples. As expected, the data size based performance estimates are catastrophically bad, resulting in negative correlations. The linear estimates do seem to extrapolate well to the test set. The rational function approximation is the only one of the three to provide positive y on this task.
5 DISCUSSION
In this work, we've proposed a new approach to predicting the performance of a prediction model as a function of training data composition that consists of measuring model accuracies for small and a range of q and fitting a parametric model V (n.,g q) := -a log(n) Σm ET <K As We show that this parametric model is natural approximation to model performance for range of models, and accurately predicts the empirical performance of models in an extrapolation setting. Our work is the first step in going beyond closed-form estimates of model performance or additivity assumptions. an open question whether the same approach can scale to large numbers of data sources, and we hope to explore this in future work.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
A. Agarwal, M. Dahleh, and T. Sarkar. A Marketplace for Data: An Algorithmic Solution. arXiv preprint arXiv: /805. wtt.1/805......... 2019.
J. Andreas, J. Bufe, D. Burkett, C. Chen, J. Clausman, J. Crawford, K. Crim, J. Del.och, L. Dorner, Eisner, H. Fang. A. Guo, D. Hall, K. Hayes, K. Hill, D. Ho, W. Iwaszuk, S. Jha, D. Klein, J. Kr- ishnamurthy, T. Lanman, P. Liang, C. Lin, I. Lintsbakh, A. McGovern, A. Nisnevich, A. Pauls, D. Petters, B. Read, D. Roth, S. Roy, J. Rusak, B. Short, D. Slomin, B. Snyder, S. Striplin, Y. Su, Z. Tellman, S. Thomson, A. Vorobev, I. Witoszko, J. Wolfe, A. Wray, Y. Zhang, and A. Zotov. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics (TACL), 8:556-571, 2020.
Ben-David, Lu, T. Luu, and D. Pal. Impossibility theorems for domain adaptation. In Artificial Intelligence and Statisties (AISTATS), 2010.
D. Cer, M. Diab, E. Agirre, I. OazezoG and L. Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. International Workshop on Semantic Evaluation (SemEval), 2017.
C. Clark, K. Lee, M. Chang, Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In North American Association for Computa- tional Linguistics (NAACL) 2019.
K. Crammer, M. Kearns, and J. Wortman. Learning from multiple sources. In Advances in Neural Information Processing Systems, 2007.
B. Dolan, C. Quirk, and C. Brockett. Unsupervised construction of large paraphrase corpora: Ex- ploiting massively parallel news sources. In International Conference on Computational Linguis- tics (COLING), 2004.
Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. In Conference on Learning Theory (COLT), 2010.
A. Ghorbani and J. Zou. Data shapley: Equitable valuation of data for machine learning. arXiv preprint
2019.
A. Ghorbani, M. Kim, and J. Zou. A Distributional Framework for Data Valuation. arXiv preprint 2xXi::2002.12334, 2020.
S. Hanneke. A bound on the label complexity of agnostic active learning. In International Confer- ence on Machine Learning (ICML). pp. 353-360, 2007.
J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv preprint 171.v:/712.004400 2017.
J. Hu. M. Xia, G. Neubig, and J. Carbonell. Domain adaptation of neural machine translation by lexicon induction. In Association for Computational Linguistics (ACL), pp. 2989-3001, 2019.
R. Jia, D. Dao, B. Wang. F. A. Hubis, N. Hynes, N. M. Gurel, B. Li, C. Zhang, D. Song, and C. Spanos. Towards efficient data valuation based on the shapley value. arXiv preprint 1027: 2019,
R. C. John and N. R. Draper. D-optimality for regression designs: A review. Technometries, 17(1): 15-23, 1975.
Kaplan, S. McCandlish, T. Henighan, T. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint 7rXiv:2001.0836. 2020.
Koehn and R. Knowles. Six challenges for neural machine translation. In The First Workshop on Neural Machine Translation, pp. 28 39. Association for Computational Linguistics, 2017.

---

Under review as a conference paper at ICLR 2021
Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation with multiple sources. In Ad- vances Neural Information Processing Systems (NeurIPS), pp. 1041-1048, 2009.
O. Ohrimenko, S. Tople, and S. Tschiatschek. Collaborative Machine Learning Markets with Data- Replication-Robust Payments. arXiv preprint arXiv:
2019.
M. Post. A call for clarity in reporting BLEU scores. In The Third Conference on Machine Trans- lation: Research Papers, pp. 186-191, Belgium, Brussels, 2018. Association for Computational Linguisties.
Pukelsheim. Optimal Design of Experiments. Society for Industrial and Applied Mathematics, USA, 2006. ISBN 0898716047.
Rosenfeld, A. Rosenfeld, and Y. Belinkov. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations (ICLR), 2020.
A. W. van der Vaart. Asymptotic statistics. Cambridge University Press, 1998.
R. Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv, 2010.
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task bench- mark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), 2019a.
A. Wang. I. Tenney, Y. Pruksachatkun,
Yeres, J. Phang, H. Liu, Htut, Yu, J. Hula, P. Xia, R. Pappagari, S. Jin, R. McCoy, R. Patel, Y. Huang, E. Grave, N. Kim, T. Fevry, B. Chen, N. Nangia, A. Mohananey, K. Kann, S. Bordia, Patry, Benton, E. Pavlick, and S. Bow- man. jiant 1.3: A software toolkit for research on general-purpose text understanding models. http:://jaat info/ 2019b.
A. Williams, N. Nangia, and S. Bowman. broad-coverage challenge corpus for sentence under- standing through inference. In Association for Computational Linguistics (ACL), pp. 1112-1122, 2018.
J. Yoon, S. Arik, and T. Pfister. Data valuation using reinforcement learning, arXiv preprint arXiv:1909.11671 2019.
10

---

X Xn - -Dilpp S. C V.Voooo Vlog(1/8) Under review as a conference paper at ICLR 2021
A APPENDIX
Proposition 3.1. The excess expected loss for ordinary least squares trained on a mixture q with data size and subgaussian follows
Vlog(1/0) vni
log(L(n, loggn)) log Tr
2'
Σ
+0
KE[A]
C(a)
with probability at least where En
Ragnop [xrx]
and Egg =
Proof We will begin by deriving the excess loss for fixed set of test examples X E mxd and training examples E Raxd We are interested in the excess loss, which can be written simple form due to strict exogenity of the least squares regression,
X (B 9)12/m Tr(X"T X*
This is classic instance of V-optimal design. We reproduce this result for clarity. Let = (B then,
|X 3m1l3mm x Tr(X* x (XT x) )/m.
The challenge now is to bound the expected loss,
L(n q) E Tr (xTx) /m.
The expectation with respect to X is straightforward, but the expectation with respect to is make use of ma- trix concentration show this term concentrates appropriate rate. Let B[X X] then by Vershynin 20000 Corollary 5.50) for sufficiently large n1,
challenging as E[X x]-1 * [ J. Since is subgaussian,
Vn
the subgaussianity parameter of X. Now can expand the empirical covariance in terms the expectation and residual = X XX/n- E using the identity (2) + A) = E-1 Eie Σー
with probability 1 5 and constant C depending
(AS
1)
L(n.q) = Tr (Σ' + 'E
Tr Σ
whenever the series convergent (|
(1Σ-1) 1lop | lopp).
For sufficiently large n, this series will converge (as a (E) is a constant and || = ((2-1/2) and the first term the series is the dominant one. Using the trace inequality TBAA- s d2 [A]|A-1 BBl
v [og(1//) 3/2
L(n, g) = -T-EE'-'''' +0
Finally, we make use of the fact that X is drawn from a mixture with component moments Σn := Exepa [xx]] "] and take the Taylor expansion of log(L(n, q)) L(n, -n obtain,
77) +0
(Vlog(1/0)
log(L(n,g)) g)) = - log(n) + log Tr Σ"
Σ Ex Ex k
vn
with probability 1 1-8.
11

---

Under review as a conference paper at ICLR 2021
Corollary 3.1. Let P be an orthogonal matrix which approximately simultaneousty diagonalizes P- D", P- E = D3 + Rx for diagonal some matrices D. Then for full-rank E and
sufficiently small Rg-
1r Σx
Σ kE[K]
Isk
E :EId
D
+o
( k
9h Rulle)
E k Th Dk,s0
Proof
Since En full rank, we can apply the local expansion (A + B)-1 A- +
Σ A-1 (BA-1) along with the trace bound [T((A- 11
Tr Σ
Exp xx) = Tr
D
(Dx + Ra))-1
k
D
(1
+
Σn
sE|d]
Where the last line uses the local expansion and trace bound for sufficiently small B.
Theorem 3.1 (vn der Vaart (1998)) Consider a twice differentiable loss & whose gradients are bounded and Donsker. Let 6, be an estimator which fulfills the approximate first-order optimality condition with minimizer 9xc.
Epm Ve(y, 0n)] 1122 and
EpIVe 2; 2; 0m =0
If 8n exist,
3, Ox and both Io_ Epl Ep(He(y, x; o00)] and Eg
vn(0,- - 0z)) + N (0, 1o. E. Tom)
Proof
First, we take the first order approximation to the population minimizer
EpiVe(y, 0w))] E,lVe r; 0x0)] + EglHely, x; bx)) (B,, 0x0) + of 1( foo
Assuming the existence of T and using the approximate first-order optimality conditions for both fxc and 6, we can solve for the parameter difference as
VT((n, -ox))
= VnilT"
2; E, Volza ( x; 0.)) R.p., Vi 0,))) + o(1 + vvl10, 00 Il)
VV 0z +(vnl - 1x
Now since V is Donsker over each coordinate and 0n, - 0x in probability, we can obtain uniform concentration the gradients (van der Vaart 1998
vn (E,(VV((, x; 0.)) Ep,, (V x;; 0..) (V 0 Ep. [Vi(y, 0x))) o(1 Substituting this into the earlier equality allows us to replace 0,, with 0x0
Vn((,,- -xx) = vnl.a (Ep.. [V(y, x; 0oo)] - Sp.Vely x; 0oo)) + o(1 vnl -xx
Since V bounded and thus has finite third moments, Ep,, [Vl a; 0xx) obeys the central limit
theorem with distribution N ((E, (VE x;; 00)) ). Finally, the lower order terms on the right
asymptotically vanish since on + O% and we obtain the stated result.
Corollary A.1. Under the conditions of Theorem 3.1 and either Eyy. (Ve(y, x; Axx)] 0 or oorrF [[ 0xc + o(n L(n,g) B[[(y, rz 0m)) El((y, 0xx) --11
17r (Tep [H((V- x1; 0z0 To E. 1o_)) o(n oln
12

---

Under review as a conference paper at ICLR 2021
Proof
Taking second order Taylor expansion,
BB 1epp1e(y, B,)ll = Ep- (e x; x0)]
+ B
(00xx - -0a) + B.p [00> - O.ly, x; (00) - 0,.) on
The first-order term is at most o(n-1) by the additional assumption. Either our asymptotic param- eter estimate is also a stationary point for the test distribution and Ep- [Ve(y, æ; 0x)) 0, or our estimator is unbiased and Ef 0..] o(n-1). The second order term can be simplified via the asymptotic normality of i, as
E|Ep- [00x 0n) x; 200 (0x 0n) = n nT'rr (Ep- (He(y, 000) Da To + o(n
Corollary 3.2. Under the conditions of Theorem 3.1
let {(,,,;0) - -logpol x)) and there
exists some 0 = 0. such that po- (y x)) = p(y x) then
Laadin auiia -) + +o(n- 2o(=-))
Σ Tr 2"
log(L(n, 4)) = log((n) + log
for Σx = Ep,, [He(y, 0") and E" == Ep. [H](y, 6*)
The statement follows almost definitionally. By the conditions of the corollary statement, the model is well-specified maximum likelihood estimator, and the fisher information and hessian coincide, De_ la Simplifying the expression in Corollary A.I and expanding Eo_, into its components gives the desired result.
Proof
Proposition 3.2. Let Bo(I,Pt) = Ex'-pp [B(( {x())]] be the probability of drawing some a! pk in the same bin as H, and assume (x, pk) bounded away from zero. Then
[(n, g) = E(((fo(x), -
[(f(x),x)]
(f()
Vlog(T-1) + dlog())
= E
+0
+ COLLOVA+L38900
Bo
22n
holds with probability at least where the expectation is taken with respect to draws of y.
Proof Note that by definition of y, whenever [B, (x)) 0, fa(x) has mean close to f(x) with the deviation controlled by the Lipschitz constant of
(x)=(f(())] f(x) O(LVV))
and variance (considering [B, X fixed).
Var Va 1+ Var [f(z) TE Bole, [Bs X..)
1+ O+0(LöVd) Ba(r, X..)
Taking the second order Taylor approximation to l at f(x) we get
= (
(M))) (B(((()) {s(x))) [,0 -U (BU -d(c)))))))
=3)" X.)/ +
13

---

Under review as a conference paper at ICLR 2021
The third line follows from the expectation bound, well applying the varsaner decompo- sition to the second order term. The last line follows from the variance identity above, and applying the same bias-variance decomposition on the (x) {=)))) term.
As with the linear regression case, cannot simply take expectations there is small but nonzero probability that (2, Xn) inequality, each of the 8-a bins will concentrate towards their expected values
] is zero. We will show this happens with low probability. By Hoeffding's
P
P (IBAaCE ([IBa(e,,xx))0'> Xn))n= -
- Ru Ba (o'))]
Vlog(2y 15--1
<
11-0%
2n
Applying the union bound, we have concentration at the same rate over 8-4 bins with probability Now note that we can write
1
- e" (f(x)), BsBs(a, (x'))( + [B -.)la--B(BBae (x
(f(t) X.)1"
We can take the Taylor approximation of the ratio at the expectation, which gives us
[f" x) x) Bo = 1 E" (f((()), (x'l)) + ((( -EB (x'xx)
Now with probability at least Y
VVog(r + d logg())
E (f(())), BsBs(z, {2))) n
+0
22n
Plugging this into carlier expression, and expanding the expectation in terms of each of the data sources completes the proof.
14


Reviews:
[{"review_id": "trDVHXbg-Ib", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work studies the problem of predicting model performance with more training data when the data are collected from different sources. The predictor is a function of the number training examples, and the ratio of examples from each source. The predictor needs to be built from a small number of training examples the observed model performance, and applied to larger numbers of training example without actually training the model. The predictor can be used to decide a good data collection policy which is expected to have best model performance. The proposed solution is a simple parametric form of the predictor, which is log-linear in the log of # training examples, and log-rational in the source distribution vector. The solution is motivated by recent literature about the same task for single source. The correctness of the solution is proved for several cases: linear regression, M-estimator and nonparametric binning. The performance of the predictor is then evaluated for a number of real-world tasks: linear regression for Amazon book rating, semantic parsing, machine translation and multitask question answering. The performance is measured by r2 score between the actual performance and predicted performance. The proposed predictor has a clear advantage over the baseline of using a linear predictor.\n\nPros:\n\n1. The work is built on top of recent studies with single source and the extension to multi source is timely. \n2. The discovered relation between the expected model performance and the number of training examples from multiple sources is neat and interesting. \n3. The proposed rational function of the distribution vector is justified for a few cases theoretically.\n4. The empirical evaluation uses diverse real tasks, covering different loss functions, including the ones where the assumptions in the theoretical analysis are not satisfied. And the proposed solution has a reasonably robust performance and is much better than the baseline. The behavior of the predictor across the tasks meets expectation.\n\nCons:\n1. \" Following earlier observations by Hestness et al. (2017), we expect a log-linear relationship\nbetween L(n, q) and log(n) for any fixed q,\" It will be good to specify the condition of the loss function required by the reference.\n2. The experiment applies prediction to 2-6$\\times$ of the training examples used for fitting the predictor at max. It will be useful to study the case when this gap is larger, to understand the power or limitation of the proposed predictor. \n3. The distribution vector is randomly sampled in both training and testing. It will be useful to test the case where the distribution vector is sampled differently, to further evaluate the extrapolation ability.\n4. Combining 2 and 3, it will be very useful to test the limit of the proposed approach and know the minimal required observations for the predictor to work well.\n5. Page 8: typo BLUE. \n\n===================\nUpdate after discussion: The difference between excess loss and absolute loss is an issue I overlooked. After discussing with other reviewers, I realized a new question: how to compute the excess loss L(n, q) without knowing the the best possible estimator (oracle) in the class? This is a key step in the proposed algorithm when fitting the predictor. The paper has not explained it. As of now I do not think that is possible, except if strong assumptions are made, such as the oracle has 0 loss. In the experiments, if the authors use the estimator trained with full data as the oracle to compute the excess loss, that would invalidate the practical usability of the approach because the goal was to predict the performance without having full data to begin with.", "extra_scores": {}, "extra_text": {}}, {"review_id": "hyI3VGpbMdJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a model to predict the model performance given the sample size n and the composition of the source of the data, q. They argued that the excess test loss should be separable in n and q.  The paper is interesting and well presented. I have no difficulty following the authors arguments. The proposal could be useful for some practitioners. \n\nHowever, I am not convinced that log(L(n, q)) should be separable in n and q in general. The authors did show in some special cases in Figure 1 and Sections 3.1~3.3 that n and q are separable, but a more general theory is lacking. The baselines in the experiments are all relatively weak. In fact, comparing the results between Datasize and Rational, it seems plausible to me that Rational achieves good performance simply due to the added flexibility in the q term, rather than modeling the n and q relationship correctly. A more suitable baseline would be to use a DNN model on [n, q], without separating n and q, and if Rational achieves on par performance with DNN on Train, that suggests to me that the functional form of Rational is probably correct.\n\nOn page 2, the authors wrote \"The performance of this classifier is evaluated on a test distribution which may differ from the\ntraining distribution by a covariate shift (i.e., p(y|x) = p_test(y|x)).\" Is there a typo here? Do the authors want to write \"p(y|x) = p_test(y|x) but P(x) \\neq p_test(x)\"?\n\nOn page 3, what is the expectation taken over in L(n, q)?\n\nOn page 3, I can only see 3 colors in Figure 1 (a), yet there are 5 lines in the legend.\n\nOn page 4, in the formulation of C(q), lambda appears in the denominator. How to initialize lambda such that sum(lambda q) is not too small, leading to exploding C(q)? Is there any constraint on alpha to guarantee the sublinear relationship established in Hestness et al. (2017)? Is there any constraint on lambda's? How many lambda's should we use? It seems that based on the propositions, there should optimally be dk lambdas. This seems like a large number.\n\nOn page 7, in Table 1, do we extrapolate only with respect to n? If so, it would be great to also present a study with extrapolation on q.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0ymkiR73NvM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper casts generalization under covariate shift as a prediction problem. Concretely, given a model trained on $n$ samples from a mixture of $K$ distributions, can we predict the excess risk of the model on data drawn from a different mixture? \nMotivated by recent empirical work, the paper posits a simple functional form for the model: log-linear dependence on dataset size and an additive offset given by a \\emph{rational} function of the mixture coefficients. The author's demonstrate theoretically that this form arises naturally for linear regression, M-estimation, and a non-parametric binning estimator. Empirically, they show fitting this model to data gives better predictive performance than methods based on dataset size or using linear functional forms.\n\nPros:\n- This paper addresses a question of great interest to much of the ML community: generalization under distribution shift.\n- The approach taken here, i.e. using theory to posit \"scaling laws\" or models whose parameters are fit to then fit to empirical data is an interesting alternative to reasoning about performance under distribution shift from first principles alone.\n- The quantitative predictions from the models presented here may be of interest to people using active learning to gather new data to improve model performance. \n- The paper and proofs read well and don't appear to have correctness issues. (I did not carefully verify Proposition 3.2 due to time). \n- In the experiments presented, the rational function model gives fairly accurate predictions, and certainly better predictions than methods based on dataset size alone.\n\nCons:\n- My main complaints with this paper is the lack of limited baselines and the limited scope of the experimental results.\n- Baselines: The dataset size and linear baselines are good starting points, but the the experiments not include any non-linear baselines. This makes it difficult to determine whether the rational function model is really fundamental or necessary. Would, e.g., a generic function approximator perform just as well?\n- Experimental scope: Scaling laws like the ones presented in this paper are useful and interesting in so far as they capture some \"universal\" phenomenon. \n-- In each of the settings considered, the experiments focus on one model, e.g. pretrained BERT for question answering. How general is this method beyond the models considered? Does the rational model give predictive power on each task across a diverse set of models, or is there something particular about the ones presented?\n-- All of the experiments take place with text data. Does a similar result hold, e.g., for CNNs trained with natural images?\n\nOverall, I very much enjoyed reading this paper, and it offers several avenues for further inquiry. I vote to accept.\n\n==============\n\nUpdate after rebuttal:\nThanks to the authors for their response. I enjoyed this paper, and I'm keeping my score unchanged. In terms of multiple datasets/diverse models, I appreciate the various models/dataset/baselines currently included as a proof of concept. However, I'd be very interested in a more systematic study with significantly more models and datasets to understand better precisely when excess error as function of data composition and size can be reliably extrapolated and the extent to which the trends observed are \"universal.\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "-IoBo1Lpf0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary of the contribution\nThe paper addresses the issue of combining datasets from different sources, and taking into account the fact that they do not share the same precision. \nIt aims at predicting the performance of an algorithm trained on those data, through a simple formula taking into account the size of the data, and its composition through the mixture weights of the distribution from which it comes.\nAs the formula also depends on the estimator trained and the quality of data, it is trained to be estimated fully.\n\n### Strengths\nThe paper and the topic are both interesting. The maths look correct, although I have not checked it all.\n\n### Weaknesses and concerns\nThe paper lacks clarity in general.\n1. Context\nThe context of mixed-source data is not very clear: \n\t- from Section 2, it appears to be applicable e.g. for datasets coming from different sensors, but predicting the same output, while the experiment section makes use of very different datasets, not even predicting the same target; \nI encouraged the authors to provide specific examples in Introduction or Section 2 for clarity. The experiments seem to be related to multitask learning, as mentioned shortly for one application but looks to be true also for others;\nit could also possibly be related to domain adaptation. In both cases, the experiments should include comparison to state-of-the-art approaches in those learning problems.\n\t- It is also not clear how the different sources are mixed together: does the proportion $q$ apply to the number of observations in each source? \n\t- Moreover, there is mention in the Introduction of the goal of optimal data collection policy. \nHow can the proposed work be used in practice for that goal, since we do not know in advance what is the quality of each dataset, or how well does an estimator perform?\n2. Main goal of the work \nThe title and most of the work is a bit misleading, as it mentions predicting the performances of an estimator, while in fact it predicts its excess loss, that is, how close an estimator is from the best possible estimator (oracle) in the model; \nwhile this is a good approach to improve the fitting of an estimator (and its parameters) inside a family, it does not give any clue as its actual performance. \nIt is also not clear what the learning problem at the beginning of Section 3 does, and how it is used in practice.\n3. Related work\nAppart from the learning problems mentioned before, here are possible issues with related work\n\t- Active learning does not qualify as related work, as it is usually concerned with the labelization of unlabeled samples in a single source dataset when labelling is expensive. \nIf there exist work on that domain that is specific to the problem of combining data from different sources, please provide a more specific reference.\n\t- There exist however other works for the issue of performance evaluation through a learning problem, e.g. \"Per Instance Algorithm Configuration of CMA-ES with Limited Budget\" by Belkhir et al (2017);\n it should be interesting to compare to the proposed work.\n4. Experiments\nThe experiments show the following issues:\n\t- As the proposed formula estimates the excess loss and not the performance itself, there should also be a comparison of the actual performances obtained both in the full training set and on the test set.\n\t- Amazon sentiment: \n\t\t- regression is applied on the sentiment prediction, which is a classification task\n\t\t- the estimator considered is ridge regression, which does not fit Proposition 3.1 setting and no comments are made as to how the penalty would impact (or not) the theoretical properties of the work\n\t\t- $C(q)$ contains 4 terms, while it said that there are 3 data sources in training\n\n### Minor comments:\nProposition 3.2 is not written in the same format as the others, and makes it hard to see the link with n (or log n). Please re-write it or explain where the n component is.\n\n### Overall evaluation: \nThe paper is interesting and tackles an important problem. However, it lacks clarity in many aspects.\n\n=====POST-REBUTTAL COMMENTS======== \n\nI thank the authors for answering my questions. Their answers did clarify some aspects, such as the ratio of mixed sources, the optimal data collection, and the relation of the proposed work to active learning and multitask. However, the answer provided on my comment about the distinction between excess loss and absolute loss is not sufficient. I believe this is an important point, and calls for a major revision, not just a promise to clarify the point throughout the paper. The same comment applies to the experiment when comparing methods on the basis of computing the excess loss and not showing the absolute loss, which is the actual measure of quality of an estimator.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the following broad question: How can we predict model performance when the data comes from different sources? The reviewers agreed that the direction studied is very interesting. While the results presented in this work are promising, several reviewers pointed out some weaknesses in the paper, including a confusion between absolute loss and excess loss, and the limited scope of the experiments. Overall, this paper does not appear to be ready for publication in its current form. In my personal opinion, if the concerns raised by the reviewers are appropriately addressed, this work could be publishable in a high quality venue.
Decision: Reject