A Constructive Prediction of the Generalization Error Across Scales
Authors: Jonathan S. Rosenfeld
Amir Rosenfeld
Yonatan Belinkov
Nir Shavit
Venue:  (2020)
Keywords: neural networks
deep learning
generalization error
scaling
scalability
vision
language

Abstract: The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data.

Published as a conference paper at ICLR 2020
A CONSTRUCTIVE PREDICTION OF THE GENERALIZATION ERROR ACROSS SCALES
Jonathan S. Rosenfeld Amir Rosenfeld
Yonatan Belinkov
Nir Shavit
(jonsr, bel inkov, shani (haanr)ecssil mit.edd
ami rêcse. yorku. ca
Massachusetts Institute of Technology
York University
Harvard University
4 Neural Magic Inc
Tel Aviv University
ABSTRACT
The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g.. width, depth), we are able simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and lan- guage tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data.
INTRODUCTION
With the success and heightened adoption of neural networks for real world tasks, some questions remain poorly answered. For a given task and model architecture, how much data would one require to reach a prescribed performance level? How big a model would be needed?
Addressing such questions is made especially difficult by the mounting evidence that large, deep neural networks trained on large-scale data outperform their smaller counterparts, rendering the training of high performance models prohibitively costly. Indeed, in the absence of practical an- swers to the above questions, surrogate approaches have proven useful. One such common approach is model scaling, where one designs and compares small-scale models, and applies the obtained chitectural principles at a larger scale (e.g.. [iu et al., 2018) Real et al. 2018) Zooh et al.) 2018). Despite these heuristics being widely used various degrees of success, the relation between the performance of a model in the small- and large-scale settings is not well understood. Hence, explor- ing the limitations or improving the efficiency of such methods remains subject trial and error.
In this work we circle back to the fundamental question: what is the (functional) relation between generalization error and model and dataset sizes? Critically, we capitalize on the concept of model scaling in its strictest form: we consider the case where there is some given scaling policy that completely defines how scale up a model from small to large scales. We include in this context all model parameters, such that traversing from one scale (in which all parameters are known) to another requires no additional resources for specifying the model (e.g., architecture search/design).
We empirically explore the behavior of the generalization error over wide range of datasets and models in vision and language tasks. While the error landscape seems fairly complex first glance, observe the emergence of several key characteristics shared across benchmarks and domains. Chief among these characteristics is the emergence of regions where plwer-law behavior approxi- mates the enror well both with respect to data size, when holding model size fixed, and vice versa.
Motivated by these observations, we establish criteria which a function approximating the error landscape should meet. We propose intuitive candidate for such function and evaluate its quality, both in explaining the observed error landscapes and in extrapolating from small scale (seen) to large scale (unseen) errors. Critically, our functional approximation of the error depends on both

---

Published as a conference paper at ICLR 2020
model and data sizes. We find that this function leads to high quality fit and extrapolation. For instance, the mean and standard deviation of the relative errors are under 2% when fitting across all scales investigated and under 5% when extrapolating from slimmed-down model (1/16 of the parameters) on a fraction of the training data (1/8 of the examples) on the ImageNet (Russakovsky et al. 2015) and WikiText-103 (Merity al. 2016) datasets, with similar results for other datasets.
To the best of our knowledge, this is the first work that provides simultaneously:
A joint functional form of the generalization error landscape- as dependent on both data and model size with-with few, interpretable degrees of freedom (section (5)
Direct and complete specification (via the scaling policy) of the model configuration attain- ing said generalization error across model and dataset sizes.
Highly accurate approximation of error measurements across model and data scales via the
functional form, evaluated on different models, datasets, and tasks (section 6) ).
Highly accurate error prediction from small to large model and data (section
We conclude with a discussion of some implications of our findings as a practical and principled tool for understanding network design at small scale and for efficient computation and trade-off design in general. We hope this work also provides a useful empirical leg to stand on and an invitation to search for theory of generalization error which accounts for our findings.
RELATED WORK
Model scaling: A number of studies have explored the effect of model scaling on performance.
For instance, image classification networks can be scaled by depth (number of layers: He et al. 20166) or width (number of channels; Zagoruyko Zagorruk & Komodakis [016)] Howard et al.) 2017). More recently,
Tan & Le (2019) demonstrated how scaling width, depth, and input resolution has combined positive effects larger than scaling each factor in isolation. However, this relationship has yet to be quantified in predictive form - by how much will eтror change with model scaling? In this work, we focus on finding a constructive functional form for determining the model given a specified performance.
Data scaling: It has long been recognized that more data improves performance, and various studies report such trends in both computer vision (e.g., Zhu et al.] 2012); [Sun et al.) 2017) and language processing tasks (e.g., [Banko & Brill, 2001 Talmor Berant 2019). A number of prior studies observed power-law relations between the generalization error and training data size (Cho) al. 2015 Miceli Barone al., 2011 Johnson et al. 2018). Most relevant to our work. Hestness 2017) explored the effect of data size on the generalization error in vision, language, and speech tasks, and observed strikingly consistent power-law behavior However, while these studies point to the empirical existence of a power law in terms of data, they do not offer tools for predicting the performance given specified model. Nor do they offer low-cost methods to specify the model configuration which would attain the power law with data dependency. Indeed, Hestness et al. had to search over models and their configurations at large scale to exhibit their findings, incurring prohibitive computational costs.
et al.
large set of experiments.
In contrast, we demonstrate a constructive recipe, where we directly predict the test performance at large scale and specify the full model configuration which attains (with no need for large-scale search), given performance at small scale.
Predicting model performance: Since training models full data/model scale may be compu- tationally prohibitive, a line of work tries to predict the performance of a given model on a given dataset, without training the model, for example by using a bank of previously trained models, dataset, and their associated performances (Istrate et al.) 2019). Others have proposed to estimate performance on small data (Klein et al.) 2017) or model sizes (Zoph et al. 2018 Real et al. 2019) in the context of neural architecture search (NAS). In this case, the small-scal evaluation is used to compare models at small cost, to expedite the search process; see Elsken et al. (2019) for a re- cent survey. Our work complements previous approaches by demonstrating functional form that can predict large-scale performance from small-scale measurements. Moreover, our method may be integrated in NAS, addressing some of its current limitations (as discussed in section 8).
2

---

Published as a conference paper at ICLR 2020
The datasets and models used in this work, along with their original training data size and the range of explored scales. For more information, see appendix A
Table 1:
(a) Training data size (number of words) and model size (number of parameters excluding word embeddings) for language modeling tasks.
Dataset
Size (N)
Scales (n)
Base Model
Size (M)
Scales (m)
PTB
0.9M
AWD-LSTM AWD-LSTM
20M
2 2
4
WikiText-2
2M
20M
0sk55
05
WikiText-100 103
100M
Transformer-Xe XL
41M
(b) Training data size (number of images) and model size (number of parameters) for image classification tasks.
Size (N)
Size (M)
Dataset
Scales (1)
Base Model
Scales (m)
ImageNet
1.2M
2 N, 05556
ResNet-50
25.5M
4~
CIFARIO
60K
WRN-44-16
0.7M
MM -3 5
CIFAR100
60K
WRN-44-16 16
0.7M
2 2 N,
DTD
5640
WRN-44-166
0.7M
4-*
0
Aircraft
10K
WRN-44-16
0.7M
-
UCFI01
13K
WRN-44-16
0.7M
Theoretical error bounds: Much attention has been given theoretical explanations of the gener- alization capabilities of deep neural networks (Neyshabur et al., 2017a bj Allen-Zhu al. 2018a) 80 b) Arora et al. 2018). While fully engaging with this literature beyond our scope, we note that recent studies have derived bounds involving power-lav law dependencies in both model (Yarotsky) 2018) and data size (Liang et al., 2019). We leave it as an open question for future work to find theoretical explanations for the empirical behavior and the functional form investigate in this work.
EXPERIMENTAL SETUP
Notation: Let D, (x., yd Vi denote a labeled (training) dataset with samples datapoints. Let fm denote a neural network whose size is the number of parameters m, such that ý fm (x)) is the predicted label. Let (n, m) be the generalization error function of n and m, measured by a performance metric (c.g., top-1 accuracy or cross-entropy loss) on a held-out test set. We refer to this error function as the error landscape.
3.1 SCALING POLICIES
Dataset scaling: We wish to scale datasets while preserving the original distribution. For image classification, we uniformly subsample all classes by constant ratio, thus preserving the relative sample size per class. We limit the maximal sub-sampling to avoid eradicating any class. For language modeling, where the number of classes (vocabulary items) has very long tail distribution, we randomly sample sentences such that the total number of sampled words will be certain fraction of the original dataset. Table[report.. the data scales we use. In all tasks the held-out test set remains untouched for evaluating the eтror.
Model scaling: We are critically interested in method where moving across scales is defined by some scaling function, such that no additional significant computation would be incurred. We thus consider the case where the model architecture is given and the model size determines how to scale it. For instance, one may scale width (number of channels in convolutional networks, hidden state size in recurrent networks). depth (number of layers), do compound scaling (Tan & Le 2019), or more generally define function tying the model degrees of freedom and size. We focus primarily on width scaling in our experiments; the model scales are reported Table [ We also perform selected depth scaling to demonstrate flexibility with respect the scaling method.
3

---

Published as a conference paper at ICLR 2020
0.80
004
log10
0.8
0.65
110
0.55
12 10 8 y 2 log 1og2 madel fracton))
lag2 fraction)
& v 0
-2 log2(data fraction)
fog2ídata fraction)
(a) Wikil03 emorr (cross entropy) landscape.
(b) CIFARIO emorr (topl) landscape.
Figure Error landscapes in log-log-log scale. Each point (blue dot) is the error resulting from training with model/data configuration m, The surface is linear interpolation between the points, which then projected on the (m, c), (n,, c) and (m, n) planes. See Appendix C for details.
0.00
0.75
(og(01 rrow entroyy lown
lng100
0.66
Ing2imuMi
10.0
.1.00
log2tn/k)
Ing2iniki
-10.0
+5.0
0.50
-1.-
0.55
2.0
1.0
0.0
0.0
4.0
0.0
-1.4
log21 data
log21 tracioon
log21
lodel
(a) Wikil03 cross entropy vs. data and model size.
(b) CIFARIO topl error vs. data and model size.
Figure 2: Error vs. data size (left part of each subfigure) and model size (right part) for Wiki103 and CIFARIO. Solid dots measurements, dashed lines are best fit saturating power-law.
Hyper-parameters: For similar reasons wish avoid hyper-paraminn search at large scales, and thus avoid the temptation to tune hyper-parameters parameters accordingly (Learning rate, regularization, hold all hyper-paramettrs fixed. This enables
etc.). Therefore, that fits the error landscape and can be used to predict the error across scales while completely defin- ing the model attaining We consider pros and cons of this approach the discussion (section 8)
construct functional form
3.2 TASKS, MODELS, OPTIMIZERS AND DATASETS
We experiment with both vision and language tasks. We use 6 benchmark datasets for image classi- fication and 3 for language modeling. For image classification, we train ResNet (He et al. (016) and WRN models (Zagoruyko & Komodakis) 2016) with stochastic gradient decent (SGD). In section 6.2 we explore the effect varying architectures and optimizers for a fixed task (CIFARIO0 adding
VGG16 16 (Simonyan & Zisserman 2014) and DenseNet (Huang et al.) 2017) models trained with both
Adam Kingma Ba 2015) and SGD. For language modeling, we train AWD-LSTM (Merity et al,
2018) and Transformer XL models (Dai al.) 2019) with SGD and Adam optimizers respectively.
Summary statistics are shown Table | along with the range of explored scales. Appendix A gives additional information.

---

Published as a conference paper at ICLR 2020
4 OBSERVATIONS ON THE ERROR LANDSCAPE
Figures
respectively show an example test error landscape for width scaling of Transformer-Xe XL WikiText-103 and WRN-44-16 CIFARIO. Various additional such land- scapes are found appendix|C] showing largely consistent patterns. Examining the error landscapes yields the following observations:
La and
lb
O1 Model Scaling
01.1 For given dataset size, scaling up the model results in initial decrease in test error, which then saturates to a level determined by the dataset size This behavior has been noted by Tan & Le (209)) across varied model scaling methods, although they have not engaged with the dependency on dataset size.
01.2 The rate of error decrease with model size appears well approximated by a power-law.
These two observations together be summarized the following relation:
e(m, n)) b(n)m 6(n)m=-P()) Gm (12)
where b, 3, Cm may depend on the data size n, s.t. as m grows, + Cm. Example fits to this form (allowing b, 3, Cm to be fit per n) are in figure 2a (right) and figure 2b (right).
02 Data scaling
02.1 For a given model size, scaling up the dataset results in an initial increase in performance, which then saturates to a level determined by the model size.
O2.2 The rate of error decrease with dataset size appears well approximated by power-law. Hes- tness et al. (2017) also noted a similar relationship, but did not functionally tie the saturation level to the dataset size.
These two observations together can be summarized as the following relation:
<(m, n) x a(m)n (m)) + (m))
(2)
where a, ax, Gn may depend on the model size m, s.t. grows, Cp- Example fits to this form (allowing a, Ot, Cp to be per m) are seen in figure 2a (left) and figure 2b (left).
03 Joint properties The behavior of the error when scaling model size while holding data size fixed, and vice versa, extends to the entire error landscape in a well-behaved manner, such that the manifold e(m, n) is smooth everywhere as a function both model and data scales.
FUNCTIONAL APPROXIMATION OF THE GENERALIZATION ERROR
5.1 CRITERIA
Motivated by the above observations, we now consider functional approximation for the error land- scape. In particular, let consider function families meeting the following criteria which augment and restrict our observations:
C1 As either model or dataset size goes to zero, the expected performance is equivalent random-guess error level éo [
C2 For given dataset size, scaling up the model will result in an initial increase in perfor- mance, which will then saturate, taking the form in equation [
C3 For a given model size, scaling up the dataset will result in an initial increase in perfor- mance, which will then saturate, taking the form in equation 2
C4 There exists an irreducible error Coe, intrinsic the dataset.
C5 The function must be smooth everywhere and monotonic non-inereasing in terms of model and data size (observation 03
While there are many possible function families meeting the above criteria, below we propose simple function family for our evaluation. We do not claim that this is in fact the true underlying dependency, but rather that it serves as a good approximation of the error landscape- consistent with these criteria.
At some point error increase ensues; this point differs between datasets, see Appendix C for examples. Best guess when m (Con) n (cmo) need not coincide, but can, c.g., balanced dataset.
5

---

Published as a conference paper at ICLR 2020
5.2 PROPOSED FUNCTION FAMILY
As a first insightful step, consider the implications of satisfying C2 and C examining the limiting behavior as m or n grow, we have:
simultanoussly By
As m grows large:
(m(n) a(m)n a(m)n-a(m) a(m) c, (m)
As grows large:
Ca(m) x b(n)m - B(m) + (n)
Thus, a consistent form satisfying and simultancously is:
e n) R a(m)n am() + b(n)m - B(n) + Coo
(3)
where Coo is a constant not dependent on either m or n.
Let us now examine the simplified case where a, b, at B are constant:
É(m, n)) = an + bmn -8 + Coo
(4)
where a 2 0 and / 2 control the global rate at which error decreases with data and model size, respectively, a >> 0 and b are form of unit conversion between data and model sizes and error, and Coo > 0 is the asymptotic lower value attainable. This function is a special case of equation and meets criteria C2 and C3 by construction. Importantly C4 and C5 are also met.
However, by giving up the dependence of a, b, a, 3 on m, n., this function does not meet criterion CI We thus need model the transition from the initial random-guest level the power-law region. We propose to parameterize the transition using the following envelope (complex) function:
ε(m,, n)) in
an + bm - + Coo
E(m, n) = 60 7(m. n)
= an-a + bm- B + Coc in
(5)
where V-T. Here the simple pole at controls the transition point from the initial random-guee gugsss level co as (m, n) increase. As (m, n) grow, غ --> Coe and the final irreducible error 6so - CyCoe?) is approached. The random-guess error, cp. is known parameter determined by dataset statistics (e.g, (Netasses 1 1)/Nansses for balanced dataset), Note that due to our choice of rational envelope, we can divide by constant the form in equation Without loss of generality, let choose 1.
Note that while the forms equations 3 and are well motivated, the approach taken for modeling the transition is solely convenience one. In fact, the transition(s) as function of and may be captured in the functional forms a, 6, a, 3 or another envelope mechanism. We leave a more refined investigation of the nature of the transitions to future work.
ERROR LANDSCAPE ESTIMATION
We wish empirically estimate the quality the proposed functional parameterization as the true error landscape. Let ě(n, 0) be the parametric function family (equation 5) approximating the error landscape (n, m)) where 0 {ax, 3, b, Goo ") Define the divergence (n, m: 0) as the relative difference between the estimated error ē(m, n; 0) and the true error <m,, nt):
é(m, n: 0) e(m,n)
8(n, m: 0) &
el emm, n)
We fit least squares regression model find the best parameters minimizing the divergence. In this section, we fit the function using 10-fold cross-validation across all model/data configurations m, from I and evaluate the fit quality. (In the next section, we perform extrapolation experiments, unseen points.) We perform the fit separately for each dataset and evaluate its quality by the mean and standard deviation o of the divergence over all points (m, n). See Appendix B. for experimental details.
(see T
As figure shows, estimated test accuracy is highly correlated with actual test accuracy for various datasets, with worst-case values 1% and < 5% Note that the number of free parameters is small (i0) < 6) compared to the number points (42-49 model-data configurations). demonstrating the appropriateness of the proposed function for modeling the complex error landscape.

---

Published as a conference paper at ICLR 2020
12-
atrcntt:
2:3-700-1
2:3.400:2
(Leartt::
entimtted box
25-
0.4
45-
6.8
4.-
0.2
15
10
0.3
30
2.
4.2
3.0
1.
measured teet lann
meesured topt wenor
(a) Estimated vs. actual cross-entropy loss for various language modeling datasets.
(b) Estimated vs. actual test error for various image classification datasets.
Figure Error estimation results, using 10-fold cross-validation on all configurations in each dataset. For reference, in blue is the identity line. The legend shows μ and standard devi- ation a of the divergence (# one std). See Appendix for the actual and estimated landscapes in each dataset.
3.4
1.0
-05
estmm estimated ated topl error
0.0
0.8144
-.7
0:02:0APP
080
0-128: 0.15.8
0.6
-11
-10
0.4
extmmate
0.4
bog Stmodd (racien)
0.0
0.0
I0g2)DS fraction)
3
0.0
0.2
0.6
0.8
0.4
measured top1 error
measured top1 errar
(a) Error landscape when scaling depth (at constant baseline width).
(b) Width scaling at different constant depths (D).
(c) Depth scaling fit at different constant widths (W).
Figure 4: Error landscape estimation results CIFARIU for width and depth scaling, showing small and comparable fit errors in both cases. Numbers in legends denote mean/van variance of the estimation divergence.
6.1 PROBE INTO DEPTH SCALING
Here we verify that our results extend to another canonical scaling policy, namely depth scaling- Figure 4aluee/asssa shows the error landscape with depth scaling on CIFARIO, exhibiting the same character- istics width scaling. Figures 4b and 4c show error landscape estimation results for both cases of width and depth scaling, exhibiting small and comparable fit errors (confidence intervals < 3%).
Since the difference in approximation quality is effectively indistinguishable when scaling depth or width orthogonally, we expect compound scaling to adhere to the same functional form. Indeed, we verified this the publicly available (model scaling only) results for EfficientNet (Tan & Le 2019
6.2 THE VARIETY OF OPTIMIZERS AND ARCHITECTURES
deliberate variety of architectures (ResNet, WRN, LSTM, Trans- former) and optimizers (Adam, SGD variants), following standard implementations in the literature as recommended for each dataset/model setting; see Appendix A
Our study
For image classification, set éo (Nelasses 11/Nelasses (the balanced dataset case). For language modeling, estimate to another parameter, such that 0 {a, B, coo: 7, 6o) in this case.

---

Published as a conference paper at ICLR 2020
1.0
7.0
extragolated
extrapolated
6.5
fraction: log2imaM)
1.0.5%
error
estimated test loss
6.0
011.689%
0.4
4.5
4.0
-10
0.2
3.5
model fraction 1/16
model fraction 1/16
data traction 1/8
data fraction 1/8
5-5 34-32-1 0 data fraction: log2(ruN)
0.0
0.2
0.4
0.6
0.8
10
3
test
(a) Illustration.
(b) Extrapolation on ImageNet
(c) Extrapolation on WikiText- 103,
Figure 6: Extrapolation results. (a) Illustration of the extrapolation setup, where fit subset of the points (in green) and predict on larger points (in red). (b) and (c) show example results on one configuration in two benchmark datasets. Comprehensive results are given in Appendix D
However, the model/optimizer settings differ in multiple aspects across the different tasks ren- dering the comparison of, say, different optimiz- ers, challenging. In this section we verify that the functional form holds when varying the optimizer and/or the architecture on the same task, namely image classification on CIFAR100.
10:
0.8
estimated topa ertor
0.6
In addition to the previously examined setting of
WRN with SGD, we add four more settings: two well known architectures (VGG and DenseNet), each trained with both SGD and Adam optimizers. See Appendix A for experimental details. Figure 5 exhibits consistent, accurate, fit values across all ar- chitecture/optimizer settings, with mean divergence of μ < 1% (std: o 6%; confidence intervals 4%).
0.4
wr//spt
d:1.440.11
rap/adan:
4:0:021.4
a:1.220.5
v22/5od
pe9.921.5
a11.359.1
p.8.5e3
014.552.75
pi8.522.6
0.0
0.6
messured top1
Figure 5: CIFARI00 Error estimation re-
sults with three architectures (WRN, VGG, DenseNet) and two optimizers (SGD, Adam).
EXTRAPOLATION
In this section, evaluate the ability of our functional approximation extrapolate beyond model/data configurations. The primary question we ask is: can we predict the error of a large model/data configuration from the errors of smaller-scale model/data configurations? To do this, we fit the least squares regression on a subset of the configurations and predict the error on larger, unseen configurations. More formally, let (m,,71,) denote given model/data configuration. We first estimate parameters 0,j by fitting the function in equation 5 on all points of at most that size (m S m4, n S n.). Then we predict the error c(m, n) in all points corresponding (m m,, > n.)) using estimated 0.y. Finally, we measure the divergence 8(m, n) between the estimated error and the actual error all larger configurations. This process is illustrated figure ba
larger configurations
Figure 6b shows the results of one such extrapolation experiment, on ImageNet. In this case, we have fit the functional form on configurations of model size m < m, M/16 and data size < n2 N/8, and predicted the error all larger configurations. As the figure shows, the extrapolation is highly accurate, with a mean divergence of μ 4.5% (std: o = 4.7%). Figure 6c very good extrapolation, with
reports a similar experiment mean divergence μ = 0.5% (std: o 1.7%). Note that each extrapolation is run 10 times with different random initializations of 0,g in the least squares with negligible effect the prediction.
WikiText- 1033. Here, again,
8

---

Published as a conference paper at ICLR 2020
In practice, we may be interested in extrapolation quality with different subsets of configurations. Appendix D provides detailed extrapolation results on multiple subsets of configurations, for both vision and language datasets. Generally, the extrapolation performs well once not ill ill-posed, which may be caused by lack of signal in the region of the initial "random-guess" level, or in degenerate cases like having fewer measurements than the number of free parameters in O.
DISCUSSION AND CONCLUSION
In this work, through insights gained by the joint examination of the dependencies of generalization error on both model and data size, we arrive at criteria for functions consistent with the form of the generalization error under a given scaling policy. We consider one such function and find it to be in very good agreement with the actual behavior of the error landscape. Indeed, the agreement is strong enough that extrapolation from small to large scale becomes feasible: the function predicts the behavior of the generalization emror in practice for the practical case of scaling models and data. We discuss several example implications of knowing such functional form.
Small-scale network development: At the of small fidelity searches is the notion of perfor- mance rank comparison between models. However, small scale and large scale ranks are not assured to be consistent. If indeed functional form such as empirically found in this work holds very gen- erally, then in contrast, one can safely assess scaling rank between models at small scale, with the assurance that it remains consistent. This suggests that would be well served by searching over scaling policies; a pertinent example of such a success is Tan & Le (2019). The functional form also explains the limitation of small-scale search: once reaching the random-guess error level, where the sensitivity to scaling vanishes, the informativeness of ranking diminishes. Finally, the functional form allows direct usage of differentiable methods for NAS.
Principled design: Knowing the error landscape function facilitates reasoning about the choice of (m, n)) attaining a specified error level. In other words, for any given error level, one can solve Eq. 5 for m, n based on small-scale measurements. Thus, one quantitatively answer design questions regarding the expected (in particular, large-scale) relations between m, n., and c. In fact, Eq. 5 provides direct ansewrs questions such "how much data would one require reach prescribed performance level?" or "how big model would be needed?" Imposing constraints is also straightforward. For instance, consider the following question: "What is the maximal model size possibly needed (useful). when the data is limited in size, n nim (for a given model architecture and scaling policy)?" For fixed dataset size, model scaling eventually contributes marginally to error reduction and becomes negligible when byn- < niam (Eq. 5). Define the relative contribution
threshold T satisfying T = meeting threshold is:
(For example, T 10.) Then the maximal useful model size bmm
Piem
Myman (T) = (BT) a/B 1
Similarly, The maximal useful amount of data for a limited sized model mism is:
Timaa (T) (1/6T) 1/ax
3/a m
Moreover, Eq. 5 alllow for complex design trade-offs. Generally, given some design-tradeoff cost function ('m,, n, e). one can minimize such cost s.t. Eq. For example, consider the case of opti- mizing for efficient computation which has both practical and environmental importance (Schwartz) et al., 2019). Since the number of FLOPs during training is x m n (for constant epoch budget), the trade-off cost function may be formulated as C(FLOPS, c) C(mn, c). Further, since constant error contour is very well approximated by c 1 + 6 m (Eq. 5) dataset and models may be scaled with optimal resource efficiency with no effect on performance by solving for:
1 b
argmin m,
m 72
s.t.
n m
The solution gives the optimalccompptaaiolllfffcicnn ratio of model data size:
63 n m
1.

---

Published as a conference paper at ICLR 2020
Limitations: We have made few simplifying assumptions in choice of approximating fune- tion, in particular in how to model the transition from the initial random-guess guesss error level and the union of the random-guee guesss level of the two scenarios (small model with large data and large model with small data). We leave a more detailed examination of the behavior of the transitions from random-guess error levels and refinements of the functional form to future work.
Critically, the restrictive nature of our scaling framework (all parameters and hyperparameters de- scribed by policy) is both blessing and challenge. The blessing comes in fulfilling the goal of finding simultaneously both the form of the generalization error and the full specification of the model and hyperparameters that attain it across scales. The challenge is that we have demonstrated in this work only the case of constant model configuration and hyperparameter choice (Zela et al., 2018) may entail the potential to for- mulate lyperparameter-saalinn too fall under the scope of the form we find in this work. This too will be the subject of future work.
We conjecture that the relation between
policies similar in nature to the model-scaling polices, and that these
We hope that this work will bring the actual functional form of the generalization error in this practical case of scaling to the fore, both in practice and as empirical leg to stand on in the quest for its theoretical origins.
ACKNOWLEDGMENTS
We thank Alexander Rakhlin, Alexander Madry, Kai Xiao, Lu Mi, Viaks Garg, Dan Alistrah, and Tommi Jaakkola for discussions and their help. We also thank the anonymous reviewers for their valuable feedback. J.R. was partly supported by the Eli and Dorothy Berman Fellowship well as grants NSF IIS- 1447786, NSF CCF-156333
and China-Singapore Suzhou Industrial Park. A.R. was partially supported by the Air Force Office of Scientific Research USA (FA9550-18- 18-1005)))) though a grant to John K. Tsotsos. Y.B. was partly supported by the Harvard Mind ,Brain, and Behavior Initiative.
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li. and Yingyu Liang. Learning and generalization in overparameter- ized neural networks, going beyond two layers. arXiv preprint urXiv:1811.04913 2018a.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. On the convergence rate of training recurrent neural networks. arXiv preprint arXiv:/8/0. 12065, 2018b.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:1802.05229. 2018.
Michele Banko and Eric Brill. Mitigating the paucity-of-dan problem: Exploring the effect of training corpus size classifier performance for natural language processing. In Proceedings of the first international conference on Human language technology research, pp. 1-5. Association for Computational Linguisties. 2001.
Hakan Bilen, Basura Fernando, Efstratios Gavves, Andrea Vedaldi, and Stephen Gould. Dynamic image networks for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3034-3042, 2016.
James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. Quasi-recurrent neural net- works. In International Conference on Learning Representations, 2017.
Junghwan Cho, Kyewook Lee, Ellie Shin, Garry Choy, and Synho Do. How much data is needed to train a medical image deep learning system to achieve necessary high accuracy? arXiv preprint arXiv:1511.06348, 2015.
Mircea Cimpoi. Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures the wild. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 3606-3613, 2014.
10

---

Published as a conference paper at ICLR 2020
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Anmual Meeting of the Association for Computationnl Linguistics. pp. 1978-29888 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. Nt.18633/11/19/ 1285... URL.
https: / www. ac web www.:wwwbb.............. wwr.oorgg//aaa.oo weweanhhoooyy/oono ogy/ 9- 1285
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey.
Journal of Machine Learning Research, 20(55)1121,, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. pp. 770-778, 2016.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint 1rXiv:1712.00409, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: the marginal value of training the last weight layer. In International Conference on Learning Representations, 2018. URL.
Sattps://openreett
net www.:neeee/oooooooeoooo 2o
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 17Xiv:1704.04861, 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. 1(2):3, 2017.
Roxana Istrate, Florian Scheidegger, Giovanni Mariani, Dimitrios Nikolopoulos, Costas Bekas, and A Cristiano I Malossi. Tapas: Train-less accuracy predictor for architecture search. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3927-3934, 2019.
Mark Johnson, Peter Anderson, Mark Dras, and Mark Steedman. Predicting accuracy on large datasets from smaller pilot data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 450-455, Melbourne, Australia, July
2018. Association for Computational Linguisties. doi: 10. 10.18653/v1/P18 2072. URL https: 7/ 7/ww www. aclweb. org/ wwthhooocoo/111200000
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian op- timization of machine learning hyperparameters on large datasets. In Artificial Intelligncce and Statistics, pp. 528 536, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Tengyuan Liang, Alexander Rakhlin, and Xiyu Zhai On the risk of minimum-norm interpolants and restricted lower isometry of kernels. arXiv preprint arXiv:1908. 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.0903: 09055, 2018
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained 2013.
visual classification of aircraft. arXiv preprint
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1rX.::16090078433 2016.
Stephen Merity. Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. In International Conference on Learning Representations, 2018. httpss:::
// //openrevis ew. www. wttp:t//ooruummddddyyooopereee

---

Published as a conference paper at ICLR 2020
Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. Regularization techniques for fine-tuning in neural machine translation. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 1489-1494, Copenhagen. Denmark, URL.
September 2017. Association for Computational Linguistics. doi: 10.1 (0.18653/11/D17-11 https: www a web. org/ orr.oorg//aaat antho http://aathooooooooooooooeo hgtp//1-yy////////00
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Kleventh Annual Conference of the International Speech Communication Association, 2010.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization deep learning. In Advances in Newral Information Processing Systems, pp. 5947-5956,
2017a.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to opectrall-normaaliza margin bounds for neural networks. arXiv preprint arXiv:1707.095644 2017b.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Real, A Aggarwal, Y Huang, and QV Le. Aging evolution for image classifier architecture search. In AAAI Conference on Artificial Intelligence, 2019.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. arXiv preprint /rXiv:/882.001548, 2018.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, pp. 506-516, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)211 2015.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. arXiv preprint arXiv: 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1xXiv:1409.11. 2014.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 1212.0222.0422 2012.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. Proceedings of the IEEE international conference on computer vision, pp. 843-852. 2017.
Alon Talmor and Jonathan Berant. MultiQA: An empirical investigation of generalization and transfer in reading comprehension. In Proceedings of the 57th Anmual Meeting of the Associ- ation,fr Computational Linguistics, pp. 4911-499 1-141,,,, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 1..1865/v11/19... 1485. URL www awwwwb.wwb.... wtwp:/wwww..www.r..//ooo..oo.. org/
lanthology/e19-11 1485
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-61144 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan. and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998 60088.60088000000 Curran Associates, Inc., 2017. URL attention all--ou--eeeee pdf
http::/ www..i....pppppeem
12

---

Published as a conference paper at ICLR 2020
Zifeng Wu, Chunhua Shen, and Anton den Hengel. Wider deeper: Revisiting the resnet model for visual recognition. arXiv preprint arXiv: 161.10001,,00800,, 2016.
Dmitry Yarotsky. Optimal approximation of continuous functions by very deep relu networks. arXiv preprint wrxiv:/80.003666 2018.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
1rXiv:1605.07146, 2016.
Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint arXiv:1807.0600 0696666 2018.
Xiangxin Zhu, Carl Vondrick, Deva Ramanan, and Charless C Fowlkes. Do need more training data or better models for object detection?. In BMVC, volume 3, pp. 5. Citeseer, 2012.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pp. 8697-8710, 2018.
13

---

Published as a conference paper at ICLR 2020
A DATASETS AND MODELS
A.1 IMAGE CLASSIFICATION
A.1.1 DATASETS
We evaluated our predictions on several popular image classification datasets: ImageNet (Rus-) sakovsky et al. 2015): a large-scale recognition benchmark consisting of natural images of 1000 ob- ject categories with I. 1.28M training images spread roughly uniformly over the categories. It has 50K validation and 100K testing images. It has been the most popular large-scale benchmark for image classification methods for the better part of the last decade. CIFAR 1F/AR00/100 (Krizhevsky al. 2009): 60K natural RGB images of 10 classes (100 for CIFAR 100) with a train/test split of 50K/10K, For each of the following datasets, we the version collated, resized, and split into train/validdationtest sets by |Rebuffi et al.] (2017). DTD (Cimpoi et al.) 2014): a texture database of 47 categories and 5640 images. Aircraft (Maji et al. 2013): 10K images of 100 different aircraft classes. UCFI01
(Soomro et al. (2012): originally video action recognition dataset, converted using the method of
Bilen al. (2016) into single image per video. It contains 13,320 images of 101 action classes.
A.1.2 MODELS
We experiment with four models for image classification. We use different variants of the popular ResNet architecture (He al., 2016) in the main experiments. For ImageNet and build on the code from the PyTorch framework (Paszke et al. (2077) to vary the model width. For all other datasets we use WRN-44-16 (Wu et al. 2016) of varying widths, modified from the implementation of Holfer et al. (0188)
ResNet-50
Scaling the models' width performed by multiplying the number of channels in each convolutional layer and the width of the hidden linear layers by constant factor and rounding to the nearest integer. The ranges of width scales (and data scales) for the main experiments are detailed in Table Tb
In section 6.2, we perform width scaling for two additional architectures, VGG16bn (Simonyan & Zisserman 2014) and DenseNet (L=400, k=32) (Huang al. 2017). The VGG and DenseNet models were also modified for width scaling from the implementation of Hoffer et al. (2018). The model scales in this case are 4 k S 5, for both VGG and DenseNEt.
Depth-scaling, in the CIFAR10 case (section 6.1 is performed by appending extra layers within each block.
A.1.3 TRAINING
In the main experiments, training is done via SGD with momentum of 0.9, weight decay of lc-4 and initial learning rate of 0.1. For ImageNet we train for 90 epochs, decreasing the learning rate by multiplicative factor of 0.1 after and 30 and after 60 epochs. We use a batch size of 16. For all other vision datasets we use batch-size of 128. We begin training with a learning rate of 0.1, run for 200 epochs, and reduce by multiplicative factor of 0.1 after 80, 120, and 160 epochs.
For the VGG and DenseNet experiments on CIFAR100 in section 6.2 we train with both SGD and Adam optimizers. We train VGG for 170 epochs and Densenet for 300 epochs. Adam hyperparam- eters are default, with an initial learning rate of le-3. When training with SGD, we retain initial learning rate, batch size, momentum, and weight-decay, and le-4 respectively) and follow standard stepped learning rate schedules: For VGG, learning rate multiplicative factor of 0.1 after 80, 120, and 160 epochs; For DenseNet, learning rate multiplicative factor of 0.1 after 150 and 225 epochs.
in the main experiment (at 0.1, 128, 0.9,
A.2 LANGUAGE MODELING
A.2.1 DATASETS
We evaluate on several datasets commonly used for (word-level) language modeling: Penn Tree- bank Miikolov et al. 2010), WikiText-2 (Bradbury et al. (017).. and WikiText-103 (Merity et all] 2016). The PTB relatively small language modeling dataset of news texts, with vocabu-
14

---

Published as a conference paper at ICLR 2020
lary of 10K unique words and about 900K/70K/80K training/validaaionneest words. WikiText-2 is drawn from Wikipedia articles and it is both larger and richer, with a vocabulary 33K words and 2M/210K/240K viiniing/alidaation/estt words. WikiText-103 is also based on Wikipedia, but larger still, with a vocabulary of 270K words and 100M training words (and the same validation and test sets WikiText-2).
A.2.2 MODELS
We experiment with two standard models for language modeling: Transformer-XL (Dai et al. 2019) and AWD-LSTM (Merity et al. 2018). Transformer-XI. is recent language modeling architecture that is based transformer self-attention (Vaswani al. 2017). but modified to better learn de- pendencies beyond fixed length by adding segment-level recurrence mechanism. It has achieved state-of-the-an results multiple benchmarks. We the official PyTorch implementation" with their base configuration: 16 layers, embedding size of 410, inner dimension of 2100 in the fully- connected layers, and 10 attention heads. Training is done with Adam. See the implementation for other details. For scaling experiments, we decimate the inner dimension. We use Transformer-XL. for 1ik
AWD-LSTM is a long short-term memory (Hochreiter & Schmidhuber 1997) language model with adaptive weight averaging. We use the official implementatior with the recommended configura- tion: 3 layers, embedding size of 400, and hidden state size of 1550. Training is done with SGD. We use AWD-LSTM for PTB and WikiText-2 and follow the recommended settings for these two datasets. For scaling experiments, decimate the hidden state size.
15

---

Published as a conference paper at ICLR 2020
B
ERROR ESTIMATION EXPERIMENT
B.1 EXPERIMENTAL DETAILS
In the experiment described section 6 we fit least squares regression model find the best parameters minimizing the divergence o n) evaluated at configurations n as in Table I
0* arg min
16(m, n: 0)1
We quantify the quality of the fit by the mean μ and standard deviation or the fitted divergence by performing standard 10-fold cross validation over all points (m, n) with confidence intervals reported as +:1 std over the folds.
B.2 FOUND THETA VALUES
Table Optimal values of found by the least squres regression fitting the functional form.
(a) Image classification (fitting top I error).
Coc
ImageNet
0.75
0.61
0.76
3.63
18.50
CIFAR10
0.66
0.53
5.87
10-02 02
7.14 10-14
19.77
CIFARI00
0.70
0.51
0.15
0.71
6.93
DTD
0.40
1.16
4.30
10-08
1.27
0.85
Aircraft
1.10
0.83
3.47
10
5.16 10 10
1.13
UFC101
0.93
0.54
4.68
10 02
1.16
10
2.98
(b) Language modeling (fitting cross entropy loss).
a
3
Coo
PTB
0.81
0.34
0.15
5.00
6.27
6.10
WikiText-2
1.01
0.22
0.99
8.23
10.38
6.21
WikiText-103
0.74
0.56
0.33
9.04
16.34
6.60
16

---

Published as a conference paper at ICLR 2020
C ADDITIONAL ERROR LANDSCAPE MEASUREMENTS AND ESTIMATIONS
In this appendix, we provide error landscape measurements and estimations for all datasets, corre- sponding the experiment in section 6 The results are shown in 3D graphs similar figure | In each such graph, the z-axis is the logarithm of the generalization error as a function of two indepen- dent variables: the model size m and the data size n.
The 3D graph is deliberately portrayed in log-log-log scale, as we cover a very large range of data scales and model scales and correspondingly wide range of errors. This view useful one when one wishes to evaluate both large dynamic ranges (simultaneously both very large and very small values) and especially vivid in portraying power-law like dependencies; power-law naturally forms straight line in a log-log view.
In each figure, subfigure (a) shows the measured error landscape is in log-log-log scale, where each point (blue dot) is the error resulting from training with model/data configuration m, Subfigure (b) shows the best-fit estimated error landscape. The surface a linear interpolation between the points, which is then projected on the model-error (m, data-error (n, c), and model-data (m, n) planes. The contour plots on each one these planes are the projections of the error landscape surface, and useful in considering the behavior of the surface when holding dimension constant.
We call to attention several interesting observations on the datasets explored:
As quantified rigorously in section 6 the fits perform well across error ranges. In these surfaces, one also gets qualitative sense of the fit adequacy across the wide ranges of the dataset and model scales directly. While perhaps slightly difficult to asses the surface di- rectly, helpful view is to consider the similarity between the projections of the actual and projected surfaces.
With increasing model size, indeed typically the error does remain saturated. However, in one of our tested datasets (figure 12] there was renewed slight increase. We verify that this is indeed over-fitting, in the sense that there is no corresponding increase in the training error. We note that the functional form we find can actually be used to veer clear of the m, regions where such over-fitting may occur.
The simplifying approach taken by considering the random guess levels (and associated transitions) for small models or small data as identical, seems work fairly well with some deviation apparent by examining figure [5] Indeed the simplification can hold well for balanced datasets, but need not for imbalanced ones such as in the task language modeling. Thus, relaxation of this simplification is expected to be important conceptually and practically.
17

---

Published as a conference paper at ICLR 2020
imagenet: actual test err
Imagenet: estimated test err
log 10101 em
-0
10g
0.4
0.4
2 (ppram
22
1og2( @pgara))))
22
JDg fraction)
18
log2(DS fraction)
16
16
Actual error landscape.
(b) Estimated landscape.
Figure 7: ImageNet erro landscape.
cifar10 O: actual test
cifar10: estimated test
-0.4
0.4
10g10
log10/er/ (()))
0.8
0.8
12
2
2g 20
-5
log2 @param)
-5
log21 (param)
16
20
IDS fraction)
14
log2(DS fraction)
14
-1
12
(a) Actual error landscape.
(b) Estimated error landscape.
Figure CIFAR1O error landscape.
decathlon_cifarl00:: actual test err
decathlon_cifar100: estimated test err
0.2
(e)))))))
02
log10 01 (())
0.4
22
22
tog2 parram)
20*
10921 p9gi(spal param))
20
1g
18
-2 IDg2(DS fraction)
14
log2(DS fraction)
-2
14
12
12
(a) Actual errrr landscape.
(b) Estimated error landscape.
Figure 9: CIFAR100 error landscape.
18

---

Published as a conference paper at ICLR 2020
decathlon_ce dtd: actual test err
decathlon dtd: estimated test err
0.02
0.04
0.04
00.06
0.00
0.08
0000
0.10
-0.10
0.12
0.12
22
log 2 (param)
20
tog #param)
20
18
26
16
IDS fraction)
-2
14
log2(DS fraction)
-2
12
12
Actual error landscape.
(b) Estimated landscape.
Figure 10: DTD error landscape.
decathlon airccaft:: actual test
decathlon_aircrann aircraft: estimated test
0.050
0.050
0.07
0 100
0.10
0.125
0.150
0.150
0.175
00.175
log2 @param)
1og2 1og2 (param)
20
-5
16 18
16
IDS fraction)
24
log2(DS fraction)
(a) Actual error landscape.
(b) Estimated error landscape.
Figure 11: Aircraft error landscape.
decathlon_ucfll actual test err
decathlon_ucf101. estimated test err
-0.1
0,1
Toglo(er))
logloferr)
0.3
0.4
-044
22
22
tog2 (parmm)
20
10921 leg2(spaa param))
20
1g
18
-2 Iog2(DS fraction)
14
-2 log2(DS fraction)
14
12
12
(a) Actual errrr landscape.
(b) Estimated error landscape.
Figure 12: UFC101 error landscape.
19

---

Published as a conference paper at ICLR 2020
PTB: actual test loss
PTB: estimated test loss
0.76
0.76
0.74
0.74
em
0.70
0.70
0.60
0.68
0.66
log2 imadll fraction)
logllmodel fraction)
-2 IDS fraction)
l0
-2 log2(DS fraction)
20
-2
-2
Actual error landscape.
(b) Estimated landscape.
Figure 13: PTB error landscape.
wiki2: actual test loss
wiki2: estimated test loss
0.78
0.76
0.76
0.72
0.72
0.70
0.70
0.68
0.68
0.66
0.66
2.64
0.64
Y2 1o i ' 0
1 y V
log 2og2 fraction)
log fraction)
-5
-5
IDS fraction)
-3
log2(DS fraction)
-1
-2
(a) Actual error landscape.
(b) Estimated error landscape.
Figure 14: WikiText-2 enror landscape.
wikil03: actual test loss
wiki103: estimated test loss
0.80
0.80
10gl0 (ogl0(em))
logl0(err)
0.65
0.65
0.55
0.55
log modell fraction)
-l a & 2
model fraction)
b >2
-2 Iog2(DS fraction)
-2 log2(DS fraction)
-12
(a) Actual error landscape.
(b) Estimated error landscape.
Figure 15: WikiText- 103 error landscape.
20

---

Published as a conference paper at ICLR 2020
D ADDITIONAL EXTRAPOLATION RESULTS
Here we provide detailed extrapolation results, for all datasets, All figures are structured in a similar way. Each subplot shows estimated (y-axis) vs, actual error (x-axis) (0 to scale on both axes). Each subplot is located at the coordinate of the maximal data and model given for the task of performing the fit the functional form equation [ This is the point the top-right comerr of the green dots in the illustration in figure 6a The target is to find the error-landscape values for unseen, larger scales of both model and data (red points in the same illustration). Going from left to right in each figure indicates observed measurements of the error from models of an increasing fraction w.r. the full size. Going from bottom-to top indicates observed measurements of the error from dataset sizes of increasingly large fraction of the full dataset.
In each subplot, every point shows the estimated vs. actual error on a model-data configuration. Points that were given for fitting the function are colored in green, while unseen points that were not used are in red. The red points show the estimation error vs. actual error when extrapolating to all larger models and data sizes. In each subplot, the mean and standard deviation over all divergences 5 at target points are given in text.
Each experiment fit of the parameters repeated 100 times, with different random initializations of 0. The shaded bands show one standard deviation across these runs.
The quality of the extrapolation is critically dependent the signal provided in the (green) fitted points. Two limiting factors are evident by examining the figures below, which both play a role in the well-posedness of the solution:
The proximity to the initial random guess level. Only upon transitioning from the initial error platcau, does meaningful signal about the scaling rates become available. Indeed, for scales prior still in the region or close to the initial error level, one sees poor extrapolation
results; see figures 18
19 and 21 and the vivid origin of this phenomena by examining
figures 11 10 and 12
A second source of ill- ill-posedness is tied to the number of configurations used for the esti- mation of 0. Clearly, when this is small, one cannot expect the extrapolation to be stable. In fact, at least two measurements in each scaling dimension (model/data) are needed, and no less than the number of parameters in in total. Indeed, for all the plots in this ap- pendix, the smallest scale of m, is omitted form the graph such that the lowermost row and leftmost column span exactly two model and data scales correspondingly. Of course, there is nothing tying directly the number of points and scale of configurations measured, and one can decouple these two factors by taking closer spaced samples at small scale.
When both the above factors are not limiting the mcasurement, one readily sees that for divergences of more than a few percent, it is sufficient to measure model/data config- urations which are far-ranged from the configurations which one wishes to extrapolate to
21

---

Published as a conference paper at ICLR 2020
imagenet
1.00
1-15.125.8
Jr 03.00.00.0
10.1000.0
M.5.800.0
#:53=002
0.75
0:11.3+3.6
0:5.8+0.0
o2 22.00
0.0000.0
000.0=0.0
0.00
1.00
4-12.025.3
6.30.000.0
22..520.22
#:.0200.1
0:3.320.0
0:11.723.7
66..
02.520.0
0...20.11
0:1.9.0.0
88 00.020.0
0.00
0...0.00.00
4...000.00
2.200000
n-11.660.66
Model Frection (og2mm3
0.50
o.9.9t4.1
09.740.0
04.720.0
0.5.20.00
0:3.320.0
a3.120.2
0.00
0113300.0
6.8.620.1
0=000..0
u:4.20..1
0.25
6:13.9x2.4
07.3+0.0
03.0.0.0
0.300...0
0.00
0.25
0.00
0.280.00.00
3135288.6
26.9:14.
n4.42158
0.50
0.25
6:22.5+0.0
6:16.0+5.
0:21 6+9.9
0:7.7+5.3
0.00
0.5 Dataset Fraction liog20NN+
Figure 16: ImageNet extrapolation results.
22

---

Published as a conference paper at ICLR 2020
decathlon_cifar100
100
14.6:114
I-112003
00.6200.
0.0.000.0
1.0
21.620
k3.320.3
4:37720.
IC3. 910.1
03.4
01.7=0.0
0.0
1200.57.2
p:10.620:2
124.320.1
p7.0201
j65 220.1
Model Frection (1ay2mmeM)
0.0
7-7.7771
2o23J200.1
12.2200.1
320.0
920.0
8:5.300.1
0.0
1.8.7.00.0
1--06r0.11
p:.01.0.00
G7.412.6
a20.020.0
m33. 0:0.00
e6.110.0
09.0.0.0
0.0
133.71113
0.5
1.14.1=1.0
a:4.8= 2.6
m9 3+3.0
288.9+4.1
0.8.++5.1
0.0
0.0
0.5
1.0
0.0
0.5
10
0.0
0.5
1.0
0.0
0.5
10
0.0
0.5
1.
Dataset Fraction (log2(rN))-
Figure 17: 100 Extrapolation Results
23

---

Published as a conference paper at ICLR 2020
Srrathlon_aircrase
10
0132=0.
2.4.220.00
52000
8:100
e:17.5e0.1
016
0.14.80..0
00.000..0
1.0
11.82022
1:10.100.0
12..0000.0
20.0
1c300000.00
0:17.5e0.2
0.6.9=0.00
d.15.6+0.0
68.1+0.0
0m0.0
0.0
[213320033
p.12.220.0
[210.820.00
p:50020.0
1c.012200
Model Frection (1ay2mmeM))
e:17.320.3
0.0
2..2200
1110000
112.7000
207.200.0
M-3.9000
0.5
6:15.420.7
0.0
::3.+000
p.1.6.0.00
00001100
0:15.720.2
=15.320.0
14..5...0
m9.710.0
120.0
0.0
0.5
0:13.0+1.5 1.5
a16.3+0..0
1.13.+00.0
04.7+0.0
0.4.+00.0
0.0
0.0
0.5
1.0
0.0
0.5
10
0.0
0.5
1.0
0.0
0.5
10
0.0
0.5
1.0
Dataset Fraction (log2(rN))-
Figure 18: Aircraft extrapolation results.
24

---

Published as a conference paper at ICLR 2020
decathlon_dke _tt
10
#5.6111
82700..0
M3.320.
#2.000.0
0.5.240.9
03.220..0
4:0.0
0.0.0...0
1.0
27.7204
3:3.00000
12.2100.0
0:00.110.0
A00.010.0
0.6.9+0.3
03.4+0.0
6.1.+0..0
000.0
0.0
[27.1204
p.4.220.0
J22.20.0
6-1.00000
je2.12000
Model Frection (1ay2mme0))
0.0
6.0000
pF1720..
11.00.0
110.0
M-302000
0.5
0.0
128.700.0
0.10000.0
.9.210.00
16..0.00
m1. 7:0.0
c. r.00.0.0
120.0
0.0
0.5
.9.3+0..0
0.8.00..0
m1 3+0.0
05.4+0.0
0..3+00.0
0.0
0.0
0.5
1.0
0.0
0.5
10
0.0
0.5
1.0
0.0
0.5
10
0.0
0.5
1.
Dataset Fraction (log2(rN)=
Figure 19; DTD Results
25

---

Published as a conference paper at ICLR 2020
cifar10
1.0
6.0
(336.5813.7
0:33+00.
00.040.0
1.0
PS.7003
03364s11.9
610..a0.1
05 08.3
02.0a0.0
0:0020.00
6.0
1.0
33.3502
0288449.2
11005000
0:1.0
03.6300.0
0:1.200.0
Model Freetion (isghee
D24.155
M85201
0.0
x21.5
38.996.0
m6.3=00.
03.8.0.0
m:1.20.0
233
03.7001
0.0
015.557.3
0...2...1
m2.0.6.0
1.0
22.0262
G442001
0.0
03.026.4
(046.800.0
e12.6a0.2
04.240.0
0:100
018.=21.7
6225
d147.1.
666,500.1
0:3220.11
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Delnsel Practior (lgpi-
Figure 20: CIFARIO extrapolation results.
26

---

Published as a conference paper at ICLR 2020
decathlon_ucf10
100
62
361420.3
1=157=0000
e:50.267.0
0.4.720.3
610.0
0..00...0
1.0
36.126.4
410..
22.0200.
20.0
Ac-13100
0:47.1=7.3
6..8+00.0
0.0.0
0.0
P:33082022
12.0.120.0
p08.220.3 20.3
164. 144.020.1
Model Fraction (1ay2mmeM)
0.0
p2 826.1
1--30000.0
p:l 520.0
020.0
0.5
0.30.9:7.6
0.0
12..5.6..3
p=7.0+0..0
o.22.9:7.3
7:4 620.0
m3. 310.0
a6 26.00.0.0
o2.320.0
0.0
0.5
08.0+2.0
4:49 4+0.5
m8 3+0.0
08.0+0.00
0.8 0.8.++0.0
0.0
0.0
0.5
1.0
0.0
0.5
10
0.0
0.5
1.0
0.0
0.5
10
0.0
0.5
1.0
Dataset Fraction (log2(rN))
Figure 21: UCF101 extrapolation results.
27

---

Published as a conference paper at ICLR 2020
PTB
0:2 920.7
0:0.200.0
010.420.0
0:0.0.0.0
3
ml.100.0
030.9
0:0.6
0000..0.0
1.16600.
M=1.2.0.0
a.0.100.0
P.1000000
Mouel Frection (1a22/maM)
710.7
m1.620.0
011020..0
0:0.820.0
0:0.0
Po11.20000
F.1.00000
520.00
6:0.6
ml 1.0.0
0:1.2+0.0
0.1.100.0
m1.00..0
2-1.111
p.007.00.0
p.0.700.0
R.1.200.0
a:3.1=0.0
e11.7+0.3
5=0.0
e1.55e0.0
3
1.0.200.7
p:300112
6110
5-
62.6+0.6
0.7
10.7
3
4
Dataset Fraction (log2Ir/N)-
Figure 22: PTB extrapolation results.
28

---

Published as a conference paper at ICLR 2020
wiki2
p-10.123.5
3
0:8. 2..6622.
0:0 0.0.0
010.320.0
0:0.0.0.0
28.9
m9 9.000
0:.3000.0
0:0.6
0000..0.0
P=4.522.1
m1.20000
5=1.500.0
Mouel Frection (1a22mmaM)-
711.6
m:5..0.0
01.120.0
0:0.9.0.0
0:0.740.0
P.30000.00
P.1.00000
2:2.4.0.0
65.4+1.4
m3.1:0.0
e:1.5e0.0
0.1...0.0
m1.+0..
P.3.2=2.0
po.42200.0
p18:0.0
R.120..0
a:1.7=0.0
04.9+11.1
03.00000
e1.7+0.0
3
212.1
4.202.0.77
510.8
A:3.J10.3
5-
9+2.0
033
6.26+0.77
3
4
Dataset Fraction (log2/r/N)-
Figure 23: WikiText-2 extrapolation results.
29

---

Published as a conference paper at ICLR 2020
wiki103
1-124422.8
0.7.21.9
9:0.9.0.0
0:1.620.0
0:0.0.0.0
3
04.4+1.4
09 800.0
010.9
e1140..0
m0.0..0.0
p.2.111
p=1.0=0.0
a:40000.0
Mouel Frection (1ag2)muMi-
6
m:1750.0
o:1.2.0.0
0:1.9:0.0
m: 110.
00.7E11
6..0.00.00
0F.0.42..0
04.2+0.4
055 0.0.0
02.0+0..0
0.2.60..0
m2.5+00.0
2..0000.7
p:07=00.
0.21200.0
p.4.120.0
J:5 2.0.0
e...+0.11
m2..000.0
03.0+0.0
0.3.+0.00
8:3.++0.0
3
1..05.010.0
0::00.80.00
4o:10.00.7
22-21420.44
1-13.911
5-
629.7+0.0
a.4.5e0.5
3
Dataset Fraction (log2/r/N)-
Figure 24: WikiText- 103 extrapolation results.
30


Reviews:
[{"review_id": "rkeru_QAYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a functional form for the relationship between <dataset size, model size> and generalization error, and performs an empirical study to validate it. First, it states 5 criteria that such a functional form must take, and proposes one such functional form containing 6 free coefficients that satisfy all these criteria. It then performs a rigorous empirical study consisting of 6 image datasets and 3 text datasets, each with 2 distinct architectures defined at several model scales, and trained with different dataset sizes. This process produces 42-49 data points for each <dataset, architecture> pair, and the 6 coefficients of the proposed functional form are fit to those data points, with < 2% mean deviation in accuracy. It then studies how this functional form performs at extrapolation, and finds that it still performs pretty well, with ~4.5% mean deviation in accuracy, but with additional caveats.\n\nDecision: Accept. This paper states 5 necessary criteria for any functional form for generalization error predictor that jointly considers dataset size and model size, then empirically verifies it with multiple datasets and architectures. These criteria are well justified, and can be used by others to narrow down the search for functions that approximate the generalization error of NNs without access to the true data distribution, which is a significant contribution. The empirical study is carefully done (e.g., taking care to subsample the dataset in a way that preserves the class distribution). I also liked that the paper is candid about its own limitations. A weakness that one might perceive is that the coefficients of the proposed functional form still needs to be fit to 40-ish trained NNs for every dataset and training hyperparameters, but I do not think this should be held against this work, because a generalization error predictor (let alone its functional form) that works for multiple datasets and architecture without training is difficult, and the paper does include several proposals for how this can still be used in practice.\n(Caveat: the use of the envelope function described in equation 5 (page 6) is not something I am familiar with, but seems reasonable.)\n\nIssues to address:\n- Fitting 6 parameters to 42-49 data points raises concerns about overfitting. Consider doing cross validation over those 42-49 data points, and report the mean of deviations computed on the test folds. The extrapolation section did provide evidence that there probably isn't /that/ much overfitting, but cross validation would directly address this concern.\n- In addition, the paper provides the standard deviation for the mean deviations over 100 fits of the function as the measure of its uncertainty, but I suspect that the optimizer converging to different coefficients at different runs isn't the main source of uncertainty. A bigger source of uncertainty is likely due to there being a limited amount of data to fit the coefficients to. Taking the standard deviation over the deviations measured on different folds of the data would be better measure of uncertainty.\n\nMinor issues:\n- Page 8: \"differntiable methods for NAS.\" differentiable is misspelled.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJl_1dRl5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the relation among the generalization error of neural networks and the model and data scales empirically. The topic is interesting, while I was expecting to learn more from the paper, instead of some well-known conclusions. If the paper could provide some guidance for model and data selection, that would be an interesting paper for the ICLR audience.  For instance, how deep should a model be for a classification or regression task? What is the minimum/maximum layers of a deep model? How much data is sufficient for a model to learn? What is the minimum/maximum size of the data set? Do we really need a large data set or just a subset that covers the data distribution? What's the relation between the size of a model and that of a data set? By increasing the depth/width of a neural network, how much new data should be collected for achieving a reasonable performance? How about the gain of the task performance? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BJxakKXRFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a functional form to model the dependence of generalization error on a held-out test set on model and dataset size. The functional form is derived based on empirical observations of the generalizing error for various model and dataset sizes (sections O1, O2, and O3) and on certain necessary criteria (C1, C4 and C5). The parameters of the function are then fit using linear regression on observed data. The authors show that the regressed function \\(\\epsilon(m,n)\\) is able to predict the generalization error for various \\(m\\) and \\(n\\) reasonably accurately.\n\nMajor Points:\n- While the current experiments are a good start, I do not think they are extensive enough to count as strong evidence for the  power-law form of \\(\\epsilon(m,n)\\). I would ideally like to see results on more optimizers, at the very least for Adam, even if for fixed hyper-parameters. As far as I understand, this involves only minor changes in the code since reasonable hyperparameters required for the convergence of Adam have been extensively studied. If the form still holds true then the results from this work can be more reliably used for small-scale network development and in making trade-off choices (as discussed in section 8).\n- Given the current form of the paper, the abstract and introduction should be modified to reflect the fact that only limited architectures and optimizers were experimented with, and the claims of the paper are not experimentally validated in general.\n\nMinor Points:\n \n- It would be nice if more network architectures were analysed (such as VGG and DenseNets). \n- It would be nice if different stopping criteria were analysed.\n- It would greatly benefit the reader if eq. 5 were expanded.\n \nOverall, I think this is a well written paper and provides good insight into the behaviour of the error landscape as a function of model and dataset size. The paper’s primary drawback is the restrictive setting under which the experiments are performed. Therefore, I am not convinced that the power-law form of the generalization error would hold when the experimental settings are marginally different (like when using the Adam optimizer or a VGG-like architecture). I think this work would have much greater impact if the authors can show that the power-law form holds for a larger variety of architectures and optimizers thus allowing researchers to more confidently incorporate the results of this work into the design and training deep neural networks.\n\nRebuttal Response\nI would like to thank the authors for their response. The results of additional experiments as described in Section 6.2 and in Figure 5 do indeed provide stronger evidence of the power-law form of the error function. In light of this, I have changed my original rating. \n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a very interesting idea for estimating the held-out error of deep models as a function of model and data set size. The authors intuit what the shape of the error should be, then they fit the parameters of a function of the desired shape and show that this has predictive power. I find this idea quite refreshing and the paper is well written with good experiments. Please make sure that the final version contains the cross-validation results provided during the rebuttal.
Decision: Accept (Poster)