On the Variance of Neural Network Training with respect to Test Sets and Distributions
Authors: Keller Jordan
Venue: ICLR 2024 poster (2024)
Keywords: stochasticity
distribution shift
randomness
deep learning
variance
random seed
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Neural network trainings are nondeterministic; this paper presents and experimentally justifies a statistical model for the variation between independent runs of training.

Abstract: Neural network trainings are stochastic, causing the performance of trained networks to vary across repeated runs of training.
We contribute the following results towards understanding this variation.
(1) Despite having significant variance on their test-sets, we demonstrate that standard CIFAR-10 and ImageNet trainings have little variance in their performance on the test-distributions from which their test-sets are sampled.
(2) We introduce the independent errors assumption and show that it suffices to recover the structure and variance of the empirical accuracy distribution across repeated runs of training.
(3) We prove that test-set variance is unavoidable given the observation that ensembles of identically trained networks are calibrated (Jiang et al., 2021), and demonstrate that the variance of binary classification trainings closely follows a simple formula based on the error rate and number of test examples.
(4) We conduct preliminary studies of data augmentation, learning rate, finetuning instability and distribution-shift through the lens of variance between runs.

Published as a conference paper at ICLR 2024
ON THE VARIANCE OF NEURAL NETWORK TRAINING WITH RESPECT TO TEST SETS AND DISTRIBUTIONS
Keller Jordan" Independent researcher
ABSTRACT
Neural network trainings are stochastic, causing the performance of trained net- works to vary across repeated runs of training. We contribute the following results towards understanding this variation. (1) Despite having significant variance on their test-sets, we demonstrate that standard CIFAR-10 and ImageNet trainings have little variance in their performance on the test-distributions from which their test-sets are sampled.
(2) We introduce the independent errors assumption and show that it suffices to recover the structure and variance of the empirical accuracy distribution across repeated runs of training. (3) We prove that test-set variance is unavoidable given the observation that ensembles identically trained networks calibrated (Jiang al., (021), and demonstrate that the variance of binary classification trainings closely follows a simple formula based on the error rate and number of test examples. (4) We conduct preliminary studies of data augmen- tation, learning rate, finetuning instability and distribution-shift through the lens of variance between runs.
INTRODUCTION
Modern neural networks (Krizhevsky et al. 2012; He et al. 20166 Vaswani et al. (017)) are trained
using stochastic gradient- based algorithms, involving randomized weight initialization, data/batch ordering, and data augmentations. Because of this stochasticity, each independent run of training produces different network which may have better or worse performance than the average.
This variance between independent runs of training is often substantial. Picard (2021) shows that for a standard CIFAR-10 (Krizhevsky et al., (0099) training configuration, there exist random seeds which differ by 1.3% in terms of test-set accuracy. In comparison, the gap between the top two methods competing for state-of-the-art on CIFAR-1O has been less than 1% throughout the major- ity of the benchmark's 's lifetime"] Prior works therefore view this variance as an obstacle which
impedes both comparisons between training configurations (Bouthillier et al. 2021 Picard 2021)
and training pipeline reproducibility (Bhojanapalli al.) [202]] Zhuang al. 2022) To mitigate
stochasticity, Zhuang et al. (2022) study deterministic tooling, Bhojanapalli et al. (2021) develop
regularization methods, and many recent works (Wightman al. [022] Liu al. 2022 report the
average of validation metrics across multiple runs when comparing training configurations.
This paper contributes the following results towards understanding the structure and origin of vari- ance in stochastic neural network training.
1. We demonstrate that random seeds which are "lucky" with respect to split of test data perform no better than average on a second split. (Section 3.1)
2. We introduce the independent errors assumption, and show that it recovers the structure and quantitative variance of the empirical test-set accuracy distribution. (Section 3.2
3. We present formula for estimating the variance of the distribution-wise accuracy using only statistics observed on finite-sample test-set. (Section 3.3)
4. We prove that the class-calibres calibration property (Tiang et al. 1021 of neural network train- ings implies finite-sample variance, and derive from it a formula which accurately predicts variance using only a single run of training for the binary classification case. (Section 3.4
Work performed while at Hive AI
https://paperswii wwwoweewwwwwwhh.occ.mo.mmo
aasi on

---

Published as a conference paper at ICLR 2024
We define the distribution- variance of training algorithm to be the variance between runs of its performance on the test distribution from which the test-set is sampled. Although this quantity cannot be directly calculated from a finite test-set, we develop a formula (Equation[3) which provides an unbiased estimate of it using statistics collected on a test-set. Using the formula, we estimate that despite the fact that standard CIFAR-10 1-00 training has relatively large test-set standard deviation of 0.15%. its distribution-wis wise standard deviation is only 0.03%. We find similar results for a standard ImageNet (Deng et al., 2009) training (Section C). We conclude that the distribution-wir variance of standard neural network trainings is significantly less than their test-set variance.
To understand the reason why standard trainings have more variance on their finite test-sets than on their test- dest-istributions. we turn to the class caasscaalibatiion property discovered by Jiang et al. (2021). We first prove mathematically that although this property generates no constraint on the distribution- wise variance, it does imply lower-bound the test-set variance (Theorem 4. We then empirically demonstrate that this lower bound closely matches the test-set variance across hundreds of binary classification trainings (Figure This finding yields a tool for estimating variance without the need for multiple runs of training. which is more accurate than the commonly used binomial approxima- tion (Dietterich 1998 Raschka 2018).
Our experiments and conclusions focus on standard CIFAR-10 and ImageNet trainings. We also investigate two exceptional scenarios where conclusions do not fully apply: trainings where there is a distribution shift between the training and test distributions (Section C. and trainings with pathological instability (Section[4.1) Both of these cases have significant distribution-wise variance, unlike the standard training scenarios.
Finally, we conduct preliminary studies regarding the effect of learning rate (Section 4.3 and data augmentations (Section 4.2) on variance. When increasing the learning rate, accuracy begins to de- cline at the same point at which sigmificant distribution wise variance appears. Data augmentations significantly reduce variance.
1.1 RELATED WORK
A number of prior works investigate which sources of stochasticity are most responsible for variation between runs of training. Fort et al. (2019) observe that when using a below-optimal learning rate, randomized data ordering has a smaller impact than model initialization on the churn of predictions between runs. [hhojanpaali et al. (202T) similarly find that fixing the data ordering has no effect, while fixing the model initialization reduces churn. Bouthillier al. (202T) report that data ordering has a larger impact than model initialization. Finally./uummers Summmers & Dinneen (202T) find instead that
most variation can be attributed to the high sensitivity of the training process to initial conditions, with a single bit difference in starting parameters being sufficient to lead to the full quantity of
disagreement between runs. We include a replication study of Summmers & Dinneen (021) in the appendix (Section[]
Dodge et al. (2020) study variation between runs of BERTLARGE finetuning, and achieve substantial gains validation performance via the strategy of re-running finetuning many times and taking the best-performing result. We demonstrate (Section [.1 that for the case of BERT BART. BaSe, the low amount of genuine distribution-wise variance between runs indicates that any performance gains yielded by this strategy only amount to overfitting the validation set. On the other hand, for BERTLARD demonstrate that there is significant distribution-wiss variance, supporting the use of multiple runs of training as Dodge et al. (2020) suggest. MMsbach et al. (2020) also study the finetuning instability of BERTLARGE: and suggest to mitigate it by warming up the learning rate, training for longer with smaller learning rate, and using bias correction for Adam (Kingma & Ba 2014).
we
Many of our theoretical results draw upon the class-calibration property discovered by Jiang et al. (221)) who use it to provide a theoretical proof for their empirical observation that the rate of dis- agreement between two identically trained networks matches the error rate of each network. This builds on earlier works studying neural network ensembles, which reported that they are more well- calibrated (Lakshminarayanall et al. 2017 Nixon et al. 2020) and achieve higher accuracy under distribution shift (Ovadia et al. (2019) compared to individual networks. Also related is the observa- tion of Mukhoti et al. (2021) that the usefulness of an ensemble's uncertainty scores depends upon the variance between the individual networks.
2

---

Published as a conference paper at ICLR 2024
CIFAR-10 trainings
0 epochs
4 epochs
16 epochs
64 epochs
6%
8%
10%
12%
14%
78%
80%
92%
94%
Test-set accuracy
Figure| 1: Accuracy distributions. The test-set accuracy distributions across our four training durations, displayed unsmoothed histograms for 60,000 repeated runs of training each. The differences between the "luckiest" and most unlucky run (max minus min accuracy) 13.2%. 6.6%. 1.7%. and 1.4% for the 0, 4. 16, 64-epoch training durations, respectively. standard deviations are 1.87%, 0.56%, 0.19%, 0.15%.
Several prior works (Baldock et al., 2021, Ilass et al. 2022, Lin et al. 2022) study the effect of randomly varying the data used to train neural network, across large number of training runs. Our study differs from these works in that we consider the simpler scenario of running a single training algorithm many times without varying anything except the training seed.
Broadly, our work is related to research aiming to understand the relationship between pairs of neural networks produced by repeated
of training. This topic is of both theoretical and practical inter- est, and has been studied from variety of angles, including the similarity of internal representations (Li et al., [0155] Kornblith al.) 2211)) degree of correlation between predictions (Fort 2019 Jiang et al. 2021), similarity of decision boundaries (Somepalli et al. 2022), path-connectivity in weight-space (Draxler al.) 2018 Garipov al.) 2018), linear mode connectivity (Frankle al.,
2020). and linear mode connectivity modulo permutation symmetries (Tatro et al. 2020 Entezari
ct al. 2021 Ainsworth et al. 2022 Jordan et al. 2022).
SETUP
Notation. This work studies neural networks trained to solve classification problems. We as- sume test-set examples are sampled independently from distribution D over X' x y where y kl the set of classes and X' is the input space. When we refer to a training al- gorithm configuration,
understand it to include everything necessary for training besides the random seed. That is, a training configuration already includes the choice algorithm, dataset, network architecture, and hyperparameters, so that only the random seed mains to determine the outcome of training. Let 7. be the hypothesis class of all functions of the form h
optimization
+ V. Following Jiang al. (2021). 1), for a stochastic training algorithm we write h ~ HA to denote hypothesis sampled from the distribution induced by the algorithm, iee. by running the algorithm and collecting the trained network. We write eTTz,y (h) 1(h(x) * y} to denote the condition that the neural network h makes an еттог on the example (I, 8). so that En-HA ( (err
the proportion of runs of training which make an error on (æ, y). We addi- tionally write err (h) = ((r.))> jerra,y (h)] to denote the true error rate on the test distribution, so that the distribution-wer wise variance is Var (crr(h)). For test-set 5 = ((Z1, Vi).... ...., (In, yn)) we write errs(h)
Σ crT... (h) to denote the test-set error. Finally, we write (AA) En~Ha BawHa/err(hh (err(h) to denote the average true eтrоr of the training algorithm.
Core experimental setup. For our core experiments (Section 3}. we train ResNets on CIFAR- - We study four different training durations: O, 4, 16, and 64 epochs. The 0-epoch case corresponds to evaluating the network at initialization; this naturally has random chance-level average accuracy of 10%, but some random initializations reach high 14% and as low as 6% accuracy. A complete description of each training configuration is provided in Section A We execute each configuration 60,000 times and collect the resulting test-set predictions, yielding the accuracy distributions shown in Figure

---

Published as a conference paper at ICLR 2024
Epochs=0
Epochs=4
Epochs=16
Epochs=64
Accuracy (odd examples)
01%
16%
22.5%
80%
92.0%
95.0%
12%
79%
78%
91.5%
94.5%
77%
0%
4%
476
8%
12%
16%
77% 78% 79% 80% 81%
90.5% 91.0% 91.5% 92.0% Corre ation 6%
93.5% 94.0% 94.5% 95.0% Corre stion 3.1%
Correlation 95.4%
Correlation 74.6%
Accuracy (even examples]
Figure 2: Error rates on disjoint splits of test data become decorrelated when training to convergence. We evaluate a large number of independently trained networks on two splits of the CIFAR-10 test-set. When mader-training there is substantial correlation, that "lucky" run which over-performs the first split is also likely to achieve higher-than-aa than-gee accuracy on the second. we increase training duration, the two error rates decorrelate from each other.
DELVING INTO VARIANCE
3.1 Do LUCKY RANDOM SEEDS GENERALIZE?
In Figure I Twe observed that our standard CIFAR-10 training configuration has significant variation between runs. Even when training for a long duration, we found pairs of random seeds which produce trained networks whose test-set accuracy differs by more than 1 1%. In this section, argue that this variance is merely a form of finite-sample noise caused by the limited size of the test-set, and does not imply almost any genuine fluctuation in the quality of the trained network.
Suppose we view the random seed as a training hyperparameter, Then we have observed that it can be effectively "tuned" obtain improved performance on the test-set on average, our training configuration attains an accuracy of 94.42%, but we found random seeds which reach above 95%, which is more than 10% fewer errors. However, this improvement on the test-set alone is not enough to conclude that the random seed genuinely affects model quality. What remains to be seen is whether these performance improvements can generalize unseen data, or we are merely over- fitting the random seed to the observed test-set.
To find out, we perform the following experiment. First, we split the CIFAR-10 test-set into two halves of 5,000 examples each. CIFAR-10) is already shuffled,, so for convenience we simply use the odd and even-indexed examples as the two halves. We can view the first half as the hyperparameter- validation split and second as the held-out test split. Next, training, with identical configurations other than the varying random seed. We measure the perfor- mance of each trained network both splits of data. If lucky random seeds do generalize, then should observe that runs of training which perform well on the first split also perform better than average the second split.
execute many independent runs of
To additionally determine the effect of training duration, we repeat this experiment for trainings of 4, 16. and 64 epochs, using 60,000 independently trained networks for each duration. We view the results in Figure 2 For short trainings, the two splits are indeed highly correlated, such that runs which perform well on the first split also tend to do well on the second. But when training for longer, this correlation nearly disappears. When training for 64 epochs, for example, our highest- performing network on the first split does not even perform better than average on the second split. And on average, the top 1 1/4 of runs with respect to the first split only perform 0.02% better than average on the second split.
This result has the following practical implication. Suppose we want to obtain good CIFAR-10 model. Noticing significant variation between runs (Figure [ we might be tempted to re-run train- ing many times, in order to obtain networks with better test-set performance. However, according to Figure 2 this would be useless, because improvements on the test-set due to re-training will have near-zero correlation with improvements unseen data. These networks would be "better" only in the sense of attaining higher test-set accuracy, but not in the sense of being more accurate on unseen data from the same distribution.
4

---

Published as a conference paper at ICLR 2024
Epochs=0
Epochs=4
CIFAR-10
Empirical Hypothesis
1.2%
Empirical
Hypothesis
Text Test-set ser--set accuracy stdl
1.0%
0.8%
8%
12%
16%
78%
79%
Epochs=16 16
Epochs=64
0.6%
0.4%
0.2%
0.0%
91.0% 91.5% lest-Set Accuracy
92.0%
94.0%
94.5%
95.0%
4 8 16 32 128256 Training epochs
Independent errors explain variance when training to convergence. (Left:) We compare the em- pirical distribution of test-set accuracy with that generated by simulated equal number samples assuming the hypothesis of independent error.. The assumption wrong for short trainings, but becomes close fit as training progresses. (Right:) The assumption of independent errors accurately predicts the variance between runs of test-set accuracy when training to convergence.
Figure 3:
3.2 INDEPENDENT ERRORS
In the previous section we showed that when training to convergence, disjoint splits of test data become nearly decorrelated, in the sense that networks which randomly perform well split do not perform better than average on another. We now formalize an even stronger condition, which states that independent runs of training make errors which vary independently on each example.
Definition 1. The training algorithm makes independent errors test-set for every pair of test examples (Ii, Us) and (I Wi).
Cov (eIT (h), eTT u, (h)) = 0 h~HA
(1)
We demonstrate this condition in Figure 4 The
error on each test example (I,, yt). as func- tion of training stochasticity, is Bernoulli vari- able with mean E; = E, (err (h)]. The test- set error errs(h) is the average of such Bernoulli variables. If HA makes independent errors, then each Bernoulli variable is independent, so that the overall test-set error must have variance equal to 1 Ee Vara (h)). On the other hand, if the independent emrors property is violated, then variance may be higher.
Figure 4: A pair with independent errors. (Lefi) is image 776 of the CIFAR-10 test-set. Out of 64-epoch training.
of 60,000 independent 21,736 networks (36 (6.2%) correctly predict this ex- ample. (Right) is image 796, which is correctly predicted by 36,392 networks (60.7%). The num- ber of networks which predict both correctly the
time is 13,103 (21.83%), which has statistically insignificant difference to quantity 0.362 0.607 = 21. 21.97%, which is the expectation if their errors are independent.
(h)).
We next compare the distribution of accuracy-values that we observe in reality to what would be pre- dicted by independent errors. Using our large collection of trained networks, we compute estimates of $1,...,610, for our four training configurations. From these estimates, using the assumption of independent errors, we both (a) compute predictions of test-set accuracy variance using the for- mula / Σi= (1 (q), and (b) simulate samples of the test-set accuracy distribution. In Figure we demonstrate that both of these become close fit with reality as we train to convergence. For short trainings, the real accuracy distribution contains excess variance which is unexplained by in- dependent errors, but for long trainings this excess disappears, so that the hypothesis becomes true in the aggregate sense of predicting the shape and standard deviation of the accuracy distribution. Additionally, in Figure 14 we find that only a small number of visually similar pairs violate the assumption of independent emrors. and in Figure 13 we show that it compares favorably to the bino- mial assumption. In the next section we explore how small deviations from the independent errors assumption can be used to estimate variance with respect to the test-distribution.
5

---

Published as a conference paper at ICLR 2024
3.3 ESTIMATING DISTRIBUTION WISE VARIANCE
we showed that accuracy is decorrelated between disjoint splits of test-data, and argued that this implies small genuine variation in model quality between runs of training. this section we clarify our notion of model quality, and present a method of directly estimating the true variance between runs, on the full test-distribution rather than a finite test-set.
In Section 3.1
Neural networks are typically evaluated by their performance a test-set. However, what really matters is performance on the test-distribution D, because this what determines the expected performance on new batches of unseen data. Therefore, our notion of model quality is based on the distribution-wii error err(h) = B( (x(() * vHI, which we call the true error. The test-set is finite sample from D, that test-set error is noisy binomial approximation of true error.
Estimating the mean true error across train- ing stochasticity is relatively easy, because test- set accuracy is an unbiased estimator, as we have
CIFAR-10
1.0%
Es-D" [ (h)] = Ex [err(h)]. Estimating the variance a² := Vara (err(h) is more challenging,
Test-set (observed)
Accuracy stddev
Test distribution (estimated)
since the variance test-set accuracy an overesti- mate (proof in Section B.I).
0.6%
Theorem 1. In expectation, variance in test-set ac- curacy overestimates variance true error.
0.4%
0.2%
E SNDD"
Var (err)(())) A-HA
2
Var (err(h)) h-HA
(2)
0.0%
2
32 Training epochs
16
256
We investigate how to obtain unbiased estimate of the true variance Var (err(h)). We recall the results of Section 3.2 When training to conver- gence, we found that test-set accuracy follows a dis- tribution which assuming that all test examples have independent er- rors (Definition [ Figure 3). The distribution-wise variance in this case should be essentially zero. On the other hand, for shorter trainings we observed substantial test-set variance in excess of that predicted by Definition [ i.e., Varh~ (errs (h)) + E Var, (erTz..g. (h)). For example, the hypothesis predicted that our 4-epoch configuration should have a standard deviation of 0.22%, but the observed value was much larger at around 0.56%.
Figure 5: Test-set variance overestimates true variance. We use Equation estimate the true distribution-ve wise variance Vara-w, (err(h)). becomes 22 x smaller than the test-set variance when training to convergence.
be approximately recovered by
This suggests that these shorter trainings may have significant true variance between runs.
To formally connect the variance in true error to the excess in test-set variance over that predicted by the assumption of independent errors, we provide the following theorem (proof Section B.2
Theorem 2. The following quantity is an unbiased estimator for true variance Varh~H (err(h)) (err(l))))r(
05
11 n 1
Var (crTs(h)) AwHA
Σ
Var (eIT (h))
(3)
n
h~H
We calculate this estimate using many of training, with the fixed CIFAR-10 test-set. In Fig- ure 5 we compare à² to the test-set variance Var (h)r across a range of training durations. When training for 4 epochs, we estimate that the standard deviation of true error is / = 0.52%, indicating significant differences model quality between trainings of this duration. In contrast, when training for 64 epochs, estimate that the true standard deviation is only 0.033%. In compar- ison, that configuration's test-set standard deviation is 0. 149%. We obtain a similarly small estimate for ImageNet trainings in Section These findings indicate that when training convergence, there little variation in model quality (i.e., expected performance with respect to new batches of data from the cist-istributtion) between runs of training.
Having confirmed that true distribution-we wise variance is small, it still remains to explain why there high variance on the finite test-set in the first place. We investigate this in the next section.

---

Published as a conference paper at ICLR 2024
3.4 CALIBRATION IMPLIES PEINIEE-SAMPLS VARIATION
In this section we prove that variance in test-set accuracy between runs of training is an inevitable consequence of the observation that ensembles of trained networks well-calibrated (Tiang al. 2021). Our analysis leads us to a simple formula which accurately estimates variance for binary classification problems.
Classical machine learning algorithms based on convex loss functions can have neither test-sct nor test-distribution variance, because training always converges to a single global optimum. As an example, we show in Figure 7 (left) that repeated runs of training of a regularized linear model on CIFAR-10 leads below 0.01% % variance in test-set accuracy. On the other hand, neural networks have many optima (Auer et al. 1995; Choromanska et al. 2015), so that every run of training can potentially lead different solution. Despite this theoretical property, the previous section we showed that the variance in true emror between runs of training is in fact quite small. What then explains the significant variance in test-set error?
We show that the following property, which Jiang et al. 2021 demonstrated approximately holds for neural network trainings in practice, is connected to their high variance on test-sets.
Definition The training algorithm A satisfies class-wise calibration (Jiang et al. 202/ if for every class E y and confidence level q E 1].
P( = Ph-HH (h(x) =c) =c)= -))=)=1 =
(4)
As an explanatory example, if we let S' C S be the subset of test images which are classified by 30-40% of independently trained neural networks "cat," then 30-40% of S' really will be cats. Theorem 3. Given stochastic learning algorithm A for binary classification, if it is class-wise calibrated, then its expected variance on a lest-sel of size n is
err(A)
E SNID
Var (errs(())) A~HA
+ (1 (1- Var (err(h)) hoHHA
(5)
2n
Proof is provided in Section tion we showed that the true variance Vara (crr(h)) is small in practice. Therefore, the above theorem practically reduces to the following simple formula:
B.3 In the previous sec-
CIFAR-10 binary classification tasks
0.20%
Es Var (errs(h)) As err(A)/2n
(6)
0.18%
In Figure 6 we use this to predict the variance in test- set accuracy across 511 different binary classifica- tion tasks derived from CIFAR-10, 10, where substi- tute the average test-set errors as a cheap approxima- tion the error (rr(A). It good empirical fit, with R² = 0.996 across the collection of tasks. We also compare the commonly used binomial as- sumption (Dietterich] 1998 Raschka 2018), which yields a larger variance estimate of crr(1 c /n. The formula based on class-wise calibration gives variance estimates which are 70.5x more accurate than those based on the binomial assumption, in terms of their average squared distance to the em- pirical variance.
Predicted stedev
0.14%
0.12%
0.10%
0.08%
0.06%
0.04%
Verri emri (binomial assumption)
0.02%
Vertinn (class-wise calibration)
0.04% 0.06% 0.08% 0.10% 0.12% 0.14% Observed stddev between runs
We additionally provide the following lower bound for general multiclass classification.
Figure Predicting test-set variance. Across hundreds of tasks, the formula given by class-wise calibration accurately predicts the standard devia- tion of test-set error. In contrast, the formula given by the binomial assumption inaccurate.
Theorem 4. Given a learning algorithm A for k- way classification, if class-wise calibrated, then its expected variance on test-set of size is
(AT(A)
I8 SDDD
Var (errs(h)) A~HA
2
nk

---

Published as a conference paper at ICLR 2024
CIFAR Linear model
CIFAR-10
BERT
BERT
0.57
ning daca
0.90
1est
080
Augitest
0.75
Reghtesst top 1555
41.3 50
41.4%
43.5%
00%
57%
01%
0.56
2.57
0.85 0.50
0.90
0.75
0.50
0.85
0.30
scuracy
Validation
Figure 7: (Far left:) Training regularized linear model has very little variance between runs. (Center left:) Removing either data augmentation, or 80% of training data from CIFAR-10 training, reduces the average accuracy to around 87.5%. But the former produces much more variance between runs than the latter. (Right two:) When finetuning BERT BASe MRPC, variations between runs in terms of performance the validation and test sets are not strongly correlated. On the other hand, BERTLARGE has significant true instability.
Proof is provided in Section B.4 Although it is still largely a mystery why neural network trainings satisfy class-wise calibration, together these theorems show that their variance on finite test-sets an inevitable consequence of that property.
ADDITIONAL EXPERIMENTS
In this section we apply the tools developed in the preceding sections to BERT finetuning, and conduct preliminary investigations of the effect on data augmentation, learning rate, and distribution- shift on variance. We additionally include replication study of [Summers & Dinneen (2021) in Section D which confirms their findings that variance is caused by extreme sensitivity to initial conditions rather than any particular stochastic factor like the initialization data ordering.
4.1 BERT FINETUNING
In this section we study BERT (Devlin et al. 2018) finetuning, where previous works have reported significant variance between runs (Devlin et al. 2018 Dodge et al. 2020 Mosbach et al.) 2020). Our contribution is to use the tools we developed in Section 3 to clearly differentiate the behavior of BERT LARGE from BERTBASE- arguing that although both models exhibit high variance in their test-set error rates, only the former has high variance in its true eгror rate.
For experiment, we finetune pretrained checkpoints of both models 1,000 times each the MRPC (Dolan & Brockett 2005) task. The test-set error rate of BERTBASE has a standard deviation of 0.80% between runs of finetuning, and BERTLARGE has stddev of 2.24%. In Figure 7 (right) we show that for BERT the validation and test splits of MRPC are close to decorrelated in terms of finetuned model performance, recalling Section 3.1 In particular, the top 15% of seeds in terms of validation-set performance achieve only 0.09% higher performance than average on the test-set. This observation is reinforced by the use of Equation 3 which computes a distribution- wise stddev of only 0.21 % for BERT compared to 2.08% for BERTIARGE, where there is clear correlation. We therefore conclude that, despite both models appearing to have high test-set variance, in fact only BERTLARGE has substantial variance in its true error rate. Our contribution ends at making this distinction; we do not speculate as to the reasons underlying this instability.
4.2 THE EFFECT OF DATA AUGMENTATION
In this section we look at the effect of data augmentation variance. In Figure 7 (center left) we compare two ablations: first, removing fixed 80% of training data, and second. removing data aug- mentation. While both configurations achieve similar mean accuracy of 87.5%. the augmentation- free training has over 3.5x more variance between runs. We also observe that the ensemble accuracy of the augmentation-fee free networks is higher, reaching 91.2%, compared to the reduced-dic ensem- ble, which achieves only 89.8%. Based on these observations, we speculate that one role of data augmentation may be reduce variance between runs.
8

---

Published as a conference paper at ICLR 2024
100%
-IF 64 epochs
0.8%
Observed Hypothesis
Accuracy stddev
Accuracy mean
0.6%
95%
0.4%
90%
0.2%
85%
0%
2 10
2-8
2 2-4 0.25 1.0 4.0
2-10
2
0.25
1.0
4.0
Learning rate
Learning rate
Figure 8: Accuracy is maximized by the largest learning rate without excess variance. Across learning rates, we plot the ohserved stddev of test-set accuracy and that predicted by the independent errors assumption. We observe that the best learning rate is the largest one which does not induce significant excess variance.
4.3 THE EFFECT OF LEARNING RATE
In this section we investigate the relationship between learning rate and variance. Our experiment execute 1,000 64-epoch CIFAR-IO trainings for each binary-power learning rate between 2-10 10 and 22. For each setting, we measure the mean and variance of test-set accuracy. We observe that the learning causes the standard deviation of test-set accuracy to increase from 0. 0.148% to O. 0616.. This may seem insignificant, but Equation estimates that it coincides with 5x increase in distribution-wer wise variance. We therefore conjecture that, as general property of neural network trainings, the optimal learning rate the largest one which does not induce significant distribution wise variance.
0.5 yields both the highest mean and the lowest variance. Raising it to 1.0
4.4 THE EFFECT OF DISTRIBUTION SHIFT
In this subsection we summarize our results on distribution shift which are fully described in Sec- tion C Our experimental setup is to train 1,000 ResNets on ImageNet (with identical hyperpa- rameters), and then use Equation to estimate the distribution variance on the validation set, ImageNet-V2 (Recht et al. 2019). and three distribution-shifted sets. We find that the main valida- tion set and ImageNet-V2 have standard deviation at or below 0.07%, while the three distribution- shifted sets all have at least 6x more variance. That is, distribution wise variance is large for pre- cisely those test-sets which have a shifted distribution relative to the training set.
DISCUSSION
central focus of paper was the distinction between model's observed error rate on test-set, and its true error rate on the test-distribution. We showed that although the true error rate is not directly accessible (since we can't sample an infinite number of test examples), it is nevertheless possible to acquire an unbiased estimate of its variance via Equation 3 And we found that for standard trainings, this quantity is quite small, with the standard deviation of true error being only around 0.03% for both CIFAR-10 and ImageNet. Our takeaway is that for standard trainings, even though some random seeds lead to higher or lower performance on the test-set, they are all nearly equal on the test-distribution.
However, we found two exceptions to this takeaway. The first fairly obvious exception is trainings which have pathological instability, such as BERT finetuning where the accuracy can vary by more than 15% (Section 4.1 Dodge et al. (2020). The second more important exception trainings whose test distribution shifted relative to the training distribution, for which case observed many times more variance (Section C). Understanding why variance appears alongside distribution shift is task whose solution we look forward to in future work.

---

Published as a conference paper at ICLR 2024
REFERENCES
Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.00 04836, 2022.
Peter Auer, Mark Herbster, and Manfred KK Warmuth. Exponentially many local minima for single neurons. Advances in neural information processing systems, 8, 1995.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 24:10876-10889, 2021.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang. Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Srinadh Bhojanapalli, Kimberly Wilber, Andreas Veit, Ankit Singh Rawat, Seungyeon Kim, Aditya Menon, and Sanjiv Kumar. On the reproducibility of neural network predictions. arXiv preprint arXiv:2102.03344, 2021.
Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et Ac- counting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems, 3:747-769, 2021.
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pp. 192-204, PMIR, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. IEEE, 2009.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O): 8-bit matrix multiplication for transformers at scale. arXiv preprint (rXXi:2208.073399 2022.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 2018.
Terrance DeVsi and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 170.:0470000455 2017.
Thomas G Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, (177):8955-1223 1998.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint ar arXiv:2002.06305, 2020.
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (/WP2005), 2005.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp. 1309- 1318. PMLR, 2018
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296, 2021.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv preprint 19Xiv:1912.000 2019.
10

---

Published as a conference paper at ICLR 2024
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode nectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp- 3259-3269 PMLR, 2020.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information pro- cexsing systems, 31, 2018.
G. H. Hardy, J. E. Littlewood, and G. Pólya. Inqqualities. Cambridge Mathematical Library. Cam- bridge University Press, Cambridge, 1934.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. $340-8349, 2021.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Data- models: Predicting predictions from training data, arXiv preprint arXiv:2202.00622 2022.
Yiding Jiang, Vaishnavh Nagarajan, Christina Back, and Zico Kolter. Assessing generalization of sgd via disagreement. arXiv preprint arXiv:2106. 13799. 2021.
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renor- malizing permuted activations for interpolation repair. arXiv preprint arXi::2211.08403 2022.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980. 2014.
Simon Kornblihh, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 and cifar-10 (canadian institute for advanced research), 2009. URL http: http://www.......toooo...o MIT License.
CS. www.ttoroooooooooo edw wot.oott...edd///rr///idiii..... html
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2012.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems, 30, 2017.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. ffev.
com http:///llffee/ff/////ee//// 2022.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hoperoft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint arXiv:1511.0733 2015.
Jinkun Lin, Anqi Zhang, Mathias Lécuyer, Jinyang Li, Aurojit Panda, and Siddhartha Sen. Mea- suring the effect of training data on deep learning predictions via randomized experiments. In International Conference Machine Learning, pp. 13568-13504. PMLR, 2022.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976-11986. 2022.
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines, arXiv preprint arXiv:2006 2088048044488444 2020.

---

Published as a conference paper at ICLR 2024
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deep deter- ministic uncertainty: A simple baseline. arXiv preprint arXiv:2102.11582, 2021.
Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran. Why are bootstrapped deep ensembles not better? In "I Can't Believe It's Not
2020 workshop, 2020.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
David Page. How to train your resnet 4: Architecture, 2019. URL https::/myrt.e.aat
David Picard. Torch. manual -seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision. arXiv preprint arXiv:2109.0820) 2021.
Sebastian Raschka. Model evaluation, model selection, and algorithm selection machine learning- arXiv preprint arXiv: 181 12808 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-5400 PMLR, 2019.
Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Yeh-Chiang, Yehuda Dar, Richard Baraniuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13699-13708, 2022.
Cecilia Summers and Michael Dinneen. Nondeterminism and instability in neural network opti- mization. In International Conference on Machine Learning, pp. 9913-9922. PMLR, 2021.
Norman Tatro, Pin- Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Op- timizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 1300003111 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Haohan Wang, Songwei Gc, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476, 2021.
Donglin Zhuang, Xingyao Zhang, Shuaiwen Song, and Sara Hooker. Randomness in neural network training: Characterizing the impact of tooling. Proceedings of Machine Learning and Systems, 4: 316-336, 2022.
12

---

Published as a conference paper at ICLR 2024
A TRAINING HYPERPARAMETERS
A.1 CIFAR-10
For our core experiments we train a ResNet on CIFAR-10. Our network architecture is the same as was used in Ilyys et al. (2022); namely, a 9-layer ResNet originally derived from Page (2019). Our O-epoch configuration corresponds to a randomly initialized network. Our 4, 16, and 64-epoch configurations
train using SGD with a learning rate of 0.5, a momentum of 0.9, and a weight decay of 5e-4, with the learning rate linearly ramped down to zero by the end of training. We train using random flipping, 2-pixel translation, and 12-pixel Cutout (DeVrics & Taylor 2017) data
augmentations. We use batch size 500 and load data using the FFCV (Leclere al. (022) li- brary. Our training script is publicly available at ffev-ci far/ba ob/ master/aaatammmm eeeettriinn.neeeeeeeeeeiaeeeaiat py The 64-epoch configuration attains average curacy of 94,42% without any test-time augmentation. We execute each configuration 60,000 times, generating 240,000 sets of test-set predictions, which form our object of study for Section
com/ KellerJordan/
A.2 IMAGENET
For our ImageNet experiments we train ResNet-18s using standard random flip and random resized crop data augmentations. We train resolution 192 for 100 epochs with batch size 1024, using SGD-momentum with learning rate 0.5, momentum 0.9, and weight decay 5e-5. We linearly ramp the learning rate up from 5e-5 0.5 by epoch 2, and then down zero by the end of training. We use the FFCV (Leclere et al. 2022) dataloader here as well.
A.3 BERT FINETUNING
We finetune BERTBASE and BERTLARGE on the MRPC dataset. We train for 3 epochs batch size 16, using Adam (Kingma & Ba 2014) with default hyperparameters other than the learning rate, which is linearly annealed from maximum of 2e-5 down zero by the end of training.
B PRooFs
We first prove the following two lemmas, using the notation of Section 2
Lemma 1.
The variance of the overall error rate is equal to the expected covariance between error rates pair of examples.
Var (err(h)) hwHA
E
Cov (err.1.91 (h), err er (h)) AmHA
13

---

Published as a conference paper at ICLR 2024
Proof.
Var (err(hi)) [(rr((h) Rn [e(eer(h))))]] b~HA
A-Ha
27
IE
lerr,,, (h)]
E
lerra (1:]]
x.y
27
IE
Jerr.,y (h)]
E
[E y. (hi]l
h
z.y
IE
lerr (h)
err s. (h)]]
h
IE
(erT 21 (h) 18 err (h)]]
(erT (h)
E alrr (b)]]
h 11.2//1
12.1/2
h
E
E
(erTz, El (h)] (h) Elerr (h)])
h 21.81 #2:82
E
(eITzI.1 (h) Elerr um (erose eer (h)))
21.81 22.222 h
Cov (err (h), err erTe (h)) uwHHA
E (1.
Lemma 2. For an IID test-set S = ((x1, (u),...,(ze (Fa, Un)), the expected variance in the test error rate can be decomposed into a mixture of distribution-wis and example-wise variances.
Var (erTs(hi)) A~HH
=(1 (1-1//n)- Var (err(h)) (1/n)- A-H
Var (eIr (h))
SnDD
(r.N)/ h-HH
Proof.
Var (errs(1)) E Var 1 Σ M
E
er er (h)
S~ID"
1=1
Cov an .r (h), a (h))
Cov (h), eTT (hi)
n2
à
Var(erra, (h))
+
E
Cov (h), err (1))
2
n
1=1
j
E seath
Var(err (hi)
+
Cov(ertanor (h), erra, uss (h))
n12
(mt (13))
i=
jæi
+ n(n-1
E
Var(errz (A))
E (.).2.22
Co (h), eIT (h))
n
n²
(1/n)-
Var u~HA
(h))
+ (1 1/n) b~HA
Var (crr(h))
(D.0)~
Where the last step Lemma
14

---

Published as a conference paper at ICLR 2024
B.1 THEOREM
Theorem 1. In expectation, variance in test-set accuracy overestimates variance in true enror.
E
Var (errs(())
2
Var (err(h))
SDD
h~HA
Proof.
The difference between the two terms is
E SwDD
Var (errs((i)) AwHHA
Var (err(h)) AwHa
1
E
Var (ei)) hilla
Var (err(hi)) hoil
n
(a
1
Var(ertu.y)))))
Cov(erT21.11 (h), ez (hi)
E
E
n
2.y
1
0.5
E
Var(err 21:1 (hi)
0.5
E
Var(err (hi)
n
02.3/2
E
Cov(ertz (h), ert (1i))
E
Var (hi) + Var(err (h)) Cov(erT (h), erT za
2n (m1.an (as.ya)
1
Var(err (h)) Var(err (1)) 2 2 /arlerr (1)) / Var(erraa.ee (h))
E
2n (m2.a))
h
1
(Vaareerrasnnn Varlertoawaa (h))
2
E
2n (22.0)
0.
where the first two steps use Lemma and then Lemma
B.2
THEOREM 2
Theorem 2, The following quantity is an unbiased estimator for true variance Varnw (err(h)).
03 n -1 1
Var (crrs(h))
Σ
Var (erT (h))
A~HH
n²
h~H
Proof.
Var (err(h)) =
n
E SmDD
Var (errs(h)) hnH
(1/n)
E
Var AnHA
(errz.g(())
h~HA
(x.
Var (crr
- (1/n)
E SwDD
Var (eIranve (h)) MwHt
SnDDa
AwHHa
1
n
E
Var (errs(h))
Var (erT (hi) hwHA
SwDn
h~Ha
n²
Where the first equality a rearrangement Lemma
The quantity 03 is also equal (*) ΣT Ejye CoV (crTa..y. g. (h), erT (h)). Comparing this formula to Lemma may help provide intuition for why it is estimator for the true variance. The formulation given in Theorem [ looks less intuitive, but the benefit that we only have to calculate separate variances, rather than (?) separate covariance values.
15

---

Published as a conference paper at ICLR 2024
We note that the proofs of Theorem[aan and Theorem do not assume anything about the error function eIt (h) so, e.g., they are also true for regression tasks.
B.3 THEOREM
Theorem 3. Let A be training algorithm for binary classification which satisfies class-wise cali- bration (Definition 2j.
Then the expected variance on an IID test-set of size n is
err(A)
E
Var (errs(t)) A~HA
(1-1/ni)
Var (err(h))
SMD
2n
hoHH
Proof. Define the random variable g(x) = (L(h(()) = lt to be the proportion of training runs which classify x as positive. We first compute formula for (AA) terms of q. By the usual laws of conditional expectation we have
err(AA
(err (h.) EE[ 2.
(ert (h) [d]]
z.y.h
EE[ [1(((x) + v) q]]
x.y.h
EE| E 14y=0)1(1[) = 1 + 1) 0} [d]]
m.u.h
[B| >.
[E( 1 0)1[((2) 1) +1y = 0} | []]
(1 [[ - (1(()))))))))))))))))))))))))))) 03 |4] |g]
EE [g =.8
1(y= |
El
+1 (eeEEEEE [1 1} d]]
Elg(1 E [1fy
+ (1- 11 1) a))]-
z.y
2.1
Using the assumption of class- wise calibration, this equal Eia(1 - 4) + (1 2)4] = E - 9)}
9
Next we analyze the example-wise variance. We have A x.y
E
(ar(err=.. = [E[ E [Var(L(h(z)) + v))) g(z) d]]
2,3 h
E E [E[L(h(T) # v) - -1111(((h(a E[((h(x) + v}l) al)
E E [E[()(t))
1 [) = 03 al
2.
EE E (q(1-4) [q]
=
E[g(1 ()]
The second equality uses the formula for variance of Bernoulli variable. The third equality uses the fact that, regardless of whether y 0 or y 1, the product E,, (h(z) * (1)(11E,, (1((xx) * v)l) equal to 1(/(z)) = [x = 0}}. The fourth equality applies the assumption of class-wise calibration.
Finally using Lemma 2 and the previous two results we get
Var (errs(h))
=11 =11-//n) Var (err(h)) + (1/n)
1/n)-
E
Var (err (errz.g(h)) A~Ha
S~DD
A~Ha
h~H
(m
(1 -1-1/nn-
Var (err(h)) (1/n) Blq(1 9)]
h~H
err(A)
+ (1 (1- (1-1/n) Var (crr(h))
2n
h~
2That is, other than it being non-pathological enough to allow the interchanges of expectation via Fubini's thcorem, e.g., nonnegative suffices.
16

---

Published as a conference paper at ICLR 2024
B.4 THEOREM 4
Theorem 4. Let A be training algorithm for k-way classification which satisfies class-wise cali-
bration (Definition 2). Then the expected variance on an IID test-set of size n is at least
Var (orrs(11)) Anild
err(A)
E
2
SmD
nk
Proof. For each class {1....,k) define the random variable ge(x) En-H, (1((()) = cc
be the proportion of runs of training which classify c. Let q(x) (q) (z.).... 94 (z)) be the vector of these variables. The laws of conditional expectation yield the following expression for the expected error.
err(A) E (erTz,y(h)) s.y.h
E (errn,y()) (h) ((x) =
a.A.A
El [1(h(x) # y} |q]]
a.N.A
k
E
E g.y.h
Σ [(==
E
E
Σ ((1=))
# +c 19
h
c=1
(e)
[1 c) 1
a.
go - 4.)
E
The last step uses the assumption of class-wise calibration. We next derive a related expression for the sxamplewwise variance.
E Var (h))] = E (ar + v))) <(x) 4]]
2.9
q Z.
El # y) [0(1 - E())(()) + w) 1 ()) [d]
2.N
EE IE [g,, (1 - 9.) 9]l
&
E
E
-
k
E
[L 2,21
E
cJ 1 a) q6 go)
de (1 a.)
E
We now analyze the ratio between
> D (2 go) and E Σ& 9a(1 9c)
17

---

Published as a conference paper at ICLR 2024
Without loss of generality, let 91 S 92
9k be in nondecreasing order. Then we have
*
*
(c (1 - 9e) = k.
qe(1 - ge)
k c=1
1 x
1
2 k
Σ
9e
Σ
9e gc)
k c=1
h:
9c(1 96)
The inequality step is due to an application of Chevychev's sum inequality (Hardy et al. 1934).
which is possible because the series q1 (1 q1)........................ 9b q&) is nondecreasing, which we prove as follows.
We first recall that E 9 = L, and that we assumed without loss of generality that 91 S .... S gr- xx1 22) on the interval [0, 1 1/2,.
For the first h 1 terms, the monotonicity of the mapping
combined with the fact that 9. S 1/2 for E (1..... k (5-1), 1), implies Q1 (1 ④1) S.... S - (1
9e-1). It remains to show that qk- (1 9k-1) S qu (1 qz).. If qo. S 1/2, then this is again due to the monotonicity of + x(1 (1-z) on [0, 1/2.. Otherwise qu 2 1/2, then combining 9h-1 S 1 1-00 Qh
and (1 9 & 1/2 yields 1k-1(1 9k-1) S (1 9k (x) -11-(1
9x)) g& (1 4k). Either way,
have shown that (1 - -1) I ... S 9 (1 qk) is in nondecreasing order, allowing the application
of Chevychev's sum inequality above.
Putting Lemma 2 together with the above results, as follows, yields the theorem.
Var (cIrs(hi)
-11 1/n)
Var (err(h)) (1/n)- N~HA
Var (CITz, MwHH
S~DD
M~H
(m
2 (1/n)
E
Var (h))
k
E
ce -qo)
-==1
1
E
do 9c)
nk 9
c=11
(rr(A)
nk
B.5 REPLICATION OF THE MAIN RESULT FROM JIANG ET AL. 2021
Because it is theoretically related to our results, we include a simplified proof of the main result from et al. (202)), which is Theorem 4.1 of that work.
Theorem 5 (Tang al. 2) Let A be a stochastic training algorithm. If is class-wise cali- brated, then the error rate is equal to the expected disagreement rate between two trained networks:
erT(A)=
E (a,
[1fh1(z) + hz(2)H
18

---

Published as a conference paper at ICLR 2024
Then the laws of conditional expectation yield the following expression for the disagreement rate.
Proof. Let X x + [0, 1| be defined in Section B.4
E ba (a.))(-))))
(1)1(()) h2(2))
E (1) + h2(2)) ((x) - al
[1 (x) (2
(2..0))))
k
E
qe 1(b2(z) + + ct11
(2:(3.9)
c) q 11h2(x)
E
9c
h2
E
Σ
9c(1
qe)
k
1e(1 - q.)
E
Σ
Each conversion of conditional expectation over H. to a formula involving g uses the assumption of class-wise calibration. This formula for the disagreement rate is the same that arrived in Section B.4 for the error rate, so we have shown that the two are equal.
ImageNet-Sketta
ObjectNet
ImageNet-R
ImageNet-V2
ImageNet
//
150
Observed
Hypothesis
100
50
18%
23%
29%
30%
58%
71%
Figure 9: Distribution shift produces excess distribution-wise variance between runs. Across 1,000 of ImageNet training, both the ImageNet validation set and ImageNet-V2 have accuracy distributions close to that predicted by the independent errors assumption. On the other hand, distribution-shi shifted sets have significant excess variance, which indicates genuine differences between trained models, in light of Theorem 2
C IMAGENET AND DISTRIBUTION SHIFTS
In this section we show that shifted distributions of test data have increased variance in their accu- racy distributions between runs of training. We additionally confirm that the findings of Section generalize to standard ImageNet training.
Our experiment is as follows. We independently train 1,000 ResNet- 1 88 on ImageNet using stan- dard configuration (see Section A.2), attaining an average accuracy 71 71.0%. We study the predic- tions of these networks the ImageNet validation set, ImageNet-V2, and three distribution-shiitee datasets.
We first look at the ImageNet validation set. In Figure (rightmost) we observe that the true distri- bution closely matches the one predicted by independent errors. The observed standard deviation is 0.118%, but using Equation B we estimate distribution-winn variance to be 12x smaller, at 0.034%. This value is close to what we found for CIFAR-10 10, confirming that both training scenarios adhere our conclusions from Section
19

---

Published as a conference paper at ICLR 2024
CIFAR 10 epochs=1
epochs=64
Randem factor Isolation
1.4%
Only vary model
1.2%
only very model init
Only vary data order
only vary data order
1.0%
Only vary data augs
Only data
Arcurayy
Vary three
Vary all three
0.6%
0.2%
0.0%
52%
54% Test-sel
56%
58%
60%
93.3%
94.0%
2
8
16
32
128
Training epochs
Figure 10: One source of stochasticity sufficess for long training. When training only epoch, varying all three sources of randomness induces a standard deviation 1.33% test-set accuracy between runs, while each source alone induces 25-40% less variance. But when training for 64 epochs, varying any one source induces as much variance as all three together. Each distribution corresponds to 4,000 runs of training.
Next we consider ImageNet- V2 (Recht et al., 2019). This dataset is intended to have the same dis- tribution of examples as ImageNet, and we demonstrate that its accuracy distribution has similar statistical properties as well. Specifically, we find that the distribution predicted by independent errors also closely matches the true distribution, and Equation Equation3eestinne esttmatts a distribution wise stan- dard deviation of 0.071%; %. which is larger than what we found on the ImageNet validation set, but still relatively small. We note that the accuracy distribution for this dataset is wider, but assuming independent errors this can be explained simply by the fact that it has 5x fewer examples than the ImageNet validation set.
By contrast, ImageNet-R (Hendrycks al. 2021), ObjectNet (Barbu al. 2019) and ImageNet-
Sketch (Wang et al. 2019) have different statistical behavior compared to the first two datasets. These datasets constructed to have shifted distributions relative to ImageNet. We find that their accuracy distributions have variance significantly in excess of that predicted by independent errors. We estimate using Equation 3 that these three test-sets have large distribution standard devia- tions of 0.1811 %, 0.17,% and 0.257% respectively, indicating significant differences between runs of training.
We additionally investigate correlations between pairs of these five datasets (Figure 12 The strongest correlation is between ImageNet-R and ImageNet-Sketchh, with H²² 0.14 (p 10 10-*). Manual inspection shows that both ImageNet-Sketch and ImageNet-R contain many sketch-like images, suggesting that similar features may induce correlation between distributions. All other pairs have R < 0.01. For example, ImageNet-Sketch is decorrelated from ObjectNet, with R2 0.001 (p - 0.336).
Overall, our findings suggest that training instability is in some sense a relative notion. ImageNet training is stable when evaluated on the main distribution, with a small standard deviation 0.034% on the distribution of the ImageNet validation set. But it is unstable ImageNet- having 58x as much distribution-wise variance, at a standard deviation of 0.25%%. This serves caveat the title: from the perspective of the main training distribution, variance harmless in that every trained network has almost the same performance, but from the perspective of shifted distributions, there significant differences between runs.
shifted distributions, with
D REPLICATION STUDY OF SUMMERS & DINNEEN (2021)
D.1 THE THREE SOURCES OF RANDOMNESS
Training neural networks typically involves three sources of stochasticity, namely, model initializa- tion, data ordering, and data augmentations. In this section investigate how each of these sources contributes to the final variance between runs that we observe at the end of training.
We develop CIFAR-10 training framework that allows each source to be independently controlled by one of three different seeds. For example, when the aata-augmentttionn seed is fixed and the data- order seed varied, the set of augmented images seen by the network throughout training will
20

---

Published as a conference paper at ICLR 2024
Epochs=128
Epochs=8
Poke during training
6%
nifferent seeds
seed, pake first layer
59
Disagreement rate
Same seed, pake last layer
4%
3%
2%
1%
0%
2000 Step poked
Disagreement rate
Figure 11: Training has high sensitivity to initial conditions. (Left:) For short trainings, pairs of runs which differ only by one network having been "poked" (i.e., had a single weight changed slightly at initialization) disagree on 7.0-7.5% of predictions. Pairs of runs with fully different random seeds disagree more, on ~8855 of predictions. For long trainings. there is almost difference. (Right.) The earlier network is poked during the training process, the more predictions will disagree with the network that trained unperturbed with the same random seed.
remain the same, but be presented in different order. When all three seeds are fixed, training is deterministic, so that repeated runs produce the same network. Standard training equivalent to allowing all three seeds to vary.
We fix two seeds, and vary just the third (e.g. varying only the data order while keeping the model initialization and data augmentations fixed). Our naive intuition is that each factor contributes some part to the overall variance, so that this should decrease variance relative to the baseline of varying all three seeds.
Our results show that for short trainings 1-16 epochs, this intuition correct (Figure T0p. For example, when training for 4 epochs, if fix the data order and augmentations, while varying only the model initialization, then variance in test-set accuracy is reduced by 26%, with the standard deviation going from 0.45 + 0.01% 0.38 + 0.01%.
However, for longer trainings of 32 epochs or more, varying only one of the three random factors produces approximately the
variance the baseline of varying all three. For example, across 8,000 runs of training for 64 epochs, varying just the model initialization (with data ordering and augmentation fixed) produces standard deviation of 0.158% almost the same as the bascline, which has 0. 160%. At 8, 000 this is not a statistically significant difference; it is possible that the true values are the same, or that they differ by a small amount. We conclude that for this training regime, any single random factor suffices to generate the full quantity of variance, rather than each factor contributing to overall variance.
D.2 SENSITIVITY INITIAL CONDITIONS
In the previous section, we showed that when training convergence, varying just the model ini- tialization (or just the data ordering, or augmentations) produces approximately the same quantity of variance between runs as baseline fully random setup. In this section we find that even varying a single weight at initialization suffices. Our findings replicate the work of [Summers & Dinneen (2021), who reach similar conclusions.
Consider multiplying single random weight in the network by 1.001. We call this "poking" the network. This a tiny change; recent work quantization (e.g.. DDttmerss et al.) (022) implies that trained models typically have all their weights modified more than this without losing accuracy.
Nevertheless, in Figure [] we demonstrate that poking the network early in training produces large difference in the final result. Our experiment is to run two trainings with the same random seed, but with one network being "poked" at some point during training. We measure the disagreement rate between the two networks, i.e., the fraction of their test-set predictions that differ. For short trainings, poking induces much less disagreement than changing the random seed. But when training for 128 epochs, poking alone produces average disagreement of 5. 3.14%, barely less than the 5.19%
21

---

Published as a conference paper at ICLR 2024
produced by using two entirely different random seeds. We have also observed that varying just the first batch data, or the numerical precision of the first step (e.g,, fp16 vs. fp32) has a similar effect. We conclude that almost all variation between runs is not produced by specific sources of randomness like model initialization, data ordering, etc., but is instead intrinsic to the training process, which has extreme sensitivity to initial conditions.
E ADDITIONAL FIGURES
R-2-2 3.000 1p-00 0.0011
I""2+0 (3-03331
#"2-0 004 12-0
1"2- 10-2.228
B""2-d 023 12-0.221
2010
75,09
2h
11.0%
Ve
0.2000002 (p-0.130)
R 009 (p-0.0021
R-2-0 34040 (p-0.000) (000)
R-2-0,002 (p-6.146)
2-2 (p-0.336)
madgetes SM455
26
masgores SAARL
Onjecteee
25.79
28
18.3%
15.0%
22.07
17.0% 3735
263% IPL 28.00
2835 ONL 28:00 Imsgehet
81.%
37.0L 1E 1ROW Image
InsgeNet.
Figure 12:
Correlations between distribution shifts. We visualize the accuracy values of 1,000 ResNets which were independently trained on ImageNet. Each network evaluated on the ImageNet validation set, as well as four distribution-shift datasets (IN-V2, IN-R, IN-Sketch, and ObjectNet). We display the scatterplots of accuracy on each of the (2) pairs. The following pairs had statistically significant correlations: (ImageNet-R, Sketch, ImageNet). (ObjectNet. ImageNet-V22 V22). and (ImageNet Sketch, ImageNet-R)
ImageNet), LkaachettSkkecch but one pair have weak correlations with R² 0.01. The strong correlation between ImageNet-PP R and ImageNet-Siee -keeeh with H² = 14. We hypothesize that this is caused by the fact that both sets contain many sketch-like images, so that this pair has a) similar distributions and b) shifted distributions rolative to the training set. We report two-sided p-values.
CIFAR-10 64 64-epoch training
Empirical sample
Hypothesis simulation
Binomial simulation
94.0%
94.5%
95.0%
Test-set accuracy
The binomial approximation overestimates variance. Compared to the empirical distribution of test-set accuracy, the binomial approximation predicts distribution with too much variance. We use p 0.9441 (the average accuracy) and 10, 000 (the size test-set) simulate 60, 000 samples from Binom(n, p). which we find overestimates variance by a factor 2.5x. comparison, the independent emors assumption provides an accurate estimate.
Figure 13:
22

---

Published as a conference paper at ICLR 2024
PPCC)
C(
46.76%
59.78%
58.46%
64.57%
40.89%
43.63%
25.46%
27.96%
43.88%
46.04%
Figure 14: Pairs with non-independent errors. The first column is the product of the probability (over training stochasticity) that the trained network predicts the first example correctly, and the probability that predicts the second example correctly. The second column is the probability that the trained network predicts both correctly at same time. All probabilitics are measured independent crrors condition (Definition [b would be if these two quantities were equal. Out of all (10.000) pairs of examples in the CIFAR-10 test-set, only these five deviate by more than 2% from having independent emorr..
our 64-epoch training configuration, The
23


Reviews:
[{"review_id": "U04r5lNNFV", "reviewer": "Reviewer_G5ah", "summary": "This paper shows that the variance for true test distribution among multiple training runs is actually smaller than those observed from the test dataset. They also proposed a statistical assumption and derived an estimator to estimate the stand deviation of the test distribution.", "questions": "Have we tried different architectures? Do they have similar results? I am curious do we really need to get the true variance? For example, if for all the models, we have a similar trend between variance from test-set and test-distribution, then why do we need to get that?\n\nDo we have some cases that the variance among the test distribution gives us new insight? For example, does flatness of the minimizer related with this?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to follow. The paper provides many empirical analysis to support their points and also include some theoretical analysis.", "weaknesses": "Since most parts of the paper are based on the empirical analysis, it might be better the have the experimental setup in the main paper including all the hyperparameters and all the models in those 60000. However, it might be important to include results for different architectures since some of the models may have better reproductions. It might be interesting to have more use cases for getting the variances of the true test distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UtU1bI9uwN", "reviewer": "Reviewer_Wzaj", "summary": "This paper delves into the variance observed in test-set performance across repeated runs of neural network training, a phenomenon that has raised concerns about hyperparameter comparison and training reproducibility. Key findings of the study are:\n\n- Although there's significant variance in test-set performance, the variation on the underlying test-distributions (like CIFAR-10 and ImageNet) is minimal. This suggests that in practical applications, the variance might not be as concerning as assumed.\n\n- The study puts forth an \"examplewise independence\" hypothesis, suggesting that a network's correct prediction for a given test example is akin to a biased coin flip and is independent of its predictions on other examples.\n\n- The paper states that prior works have noted that predictions from ensembles of networks, trained repeatedly, are typically calibrated. They argue that this calibration inevitably leads to some variance in test-set accuracy. For binary classifications, they provide a formula to determine this variance.\n\n- Other observations include the reduction in distribution-wise variance with longer training durations, the correlation between optimal learning rate and absence of excess distribution-wise variance, the effect of data augmentation in reducing variance, and the increase in variance when test-sets differ from the training set.", "questions": "- How does the conclusion generalize to other training setting or other network architecture (like Transformers)?\n\n- How can the insights of random seeds help practical development? E.g. if they can be used to tell the quality of individual model, to separate the accuracy into random part and true accuracy.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper covers an interesting topic: the variation of test accuracy over random seeds (stochasticity of deep learning experiments). It tells the test accuracy on the whole test set from the test accuracy on the test distribution (random subsets of test sets). An important conclusion is that accuracy gain in the whole test set from random seeds does not generalize to the whole test distribution. Another interesting conclusion is that the result of trained model on each example can be approximated by binomial distribution with biased flip probability.\n\nOverall, I find this paper interesting and novel. The paper is easy to understand.", "weaknesses": "Althouth I enjoy reading this paper, the analysis is mainly posterior: we can only get these insights after running the same algorithm for hundreds of runs. As far as I can see, the insights from this paper can tell us if a training algorithm is good despite the impact of random seed, but it cannot tell if a trained model is better than another despite the impact of random seed. If this paper can further achieve the latter application, I would be more than happy to raise my score.\n\nAnother concern is how well does the conclusion itself generalize to other settings. In Figure 1, it seems the variance becomes smaller as training goes on. But I think it is caused by the learning rate schedule: the authors say that they \"always linearly ramp the learning rate down to zero by the end of training\", so it is expected that the accuracy has less variation in the end.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hokD8Nb5mR", "reviewer": "Reviewer_yu6g", "summary": "The authors analyze the variance between trained networks on the test set and find that minor variations are insignificant and explained by finite sample noise due to a finite test set. Further, they show that when considering the test distribution, the variations in validation set performance are not well correlated with the test distribution performance.", "questions": "- In figure 2, why is (odd examples) in parentheses? I cannot figure out what this means since there are only two splits and they seem to be uniformly random.\n\n---\n\nUltimately, I like the findings of this work, but I find part of the presentation problematic as noted in the weaknesses section above. If these things can be fixed, I would probably raise my score.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The work formalizes what I think many people would ultimately suspect in that the minor variations between runs are insignificant when looked at through the larger perspective of the test distribution. Even though I think this would be expected, it is nice to see a work which empirically verifies it. \n\n- The derived bounds are useful for estimating the expected variance between runs.", "weaknesses": "- The two main itemized contributions contain way too many points to be in a list. They are paragraph sized. I do not think something of that size should be presented in a list. I believe they either need to be broken up into smaller components, or discussed in plain text instead of a list. \n\n- What is $n$ in theorem 3, 4, and 5? I suppose it should be the size of the dataset, but it would be nice to include this right by the theorems in order to avoid any possible confusion. \n\n- I think it would be interesting to see the difference in Theorem 3 when varying the number of models in the ensemble. For instance, the bias in the estimate may be small, and a two model ensemble may give good results, or it may actually require quite a large ensemble in order to be close to correct. Can you add this experiment? I believe it can be done by only randomly choosing and expanding the ensemble of the models which are already trained.\n\n- Many of the claims listed as main contributions are in the appendix. Even in the conclusion, the things in the appendix as main contributions which have been demonstrated even though there has been nothing said about them up until this point in the text. I do not think this is fair to the reader and it should be reorganized or rewritten such that this does not happen. \n\n- In section 3.4 it says: “That is, if we let S ′ be the subset of test images which are classified by 30-40% of independently trained networks as “dog”, then approximately 30-40% of the images in S ′ really will be dogs.” I do not think that is how calibration is phrased in most calibration works. In fact, all networks may correctly classify the image as a dog (meaning that the highest predicted probability is the dog class), but the predicted probability (confidence) should equal the empirical distribution of the positive class in $S^\\prime$, then the model is calibrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MxxkMkOa0q", "reviewer": "Reviewer_Nr6n", "summary": "In this work, the authors try to address the following important question: are marginal improvements in test accuracy (after a certain point) on benchmark datasets actually indicative of better models (with respect to the underlying data distribution)? Towards this end, the authors conduct a series of large-scale experiments on CIFAR-10 and ImageNet which show that trained model performance becomes uncorrelated on disjoint splits of test data. They also connect this variance in model performance with the notion of calibration of ensembles of models.", "questions": "- Is the test set split fixed? In other words, when the authors write \"we split the test-set into two halves of 5000 examples each\", does this mean there is a single fixed split of the test set? If so, this seems to not make sense in the context of the proposed formalism ($S \\sim \\mathcal{D}^n$), which considers randomly sampled test-sets.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Originality:** The experimental setups in this paper, as well as the connection introduced between model variance and calibration, are to the best of my knowledge new and contain several interesting ideas.\n2. **Quality:** The authors have run extensive experiments to verify their hypotheses (at least within the context of the benchmarks they restrict themselves to), and their theoretical predictions track the experiments quite closely (for the appropriate regimes). \n3. **Clarity:** Overall the paper is very easy to read, and justification/sufficient detail are provided for the experiments conducted in the paper.\n4. **Significance:** This paper studies the important problem of how to assess a trained model's future test performance, and introduces useful formalisms for thinking about this problem. The main drawback here is that the paper's experiments and hypotheses are restricted to two image classification benchmarks (which are widespread - this is still a useful contribution), so it is tricky to assess how well the observations will generalize.", "weaknesses": "## Main Weaknesses\n1. **Generalization of takeaways.** The practical takeaway provided in the paper in Section 3 is that marginal test performance improvements for well-trained models on CIFAR-10/ImageNet can be uncorrelated with performance on the underlying data distribution. I have some concerns with this claim (i.e. fixed test splits vs resampling), but even assuming this to be true - what can be said for more general machine learning (or even image classification) tasks? For example, in the experiments in Section 4, as the authors note there is a clear correlation between BERT-Large validation performance and test performance. I am not really sure how this should be interpreted in the context of the paper; while the analysis in the paper shows empirically that BERT-Large fine-tuning has more variance than BERT-Base, I don't see how the analysis in the paper would allow one to predict (without running many experiments) whether val performance will be correlated with test performance or not.\n2. **Hyperparameter choices/sensitivity for results.** Of course it is not possible to be entirely comprehensive with respect to hyperparameters, but I have concerns with some choices made by the authors. Particularly, the training horizons considered are 0, 4, 16, and 64 epochs. We see in the experimental results a clear trend of decreasing variance as the training horizons are extended - do any of the results still hold for longer training horizons? I would anticipate that the independence of network predictions surely shrinks as training horizons are extended. \n\n## Minor Comments\n- In the proof of Lemma 1, the exponent should be inside the expectation. Additionally, it's probably helpful to justify the penultimate step (interchange of expectation) by saying that Fubini's Theorem applies.\n- It would be better to include a definition of ECE in Section 3.4, or at least verbally describe it in the context of Hypothesis 2.\n\n## Overall Recommendation\nMy concerns with the paper are mostly with the actual significance of the practical recommendations in the paper (as discussed above). That being said, I think the paper contains sufficiently many interesting ideas and experiments to possibly warrant acceptance (I am borderline because again, I'm not sure how well these ideas generalize). Thus, I recommend **weak accept**.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)