On Bias-Variance Alignment in Deep Models
Authors: Lin Chen
Michal Lukasik
Wittawat Jitkrittum
Chong You
Sanjiv Kumar
Venue: ICLR 2024 spotlight (2024)
Keywords: bias-variance decomposition
ensemble
deep learning
Primary area: general machine learning (i.e., none of the above)

Abstract: Classical wisdom in machine learning holds that the generalization error can be decomposed into bias and variance, and these two terms exhibit a \emph{trade-off}. However, in this paper, we show that for an ensemble of deep learning based classification models, bias and variance are \emph{aligned} at a sample level, where squared bias is approximately \emph{equal} to variance for correctly classified sample points. We present empirical evidence confirming this phenomenon in a variety of deep learning models and datasets. Moreover, we study this phenomenon from two theoretical perspectives: calibration and neural collapse. We first show theoretically that under the assumption that the models are well calibrated, we can observe the bias-variance alignment. Second, starting from the picture provided by the neural collapse theory, we show an approximate correlation between bias and variance.

Published as a conference paper at ICLR 2024
ON BIAS- VARIANCE ALIGNMENT IN DEEP MODELS
Lin Chen, Michal Lukasik, Wittawat Jitkrittum, Chong You, Sanjiy Kumar Google Research, (linche, ml mlukasik, k, wi wittawat, cyou, jenoivvxeeoooge..com
ABSTRACT
Classical wisdom in machine learning holds that the generalization error can be decomposed into bias and variance, and these two terms exhibit a trade-off. However, in this paper, show that for an ensemble of deep learning based classification models, bias and variance are aligned at a sample level, where squared bias is approximately equal to variance for correctly classified sample points. We present empirical evidence confirming this phenomenon in a variety of deep learning models and datasets. Moreover, study this phenomenon from two theoretical perspectives: calibration and neural collapse. We first show theoretically that under the assumption that the models are well calibrated, we can observe the bias-variance alignment. Second, starting from the picture provided by the neural collapse theory, we show an approximate correlation between bias and variance.
INTRODUCTION
The concepts of bias and variance, obtained from decomposing the generalization enror, of fundamental importance in machine learning. Classical wisdom suggests that there a trade-off between bias and variance: models of low capacity have high bias and low variance, while models of high capacity have low bias and high variance. This understanding served as an important guiding principle for developing generalizable machine learning models, suggesting that they should be neither too large nor too small (Bishop, 2006). Recently, line of research found that deep models defy this classical wisdom (Belkin et al., 2019): their variance curves exhibit a unimodal shape that first increases with model size, then decreases beyond the point that the models can perfectly fit the training data (Neal al., 2018; Yang et al., 2020). While the unimodal variance curve explains why over-parameterizze deep models generalize well, there is a lack of understanding on why it occurs.
Varlancelx, v)
26-7
10-1
20-14
Correctlyy dlassifed
2014
Co medeyy dass clss fed?? rabe
Blanio, y)?
30
fulee
Truo
10
10
30
10"
(a) Illustration B-V alignment in B-V space (left) and output space (right).
Bits squared
(b) B-V for vision task
(c) B-V for an NLP task
Figure 1:
The bias-variance alignment phenomenon. (a) Given input and its associated label y, bias-variance alignment refers to the phenomenon that the bias and variance of a deep model satisfy log Biash..(z.y) s log Varina,(z.g) for correctly classified points, as illustrated the dashed line (see the left subfigure). In the right subtigure, each cross represents a prediction of the model ensemble } on sample x, and the center is the one-hot encoding e, of the corresponding label y. Green, yellow, and red colored clusters of crosses correspond the three groups also shown in the left subplot, with small, medium, and large bias, respectively. Bias-variance alignment implies that the three groups have small, medium, and large variance, respectively. (b) Per-example bias and variance for ResNet-50 trained on ImageNet, where each dot corresponds to test sample and colored according to whether the sample correctly classified by the model or not. Bias and variance estimated from 20 independently trained networks with different initial weights and over different bootstrap samples from the train set, following methodology from Neal et al. (2018). (c) Per-example vaas-varinnce for BERT trained on TRBC data (see Section E.7 in Appendix for details).

---

Published as a conference paper at ICLR 2024
This paper revisits the study of bias and variance to understand their behavior in deep models. We perform a per-sample measurement of bias and variance in popular deep classification models. Our study reveals curious phenomenon, which is radically different from the classical tradcoff' perspective on bias-variance, while is concordant with more recent works (Belkin et al., 2019; Hastie et al., 2022; Mei & Montanari, 2022). Given sample x and its corresponding label from dataset of test examples {( 34 JEjM let (2as),.(z.g) and (arina. be the bias and variance, respectively, of an ensemble of deep models {he}. Here, the randomness in calculating the bias and variance comes from 0, which depends on the randomness in parameter initialization, batching and sampling of the training data (Neal et al., 2018; Yang et al., 2020).
Our key empirical observations be summarized with the following two statements which we call the Vara-nceee Alignment and the Upper Bounded Variance. As we explain in the remainder of the paper, and summarized in Table 1, these observations encapsulate our empirical observations, and also capture the special cases we prove from the calibration and the neural collapse assumptions.
Bias- Variance Alignment. First, we find that for correctly classified points (3r, y) E (1r,, ve(w)
log (G. = log (aa or log Varino.(<x.::) log (i E + (E; is noise s.t. (n-) [Ec]
+hho
(Eho is constant independent of i)
(1)
where e; is random noise with mean vanishing across the dataset (i.e., Uni = 0). Specif- ically, our quantitative results show that (1) a simple linear regression of log (arine.(z..n)) on log Biash (*..s)) yields a remarkably high coefficient of determination R²; (2) the residuals of the simple linear regression exhibit an approximate normal distribution (we provide evidence for (1) and (2) in Section 3.1).
Type
Assumptions
Finding (logarithmic scale)
Finding (linear scule)
Ref.
Note
Large model
log (=..a)
Varina
Empirical
+ Correctly classified data
ss log 2
Ens
Cho i + &a
Sec. 3
Theoretical
Perfect calibration
Binsh (eunv) (aria
Sec.
Thooretical
Neural collapse
log Blan, (K) log Varlog (R)) E (1.114,4)
Blonh Blodh Varhag(-res)
(2--112 (xD(
33
Sec. b
Binury classificution
The result Biush,. R Varing. corresponds to C' in our main empiricul observation on the linear scale presented in Eq. (2) and we bound & Varis, Biusi by the calibration error in Section 4.
b Here, &t e (1,2) is class index and is (roughly speaking) the 62 norn of the prelogit ho(x)) (i.e., before softmax).
Table 1: A summary of our findings on the bias-variance alignment.
In linear scale, we can represent Equation (1) as
(ar.5 = Ch, Biashe + (Cho = c Eno En- > O is a constant)
(2)
&, OBBiss m (1: is noise s.t. (n-()ni((()) [m]] 0)
formal statement of the above formulations can be found in Proposition E.1 in Appendix E.4. Note that because the noise term & scales with squared bias, Equation (2) predicts that the sample-wise bias-variance in linear scale has a cone-shaped distribution (i.e., bias increases, an increasingly wider range of variance is covered by examples). We discuss this in more detail in Appendix E.4.
Upper Bounded Variance. Second, we find that the following relation approximately holds for all examples (i.c., for both correctly and incorrectly classified examples):
Biash (2..0) 2 Che (arino.(z.))>
V(z, v) E ((rs, vi) [))een))
where Ch, is the same constant in Equation (2). In Figure 1, we illustrate these findings on an illustrative example. Observe that for correctly classified sample points, the bias and variance align closely along the line of Biash...(z.)) = Varina i.c., the equality in (1) holds. We refer to this phenomenon as the bias-variance aligument. For incorrectly classified samples, we observe Biasho. (x.,)) > Vari hence the inequality in (3) approximately holds for all examples. It is worth noting that Eq. (3) provides an explanation for why deep models have limited variance, and in
2

---

Published as a conference paper at ICLR 2024
effect, good generalization, i..., the variance of a model is always bounded from above by the squared bias at every sample.
The paper provides both empirical and theoretical analyses of the bias- alignment phe- nomenon. We organize the paper as follows. We begin with empirical investigations, in which we observe the bias-variance alignment across architectures and datasets (Section 3). We then move on to theoretical explanations of these phenomena. We start from a statistical perspective, where we connect calibration and the bias-variance alignment (Section 4). In the process, we generalize the theory from previous works on calibration implying the (Jiang et al., 2022; Kirsch & Gal, 2022) (Section 4. Next, we show how starting from a separate perspective of neural collapse (Papyan et al., 2020) can lead to the blas-variance approximate equality result (Section 5). We conclude with the discussion of wider implications of our findings (Section 6).
equality
Our main contributions are: (1) We conduct experiments to show that the bias-variance alignment holds for variety of model architectures and on different datasets. (2) We provide evidence that the phenomenon does not occur if the model is small. This suggests that the bias-variance alignment is specific to large neural networks and provides more evidence that there could be a sharp difference between small and large models. (3) Theoretically, we prove the bias- alignment under the assumption that the model is well-calibrated (i.c., the output of the softmax layer aligns with the true conditional probability of each class given the data). As side product, provide unified definition for a variety of definitions of calibration introduced in previous works. (4) We show that the neural collapse theory predicts the approximate bias-variance alignment.
2 BACKGROUND AND RELATED WORK
2.1 BACKGROUND ON BIAS- VARIANCE DECOMPOSITION
Consider the task of learning multi-class classification model he X + M((K] RKK where X' is the input domain, ((K is the set of distributions on [K], and K is the number of classes. Let {he : X + M((K]} be an ensemble of trained models, where 0 is random variable taking values from O¹ For any input
X, we use h- x)) to represent the corresponding distribution. That is. h x) A (1 (1 [x,..... ha K x)) is the vector of predictive probabilities from model hgg Given any sample (X, E X' x KK), the bias and variance of {ho) with respect to the mean squared error (MSE) loss are defined as follows.
Definition 2.1 (Bias and Variance). Let h(- xx) & Eghg(: (-1 x) be the mean function of [ha}. The bias, variance, and bias-variance gap of the i-th entry on (X, Y), for each E [K], are defined
Bias (x,,) (1) (x,) A |h(i X) - 1(Y = it}.
(4)
Variso. (X,Y) (1) 2
(X, A Rg (ho X) h[( x)))
(5)
(X,)
Bias (x,,) - (xrr,,,(X,))
(6)
Throughout this paper, we use Biash,. and (x,,(xx,) interchangeably as synonyms, and we call Sho.(X,Y) (i) the standard deviation of the i-th entry. Moreover, the total bias, variance, and bias-variance gap are defined as
Biaspo
Bias Bi (x.)) (1) ||A(- | ex eyll2
KE|K
Varing
Varino (X.) (i) Es (: X) (x)13))
(8)
aE|K]
BVGRo.(X.Y)) Bias Varina,
where E E IR is a vector whose i-th entry is and all other entries are
is well-known that bias and variance provide a decomposition of the expected risk with respect to the MSE loss (Hastie et al., 2009, equation (2.25)). That is, Risk (X.Y) - (ha(
(-1 X) - ey 1| = Bias (x.,) + (x,.),,(X,Y))
(10)
In all of our empirical results, the randomness comes from weight initialization and hootstrapping of training set, following Neal al. (2018). 8). The effect of different sources of randomness is studied in Appendix F.6.
3

---

Published as a conference paper at ICLR 2024
Pointwise
Empirical vs. Thooretical?
Premise
Finding
In ageregate?
Calibration
Generalization ezror Disgreementt
In aggregate
Both
(Jiang
2022: 2022: Kirsch & Gal. 2022)
Generalization
Calibration
In aggregate
Empirical
(Carrell et al., 2022)
Multi domain calibration
Out of do-moni generalizaation
In aggregate
Empirical
(Wald et al., 2021)
Calibration
Bias" za Varisnce
Pointwise Pointwise
Both
This work
Neural collapse
Biae Varisnce
Both
This work
Table 2: Summary of findings about calibration, generalization and disagreements.
We focus on models trained using the cross-entropy (CE) loss throughout the paper. However, we analyze the bias and variance of these models using decomposition of the mean squared error (MSE) loss. We present results for bias and variance from decomposing CE loss in Section E.2.
We now introduce several notations that will be used throughout the paper.
Definition 2.2. The prediction, confidence, and accuracy on æ are defined by
pred, (x) arg max h(j | conf (x) h(prod, (x)) x) acck(r) = Pyix (prod, x).
SE[K]
The uncertainty of the ensemble {he} on x is Uncena (x) = E.g Eollhal x)113.
2.2 RELATED WORK
Bias-variance decomposition in deep learning, In the classical statistical learning theory of bias- variance tradeoff, increasing the model capacity beyond certain point leads overfitting (Geman et al., 1992). However, deep neural networks in practice usually contain large number of parameters but still generalize well. Towards bridging the gap between theory and practice, one of the most famous work is Belkin et al. (2019) which reveals a "double-descent" curve to subsume the U-shaped tradeoff curve. This surprising observation motivates the work of Neal et al. (2018) to measure the bias and variance in popular deep models, leading to discovery of "unimodal variance" phenomenon (Yang et al., 2020). Subsequent work includes Adlam & Pennington (2020); Lin & Dobriban (2021) that study variance under fine- ene-grained decompositions, and Rocks & Mehta (2022a;b) that analyze variance under simplified regression or random feature models.
Calibration. Calibration is fundamental quantity in machine learning which informally speaking measures the degree to which the output distribution from model agrees with the Bayes probability of the labels over the data (Guo al., 2017). Previous works proposed theory calibration equality (Jiang et al., 2022; Kirsch & Gal, 2022). In Table
implying the amneralization-diiageeeeee we summarize several related results connecting calibration with other fundamental concepts from previous works. Our work can be viewed as extending these works as we connect calibration and the bias-variance alignment.
Neural collapse. Towards understanding last layer features learned in deep network based classifica- tion models, the work of Papyan al. (2020) reveals the neural collapse phenomenon that offers clear mathematical characterization: Within-class features collapse to their corresponding class means, and between-class separation of the class means is maximized. This observation motivates a sequence of theoretical work justifying its occurrence (Fang et al., 2021: : Zhu al., 2021; Tirer & Bruna, 2022; Poggio & Liao, 2020; Thrampoulidis et al., 2022), and practical work on leveraging the insights improve model performance (Liang & Davis, 2023; Yang al., 2023; Li et al., 2022).
EMPIRICAL ANALYSIS OF BIAS- VAAS-NARANCE ALIGNMENT
We begin by providing quantitative measure in Section 3.1 on the alignment of bias and variance illustrated in Figure lb. Then, we provide empirical evidence in Section 3.2 that the bias-variance alignment phenomenon occurs than ImageNet. Finally, in Section 3.3 we study the effect of network size, showing that bias variance alignment is phenomenon for over-parameterized models.
prevalently for networks beyond ResNets, and for datasets other
4

---

Published as a conference paper at ICLR 2024
3.1
QUANTITATIVE REGRESSION ANALYSIS OF BIAS- VARIANCE ALIGNMENT
Model name
Slope
Restltt 56 (CIFAR 10)
Restict (CIFAR: 10)
ResNett 50 limageliet
Restict 110 (CIFAR 1001
Oroered values
ResNet-8 (CIFAR-10) 10)
0,979
0.882
RosNet-56 (CIFAR-1O)
0.964
ResNet-110 (CIFAR- (00)
0.986
0.901
RosNet-50 (ImageNee)
0.977
0.897
Table 3: Coefficient of determi- nation (R22 and slope for linear regression of log Varing.(a;aa) on log (ia.5..(
-2.5 0.0 2.5 Theoretical quantles
0.0 2.5 Theoretical quanbles
-2.5
-2.5 0.0 2.5 Theoretical quanbles
2.5 Theoretical quantles
-25
Figure The Q-Q plots of the residuals of linear regression of log Variha (.)...)) on log Bi (2....)) of the four models.
We start by conducting a quantitative regression analysis of bias and variance on the logarithmic scale for verifying the bias-variane, alignment phenomenon in Eq. (1). First, in Table 3, present the coefficient of determination R² and the slope of the linear regression of log Varia, (w) on log Biass,, (z..5..( for the following models and datasets: ResNet-56 (on CIFAR-10). ResNet-8 (on CIFAR-10), ResNet-50 (on ImageNet), and ResNet- 10 (on CIFAR-100). Notice how the coefficient of determination for all four settings extremely close I (at least 0.977) and the slope is also very close to 1, demonstrating a very strong linear alignment of the two quantities. We next analyze the normality of the residuals of the logarithmic linear regression. The Q-Q plots are shown in Figure We observe that the residuals of the linear regression on the four models are all approximately normal, especially for the data points whose sample quantile between 1.5 and 1.5.
3.2 PREVALENCE OF VTAS-VARIANCE ALIGNMENT ACROSS ARCHITECTURES AND DATASETS
In Figure 1b, we showed that the varianee alignment occurs for ResNet-50 trained on ImageNet. Here, provide additional evidence in Figure that the bias-variance alignment occurs for other model architectures and datasets.
In particular, Figure 3a and Figure 3b show the sample-wise bias-variance of EfficientNet-BO (Tan & Le, 2019) and MobileNet-Ve V2 (Sandler et al., 2018), respectively, trained on the ImageNet dataset. Figure 3c and Figure 3d show the sample-wise bias- variinee of ResNet- 110 trained on CIFAR- and CIFAR-100, respectively. In all cases, we observe a similar pattern, demonstrating the prevalence of the bias-variance alignment with respect to network architectures and choice of dataset. Finally, in Figure 12 we confirm the observation on the NLP benchmark, where we finetune BERT models on TREC the dataset.
Blicientfis-.00
10f
RolNel-112
Daaliab-112
10.
10-1
1n
20-
10m4
30:31
Correclyy
Comectly
Correclly claaifid? Tre
1090
10
The
30-41 1010t 10-1
10-19
19-1
2anh p0rt 2prt Bas Squared
30rth
9ls Squaros
Blas squared
Bia Squared
(a) ImageNet
(b) ImageNet
CIFAR-I0
(d) CIFAR-100
Figure 3: Sample-wiss wise bias and variance for (a, b): Varying model architectures trained the ImageNet dataset, and (c, d): Two additional datasets, namely CIFAR-10 and CIFAR-100. For ImageNet, CIFAR-10 and CIFAR-100. bias and variance are estimated from 10, 100 and 100 independently trained models, respectively.
3.3 ROLE PAAMPARRRTTERZAAIOOON
We investigate the impact of model size the bias-variance alignment phenomenon, demonstrating that it only occurs for large and wve-pparamteriized models.
To vary the size of the model, we consider ResNets with depths of 8, 20, 56, and 110, which denote as ResNet-(S, (8, 20, 56, 110). To obtain even smaller models, we reduce the width of ResNet-8 from 16 8, 4, 2, and 1. Here, the width refers the number of filters in the convolutional layers of
5

---

Published as a conference paper at ICLR 2024
Re Specifically, the convolutional layers of R can be divided into three stages with 16, 32, and 64 filters, respectively. Therefore, ResNet-8 with width k, denoted as ResNet-8 [.[[)] refers to ResNet-8 with k. 2k, and 4k filters in the three stages, respectively.
Due to space limit, we choose three models, namely ResNet-8- WI, ResNet-8-W16, and ResNet- 10- W16 from the eight model sizes discussed above, and present their sample-wist bias and variance evaluated on CIFAR-10 in Figures 4a to 4c. These three models represent a small, medium, and large size model, respectively. Results for all the other model sizes are provided in Figure (see Appendix). We can see that bias-variance alignment becomes increasingly pronounced with larger models.
36-7
1214
1004
10-4
Pet nodt
petee same anecd
and
neact shaeed severt oabers Fraetet
Das squared
0.0 oquzned
Blx squared
(a) ResNet-8-WI
(b) ResNeet--WW66
(c) 10-
(d) Varying model size
Figure 4: Figures 4a to 4c: Sample-wise bias and variance for networks of varying scale trained on CIFAR-10. 10. Here, ResNet-[pl [p]--pp]WWW-- [p]-W[q] refers to ResNet with p layers and width q. The model size monotonically increases from the leftmost figure the rightmost figure. For all cases, the bias and variance are estimated from 50 independently trained models. Figure 4d: Averaged bias and variance over all test samples under varying model sizes.
The results in Figure 4 suggest that the emergence of bias dlas-variance alignment is associated with per-parrameterization, which refers to large capacity deep models that can perfectly fit any training data. Classical theory suggests that bias and variance exhibit a trade-off relation, where larger model size reduces bias and increases variance. However, Yang al. (2020) shows that this trade-off holds only in the regime where the model size is relatively small. For over-parameterized models, the variance does not continue to grow: rather, it exhibits a unimodal shape. To examine bias-variance alignment in association with the unimodal variance phenomenon of Yang et al. (2020), we compute the averaged bias and variance over all test samples for results reported in Figures 4a to (and also results of five additional models reported in Figure 5), and plot them as function of model size in Figure 4d. The result in Figure 4d aligns with the observation in Yang et al. (2020), namely, bias is monotonically decreasing and variance curves is unimodal. Moreover, the model that exhibits strong bias-variance alignment in Figure 4, namely ResNet-56- W16, clearly is outside of the classical regime of bias-variance tradcoff shown in Figure 4d. Meanwhile, model in the classical trade-off regime, c.g. ResNet-8-Wi 11,, does not have bias-variance alignment.
CALIBRATION IMPLIES BIAS- VARIANCE ALIGNMENT
In this section we show how model calibration can imply bias-variance alignment. We start from unifying the different calibration views from previous work, and then show how each of these assumptions implies different version of the bias-variance correlation.
4.1 A UNIFIED VIEW OF CALIBRATION
In previous work, various definitions of calibration have been introduced. In the following, we present a general definition that encompasses a wide range of these definitions. Suppose that there a collection of sub-o- -abbebas (Eieej of (X), where or oXX) is the o-algebra generated by the random variable X. In this section, we refer to the o-algebra generated by a random variable or event using the notation ((.).
In Definition 4.1, the alagebras V; must be sub-o-algebras of (xX). This is because we want the random variahles and events we conditioning to be functions of X. Thus, the conditional expectation in the definition of calibration averages over all data examples that have the same value of function of X.

---

Published as a conference paper at ICLR 2024
BCF (ail
CWCF Sihhej
(Kirsch Gal, 2022, Expoctation of Eg. (6))
This work (Definition 4.2)
(Guo al., 2017, Eq. (2))
Appendix F:2)
(Kirsch Gal, 2022, (36))
(Guo al. 2017, Eq. (3)) (Nacioi et al., 2015) (Nixon al., 2019) (see our Appendix E.3)
This work (Definition 4.2)
Table 4: Summary of the various definitions of ECE; (2
fC heik] and CWCE (ee)))))))))) that have been
proposed in previous work. Our unified definition subsumes these previous definitions.
Definition 4.1 (Perfect (confidence) calibration). A function h X + .M((K) has perfect calibra- tion with respect to { HELK if the following equation holds for all i E [K]
(i X) Σ.j =0 A .:) & Mii x) - y (i x).
(11)
The function has perfect confidence calibration if the following equation holds: E[A(pred, (X) XX (x)) = E[(conf - accs (X)) pxred = 0.
(12)
The sub-o-algebras NE))E control the granularity of perfect calibration. In other words, (11) in Definition 4. says that A(i X) vanishes on average, and {>N hejai specifies the set of samples that we average over. Here are examples of ways of choosing {Eihet
Sample-wise perfect calibration > = (XX). .. In this case, Definition 4. is equivalent to h(i X) P X) for every i [K]. In other words, perfect calibration holds for every sample X.
Pre-image perfect calibration Σ, (h(i ( | X)). This may be the most widely used definition of calibration (Guo et al., 2017; Kirsch & Gal, 2022) and characterizes the calibration averaged over all samples that share common prediction h(- X)..
Bin-wise perfect confidence calibration Σhin = a (I{[hh( x)1H) where M is a positive
integer (Guo et al., 2017). The map x + [Mh(i æ:) assigns x to M bins (if h(i xx) >
for all 2): (0, ] AT. 2 w.... Ar 1] according to the value of h(i x). Then the samples that fall into the same bin are averaged over.
Definition 4.2 (Calibration Errors). If we define A(i ()) as in (11), the expected calibration
error (ECE3 (ECE)).he)))
(2chcj11) and the class-wise calibration error (CWCE) ( of function h X - M((K]) with respect to [E()E]] on the data distribution P E MM x y) are given by
(2.hetk) (h) A E [[A(pred, (X) X)) E Epred, (x)]]
ECE, 7
(E.heE CWCE,
(h) A E |E
X) Ex]l
We elucidate how our unified definition subsumes the various definitions in previous work, and we summarize it in Table 4. In Appendix F.2 and Appendix F.3, show concretely what ECE, (%, heix looks like with respect Xpre and Ebin, respectively.
4.2 CALIBRATION MEETS THE BIAS- VARIANCE DECOMPOSITION
In this subsection we then show that, under the model calibration assumption, there is a correlation between squared bias and variance. Moreover, in the case of imperfect model calibration, the discrepancy between the squared bias and variance can be bounded by the calibration error.
Recall the definitions of bias and variance in Section 2.1. Theorem 4.3 characterizes the discrepancy between the squared bias and variance, and provides an upper bound for it by the class calibration error. Corollary 4.4, which follows the theorem, shows that in the case of perfect calibration, the variance is upper bounded by the squared bias. Moreover, if the model outputs a completely certain prediction (outputs one-hot vector), it is theoretically guaranteed that the squared bias equals variance, i.e., the bias- correlation appears. If the model does not output completely certain prediction but a highly certain prediction, an approximate bias-variance correlation follows.
Theorem 4.3. If { is an ensemble whose mean function h(i X) is E.measurable, then we have Eyix (BVG 2h( X)A(i X) + Px (i) X)- Egha(i x)²,
E E [yix
Pyyx (i X) + Ep x)2 |x] < 2 2CCE (E. hetK) (t).
7

---

Published as a conference paper at ICLR 2024
Table 5: Empirical summary of values for CWCE (2 and the bias and variance, both calculated w.r.t Ebin (where
CWCE !)
Width factor
LHS of (13)
RHS of (13)
1/2
0.64
0.90
0.45
1/4
0.63
0.74
0.37
20 equally spaced bins) from ResNet-8 models on CIFAR-10 across varying width.
1/8
0.66
0.70
0.35
1/16
0.76
0.90
0.45
where CCE, CCEEEiiee (1) A E [E] X) Ei]l is the class calibration error for class If NEERK] ( for total squared bias and variance, the following equations holds
E Eyix (VGho,(X,y)) [ -E [Uncen, (X)
+
Σ E [E X) E.j((i X) Σ]
EE[R]
E Erix [BVG [BVG - Uncen, (X) Σ S CWCE (EEII
(13)
We observe that, without any additional assumptions, M(j]X) is automatically E and D. measurable, If one chooses Enj Egamp = (X),, then [[[y|x[] []] Σ] Byjx[]- this case, Theo- rem 4.3 holds for every example X.
Corollary 4.4. If h has perfect calibration with respect to {E<)ie(K) then E[ | X) E and therefore we have
E Eyix (X,,) Σ5 P (1 X) - (i x)² E
= E [M6i X) Eshali X)) Σg = E [Eo [ha (i X) - ha
X))] E,
(14)
E (BVG
] E [Uncesa (X) Σ] 2 0.
(15)
Moreover if ho oulputs one-hot vector, huve [I = 0. In other words, in expectation E By
X)1 1
1 and therefore
E [ryx (BVG bias equals variance.
Σ]. the squared
Corollary 4.4 shows that / has perfect calibration, then in expectation E|Ey [+] E). the variance
is upper bounded by the bias and the gap between them Unceeo (X). If ho(- X) is highly confident (i.e., max h- [X) ss 1), then Biash.. (X,,)) RS Varin..(x,)) The extreme case that hy outputs a one-hot vector.
Corollary 4.5. If h has perfect calibration with respect to {Ei and ho(i X) + for every 0 (a is either 0 or 1), then E [Ev] [3)).(X.)) (x, (i)] E. + 0.
Corollary 4.5 shows that when ho X) is highly confident (ha(i X) + a E (0,1)), then (b..(X.Y) y) (i) - (x.)(x.,)(1)) + 0 in the mean [[y|i|+ 2,], which is entrywise varianae correlation. Moreover, since
E [E [Bho, - (x)) Eg]
= E [1 = i] - hiill X) - Sho, (i)) + 1[Y (i)((i) X) (hh,, E. [
we have hii X) Sha (1); and if (Y = Σ,) Ri we have h(i X) (i))
(Y = |,,)
Experiments confirm our theory. We now present empirical results that support our theory. We em- pirically verify the inequality (13) across models of varying widths when using Ebin (where we use 20 equally spaced bins) for the definitions of calibration, uncertainty, bias and variance. Note that calibra- tion requires estimating the true probability distribution, and so direct computation for 2.pred or Samp infeasible. Table 5 we empirically verify the relationship between bias, variance, and uncertainty when using Ibin The left-hand side of (13), which is E [Ev]x(BVG (x.,).,.).)).. - UnceR (X) E||
computed from the bias and variance values. The right-hand side of (13) is 2 2CCEE (22hin helx We see that across architectures, Equation (13) of Theorem 4.3 holds.
5 NEURAL COLLAPSE IMPLIES APPROXIMATE BIAS- VARIANCE ALIGNMENT
Neural collapse (Papyan et al., 2020) is a phenomenon pertaining the last layer features and classifier weights of trained deep classification model. This section considers statistical modeling of the
8

---

Published as a conference paper at ICLR 2024
prediction of the network ensemble {h₂) an arbitrary test data (X,Y) that is motivated from neural collapse, upon which we derive bound on the ratio between Biash,, and (X,,),
Modeling assumption motivated by neural collapse. We assume that each model ha in the en- semble can be written as he(- 2) = softmax thr (x)), where = (T, W) denotes trainable parameters. In the above, we refer to (x) E IR as the feature vector, and W as the classifier weight. The neural collapse phenomenon states that during training, W converges to a rotated and scaled version of the simplex equiangular tight frame (ETF) matrix WETF, That is,
K K-1
1 LK-KRR - RIKIK) (16)
x WETFR' R.T, where W = (WFFFF
BTF) WKK
IK
and R E RdxX is an orthogonal matrix (i.e., R = Ix). In above, Ix E IRRX is an identity matrix, and k R is column vector with all entries being 1. We summarize the properties of WETF in Appendix G.I. Moreover, for any training data (æ*rain, yrrain), ). neural collapse predicts that the feature y (a is aligned with its classifier weight, i.c. By (ztrain) sRw glaan for some
8 > 0 independent of (xtrain "etraim."rrinng train..
For a test sample (X, Y). neural collapse does not predict the distribution of its feature 4/ (X). However, it is reasonable to assume that it slightly deviates from the training feature of class Y. This R (SwETT + z)), where is the noise vector, which leads to motivates us to assume that v ETF + v)). Hence, the prediction of the network ensemble {ho}
he(- X) = softmax (aaaax(WEETT ( may be modeled as follows.
Assumption 5.1. We assume (h X)} = (softmax (WETF (owF1F + =])} for any test sample (X,Y) Y), where has i..dd entries drawn according to -BV K-I Gumbel(u, 8).
K
Appendix G.2 shows that the assumption aligns with the observation in practical networks. Bias-variance analysis. Theorem 5.2 computes the entrywise bias and standard deviation under Assumption 5.1, for entries corresponding to the true class.
K/KK
Theorem 5.2. Consider the model ensemble in Assumption 5.1. Let K' = K - c K
and
ØR K" (c- ) (cK' log(ek")-1) + (c- (c- EK'1
[K' 1/-
cK'
===
Then we have (x,)(X (c) 11 and (Xo, (Y) cv where (xx)(x.))()))) and (x.)( are defined in (4) of Definition 2.
ded [ddgc(e) ( + [kk((c)
de
Due to technical difficulties, Theorem 5.2 does not provide bias and variance for entries other than those corresponding to the true class. However, under the special case of binary classification, we are able to provide bias and variance for all entries as follows.
Corollary 5.3, If K 2, then E {1,2) we have (i))( [log(e)c-c+]] (c-1)²
(ho.(X.Y))
where = 25 Furthermore, we have: (1) on the linear scale,
(x.x(x))) is
(e-1)
Shig (xx)(
a decreasing function of and 1.74
(x,)
2s-1 ea
or equivalently, 1.742
Bias (X.3")
Shy (x)
Varing (X,Y)
(2s-1)² 1)? eb
log (2) on the logarithmic scale, log (xring.
E (0.557,2) 2).
Corollary 5.3 illustrates the entrywise bias and standard deviation for binary classification. We prove the approximate correlation between the entrywise bias and standard deviation by providing an upper and lower bound for the ratio of the entrywise bias to the standard deviation.
6 CONCLUSION
We show that bias and variance align at a sample level for ensembles of deep learning models, suggesting nuanced bias-variance relation in deep learning. We study this phenomenon from two theoretical perspectives, calibration and neural collapse, and provide new insights into the bias-variance alignment.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
We would like to acknowledge helpful comments from Aditya Krishna Menon (Google Research) and Christina Baek (Carnegie Mellon University).
REFERENCES
Taiga Abe, E Kelly Buchanan, Geoff Pleiss, and John P Cunningham. Pathologies of predictive diversity in deep ensembles. arXiv preprint arXiv:2302.0077 2023.
Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine-grained bias- variance decomposition. Advances in neural information processing systems, 33:11022-11032 2020.
Alexander Atanasov, Blake Bordelon, Sabarish Sainathan, and Cengiz Pehlevan. The onset of suriance-limites behavior for networks in the lazy and rich regimes. arXiv preprint 1wX11:22/2.12147, 2022.
Christina Baek, Yiding Jiang, Aditi Raghunathan, and Zico Kolter. Agreement-on on-thhht---te--eee Predicting the performance of neural networks under distribution shift, 2023.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical variance trade-off. Proceedings of the National Academy of Sciences, 116632):15499-15888 15854, 2019.
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Verlag, Berlin, Heidelberg. 2006. ISBN 0387310738.
Gavin Brown, Jeremy L Wyatt, Peter Tino, and Yoshua Bengio. Managing diversity in regression ensembles. Journal of machine learning research, 6(9), 2005.
A. Michael Carrell, Neil Mallinar, James Lucas, and Preetum Nakkiran. The calibration generalization gap. 2022. URL https: arriv. org/ abs 22200 01964.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACI-HLLT, pp. 4171- 4186, 2019.
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse imbalanced training. Proceedings of the National Academy of Sciences, (18(43)ee210306) 18, 2021.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural compulation, 4(1):1-58, 1992.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Neha Gupta, Jamie Smith, Ben Adlam, and Zelda E Mariet. Ensembles of classifiers: a vias-nariance perspective. Transactions on Machine Learning Research, 2022.
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan Tibshirani. Surprises in high-
dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022.
Tom Heskes. Bias/variance decompositions for likelihood-brat estimators. Neural Computation, 10 (6):1425-1433, 1998.
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin- Yew Lin, and Deepak Ravichandran. To- ward semantics-basee answer pinpointing. In Proceedings of the First International Confer- ence on Human Language Technology Research, 2001. URL https: wwwweb..wwwbcoom org/ anthology/HOO 1069.
10

---

Published as a conference paper at ICLR 2024
Alan Jeffares, Tennison Liu, Jonathan Crabbé, and Mihaela der Schaar. Joint training of deep ensembles fails due to learner collusion. arXiv preprint arXiv:2301.11323. 2023.
Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and Zico Kolter. Assessing generalization of SGD via disagreement. In International Conference on Learning Representations, 2022. URL https: / /pp.reeiee net/ ddwrrummddddwooooooe0
Andreas Kirsch and Yarin Gal. A note on "assessing generalization of sgd via disagreement". Transactions on Machine Learning Research, 2022.
Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos FemandezzGrannaa, Zhihui Zhu, and Qing Qu Principled and efficient transfer learning of deep models via neural collapse. arXiv preprint arXiv:2212.12206, 2022.
Xin Li and Dan Roth. Learning question classifiers. COLING 2002: The 19th International Confer- ence Compulational Linguistics, 2002. URL https: https:://wwwwwaa/ acwwbb.........................b org/ anthology/ C02-1150.
Tong Liang and Jim Davis. Inducing neural collapse to fixed hierarchy- aware frame for reducing mistake severity. arXiv preprint arXiv:2303.056689 2023.
Licong Lin and Edgar Dobriban. What causes the test error? going beyond bias-variance via anova. The Journal of Machine Learning Research, 22(1))6925-7006, 2021.
Neil Mallinar, James Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward refined taxonomy of overfitting, Advances in Neural Information Processing Systems, 35:1182-11 1195, 2022.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75 (4):667-766, 2022.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: On the strong correlation between duitt--diitribution and in-distribution generalization, 2021.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In 7wenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste- Julien, and Ioannis Mitliagkas. A modern take on the bias-variance tradcoff in neural networks. arXiv preprint arXiv: arXiv: 1810.0891, 2018.
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In CVPR workshops, volume 2, 2019.
Luis A Ortega, Rafael Cabañas, and Andres Masegosa. Diversity and generalization in neural network ensembles. In International Conference on Artificial Intelligence and Statistics, pp. 11720-11743, PMLR, 2022.
Vardan Papyan, XY Han, and David L Donoho. Prevalence neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
David Pfau. A generalized bias-variance decomposition for bregman divergences. Unpublished Manuscript, 2013.
Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXiv:2101.00072, 2020.
Jason W Rocks and Pankaj Mehta. Bias-varianee decomposition of overparameterized regression with random linear features. Physical Review E, 106(2):025304, 2022a.

---

Published as a conference paper at ICLR 2024
Jason W Rocks and Pankaj Mchta. Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models. Physical Review Research, 4(1):113201, 2022b.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-6114 6114. PMLR, 2019.
Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. Advances in Neural Information Processing Systems, 25-27225-27238, 2022.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Machine Learning, pp. 21478-21505, PMLR, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On calibration and out-of-domain generalization. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: / oppnreveeeev ew. net/ forum? iddxxYJ25-yyRR.
Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neural collapse inspired classifier alignment for few-shot class incremental learning. arXiv preprint arXiv:2302.030044 2023.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. In International Conference Machine Learning, pp. 10767-10777. PMLR, 2020.
Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: bias-variance tradeoff perspective. arXiv preprint 1rXiv:2102.006:: 2021.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-28 298344 2021.
12

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Societal Impact Statement
13
B Limitations
13
C Further related work.
14
D Practical applications.
14
E More Empirical Analysis of Bias-variance Alignment
15
E.1
Additional results on rolc of over- paramcterization
15
E,2
Bias-variance decomposition of cross-entropy loss
15
E,3
Permutation test for the residual and the log-bias
15
E,4
Linear vs logarithmic scale
16
E.5
Correlation to prediction uncertainty and logit norm
17
E.6
Effect on the sources of randomness
17
E.7
TREC experiments with BERT
18
Further Results on Calibration and the Bias- Variance Correlation
F.I
Perfect calibration does not necessarily imply perfect confidence calibration Pre-image expected calibration error
F2
F3
Bin-wise expected calibration error
Proof Corollary 4.5
20
E5
Proof Theorem
20
R6 No bias-variance correlation in Kullback Leibler convergence.
21
F.7
Thcorem 4.3 implies gencralization disagreement equality (GDE)
22
G
Further Results Neural Collapse and the Bias- Variance Correlation
23
G.I Properties of simplex equiangular tight frame (ETF)
23
G.2 Verifying Assumption 5.1
24
G.3 Verifying Corollary 5.3: Binary classification
25
G.4 Relationship between Gumbel and exponential distribution
25
G.5 Proof of Theorem 5.2
26
G.6 Proof of Corollary 5.3
27
H Future Work
28
A SOCIETAL IMPACT STATEMENT
This paper aims to expose a peculiar bias-variance alignment phenomenon and characterize it both theoretically and empirically. We do not foresee any negative societal consequences from this work.
B LIMITATIONS
We identify the following limitations of our work:
We only considered the squared loss and cross entropy loss functions. It would be interesting extend our results to other loss functions, such the 0/1 loss.
13

---

Published as a conference paper at ICLR 2024
Our theory is based on the binary classification assumption. We plan to extend it to multi-
class classification in future work.
It would be interesting to study the bias- Strsinarinnce alignment theoretically in an end-to-end
manner, using tools such as the neural tangent kernel theory or a mean-field analysis.
We conducted our experiments in the image classification domain. It would be interesting verify our findings in other domains, such as natural language processing (NLP).
We believe that these limitations do not detract from the overall significance of our work. Our findings provide new insights into the bias-variance alignment phenomenon, and we hope that this paper will stimulate further research on this important topic.
C FURTHER RELATED WORK.
It was shown that statistics such as accuracy (Miller et al., 2021) and disagreement (Baek et al., 2023) are highly correlated when contrasted across in-distribution and out-distribution data. This points at a potential extension to our work to consider how our findings translate to the out of domain data.
Gupta et al. (2022) investigate the thcoretical underpinnings of ensemble methods for classification tasks, extending the bias-variance decomposition to derive generalized laws of total expectation and variance for nonsymmetric losses. Their work sheds light on the mechanisms by which ensembles reduce variance and potentially bias, providing valuable insights for improving the performance of ensemble classifiers. Ortega et al. (2022) provide a comprehensive theoretical framework for understanding the relationship between diversity and generalization error in neural network ensembles. They analyze the impact of diversity on ensemble performance for various loss functions and model combination strategies, offering valuable insights for designing effective ensemble learning algorithms. Brown et al. (2005) focus on managing diversity in regression ensembles. They introduce control mechanism through the error function, demonstrating its effectiveness in improving ensemble performance over traditional methods. This work provides insights into systematic control of the iiasvarrianc--ovvarince trade-off in regression ensembles. Abe et al. (2023) challenge conventional wisdom on predictive diversity in deep neural network ensembles. While diversity benefits small models, the authors find that encouraging diversity harms high-capacity deep ensembles used for classification. Their experiments show that liversity-encouragee suggesting that the best strategy for deep ensembles may involve using more accurate but less diverse component models. In contrast to traditional ensemble methods, deep ensembles of neural networks offer the potential for direct optimization of overall performance. However, Jeffares et al. (2023) reveal that jointly minimizing ensemble loss induces base learners to collude, inflating artificial diversity. This pseudo-diversos fails to generalize, highlighting limitations in direct joint optimization and its impact on generalization gaps.
regularizers hinder performance,
D PRACTICAL APPLICATIONS.
We believe that our findings on the bias-variance alignment can be used to develop new methods for validating deep learning models and selecting generalizable models in practice. One practical application is estimating the test error of deep learning model using variance. This is possible because our finding is that bias and variance are aligned, and so we can estimate bias from variance. This means that even when the true labels of the test data are unavailable, we can still get good estimate of the test error by measuring variance over multiple models on the test data. Compared to Jiang et al. (2022) which analyzed the alignment between disagreement and test error across the entire dataset, our method is a per-example approach and thus enables example-level validation of deep learning models. This is a novel way of validating deep learning models and selecting generalizable models in practice even when the true labels of the test data are unavailable. Additionally, our method is simple to implement, so it could be easily adopted by practitioners. Moreover, inspired by our result, one could consider practical algorithms leveraging the observation of bias and variance alignment. As one example, one can consider routing between ensembles of models. Given two ensembles of models, one could dynamically route between such two ensembles, depending which one yields lower variance. Given the above possible applications, we believe that our work has the potential to make significant contribution to the field of deep learning.
14

---

Published as a conference paper at ICLR 2024
E MORE EMPIRICAL ANALYSIS OF
ALIGNMENT
E. ADDITIONAL RESULTS ON ROLE OF OVER PERE-ARRMMETERIZZAIOON
In Section 3.3 we showed that the bias-variance alignment phenomenon becomes more pronounced for over-pane per-parameterized models, by plotting sample-wist wise bias-variance for three models of varying sizes in Figure 4. Here we present results on five additional models of varying size Figure 5 that complement the results in Figure 4.
Tamhat-l-W12
Fanch st-3-WWW
1319
1014
1331
109
13-1
1321
1014
1010
100
1344
32-4 1ird
107
10-9 Inc: 10-1 1819 1044 squared
squared
Elsd squared
squared
12:l
1211
10
60-13
10:14
1002
10:1
1971h
20111 20-1)
30-17 10r11
985 stac
0: oquared
Bl3 Squiteed
B40 squiieedd
Figure 5: Sample-wise bias and variance for networks of varying scale trained on CIFAR-10.
E.2 BIAS- VARIANCE DECOMPOSITION
PROSS-ENTROPY LOSS
Deep neural networks for classification tasks are typically trained with the cross-entropy (CE) loss. Here, we investigate whether the bias and variance from decomposing the CE loss also exhibit the alignment phenomenon. The risk with respect to the CE loss can be decomposed as follows (Pfau, 2013):
(ey log(ho(- x))) (ey Rb. (X) DoDKL (h(- (X) |I ho(- X))).
(17)
Risk
Blas?
Variance
where DKL denotes the KL divergence. In the above equation, h(-lX) is obtained by taking the expectation the log proobbbbbiilstis and then applying a softmax function. In other words,
exp Eg log(he X)))
hit X)) Σn exp Eg log(he X)))
(18)
Intuitively, h(-lX) represents the average prediction under the KL divergence, assigning a probability proportional to exp Eg log(ho( (i X)) to each class i. The bias term measures the KL divergence between the true distribution ey and the average prediction h--[X), quantifying the deviation of the ensemble's 's mean prediction from the actual class distribution. The variance term, the other hand, captures the average KI. divergence between the individual predictions in the ensemble and the mean prediction [1- XX, reflecting the overall variability of the ensemble's predictions.
In Figure 7 we present the sample-wise bias and variance from decomposing the CE loss under the same setup as that in Figure 1b. In other words, the only difference between Figure 7 and Figure 1b is that the bias and variance are computed from decomposing the CE and MSE loss, respectively. It can be seen that the bias no longer aligns well with the variance. In Appendix F.6, we theoretically explain this phenomenon.
E.3 PERMUTATION TEST FOR THE RESIDUAL AND THE LOG-BIAS
We perform linear regression of log Varia,, on log Biash, for the following models and datasets: ResNet-56 (on 100, ResNet-8 (on CIFAR-10), 10), ResNet-50 (on ImageNet), and ResNet- 10 (on 110) We would like to test whether the residual is linearly correlated of the exogenous variable log (s...)) To this end, we perform permutation tests with the Pearson's
15

---

correlation coefficient on the residuals and their corresponding log Biasko values. The null hypothesis is that the residual and log (iasse.(...s)) ) not correlated. We plot the null distribution, the statistic, and the p-value of the test results on the four models in Figure 6. The results show that Published as a conference paper at ICLR 2024
ResNet-56 (CIFAR-10)
ResNet-8 (CIFAR-10)
ResNet-50 (ImageNet)
ResNet-110 (CIFAR-100) p-value: 0.9947
p-value: 0.9949
p-value: 0.9957
p-value: 0.9999
Null Distribution
Distribution
Null Distribution
Null Distribution
75
Statistic
Statistic
Statistic
Statistic
20
50
20
25
0002550000 0.025
-0.025 0.000 0.025
-0.02
0.00
-0.0250.000 0.025
Figure 6: Null distribution, the statistic and the p-value of the permutation test results for the residual against log Bias ha (ar.))) in linear regression of log Varing.(z.:111 on log (Fr,)..
ResNet-50
ResNet-50
10a
0.20
10-3
Variance
Variance
0.15
10-6
10-9
0.05
0.00
10-6
10-4
10-2
100
0.0
0.2
0.4
0.6
0.8
Bias squared
Bias squared
Figure 7: Sample-wise bias and variance of the CE loss.
Figure 8: Sample-wist bias and variance plotted in linear scale (with correctly classified samples only).
the null hypothesis is not rejected, suggesting that it may be true.
E.4 LINEAR VS LOGARITHMIC SCALE
In Section 1, the bias-variance alignment is presented first in the logarithmic scale (see Eq. (1)) and subsequently in the linear scale (see Eq. 2). Here, provide rigorous analysis their connections. In addition, we explain the implication of the bias-variance alignment when plotted in the linear scale, which is complemented by empirical results.
Connection between bias-variance alignment in linear vs logarithmic scale. First, we provide formal statement on the connection between the linear and log scale of the bias-variance alignment. Proposition E.1. If log (s...) log Bias Biasha +Eng + 9; where e, is independent of
Bias? he (x... and (n-unit(()))ee)) [ec] = O, we have
Varino.aas.s)) Ch, Bias (a + +Dig Bias ((4.01) lh
where Ch Fno Ee-Una((()))e > O. Dng = e Fio 0, E E[eey] and Ep-unt((()((n))
Proof. We exponentiate both sides of log (aran) log Biash + Eno + es where [.-uni((a)eee] and obtain
Varing.(raa)) = Eng ( : Eno Biash (T. E(e5t])
= =Che Biash + Dio Biass (wr.a)) Bi
where 7/ has mean 0 by definition.
16

---

Published as a conference paper at ICLR 2024
ResNet- C-010-0-6 U-W
RecNat- W16
ResNet-11c.he
Verferne
Ven cenanees
0.2
0.10
022
0.03
0.1
61
0.00
0.3
0-4
0.4
squared
squared
squared
(a) ResNee-8-W1
(b) ResNec-8-WW6
(c) ResNeet-
Figure 9: Same as Figure 4(a-c) but plotted in linear scale and with correctly classified samples only.
Sample-wise bias and variance plotted in linear scale. Unlike in the log scale where the noise term (see Eq. (1)) is independent of the bias and variance, in linear scale the noise term & is multiplied by factor that scales with the squared bias (see Eq. 2). This implies that instead of aligning along straight line, sample-wise bias and vairance in linear scale has a cone-shaped distribution. That is, as bias increases, an increasingly wider range variance covered by the samples and such range forms cone. To illustrate this, we regenerate the plot of Figure 1b but with linear (instead of log) scale in both the and y axis, and the result is shown in Figure 8 (we also removed the incorrectly classified data points from the plot). Furthermore, to observe the effect of model size on ftas-variance alignment linear scale, we regenerate the Figure 4(a-c) with and y axis switched to linear scale and present the result in Figure 9.
E.5 CORRELATION TO PREDICTION UNCERTAINTY AND LOGIT NORM
Figure lb demonstrates that the bias and variance of varying sample points exhibit the alignment phenomenon for points that are correctly classified. Here, addition to the correctness of prediction, also examine the relation between the alignment phenomenon with the prediction uncertainty and the logit norm.
ResNet.50
uncertainty
logit_norm
10-4
10-4
10-4
Vararce
10'9
Variace
20
0.4
Van airrrce
10'7
100
0.5
120
Carrectly dossified: lalse
10-19
10-49
10-19
10
10 10 10a
10
10
100
10
10
10*
Bias squared
Bies squared
Bliss squared
(a) Correctness
(b) Uncertainty
(c) Norm of Logit Vector
Figure 10: Same as Figure 1,, but with each sample colored according to (a): Correctness of model prediction, (b): Uncertainty of model prediction, and (c): L² norm of the logit vector.
Figure 10a is the same as the one in Figure 1b for the reader's reference. In Figure 10b, show how the uncertainty in model predictive distribution, i.c., (nce; (see Definition 2.2), correlates with bias and variance. It can be seen that samples with large variance are those with large uncertainty scores. We give formal relation between bias, variance, and uncertainty in Theorem 4.3. Finally, Figure 10c shows the lack of correlation between bias/variance and the g norm of the logit vector.
E.6 EFFECT ON THE SOURCES OF RANDOMNESS
The decomposition of the generalization into the summation of bias and variance requires one to specify a source of randomness in obtaining a collection of models. In classical bias-variance tradeoff, this source of randomness is usually taken to be the sampling of the training dataset. Correspondingly, the numerical estimation of bias and variance can be achieved by sampling a given dataset via bootstrap (see e.g. Neal et al. (2018)). This is the approach that we adopt in all numerical experiments in this paper, other than those in this section. On the other hand, modern deep networks often have other sources of randomness as well, such as the initialization the model parameters, random sampling of the batches in the training process.
17

---

Published as a conference paper at ICLR 2024
In this section, study whether the emergence of bias-variance alignment is due purely to the randomness in sampling the training dataset, or other sources of randomness may also give rise to a similar phenomenon. Towards that, conduct experiments to train multiple deep neural networks without data bootstrapping. In such cases, the randomness the collection of networks comes only from random initialization and data batching. The result is shown in Figure 11. Comparing it with Figure 4, where the only difference lies in the bootstrapping of training dataset, it can be seen that the source of randomness have very small impact on the bias-variance alignment phenomenon.
10
10
0310
(a) ResNeet8-WI
(b) ResNee-8-W16
(c) Re--ee--100-W16
(d) Varying model size
Figure 11: Same as Figure but without bootstrapping of training dataset. Hence, the randomness in computing the bias and variance comes from random initialization and data batching, and there is no randomness in sampling of training dataset.
E.7 TREC EXPERIMENTS WITH BERT
We show that NLP datasets with Transformer-based model (Vaswani al., 2017), more specifically BERT (Devlin et al., 2019), the bias-variance alignment observation holds. This is shown Figure 12 where we consider the TREC dataset (Hovy al.. 2001: Li Roth, 2002) with its fine-grained labels (i.e., 47 classes) and vary the number of layers in the BERT model.
In this experiment, each of the ensembles consists of 20 BERT models. In each case, each of these models was initialized from the same pre-trained checkpoint, and trained for 20 epochs with learning rate of 2 10 using Adam. We use a polynomial decay learning rate schedule with the number of warm-up steps set to be 10% of the number of total update steps. Training batch size was set to 8.
10-1
10-2
Variance
Variance
10
10-5
10-6
10-2
10"
10-5
10 -3
10-1
Bias squared
Bias squared
BERT with layers
(b) BERT with 8 layers
Figure 12: Sample-wise bias and variance of BERT fine-tuned on TREC. We confirm the bias-variance alignment phenomenon.
F FURTHER RESULTS ON CALIBRATION AND THE BIAS- VARIANCE CORRELATION
F.1 PERFECT CALIBRATION DOES NOT NECESSARILY IMPLY PERFECT CONFIDENCE CALIBRATION
Perfect calibration does necessarily imply perfect confidence calibration. To illustrate this, consider the following example: let y [1 and let X be a uniformly random variable on {1, 2). Let the probability of Y given X be PP(i X) 1(X A i). and let the classifier be
18

---

Published as a conference paper at ICLR 2024
defined h(i X) 1 =t}. In addition, let Σ, represent the trivial o-algebra for all in the set 1, 2. In this case, we have ( X) Σ A X)] = E 1(X = - 1] = 0, which
means that h has perfect calibration with respect {2, = 1.2. However, since pred, (x) have (pred, (X) X) E'red, = [ (pred, (X) X)] = ( x)] = 1. Therefore, h does not have perfect confidence calibration.
This is not true even for preimage perfect calibration Drre
Consider the following counterexample
3
4
hii 2)
1
0.3
0.25
0.2
0.25
2
0.3
0.5
0.2
0
(y)x(1|a)
0.4
0.25
0.1
0.25
2
0.2
0.5
0.3
0
We set P(X = 1) ( = 2) = 1/2. It is clear that
E [M(i X) P (i) X) 11i X)] =0
(19)
holds for for 2,4. If [h(1 X) Py|x(1 X) h(1 X) 0.3] 0.3 0. [ (1 X) h(1 X) 0.3 0.3
1. we have hii
X) 0.3. Therefore.
0.4+0.2 0. Similarly, can show that (19) holds for 4. For 20 >>
pred, (X)
a (h(pred (X) x)) (X). Note that in this example, conf (X) can take only two values 0.3 and 0.5. Since
B[(conf (X) acc)(X)) confx(X) 0.3]
(20)
[coof (X) - accı (X)) X = 1] = 0.1 + 0,
(21)
perfect confidence calibration not satisfied.
F.2 PRE-IMAGE EXPECTED CALIBRATION ERROR
The expected calibration error (ECH) with respect to {Epre) RE(K recovers the ECE from Equation (2) in (Guo et al., 2017). Recall the definition of conf and acc in Section 2. The ECE with respect to (EP ]æe[K]
(Xxhenk
ECEp
(22)
E EE A (pred, X) Spre
pred, (x)1
(23)
E [[A(pred, (X) X) luprrdd, (X) X)]]
(24)
EE [[(confa (X) acca (X)) conf, (X)]]
(25)
Equation (25) follows from A(pred, (X) X)
conf (X) acc, (X) and h(pred, (X) X)
(X). This recovers the definition of the ECE in (Guo et al., 2017, Equation (2)).
F.3 BIN-WISE EXPECTED CALIBRATION ERROR
The expected calibration enror (ECE) with respect to (Ibin) ][[[]] recovers the ECE from Equation (3)
in (Guo al., 2017). Let E, represent the event j-1 confx s 1 } = {[M conf, (X)]
j).
19

---

Published as a conference paper at ICLR 2024
The ECE with respect to {
he[R]]
is
ECE2
(26)
Σ P(E,)E
E
[lA(prod, X) E ppred,
Eg]
(27)
JEIM
Σ
acc acCa(X)) (X)c((X)) Ey]l
(x))))))))))) | [M confA (x)](l Fgy
(28)
JE[M]
Σ
P(B;) [B[[confa(X)
(29)
JE[M]
Equation (28) follows from (pred,, (X) X) confx confx(X)= acca (X) and [ (X)
X]] [M confx Equation (29) is the ECE with respect to {>, bin jiEjK] on the population P(X,Y) Y). one wants to estimate the ECE from an empirical distribution formed by sampling n i..d.
samples {<r v.) [e(n from P(X, Y), then P(E,) is
[Byl where Bj denotes the set of the indices of
the samples whose confidence falls into the bin 1-1 1.
Under the empirical distribution, we have
M
M
E faccs (X) E,1
1 y
(30)
Byl
ElconL (X)
Eg]
conf
(31)
By
iEB,
We recover the definition of the ECE in (Naeini al., 2015) and (Guo al., 2017, Equation (3)).
F.4 PROOF OF COROLLARY 4.5
Proof of Corollary 4.5. By (14) and the bounded convergence theorem, we have
E [x(x - o(i) Ex]
(32)
I.. Eo [ho X)(1 ha X))] E.] + 0.
(33)
Let us write 8(1) = (i) (i). It follows that
E [x(x)3(1) - E.]
(34)
E
Eyjx ((o(i) + 5(1)) - o())) E.]
(35)
E
Ex]x ()()(((()))) + 22 E.
(36)
2 E Exix
[2 E 2.
(37)
As result, obtain E [Byx () Σ. 0, which implies E [Ey]x(5(1)] Σ.] 0 since I
convergence of random variables implies L convergence.
F.5
PROOF OF THEOREM 4.3
Proof of Theorem 4.3. We have
Biash (1) h(i x)2 + 2 2-1 0th(i X),
(38)
Vari (X,)) (i) = Egho(i) x)² - hi( x)2.
(39)
Therefore we get
(x.)) Varin..(x.x) =2 (hii Xx) 11Y ith(i X))
(40)
+ +1Y = - Esho(i X)2
(41)
Taking the expectation over the conditional distribution of Y X yields
Eyix
- Varina,(x,)) (i)
(42)
= 2h X) X) + Pyix X) - Echo(i xy2.
20

---

Published as a conference paper at ICLR 2024
Then further take the conditional expectation E/- Σg] and re-arrange the terms, and obtain
E [ylx [Bixs (X., (X.Y) (i) (x (i) (y X) + (i xx2² ]
(43)
-2h(i (A) X) E.]
Taking the absolute value and then the outer expectation gives
E E
Exlx [Bias (xx,)) - Varia (x.,7 Prix(i| X) + Egho(i x)2 Σi]
= 2E [h(a X)
[ X) Ei]l
(44)
s 2E [E [i X) [.]]]
Summing (43) over i [K] and taking the outer conditional expectation [B|: [X] gives
E Exjx Bias (X..)) Varing.(Xy))
Σ
(45)
= 1 - E Er Be (: XII = +2
Σ
E [E X) E.J/(i X) [
(46)
KE[K]
Re-arranging the terms and taking the absolute value yields
Erix x [Bias (X,Y) Varing + Ey x Eollhoo X)] Σ
(47)
E (i X) Ed]l h(2 X)] s CWCC, (E))EeA) (
$
(48)
xE[K]
F.6 No VIRLNNAR CORRELATION IN KULLBACK-LEIBEE CONVERGENCE,
The expected Kullback- Leibler (KL) divergence Eg (RI,eEy || hol X)) can also be decomposed (Heskes, 1998; Zhou al., 2021; Yang et al., 2020) into the bias Biasho.(X,)) and the variance Varing.(X,))
2 Bias hx,,8) = (ey |l (X),
Variny,(x,y) DKL || ho X)) Biash
where the mean function h and the partition function thereof are defined by 1
h(i X)=
Z exp(Eo log ha X)),
(49)
Z =
Σ KE|K]
exp(E, log (i X)).
(50)
We can see Vari,,, = log Z.
The following Proposition F.1 demonstrates that there is correlation between bias and variance in KL divergence, unlike in mean squared error. Specifically, we prove that the ratio of expected bias to expected variance in the decomposition of KL divergence
take any value in the range of (0, co). Proposition F.1. There exists a data distribution PXX, Y) such that for any value r E oo), there is an ensemble {ha j such that its mean function Eghg has samplewise perfect calibration, and the Evix Bi
ratio of expected bias to expected variance under the KL divergence Evix Vatiag
Proof. Suppose that there are 2 classes and for every v, P(i x) 1/2 (i 1, 2). Moreover,
detine h1 (1 x) h2 x) = e and h (2 x) h2(1 x) = E, and set 0 to uniformly
random variable on {1,2]. Then the mean function satisfies h(1 x:) = h2(2 x) 1/2, which does not depend on E. The expected bias Erjx Biash is log 2. The partition function Z
equals 2 exp((log + log(1 (1 e)), from which we obtain the variance Eyjx Varino.(x.)) Variag,(X,y) = - log log s(1 - =). As E - 0+, the variance Varino.(X,y) tends to 30. As E 1/2, the variance Variae.(x,)) vanishes. Therefore the ratio Eyix Biash, can be any value in the range of (0, oo).
Byix Varing
21

---

Published as a conference paper at ICLR 2024
In Proposition F.1, can let approach O. In this limit, there exists collection of ensembles {ho te (which depends on r, hence the term "collection") such that By x (
Conversely, we Eyix Bias ((X.)) - Therefore, either bias variance can let approach infinity, in which case Eyix ((X,) can be arbitrarily large relative to the other, implying that there is no alignment of bias and variance under the KL divergence.
Eyix (X.0)
F.7 THEOREM 4.3 IMPLIES GENERALIZATION DISAGREEMENT EQUALITY (GDE)
In this section, we show that Theorem 4.3 implies the generalization disagreement equality (GDE), which is the main result of (Jiang et al., 2022; Kirsch & Gal, 2022). We first recap the GDE using the notation of this paper. We begin with defining the test error, disagreement, and class aggregated calibration error (CACH) originally defined in (Jiang et al., 2022; Kirsch & Gal, 2022).
Definition F.2 (Test error, disagreement, and class aggregated calibration error (Jiang et al., 2022: Kirsch & Gal, 2022)). Let (he X' - )M[K]) be an ensemble of trained models, each of which has deterministic prediction, i.e., h- æ)) is one-hot vector for Væ: E X. Let h(- a:) A Egho( xx) be the mean function of {ho}. Then, the test error, disagreement, and class aggregated calibration error (CACE) of h are defined follows:
TestErrp (he) Ex ( ( X) H. Disr (ho, hor) = B(x (ho * ho (X))),
CACE '''
P(Y i, X) 4) q Σ [E[K]
Σ P(h(i X) q) dg. RE[R]
Note that while the test error Test Errp and disagreement Disp(ho, har) are expected values over P, they still have randomness due to 0.
Moreover, note that Jiang et al. (2022) use integer i E [K] to denote the prediction of hg. However, we use a one-hot vector 6 R.K, We will see the mathematical convenience of representing the prediction with one-hot vector in proof of Theorem F.3. In particular, our proof of Theorem F.3 shows that in expectation, the disagreement is equal to the variance (defined in Equation 8) and the test error equals half the risk (defined in Equation 10).
Theorem F.3 (Theorem 4.2 of (Jiang et al., 2022)). he outputs an one-hot vector (as assumed in (Jiang et al., 2022)) and 0, O' are ii.dd, The following inequality holds:
Eeger or [Dise(hg he }] Eg [ [Tee (he)]] S CACEp,
If the ensemble {ho} satisfies the pre-image perfect calibration (EiA(i
| X) E, 0. for
di E [K), the following generalization disagreement equality (GDE) holds:
Eg,er [hg, her ]] = Egg [TestErry(he]]]
Proof. We first show that the disagreement is equal to variance in expectation:
(Xa (X)I
Eg Disp (ho, hor )] = Eo EX
=Exxyy~p
Eoll (X) 112] [hE (X XI]] =xx,y)> Varino
Next, we show that the test error is equal to half the risk in expectation:
|10( X) ex
Eg (TestFErr(hoo]] Egg E(
Riskha.(X.Y)
Bias
Varina
=E(xY
-Ex
2
2
2
22

---

Published as a conference paper at ICLR 2024
We obtain the following equation by subtracting the above two equations:
Eg.0r (Disp (he. her -ol (rrTeEeE))))
Bias (X..)
Varino,(X.M)
Varino.
2
BVG (X,,) -x
2
Apply Theorem 4.3 with Σ; E (h(i) X)) and using Uncero (X) 0 (because ho(: 2)) a one-hot vector for Vo E X), we obtain
is
BVGho,(X.Y)
EEA () Ee X)]
(51)
2
KE[K]
We see immediately that if ( X)
- = 0 for Vi E [K], the GDE is satisfied.
The right-hand side of Equation (51) equals
(E X) Ep""]((i X)]
KE|K]
Σ (E(K)
(q - P(Y =i h(E (X) a)) ql((h(i x) = g)dq
7 Σ P( 9)
Σ SE[K]
XX) v)
qdq.
SE[K]
In the first equality, we expand the expectations. To compute the outer expectation, we condition on the prediction X(] returned by the model for class index and integrate with respect to the conditional probability distribution h(] The inner expectation is taken over all X such that the model for class index i returns q, the value that we condition on. In the last equality, apply the conditional probability rule PPY = i, Aii X) 4) IP(Y i h(i X) = ())P( X) = 4).
Taking the absolute value gives
Eg.or Diss her - Eg (ros))rr
6
9 Σ
P(h(i X) = 9)
Σ
P(Y = i, X) g)
gdlg
EE[K]
kE[K]
9 Σ P(h(i x)
Σ
P(Y = i, 9)
dq
XE[R]
kE(K]
CACE
where the last inequality uses E [0, 1].
G FURTHER RESULTS ON NEURAL COLLAPSE AND THE BIAS- VARIANCE CORRELATION
G.1 PROPERTIES SIMPLEX EQUIANGULAR TIGHT FRAME (ETF)
ETF has the following properties: First, it is symmetric. Second, the inner product between any two distinct columns is equal to ক- Third, this pairwise distance is maximized, i.e., there does not exist any matrix where the inner product between any two distinct pairs of columns are smaller than K-I-
23

---

Published as a conference paper at ICLR 2024
G.2 VERIFYING ASSUMPTION 5.1
As stated in Assumption 5.1, we assume that the logits of a deep network for any test sample (X, Y) are drawn from the following distribution:
softmax (W (swy
ETF + +2))
= softmax (V (
K
1 [KK
[k1kk
$1 K - (or
K 1x +
K
use the fact
IK
1
1x =0)
K
=
softmax
K KK-1
K
IK
K
[klk
$1 K - cy t0 (ex+t)
(52)
= softmax
K-1 BY +
sK
K K-1 U
K
K
1x1k
(8 v
K
K-1 K
8K
K
= softmax
K- eY
+
K 1
where the final equality follows from the fact that the term V K-1 K k (ov x +u) is parallel to 1g and the addition of this term does not change the value of the softmax function. In particular, above is random vector with i.i.d. entries drawn according to -3 R-T
Gumbel(J,, 3). In this section, we verify this assumption from two perspectives. First, we will plot the distributions of logits in practical neural networks, and show that they align with (52). Second, we will show through simulation that, if the logits are generated according to (52), then we observe bias-variance alignment.
Distribution of logits. From (52), the logits corresponding to the correct class (i.e., Y) and any incorrect class Y * Y are given by
sK K-1
K 1 uy, and
K K
UY
(53)
respectively. In particular, since Gumbel distribution has unimodal shaped probability density function, the distribution both the positive and all the negative logits have unimodal shape according to (53). To verify that this aligns with the practical observations, we calculate the distributions of logit values on various datasets and model architectures, for both positive classes and negative classes. The results are presented in Figure 13. We observe unimodal logit distributions for both positive and negative classes in all cases. On the other hand, one may notice that while (53) predicts the positive and negative logits have different biases but the same variance, in many cases from Figure 13, the positive and negative logits have notable different variances. Hence, Assumption 5. is used as simplified model that makes our theoretical analysis tractable, but not meant perfectly model the distribution of logits in practice. We will show next that such a simplified model is sufficient for producing the bias- varianee alignment phenomenon that we observe in practice.
Synthesizing bias-variance alignment. To justify Assumption 5.1, we synthetically generate a collection of logit vectors according (53), and plot the sample-wise bias and variance obtained from the logit vectors. Specifically, given any number as the number of samples, and K as the number of classes, we first generate collection of random labels where each label is drawn uniformly at random from [K]. For each label, we sample T logit vectors independently according to (53) (for the Gumbel distribution, we take μ = 0 and 3 1). Here, T' is interpreted as the number of independently trained models for estimating bias and variance.
The results with n = 200, K = 2, and T 10, under varying choices of s E (5, 10, 20, 100} are reported in Figure 14. In all cases, observe clear biri-variance alignment. In particular, did not draw correctly and incorrectly classified samples in different colors (as in Figure 1b) because all the samples in these cases correctly classified.
24

---

Published as a conference paper at ICLR 2024
classes Negalive classes
Positive Negalive classes
Positive classes
0.08
0.08
Negarive classes
Density
0.06
Densiyy
0.06
0.04
0.04
0.02
0.02
-20
20
-20
20
40
20
40
Logit
Logit
Logit
(a) CIFAR 10 ResNet
(b) CIFAR ResNet 56
(c) CIFAR 10 ResNet 110
0.10
Positive
0.08
Negative clarses
0.06
Negative classes
Negative classes
Density
Density
0.00
0.04
0.02
0.02
0.00
0.00
-20
20
-20
20
60
-20
Logit
Logit
Logit
(d) CIFAR 100 ResNet
(e) CIFAR RReNNet
(f) CIFAR 100 ResNeet
0.20
0.20
Positive classes
0.25
Pasitive classes
Positive classes
0.15
Negative classes
0.20
Negative classes
0.15
Negative classes
Denstor
Densiy
0.10
0.15
0.10
0.10
10
20
30
20
40
20
40
60
Logit
Logit
Logit
(g) ImageNet- - MobileNet V2
(h) ImageNet EfficientNet BO
(i) ImageNet ResNet
Figure 13: Distribution of logits for positive and negative classes.
10-21
10-105
10-7
Variance
Variance
Variance
Variance
10-150
10
10
10-28
10
10-9
10
10-21
10-17c
10
12
10-7
10
10
474 100
Bias squared
Bias squared
Blas squared
Bias squared
Figure 14: Sample-wise bias and variance for synthetic data generated according to (53). From left to right, is varied in the set (5, 10,20, 100).
G.3 VERIFYING COROLLARY 5.3: BINARY CLASSIFICATION
We note that the Neural collapse theory relies the binary classification assumption. To ensure that the bias-variance alignment results hold for such a setup empirically, we construct a binary classification problem based on the CIFAR-10 10 dataset: each example in the first five classes is assigned label 0, and each cample in the last five classes is assigned label 1. We call the resulting dataset CIFAR-2. The results are shown in Figure 15.
G.4 RELATIONSHIP BETWEEN GUMBEL AND EXPONENTIAL DISTRIBUTION
Lemma G.1. Let Gumbel(p, 3). Then, X/8 Exp(er/8).
Proof. Recall that the cumulative distribution function (CDF) of the Gumbel distribution is given by
P(X < x) -x--1/1
(54)
25

---

Published as a conference paper at ICLR 2024
10
Correct
10-
Correct
False
False
10:4
10
Variance
10
1.ogg 12 www.eeleeeee
Variance
10-7
10
10
10
10
10 11
10
10-10
10-6
10
10-14
10-9
10"4
10
Blas squared
Blas squared
(a) ResNet
(b) ResNet 14
10-1
Correct
10
Correct
False
False
10-3
True
10-1
True
Vanaacee
10:5
Variance
10-3
10
10
10
10-11
10-13
10-13
10-9 squared
10-1 10-1
10-12
10-h
10-1
10"
squared
(c) ResNet 56
(d) ResNet 110
Figure 15: Squared bias and variance computed based on various model sizes on the CIFAR-2 problem. See Appendix G.3 for details.
Thus we get
P(e- X/8 se-z10)= (x-11/1
(55)
Substituting = =x/3, we get
P(e P(e- X/8 St) = - -teii0
(56)
This is the CDF of Exp(eu/9).
G.5 PROOF OF THEOREM 5.2
Proof of Theorem 5.2. As the first step, we compute the output of function he
(x()) sofumax (X))
softmax ( ( ETF R
(sw
ETF
ETF
+z)))
softmax
V K K-
8
ex
- 1k t0)
(57)
K
K
sK K-TeY +
K
softmax
K 1-
Let us denote t A ho X). Without loss of generality and for the ease of presentation, we label the Y-th entry as the first entry (Y = 1). Moreover, we introduce the shorthand notation K-1
26

---

Published as a conference paper at ICLR 2024
Then we have
sK
softmax
K 61 + u
(58)
ae
20
eux
=(Ec") aee + e +
edk
ae gd +
eukx"
ae eu + + euk
where = esK/K", In light of Lemma G.I, since - -Bu, ~ Gumbel(p, 8) are ii.d., then vs ex ~
Exp(A) where X - el//3
Let us look at the first entry wa of softmax(rei u). equals
ae"1 11
W1 ac + 6d2 +
6
+ e t............. te
a a (K - 1)F
c F
(59)
where c K-1 e Kjj! 1),2) follows the F distribution (this is because e Exp(A) x2x
Kr and (e (e"24 (ex) ((2(K))))))))
(e"2 F(2(K
ont /2
e²
The expectation of wc/cis given by
E
Er-D(2K'2)
(60)
c+
x K'-1
K' ("Betal K" K', J
(c+ x) + 1/K')'41 de
(61)
K' (c- )x+)
-k'
(cK' - K' log(ck") -
(62)
ck' -1
(c Kr
-K' K-
(K" - -)(e-)c -
(63)
K'
j
(c)
(64)
As result, the squared bias of the first entry W1 is
Bna,( = [Ewn 1|
lc0x (c) -1 1/-
(65)
To get the variance of the first entry w1, we compute its second moment as the first step:
1
= cc2 d dc
1
= 2 (c)
E L = dZ
E
(66)
(c F) F)2
C
dc
Therefore, it follows that
e² ox (0)2 d
Vari (1)= Biw, 2
(67)
dc
which yields
(ho.( = /aarno. = ( (c)
+ Φr (c)2 --((dor((() de
(dox:
(68)
G.6 PROOF OF COROLLARY 5.3
Proof of Corollary 5.3. If K = 2, get Ep-r(22))++))
c log(c)-1 As in the Proof of Theo- (c-1)²
rem 5.2, without loss of generality and for the ease of presentation, label the Y-th entry the
27

---

Published as a conference paper at ICLR 2024
first entry (Y 1). We then have the following expectations:
By [[] = c(c - -og(e)) - 1)
Bu[wz] = -c clog(c) +
(c 1)2
(c-
1
?² - 2c log(c) -
BF-F(2.2)
BF-2222)
(c + F)2
de
F
(c 1)3c
To obtain the variance of wi; W2, we first calculate the second moment:
B, [u[l]] 2 2c 2elog(c)
1)
(69)
(c-1)0c
Bn[wg]] = Eu((1 - )] c² 2=-2clog(c)
(70)
(c - 1)3
As result, we have
c c((c- 1)2 (c))
Var, [w]] = Var, [wg]]
(71)
(c- 1)4
Therefore, we obtain
(x,,(X,)))1) = |E,, [wi] - 1| = [log(c)--cr - c
(c (c- 1)2
Bn (X,x = [wyl - 0 -Bxo,(X,))11)
Velle- - 1)2 c log (c))
(x,) (x,r Var, [wy
(c - 1)2
The ratio
Sh (x.,(()
|---olo(c)+11 is a decreasing function of E (1, and (dle-1))--coog(11
Sho (1x)(x.)))
lima-+1
v3. Therefore, the ratio (x..(x.)))) (ha.(X.
(x)
S v3 1.74 for To show
9 (i)
v (ffle-1)-clog((00 loge-1 Vc
1-cte +1
2x-1 it suffices to prove
6"
(-c+ clog(c) + 1)² 2 ((c- - 1---1)--a - clog((c)) (log(c) - 1)²
which is equivalent
log(c) (-2c + e log (c) 2c 2o (c) + (3c - 1) log(e) + +2) - log(c){(c) 2 0.
Since fc() - + (c) lc)2(c) - log(c) + + and f(1) = 0, complete the proof for
Pno.ex.Y (i) ()) >
logo-1 c-1 Ve
log g On the log scale, log Varia, (() log
log
(=etelox(c)+1))
(c-1)*
is monotone
<((e clou?(e))
(e-1)4
function for Ve As e approaches I from the right, the limit of the function is log(4) As approaches infinity, the limit of the function
0.557.
[1g(12)
H FUTURE WORK
In the realm of future research, an intriguing avenue lies in the exploration of the bias-variance alignment phenomenon within diverse machine learning model ensembles. While our present work sheds light on this phenomenon in the context of deep neural network ensembles, there remains a substantial gap in our understanding of its presence or absence in other model types such decision trees and support vector machines. An essential direction for future investigation involves unraveling the root causes of bias-variance alignment, particularly as it pertains to distinct types of overfitting in machine learning models- be it benign, tempered, or catastrophic (Mallinar et al., 2022). Our conjecture extends towards discerning whether bias-variance alignment is ubiquitous trait across
28

---

Published as a conference paper at ICLR 2024
various overfitting scenarios or if its manifestation is contingent the specific type of overfitting. Notably, we hypothesize that catastrophic overfitting may exhibit a bias-variance tradeoff, while benign and tempered overfitting might demonstrate unique bias-varianee alignment. Unraveling these intricacies is pivotal for advancing our comprehension of ensemble behaviors across different machine learning paradigms. Moreover, Atanasov al. (2022) demonstrates an inverse relationship between the feature learning strength and the variance across initializations. Exploring the interplay between bias, variance, and feature learning strength in neural networks is also intriguing area of future research.
29


Reviews:
[{"review_id": "mQjjkD9Hiu", "reviewer": "Reviewer_ehNn", "summary": "The paper discusses the *bias-variance alignment* phenomenon with respect to an ensemble of deep models in classification tasks. Essentially, the phenomenon states that the logarithms of bias and variance (within the ensemble and at a given sample) of predicted probabilities are well described by a linear relation with coefficient $1$, i.e. $\\log \\mathrm{Vari} \\approx \\log \\mathrm{Bias}^2 + C$. The authors first empirically demonstrate the phenomenon on image classification tasks. Then, they proceed to a theoretical justification of the phenomenon from two independent perspectives: calibration and neural collapse. For the calibration perspective, the authors propose a definition of calibration that unifies other definitions previously used in the literature and then utilize it to show that bias-variance gap can be bounded by the calibration error. For the neural collapse perspective, the authors propose a simple model of ensemble predictions based on Gumbel noise added to the perfect neural collapse prediction (i.e. ETF complex vertices) and explicitly derive bias and variance given the scale $s$ of last layer features, subsequently showing that for all values of $s$ bias-variance alignment approximately holds.", "questions": "I have a few questions mostly related to the Neural Collapse part. \n- Does figure 13 contain only correctly classified samples or all of them? In the latter case, it would be interesting to know why the approximately vertical shape of wrongly classified points (e.g. a \"blue tail\" on figure 1.b) is not present in the synthetic data. \n- *Choice of Gumbel distribution*. Overall, the validation of assumption 5.1 presented in sec. G.2 seems to support the hypothesis that we may write $\\psi_\\tau(X)=R(sw^{\\mathrm{ETF}_Y+v})$ with random vector $v$ whose entries are i.i.d., zero mean, and have unimodal shape. For example, Gaussian distribution satisfies these criteria. So is there some additional evidence for choosing Gumbel distribution, besides it enabling an analytical solution? \n- After eq. (10) it is mentioned that the paper considers (except sec. E.2) MSE loss on top of classifier probabilities. However, I did not find any mention that this is also applied to the training of neural networks used for empirical results - that would be quite non-standard from a practical point of view (CE loss is usually used in practice). Could you please provide details on how the networks were trained?\n- If, in the previous question, CE loss was used, I would expect that scale parameter $s$ of ETF complex grows with training time in the terminal phase of training (it is required for reduction of train loss to extremely small values). Then, by training the network for a very long time, one can hope to cover a wide range of $c$ values from Corollary 5.3, and thus experimentally check whether theoretical prediction that $\\log \\mathrm{Bias}^2 / \\log \\mathrm{Var}$ changes from $0.557$ to $2$. It would be interesting to see whether the theory can work on such a fine level of detail. However, this also contradicts \\textit{bias-variance alignment} statement (1) since it requires $\\log \\mathrm{Bias}^2 / \\log \\mathrm{Var} \\to 1$ as $\\log \\mathrm{Var}\\to-\\infty$.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Overall, the paper is well-written and organized, which allowed to compactly describe a number of various contributions. \n\nThe contributions of the paper present are very balanced and include \n- Formulation and explanation of a new phenomenon. It is done via a nice illustration (Figure 1, both a. and b.), intuitive explanation of the phenomenon, clear definitions and the setting (sec. 2.1), and a focused related work section.\n- Very convincing empirical evidence supporting it. In my experience, it is quite difficult to find simple relations that accurately describe modern neural networks on realistic-data (i.e. not toy models). Yet, the quality of the linear relation on figure 1.b (and other plots of the same type) is surprisingly high. \n- The authors found two quite simple theoretical scenarios that can replicate the bias-variance alignment. Moreover, these scenarios are quite distinct: the calibration part requires no specific assumptions and provides rigorous upper bounds, while the neural collapse part uses very specific model which admits an exact and explicit solution (i.e. not just an upper/lower bound).", "weaknesses": "Overall, I have not found significant drawbacks.\n\nThere are a few moments which could be corrected to improve clarity and reading experience: \n- Formally, eq. (52) is correct only up to the addition of a vector parallel to $\\mathbf{1}_K$. Indeed, such vectors, when passed to softmax, can be ignored and therefore does not affect the subsequent calculations performed in the paper. However, it would be better to mention this to avoid confusion. \n- The colors for negative and positive classes in figures 12,13 are swapped compared to the rest of the plots in the papers. Though completely stylistic, it would be nice to have a consistent color scheme to make paper reading more comfortable.\n- It took me some time to verify that variable $F$ appearing in eq. (59) indeed has $F(2(K-1),2)$ distribution. Since this fact is central for the following computation and, to my understanding, motivates the choice of Gumbel distribution for $v$ in assumption 5.1, it would be better to discuss and explain this explicitly in appendix.  \n\nAlso, a couple of minor typos:\n- Histograms for negative classes have disappeared in the middle row of Figure 12. \n- A typo \"$K$ as the number of samples\" at the end of sec. G.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FecenxH0Db", "reviewer": "Reviewer_18mF", "summary": "The paper studies the trends in the bias and variance of an ensemble of classification models (including their pointwise trends). It empirically establishes two phenomena: the bias variance alignment and upper bounding variance with bias. On a theoretical front, the paper shows that the bias variance gap in expectation can be bounded by a variant of calibration error and therefore when the models are calibrated the exhibit the above phenomenon. Furthermore, assuming that for test data the model prediction is perturbed from the model obtained during training after neural collapse and the upper bounding variance phenomenon for binary classification is proved.", "questions": "-", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "a) Through experiments, the paper establishes the alignment of the bias and variance. The role of overparameterization in the bias variance phenomenon is well demonstrated. This provides an avenue to understand generalization for over-parameterized models. \n\nb) The Assumption 5.1 on model prediction under neural collapse is very interesting. Using this to show the bound the ratio of variance and bias is novel. \n\nc) A general definition of calibration under a sub-$\\sigma$-algebras is a nice contribution.", "weaknesses": "a) The main weakness of the paper is the comparison with related work. For example, theorem 4.3 seems equivalent to the theorem 4.2 in [1].  In the appendix it is claimed that the theorem 4.3 implies the result of [1].  However, it seems more of an equivalence than implication.  The authors should comment on this aspect in the main paper to be more transparent. \n\n[1] Y. Jiang, V. Nagarajan, C. Baek, and Z. Kolter. Assessing generalization of SGD\nvia disagreement. In International Conference on Learning Representations, 2022.\n\nb) Assuming that the model predictions are well calibrated seems is a very strong assumption, thus using this to bound the bias variance gap makes the result less interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qgj3lNc6Cr", "reviewer": "Reviewer_iQea", "summary": "This paper investigates the bias-variance decomposition of deep ensembles. The paper claims that, in the deep learning regime, the relationship between (squared) bias and variance is (a) approximately linear for correctly classified test examples and (b) bounded for all examples ($B^2 \\geq k \\cdot V$). These claims are evaluated on standard deep learning image benchmarks. The role of over-parameterization is also investigated. Two theoretical discussions are included: one showing that certain relationships between bias and variance can be made explicit under calibration assumptions and the other providing some results from the perspective of neural collapse theory.", "questions": "* Is the $C_{h_{\\theta}}$ in eqn (2) the same as the one in eqn (3)? If so, this should be made more clear. \n* I would be interested in the authors' view on the consequences of the findings of this paper and how it should impact future work. For example, how do these results impact the intuitions we may have developed in the domain of ensembles of \"simple\" models (e.g. decision trees)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* This paper addresses an interesting problem which is the bias-variance trade-off in the regime of deep ensembles which has some distinct characteristics compared to the regime from which it emerged (decision trees, random forests, etc). \n* It makes an interesting and (to the best of my knowledge) novel claim that bias and variance are _aligned_ (i.e. have some consistent behaviors in their relationship) in the deep ensemble setting. \n* Experiments are performed on standard image benchmarks of CIFAR and ImageNet making them relevant to many previous deep ensemble works. \n* Aspects of the connection to calibration were useful and provided concrete results that seem to reinforce the authors claims.\n* I found that the introduction did a good job of setting up the structure of the paper and it's main claims.", "weaknesses": "I am quite borderline on this paper. I found the topic and claims of the paper interesting and worthwhile and I certainly think there is value to the community in this type of analysis. However, the exposition of many of the ideas could have been better. I think there were too many ideas (often quite disjoint) that were densely squeezed into the 9 pages requiring a terse, notation-heavy delivery that was challenging to follow at times. I thought some ideas were under-explored (e.g. the role of overparameterization) while others were over-explored (e.g. I felt the section on neural collapse added little to the paper). Overall, I think the paper would greatly benefit from prioritizing its more impactful claims and resolving them more comprehensively. I list my more specific issues below. \n* **Title** - I'm not sure about the use of the word alignment. You wish to make a statement about the relationship between two variables (i.e. they are linear or proportional). I don't think aligned quite means this. Additionally, this is an overloaded word with connotations in e.g. AI safety. \n* **Related work** - I found the related work to be _highly_ incomplete without any inclusion of previous works in the ensemble or deep ensemble literature. For example, many previous works (some recent [1,2], others older [3]) have explored various overlapping aspects of decomposing ensembles into bias and variance. Recent works have empirically noticed and explored the role of overparameterization on variance [4]. Other work has noticed that optimizing to increase ensemble variance causes a simultaneous increase in the predictive bias [5]. I would recommend that the authors perform an extensive literature review to appropriately position themselves relative to this and other works from the ensemble literature. \n* **Evidence for empirical claims** - Ultimately the two main hypothesized relationships proposed in this paper are empirical in nature (despite being possibly motivated by theory). Therefore I would have expected a more extensive empirical evaluation of their claims. Only CIFAR and ImageNet datasets are included, which seems insufficient for the claim that this is a general relationship. I would be more convinced if either (a) the claims were reduced to just standard benchmarks on image data or (b) more comprehensive experiments were performed (i.e. more datasets and more architectures). Additionally, I found it challenging at times to tell if the results included had sufficient detail to evaluate the claims. For example, should Fig 3 not include colors so that we can ensure the outliers are also misclassified? Should the actual bound not be included so that we can verify eqn (3)? I think the presented results could be more clearly linked to the claims they are investigating. \n* **Overparameterization** - I think the role of overparameterization in deep ensembles and how it seems to invalidate the theory developed in the original ensembling literature (e.g. random forests) is fascinating. Therefore I was disappointed that this analysis didn't go into more depth and consisted of simply evaluating ResNets of various widths on CIFAR. Given that the authors decided to include this in the paper I would have liked to have learned more than what was already established in works such as [4].\n* **Calibration** - The relationship between calibration and bias-variance was interesting to explore. However, given that the main claims of the paper are empirical I thought this section could have been less detailed to provide more time to actually verify and evaluate the empirical claims in depth. I thought much of this section could have been presented in a more clear way that more succinctly linked to its purpose (i.e. its relationship to bias-variance alignment and upper-bounded variance).  I think it would be possible to restructure this section such that important points are clear with some more of the technical detail (when less relevant to the main point) placed in the appendix. \n* **Neural collapse** - In contrast, I thought the section on Neural collapse added very little to the paper. The assumptions are restrictive and the results don't seem to be particularly useful. Also, it seems that the section is too short to have space the provide sufficient detail and background on the topic. I would suggest moving most or all of this section to the appendix and using the additional space to expand on the empirical evaluation. \n\n\nOther points\n* The two claims on page 2 are empirical claims but are presented in a way that suggests they are theoretical truths (I misunderstood this on my first read). I would suggest making this more clear in the text. \n* Eqn (10) should either contain a proof (in the appendix) or cite a source (since the authors claim it is well-known). \n* Many of the plots are very small (especially the text) making them hard to read. \n* Tables 1 & 4 formatting is unusual. Why the double line at the top? I would try to avoid vertical lines in the body of tables (see NeurIPS style guide). \n* Typo - p9, l5: \"In above\" -> \"In the above\"\n\n\n[1] Gupta, N., Smith, J., Adlam, B., & Mariet, Z. E. (2022). Ensembles of Classifiers: a Bias-Variance Perspective. Transactions on Machine Learning Research.\n\n[2] Ortega, L. A., Cabañas, R., & Masegosa, A. (2022, May). Diversity and generalization in neural network ensembles. In International Conference on Artificial Intelligence and Statistics (pp. 11720-11743). PMLR.\n\n[3] Brown, G., Wyatt, J. L., Tino, P., & Bengio, Y. (2005). Managing diversity in regression ensembles. Journal of machine learning research, 6(9).\n\n[4] Abe, T., Buchanan, E. K., Pleiss, G., & Cunningham, J. P. (2023). Pathologies of Predictive Diversity in Deep Ensembles. arXiv preprint arXiv:2302.00704.\n\n[5] Jeffares, A., Liu, T., Crabbé, J., & van der Schaar, M. (2023). Joint Training of Deep Ensembles Fails Due to Learner Collusion. arXiv preprint arXiv:2301.11323.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hpqrZYwKk4", "reviewer": "Reviewer_1psh", "summary": "The paper conducts extensive experiments across realistic datasets and architectures (mostly on vision tasks) to demonstrate that the bias of a classifier correlates strongly with its variance in the regime where the number of parameters is large. They go on to show that smaller networks deviate and have worse bias-variance correlation. They prove theoretically that such an alignment should happen under the assumption that the network is well-calibrated. Finally, they show that the theory of neural collapse also predicts such an alignment.", "questions": "My understanding is that real-world models do not generally exhibit calibration. Given this, would you still expect the bias-variance equality to hold approximately for uncalibrated models as well?\n\nI don't really understand why you're saying that in Figure 7, Bias and Variance don't align well. There definitely seems to be nontrivial correlation. Is it just that it is worse than Figures 3, 4?\n\nIt would be interesting to study the bias and variance as a function of feature learning strength in the network. The scale of the variance over initializations is shown to depend inversely on feature learning strength in e.g. https://arxiv.org/abs/2212.12147", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper provides extensive empirical evidence for the bias-variance alignment phenomenon across different model architectures and datasets. The figures are very compelling, esp Figures 2-4. The two theoretical motivations are solid in the opinion of the reviewer. The connection to calibration is certainly interesting. Overall, this paper provides new insights into the bias-variance relation in deep learning and I recommend acceptance.", "weaknesses": "I don't have many comments on this. The primary datasets that this paper focuses on are in image classification - it would be good to see results on other domains like NLP or even simple polynomial curve fitting. \n\nIt would also be good to be more explicit about what bias and variance you're talking about. Adlam and Pennington (as cited in our paper) show that double descent can be clearly understood in linear models by using a \"fine-grained\" bias-variance decomposition. There, there is variance due to initial parameters, train set, and label noise. Its extremely hard to understand which sources of variance you're talking about in section 2.1. It's clear that initial parameters is certainly one of them, since you are defining $h(\\cdot | x) = \\mathbb E_{\\theta} h_{\\theta}(\\cdot | x)$, but it is not clear how the variance over train set is entering this (or whether it is at all).\n\nThe change to CE loss in appendices E.2, F.6 is not very readable. It seems that the effect mostly goes away. Given the ubiquity of CE in modern ML, it would be nice to expand and make these currently tiny sections more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)