Great Minds Think Alike: The Universal Convergence Trend of Input Salience
Authors: Yipei Wang
Jeffrey Mark Siskind
Xiaoqian Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: explainable artificial intelligence
saliency maps
model distributions
Primary area: interpretability_and_explainability
TL;DR: Leveraging input saliency maps, we discover that with increasing capacities, the distributions of models converge to the almost-shared population mean, and thus the limiting of model can be estimated through the population mean of small models.

Abstract: Uncertainty is introduced in optimized DNNs through stochastic algorithms, forming specific distributions. Training models can be seen as random sampling from this distribution of optimized models. In this work, we study the distribution of optimized DNNs as a family of functions by leveraging a pointwise approach. We focus on the input saliency maps, as the input gradient field is decisive to the models' mathematical essence. Our investigation of saliency maps reveals a counter-intuitive trend: two stochastically optimized models tend to resemble each other more as either of their capacities increases. Therefore, we hypothesize several properties of these distributions, suggesting that (1) Within the same model architecture (e.g., CNNs, ResNets), different family variants (e.g., varying capacities) tend to align in terms of their population mean directions of the input salience. And (2) the distributions of optimized models follow a convergence trend to their shared population mean as the capacity increases. Furthermore, we also propose semi-parametric distributions based on the Saw distribution to model the convergence trend, satisfying all the counter-intuitive observations. Our experiments shed light on the significant implications of our hypotheses in various application domains, including black-box attacks, deep ensembles, etc. These findings not only enhance our understanding of DNN behaviors but also offer valuable insights for their practical application in diverse areas of deep learning.

Great Minds Think Alike: The Universal Convergence Trend of Input Salience
Yipei Wang, Jeffrey Mark Siskind, Xiaoqian Wang
Elmore Family School of Electrical and Computer Engineering
Purdue University
West Lafayette, IN 47907
wang4865, qobi, joyaaoogaaaaguuud.....e
Abstract
Uncertainty is introduced in optimized DNNs through stochastic algorithms, form- ing specific distributions. Training models can be seen as random sampling from this distribution of optimized models. In this work, we study the distribution of optimized DNNs as family of functions by leveraging a pointwise approach. We focus on the input saliency maps, as the input gradient field decisive to the models' mathematical essence. Our investigation of saliency maps reveals a counter-intunee each other more as either of their capacities increases. Therefore,
trend: two stochastically optimized models tend to resemble
hypothesize
several properties of these distributions, suggesting that (1) Within the same model architecture (e.g.. CNNs, ResNets). different family variants (e.g., varying capaci- ties) tend to align in terms of their population mean directions the input salience. And (2) the distributions of optimized models follow convergence trend to their shared population mean as the capacity increases. Furthermore, we also propose semi-parametric distributions based on the Saw distribution to model the conver- gence trend, satisfying all the counter-intuitive observations. Our experiments shed light on the significant implications of our hypotheses in various application domains, including black-box attacks, deep ensembles, etc. These findings not only enhance our understanding of DNN behaviors but also offer valuable insights for their practical application in diverse areas of deep learning.
Introduction
The advancement in computational power has significantly enhanced the capabilities of Deep Neural Networks (DNNs), leading to their unparalleled expressiveness and success in a multitude of applica-
tions across various fields (Krizhevsky et al., 2012 He et al. 2016 Rajkomar al., 2018 Berner ett al. 2019: Rombach et al.)
et al. [2023] Thirunavukarass et al. 2023). Despite these
achievements, DNNs remain enigmatic, not only end-users but also researchers and practitioners (Ribeiro et al. 2016 Rudin 2018 Preece et al. 2018). Due to the over-parameterizzaion nature of modern DNNs, they are capable of reaching zero loss the training distribution (Goodfellow et al. 2014b) Allen-Zhu et al. 2019 Du et al. 2019). Furthermore, the inherent stochastic nature of training algorithms means that even when using the same training data, DNNs tend converge to various minima (Huang et al. 2017 Liu et al. 2020). Thus even though these models may exhibit comparable performance in terms of metries like testing loss or accuracy, their underlying mechanisms can still differ significantly. Because of the stochastic nature of the training procedure, optimized DNNs collectively form distribution over the functional space C¹ (X)), and training DNNs from scratch is thereby equivalent to randomly sampling from such distribution without any guarantee. This inherent opacity, combined with the high dimensionality and nonlinearity, limits our understanding of the internal mechanisms of DNNs.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Random Distribution
(b) Convergent Distribution
Figure 1: A synthetic illustration of the distribution of the directional gradients of stochastically optimized models of the same input data. The subfigures demonstrate (a) an intuitive, stochastic scenario, where the distributions of different model families are not closely dependent. and (b) the converging distribution trend introduced by our hypothesis. Different colors represent different model families, and points represent different optimized models.
In response to these challenges, study the aforementioned distributions. By adopting pointwise approach, our focus is on the distribution of input salience (Simonyan et al. 2013) from the context of eXplainable Artificial Intelligence (XAI), which aims demystify the inner workings of these complex models (Gunning and Aha 2019 Arrieta et al. [020] Van der Velden et al. 2022). Saliency maps, particularly in the form of imput gradients, represent the data points within the gradient fields of DNNs. Thus the study of gradients can offer a deterministic view of the landscape of model predictions. This approach allows us examine the intricate nuances of DNNs in more structured and analytical manner.
For clarity, in the following context, we distinguish between the term model architecture (e.g- skip/direct connections) from the term model family. The latter refers to a specific collection of models F. that differ only in capacity to be in the same family if they differ only in parameter values. Given an input, varying model families result in distinct distributions. A synthetic visualization of such distributions is shown in Figure a Different models are depicted by the points. However, the relationship between different model families, represented by various colors, remains elusive. In this work, we introduce and verify several hypotheses to uncover a striking pattern. (1) Within the same model architecture (e.g., CNNs, ResNets), different family variants (e.g., varying capacities) tend align in terms of their population mean direction. (2) As the model capacities increase, the variance within the distribution of the same family diminishes. This leads converging trend of the distributions. Both hypotheses are illustrated in Figure b)). Additionally, we introduce a semi- comi-parametric approach to model these distributions, providing detailed quantification of the convergence.
determined by width and depth. Two models are said
The similarities observed in input salience have direct implications for understanding the important vulnerability of DNNs regarding gradient attacks (Szegedy et al.) [013) Goodfellow al., 20) In particular, in black-box attack settings, the gradients of the target model are not directly accessible. A higher degree of salience similarity naturally enhances transferability (Chen et al. 2023). Our findings elucidate why models with larger capacity consistently exhibit superiority in terms of adversarial robustness compared smaller models (Madry et al. 2017: Gustafsson et al. 2020 Li al. 2020
Bubeck and Selike| 202T). Moreover, given that the mean direction is aligned across different models, is possible approximate this mean direction by randomly sampling from a set of independently optimized models. We demonstrate that these estimated mean directions can attain a near-perfect cosine similarity of almost 1.0, even between completely independent models or ensembles, in high-dimensional space. Moreover, note that deep ensembles essentially calculate this population mean direction (Lee et al. 2015 Lakshminarayanan et al. 2017 Fort et al.) 2019 KKnndatyuk et al.) 2020), where the mean of group of independently trained models can improve the performance. As a consequence, the insights of our hypotheses also shed light on this phenomenon which, although empirically successful, has been somewhat enigmatic in terms of their source of capability (Lobacheva jtt al., 2020; Deng and Shi 202T Abe et al., 2022 Theisen et al. 2023). Furthermore, since deep ensembles approximate the aligned mean directions much faster than scaling up single models, this

---

also demystifies the significant black-box attack transferability of deep ensembles (Yang et al. 2021 Chen al., 2023). Our research thus not only advances the understanding of model behavior in practical applications but also contributes to the broader field of Al trustworthiness and efficiency. Our main contributions can be summarized as follows:
We reveal an appealing phenomenon where the mean distribution directions of input salience across different model families have extremely high resemblance.
We empirically demonstrate the distribution converges towards the mean direction as model capacity increases.
Incorporating both empirical observations and theoretical analysis, we hypothesize distribu- tional properties of optimized models, quantifying the aforementioned phenomena.
The hypotheses effectively explain many hitherto unclear phenomena such as black-box attack transferability, the efficacy of deep ensemble methods, etc.
2 The Convergence of Input Saliency
2.1 Salience Similarities
Notation. Let X' x D = D denote the dataset, where C Rd is the input set and V = [c] is the set of labels and N4. is the number of classes. Following the benign overfitting phenomenon (Bartlett et al. 2020 Papyan et al. 2020 Cao al. 2022), we let F {S]Cs; Xerain Verain) < 10 denote family of optimized models, distinguished by different architectures, such vanilla sequential CNNs, skipping blocks, etc. C denotes the expected cross-entropy loss for the training distribution. For simplicity,
focus f IRd IR which predicts the logit specifically for the
targeted class. This to stay consistent with XAI methods. We demonstrate in Appendix A that the difference between logit and probability (Wang and Wang 2022 does not affect the observed phenomena. Unless otherwise indicated, experiments are carried out over the test set = Neest, y Veest- Within the same architecture, model capacity is determined by both the width and the depth. Since is more difficult to model depth as a single variable, we model varying depth as different families F but model varying width ls parameter of the family, F(k).
Mail Cap
Mali
(a) CNN, CIFAR-10 (b) ResNet, CIFAR-10 (c) CNN, CIFAR-100 (d) ResNet, CIFAR-100
Figure 2: The individual similarity Pina( (1), (2)
Ex [CosSim (x)) (2)
where ((1) F(kx), f(2) ( E F(k2). 1IFAR-10/100 and CNN & ResNets tested here.
10
Hodel Caporty
70
Tickl Crpwery
6.15
0.00
40
82
0.0
80
0.20
150
180
250
Modid ty
Madel Capccity
Madel Capecity
Madia Capucity
CNN, CIFAR-10
(b) ResNet, CIFAR-10
CNN. CIFAR-100
ResNet, CIFAR-100
Figure The expected similarity p(k1, k2) between model families of varying capacities k kz. Here the datasets are CIFAR-10/100, and the models are CNN and ResNets.

---

The Increasing Similarity. Let CosSim : R x R + [-1,1] denote the cosine similarity metric, then the individual similarity between the input salience of two given models f(l)) e F(ki), E F(k2) given input æ is Pind(f f(3); ;)) = (aa
Va (x)) Tak-
ing the entire testing set into consideration, denote (ina(s (2)) = Exxx (f(1) , [(0); In Figure 2 the expectations over the testing set x CosSim V V= (x), Va f (x)) with vary- K are illustrated. Here, we define K [j2' {8, 10, 12, 14, 16, 20. ................................ 384, 448) to balance between finer linear scaling and coarser exponential scaling. It can be observed that the similarity between two stochastically optimized models (W, has an increasing trend with respect to both ki k2- Two different architectures CNN and ResNet included. To rule out the influence of any single model, we define the similarity between families F(k)), F(2)) for given input æ by taking the expectation of the two models:
ing k1, K2
S J - 7, S i 63
p(k kzt := (E Cos (v. (x)
(1)
The global similarity between models of widths k1, k2 is then denoted by p(k1, k2) Exexp(Kk,, ky; x)). Note that estimating this value requires training numerous f F(k) ) for each k E K. Therefore, we carry out the experiments over K' = {(22* : = 52 = 1, 4,5] = {10, 20, 40, 80, 160) C K. For each kı,ky E K', 100 models are sampled respectively to em- pirically estimate the expectation over F(k1),F(ke)) As observed in Figure 3 p(12, k2) has an increasing trend w.r.t. both k,, k2g Compared with Figure the average similarity over the model families is similar to the individual cosine similarity for the same & values. As result, studying the similarity of two randomly sampled models instead of the expectation over Fs can significantly alleviate the computational burden. This is further discussed in detail in Section 3.2 Besides, It trivially follows that Vk1 > k2. p(k1, k2) > p(k] k2) > p(k2, k2), which suggests that larger models tend to resemble smaller models even more than smaller models themselves. - Even if the two smaller models only differ in the random seeds during training. Please refer to Appendix (A for the results of more datasets, where such increasing trends still exist.
2.2 The Spherical Distribution of the Salience
V. V [(]]] and (
V (1) (nn)
Since the cosine similarity can be written as the inner product between which are high-dimensional unit vectors, we explore the properties and potential distributions of F through the perspective of spherical statistics. Given an input æ E X. we denote by Gx( the set of all possible gradient directions of input regarding the models f in F(k). Formally, let
(x) =(u = V
E F(K) Ve E X
(2)
Then the similarity is re-written as the inner product p(k), kz;; = Eu, EG2, (0)) (w) [u]uu).
The Intra-Famity vs. Cross-Family Paradox. An interesting paradox is raised as p(...) increases with both inputs. Naturally, one would reasonably deduce that two models resemble each other since they are from the same family (i.e. having the exact same structure and only differ in training seeds). However, since p(kg, ky) > p(k k1), the cross-model family similarity becomes greater than the intra-model family similarity. To uncover the mystery of the observations, we present the intuitive understanding and the rigorously analyzed hypotheses as follows.
E F(ki) should
The Intra-Family Hypothesis. Note that for intra-family scenario, u. E G1 (x) are i.i.d., the similarity can be written as p(k, k; x) = (<) ("EE,, [o]) = [|E (az) [u]l which denotes the square of the population mean resultant length (Mardia et al. 2000) of Gie (ar). The population mean resultant length pkkk k; æ) quantifies the degree of dispersion of G. (x). where a larger length suggests more concentrated distribution. In directional statistics, the degree of dispersion is usually quantified by the spherical variance 2(1 Vp(k, k;; x)) the total variation 1 p(k, k; x). Since p(k. k; x) also increases w.r.t. k. this suggests an increasing concentration of input salience of models as the width k of the model increases. In conclusion, the larger the models are, the smaller the spherical variance of the salience is. Formally, we propose the following hypothesis.
Hypothesis I (HI): Let denote the width (capacity) of the model and Gx( {u
(m)(m) VF E F(k) denote the set of input gradient directions regarding æ.. Then

---

G (1r) [u]] = Vplk, k; x ())kk()))) x) and p(k, k) increases with k. Here p(k,
(ms(uu
denotes the unit mean direction of Gy(a)).
Note that HI also holds for the change of model depths, which is positively related the dispersion degree of the distribution. However, the change in model depth inevitably affects model width. Thus, part of H2.
we only provide empirical verification in Section but do not include
The Cross-Family Hypothesis. Unlike the intra-family similarity, the increasing cross-family similarity is where the phenomenon becomes counter-intuitive. Then due to the increasing intra-family similarities, when k1, ky increase, 41 E Gv, (a) becomes closer to [uki; x), U2 E Gk2 (x) becomes closer to [K x) However, the cross-familty similarities suggest that as W1, U2 approach their respective mean directions, their similarity increases as well. This indicates that the mean directions (=(k;;)) pk(k2; a)) are similar, too. Formally, this intuition is considered as follows. For k h2, when fu(k;; æ)) and (kb;; x)) are sufficiently similar, (k(k;;; x²" (= x) 2 [11;; (l)l:())
Thus we have
p(ki, m) =F EBt, (z) (2)), (2) [u u2] = Eu (2)) (21) E (a) [u2]
(3)
BllEur (a)) (m)] Eu (w) [uell - v p(ki, kii p) k23
which is monotonic w.r.t. both ki, Ag. Formally, this summarized as
Hypothesis II (H2): Let Gk, (a)), Gky (a) denote the input gradient directions of two model families where k * k2. Then pu(k;; x) se p x) regardless of k1, k2-
The two hypotheses are both empirically verified. For a smoother flow of the presentation, we defer the detailed experiments to Section 3 The basic ideas of H1 and H2 are illustrated (ag
2.3 The Directional Distribution of Gradients
Given the analysis and hypothesis above, one can have an overview of the models' internal mech- anisms. As the model capacity increases, models are distributed in more concentrated manner, while the mean direction stays almost invariant. To better understand the models' behavior with the stochasticity, we delve into the distribution of G5 (x) and present semi semi- parametri analysis with experimental verification. A general form of centralized symmetric distribution over hypersphere
is known as the Saw distribution (Fishrr et al. 1993) p(u: 11) viu"" 1)) where μ is the mean direction with || 1, & E C([-1, and 7 -ss- rb(u² p.)du is the normalization term for distributions. Due to the symmetry assumption, the shape of the distribution is solely determined by sb. For example, a monotonically increasing & suggests that u is distributed more densely near the mean direction and sparsely distant from the mean direction. Considering the concentration trend of gradients, we hypothesize that (-) of (x) not only monotonically increases with the input, but also increases faster with greater & values.
Z
Marginalization. For Vu E GA (z). it can be decomposed to u = t- (((x)()) + VI- 2 m(x)+
where (xx) 1 is a unit tangent to S 5d-1 at J.. Then t u pzx). This is shown in Figure 4 (b). Note that (=x)) independent from 4, then the distribution of is the marginal distribution over the intersection between S and the hyperplane spanned by fxz))² which a (d ((d-2--uae 2)ddimensional hypersphere. According to the symmetry assumption of Saw distribution, conditioned on a fixed similarity f, the distribution of uilt over the dashed 5d-2 does not affect tb. Therefore, we focus on the marginalized distribution of L. Note that the radius of the intersection Sd-2 is v1 21-22 P2, we thus have du 2x (4-1) (1-2" 2 (d-3/3/2 di, where the density of is observed by the integral over the corresponding (d 2) 2)-hypersphere. As a result, the marginal distribution of t has the PDF
[((d-1/22
(x = Uhx (t: x) ())
(4)
where Z' is a constant normalization term. Note that (1 ()d--3)/2 is symmetric bell curve centered at t- = O. Equation (4) can thus be viewed as using e/x (t; x)) to reweight its PDF Pongin (t) = T(d/2)
(1 (1- 11- 3d- ()/2)) Note that here P& (t: x:) is the distribution of t u (kk: æ),, which can be empirically estimated, the shape of the function tha becomes empirically accessible with varying values. The empirical studies and verification are provided in Section 3.3
5

---

p(kgiz), ((x,1)))
u1
12 (x))
M2
dt
du
54-1
(n) The illustatton of cross- family distributions kz).
The detemmoooootiin (lnff)and marginalization (fight) Gy(r).
Figure 4: (a) presents an illustration of HI and H2. Blue and green caps represent i1 E Gk, (2) (green) and u2 E (2)(2) (blue) regions !
HI: larger ks lead to smaller spherical variances; H2: the mean directions are extremely similar. (b) illustrates (left) the decomposition of u to the mean direction and the orthogonal direction; and (right) the marginalization of the distribution from u to t.
Empirical Verification of Hypotheses
In this section, we provide comprehensive experimental results to verify the aforementioned hypothe- ses. First, we introduce the detailed setups of our experiments. They are carried out on Intel(R) Core(TM) 19-9960X CPU @ 3. 3.1GH with Quadro RTX 6000 GPUs.
Datasets & Models. Due to the massive size of experiments, here we mainly follow the setups of the benign overfitting (Nakkiran et al. (021). which also present comprehensive study of optimized DNNs through CIFAR-10 10 and CIFAR-106 (Krizhevsky et al.] 2009). Besides, we also include Tinylmagenet-200 (Le and Yang 2015) compromise between the computational efficiency and the dataset complexity. As for models, we include CNNs and ResNets as in (Nakkiran et al.) [021). These two models represent the two typical architectures the direction connection and the skip connection. We also notice a striking capacity gap between them in the original implementation. Therefore, we term them CNNSmall (CS) and ResNetLarge (RL), respectively, and include CNNLarge (CL), and ReslltSmall (RS) to fill the gap. The comparison between the small and large families also shows the influence of depths. As for the training process, following Nakkiran al. (2021), ) we use stochastic gradient descent (SGD) as the solver, with batch size of 128. The input data are normalized, but not augmented. We start with the initial learning rate fu 0.1 and update it with where is the epoch. Please refer to Appendix B for more experimental details.
Th 70/v /1 1 +
3.1 The Mean Direction Similarity (H2)
H2 can be verified independently from HI and can provide simplifications and insights verifying HI. We hence focus on H2 first. As stated in H2, the mean directions of different model families are consistently aligned, i.e., a p(kg xt). For each family F(k) where K', A 100 models with different random seeds are trained. The population mean is then estimated through
1 M
M
Vaf((w)) [=f((=)]
//1
1
M VaJi(æ) [=f((w]]|
jak:: x)
p(k;)), F(K).
(5)
Then the similarity of mean directions naturally (ūik;;; =))² ( (k); Note that when k; kg- the 100 models are partitioned by the seeds to avoid overlapping. The results of the expectation over X' are visualized in Figure It be found that the mean directions have extremely high resemblance within each architecture, as proved by the high cosine similarities. It should be noted that with high dimensionality (e.g. d 3072 for CIFAR), a cosine similarity close I is an extremely significant result. We demonstrate this with the uniform distribution on the hypersphere in Appendix D The observations verify the hypothesis all Gy (a) almost share the same mean direction within the model architecture. This not only hold across different widths determined by k but also holds across different depths (i.e. CS vs. CL, RS vs. RL). Therefore, the mean direction mostly related to the certain model architecture instead of any single model f E F, making it an intrinsic
"The caps to illustrate the variance differences. Actual distributions the entire hypersphere.

---

04
Caparity
Capanity
Noviel Capartly
3.6
300
300
61
(a) CIFAR-10
(b) CIFAR-100
(c) Tiny Imagenel
Figure 5: The heatmap visualization between the estimated population mean directions from different model families. Each entry is computed by E (L) (x; F), (x; F))) The results are generated from CIFAR-10/106 and Tinylmagena datasets. E (CCS, CL, RS, RL).
RL
0.8
0.5
0.4
Smilarrty Metric
0.4
0.4
0.3
0.2
0.2
0.7
And
0.1
0.1
200 Width
400
200
400
200
400
200
400
Wlith
Width
Width
Figure 6: Illustration (red) Pina (f)) ((2)), f(), f E F(k)) and (blue) p(k, k)) on CIFAR-10.
property of the model architecture. With this property, how different model architectures differ in mechanisms can be studied by looking deeper into the population mean direction of saliency maps. For instance, can be observed that the ResNet architecture admits closer relation between models of different depths, compared with the CNN architecture.
3.2 The Decreasing Spherical Variance with (HI)
Expectation over F vs. Conditioned f f F. As previously discussed, the spherical variance of distributions over the hypersphere can be measured by the population mean resultant length the mean directions. This can be
/p(k, k; æ), which, unfortunately, requires an estimation expensive to study for comprehensive set K of & values. The experiments on subset K" (10, 20, 40, 80, 160) are already carried out in Figure 3 shown as the diagonal elements. As increases, the resultant length increases monotonically, indicating decreasing spherical variance and a more concentrated distribution around the mean directions.
The computational burden of taking the expectation over F can be alleviated by considering randomly picked f. In order to compare and Pind, we consider the model-dependent set s(f) {pma(f, fi
E X} for each f F. Here compute the expected Wasser- stein distance Ey (F)) (D) (())))) This is estimated by the (M) distinct pairs of models. The distances of all 60 (dataset, model family) pairs lie below 0.035. Such observation suggests that after taking the expectation over X, the differences across individual models can be mitigated. Please refer to Table [ for comprehensive results on all model families and datasets. As a consequence, it suffices to use (f f) for some f E F(k) approximate p(k, k). This is in fact the diagonal elements of Figure 2 A comparison between the diagonal elements Pk.k and (1), appendix for other datasets. Pind evaluated over & E K, while p evaluated over k E K' C K. It can be found that after taking the average over X' even though is a little smoother than Pind- they are very consistent. This verifies that resultant length increases with h K much more comprehensive set of models. Thus HI is empirically verified.
f()), (2) E F(k) over CIFAR-10 10 is presented in Figure 6 Please refer to the

---

Single Models
1.0
100
Pag1
Pa
702570 LOG5
2.5
1.5
2.0
1.0
30 033
4.2 33
04
ChNSsmall
Ch0laange
Rechettmaal
HanfartLange
K-10 10
K- 40
00
Figure (left) The illustration of the frequency of
The illustration of the relation be- tween the expected testing loss [x[[4] and the
Figure 9:
the mixture TE. where k E {10, 20, 40, 80, 160).
Specifically, the black histogram represents the distribution Porigin- The dashed curves are the
marginal expectation Ex where models
from (a) single models with varying capacities (b)
approximated PDF pę obtained by KDE. The re-
deep ensembles with varying member Each
sults are generated using CNNSmall and CIFAR-
color represents a model family. In particular,
10. (right) The illustration of log Pt. which is
in (b), different marker shapes indicate different
Porign
linearly related to log thk-
l E [10, 20, 40, 80] of the ensembles.
3.3 The Shape of the Saw Distribution
Given (x))). the marginalized distribution px[ w) be approximated by (xx) = {ī u² ja(k; ))u E (x))) In order to obtain the global results over test dataset X, consider the the mixture distri- butions (t) = x Exs Ph (t; x) We plot the histogram of Ta with k E K' for CNNSmall and CIFAR-10 10 in Figure 8 The left figure shows Porigin and the estimated pk, visualized by different colors. Qualitatively, Pk (t) has higher means with greater & values. The reason why the density of pa is not centered at = 1 (i.e. u (1) is because t increases, the size of the (d 2)-hypersheere decreases with Parigin (t) = (1 (2)(d-31/2 (TT((d1)/2)) which shown as black histograms. This much faster than the increase of tha. The shape of the is determined by pk Porigin with normalization. From the right figure, by comparing Ps with Pepr, it can be empirically verified that thy (t) increasing vastly. It is also observed that larger ks lead to faster increase of the and higher E[t]. This also provides a quantitative understanding of HI and H2. The results of other model architectures and datasets can be found in the appendix.
unions of different input samples Ty U2E (x This is an approximation
r(d/2)
Verification of the Symmetry. Saw distributions study the marginalized value u" μ to directly focus on the degree of concentration of the gradients. This naturally leads to ro- tationally symmetric distributions since the distribution on the intersection between 5d-11 and the hyperplane does not affect the distribution of t. We thus carry out an empirical study of the distribution on the intersection (i.e. conditioned on 6). Specif- ically, we train 1000 CNN models with K = 40 and seeds 1-1000 CIFAR-10 10 and compute regarding each test sample. The distribution of the first sample is visualized in Figure 7 We partition the range of into 10 intervals by every 10 percent of the frequency, and inspect the direction of the mean of the gradients in each interval, each direction is estimated by 100 models. If these conditional mean directions are consistent with the population mean direction, then the gradients are symmet- rically distributed on each dd hypersphere (R7(right)), thus verifying the rotational symmetry. We investigate the cosine similarities between the conditional and unconditional mean directions on the first 1000 samples. The 10 x 1000 similarity values have a mean and std at approximately 0.970 and 0.013 respectively. Thus the rotational symmetry empirically verified.
Partitons
0.15
0.20
0.25
Figure 7: The marginal distribu- tion of t of the first test sample of CIFAR-10 10. Red dashed lines parti- tion the range of every 10 percent of the frequency.
8

---

10
14
Mode
05
06
-06
066
04
64
226
192
0.
0.
0.6
06
Target Modw 2
Target Modw
Target Model
Target Model
(a) CIFAR-IO, CS
CIFAR-IO, CL
(c) CIFAR-IO, RS
CIFAR-IO, RL
The results of single model black-box attack. The value of each entry a(k1 k2) for different model capacities, where 11 is the width parameter of the source model and ky is the width parameter of the target model.
Figure 10:
Precddan Decay Rabo
IN CESE Mask dometCiaa
Precetton Decruy Rabto
2.x10"
Mask MANNITTA
CAIE
Precetton Deeay Rasto
AOUR
Atrck HEMNIC)L
IN CARE
Precetton Derry Santo
Missk plaiett
Care
107
6x
6x11
xx
2x100
Target
Target
Target
Tergot
(a) CIFAR-10, CS
(b) CIFAR-10, CL.
(c) CIFAR-10, RS
(d) CIFAR-10, RI.
Figure 11: The comparison between the single-model attack from the largest model (red), the single- model attack from the very same capacity (green) and the attack by the mean direction (blue).
4 Applications of Hypotheses
4.1 Deep Ensemble: Why Does It Work?
After verifying the hypotheses, we explore possible applications and implications of the discovered phenomena. The deep ensemble method makes use of the stochasticity to of models by incorporating the predictions from m members. While deep ensembles have been verified be effective in improving performance, the source of such capability remains mysterious. Note that ensemble members are i.i.d. optimized models with SGD, which correspond the population mean of our hypothesis. We thus provide another perspective in understanding the capability of deep ensembles.
For single models, the model capacity increases, benign overfitting suggests that the testing loss decreases, too. We deduce that this is because the distribution of larger models becomes more concentrated, and combined with H2, the closeness the aligned population mean is highly related to the models' testing performance. As shown in Figure D().. it can be observed that the expected loss Ex [C] and the marginal expectation Ex are highly correlated. Similarly, deep ensemble approximates the population mean much more effectively by increasing the number m of members than scaling up single model by k. We thus scale up the deep ensemble by changing the number of ensemble members. The results are shown in Figure (b). It can be found that (1) the correlation between Ex [C] and E [t] is not only significant, and (2) the correlation pattern is shared between two completely different scaling mechanisms, single model scaling and model ensembles.
4.2 Black-Box Attack via Saliency Similarity
The understanding of adversarial attacks can benefit from the behaviors of the input salience of optimized models given their close relation to input gradients. We verify the aforementioned similarities through the black-box attacks, where the adversarial samples generated from the gradients of source models while the gradients of the target models are not available. Let f()) E F(ki) denote the source model and ((2) E F(k2) denote the target model. We define the attack rate from

---

f() to f(2) similar to Pind as
a( ((
E wEX
((2) (x sign V V (x) (x))) ((2) (()) (x)
(6)
which is the performance decay of ((2) when attacked by model f()) Small a(ki, k2) values suggest successful attack from f(1) to f(2). The results are shown in Figure 10 where the attack step is set to 0.05. In each heatmap figure, the y-axis represents the width of the source models, while the x-axis represents the width of the target models. It can be observed that larger models succeed in attacking smaller models, but the opposite is not true. To attack large model, the gradient needs to be generated from a model of a comparable level in terms of capacity.
Mean Direction Attack. According to the verified hypothesis H2, for any two individual models (()) E F(k2),, f(2) E F(k22) the mean directions (k; æ)) is closer to both of them than themselves, regardless of k, ki, k2. It is then suggested that using the mean gradient can perform more successful black-box attacks. We employ the mean direction ju( (600 æ) to attack models of different capacities, and compare the results with the attack from the largest single model (red) and the attack from the models of the identical structure (green). The results are shown in Figure where it can be observed that the mean direction of salience transfers much more successfully than single models.
Conclusions
In this paper, we introduce hypotheses explain the observations on the input salience convergence w.r.t. the model capacities. Under the same model architecture, stochastic algorithms such as SGD, result in certain distributions of optimized models. We hypothesize and use pointwise methods to verify that such distribution follows a Saw distribution with aligned population means, which is invariant from the model families. Besides, the variance of the distribution decreases as the model capacity increases, suggesting a convergence trend of the models' internal mechanism - the larger the models are, the less variant they tend to be affected by the randomness from the stochastic algorithm during the training phase. Furthermore, since the distributions converge towards the aligned population mean direction, the limiting points can be estimated by the population mean of models. Based on this, we present comprehensive experiments on the properties of the limiting model and demonstrate its capability in various domains, such as the black-box attack transferability, and the explanation of the effectiveness of deep ensembles. However, it is admitted that, due to the high computational burden. although improved from CEFAR-10/100 to Tinylmagenet compared to (Nakkiran et al. 2021), our experiments are limited to rather small datasets.
Our introduced hypotheses also lead to various interesting topics. Note that the aligned mean direction stays invariant to the model families, which indicates such population mean is more related to the essence of the dataset itself rather than any single model. Leveraging this property can bring deeper and more comprehensive understanding of the relation between data distributions and models.
6 Related Work
In terms of the convergence trend of DNNs, existing works focus the convergence of single models throughout the training process. The parameters of DNNs have been demonstrated to converge to
global minima throughout the training progress (Goodfellow al. 201 4b; Li al. 2018 Allen-Zhu
eet al. 2019 Liu et al. 2020 Damian et al. 2021 Refinetti et al. 2023 Suh and Cheng 2024). Recent years, the studies of benign overfitting also suggest that increasing model capacities can improve
the performance instead of exacerbating the overfitting issue (Bartlett et al., 2020 Nakkiran et al.
2021: Cao et al., 2022). While the studies of input gradients span into an abundant but extremely complicated spectrum. Among them, the area that is the most related to our work is the XAI domain, where the input gradient and variants are crucial in revealing the models' internal mechanisms
(Simonyan et al. 2013] Springenberg et al.) [014]] [ellaraju et al. [017]] (Sundararaaan et al. 2017 Adeebay et al. [2018] et al. |l.| [002]]. On the other hand, the studies of the distribution of optimized
models have received little attention. Such topics are slightly dipped in the efforts to demystify
the source of capability of deep ensembles (Lee et al.] 2015 Fort et al. 2019 Allen-Zhu and Li]
2020: Kobayashi et al. [2021] Abe et al. 2022] Ganaie et al. 2022 Theisen et al. 2023) and their implications (Lakshminnrayanan et al. [017] [eiger et al. 2020 Yang et al. 2021 Chen et al.) 2023). Thus, to our knowledge, the studies the distribution of optimized models remain a novel topic.
10

---

Acknowledgement
This work was supported, in part, by the Defense Advance Research Projects Agency (Prime contract award number: HR001 1222003, subcontract award number: 2103299-01, grant number: 13001 129), the EMBRIO Institute, contract #2 20200,, a National Science Foundation (NSF) Biology Integration Institute, and NSF IIS #1955890, IIS #2146091, IIS #2345235. The content of the information does not necessarily reflect the position the US Government. No official endorsement should be inferred. Approved for public release; distribution is unlimited.
References
Abe, T., Buchanan, E. K., Pleiss, G., Zemel, R., and Cunningham, J. P. (2022). Deep ensembles work, but are they necessary? Advances in Neural Information Processing Systems, 35:33646-33660.
Adcbayo. J., Gilmer, J., Muelly, M., Goodfellow, L., Hardt, M., and Kim, B. (2018). Sanity checks for saliency maps. Advances in neural information processing systems, 31.
Allen-Zhu, Z. and Li, Y. (2020). Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:20/2.098/6.
Allen-Zhu, Z., Li, Y., and Song, Z. (2019). A convergence theory for deep learning via over- parameterization. International conference on machine learning, pages 242-252. PMLR.
Arricta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., al. (2020). Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion, 58:82-115.
Bartlett, P. L., Long, M.M. Lugosi, G., and Tsigler, A. (2020). Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117448):30033-30070
Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P.. Dennison, C., Farhi, D., Fischer, Q.. Hashme, S., Hesse, C., et al. (2019). Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.0666
Bubeck, S. and Sellke, M. (2021). A universal law of robustness via isoperimetry. Advances in Neural Information Processing Systems, 34:28811 11888112200 11-8822..
Cao, Y., Chen, Z., Belkin, M., and Gu, Q. (2022). Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems, 35:25237-25250.
Chen, H., Zhang, Y., Dong, Y., and Zhu, J. (2023). Rethinking model ensemble in transfer- based adversarial attacks. arXiv preprint arXiv:2303.00115.
Damian, A., Ma, T., and Lee, J. D. (2021). Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems, 34:27449-27461.
Deng. D. and Shi, E. B. (2021). Ensembling with fixed parameter budget: When does it help and why? In Asian Conference on Machine Learning, pages 1176-1191. PMLR.
Du, S., Lee, J., Li, H., Wang, L.,, and Zhai, X. (2019). Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675-1685. PMLR.
Fisher, N. L. Lewis, T., and Embleton, B. J. (1993). Statistical analysis of spherical data. Cambridge university press.
Fort, S., Hu, H., and Lakshminarayanan, B. (2019). Deep ensembles: A loss landscape perspective. arXiv preprint arXiv: 19Xx:2991200275:
Ganaie, M. A., Hu, M., Malik, A., Tanveer, M., and Suganthan, (2022). Ensemble deep learning:
A review. Engineering Applications of Artificial Intelligence, 115:105151.
11

---

Geiger, M., Jacot, A., Spigler, S., Gabriel, F.. Sagun, L., d' Ascolii S., Biroli, G., Hongler, C., and Wyant, M. (2020). Scaling description of generalization with number of parameters deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2020(2):03300..
Goodfellow, J., Shlens, J., and Szegedy, (2014a). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
Goodfellow, I. J., Vinyals, O., and Saxe, A. M. (2014b). Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv:1412.6544,
Gunning, D. and Aha, D. (2019). Darpa's explainable artificial intelligence (xai) program. Al magazine, 40(2):44-58.
Gustafsson, F. K.. Danelljan, M., and Schon, T. B. (2020). Evaluating scalable bayesian deep learning methods for robust computer vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 318-319.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778.
Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, E., and Weinberger, K. Q. (2017). Snapshot ensembles: Train I, get for free. arXiv preprint arXiv: 000000
Kobayashi, S., von Oswald, J., and Grewe, B. F. (2021). On the reversed birs-variance tradcoff in deep ensembles. ICML.
Kondratyuk, D., Tan, M., Brown, M., and Gong, B. (2020). When ensembling smaller models more efficient than single large models. arXiv preprint arXiv:2005.00570.
Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems, 30.
Le, Y. and Yang, X. (2015). Tiny imagenet visual recognition challenge. CS 231N, 7(7):3.
Lee, S., Purushwalkam, S., Cogswell, M., Crandall, D., and Batra, D. (2015). Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06311
Li, H., Xu, 7.,, Taylor, G., Studer, C., and Goldstein, T. (2018). Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31.
Li, Y., Bai, S., Zhou, Y., Xie, C., Zhang, 7., and Yuille, A. (2020). Learning transferable adversarial examples via ghost networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11458-11465.
Liu, S., Papailiopoulos, D., and Achlioptas, D. (2020). Bad global minima exist and sgd can reach them. Advances in Neural Information Processing Systems, 33:8543-8555 8552.
Lobacheva, E., Chirkova, N., Kodryan, M., and Vetrov, D. (2020). On power laws in deep ensembles. Advances In Neural Information Processing Systems, 33:2375-2385.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2017). Towards deep learning models resistant adversarial attacks. arXiv preprint 1xXvv:706.0608.
Mardia, K. V., Jupp. P. E., and Mardia, K. (2000). Directional statistics, volume 2. Wiley Online Library.
Muller, M. F. (1959). A note on a method for generating points uniformly on n-dimensional spheres. Communications the ACM, 294):19-20.
12

---

Nakkiran, P.. Kaplun, G., Bansal, Y., Yang. T., Barak, B., and Sutskever, I. (2021). Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 (12): 124003.
Padmaja, B., Moorthy, V., Venkateswarulu, N., and Bala, M. M. (2023). Exploration of issues, challenges and latest developments in autonomous cars. Journal of Big Data, 10(1):61.
Papyan, V., Han, X., and Donoho, D. (2020). Prevalence neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652- 24663.
Preece, A., Harborne, D., Braines, D., Tomsett, R., and Chakraborty, S. (2018). Stakeholders in explainable ai. arXiv preprint 10.Xv::8100001110
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. Liu, X., Marcus, J., Sun, M., et al. (2018). Scalable and accurate deep learning with electronic health records, NPJ digital medicine, 1(1):18.
Refinetti, M., Ingrosso, A., and Goldt, S. (2023). Neural networks trained with sgd learn distributions of increasing complexity. In International Conference on Machine Learning, pages 18843-28866 PMLR.
Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 133--1144.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695.
Rudin, C. (2018). Please stop explaining black box models for high stakes decisions. Stat, 1050:26.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626.
Shah, H., Jain, P., and Netrapalli, P. (2021). Do input gradients highlight discriminative features? Advances in Neural Information Processing Systems, 34:2046-2059.
Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualis- ing image classification models and saliency maps. arXiv preprint arXiv: 1312.6034.
Smith, I., Ortmann, J., Abbas-Aghababzzadeh, F., Smirnov, P., and Haibe-Kains, B. (2023). On the distribution of cosine similarity with application to biology. arXiv preprint arXiv:2310. 13.19944.
Springenberg. J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2014). Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6880
Suh, N. and Cheng, G. (2024). A survey statistical theory of deep learning: Approximation, training dynamics, and generative models. arXiv preprint arXiv:2401.07/533
Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319-3328. PMLR.
Szegedy, C., Zaremba, W. Sutskever, I., Bruna, J., Erhan, D.,, Goodfellow, I., and Fergus, R. (2013). Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199
Theisen, R., Kim, H., Yang, Y., Hodgkinson, L.., and Mahoney, M. (2023). When are ensembles really effective? arXiv preprint arXiv:2305./2315.
Thirunavukarasu, A. J., Ting, S. J., Flangovan, K., Gutierrez, I.., Tan, T. F,, and Ting, D. S. W. (2023). Large language models in medicine. Nature medicine, 1930
13

---

Van der Velden, B. H., Kuijf, H. J., Gilhuijs, K. G., and Viergever. M. A. (2022). Explainable artificial intelligence (xai) in deep tearning-basee based medical image analysis. Medical Image Analysis, 79:102470.
Wang, Y. and Wang, X. (2022). "why not other classes?": Towards class-cont back-propagationn explanations. Advances in Neural Information Processing Systems, 35:9085-9099
Yang, Z.,, Li, L., Xu, X., Zuo, S., Chen, Q., Zhou, P., Rubinstein, B., Zhang, C., and Li, B. (2021). Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Processing Systems, 34: 14:17642-77655
14

---

both k2, R2, hold across different datasets. It is also worth noticing that compared with the results of CIFAR-IO shown in Figure 2 the resulting Pind (k), k2) of CIFAR-I00 and TinyImagnet (Figure (2) shows another peak at around sJ 10 (see Figure e2))). This is related to the deep double descent phenomenon (Nakkiran et al. 202T), where when the complexities of the model and the dataset comparable, the overfitting issue is at peak. For smaller such as CIFAR-IB 10 or larger models such as ResNet, this issue becomes much less significant, as very small value is already sufficient for the data distribution. Also, as the complexity of the data distribution increases, the cosine similarity decreases inevitably for the same model complexity. This suggests less concentrated distribution of the optimized models of the same capacity compared with less complex dataset. Correspondingly, the results of the expectation over the model sets for K' [10, 20, 40, 80, 160) are shown in A Additional Results of the Similarities
The illustration of CIFAR-100 & Tiny Imagenet. In addition to the cosine similarity Pind k2)
shown in Figure 2 we include more sophisticated datasets such as CIFAR-100 100 and Tinylmagenet It is observed that the observed phenomena, where (ind k2) tend to increase w.r.t.
in Figure 12
Figure 13
Coundyy
Countya
(a) CS. CIFAR-10
(b) CL. CIFAR-10
(c) RS, CIFAR-10
(d) RL, CIFAR-10
875
Pobe Grece
Mode Ceanty
Rodel Calerr
Modi Cowtry
(e) CS, CIFAR-100
(1) CL. CIFAR-100
@ RS, CIFAR-100
(h) RL, CIFAR-100
430
320
635
10
Camdl
(i) CS, Tinylmagene
(i) CL, Tinylmagenet
(k) RS, TinyImagenet
()) RL, Tinylmagenet
Figure 12: The expected similarity p(k:, kiy) between models of varying widths k1, k2- Here we include CNNSmall, CNNLarge, ResNetSmall, and ResNetLarge as F. The values k1 k2 determine the widths in each layer. Here the datasets CIFAR-I0 (top), CIFAR-100 (middle) and tinylmagenet (bottom).
The Ablation of the Training Process. To verify that the observed increasing trends of Pind (k1, ky) with model capacities are caused by the training process of DNNs instead of some normalization issue, we compare the similarity for models with initialized parameters. The results are shown in Figure 14] All three datasets and four model families are included. It can be clearly observed that, when the model parameters are initialized, the similarity between input saliency maps of different models are distributed randomly. The cosine similarity values are very concentrated around O, which the mean of random distribution. This verifies that the aforementioned increasing trends are caused by the optimization of models instead of normalization process.
15

---

0.6
C4
230
10-
Mode: Capacity
20-
20
5.25
0.55
0.20
0.
Mone
Voli
235
40
0.15
5.0
0.10
160
103
102
Modil Capacit x
Modal Capacity i
Model Capacity
Model Capacity ir
(a) CS, CIFAR-10
(b) CL, CIFAR-10
RS, CIFAR-10
RL, CIFAR-10
0.30
Motel Capnerty
20-
6.15
Wecee Dapucity
20
020
025
20
20
6,10
530
80
80
100
160
160
150
150
Cxpectt
(e) CS, CIFAR-100
(I) CL. CIFAR-100
(g) RS, CIFAR-100
(h) RL, CIFAR-100
0.12
020
Nodee Daperty
Expecty
0.1
368
0.08
0.10
3.0
40
0.08
304
0.04
160
160
160
150
0.02
Model Capacit it
Model Capacity
Model Capacity
Model Capacity k
(i) CS. Tinylmagenet
(i) CL, Tinylmagenet
(k) RS, TinyImagenet
(1) RL, Tinylmagenet
Figure 13: The expected similarity p(ki, between models of varying widths 1 k2. Here include CNNSmall, CNNLarge. ResNetSmall, and ResNetLarge as F. The values of k1, k2 E [10, 20, 40, 80, 160) determine the widths each layer. Here the datasets CIFAR-10 (top), CIFAR- 100 (middle), and tinylmagenet (bottom).
Softmax Activations. Apart from the normalization concern, recent work in (Wang and Wang 2022) demonstrate the difference between the input salience of the predicted logits and probabilities. As result, we clarify that, although we define f as the predicted logit (before softmax activations), this choice does not affect the observed increasing trend, matter when the input salience is generated concerning the logit, probability, or the loss. The results of Pind (k1 K2), generated from the saliency maps w.r.t. the predicted probability (after softmax activations), are shown in Figure 1.5 It can be found that Pind (1, kz) still increases with both kn and lzz
B Experiment Details
Model Details Throughout the experiments, we use CNNSmall, CNNLarge, ResNetSmall, and ResNetl as model families. Within each family, model width is controlled by the parameter And the model depths are controlled by the "Small" vs. "Large" suffixes. For CNNs, CNNSmall consists of convolutional layers with channels [ks, 2k, 4k, Sk], while CNNLarge repeats each layer twice: [k, A, 2k, 2k, 4k, 4k, 8k, 8k). The details of CNNs Tinylmagenet, since the input data of size 64 x 64, we increase the stride of the second MaxPool2d layer (Layer 10) to 4. As for ResNets, we modify the width of ResNet-10 for Seall and ResNet-18 for ResNetLarge. Note that & = 64 ResNetSmall corresponds to ResNet-10 10, while k = 64 ResNetLarge corresponds to ResNet-18. The sizes of models are illustrated in Figure 16 the # of trainable parameters.
shown in Table 2 Additionally, for
It should also be noted that, ideally, CNNSmall and CNNLarge, ResletSmall and ResNetl are considered as the same families due to the same architecture. However, since widths can adjusted independently of the depth, while the adjustment of depth inevitable affects the width, we split them.
16

---

-042
Back
Seed,
(a) CIFAR- 10, CS
(b) CIFAR-IO, CL
(c) CIFAR 10, RS
(d) CIFAR 10, RL
60
(e) CIFAR-100, CS
( CIFAR-100, CL
CIFAR-100 RS
(h) CIFAR 100, RL
4.00
Tinylmagenet, CS
Tinylmagenet, CI.
Tinylmagenet, RS
TinyImagenet, RI
The expected similarity p(k1, k2) between models of varying widths 1 12 Here we include CNNSmall (CS), CNNLarge (CL), ResNetSmall (RS), and ResNetLarge (RL) as F. The values of kı, k2 determine the widths in each layer. the models are initialized to random values without any optimizations. Here the datasets are CIFAR- 10 (top), CIFAR-100 (middle), and Tinylmagenet (bottom).
Figure 14:
Table 1:
The average Wasserstein distance E FeD ( (f((), S(f(2)) (((()) with the standard
deivation for all model families [CS, CL, RS, RL x (10, 20, 40, 80, 160) over CIFAR- 10/100 and TinyImagenet datasets. Note that here the baseline should be 2 since the cosine similarity lies in
[]. 1]. is observed that deeper models usually have larger distances (CS vs. CL, RS vs. RL). We deduce that this because of the training for deeper models more difficult.
CIFAR-10
k 10
20
40
A: 80
k 160
CS
0.0081 0.0051
0.0100
0.0060
0.0106
0.0062
0.0135 + 0.0094
0.0153 0.0106
CL
0.0143 + 0.0091
0.0171 1 0.0144
0.0322 1: 0.0295
0.0334 1 0.0287
0.0345
0.0271
1: 0.0059
0.0092
0.0073
0.0043
RL.
0.0096 + 0.0054
0.0107 = 0.0065
0.0053 0.0090
0.0090 + 0.0057
0.0163
0.0121
CIFAR-100
40
160
0.0078 + 0.0052
0.0056 0.0034
0.0054
0.0031
0.0060 : 0.0034
0.0085 U. U.0056
CL.
0.0098 1 0.0059
0.0099
0.0060
0.0156
0.0106
0.0184 1 0.0130
0.0222
0.0146
RS
0.0071 + 0.0034
0.0066 + 0.0029
0.0062
1: 0.0030
0.0061
+ 0.0029
0.0055
0.0029
RL
0.0087 0.0042
0.0037 0.0079
0.0078 0.0037
0.0029 0.0065
0.0078 0.0044
TlayImagenet
20
40
k:
160
CS
0.0033 + 0.0018
0.0021 = 0.0010
0.0018 +: 0.0007
0.0028 + 0.0013
0.0060
0.00-49
CL
0.0016
0.0032 0.0014
0.0129 0.0083
0.0201
0.0155
RS
0.0083 1 0.0052
0.0058 1 0.0038
0.0049 1: 0.0025
0.0045 1 0.0023
0.0041 0.0018
RL
0. 0.0062 + 0.0010
0.00G0 0.0035
0.0055 0.0030
0.0055 = 0.0031
0.0057 0.0031
But our experiments in Section 3.1 verify that the depths do not affect the population mean of model
17

---

6.0
6318
ACT
Crperity
Ueael Coundty
Meddd Couetry
(a) CIFAR-10. 10, CS
(b) CIFAR-IO, CL
(c) CIFAR-10 10, RS
(d) CIFAR-IO, 10, RL
837
638
835
1 way
Coundyy
(e) CIFAR-100, CS
(I) CIFAR-100, CL
100, RS
(h) CIFAR-100, RL
020
4014
814
A233
Pak Gamp
Mead Coaddyy
Nakel Caariyy
(i) Tinylmagener CS
(i) TinyImagenn Imageneett CI.
(k) TinyImagenct RS
(I) Tiny RL
Figure 15: The expected similarity k2) between models of varying widths k1, 12. Here we include CNNSmall, CNNLarge, ResNetSmall, and ResNetLarge as F. The values k k2 determine the widths in each layer. In particular, all the cosine similarities are between the input saliency maps of the predicted probabilities instead of the predicted logits. Here the datasets are CIFAR-I0 (top), CIFAR- 100 (middle) and tinylmagene (bottom).
1e8
CNN Small
of Trainable Parameters
6
CNN Large
ResNet Small
ResNet Large
4
3
2
100
200
300
400
500
Size
Figure 16: The of trainable parameters of models vs. the width parameter l: for each architecture.
18

---

Table 2: CNNSmall Model Details
Layer
Type
Parameters
0
Conv2d
3 inch, outch, 3, stride 1, padding
1
BatchNorm2d
2
ReLU
3
Conv2d
k inch, 2k outch, ks 3, stride 1, padding 1
4
BatchNorm2d
5
ReLU
MaxPool2d
ks 2, stride 2
7
Conv2d
2k inch, 4k outch, ks 3, stride 1, padding
BatchNorm2d
9
ReLU
10
MaxPool2d
ks 2, stride 2 4k inch, 8k outch, ks 3, stride 1, padding
11
Conv2d
12
BatchNorm2d
13
ReLU
14
MaxPoo12d
ks 8, stride 8
15
Flatten
Linear
in features=80, out_features n_class
1-11
WAc-211
(a) Depth-Witg Wise Similarity
(b) Width-Wise Similarity
Figure 17:
The cosine similarities between CNN models with different widths (left) and depths (right).
Widths and depths are enumerated in {16, 32, 64, 128, 256} and {1, 2, 3, 4, 5]. Note that to scale depths without affecting widths, here we fix the widths of all layers to k instead of [k, 2k, 4k, 8k) as the manuscript
distributions. In experiments, set X' the first 1000 samples of the unshuffled testing of each dataset (CIFAR-10/100, Tinylmagenet).
C Additional Settings
This work aims to reveal the convergence trend of the distribution of model behaviors under the stochasticity of the training criterion. This does not limit the conclusion to the specific criterion described above. Distinct training criteria can lead to different distributions of trained models. However, these different distributions of trained models all satisfy the revealed trend. To verify this, we present additional experiments to investigate possible variants such as (1) depths and widths; (2) learning rates; (3) batch sizes; (4) solvers (5) initializations, and (6) other model architectures. Note that studied in Figure/othe manuscript, Pind can be computationally efficient compromise of Therefore, we studied Pind these additional experiments. Besides, there exists enumerous possible combinations of different variants of different aspects. As result, here we only vary these settings partially since enumerating the entire grid is infeasible. The results as detailed as follows.
Depths and widths. The scale of depths is as straightforward the width since modifying depths may change widths as well. Therefore, in the manuscript we study the influence of depth by setting -small and -large variations. Here we present additional results that study the influence of depths continuously, with 1-5 layers, each of which is followed by max max-pooling layer with stride 2. Finally, an adaptive pooling layer is appended at the end. To rule out the influence of widths (channels), all layers have the same width, determined by k. e.g., For the 4-layer scenario, the intermediate layers have widths [A:, ki k. instead of [k:, 2k: 4k, 8k) the manuscript. The results shown in Figure I7 It can be found that (1) Given fixed depth width, the influence of the
19

---

The cosine similarity between CNNs
(h) The cosine similarity between ResNets
ResNetLaage
CHNSmall
RerketLarge
0.03-
3.0
1.0
0.02
E.D
2.010
0.7
0.6
3,30
0.00
Ind
10r
10a
102
10"
dbi
witth K
wicth
(c) The Loss of Models
(d) The Accuracy of Models
Figure 18: (a) and (b) illustrate the cosine similarity between (a) CNNs and (b) ResNets with different batch sizes {64, 128, 256, 512}. (c) shows the loss and (d) shows the accuracy trained models.
other factor is similar when scaled up. (2) Depths are slightly different from widths. Larger widths lead to higher similarities, while closer structures in depths have higher similarities. For widths (left), the similarity always increases left-to-right and hop-to-bottont But for depths (right), pairs near the diagonal have higher similarities.
Batch Sizes. We investigate the influence of batch sizes, varying in {64, 128, 256, 512] The results shown in Figure 18 It can be observed that although different batch sizes lead to different performance (e.g. testing accuracy), the convergence trend holds all scenarios.
Learning Rates. We test different learning rates how they affect the results. We include 1e-1, 1, 1e-2, default, where "default" refers to the criterion used in the manuscript. As shown in Figure I9] the revealed trend is preserved all learning rates. It is also worth noticing that learning rates aftect ResNets more than CNNs.
Solvers. Apart from SGD, we include Adam, AdamW, and SGD w/ momentum. For Adam and AdamW we set the learning rate to 1e-3, while SGD w/ momentum uses learning rate of le-1 with a momentum of 0.9. The results are shown in Figure 20) Although diffcrent solvers lead to models of different performances, they all preserve the same convergence trend.
Table 3: The comparison between the similarities between single models of different criteria.
(a) Diff. Init.;
(b) Diff. Init.;
(c) Same Init. Oo: Diff Order
(d) Same Init. 0ı; Diff Order
Same Order
Diff. Order
# of pairs mean of Pind
100
4095
4095
4095
0.0758
0.0753
0.0879
0.0855
std. of Pint
0.0038
0.0037
0.0042
0.0048
Initializations. Given training scheme and model family F(k), the training procedure leads to distribution of trained models p(f). When the initialization fixed to 0, the training procedure essentially sampling from the conditional distribution p(f(0) instead of the unconditional distribution p(f)) We then studied the difference between the unconditional distribution p(f) and the conditional distribution p(f)0o). We focus on two conditional distributions p(f(0o) and p(f(0)), where 0
20

---

00.01 0cf.ul
15SP 94 RE07
52 28 82
A62
VIS
PAD
The cosine similarity between CNNs
VI
wan
(b) The cosine similarity between ResNets
ResNetLarge
Resheet arge
Xo44
3.0
1.0
3,010
ven
0.02
0.5
0.7
85-512
066
3,00
0.00
Aut
20
107
200
101
101
101
10
with
att
mitth k
michh
(c) The Loss of Models
(d) The Accuracy of Models
Figure 19: (a) and (b) illustrate the cosine similarity between (a) CNNs and (b) ResNets with different batch sizes {64, 128, 256, 512}. (c) shows the loss and (d) shows the accuracy trained models.
10238821022
522
No.
No.
(a) Cosine Similarity
Accuracy
Loss
0.8
0.7
(b) Performance
Figure 20: (a) The cosine similarity between CNN models with different solvers (SGD, Adam, AdamW, SGD w/ Momentum). (b): The accuracy and loss of trained models.
21

---

Similarity p(kg.kg)
Similarity of Means
0.07
0.75
10
10
0.06
0.70
20
20
0.65
0.05
4
A2
0.60
0.03
0.50
80
80
0.45
10
40
10
Figure 21: The cosine similarity between Vision Transformers (ViTs) on CIFAR-10. The capacity controlled by & {10, 20, 40, 80), where the embedding dimension is 4k, separated to k/2 heads. The left subfigure shows the mcan of the similarity. The right subfigure shows the similarity of the population mean.
represents the initializations under seed=1 and 0g represents those under seed=0. Other settings ~ p(f(fu) and fl. ........ . ~ p(f(01). The superscript are identical. We thus have .'. , f indicates the initialization seeds and the subscript indicates the training seeds.
First, we notice immediately
the training seeds for both On and 0, are 1 100. This means that vi, fi J differ only in initializations. We inspect (a) Pind ( f (100 pairs) to see if they have 4950 pairs). Besides, within
exceptional similarity compared with (b) (ina(J. fJ), # ((120) the same condition, all models only differ in terms of the orders of the training batch. We thus also inspect the similarity of all models of the same condition: (c) Pind(f) f)), + j and (d)
Pmnaft f)), i # j. Each them has (?2) = 4950 pairs.
As demonstrated in Table (i) the comparison between (a) and (b) indicates that with different initializations, the same order batches in the training procedure does not contribute higher similarities. (ii) The comparison among (b)(c) indicates that the same initialization indeed leads to higher similarity even though the order of batches is distinct. It should noted that the contributions of batch orders and initializations are also affected by the number of epochs. Intuitively, more training epochs should lead smaller contributions from the initializations but greater contributions from the batch orders.
Vision Transformers. Vision Transformers (ViT) have risen in recent years as another powerful architecture for CV tasks. Here we include brief study of ViT' to demonstrate that the discovered phenomenon holds across different architectures.
Specifically, we train vision transformer (ViT) models on CIFAR-10 with varying capacities controlled by {10,20 20, 40, 80). CIFAR-10 has an input size of 32 x 32 pixels, thus the patch size is set as 4 x 4, resulting in 8 x 8 patches. The embedding size is set to 4k, divided by k/2 heads, and we set the depth 8. The seeds vary in 1-100 and results in 100 trained models of each k. We study the mean of the similarity p(k1, k2) (i.e. the same experiments as Figure 3 in the manuscript) and the similarity of the population mean (i.e. the same Figure the manuscript). The results are shown in Figure 21 It can be observed that although distinct from convolutional layers, the transformer structure also has the discovered convergence trend. It can also noted that the degree dispersion of ViTs is much higher than CNN-based models.
In conclusion, although training schemes affect the resulting distributions of models, the influence of the model capacity stays invariant across different criteria.
D Uniform Distributions on the Hypersphere
According to (Muller 1959). due to the spherical symmetry property the zero-mean Gaussian distribution, the cosine similarity between two Gaussian variables actually uniformly distributed
22

---

over the hypersphere S in R. Therefore, the cosine similarity between two i.dd. multivari- Gaussian tensors is essentially the inner product between two i.idd uniform tensors over the hypersphere. Formally, suppose X,Y to be high-dimensional i.i.d. random variables of dimen- sion d that follow the Gaussian distribution (O, laxd). The cosine similarity is Z x²y
WLOG, it suffices to consider the scenario where y e1 (1,0,-., ,0). And it is written as [MM]
Z = c
X X1 |X||
X1
X Note that x? ~ x2),,, Ei= x² (d-1).
|xil
E X7
As result, Z² x x?
follows beta distribution Z ~ Betal d=1). The pdf is thus
fz) x x-1/2 (1-a
And then when Z > 0,
(1)/2)
fz(z)=fzz((t) =fz=(")1221
(x²) -1/((1 - x22) (d-3123)
[2xl
(8)
B(1/2 (d 1)/2)
2
(1 x²) 22)
d-1) B 21
According to (Smith et al. 2023). let u = (1 2,, x2² = =(2u 1)?.
Then this can be simplified
2
(1 (2u 1)2) (3-32/2 dr
fu(u)
(10)
B d-1)
du (d- 1)/2 (1 ud
2d u
(11)
B
d-1
2
u
1)/2 (1 2) (d-11/22 13/2-1
(12)
B d-1 )22-d
2
d- d
1
Beta
(13)
2
This is because
d 1 d
- 1
d 1-1 d+1,
B
B
.2
(14)
2
2
r d-
) d+1
2
2
2
(15)
I'dd
r d (( ) (+)
-2
(16)
2 π 1/21 d=ه ((
2
22 I ((
B( 1 d 1 222-d
(17)
I(d)
Finally, we are able to conclude that 1+Z Beta f - where Z is the cosine similarity between two i.i.d. d-dimensional Gaussian vectors. This result suggests that the distribution of Z will become very concentrated around O. And this concentration exacerbates exponentially with the dimension d. Given cosine similarity level p. the probability P(Z > p) can be extremely small, and also deceases exponentially with p. too. We visualize the relation between the probability P(Z > p) and p with varying dimensions in Figure 22 In the low-dimensional space such d = the distribution of the cosine similarity is close to uniform as humans' intuition. However, as the dimension increases, the cosine similarity is very unlikely to maintain high, as demonstrated by the curves of d = 3 x 8 x 8 = 48 (orange) and d = 3 x 32 x 32 = 3072 (green).
2
"This is because of the rotation-invarianee Let O dxd be any orthonormal matrix, then after the rotation we obtain OX [[X
T Since OX (N( 7), too, we know that OX and [[] are from the identical [[
distribution.
23

---

The CDF the cosine similarity with different dimension
PIZ-p), a.k. area/Sphere
10
10-5
10-8
10-11
d=3
10-14
d' 192
d= 3072
0.0
0.2
0.8
1.0
Cosine Similarity p
Figure 22: The relation between the probability (P p) and the cosine similarity value
E
Supplementary Experiment Results
Note that all the experiments are carried across three datasets CIFAR-10/100 and Tinyemagenet, along with four model families CNNSmall, CNNLarge, ResNetSmall, and ResNetLarge. Due the space limit, only part of the selected results can be put in the manuscript. Therefore, we defer the results with different models/datasets into this section for the audience' reference. All the conclusions drawn from the experiment results shown in the manuscript hold for the results demonstrated here.
In Figure 23
we present the complementary results of Figure on CIFAR-100 and Tinylmagenet. can be clearly observed that Pind and have very similar values, after taking the expectation over X.
Figure 25 shows the results of black-box attack results between different models. All other settings are identical to the results shown in Figure I0. but with CIFAR- 100 and TinyImagenet instead of CIFAR-10. It is observed that the capacity of models has significant influence to the models' robustness and black-box attack transferability. And this trend is highly correlated to the similarity
Pind, P demonstrated in Figures 2
12
13 and 15
Similarly, Figure 26 shows supplementary results Figure 11
All settings are identical except for are tested instead of CIFAR-10. It can be observed that using the estimated mean gradient direction (blue), the performance drops much more significantly than the attack from single models of either the exact same family as the target model (green) the largest single model tested (k 448, red). Note that due to the complexity of Tinylmagenet, overfitting phase (i.e. when the target models' capacities comparable to the dataset (Nakkiran cct al. 2021)), the single-model black-box attack results in opposite effect the prediction actually increases.
the datasets. CIFAR- C1FAR-100 and
In Figure we present CIFAR- 100 and Tinylmagenet results as supplementary of Figure 9 can be observed that the testing loss is highly correlated to the expectation of u (z)).. Such phenomena are also consistent across different model families and datasets. For both single models and ensembles, the closer they performance they have, Note that here for the sake consistency, we approximate (xx) through jū( 71160;;a æ). Therefore, in the ensemble experiments (left of each subfigure), & = 160 is omitted.
the convergent limiting point (i.e. larger EE({), the higher testing
24

---

CS
RS
RL
Similartt Metricc
0.3
0.2
0.2
0.1
0.1
0.0
0.0
200
400
200
400
200
400
Width
Width
Wieth
Width
(a) CIFAR-IAE 100
CS
CL
RS
RL
0.3
Siilerrty Mettic
0.20
0.10
0.2
15
0.15
0.10
0.05
0.1
0.05
0.05
0.00
4.00
400
200
400
200
400
Width
Wicth
Width k
Width
(b) Tinylmagenet
f(2)), Figure 23: Illustration of (red) f(2)), f(), f F(k) and (blue) p(k, k) (a) CIFAR 100 and (b) TinyImagence
Single Models
Ensembles
Single Mode's
Ensembles
Testing Loss
Rosting Loss
Betinn
Pestngg
00
n2
Norginal NarghnalBBgperatoe Experrttoon110 =[t]
Mtaggnaal Expectation. (15
Norginal Expectation [t]]
Monghnal Expoctation 100
E3iLaage
ange
chklange
BushistLagge garge
(a) CIFAR- 100
(b) TinyImagenet
Figure 24: The illustration of the relation between the expected testing loss Ex[[C [C] and the marginal expectation [x](). Both (a) CIFAR-100 and (b) TinyImagenet results shown supplementary Figure 9 Models are from (i) single models with varying structure; and (ii) deep ensembles with varying members. Each color represents model family.
25

---

1.0
10
1.5
0.5
0.
0.5
08
04
04
0.4
130
192
82
192
62
192
62
192
42
354
5.0
304
354
304
Target Model
Targe: Mocelk
Target Model
Target Modelk
(a) CIFAR100, CS
(b) CIFARIO0, CL
(c) CIFAR1O0 RS
(d) CIFARIOO, RI.
2.0
1.0
17.
12
Moomle
24
06
0.6
0.6
06
0.4
0.4
0.4
04
128
128
126
258
256
250
238
50
00
6.0
Model
Model
(c) Tinylmagenet, CS
(f) Tinylmagenet, CL
Tinylmagenet,
(h) Tinylmagenct, RL
Figure 25: The results of single model black-box attack. The value of each entry is f(
f(l) E F(k2), f(2) E F(k22 for different model capacities. Here k is the width parameter of the source model and ka is the width parameter of the target model.
by CCGNN
Decny CGoa
Ped pten Dooa Betbr
spaccop Gamppii Aponttis
Prestctton becay
60
10
Prediction
2.a
407
4x
201
101
10'
102
Target Mocel
Target
Target
Target
(a) CIFAR100, CS
(b) CIFARI00, CL
(c) CIFARIO0, RS
(d) CIFARI00, RL
101
Atach
Mirch tywitre Jercc ay
10-
fhem Boll Cootole
Artack al
Pralccin Dooay Bater
6x.10
Decay
10?
2x110-3
Predictin
10
3x
2x10
16a
10a
Mode: Witth
Modal Wilth
Wi
(e) Tinylmagenet, CS
(f) Tinylmagenet, CL
(g) TinyImagenet, RS
(h) Tinylmagenet, RL
Figure 26: The comparison between the single-model attack from the largest model (red), the single- model attack from the very same structure (green), and the attack by the mean direction (blue). The top row shows the results CIFAR- 100, and the bottom row shows the results of Tinylmagenet. Some figures' y-axis are set logarithm for clarity.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract mainly points out the hypotheses made in this work. They all properly discussed and empirically verified the manuscript.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer the conclusion section for the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

The theoretical analysis in this paper is rigorous. Hypothesis are both theoreti- cally explained and empirically verified.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in the appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The information needed to reproduce the experiments is discussed in both the manuscript and the appendix. The code is also provided in the supplementary materials. The repository will be publicized upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
28

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code is provided the supplementary material in the submission. The repository will be publicized upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
mublicc/quiees/Codeebbbass casssiinooolyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppplcccuui.eeaeeecccuea wwweedeee/eooebbmmmceee ottpee///ooooeeuueisssoooooooooo www.ssssiicggoooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details of the experiments are discussed in both the manuscript and the appendix. The code is also provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The potential statistical significance lies in the variance among different ensemble members. This is studied Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to the first paragraph of Section 3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpss//neuriips
Answer: [Yes]
Justification: The authors have reviewed the code of ethics and this work conforms with the code of ethics of NeurIPS.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Focusing on fundamental, theoretical and general topics, this works does not have societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
30

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no such risk of this work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: This work only uses general packages such Numpy, PyTorch, torchvision. etc.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "ESb1C26c8L", "reviewer": "Reviewer_YpLK", "summary": "This paper investigates the distribution of trained deep neural networks through the lens of input saliency maps. It empirically shows that as the capacity of either of two stochastically optimized models increases, they tend to resemble each other more. Therefore, the authors hypothesize that within the same architecture (Resnet or CNN), different variants align in their input salience directions, and converge towards a shared population mean as their capacity increases. Furthermore, they propose a semi-parametric model, based on the Saw distribution, to capture this convergence trend. These findings enhance the understanding of various applications such as black-box attacks and deep ensembles.", "questions": "1. Have the authors conducted experiments to validate their findings across different optimizers and related hyperparameters? \n2. Stochasticity can be introduced in the training of Neural Networks in various ways. Changing the initialization through different seeds is one possibility but other possibilities include for example training on different subsets of the data, large or small batch sizes, large or small learning rates, data augmentation or label noise. Have the authors validated their findings for different sources of stochasticity? \n3. In Figure 8, why does the Large ResNet exhibit a higher test loss than the small CNN? \n4. The explanation for the effectiveness of deep ensembles suggests that they better approximate the population mean. However, this assumes that the mean corresponds to a good generalizing model. Could the authors provide evidence or an explanation to support this assumption?\n5. The introduction mentions \"input salience\" several times without providing a definition. For better clarity and understanding, it would be helpful to include a definition of \"input salience\" in the introduction.\n6. In section 2.1, the authors state, \"We focus on $f: \\mathbb{R}^d \\to \\mathbb{R}$ which predicts the logit specifically for the targeted class.\" Could the authors clarify this statement? Does it mean that the analysis after training considers only the logit corresponding to the true class?\n7. In section 2.2, $\\mathbf{u}_1$ and $\\mathbf{u}_2$ are sometimes used to represent different samples from the same set $\\mathcal{G}_k$, and sometimes from different sets.  It would be better to use a different notation to differentiate between the two cases. \n8. In Hypothesis I, line 147, the term $\\mu(k;x)$ is only introduced later in the text. \n9. In line 234, could the authors clarify the definition of $\\rho_{ind}(f,f;x)$ and explain its relationship with $\\rho(k,k;x)$ ?", "limitations": "The limitations of the study are correctly addressed in the manuscript.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written.\n2. The experiments are comprehensive and well-designed.\n3. The input saliency maps perspective on the ensemble problem is novel.", "weaknesses": "1. The primary finding is not entirely novel. Previous research has demonstrated that the performance of neural network ensembles converges to that of a single model as the model size increases [1,2]. Similarly, as the size of the models increases, the variance in predictions and disagreement among ensemble members diminishes [1].\n2. Several details about the experimental setup are omitted. These include the optimizer used, learning rate, learning rate schedule, batch size, number of iterations, weight decay, and data augmentation techniques.\n3. The choice of optimizers and hyperparameters can significantly alter the distribution of trained models. However, the study lacks an analysis of their findings across different optimizers, hyperparameters and sources of stochasticity.\n4. The experiments are conducted in small-scale settings, which may limit the generalizability of the results.\n\n[1] Geiger, Mario, et al. \"Scaling description of generalization with number of parameters in deep learning.\" _Journal of Statistical Mechanics: Theory and Experiment_ 2020.2 (2020): 023401.\n\n\n[2] Kobayashi, Seijin, Johannes von Oswald, and Benjamin F. Grewe. \"On the reversed bias-variance tradeoff in deep ensembles.\" ICML, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer YpLK\nWe thank the reviewer for the invaluable feedback. We address the questions as follows.\n\n**[Novelty (W1)]**\n\nWe summarize the work in [5,6] and how the novel contributions of our work on deep ensembles differ from them. It should also be noted that exploring the mechanism of deep ensembles is not our main contribution, but an application and verification of our work.\n- [5] studies the failure modes of deep ensembles and. However, [5] focuses on empirical observations of the performance of ensembles. This is never claimed as our contribution. Note that the comparison between the **performance** of single models and ensembles is not novel and can be found in many works discussed in our manuscript (e.g. [7]). However, the reason behind deep ensembles' performance gain is still mysterious [5]. Our work hypothesizes and verifies that (1) single models' performance gain is related to the convergence trend, and (2) the convergence is towards the shared mean of **all capacities**. This provides a novel perspective to understand deep ensembles and why an ensemble of many *small* models can achieve the performance of a single *large* model.\n- [6] studies the double descent phase of DNNs using NTKs. The analysis based on NTK verifies similar results for deep ensembles as in [5]. As a result, the novelties of our work discussed above still hold. The similarity of mechanisms between models with different capacities and how the convergence is towards the shared population means across capacities are also novel compared with existing work. Additionally, NTK is a powerful tool yet with limitations and strong assumptions. The analysis is on the global loss regarding the data distribution (Eq. 4 in [6]). However, our results are on the direct prediction of any single input and its neighbors.\n  \nWe will add these works to the related work section for a more comprehensive review of existing work.\n\n**[Experiment Setups (W2)]**\n\nTo stay consistent across experiments, we follow the setups of [1] as claimed in L195-197. We use SGD solver with a learning rate $\\gamma = 0.1/\\sqrt{1+epoch}$. No weight decay, momentum, or data augmentation are included to minimize variants. The batch size is 128. All models are trained for 200 epochs.\n\n**[Sources of Stochasticity (W3, Q1, Q2)]**\n\nTraining schemes do not affect the discovery. Please refer to the global response for details, including comprehensive additional experiments.\n\n**[Scale of Experiments (W4)]**\n\nThis work is mostly based on the benign overfitting and double descent phenomena (e.g. [1-4]), where large models counterintuitively do not exacerbate overfitting. We aim to provide insights into this mysterious capability of DNNs from the perspective of XAI. Hence the settings of our work are mostly based on existing studies on this topic, too.\n\nThe complexity of the datasets used in our experiments is a strength, not a weakness. Unlike existing works, which mainly use CIFAR (e.g. [1]), we test our hypotheses on TinyImageNet, which includes 200 classes and 110k samples, offering a more complex and realistic data distribution. The same trends observed in CIFAR datasets are also evident in TinyImageNet, suggesting these trends will hold for even more complex datasets like ILSVRC.\n\nWe admit that carrying the experiments over ILSVRC can be infeasible for the studies of benign overfitting. As explained in Appendix B, with $k=64$, ResNetLarge is already equivalent to ResNet-18. We scale $k$ up to 448, which makes it impossible to train a group of models to estimate the empirical mean. For example, in [4], a study of overfitting carried out experiments on CIFAR and SVHN.\n\n**[Deep Ensemles and Single Models (Q4)]**\n\nAs the capacity increases, the distribution becomes more concentrated (illustrated in Fig. 4(a), verified in Fig. 3,5,6, etc.). This suggests that *as a model approaches the population mean, the performance increases*. To verify this, we carry out the experiments on deep ensembles since they approach the population mean of the model directly instead of by increasing capacity like single models. Fig. 8 shows that although approaching the mean differently, deep ensembles and single models have similar trends. This provides support for the connection between the population means and better generalizability.\n\n**[Clarifications]**\n- Q3 [ResNet]. ResNets have higher expressiveness compared with CNNs and rely heavily on data augmentations. Since model performance is beyond our scope, data augmentation is not included. Fig. R5 shows additional results of ResNets with data augmentations. It improves the performance of ResNets greatly. However, the convergence trend is not affected.\nNLL is also affected by confidence. ResNets tend to be more confident in wrong predictions, leading to higher losses.\n- Q5 [Input Salience]. Input salience is usually used by the XAI community to refer to the input gradient as a saliency map, and also used to refer to attribution-based methods. We will formally introduce the definition of the concept.\n- Q6 [Logit]. Logits refer to the prediction of the target class before softmax. We also include the results of the post-softmax output in Appendix A. It shows that this does not affect the results. Note that taking the gradient of the output probability of the target class is equivalent to the gradient regarding the negative log-likelihood, normalized by the probability.\n- Q7 [$u_1,u_2$]. We will use $u_1,u_2$ to refer to gradients of different sets and $u,v$ to refer to those of the same sets.\n- Q8 [$\\mu(k;x)$]. We will move the definition of $\\mu(k;x)$ to H1 for a better presentation.\n- Q9 [$\\rho_{ind}$]. $\\rho$ is defined in eq. (1). It refers to the similarity between models of capacity $k$ and is approximated by taking the average over 100 models each. $\\rho_{ind}$ refers to the model-dependent similarity computed regarding the two models (in L233). We will define it more clearly to avoid ambiguity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lYUJOlKBQN", "reviewer": "Reviewer_fWRt", "summary": "This paper studies the distribution of input saliency maps of trained neural networks at varying depth and width. The authors observe that as the model capacity increases, these distributions converge towards and become concentrated around a shared mean direction. The authors also state two hypotheses for the behavior of these distributions and provide empirical evidence for them.", "questions": "- Have the authors explored the effects of different optimization hyperparameters (e.g. learning rate, batch size) or optimization algorithms (e.g. Adam) on the convergence trends observed in the paper?\n- The experiments in this paper are focused on computer vision. Do the authors expect similar conclusions to hold in other domains as well (e.g. NLP)?", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper clearly formulates and discusses two hypotheses (H1,H2) that help capture the empirical observations.\n- While the experiments are small scale, they are sufficient to support the authors main claims and observations.", "weaknesses": "- The authors do not distinguish between the stochasticity introduced by random initialization and the stochasticity introduced by optimization. In Appendix A, they discuss the similarity scores at initialization, but a more interesting ablation would be to compare fixing the initialization and then randomizing the batches with fixing the ordering of the batches but randomizing the initialization.\n- Related to the above point, there is no discussion of the effect of the learning rate or batch size which should greatly affect the final results. This would help disentangle the randomness introduced by initialization vs optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer fWRt\n\nWe appreciate the reviewer very much for the acknowledgment of our work. To further strengthen the reviewer's confidence in our findings, we address the reviewer's remaining questions as follows.\n\n**[Fixed Initializations with Random (W1)]**\n\nWe thank the reviewer for the suggestion of separating the stochasticity of random initialization of the model weights and the stochasticity of the randomization of batch orders in SGD.\n\nIn order to address the reviewer's concern, we carry out additional experiments, where models are all initialized with the same weights by setting `seed = 0` before the initialization. After the initialization, we set again manually to 1~100 in the training process. The experiments are carried out using CNNSmall models with $k=20$ on CIFAR-10.\n*It should be noted that this is actually equivalent to uniformly sampling from the original distributions. Therefore, we expect an almost identical distribution compared with the models used in the manuscript, where seeds are determined at the beginning of all steps.* Since the normalized gradients $\\mathbf{u}$ are of 3072 dimensions, it is infeasible to compare the distribution directly. To verify the uniformity, we inspect $t = \\mathbf{u}^T\\mathbf{\\mu}(\\mathbf{x})$ described in L177-190 of the manuscript. This marginal distribution is 1-D on $[-1,1]$ and we study the Wasserstein distance between the distribution of models with identical (random) initialization and the distribution of models with different (random) initializations. The distance is approximately $0.0225$, which means that the distributions are very close in terms of the dispersion degree.\n\n\n**[Other Sources of Stochasticity (W2, Q1)]**\n\nWe agree with the reviewer that different training criteria such as the learning rates, the batch sizes, the solvers, etc. can affect the resulting models. However, it should be noted that a criterion determines a distribution of trained models. And the phenomenon discovered in this work holds for all kinds of criteria instead of the one used in the manuscript.\n\nTo completely resolve the reviewer's concern, we carry out additional experiments to investigate the influence of different training options, including learning rates, batch sizes, solvers, and data augmentations. The experiments are described in the global response in detail, and the results are presented in the attached pdf file.\n\n**[Other Data Types]**\n\nWe thank the reviewer very much for the question regarding other data types. We clarify that text data is not commonly studied in the literature on benign overfitting. Additionally, conventional NLP training is typically conducted in an unsupervised manner, making it difficult to carry out similar experiments. However, since NLP data are now often trained using transformers, we investigate whether the transformer and attention mechanism exhibit this property as a starting point.\n\nSpecifically, we train vision transformer (ViT) models on CIFAR-10 with varying capacities controlled by $k\\in \\\\{10, 20, 40, 80\\\\}$. CIFAR-10 has an input size of $32\\times 32$ pixels, thus we set patch size to be $4\\times 4$, resulting in $8\\times 8$ patches. We set the embedding size to $4k$, divided by $k/2$ heads, and we set the depth to $8$. We then vary seeds in 1~100 and train 100 models of each $k$ and study the mean of the similarity $\\rho(k_1,k_2)$ (i.e. the same experiments as Fig. 3 in the manuscript) and the similarity of the population mean (i.e. the same as Fig. 5 in the manuscript). The results are shown in Fig. R6.\nIt can be observed that although distinct from convolutional layers, the transformer structure also has the discovered convergence trend. It can also be noted that the degree of dispersion of ViTs is much higher than CNNs.\n\nTherefore, based on the additional experiments on transformers, we can deduce that the convergence trend may also hold for NLP tasks where transformers are extensively used.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h5EFYrpPC4", "reviewer": "Reviewer_RiRS", "summary": "This paper studied the uncertainty introduced in stochastically optimized DNNs via the input saliency maps. By empirical evaluations, the authors discovered that 1) within the same model architecture, models with different capacities tend to align in terms of their population mean directions of the input salience, 2) the distributions of the optimized models concentrated more around their shared population mean as the capacity increases. These observations are very interesting and shed some light on the applications of DNNs such as black-box attacks.", "questions": "Besides the points I made in the Weakness section, I have one more questions. When practitioners scale their models, they usually increase both depth and width simultaneously (especially in the context of scaling law). In your experiments, you mostly studied the effect of increasing width. Could you add results in the flavor of Figures 2 & 3 where the model capacity is parametrized by both the depth and width following some scaling law relationship?", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors took a suitable approach to study the uncertainty introduced in stochastic training of DNNs. The observations made by them are very interesting and inspiring, and well supported by empirical evidences. The authors did a good job explaining their main discoveries using figures and equations. These findings could shed light on the practice of DNNs.", "weaknesses": "The main complaint I have on the paper is its presentation. Although the observations quite interesting to me, I found the writing of the paper can be improved.\n\n1. More context is needed for Figure 1.\n1.1. How are the dots plotted? I assume you did something like PCA to reduce the gradient dimension to 3 and then plot them on the sphere.\n1.2. You said on line 55 that the second hypothesis is illustrated in Figure 1(b), but there is no indication of model capacities in the figure.\n\n2. Even though the authors tried to distinguish between model architecture from model family, the introduce of width and depth complicates the discussion and makes the definition less clear. It would be good to refine the presentation here. For instance, you could give a concrete example like ResNet_depth10_width10_seed0 and ResNet_depth10_width10_seed1 has the same model architecture and belong to the same model family. ResNet_depth10_width10 and ResNet_depth10_width20 has the same model architecture and belong to different model families, etc.\n\n3. It is unclear to me what is defined on line 96.\n\n4. The presentation of the paragraph on line 101 and Figures 2 & 3 can be improved.\n4.1. Why do you use the notation $\\rho_\\text{ind}$? What does ind mean?\n4.2. Can you give clear, different names to $\\rho_\\text{ind}$ and $\\rho$, and make both of them display equations?\n4.3. In the captions of Figures 2 & 3, can you use the $\\rho$ notations and refer to their display equations, respectively?\n4.4. The x and y labels in Figures 2 & 3 should be $k_1$ and $k_2$.\n\n5. $\\mu(k; x)$ is not defined in Hypothesis I.\n\n6. Some $k_1$ should be $k_2$ on line 158 and in the display equation below. And why do you need $k_1 > k_2$ for this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer RiRS\n\nWe thank the reviewer very much for acknowledging our work! We now answer the reviewer's questions as follows to make sure that all remaining concerns are resolved.\n\n**[Figure 1 Clarifications (W1)]**\n\nWe appreciate the reviewer for pointing out the potential ambiguity of Fig. 1. We clarify that Fig. 1 is for demonstration purposes and to provide a straightforward intuition for the hypotheses and the phenomena. We will update the manuscript to make sure there is no miscommunication. 1.1. The dots are plotted synthetically for a clear visualization. And 1.2. Fig. 1(b) is to illustrate the hypothesis of (i) shared means and (ii) convergence trend compared with Fig. 1(a).\n\n\n**[Model Architecture Clarification (W2)]**\n\nWe thank the reviewer very much for the valuable point. For a better presentation, We will add examples to elaborate on how model architectures and families are defined, following the ways suggested by the reviewer.\n\n**[Definition of $\\mathcal{F}$ in L96 (W3)]**\n\nL96 aims to define the space $\\mathcal{F}$ of functions we focus on in this work.\nNote that given a fixed input dimension, all models can be seen as a function $f:\\mathbb{R}^d\\rightarrow\\mathbb{R}$, regardless of the architecture, the capacity, etc. However, if the space of models $\\mathcal{F}$ is simply defined as all such functions \n$\\\\{f\\in\\mathcal{C}|f:\\mathbb{R}^d\\rightarrow\\mathbb{R}\\\\}$, many invalid models will be included. Therefore, we add a constraint to the functions in $\\mathcal{F}$ that they fit the training distribution well. We will revise the definition to a mathematical formula like $\\mathcal{F} = \\\\{f\\in\\mathcal{C}(\\mathbb{R}^d)|\\mathcal{L}(f;\\mathcal{X}\\_{train}, \\mathcal{Y}\\_{train}) < \\epsilon \\\\}$.\n\n\n**[Definitions of $\\rho,\\rho_{ind}$ (W4)]**\n\nWe thank the reviewer for the suggestion! It can indeed be a little confusing between the notations $\\rho$ and $\\rho_{ind}$. We clarify them as follows. \n\n4.1. The subscript \"ind\" in the notation $\\rho_{ind}$ refers to \"individual\". It is defined in L103 as **the cosine similarity of two models**. Note that the inputs of $\\rho_{ind}$ are $f^{(1)},f^{(2)}$, which are two functions (models). Differently, $\\rho(k_1,k_2)$ (defined in Eq. 1) represents **the expectation of the cosine similarity between two arbitrary models from $\\mathcal{F}(k_1)$ and $\\mathcal{F}(k_2)$**. It takes the width parameters $k_1,k_2$ as the input. Ideally, $\\rho$ should be used to study the similarity. And we carry out the experiments by training 100 models for a given $k\\in\\\\{10,20,40,80,160\\\\}$ for the four model architectures and three datasets. This already results in $100\\times 5\\times 4\\times 3=6000$ trained models. To demonstrate the influence of the capacity $k$ in a higher resolution, we investigate $k\\in\\\\{8, 10, 12, 14, 16, 20,\\cdots, 384, 448\\\\}$ (L106). This is computationally infeasible for $\\rho$. Therefore, we compare the results between $\\rho_{ind}$ and $\\rho$ in Fig. 6 to verify that **$\\rho_{ind}$ serves as a computationally efficient surrogate for $\\rho$**.\n\n4.2. We will term $\\rho_{ind}$ and $\\rho$ clearly as \"individual similarity\" and \"average similarity\" to avoid ambiguity. And as suggested by the reviewer, we will move the definition of $\\rho_{ind}$ (L103) and $\\rho$ (Eq. 1) together for a better presentation.\n\n4.3. Originally, Fig. 2 serves as the purpose of motivating the problem and appears before the definition of $\\rho, \\rho_{ind}$. Thus $\\rho_{ind}$ was not used in the caption. As suggested by the reviewer, we will re-arrange the figure and text to include $\\rho_{ind}$ in the caption of Fig. 2 for a better presentation.\n\n4.4. We appreciate the reviewer for pointing this out. We will revise the labels in Fig. 2 and 3.\n\n\n**[$\\mathbf{\\mu}(k;\\mathbf{x})$ in H1 (W5)]**\n\nThanks for pointing this out! $\\mathbf{\\mu}(k;\\mathbf{x})$ refers to the normalized population mean of the gradient, which is defined in L152. We will re-arrange this part and move the definition to \"The Intra-Family Hypothesis.\" for a better presentation.\n\n**[$k_1,k_2$ in L158]**\n\nNote that in L158, we are discussing the cross-family similarity regarding the population mean of each family. Since it is studied through the cosine similarity, it is symmetric between $k_1$ and $k_2$. Note that for cross-family similarity, $k_1\\ne k_2$. Therefore, the condition $k_1>k_2$ is equivalent to the condition $k_1\\ne k_2$ in H2. We will revise the manuscript to make it consistent throughout the discussion.\n\n\n**[Depths and Widths (Q1)]**\n\nWe appreciate the valuable suggestion from the reviewer. We acknowledge that the depth is also a factor that affects the model capacity. However, since the depth change usually leads to inconsistency in widths, we study the influence of depths by the -small and -large suffixes in the manuscript.\n\nTo address the reviewer's question regarding the influence of depths completely, we carry out additional experiments with different settings so that depths and widths can be altered independently by parameters $d$ and $k$, respectively. $d$ determinse the number of layers, and $k$ determines the width of the layers. Unlike the gradually increasing channel (e.g. $[k,2k,4k,8k]$) in the manuscript, here we set the same number of channels for all layers. In this way, the change in the number of layers does not affect the widths anymore. For example, when $d=3$, the layers are $[k, k, k]$. The results are shown in Fig. R1 of the attached PDF file.\n\nIt can be found that (1) Given a fixed depth or width, the influence of the other factor is similar when scaled up. (2) Depths are slightly different from widths. Larger widths lead to higher similarities, while closer structures in depths have higher similarities. This is verified by higher similarities near the diagonal entries for Fig. R1 (Right) compared with Fig. R1 (Left).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eChC49WBU3", "reviewer": "Reviewer_We4a", "summary": "This paper studies the resemblance of the functions obtained by training neural networks across various network depth and width. They do so by looking at the gradient field of the learned functions $f_1, f_2$, and defining their similarity by taking the cosine similarity between the gradients, then taking independent average over the inputs of the functions. They empirically observe that increasing the width results in higher similarity between the gradient fields, from which the deduce to properties: (1) the average gradient norm increase with width of the net, and (2) the direction of the average almost does not change when going to networks with larger width, but the variance decrease and the distribution becomes more concentrated. They also calculate the similarity measure and some properties of the gradient distribution assuming the distribution is gaussian or spherically symmetric. The major claim of. this work is summarized in hypothesis H1 and H2 on lines 145 and 160.", "questions": "What is the conclusion for training/optimization/generalization efficiency the authors look into obtain from this observation? To me the average of normalized gradient seems not to be correlated with generalization ability of neural nets. maybe the authors can add more concluding points or remarks about their empirical observation.\n\n- What is the difference between the message of Figure 3 and 5?\n\n- Are you assuming the gradient distribution is gaussian in section C?", "limitations": "The claim needs further be backed by more delicate experiments. Furthermore, the assumptions for calculation of the behavior of the similarity measure using certain classes of distributions needs to be justified at least in some regime of training DNNs. For the rest see weaknesses above.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Authors formulate an interesting hypothesis based on empirical observation regarding the behavior of the gradient field of learned functions in DNNs: the direction of the vector does not change too much when we increase the size of the network. They form this hypothesis from the observation that the average dot product of the gradient fields across networks with different width increases compared to the norm of the average of the gradient field for the smaller network.", "weaknesses": "My major concerns:\n(1) The claims are not backed my sufficient reasoning and relevant experiments. Note that the similarity measure is a function of the dot product of the average normalized gradients, so seeing an increasing trend on this quantity can be the result of two things: (*) the angle between the average of the normalized gradients doesn't change or gets smaller, or (**) the size of the average itself is increasing. Note that either of (*) or (**) is sufficient to happen to see an increasing value of cosine similarity that the authors have defined. However, observing an empirical increase in the value of similarity by going to larger nets, the authors conclude both of (*) and (**) at the same time, while it might be the case that (**) happens, ie.e. the size of the average is increasing fast, so that even though the direction of the averages are getting further apart, the overal dot product is still increasing. Therefore, in order to scientifically claim that both (*) , (**), one approach is that authors study the angle between the normalized averages (or dot product of NORMALIZED averages) separately. Specially because these claims are not backed by theoretical arguements.\n\n(2) The authors discuss the 'saw distribution' and modeling the behaviour of gradients with this dsitribution assuming the actuall gradient distribution is spherically symmetric (or close to it), and assumption that seems very strong and probably incorrect for actual neural nets. Is there a particular observation for this assumption? Authors discuss some high level calculation with this distribution on line 246, but I didn't find any calculation in appendix. \n\n(3) the writing is vague and it does not flow. In particular, the story for the theoretical calculation with gaussian distribution is not explained, and the assumptions are not justified at all. The section regarding saw distribution needs a lot of changes to become mroe clear.\n\nSome minor points:\n- the authors claim that the fact that given networks A, B where A has larger width, they claim that the observation that the average similarity of A to B is larger than that of B to itself is 'counter-intuitive' and 'paradoxical'. This is a wrong claim and needs to be omited from the paper; the reason that it is not paradoxical is you are working with 'some measure of similarity' rather than 'distance' or 'metric', for a simple illustrative example, if you define similarity of two numbers a, b to be their product ab, then if a > b > 0, ab > b^2 but it doesn't mean that because a, b are more similar, we should conclude that a is 'closer' to b than b to itself.\n\n- In Equation (5) you are estimating the normalized average of the gradient which is used in calculating your similarity measure, but I think the quantity of interest is the normalized average of the normalized gradients (because of the definition of cosine similarity that you use)?\n\nminor comments:\n- at H1 hypothesis in line 145:  the mean vector \\mu is not defined at that point of the paper\n-typo in line 158 -> should be k_1 and k_2", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer We4a\nWe thank the reviewer for acknowledging our contribution. Regarding the reviewer's concerns, we clarify that there is some communication in the empirical verifications. \n\nFirst, we would like to answer the clarification questions:\n\n**[Eq. 5 (W4)]**\n\nWe thank the reviewer for pointing out this typo! As the empirical estimation of $\\mu$, $\\tilde{\\mu}$ is computed as the **normalized average of the normalized gradients** throughout the experiments. This is the same as how $\\mu$ is defined in L152 (where $u$ is the normalized gradient). It is defined as\n$$\\tilde{\\mu}(k;x)=\\frac{1}{M}\\sum_{i=1}^M\\frac{\\nabla_x f_i(x)}{||\\nabla_x f_i(x)||})/||\\frac{1}{M}\\sum_{i=1}^M\\frac{\\nabla_x f_i(x)}{||\\nabla_x f_i(x)||}||\\approx \\mu(k;x),f_i\\in\\mathcal{F}(k).$$\nWe will revise it in the manuscript.\n\n**[Fig. 3 and 5 (Q1)]**\n\n- Fig. 3 presents the results of $\\rho(k_1,k_2)$, which is **the average of the cosine similarity between two families**. This is defined in L110-112 and serves as a starting point for the two hypotheses. The increase in the diagonal motivates and verifies H1. The increase in the rows and columns motivates and (partially) verifies H2 as mentioned by the reviewer.\n- Fig. 5 shows the results of **the cosine similarity between the averages of the normalized gradients of two families**. This is the angle between the population means of two $\\mathcal{G}$s. As also mentioned by the reviewer, this verifies H2.\n\n**[Gradient Similarities (W1)]**\n\nAfter clarifying the miscommunication, we now address the reviewer's major concerns:\n\nThe reviewer's observation of the two possible factors is insightful. Both of them are already considered **separately** in the manuscript. (ii) is discussed in the intra-family hypothesis.\nAs for (i), there were some miscommunications due to the typos in Eq. 5. The **angles between the average of the normalized gradients** are exactly what we used to verify H2. The results are presented in Fig. 5, with definitions in the caption as $\\mathbb{E}\\_{x\\in\\mathcal{X}}CosSim(\\tilde{\\mu}\\_j(x;\\mathcal{F}),\\tilde{\\mu}_{j'}(x;\\mathcal{F}))$. This is the dot product of normalized averages of the normalized gradients suggested by the reviewer. The results demonstrate an extremely high cosine similarity between the means and thus provide verifications to H2.\n\nFurthermore, we reassure the reviewer with the results in Fig. 10. The mean attack is performed by the means estimated using the family of $k=160$ (L291-297). Thus if the concerned scenario is true, the mean attack would not lead to the best results when attacking other families.\n\n**[Saw Distribution (W2,W3)]**\n\n- The symmetry assumptions (W2). To study the degree of concentration of the gradients, we study the cosine similarity $t = u^T\\mu$ between an individual gradient and the mean. This naturally leads to rotationally symmetric distributions since the distribution on the intersection between $S^{d-1}$ and the hyperplane does not affect the distribution of $t$. To resolve the concern, we carry out an empirical study of the distribution on the intersection (i.e. conditioned on $t$). Specifically, we train 1000 CNN models with $k=40$ and seeds 1~1000 on CIFAR-10 and compute $t$ regarding each test sample. The distribution of the first sample is visualized in Fig. R7(left). We partition the range of $t$ to 10 intervals by every 10 percent of the frequency, and inspect the **direction of the mean of the gradients in each interval**, each direction is estimated by 100 models. If these conditional mean directions are consistent with the population mean direction, then the gradients are symmetrically distributed on each $S^{d-2}$ hypersphere (R7(right)), thus verifying the rotational symmetry. We investigate the cosine similarities between the conditional and unconditional mean directions on the first 1000 samples. The 10*1000 similarity values have a mean and std at approximately 0.970 and 0.013 respectively. Thus the rotational symmetry is empirically verified.\n- Marginalization Derivations (W3). The marginal distribution $p_{original}$ in Sec. 3.3 is obtained by taking the marginalization of Saw distribution through integration on the hypersphere. It was not claimed as a theorem or lemma since it is not novel. We will include a detailed derivation in the appendix.\n\n**[Gaussian Assumptions (W3, Q2)]**\n\n- The Gaussian distribution described in the appendix is not related to the Saw distribution by any means. The only assumption involved in the Saw distribution is the symmetry over the $S^{d-2}$ (e.g. dashed circles in Fig. 4).\n- Appendix C serves as a sanity check or a null hypothesis on how to interpret the empirically observed cosine similarities. It is introduced in Sec. 3.1 L215, where the cosine similarities between the population means of different model families are studied. Fig. 5 shows that such cosine similarities are very high. However, the significance of these results may not be well interpreted given that in 2-D/3-D scenarios, this cosine similarity range may not be considered very high. Therefore, the analysis in Appendix C aims to uncover how difficult it is to achieve high cosine similarity as the dimension increases (e.g. Fig. 16).\n\n**[Generalizations & Population Means (Q0)]**\n\nIt can be observed that as the model capacity increases, the distributions of models become more concentrated (illustrated in Fig. 4(a) and verified in Fig. 3,5,6, etc.). This suggests that **as a single model approaches the population mean, the testing performance also increases**. To verify this claim, we carry out the experiments on deep ensembles since they approach the population mean of the model directly instead of by increasing capacity like single models. It can be found in Fig. 8 that although approaching the population mean differently, deep ensembles and single models have similar trends. This provides support for the connection between the population means and better generalizability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global Response to all Reviewers

We appreciate the reviewers very much for the invaluable feedback and insightful suggestions! First, we address the questions shared by reviewers as follows. All images are shown in the attached PDF file, indexed by the prefix `R` (e.g. Fig. R1).

##Different Sources of Stochasticity

This work aims to reveal the convergence trend of the distribution of model behaviors under the stochasticity of the training criterion. This does not limit the conclusion to the specific criterion used in the manuscript. Distinct training criteria can lead to different distributions of trained models. But these different distributions of trained models **all** satisfy the revealed trend.

**Training Details in the Manuscript**

For the sake of consistency, in this work, we follow the training criterion suggested in [1]. Specifically, stochastic gradient descent (SGD) is used as the solver, with a batch size of 128. The input data are normalized, but not augmented. We start with the initial learning rate $\gamma_0=0.1$ and update it with $\gamma_t=\gamma_0/\sqrt{1+t}$, where $t$ is the epoch.

**Additional Results**

To resolve the reviewer's concern, we present additional experiments to investigate (1) depths and widths; (2) learning rates; (3) batch sizes; (4) solvers. The results are shown in the attached pdf file. The main results should be the similarity *within* each criterion. But we also present the cross-criterion similarity results (e.g. between models trained using different learning rates/solvers/batch sizes/etc.).

We elaborate on them more as follows

- **Depths and widths**: The scale of depths is not as straightforward as width since modifying depths may change widths as well. Therefore, in the manuscript we study the influence of depth by setting -small and -large variations (L809). Here we present additional results that study the influence of depths continuously, with 1~5 layers, each of which is followed by a maxpool layer with stride 2. Finally, an adaptive pooling layer is appended at the end. To rule out the influence of widths (channels), all layers have the same width, determined by $k$. e.g., For the 4-layer scenario, the intermediate layers have widths [k, k, k, k] instead of [k,2k,4k,8k] in the manuscript. The results are shown in Fig R1. It can be found that (1) Given a fixed depth or width, the influence of the other factor is similar when scaled up. (2) Depths are slightly different from widths. Larger widths lead to higher similarities, while closer structures in depths have higher similarities. For widths (left), the similarity always increases left-to-right and top-to-bottom. But for depths (right), pairs near the diagonal have higher similarities.

- **Batch Sizes**: We investigate the influence of batch sizes, varying in {64, 128, 256, 512}. The results are shown in Fig. R2. It can be observed that although different batch sizes lead to different performance (e.g. testing accuracy), the convergence trend holds in all scenarios.

- **Learning Rates**: We test different learning rates on how they affect the results. We include {1e-1, 1e-2, default}, where "default" refers to the criterion used in the manuscript. As shown in Fig. R3, the revealed trend is preserved in all learning rates. It is also worth noticing that learning rates affect ResNets more than CNNs.

- **Solvers**: Apart from SGD, we include Adam, AdamW, and SGD w/ momentum. For Adam and AdamW we set the learning rate to 1e-3, while SGD w/ momentum uses a learning rate of 1e-1 with a momentum of 0.9. The results are shown in Fig. R4. Although different solvers lead to models of different performances, they all preserve the same convergence trend.

- Note that as studied in Fig. 6 of the manuscript, $\rho_{ind}$ can be a computationally efficient compromise to $\rho$. Therefore, we studied $\rho_{ind}$ in these additional experiments. Therefore, for the similarity of the same criterion, we only plot the upper triangular part of the similarity maps.

In conclusion, although training schemes can affect the resulting distributions of models, the influence of the model capacity stays invariant across different criteria. We will include the discussions regarding these variants of training criteria in the manuscript.



### **References**

**(The references of individual responses are also listed here)**

[1] Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. (2021). Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003.

[2] Cao, Y., Chen, Z., Belkin, M., & Gu, Q. (2022). Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems, 35, 25237-25250.

[3] Li, Z., Zhou, Z. H., & Gretton, A. (2021). Towards an understanding of benign overfitting in neural networks. arXiv preprint arXiv:2106.03212.

[4] Mallinar, N., Simon, J. B., Abedsoltan, A., Pandit, P., Belkin, M., & Nakkiran, P. (2022). Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting. arXiv e-prints, arXiv-2207.

[5] Geiger, M., Jacot, A., Spigler, S., Gabriel, F., Sagun, L., d’Ascoli, S., ... & Wyart, M. (2020). Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2020(2), 023401.

[6] Kobayashi, S., von Oswald, J., & Grewe, B. F. (2021, July). On the reversed bias-variance tradeoff in deep ensembles. ICML.

Meta-review: This paper examines the the distribution of trained deep neural networks from the perspective of input saliency maps. In particular, paper provides insightful observations regarding how models tend to resemble each other more as their capacity increases and its implications. The reviews for the paper are mostly positive. I recommend acceptance and encourage the authors to address the reviewer's comments in the final version of the paper.
Decision: Accept (poster)