Title: Great Minds Think Alike: The Universal Convergence Trend of Input Salience
Venue: NeurIPS 2024
Accepted: yes
Keywords: explainable artificial intelligence

saliency maps

model distributions
Abstract: Uncertainty is introduced in optimized DNNs through stochastic algorithms, forming specific distributions. Training models can be seen as random sampling from this distribution of optimized models. In this work, we study the distribution of optimized DNNs as a family of functions by leveraging a pointwise approach. We focus on the input saliency maps, as the input gradient field is decisive to the models' mathematical essence. Our investigation of saliency maps reveals a counter-intuitive trend: two stochastically optimized models tend to resemble each other more as either of their capacities increases. Therefore, we hypothesize several properties of these distributions, suggesting that (1) Within the same model architecture (e.g., CNNs, ResNets), different family variants (e.g., varying capacities) tend to align in terms of their population mean directions of the input salience. And (2) the distributions of optimized models follow a convergence trend to their shared population mean as the capacity increases. Furthermore, we also propose semi-parametric distributions based on the Saw distribution to model the convergence trend, satisfying all the counter-intuitive observations. Our experiments shed light on the significant implications of our hypotheses in various application domains, including black-box attacks, deep ensembles, etc. These findings not only enhance our understanding of DNN behaviors but also offer valuable insights for their practical application in diverse areas of deep learning.

Great Minds Think Alike: The Universal Convergence Trend of Input Salience

Abstract

Uncertainty is introduced in optimized DNNs through stochastic algorithms, forming specific distributions. Training models can be seen as random sampling from this distribution of optimized models. In this work, we study the distribution of optimized DNNs as a family of functions by leveraging a pointwise approach. We focus on the input saliency maps, as the input gradient field is decisive to the models' mathematical essence. Our investigation of saliency maps reveals a counter-intuitive trend: two stochastically optimized models tend to resemble each other more as either of their capacities increases. Therefore, we hypothesize several properties of these distributions, suggesting that (1) Within the same model architecture (e.g., CNNs, ResNets), different family variants (e.g., varying capacities) tend to align in terms of their population mean directions of the input salience. And (2) the distributions of optimized models follow a convergence trend to their shared population mean as the capacity increases. Furthermore, we also propose semi-parametric distributions based on the Saw distribution to model the convergence trend, satisfying all the counter-intuitive observations. Our experiments shed light on the significant implications of our hypotheses in various application domains, including black-box attacks, deep ensembles, etc. These findings not only enhance our understanding of DNN behaviors but also offer valuable insights for their practical application in diverse areas of deep learning.

1 Introduction

The advancement in computational power has significantly enhanced the capabilities of Deep Neural Networks (DNNs), leading to their unparalleled expressiveness and success in a multitude of applications across various fields (Krizhevsky et al., 2012; He et al., 2016; Rajkomar et al., 2018; Berner et al., 2019; Rombach et al., 2022; Padmaja et al., 2023; Thirunavukarasu et al., 2023). Despite these achievements, DNNs remain enigmatic, not only to end-users but also to researchers and practitioners (Ribeiro et al., 2016; Rudin, 2018; Preece et al., 2018). Due to the over-parameterization nature of modern DNNs, they are capable of reaching zero loss in the training distribution (Goodfellow et al., 2014b; Allen-Zhu et al., 2019; Du et al., 2019). Furthermore, the inherent stochastic nature of training algorithms means that even when using the same training data, DNNs tend to converge to various minima (Huang et al., 2017; Liu et al., 2020). Thus even though these models may exhibit comparable performance in terms of metrics like testing loss or accuracy, their underlying mechanisms can still differ significantly. Because of the stochastic nature of the training procedure, optimized DNNs collectively form a distribution over the functional space C 1 ( X ) , and training DNNs from scratch is thereby equivalent to randomly sampling from such a distribution without any guarantee. This inherent opacity, combined with the high dimensionality and nonlinearity, limits our understanding of the internal mechanisms of DNNs.

(a) Random Distribution

(b) Convergent Distribution

In response to these challenges, we study the aforementioned distributions. By adopting a pointwise approach, our focus is on the distribution of input salience (Simonyan et al., 2013) from the context of eXplainable Artificial Intelligence (XAI), which aims to demystify the inner workings of these complex models (Gunning and Aha, 2019; Arrieta et al., 2020; Van der Velden et al., 2022). Saliency maps, particularly in the form of input gradients , represent the data points within the gradient fields of DNNs. Thus the study of gradients can offer a deterministic view of the landscape of model predictions. This approach allows us to examine the intricate nuances of DNNs in a more structured and analytical manner.

For clarity, in the following context, we distinguish between the term model architecture (e.g. skip/direct connections) from the term model family . The latter refers to a specific collection of models F , that differ only in capacity as determined by width and depth. Two models are said to be in the same family if they differ only in parameter values. Given an input, varying model families result in distinct distributions. A synthetic visualization of such distributions is shown in Figure 1(a). Different models are depicted by the points. However, the relationship between different model families, represented by various colors, remains elusive. In this work, we introduce and verify several hypotheses to uncover a striking pattern. (1) Within the same model architecture (e.g., CNNs, ResNets), different family variants (e.g., varying capacities) tend to align in terms of their population mean direction. (2) As the model capacities increase, the variance within the distribution of the same family diminishes. This leads to a converging trend of the distributions. Both hypotheses are illustrated in Figure 1(b). Additionally, we introduce a semi-parametric approach to model these distributions, providing detailed quantification of the convergence.

The similarities observed in input salience have direct implications for understanding the important vulnerability of DNNs regarding gradient attacks (Szegedy et al., 2013; Goodfellow et al., 2014a). In particular, in black-box attack settings, the gradients of the target model are not directly accessible. A higher degree of salience similarity naturally enhances transferability (Chen et al., 2023). Our findings elucidate why models with larger capacity consistently exhibit superiority in terms of adversarial robustness compared to smaller models (Madry et al., 2017; Gustafsson et al., 2020; Li et al., 2020; Bubeck and Sellke, 2021). Moreover, given that the mean direction is aligned across different models, it is possible to approximate this mean direction by randomly sampling from a set of independently optimized models. We demonstrate that these estimated mean directions can attain a near-perfect cosine similarity of almost 1.0, even between completely independent models or ensembles, in a high-dimensional space. Moreover, note that deep ensembles essentially calculate this population mean direction (Lee et al., 2015; Lakshminarayanan et al., 2017; Fort et al., 2019; Kondratyuk et al., 2020), where the mean of a group of independently trained models can improve the performance. As a consequence, the insights of our hypotheses also shed light on this phenomenon which, although empirically successful, has been somewhat enigmatic in terms of their source of capability (Lobacheva et al., 2020; Deng and Shi, 2021; Abe et al., 2022; Theisen et al., 2023). Furthermore, since deep ensembles approximate the aligned mean directions much faster than scaling up single models, this

also demystifies the significant black-box attack transferability of deep ensembles (Yang et al., 2021; Chen et al., 2023). Our research thus not only advances the understanding of model behavior in practical applications but also contributes to the broader field of AI trustworthiness and efficiency. Our main contributions can be summarized as follows:

We reveal an appealing phenomenon where the mean distribution directions of input salience across different model families have extremely high resemblance.

We empirically demonstrate the distribution converges towards the mean direction as model capacity increases.

Incorporating both empirical observations and theoretical analysis, we hypothesize distributional properties of optimized models, quantifying the aforementioned phenomena.

The hypotheses effectively explain many hitherto unclear phenomena such as black-box attack transferability, the efficacy of deep ensemble methods, etc.

2 The Convergence of Input Saliency

2.1 Salience Similarities

Notation. Let X × Y = D denote the dataset, where X ⊂ R d is the input set and Y = [ c ] is the set of labels and c ∈ N + is the number of classes. Following the benign overfitting phenomenon (Bartlett et al., 2020; Papyan et al., 2020; Cao et al., 2022), we let F = { f |L ( f ; X train , Y train ) < 10 -3 } denote a family of optimized models, distinguished by different architectures, such as vanilla sequential CNNs, skipping blocks, etc. L denotes the expected cross-entropy loss for the training distribution. For simplicity, we focus on f : R d → R , which predicts the logit specifically for the targeted class . This is to stay consistent with XAI methods. We demonstrate in Appendix A that the difference between logit and probability (Wang and Wang, 2022) does not affect the observed phenomena. Unless otherwise indicated, experiments are carried out over the test set X = X test , Y = Y test . Within the same architecture, model capacity is determined by both the width and the depth. Since it is more difficult to model depth as a single variable, we model varying depth as different families F but model varying width k as a parameter of the family, F ( k ) .

(a) CNN, CIFAR-10

(b) ResNet, CIFAR-10

(c) CNN, CIFAR-100

(d) ResNet, CIFAR-100

0.6

0.35

0.35

0.25

0.5

0.30

0.30

Model Capacity k

- 0.4

Model Capacity k

0.25

Model Capacity k

0.25

Model Capacity k

0.20

0.3

0.20

0.20

0.15

0.2

0.15

0.15

0.10

0.10

0.10

0.1

0.05

0.05

448

0.05

89449949298

38258946

88449949498888

138259291

Model Capacity k

Model Capacity k

Model Capacity k

Model Capacity k

8

(a) CNN, CIFAR-10

(b) ResNet, CIFAR-10

(c) CNN, CIFAR-100

(d) ResNet, CIFAR-100

10

0.4

10

10

10

Model Capacity

20

0.3

Capacity

20

0.25

Capacity

20

0.15

Capacity

20

0.15

0.20

40

0.2

Model

40

0.15

Model

40

0.10

Model

40

0.10

80

80

80

80

160

0.1

160

0.10

160

0.05

160

0.05

0.05

40

40

80

Model Capacity k

Model Capacity k

Model Capacity k

Model Capacity k

The Increasing Similarity. Let CosSim : R d × R d → [ -1 , 1] denote the cosine similarity metric, then the individual similarity between the input salience of two given models f (1) ∈ F ( k 1 ) , f (2) ∈ F ( k 2 ) given input x is ρ ind ( f (1) , f (2) ; x ) = CosSim ( ∇ x f (1) ( x ) , ∇ x f (2) ( x )) . Taking the entire testing set into consideration, denote ρ ind ( f (1) , f (2) ) = E x ∈X [ ρ ind ( f (1) , f (2) ; x )] . In Figure 2, the expectations over the testing set E x ∈X CosSim ( ∇ x f (1) ( x ) , ∇ x f (2) ( x )) with varying k 1 , k 2 ∈ K are illustrated. Here, we define K = { j 2 i : 4 ≤ j ≤ 7 , 1 ≤ i ≤ 6 } = { 8 , 10 , 12 , 14 , 16 , 20 , · · · , 384 , 448 } to balance between finer linear scaling and coarser exponential scaling. It can be observed that the similarity between two stochastically optimized models f (1) , f (2) has an increasing trend with respect to both k 1 , k 2 . Two different architectures CNN and ResNet are included. To rule out the influence of any single model, we define the similarity between families F ( k 1 ) , F ( k 2 ) for a given input x by taking the expectation of the two models:

The global similarity between models of widths k 1 , k 2 is then denoted by ρ ( k 1 , k 2 ) = E x ∈X ρ ( k 1 , k 2 ; x ) . Note that estimating this value requires training numerous f ∈ F ( k ) for each k ∈ K . Therefore, we carry out the experiments over K ′ = { j 2 i : j = 5 i = 1 , 2 , 3 , 4 , 5 } = { 10 , 20 , 40 , 80 , 160 } ⊂ K . For each k 1 , k 2 ∈ K ′ , 100 models are sampled respectively to empirically estimate the expectation over F ( k 1 ) , F ( k 2 ) . As observed in Figure 3, ρ ( k 1 , k 2 ) has an increasing trend w.r.t. both k 1 , k 2 . Compared with Figure 3, the average similarity over the model families is similar to the individual cosine similarity for the same k values. As a result, studying the similarity of two randomly sampled models instead of the expectation over F s can significantly alleviate the computational burden. This is further discussed in detail in Section 3.2. Besides, It trivially follows that ∀ k 1 > k 2 , ρ ( k 1 , k 1 ) > ρ ( k 1 , k 2 ) > ρ ( k 2 , k 2 ) , which suggests that larger models tend to resemble smaller models even more than smaller models themselves. - Even if the two smaller models only differ in the random seeds during training. Please refer to Appendix A for the results of more datasets, where such increasing trends still exist.

2.2 The Spherical Distribution of the Salience

Since the cosine similarity can be written as the inner product between ∇ x f (1) ( x ) ∥∇ x f (1) ( x ) ∥ and ∇ x f (2) ( x ) ∥∇ x f (2) ( x ) ∥ , which are high-dimensional unit vectors, we explore the properties and potential distributions of F through the perspective of spherical statistics. Given an input x ∈ X , we denote by G k ( x ) the set of all possible gradient directions of input x regarding the models f in F ( k ) . Formally, let

Then the similarity is re-written as the inner product ρ ( k 1 , k 2 ; x ) = E u 1 ∈G k 1 ( x ) , u 2 ∈G k 2 ( x ) [ u T 1 u 2 ] .

The Intra-Family vs. Cross-Family Paradox. An interesting paradox is raised as ρ ( · , · ) increases with both inputs. Naturally, one would reasonably deduce that two models f (1) , f (2) ∈ F ( k 1 ) should resemble each other since they are from the same family (i.e. having the exact same structure and only differ in training seeds). However, since ρ ( k 2 , k 1 ) > ρ ( k 1 , k 1 ) , the cross-model family similarity becomes greater than the intra-model family similarity. To uncover the mystery of the observations, we present the intuitive understanding and the rigorously analyzed hypotheses as follows.

The Intra-Family Hypothesis. Note that for intra-family scenario, u , v ∈ G k ( x ) are i.i.d., the similarity can be written as ρ ( k, k ; x ) = ( E G k ( x ) [ u ]) T ( E G k ( x ) [ v ]) = ∥ E G k ( x ) [ u ] ∥ 2 2 , which denotes the square of the population mean resultant length (Mardia et al., 2000) of G k ( x ) . The population mean resultant length √ ρ ( k, k ; x ) quantifies the degree of dispersion of G k ( x ) , where a larger length suggests a more concentrated distribution. In directional statistics, the degree of dispersion is usually quantified by the spherical variance 2(1 -√ ρ ( k, k ; x )) or the total variation 1 -ρ ( k, k ; x ) . Since ρ ( k, k ; x ) also increases w.r.t. k , this suggests an increasing concentration of input salience of models as the width k of the model increases. In conclusion, the larger the models are, the smaller the spherical variance of the salience is. Formally, we propose the following hypothesis.

Hypothesis I (H1) : Let k denote the width (capacity) of the model and G k ( x ) = { u = ∇ x f ( x ) ∥∇ x f ( x ) ∥ ∣ ∣ ∀ f ∈ F ( k ) } denote the set of input gradient directions regarding x . Then

E G k ( x ) [ u ] = √ ρ ( k, k ; x ) µ ( k ; x ) and ρ ( k, k ; x ) increases with k . Here µ ( k, x ) = E G k ( x ) [ u ] ∥ E G k ( x ) [ u ] ∥ denotes the unit mean direction of G k ( x ) .

Note that H1 also holds for the change of model depths , which is positively related to the dispersion degree of the distribution. However, the change in model depth inevitably affects model width. Thus, we only provide empirical verification in Section 3 but do not include it as a part of H2.

The Cross-Family Hypothesis. Unlike the intra-family similarity, the increasing cross-family similarity is where the phenomenon becomes counter-intuitive. Then due to the increasing intra-family similarities, when k 1 , k 2 increase, u 1 ∈ G k 1 ( x ) becomes closer to µ ( k 1 ; x ) , u 2 ∈ G k 2 ( x ) becomes closer to µ ( k 2 ; x ) . However, the cross-familty similarities suggest that as u 1 , u 2 approach their respective mean directions, their similarity increases as well. This indicates that the mean directions µ ( k 1 ; x ) , µ ( k 2 ; x ) are similar, too. Formally, this intuition is considered as follows. For k 1 > k 2 , when µ ( k 1 ; x ) and µ ( k 2 ; x ) are sufficiently similar, µ ( k 1 ; x ) T µ ( k 2 ; x ) ≈ ∥ µ ( k 1 ; x ) ∥∥ µ ( k 1 ; x ) ∥ . Thus we have

which is monotonic w.r.t. both k 1 , k 2 . Formally, this is summarized as

̸

Hypothesis II (H2) : Let G k 1 ( x ) , G k 2 ( x ) denote the input gradient directions of two model families where k 1 = k 2 . Then µ ( k 1 ; x ) ≈ µ ( k 2 ; x ) regardless of k 1 , k 2 .

The two hypotheses are both empirically verified. For a smoother flow of the presentation, we defer the detailed experiments to Section 3. The basic ideas of H1 and H2 are illustrated in Figure 4 (a).

2.3 The Directional Distribution of Gradients

Given the analysis and hypothesis above, one can have an overview of the models' internal mechanisms. As the model capacity increases, models are distributed in a more concentrated manner, while the mean direction stays almost invariant. To better understand the models' behavior with the stochasticity, we delve into the distribution of G k ( x ) and present a semi-parametric analysis with experimental verification. A general form of centralized symmetric distribution over hypersphere

T

is known as the Saw distribution (Fisher et al., 1993) p ( u ; µ ) = ψ ( u µ ) Z , where µ is the mean direction with ∥ µ ∥ = 1 , ψ ∈ C ([ -1 , 1]) , and Z = ∫ S d -1 ψ ( u T µ ) d u is the normalization term for distributions. Due to the symmetry assumption, the shape of the distribution is solely determined by ψ . For example, a monotonically increasing ψ suggests that u is distributed more densely near the mean direction and sparsely distant from the mean direction. Considering the concentration trend of gradients, we hypothesize that ψ k ( · ) of G k ( x ) not only monotonically increases with the input, but also increases faster with greater k values.

Marginalization. For ∀ u ∈ G k ( x ) , it can be decomposed to u = t · µ ( x ) + √ 1 -t 2 · µ ( x ) ⊥ , where µ ( x ) ⊥ is a unit tangent to S d -1 at µ . Then t = u T µ ( x ) . This is shown in Figure 4 (b). Note that µ ( x ) ⊥ is independent from t , then the distribution of t is the marginal distribution over the intersection between S d -1 and the hyperplane spanned by µ ( x ) ⊥ , which is a ( d -2) -dimensional hypersphere. According to the symmetry assumption of Saw distribution, conditioned on a fixed similarity t , the distribution of u | t over the dashed S d -2 does not affect ψ . Therefore, we focus on the marginalized distribution of t . Note that the radius of the intersection S d -2 is √ 1 -t 2 , we thus have d u = 2 π ( d -1) / 2 (1 -t 2 ) ( d -3) / 2 Γ(( d -1) / 2) d t , where the density of t is observed by the integral over the corresponding ( d -2) -hypersphere. As a result, the marginal distribution of t has the PDF

where Z ′ is a constant normalization term. Note that (1 -t 2 ) ( d -3) / 2 is a symmetric bell curve centered at t = 0 . Equation (4) can thus be viewed as using ψ k ( t ; x ) to reweight its PDF p origin ( t ) = (1 -t 2 ) ( d -3) / 2 Γ( d/ 2) √ π Γ(( d -1) / 2) . Note that here p k ( t ; x ) is the distribution of t = u T µ ( k ; x ) , which can be empirically estimated, the shape of the function ψ k becomes empirically accessible with varying k values. The empirical studies and verification are provided in Section 3.3.

μ(x)

u

1- t²μ(x)

u2

μ(x)

u

(x)n

dt

np

0

u

Sd-1

(a)Theillustrationof the cross-

familydistributions(k>k2).

(b) The decomposition (left)and marginalization (right)of u E Gk(x).

3 Empirical Verification of Hypotheses

In this section, we provide comprehensive experimental results to verify the aforementioned hypotheses. First, we introduce the detailed setups of our experiments. They are carried out on Intel(R) Core(TM) i9-9960X CPU @ 3.10GHz with Quadro RTX 6000 GPUs.

Datasets & Models. Due to the massive size of experiments, here we mainly follow the setups of the benign overfitting (Nakkiran et al., 2021), which also present a comprehensive study of optimized DNNs through CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009). Besides, we also include TinyImagenet-200 (Le and Yang, 2015) as a compromise between the computational efficiency and the dataset complexity. As for models, we include CNNs and ResNets as in (Nakkiran et al., 2021). These two models represent the two typical architectures - the direction connection and the skip connection. We also notice a striking capacity gap between them in the original implementation. Therefore, we term them CNNSmall (CS) and ResNetLarge (RL), respectively, and include CNNLarge (CL), and ResNetSmall (RS) to fill the gap. The comparison between the small and large families also shows the influence of depths. As for the training process, following Nakkiran et al. (2021), we use stochastic gradient descent (SGD) as the solver, with a batch size of 128. The input data are normalized, but not augmented. We start with the initial learning rate γ 0 = 0 . 1 and update it with γ t = γ 0 / √ 1 + t , where t is the epoch. Please refer to Appendix B for more experimental details.

3.1 The Mean Direction Similarity (H2)

H2 can be verified independently from H1 and can provide simplifications and insights to verifying H1. We hence focus on H2 first. As stated in H2, the mean directions of different model families are consistently aligned, i.e., µ ( k 1 ; x ) ≈ µ ( k 2 ; x ) . For each family F ( k ) where k ∈ K ′ , M = 100 models with different random seeds are trained. The population mean is then estimated through

Then the similarity of mean directions are naturally ( ˜ µ ( k 1 ; x )) T ˜ µ ( k 2 ; x ) . Note that when k 1 = k 2 , the 100 models are partitioned by the seeds to avoid overlapping. The results of the expectation over X are visualized in Figure 5. It can be found that the mean directions have extremely high resemblance within each architecture, as proved by the high cosine similarities. It should be noted that with high dimensionality (e.g. d = 3072 for CIFAR), a cosine similarity close to 1 is an extremely significant result. We demonstrate this with the uniform distribution on the hypersphere in Appendix D. The observations verify the hypothesis all G k ( x ) almost share the same mean direction within the model architecture. This not only hold across different widths determined by k , but also holds across different depths (i.e. CS vs. CL, RS vs. RL). Therefore, the mean direction is mostly related to the certain model architecture instead of any single model f ∈ F , making it an intrinsic

(a) CIFAR-10

(b) CIFAR-100

(c) TinyImagenet

0.8

0.8

Model Capacity k

0.8

Model Capacity k

Model Capacity k

0.7

0.6

0.6

0.6

0.5

0.4

0.4

0.4

0.3

0.2

0.2

一

0.2

RL:160

0.1

12486

ssss..

000000000000

RRRRSRRRR

R

R

ouuusuuuu

12486

ssss..

RRRRSRRRR

R

12486

ssss..

U

RRRRSRRRR

R

12486

R

Model Capacity k

Model Capacity k

Model Capacity k

CS

CL

RS

RL

0.6

0.5

0.4

Metric

0.3

0.4

0.3

0.4

Similarity

0.3

0.2

0.2

0.2

0.2

0.1

0.1

Pind

0.1

p

0

200

400

0

200

400

0

200

400

0

200

400

property of the model architecture. With this property, how different model architectures differ in mechanisms can be studied by looking deeper into the population mean direction of saliency maps. For instance, it can be observed that the ResNet architecture admits a closer relation between models of different depths, compared with the CNN architecture.

3.2 The Decreasing Spherical Variance with k (H1)

Expectation over F vs. Conditioned on f ∈ F . As previously discussed, the spherical variance of distributions over the hypersphere can be measured by the population mean resultant length √ ρ ( k, k ; x ) , which, unfortunately, requires an estimation of the mean directions. This can be expensive to study for a comprehensive set K of k values. The experiments on a subset K ′ = { 10 , 20 , 40 , 80 , 160 } are already carried out in Figure 3, shown as the diagonal elements. As k increases, the resultant length increases monotonically, indicating a decreasing spherical variance and a more concentrated distribution around the mean directions.

The computational burden of taking the expectation over F can be alleviated by considering randomly picked f . In order to compare ρ and ρ ind , we consider the model-dependent set S ( f ) = { ρ ind ( f, f ; x ) : x ∈ X} for each f ∈ F . Here we compute the expected Wasserstein distance E f (1) ,f (2) ∈F ( k ) WD ( S ( f (1) ) , S ( f (2) )) . This is estimated by the ( M 2 ) distinct pairs of models. The distances of all 60 (dataset, model family) pairs lie below 0.035. Such observation suggests that after taking the expectation over X , the differences across individual models can be mitigated. Please refer to Table 1 for comprehensive results on all model families and datasets. As a consequence, it suffices to use ρ ind ( f, f ) for some f ∈ F ( k ) to approximate ρ ( k, k ) . This is in fact the diagonal elements of Figure 2. A comparison between the diagonal elements ρ k,k and ρ ind ( f (1) , f (2) ) , f (1) , f (2) ∈ F ( k ) over CIFAR-10 is presented in Figure 6. Please refer to the appendix for other datasets. ρ ind is evaluated over k ∈ K , while ρ is evaluated over k ∈ K ′ ⊂ K . It can be found that after taking the average over X , even though ρ is a little smoother than ρ ind , they are very consistent. This verifies that the resultant length increases with k ∈ K in a much more comprehensive set of models. Thus H1 is empirically verified.

20

Porigin

p10

600

k=10

P20

400

k=40

Frequency

15

P80

200

k=80

k=160

10

P160

601

0

（）60

8

-200

-400

5

-600

0

0°0

0.2

Estimated t

0.4

0.6

0.8

800

0.1

0°0

0.1

Estimated t

0.20.30.4

0.5

0.6

3.3 The Shape of the Saw Distribution

Given G k ( x ) , the marginalized distribution p k ( t ; x ) can be approximated by T k ( x ) = { ˜ t = u T ˜ µ ( k ; x ) | u ∈ G k ( x ) } . In order to obtain the global results over test dataset X , consider the unions of different input samples T k = ∪ x ∈X T k ( x ) . This is an approximation to the mixture distributions p k ( t ) = 1 |X| ∑ x ∈X p k ( t ; x ) . We plot the histogram of T k with k ∈ K ′ for CNNSmall and CIFAR-10 in Figure 8. The left figure shows p origin and the estimated p k , visualized by different colors. Qualitatively, p k ( t ) has higher means with greater k values. The reason why the density of p k is not centered at t = 1 (i.e. u = µ ) is because as t increases, the size of the ( d -2) -hypershpere decreases with p origin ( t ) = (1 -t 2 ) ( d -3) / 2 Γ( d/ 2) √ π Γ(( d -1) / 2) , which is shown as black histograms. This is much faster than the increase of ψ k . The shape of ψ k is determined by p k /p origin with normalization. From the right figure, by comparing p k with p PDF , it can be empirically verified that ψ k ( t ) is increasing vastly. It is also observed that larger k s lead to a faster increase of ψ k and higher E [ t ] . This also provides a quantitative understanding of H1 and H2. The results of other model architectures and datasets can be found in the appendix.

Verification of the Symmetry. Saw distributions study the marginalized value t = u T µ to directly focus on the degree of concentration of the gradients. This naturally leads to rotationally symmetric distributions since the distribution on the intersection between S d -1 and the hyperplane does not affect the distribution of t . We thus carry out an empirical study of the distribution on the intersection (i.e. conditioned on t ). Specifically, we train 1000 CNN models with K = 40 and seeds 1-1000 on CIFAR-10 and compute t regarding each test sample. The distribution of the first sample is visualized in Figure 7. We partition the range of t into 10 intervals by every 10 percent of the frequency, and inspect the direction of the mean of the gradients in each interval, each direction is estimated by 100 models. If these conditional mean directions are consistent with the population mean direction, then the gradients are symmetrically distributed on each S d -2 hypersphere (R7(right)), thus

Partitions

100

80

Count

60

40

20

0

0.10

0.15

0.20

0.25

0.30

0.35

0.40

t

verifying the rotational symmetry. We investigate the cosine similarities between the conditional and unconditional mean directions on the first 1000 samples. The 10 × 1000 similarity values have a mean and std at approximately 0.970 and 0.013 respectively. Thus the rotational symmetry is empirically verified.

SingleModels

DeepEnsembles

3.0

3.0

2.5

2.5

口

Testing Loss

2.0

Loss

2.0

1.5

1.5

1.0

1.0

0.5

0.5

中量+国

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

1.0

Marginal Expectation E[t]

Marginal Expectation E[t]

CNNSmall

CNNLarge

ResNetLarge

ResNetSmall

■

+

k=10

k=40

k=80

(a) CIFAR-10, CS

(b) CIFAR-10, CL

(c) CIFAR-10, RS

(d) CIFAR-10, RL

助死忆8

1.0

812164284

1.0

忆叽孔8

1.0

81216428

1.0

Source Modelk

0.8

Modelk

0.8

Modelk

0.8

Source Modelk

0.8

0.6

0.6

32

48

0.6

0.6

96

0.4

Source

64

96

0.4

Source

64

96

0.4

9

96

0.4

128

128

128

128

192

0.2

192

0.2

192

0.2

192

0.2

256

256

256

256

384

0.0

384

0.0

384

0.0

384

0.0

Target Model k

Target Model k

Target Model k

Target Model k

(a) CIFAR-10, CS

(b) CIFAR-10, CL

(c) CIFAR-10, RS

(d) CIFAR-10, RL

Prediction Decay Ratio α

2 ×100

Attack by u E Gmax(k) (x)

Attack by u E Gx(x)

Prediction Decay Ratio α

2 ×100

Attack by u E Gmax(k) (x)

Attack by u E Gx(x)

Prediction Decay Ratio α

Attack by u E Gmax(k) (x)

Attack by u E Gx(x)

Prediction Decay Ratio α

Attack by u E Gmax(k) (x)

Attack by u E Gk(x)

Attack by μi(max{K'}: x)

Attack by μi(max(K'}: x)

100

Attack by μi（max{K'}:x)

Attack by μi(max{K'}; x)

100

100

100

6 × 10-1

6 × 10-1

6 ×10-1

6 ×10-1

4 × 10-1

3×10-1

101

102

101

102

101

102

101

102

4 Applications of Hypotheses

4.1 Deep Ensemble: Why Does It Work?

After verifying the hypotheses, we explore possible applications and implications of the discovered phenomena. The deep ensemble method makes use of the stochasticity to of models by incorporating the predictions from m members. While deep ensembles have been verified to be effective in improving performance, the source of such capability remains mysterious. Note that ensemble members are i.i.d. optimized models with SGD, which correspond to the population mean of our hypothesis. We thus provide another perspective in understanding the capability of deep ensembles.

For single models, as the model capacity increases, benign overfitting suggests that the testing loss decreases, too. We deduce that this is because the distribution of larger models becomes more concentrated, and combined with H2, the closeness to the aligned population mean is highly related to the models' testing performance. As shown in Figure 9(a), it can be observed that the expected loss E X [ L ] and the marginal expectation E X [ t ] are highly correlated. Similarly, deep ensemble approximates the population mean much more effectively by increasing the number m of members than scaling up a single model by k . We thus scale up the deep ensemble by changing the number of ensemble members. The results are shown in Figure 9(b). It can be found that (1) the correlation between E X [ L ] and E X [ t ] is not only significant, and (2) the correlation pattern is shared between two completely different scaling mechanisms, single model scaling and model ensembles.

4.2 Black-Box Attack via Saliency Similarity

The understanding of adversarial attacks can benefit from the behaviors of the input salience of optimized models given their close relation to input gradients. We verify the aforementioned similarities through the black-box attacks, where the adversarial samples are generated from the gradients of source models while the gradients of the target models are not available. Let f (1) ∈ F ( k 1 ) denote the source model and f (2) ∈ F ( k 2 ) denote the target model. We define the attack rate from

f (1) to f (2) similar to ρ as

which is the performance decay of f (2) when attacked by model f (1) . Small α ( k 1 , k 2 ) values suggest successful attack from f (1) to f (2) . The results are shown in Figure 10, where the attack step is set to ϵ = 0 . 05 . In each heatmap figure, the y -axis represents the width of the source models, while the x -axis represents the width of the target models. It can be observed that larger models succeed in attacking smaller models, but the opposite is not true. To attack a large model, the gradient needs to be generated from a model of a comparable level in terms of capacity.

Mean Direction Attack. According to the verified hypothesis H2, for any two individual models f (1) ∈ F ( k 1 ) , f (2) ∈ F ( k 2 ) the mean directions µ ( k ; x ) is closer to both of them than themselves, regardless of k, k 1 , k 2 . It is then suggested that using the mean gradient can perform more successful black-box attacks. We employ the mean direction ˜ µ (160 , x ) to attack models of different capacities, and compare the results with the attack from the largest single model (red) and the attack from the models of the identical structure (green). The results are shown in Figure 11, where it can be observed that the mean direction of salience transfers much more successfully than single models.

5 Conclusions

In this paper, we introduce hypotheses to explain the observations on the input salience convergence w.r.t. the model capacities. Under the same model architecture, stochastic algorithms such as SGD, result in certain distributions of optimized models. We hypothesize and use pointwise methods to verify that such distribution follows a Saw distribution with aligned population means , which is invariant from the model families. Besides, the variance of the distribution decreases as the model capacity increases, suggesting a convergence trend of the models' internal mechanism - the larger the models are, the less variant they tend to be affected by the randomness from the stochastic algorithm during the training phase. Furthermore, since the distributions converge towards the aligned population mean direction, the limiting points can be estimated by the population mean of models. Based on this, we present comprehensive experiments on the properties of the limiting model and demonstrate its capability in various domains, such as the black-box attack transferability, and the explanation of the effectiveness of deep ensembles. However, it is admitted that, due to the high computational burden, although improved from CIFAR-10/100 to TinyImagenet compared to (Nakkiran et al., 2021), our experiments are limited to rather small datasets.

Our introduced hypotheses also lead to various interesting topics. Note that the aligned mean direction stays invariant to the model families, which indicates such population mean is more related to the essence of the dataset itself rather than any single model. Leveraging this property can bring a deeper and more comprehensive understanding of the relation between data distributions and models.

6 Related Work

In terms of the convergence trend of DNNs, existing works focus on the convergence of single models throughout the training process. The parameters of DNNs have been demonstrated to converge to global minima throughout the training progress (Goodfellow et al., 2014b; Li et al., 2018; Allen-Zhu et al., 2019; Liu et al., 2020; Damian et al., 2021; Refinetti et al., 2023; Suh and Cheng, 2024). Recent years, the studies of benign overfitting also suggest that increasing model capacities can improve the performance instead of exacerbating the overfitting issue (Bartlett et al., 2020; Nakkiran et al., 2021; Cao et al., 2022). While the studies of input gradients span into an abundant but extremely complicated spectrum. Among them, the area that is the most related to our work is the XAI domain, where the input gradient and its variants are crucial in revealing the models' internal mechanisms (Simonyan et al., 2013; Springenberg et al., 2014; Selvaraju et al., 2017; Sundararajan et al., 2017; Adebayo et al., 2018; Shah et al., 2021). On the other hand, the studies of the distribution of optimized models have received little attention. Such topics are slightly dipped in the efforts to demystify the source of capability of deep ensembles (Lee et al., 2015; Fort et al., 2019; Allen-Zhu and Li, 2020; Kobayashi et al., 2021; Abe et al., 2022; Ganaie et al., 2022; Theisen et al., 2023) and their implications (Lakshminarayanan et al., 2017; Geiger et al., 2020; Yang et al., 2021; Chen et al., 2023). Thus, to our knowledge, the studies on the distribution of optimized models remain a novel topic.

References

Abe, T., Buchanan, E. K., Pleiss, G., Zemel, R., and Cunningham, J. P. (2022). Deep ensembles work, but are they necessary? Advances in Neural Information Processing Systems , 35:33646-33660.

Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. (2018). Sanity checks for saliency maps. Advances in neural information processing systems , 31.

Allen-Zhu, Z. and Li, Y. (2020). Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 .

Allen-Zhu, Z., Li, Y., and Song, Z. (2019). A convergence theory for deep learning via overparameterization. In International conference on machine learning , pages 242-252. PMLR.

Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2020). Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion , 58:82-115.

Bartlett, P. L., Long, P. M., Lugosi, G., and Tsigler, A. (2020). Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070.

Berner, C., Brockman, G., Chan, B., Cheung, V., D˛ ebiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. (2019). Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 .

Bubeck, S. and Sellke, M. (2021). A universal law of robustness via isoperimetry. Advances in Neural Information Processing Systems , 34:28811-28822.

Cao, Y., Chen, Z., Belkin, M., and Gu, Q. (2022). Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems , 35:25237-25250.

Chen, H., Zhang, Y., Dong, Y., and Zhu, J. (2023). Rethinking model ensemble in transfer-based adversarial attacks. arXiv preprint arXiv:2303.09105 .

Damian, A., Ma, T., and Lee, J. D. (2021). Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems , 34:27449-27461.

Deng, D. and Shi, E. B. (2021). Ensembling with a fixed parameter budget: When does it help and why? In Asian Conference on Machine Learning , pages 1176-1191. PMLR.

Du, S., Lee, J., Li, H., Wang, L., and Zhai, X. (2019). Gradient descent finds global minima of deep neural networks. In International conference on machine learning , pages 1675-1685. PMLR.

Fisher, N. I., Lewis, T., and Embleton, B. J. (1993). Statistical analysis of spherical data . Cambridge university press.

Fort, S., Hu, H., and Lakshminarayanan, B. (2019). Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757 .

Ganaie, M. A., Hu, M., Malik, A., Tanveer, M., and Suganthan, P. (2022). Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence , 115:105151.

Geiger, M., Jacot, A., Spigler, S., Gabriel, F., Sagun, L., d'Ascoli, S., Biroli, G., Hongler, C., and Wyart, M. (2020). Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment , 2020(2):023401.

Goodfellow, I. J., Shlens, J., and Szegedy, C. (2014a). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 .

Goodfellow, I. J., Vinyals, O., and Saxe, A. M. (2014b). Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv:1412.6544 .

Gunning, D. and Aha, D. (2019). Darpa's explainable artificial intelligence (xai) program. AI magazine , 40(2):44-58.

Gustafsson, F. K., Danelljan, M., and Schon, T. B. (2020). Evaluating scalable bayesian deep learning methods for robust computer vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 318-319.

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778.

Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J. E., and Weinberger, K. Q. (2017). Snapshot ensembles: Train 1, get m for free. arXiv preprint arXiv:1704.00109 .

Kobayashi, S., von Oswald, J., and Grewe, B. F. (2021). On the reversed bias-variance tradeoff in deep ensembles. ICML.

Kondratyuk, D., Tan, M., Brown, M., and Gong, B. (2020). When ensembling smaller models is more efficient than single large models. arXiv preprint arXiv:2005.00570 .

Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25.

Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30.

Le, Y. and Yang, X. (2015). Tiny imagenet visual recognition challenge. CS 231N , 7(7):3.

Lee, S., Purushwalkam, S., Cogswell, M., Crandall, D., and Batra, D. (2015). Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314 .

Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. (2018). Visualizing the loss landscape of neural nets. Advances in neural information processing systems , 31.

Li, Y., Bai, S., Zhou, Y ., Xie, C., Zhang, Z., and Yuille, A. (2020). Learning transferable adversarial examples via ghost networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 11458-11465.

Liu, S., Papailiopoulos, D., and Achlioptas, D. (2020). Bad global minima exist and sgd can reach them. Advances in Neural Information Processing Systems , 33:8543-8552.

Lobacheva, E., Chirkova, N., Kodryan, M., and Vetrov, D. P. (2020). On power laws in deep ensembles. Advances In Neural Information Processing Systems , 33:2375-2385.

Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2017). Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 .

Mardia, K. V., Jupp, P. E., and Mardia, K. (2000). Directional statistics , volume 2. Wiley Online Library.

Muller, M. E. (1959). A note on a method for generating points uniformly on n-dimensional spheres. Communications of the ACM , 2(4):19-20.

Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. (2021). Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124003.

Padmaja, B., Moorthy, C. V., Venkateswarulu, N., and Bala, M. M. (2023). Exploration of issues, challenges and latest developments in autonomous cars. Journal of Big Data , 10(1):61.

Papyan, V., Han, X., and Donoho, D. L. (2020). Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):2465224663.

Preece, A., Harborne, D., Braines, D., Tomsett, R., and Chakraborty, S. (2018). Stakeholders in explainable ai. arXiv preprint arXiv:1810.00184 .

Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018). Scalable and accurate deep learning with electronic health records. NPJ digital medicine , 1(1):18.

Refinetti, M., Ingrosso, A., and Goldt, S. (2023). Neural networks trained with sgd learn distributions of increasing complexity. In International Conference on Machine Learning , pages 28843-28863. PMLR.

Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135-1144.

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695.

Rudin, C. (2018). Please stop explaining black box models for high stakes decisions. Stat , 1050:26.

Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626.

Shah, H., Jain, P., and Netrapalli, P. (2021). Do input gradients highlight discriminative features? Advances in Neural Information Processing Systems , 34:2046-2059.

Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 .

Smith, I., Ortmann, J., Abbas-Aghababazadeh, F., Smirnov, P., and Haibe-Kains, B. (2023). On the distribution of cosine similarity with application to biology. arXiv preprint arXiv:2310.13994 .

Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2014). Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 .

Suh, N. and Cheng, G. (2024). A survey on statistical theory of deep learning: Approximation, training dynamics, and generative models. arXiv preprint arXiv:2401.07187 .

Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic attribution for deep networks. In International conference on machine learning , pages 3319-3328. PMLR.

Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2013). Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 .

Theisen, R., Kim, H., Yang, Y., Hodgkinson, L., and Mahoney, M. W. (2023). When are ensembles really effective? arXiv preprint arXiv:2305.12313 .

Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., and Ting, D. S. W. (2023). Large language models in medicine. Nature medicine , 29(8):1930-1940.

Van der Velden, B. H., Kuijf, H. J., Gilhuijs, K. G., and Viergever, M. A. (2022). Explainable artificial intelligence (xai) in deep learning-based medical image analysis. Medical Image Analysis , 79:102470.

Wang, Y. and Wang, X. (2022). 'why not other classes?': Towards class-contrastive back-propagation explanations. Advances in Neural Information Processing Systems , 35:9085-9097.

Yang, Z., Li, L., Xu, X., Zuo, S., Chen, Q., Zhou, P., Rubinstein, B., Zhang, C., and Li, B. (2021). Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Processing Systems , 34:17642-17655.

A Additional Results of the Similarities

The illustration of CIFAR-100 & TinyImagenet. In addition to the cosine similarity ρ ind ( k 1 , k 2 ) shown in Figure 2, we include more sophisticated datasets such as CIFAR-100 and TinyImagenet in Figure 12. It is observed that the observed phenomena, where ρ ind ( k 1 , k 2 ) tend to increase w.r.t. both k 1 , k 2 , hold across different datasets. It is also worth noticing that compared with the results of CIFAR-10 shown in Figure 2, the resulting ρ ind ( k 1 , k 2 ) of CIFAR-100 and TinyImagnet (Figure 12) shows another peak at around k ≈ 10 (see Figure 12(e)). This is related to the deep double descent phenomenon (Nakkiran et al., 2021), where when the complexities of the model and the dataset are comparable, the overfitting issue is at peak. For smaller such as CIFAR-10 or larger models such as ResNet, this issue becomes much less significant, as a very small k value is already sufficient for the data distribution. Also, as the complexity of the data distribution increases, the cosine similarity decreases inevitably for the same model complexity. This suggests a less concentrated distribution of the optimized models of the same capacity compared with less complex dataset. Correspondingly, the results of the expectation over the model sets for k ∈ K ′ = [10 , 20 , 40 , 80 , 160] are shown in Figure 13.

(a) CS, CIFAR-10

(b) CL, CIFAR-10

(c) RS, CIFAR-10

(d) RL, CIFAR-10

(e) CS, CIFAR-100

(f) CL, CIFAR-100

(g) RS, CIFAR-100

(h) RL, CIFAR-100

(i) CS, TinyImagenet

(j) CL, TinyImagenet

(k) RS, TinyImagenet

(l) RL, TinyImagenet

0.6

0.5

0.35

0.30

0.25

Model Capacity k

0.4

Model Capacity k

0.30

0.25

Model Capacity

0.25

Model Capacity

- 0.20

0.3

0.20

0.20

- 0.15

0.2

0.15

- 0.15

0.10

0.10

0.10

0.1

0.05

0.05

0.05

448

448

448

89299948

883499494

68249949

Model Capacity k

Model Capacity k

Model Capacity k

Model Capacity k

0.35

0.35

- 0.30

0.25

Model Capacity k

0.30

0.25

Model Capacity k

0.30

0.25

Model Capacity k

0.25

Model Capacity k

0.20

0.20

0.20

0.20

0.15

0.15

0.15

0.15

0.10

0.10

0.10

0.10

0.05

0.05

0.05

0.05

448

6924994

88449949

6924994

884499492

Model Capacity k

Model Capacity k

Model Capacity k

Model Capacity k

0.16

0.30

0.200

0.225

0.14

0.25

0.175

0.12

0.175

Model Capacity k

0.10

Model Capacity

0.20

Model Capacity k

0.150

Model Capacity

0.150

0.125

0.125

0.08

0.15

0.100

0.100

0.06

0.10

0.075

0.075

0.04

0.05

0.050

0.050

448

0.02

384

0.025

0.025

892499

68249959499

Model Capacity k

Model Capacity k

Model Capacity k

Model Capacity k

The Ablation of the Training Process. To verify that the observed increasing trends of ρ ind ( k 1 , k 2 ) with model capacities are caused by the training process of DNNs instead of some normalization issue, we compare the similarity for models with initialized parameters. The results are shown in Figure 14. All three datasets and four model families are included. It can be clearly observed that, when the model parameters are initialized, the similarity between input saliency maps of different models are distributed randomly. The cosine similarity values are very concentrated around 0, which is the mean of random distribution. This verifies that the aforementioned increasing trends are caused by the optimization of models instead of normalization process.

8

(a) CS, CIFAR-10

(b) CL, CIFAR-10

(c) RS, CIFAR-10

(d) RL, CIFAR-10

(e) CS, CIFAR-100

(f) CL, CIFAR-100

(g) RS, CIFAR-100

(h) RL, CIFAR-100

(i) CS, TinyImagenet

(j) CL, TinyImagenet

(k) RS, TinyImagenet

(l) RL, TinyImagenet

10

0.4

10

0.4

10

0.30

10

ModelCapacity

K

20

0.3

K

acity

Capac

20

0.3

Capacity

k

20

0.25

0.20

Capacity

k

20

0.25

0.20

40

80

0.2

0.1

Model

40

80

0.2

Mode

40

80

0.15

0.10

pow

40

80

0.15

0.10

160

160

0.1

160

0.05

160

0.05

60

10

2

40

80

20

Model Capacity k

Model Capacity k

Model Capacity k

Model Capacity k

0.30

10

10

10

10

Capacity

K

20

0.15

Capacity

k

20

0.25

Capacity

20

0.15

Capacity

20

0.15

0.20

40

0.10

40

0.15

40

0.10

40

0.10

Model

Model

Model

Model

80

80

0.10

80

80

160

0.05

160

0.05

160

0.05

160

0.05

20

20

20

8

20

Model Capacityk

Model Capacity k

Model Capacity k

Model Capacity k

10

10

0.20

10

0.10

10

0.12

Capacity

Model

k

20

40

0.06

0.04

Model

acity

Capac

20

40

0.15

0.10

Model

K

acity

20

40

0.08

0.06

Model

acity

Capa

20

40

0.10

0.08

0.06

80

80

80

0.04

80

0.04

0.02

0.05

160

160

160

0.02

160

0.02

80

20

40

8

Model Capacity k

Model Capacity k

Model Capacity k

Model Capacity k

20

Softmax Activations. Apart from the normalization concern, recent work in (Wang and Wang, 2022) demonstrate the difference between the input salience of the predicted logits and probabilities. As a result, we clarify that, although we define f as the predicted logit (before softmax activations), this choice does not affect the observed increasing trend, no matter when the input salience is generated concerning the logit, probability, or the loss. The results of ρ ind ( k 1 , k 2 ) , generated from the saliency maps w.r.t. the predicted probability (after softmax activations), are shown in Figure 15. It can be found that ρ ind ( k 1 , k 2 ) still increases with both k 1 and k 2 .

B Experiment Details

Model Details Throughout the experiments, we use CNNSmall, CNNLarge, ResNetSmall, and ResNetLarge as model families. Within each family, model width is controlled by the parameter k . And the model depths are controlled by the 'Small" vs. 'Large" suffixes. For CNNs, CNNSmall consists of convolutional layers with channels [ k, 2 k, 4 k, 8 k ] , while CNNLarge repeats each layer twice: [ k, k, 2 k, 2 k, 4 k, 4 k, 8 k, 8 k ] . The details of CNNs are shown in Table 2. Additionally, for TinyImagenet, since the input data is of size 64 × 64 , we increase the stride of the second MaxPool2d layer (Layer 10) to 4 . As for ResNets, we modify the width of ResNet-10 for ResNetSmall and ResNet-18 for ResNetLarge. Note that k = 64 ResNetSmall corresponds to ResNet-10, while k = 64 ResNetLarge corresponds to ResNet-18. The sizes of models are illustrated in Figure 16 as the # of trainable parameters.

It should also be noted that, ideally, CNNSmall and CNNLarge, ResNetSmall and ResNetLarge are considered as the same families due to the same architecture. However, since widths can be adjusted independently of the depth, while the adjustment of depth inevitable affects the width, we split them.

(a) CIFAR-10, CS

(b) CIFAR-10, CL

(c) CIFAR-10, RS

(d) CIFAR-10, RL

(e) CIFAR-100, CS

(f) CIFAR-100, CL

(g) CIFAR-100, RS

(h) CIFAR-100, RL

(i) TinyImagenet, CS

(j) TinyImagenet, CL

(k) TinyImagenet, RS

(l) TinyImagenet, RL

-0.01

-0.00

0.02

x azis apo

0.02

0.01

0.02

0.08

0.06

0.04

x azi5 (apo

0.06

0.00

0.04

0.02

0.00

000

0.01

0.02

0.02

0.01

0.02

0.04

0.04

0.06

0'0-

0.02

0.06

0.08

0.08

48

3

0.02

0.015

0.04

- 0.04

0.010

-0.01

x aziS (apow

0.005

0.02

y azis apo

0.02

0.00

0.000

0.005

0.00

0.00

-0.01

0.010

0.02

0.015

0.02

Z0'0-

0.020

0.04

48

0.04

-0.02

0.02

200

200

0.01

- 0.01

0.01

0.01

x azis lapow

x azi apow

00°0

0.00

0.00

0.00

0.01

0.01

0.01

0.02

0.01

0.02

0.03

20'0-

0.02

-0.03

0.04

But our experiments in Section 3.1 verify that the depths do not affect the population mean of model

(a) CIFAR-10, CS

(b) CIFAR-10, CL

(c) CIFAR-10, RS

(d) CIFAR-10, RL

(e) CIFAR-100, CS

(f) CIFAR-100, CL

(g) CIFAR-100, RS

(h) CIFAR-100, RL

(i) TinyImagenet, CS

(j) TinyImagenet, CL

(k) TinyImagenet, RS

(l) TinyImagenet, RL

0.30

0.25

0.200

0.14

Model Capacity k

0.25

0.20

Model Capacity k

0.20

Model Capacity k

0.175

0.150

Model Capacity k

0.12

0.10

- 0.125

0.15

0.15

0.100

0.08

0.10

0.10

0.075

90°0

0.05

0.050

0.04

448

0.05

448

0.025

448

88449949498889

Q855

884499482988888

1188238848

Model Capacity k

Model Capacity k

Model Capacity k

Model Capacity k

0.25

0.25

0.200

0.175

Model Capacity k

0.20

Model Capacity k

0.20

Model Capacity k

0.175

0.150

Model Capacity k

0.150

0.15

0.15

0.125

0.125

0.100

0.10

0.10

0.100

0.075

0.05

0.05

0.075

0.050

0.050

0.025

448

448

0.025

694499

8944994

882499494985

88449949498

Model Capacity k

Model Capacity k

Model Capacity k

Model Capacity k

0.10

0.200

0.16

0.16

Model Capacity k

0.08

Model Capacity k

0.175

0.150

Model Capacity k

0.14

0.12

Model Capacity k

0.14

0.12

0.06

0.125

0.10

0.10

0.100

0.08

0.08

0.04

0.075

0.06

0.06

0.050

0.04

0.04

0.02

0.025

200

0.02

8829994949

8944994940

882499492

Model Capacity k

Model Capacity k

Model Capacity k

Model Capacity k

1e8

7

CNN Small

Parameters

6

CNN Large

5

ResNet Small

ResNet Large

4

able

3

Trainak

2

1

#

0

0

100

200

300

400

500

(a) Depth-Wise Similarity

(b) Width-Wise Similarity

Depth = 1

Depth = 2

Depth = 3

Depth = 4

Depth = 5

0.30

VPW

128

64

16

32

128

32

16

64

128

16

32

64

16

32

64

128

16

32

64

0.20

0.15

0.10

MIS

pth

2

2

2

3

0.20

0.25

0.15

0.10

AuewiS

256

256

256

256

256

0.05

0.05

163264 128 256

163264 128 256

1632 64 128 256

1632 64 128 256

1632 64 128 256

0.00

1

2

Depth

3

4

5

1

2

Depth

3

4

5

1

2

Depth

3

4

5

1

2

Depth

3

4

5

2

Depth

3

4

5

0.00

Figure 17: The cosine similarities between CNN models with different widths (left) and depths (right). Widths and depths are enumerated in { 16 , 32 , 64 , 128 , 256 } and { 1 , 2 , 3 , 4 , 5 } . Note that to scale the depths without affecting widths, here we fix the widths of all layers to k instead of [ k, 2 k, 4 k, 8 k ] as the manuscript distributions. In experiments, we set X as the first 1000 samples of the unshuffled testing set of each dataset (CIFAR-10/100, TinyImagenet).

C Additional Settings

This work aims to reveal the convergence trend of the distribution of model behaviors under the stochasticity of the training criterion. This does not limit the conclusion to the specific criterion described above. Distinct training criteria can lead to different distributions of trained models. However, these different distributions of trained models all satisfy the revealed trend. To verify this, we present additional experiments to investigate possible variants such as (1) depths and widths; (2) learning rates; (3) batch sizes; (4) solvers (5) initializations, and (6) other model architectures. Note that as studied in Figure 6 the manuscript, ρ ind can be a computationally efficient compromise of ρ . Therefore, we studied ρ ind in these additional experiments. Besides, there exists enumerous possible combinations of different variants of different aspects. As a result, here we only vary these settings partially since enumerating the entire grid is infeasible. The results as detailed as follows.

Depths and widths. The scale of depths is not as straightforward as the width since modifying depths may change widths as well. Therefore, in the manuscript we study the influence of depth by setting -small and -large variations. Here we present additional results that study the influence of depths continuously, with 1-5 layers, each of which is followed by a max-pooling layer with stride 2. Finally, an adaptive pooling layer is appended at the end. To rule out the influence of widths (channels), all layers have the same width, determined by k . e.g., For the 4-layer scenario, the intermediate layers have widths [ k, k, k, k ] instead of [ k, 2 k, 4 k, 8 k ] in the manuscript. The results are shown in Figure 17. It can be found that (1) Given a fixed depth or width, the influence of the

(a) The cosine similarity between CNNs

(b) The cosine similarity between ResNets

(c) The Loss of Models

(d) The Accuracy of Models

BS = (64, 64)

BS = (64, 128)

BS = (64, 256)

BS = (64, 512)

BS = (128, 128)

BS = (128, 256)

BS = (128, 512)

BS = (256, 256)

BS = (256, 512)

BS = (512, 512)

UOPUM

192

256

0.4

0.2

0.4

0.2

0.4

0.2

0.5

0.3

0.1

0.3

0.2

0.1

0.4

0.4

0.3

0.2

0.1

0.5

- 0.2

0.4

0.2

0.3

0.1

0.4

0.3

0.2

0.4

0.1

0.2

0.3

0.1

0.4

0.3

0.1

0.2

Aauegwes aurseg

UOPUM

128

192

256

BS = (64, 64)

0.2

E'0

T'0

BS = (64, 128)

0.2

0.3

0.1

BS = (64, 256)

0.15

0.25

0.20

0.10

0.05

BS = (64, 512)

0.15

0.19

0.05

BS = (128, 128)

0.3

0.2

0.1

BS = (128, 256)

0.25

0.20

0.15

0.10

0.05

BS = (128, 512)

0.15

0.10

50'0

BS = (256, 256)

0.20

0.15

0.10

0.09

BS = (256, 512)

0.15

0.1d

50'0

BS = (512, 512)

0.075

0.050

0.025

0.100

Aajejw(s aujseg

CNNSmall

ResNetLarge

CNNSmall

ResNetLarge

X

BS=64

0.03

1.0

***

1.0

****

0.015

BS=256

BS=512

0.02

0.9

0.9

LoSS

0.010

0.005

train

test

0.01

0.8

BS=64

BS=256

BS=512

0.8

0.7

X*

X-X-x*

*-X

0.000

*1x

0.00

0.7

*

train

test

0.6

101

102

101

102

101

102

101

102

width k

width k

width k

other factor is similar when scaled up. (2) Depths are slightly different from widths. Larger widths lead to higher similarities, while closer structures in depths have higher similarities. For widths (left), the similarity always increases left-to-right and top-to-bottom. But for depths (right), pairs near the diagonal have higher similarities.

Batch Sizes. We investigate the influence of batch sizes, varying in { 64 , 128 , 256 , 512 } The results are shown in Figure 18. It can be observed that although different batch sizes lead to different performance (e.g. testing accuracy), the convergence trend holds in all scenarios.

Learning Rates. We test different learning rates on how they affect the results. We include 1e-1, 1e-2, default , where ' default " refers to the criterion used in the manuscript. As shown in Figure 19, the revealed trend is preserved in all learning rates. It is also worth noticing that learning rates affect ResNets more than CNNs.

Solvers. Apart from SGD, we include Adam, AdamW, and SGD w/ momentum. For Adam and AdamW we set the learning rate to 1e-3, while SGD w/ momentum uses a learning rate of 1e-1 with a momentum of 0.9. The results are shown in Figure 20. Although different solvers lead to models of different performances, they all preserve the same convergence trend.

Initializations. Given a training scheme and model family F ( k ) , the training procedure leads to a distribution of trained models p ( f ) . When the initialization is fixed to θ , the training procedure is essentially sampling from the conditional distribution p ( f | θ ) instead of the unconditional distribution p ( f ) . We then studied the difference between the unconditional distribution p ( f ) and the conditional distribution p ( f | θ 0 ) . We focus on two conditional distributions p ( f | θ 0 ) and p ( f | θ 1 ) , where θ 1

(a) The cosine similarity between CNNs

(b) The cosine similarity between ResNets

(c) The Loss of Models

(d) The Accuracy of Models

Lr = (0.1, 0.1)

Lr = (0.1, 0.01)

Lr = (0.1, default)

Lr = (0.01, 0.01)

Lr = (0.01, default)

Lr = (default, default)

幼2忆28

9'0

0.3

0.6

0.15

0.4

0.5

Cosine Similarity

0.4

0.2

0.4

0.10

0.2

0.3

128

192

96

0.2

0.1

0.2

0.05

0.1

0.1

0.2

256

2

628225

= 2

12

26

295

= 2

= 2

= 2

Lr = (0.1, 0.1)

Lr = (0.1, 0.01)

Lr = (0.1, default)

Lr = (0.01, 0.01)

Lr = (0.01, default)

Lr = (default, default)

幼2忆28

0.4

E'0

0.08

0.125

0.3

Cosine Similarity

E'0

0.10

0.2

0.06

0.100

0.075

0.2

128

96

0.2

0.05

0.1

0.04

0.050

0.1

192

256

0.1

0.02

0.025

2

826

25

Zπ=

12

26

29 5

= 2

幼2忆9728

295

= 2

= 2

CNNSmall

ResNetLarge

CNNSmall

ResNetLarge

X

BS=64

0.03

1.0

***

1.0

****

0.015

BS=256

BS=512

0.02

0.9

0.9

LoSS

0.010

0.005

train

test

0.01

0.8

BS=64

BS=256

BS=512

0.8

0.7

X*

X-X-x*

*X

0.000

*1

0.00

0.7

*

train

test

0.6

101

102

101

102

101

102

101

102

width k

width k

width k

(a) Cosine Similarity

(b) Performance

4IPM

192

256

第的忆

第

Solver = (SGD, SGD)

0.4

E'0

0.2

0.5

0.1

第的

Solver = (SGD, Adam)

'0-

0.2

0.1

0.4

Solver = (SGD, AdamW)

0.3

0.2

0.1

0.4

Solver = (SGD, Momentum)

0.20

0.15

0.10

0.05

Solver = (Adam, Adam)

- 0.4

0.2

Solver = (Adam, AdamW)

0.4

0.2

Solver = (Adam, Momentum)

0.15

0.10

0.05

Solver = (AdamW, AdamW)

0.4

0.2

Solver = (AdamW, Momentum) Solver = (Momentum, Momentum)

0.15

0.10

0.05

0.100

0.125

0.075

0.050

0.025

AUIS

ques

aH

LoSS

1.0

20

15

0.9

n

10

?

0.8

SGD

Adam

AdamW

5

Momentum

0.7

*

train

X

test

101

102

101

102

width k

̸

Similarity of Means

0.07

0.75

10 -

10

0.70

0.06

20

20

0.65

0.05

0.60

40

0.04

40

0.55

0.03

0.50

80

80

0.45

10

20

40

80

10

20

40

80

K2

K2

represents the initializations under seed=1 and θ 0 represents those under seed=0 . Other settings are identical. We thus have f 0 1 , · · · , f 0 100 ∼ p ( f | θ 0 ) and f 1 1 , · · · , f 1 100 ∼ p ( f | θ 1 ) . The superscript indicates the initialization seeds and the subscript indicates the training seeds.

̸

First, we notice immediately that the training seeds for both θ 0 and θ 1 are 1 100. This means that ∀ i, f 0 i , f 1 i differ only in initializations. We inspect (a) ρ ind ( f 0 i , f 1 i ) (100 pairs) to see if they have exceptional similarity compared with (b) ρ ind ( f 0 i , f 1 j ) , i = j ( ( 100 2 ) = 4950 pairs). Besides, within the same condition, all models only differ in terms of the orders of the training batch. We thus also inspect the similarity of all models of the same condition: (c) ρ ind ( f 0 i , f 0 j ) , i = j and (d) ρ ind ( f 1 i , f 1 j ) , i = j . Each of them has ( 100 2 ) = 4950 pairs.

As demonstrated in Table 3, (i) the comparison between (a) and (b) indicates that with different initializations, the same order of batches in the training procedure does not contribute to higher similarities. (ii) The comparison among (b)(c)(d) indicates that the same initialization indeed leads to higher similarity even though the order of batches is distinct. It should be noted that the contributions of batch orders and initializations are also affected by the number of epochs. Intuitively, more training epochs should lead to smaller contributions from the initializations but greater contributions from the batch orders.

Vision Transformers. Vision Transformers (ViT) have risen in recent years as another powerful architecture for CV tasks. Here we include a brief study of ViTs to demonstrate that the discovered phenomenon holds across different architectures.

Specifically, we train vision transformer (ViT) models on CIFAR-10 with varying capacities controlled by k ∈ { 10 , 20 , 40 , 80 } . CIFAR-10 has an input size of 32 × 32 pixels, thus the patch size is set as 4 × 4 , resulting in 8 × 8 patches. The embedding size is set to 4 k , divided by k/ 2 heads, and we set the depth to 8 . The seeds vary in 1-100 and results in 100 trained models of each k . We study the mean of the similarity ρ ( k 1 , k 2 ) (i.e. the same experiments as Figure 3 in the manuscript) and the similarity of the population mean (i.e. the same as Figure 5 in the manuscript). The results are shown in Figure 21. It can be observed that although distinct from convolutional layers, the transformer structure also has the discovered convergence trend. It can also be noted that the degree of dispersion of ViTs is much higher than CNN-based models.

In conclusion, although training schemes can affect the resulting distributions of models, the influence of the model capacity stays invariant across different criteria.

D Uniform Distributions on the Hypersphere

According to (Muller, 1959), due to the spherical symmetry property of the zero-mean Gaussian distribution, the cosine similarity between two Gaussian variables are actually uniformly distributed

̸

over the hypersphere S d -1 in R d . 3 Therefore, the cosine similarity between two i.i.d. multivariate Gaussian tensors is essentially the inner product between two i.i.d. uniform tensors over the hypersphere. Formally, suppose X,Y to be high-dimensional i.i.d. random variables of dimension d that follow the Gaussian distribution N ( 0 , I d × d ) . The cosine similarity is Z = X T Y ∥ X ∥·∥ Y ∥ . WLOG, it suffices to consider the scenario where Y ∥ Y ∥ = e 1 = (1 , 0 , · · · , 0) . And it is written as

As a result, Z 2 = X 1 X 2 1 + ∑ d i =2 X 2 i follows a beta distribution Z 2 ∼ Beta ( 1 2 , d -1 2 ) . The pdf is thus f Z 2 ( x ) = x -1 / 2 (1 -x ) ( d -3) / 2 B (1 / 2 , ( d -1) / 2) . And then when Z > 0 ,

According to (Smith et al., 2023), let u = (1+ x ) / 2 ,then x 2 = (2 u -1) 2 . Then this can be simplified to

Finally, we are able to conclude that 1+ Z 2 ∼ Beta ( d -1 2 , d -1 2 ) , where Z is the cosine similarity between two i.i.d. d -dimensional Gaussian vectors. This result suggests that the distribution of Z will become very concentrated around 0. And this concentration exacerbates exponentially with the dimension d . Given a cosine similarity level ρ , the probability P ( Z > ρ ) can be extremely small, and also deceases exponentially with ρ , too. We visualize the relation between the probability P ( Z > ρ ) and ρ with varying dimensions d in Figure 22. In the low-dimensional space such as d = 3 , the distribution of the cosine similarity is close to uniform as humans' intuition. However, as the dimension increases, the cosine similarity is very unlikely to maintain high, as demonstrated by the curves of d = 3 × 8 × 8 = 48 (orange) and d = 3 × 32 × 32 = 3072 (green).

This is because

TheCDFof thecosinesimilaritywithdifferentdimensiond

area/sphere

10-2

10-5

cap

10-8

10-11

d=3

(>p),

d=192

Z)d

10-14

d=3072

0.0

0.2

0.4

0.6

0.8

1.0

CosineSimilarityp

E Supplementary Experiment Results

Note that all the experiments are carried across three datasets CIFAR-10/100 and TinyImagenet, along with four model families CNNSmall, CNNLarge, ResNetSmall, and ResNetLarge. Due to the space limit, only part of the selected results can be put in the manuscript. Therefore, we defer the results with different models/datasets into this section for the audience' reference. All the conclusions drawn from the experiment results shown in the manuscript hold for the results demonstrated here.

In Figure 23, we present the complementary results of Figure 6 on CIFAR-100 and TinyImagenet. It can be clearly observed that ρ ind and ρ have very similar values, after taking the expectation over X .

Figure 25 shows the results of black-box attack results between different models. All other settings are identical to the results shown in Figure 10, but with CIFAR-100 and TinyImagenet instead of CIFAR-10. It is observed that the capacity of models has significant influence to the models' robustness and black-box attack transferability. And this trend is highly correlated to the similarity ρ ind , ρ as demonstrated in Figures 2, 3, 12, 13 and 15.

Similarly, Figure 26 shows supplementary results of Figure 11. All settings are identical except for the datasets. CIFAR-100 and TinyImagenet are tested instead of CIFAR-10. It can be observed that using the estimated mean gradient direction (blue), the performance drops much more significantly than the attack from single models of either the exact same family as the target model (green) or the largest single model tested ( k = 448 , red). Note that due to the complexity of TinyImagenet, in the overfitting phase (i.e. when the target models' capacities are comparable to the dataset (Nakkiran et al., 2021)), the single-model black-box attack results in opposite effect - the prediction actually increases.

In Figure 24, we present CIFAR-100 and TinyImagenet results as supplementary of Figure 9. It can be observed that the testing loss is highly correlated to the expectation of t = u T µ ( x ) . Such phenomena are also consistent across different model families and datasets. For both single models and ensembles, the closer they are to the convergent limiting point (i.e. larger E [ t ] ), the higher testing performance they have. Note that here for the sake of consistency, we approximate µ ( x ) through ˜ µ (160; x ) . Therefore, in the ensemble experiments (left of each subfigure), k = 160 is omitted.

(a) CIFAR-100

(b) TinyImagenet

CS

CL

RS

RL

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0.1

Pind

p

0.0

0.0

0

200

400

0

200

400

0

200

400

0

200

400

CS

CL

RS

RL

0.3

.20

9.20

Pind

0.10

0.2

9.15

Q.15

p

9.10

q.10

0.05

0.1

Q.05

9.05

0.0

4.00

0

200

400

0

200

400

0

200

400

0

200

400

(a) CIFAR-100

(b) TinyImagenet

SingleModels

DeepEnsembles

SingleModels

DeepEnsembles

6

6

7

5

5

6

6

Testing Loss

Testing Loss

Testing Loss

SS07

4

4

Testing

5

3

3

4

4

2

2

3

3

1

+

2

2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Marginal Expectation E[t]

Marginal Expectation E[t]

Marginal Expectation E[t]

Marginal Expectation E[t]

CNNSmall

ResNetSmall

■

k=10

k =40

CNNSmall

ResNetSmall

■

k=10

k =40

CNNLarge

ResNetLarge

+

k=80

CNNLarge

ResNetLarge

+

k=80

(a) CIFAR100, CS

(b) CIFAR100, CL

(c) CIFAR100, RS

(d) CIFAR100, RL

(e) TinyImagenet, CS

(f) TinyImagenet, CL

(g) TinyImagenet, RS

(h) TinyImagenet, RL

助死忆8

1.0

的忆8

1.0

16

8

1.0

忆8

1.0

Source Model k

0.8

0.6

Model

K

0.8

0.6

Model

24

48

32

0.8

0.6

Modelk

0.8

0.6

96

0.4

Source

96

0.4

Source

96

64

0.4

Source

96

9

0.4

128

128

128

128

192

0.2

192

0.2

192

0.2

192

0.2

256

256

256

256

384

0.0

384

0.0

384

0.0

384

0.0

Target Model k

Target Model k

Target Model k

Target Model k

忆8

1.0

812164286

1.0

忆孔8

1.0

助忆8

1.0

Source Modelk

0.6

0.8

Modelk

0.8

0.6

Model

48

32

0.6

0.8

Source Model k

0.8

0.6

9

Source

9

Source

64

96

0.4

96

0.4

96

0.4

96

0.4

128

128

128

128

192

0.2

192

0.2

192

0.2

192

0.2

256

384

0.0

256

384

0.0

256

384

0.0

256

384

0.0

Target Model k

Target Model k

Target Model k

Target Model k

(a) CIFAR100, CS

(b) CIFAR100, CL

(c) CIFAR100, RS

(d) CIFAR100, RL

(e) TinyImagenet, CS

(f) TinyImagenet, CL

(g) TinyImagenet, RS

(h) TinyImagenet, RL

Attack by u E Gx(x)

Attack by u E Gx(x)

Attack by u E Gx(x)

Prediction Decay Ratio α

100

Attack by u E Gx(x)

Attack by u E Gmax(x)（x)

Prediction Decay Ratio α

Attack by u E Gmax(k)(x)

Attack by μi(max{K'}: x)

Prediction Decay Ratio α

Attack by u E Gmax(K) (x)

Attack by μi(max{K'}:x)

Prediction Decay Ratio α

Attack by u E Gmax(k)(x)

Attack by μ(max{K'}: x)

6 × 10-1

100

100

100

4 × 10-1

3 × 10-1

101

102

101

102

101

102

101

102

Prediction Decay Ratioα

100

Prediction Decay Ratioα

101

Attack by u E Gmax(k)(x)

Attack by u E Gx(x)

Attack by μi(max{K'}:x)

Prediction Decay Ratio α

Attack by u E Gmax(k)(x)

Attack by μi（max(K'}:x)

Attack by u E Gx(x)

Prediction DecayRatioα

101

Attack by u E Gmax(k) (x)

Attack by μ(max{K'}:x)

Attack by u E Gx(x)

100

6 ×10-1

100

4 ×10-1

100

3 × 10-1

Attack by u E Gx(x)

2 ×10-1

Attack by μi（max{K'};x)

Attack by u E Gmax(K)(x)

10-1

10-1

101

102

101

102

101

102

101

102

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract mainly points out the hypotheses made in this work. They are all properly discussed and empirically verified in the manuscript.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to the conclusion section for the limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The theoretical analysis in this paper is rigorous. Hypothesis are both theoretically explained and empirically verified.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in the appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The information needed to reproduce the experiments is discussed in both the manuscript and the appendix. The code is also provided in the supplementary materials. The repository will be publicized upon acceptance.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is provided as the supplementary material in the submission. The repository will be publicized upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details of the experiments are discussed in both the manuscript and the appendix. The code is also provided in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The potential statistical significance lies in the variance among different ensemble members. This is studied in Table 1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to the first paragraph of Section 3

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the code of ethics and this work conforms with the code of ethics of NeurIPS.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Focusing on fundamental, theoretical and general topics, this works does not have societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no such risk of this work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This work only uses general packages such as Numpy , PyTorch , torchvision , etc.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This work does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the resemblance of the functions obtained by training neural networks across various network depth and width. They do so by looking at the gradient field of the learned functions $f_1, f_2$, and defining their similarity by taking the cosine similarity between the gradients, then taking independent average over the inputs of the functions. They empirically observe that increasing the width results in higher similarity between the gradient fields, from which the deduce to properties: (1) the average gradient norm increase with width of the net, and (2) the direction of the average almost does not change when going to networks with larger width, but the variance decrease and the distribution becomes more concentrated. They also calculate the similarity measure and some properties of the gradient distribution assuming the distribution is gaussian or spherically symmetric. The major claim of. this work is summarized in hypothesis H1 and H2 on lines 145 and 160.\n\nSTRENGTHS:\nAuthors formulate an interesting hypothesis based on empirical observation regarding the behavior of the gradient field of learned functions in DNNs: the direction of the vector does not change too much when we increase the size of the network. They form this hypothesis from the observation that the average dot product of the gradient fields across networks with different width increases compared to the norm of the average of the gradient field for the smaller network.\n\nWEAKNESSES:\nMy major concerns:\n(1) The claims are not backed my sufficient reasoning and relevant experiments. Note that the similarity measure is a function of the dot product of the average normalized gradients, so seeing an increasing trend on this quantity can be the result of two things: (*) the angle between the average of the normalized gradients doesn't change or gets smaller, or (**) the size of the average itself is increasing. Note that either of (*) or (**) is sufficient to happen to see an increasing value of cosine similarity that the authors have defined. However, observing an empirical increase in the value of similarity by going to larger nets, the authors conclude both of (*) and (**) at the same time, while it might be the case that (**) happens, ie.e. the size of the average is increasing fast, so that even though the direction of the averages are getting further apart, the overal dot product is still increasing. Therefore, in order to scientifically claim that both (*) , (**), one approach is that authors study the angle between the normalized averages (or dot product of NORMALIZED averages) separately. Specially because these claims are not backed by theoretical arguements.\n\n(2) The authors discuss the 'saw distribution' and modeling the behaviour of gradients with this dsitribution assuming the actuall gradient distribution is spherically symmetric (or close to it), and assumption that seems very strong and probably incorrect for actual neural nets. Is there a particular observation for this assumption? Authors discuss some high level calculation with this distribution on line 246, but I didn't find any calculation in appendix. \n\n(3) the writing is vague and it does not flow. In particular, the story for the theoretical calculation with gaussian distribution is not explained, and the assumptions are not justified at all. The section regarding saw distribution needs a lot of changes to become mroe clear.\n\nSome minor points:\n- the authors claim that the fact that given networks A, B where A has larger width, they claim that the observation that the average similarity of A to B is larger than that of B to itself is 'counter-intuitive' and 'paradoxical'. This is a wrong claim and needs to be omited from the paper; the reason that it is not paradoxical is you are working with 'some measure of similarity' rather than 'distance' or 'metric', for a simple illustrative example, if you define similarity of two numbers a, b to be their product ab, then if a > b > 0, ab > b^2 but it doesn't mean that because a, b are more similar, we should conclude that a is 'closer' to b than b to itself.\n\n- In Equation (5) you are estimating the normalized average of the gradient which is used in calculating your similarity measure, but I think the quantity of interest is the normalized average of the normalized gradients (because of the definition of cosine similarity that you use)?\n\nminor comments:\n- at H1 hypothesis in line 145:  the mean vector \\mu is not defined at that point of the paper\n-typo in line 158 -> should be k_1 and k_2\n\nQUESTIONS:\nWhat is the conclusion for training/optimization/generalization efficiency the authors look into obtain from this observation? To me the average of normalized gradient seems not to be correlated with generalization ability of neural nets. maybe the authors can add more concluding points or remarks about their empirical observation.\n\n- What is the difference between the message of Figure 3 and 5?\n\n- Are you assuming the gradient distribution is gaussian in section C?\n\nLIMITATIONS:\nThe claim needs further be backed by more delicate experiments. Furthermore, the assumptions for calculation of the behavior of the similarity measure using certain classes of distributions needs to be justified at least in some regime of training DNNs. For the rest see weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the uncertainty introduced in stochastically optimized DNNs via the input saliency maps. By empirical evaluations, the authors discovered that 1) within the same model architecture, models with different capacities tend to align in terms of their population mean directions of the input salience, 2) the distributions of the optimized models concentrated more around their shared population mean as the capacity increases. These observations are very interesting and shed some light on the applications of DNNs such as black-box attacks.\n\nSTRENGTHS:\nThe authors took a suitable approach to study the uncertainty introduced in stochastic training of DNNs. The observations made by them are very interesting and inspiring, and well supported by empirical evidences. The authors did a good job explaining their main discoveries using figures and equations. These findings could shed light on the practice of DNNs.\n\nWEAKNESSES:\nThe main complaint I have on the paper is its presentation. Although the observations quite interesting to me, I found the writing of the paper can be improved.\n\n1. More context is needed for Figure 1.\n1.1. How are the dots plotted? I assume you did something like PCA to reduce the gradient dimension to 3 and then plot them on the sphere.\n1.2. You said on line 55 that the second hypothesis is illustrated in Figure 1(b), but there is no indication of model capacities in the figure.\n\n2. Even though the authors tried to distinguish between model architecture from model family, the introduce of width and depth complicates the discussion and makes the definition less clear. It would be good to refine the presentation here. For instance, you could give a concrete example like ResNet_depth10_width10_seed0 and ResNet_depth10_width10_seed1 has the same model architecture and belong to the same model family. ResNet_depth10_width10 and ResNet_depth10_width20 has the same model architecture and belong to different model families, etc.\n\n3. It is unclear to me what is defined on line 96.\n\n4. The presentation of the paragraph on line 101 and Figures 2 & 3 can be improved.\n4.1. Why do you use the notation $\\rho_\\text{ind}$? What does ind mean?\n4.2. Can you give clear, different names to $\\rho_\\text{ind}$ and $\\rho$, and make both of them display equations?\n4.3. In the captions of Figures 2 & 3, can you use the $\\rho$ notations and refer to their display equations, respectively?\n4.4. The x and y labels in Figures 2 & 3 should be $k_1$ and $k_2$.\n\n5. $\\mu(k; x)$ is not defined in Hypothesis I.\n\n6. Some $k_1$ should be $k_2$ on line 158 and in the display equation below. And why do you need $k_1 > k_2$ for this?\n\nQUESTIONS:\nBesides the points I made in the Weakness section, I have one more questions. When practitioners scale their models, they usually increase both depth and width simultaneously (especially in the context of scaling law). In your experiments, you mostly studied the effect of increasing width. Could you add results in the flavor of Figures 2 & 3 where the model capacity is parametrized by both the depth and width following some scaling law relationship?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

comments: SUMMARY:
This paper studies the distribution of input saliency maps of trained neural networks at varying depth and width. The authors observe that as the model capacity increases, these distributions converge towards and become concentrated around a shared mean direction. The authors also state two hypotheses for the behavior of these distributions and provide empirical evidence for them.

STRENGTHS:
- The paper clearly formulates and discusses two hypotheses (H1,H2) that help capture the empirical observations.
- While the experiments are small scale, they are sufficient to support the authors main claims and observations.

WEAKNESSES:
- The authors do not distinguish between the stochasticity introduced by random initialization and the stochasticity introduced by optimization. In Appendix A, they discuss the similarity scores at initialization, but a more interesting ablation would be to compare fixing the initialization and then randomizing the batches with fixing the ordering of the batches but randomizing the initialization.
- Related to the above point, there is no discussion of the effect of the learning rate or batch size which should greatly affect the final results. This would help disentangle the randomness introduced by initialization vs optimization.

QUESTIONS:
- Have the authors explored the effects of different optimization hyperparameters (e.g. learning rate, batch size) or optimization algorithms (e.g. Adam) on the convergence trends observed in the paper?
- The experiments in this paper are focused on computer vision. Do the authors expect similar conclusions to hold in other domains as well (e.g. NLP)?

LIMITATIONS:
The authors have adequately addressed the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the distribution of trained deep neural networks through the lens of input saliency maps. It empirically shows that as the capacity of either of two stochastically optimized models increases, they tend to resemble each other more. Therefore, the authors hypothesize that within the same architecture (Resnet or CNN), different variants align in their input salience directions, and converge towards a shared population mean as their capacity increases. Furthermore, they propose a semi-parametric model, based on the Saw distribution, to capture this convergence trend. These findings enhance the understanding of various applications such as black-box attacks and deep ensembles.\n\nSTRENGTHS:\n1. The paper is well-written.\n2. The experiments are comprehensive and well-designed.\n3. The input saliency maps perspective on the ensemble problem is novel.\n\nWEAKNESSES:\n1. The primary finding is not entirely novel. Previous research has demonstrated that the performance of neural network ensembles converges to that of a single model as the model size increases [1,2]. Similarly, as the size of the models increases, the variance in predictions and disagreement among ensemble members diminishes [1].\n2. Several details about the experimental setup are omitted. These include the optimizer used, learning rate, learning rate schedule, batch size, number of iterations, weight decay, and data augmentation techniques.\n3. The choice of optimizers and hyperparameters can significantly alter the distribution of trained models. However, the study lacks an analysis of their findings across different optimizers, hyperparameters and sources of stochasticity.\n4. The experiments are conducted in small-scale settings, which may limit the generalizability of the results.\n\n[1] Geiger, Mario, et al. \"Scaling description of generalization with number of parameters in deep learning.\" _Journal of Statistical Mechanics: Theory and Experiment_ 2020.2 (2020): 023401.\n\n\n[2] Kobayashi, Seijin, Johannes von Oswald, and Benjamin F. Grewe. \"On the reversed bias-variance tradeoff in deep ensembles.\" ICML, 2021.\n\nQUESTIONS:\n1. Have the authors conducted experiments to validate their findings across different optimizers and related hyperparameters? \n2. Stochasticity can be introduced in the training of Neural Networks in various ways. Changing the initialization through different seeds is one possibility but other possibilities include for example training on different subsets of the data, large or small batch sizes, large or small learning rates, data augmentation or label noise. Have the authors validated their findings for different sources of stochasticity? \n3. In Figure 8, why does the Large ResNet exhibit a higher test loss than the small CNN? \n4. The explanation for the effectiveness of deep ensembles suggests that they better approximate the population mean. However, this assumes that the mean corresponds to a good generalizing model. Could the authors provide evidence or an explanation to support this assumption?\n5. The introduction mentions \"input salience\" several times without providing a definition. For better clarity and understanding, it would be helpful to include a definition of \"input salience\" in the introduction.\n6. In section 2.1, the authors state, \"We focus on $f: \\mathbb{R}^d \\to \\mathbb{R}$ which predicts the logit specifically for the targeted class.\" Could the authors clarify this statement? Does it mean that the analysis after training considers only the logit corresponding to the true class?\n7. In section 2.2, $\\mathbf{u}_1$ and $\\mathbf{u}_2$ are sometimes used to represent different samples from the same set $\\mathcal{G}_k$, and sometimes from different sets.  It would be better to use a different notation to differentiate between the two cases. \n8. In Hypothesis I, line 147, the term $\\mu(k;x)$ is only introduced later in the text. \n9. In line 234, could the authors clarify the definition of $\\rho_{ind}(f,f;x)$ and explain its relationship with $\\rho(k,k;x)$ ?\n\nLIMITATIONS:\nThe limitations of the study are correctly addressed in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-17"}