Great Minds Think Alike: The Universal Convergence Trend of Input Salience

Authors: yipei  wang
jeffrey mark siskind
xiaoqian  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:46:25

Abstract: Uncertainty is introduced in optimized DNNs through stochastic algorithms, forming specific distributions. Training models can be seen as random sampling from this distribution of optimized models. In this work, we study the distribution of optimized DNNs as a family of functions by leveraging a pointwise approach. We focus on the input saliency maps, as the input gradient field is decisive to the models' mathematical essence. Our investigation of saliency maps reveals a counter-intuitive trend: two stochastically optimized models tend to resemble each other more as either of their capacities increases. Therefore, we hypothesize several properties of these distributions, suggesting that (1) Within the same model architecture (e.g., CNNs, ResNets), different family variants (e.g., varying capacities) tend to align in terms of their population mean directions of the input salience. And (2) the distributions of optimized models follow a convergence trend to their shared population mean as the capacity increases. Furthermore, we also propose semi-parametric distributions based on the Saw distribution to model the convergence trend, satisfying all the counter-intuitive observations. Our experiments shed light on the significant implications of our hypotheses in various application domains, including black-box attacks, deep ensembles, etc. These findings not only enhance our understanding of DNN behaviors but also offer valuable insights for their practical application in diverse areas of deep learning.

Main paper: 
 introduction: 
 The advancement in computational power has significantly enhanced the capabilities of Deep Neural Networks (DNNs), leading to their unparalleled expressiveness and success in a multitude of applications across various fields (Krizhevsky et al., 2012; He et al., 2016; Rajkomar et al., 2018; Berner et al., 2019; Rombach et al., 2022; Padmaja et al., 2023; Thirunavukarasu et al., 2023) . Despite these achievements, DNNs remain enigmatic, not only to end-users but also to researchers and practitioners (Ribeiro et al., 2016; Rudin, 2018; Preece et al., 2018) . Due to the over-parameterization nature of modern DNNs, they are capable of reaching zero loss in the training distribution (Goodfellow et al., 2014b; Allen-Zhu et al., 2019; Du et al., 2019) . Furthermore, the inherent stochastic nature of training algorithms means that even when using the same training data, DNNs tend to converge to various minima (Huang et al., 2017; Liu et al., 2020) . Thus even though these models may exhibit comparable performance in terms of metrics like testing loss or accuracy, their underlying mechanisms can still differ significantly. Because of the stochastic nature of the training procedure, optimized DNNs collectively form a distribution over the functional space C 1 (X ), and training DNNs from scratch is thereby equivalent to randomly sampling from such a distribution without any guarantee. This inherent opacity, combined with the high dimensionality and nonlinearity, limits our understanding of the internal mechanisms of DNNs. In response to these challenges, we study the aforementioned distributions. By adopting a pointwise approach, our focus is on the distribution of input salience (Simonyan et al., 2013) from the context of eXplainable Artificial Intelligence (XAI), which aims to demystify the inner workings of these complex models (Gunning and Aha, 2019; Arrieta et al., 2020; Van der Velden et al., 2022) . Saliency maps, particularly in the form of input gradients, represent the data points within the gradient fields of DNNs. Thus the study of gradients can offer a deterministic view of the landscape of model predictions. This approach allows us to examine the intricate nuances of DNNs in a more structured and analytical manner.For clarity, in the following context, we distinguish between the term model architecture (e.g. skip/direct connections) from the term model family. The latter refers to a specific collection of models F, that differ only in capacity as determined by width and depth. Two models are said to be in the same family if they differ only in parameter values. Given an input, varying model families result in distinct distributions. A synthetic visualization of such distributions is shown in Figure 1 (a). Different models are depicted by the points. However, the relationship between different model families, represented by various colors, remains elusive. In this work, we introduce and verify several hypotheses to uncover a striking pattern. (1) Within the same model architecture (e.g., CNNs, ResNets), different family variants (e.g., varying capacities) tend to align in terms of their population mean direction. (2) As the model capacities increase, the variance within the distribution of the same family diminishes. This leads to a converging trend of the distributions. Both hypotheses are illustrated in Figure 1(b) . Additionally, we introduce a semi-parametric approach to model these distributions, providing detailed quantification of the convergence.The similarities observed in input salience have direct implications for understanding the important vulnerability of DNNs regarding gradient attacks (Szegedy et al., 2013; Goodfellow et al., 2014a) . In particular, in black-box attack settings, the gradients of the target model are not directly accessible. A higher degree of salience similarity naturally enhances transferability (Chen et al., 2023) . Our findings elucidate why models with larger capacity consistently exhibit superiority in terms of adversarial robustness compared to smaller models (Madry et al., 2017; Gustafsson et al., 2020; Li et al., 2020; Bubeck and Sellke, 2021) . Moreover, given that the mean direction is aligned across different models, it is possible to approximate this mean direction by randomly sampling from a set of independently optimized models. We demonstrate that these estimated mean directions can attain a near-perfect cosine similarity of almost 1.0, even between completely independent models or ensembles, in a high-dimensional space. Moreover, note that deep ensembles essentially calculate this population mean direction (Lee et al., 2015; Lakshminarayanan et al., 2017; Fort et al., 2019; Kondratyuk et al., 2020) , where the mean of a group of independently trained models can improve the performance. As a consequence, the insights of our hypotheses also shed light on this phenomenon which, although empirically successful, has been somewhat enigmatic in terms of their source of capability (Lobacheva et al., 2020; Deng and Shi, 2021; Abe et al., 2022; Theisen et al., 2023) . Furthermore, since deep ensembles approximate the aligned mean directions much faster than scaling up single models, this also demystifies the significant black-box attack transferability of deep ensembles (Yang et al., 2021; Chen et al., 2023) . Our research thus not only advances the understanding of model behavior in practical applications but also contributes to the broader field of AI trustworthiness and efficiency. Our main contributions can be summarized as follows:• We reveal an appealing phenomenon where the mean distribution directions of input salience across different model families have extremely high resemblance. • We empirically demonstrate the distribution converges towards the mean direction as model capacity increases. • Incorporating both empirical observations and theoretical analysis, we hypothesize distributional properties of optimized models, quantifying the aforementioned phenomena. • The hypotheses effectively explain many hitherto unclear phenomena such as black-box attack transferability, the efficacy of deep ensemble methods, etc.2 The Convergence of Input Saliency (f (1) , f (2) ) = E x∈X [CosSim(∇ x f (1) (x), ∇ x f (2) (x))],where f (1) ∈ F(k 1 ), f (2) ∈ F(k 2 ). CIFAR-(1) ∈ F(k 1 ), f (2) ∈ F(k 2 ) given input x is ρ ind (f (1) , f (2) ; x) = CosSim(∇ x f (1) (x), ∇ x f (2) (x)). Tak- ing the entire testing set into consideration, denote ρ ind (f (1) , f (2) ) = E x∈X [ρ ind (f (1) , f (2) ; x)].In Figure 2 , the expectations over the testing set {8, 10, 12, 14, 16, 20, • • • , 384, 448} to balance between finer linear scaling and coarser exponential scaling. It can be observed that the similarity between two stochastically optimized models f (1) , f (2) has an increasing trend with respect to both k 1 , k 2 . Two different architectures CNN and ResNet are included. To rule out the influence of any single model, we define the similarity between families F(k 1 ), F(k 2 ) for a given input x by taking the expectation of the two models:E x∈X CosSim(∇ x f (1) (x), ∇ x f (2) (x)) with vary- ing k 1 , k 2 ∈ K are illustrated. Here, we define K = {j2 i : 4 ≤ j ≤ 7, 1 ≤ i ≤ 6} =EQUATIONThe global similarity between models of widths k 1 , k 2 is then denoted by ρ(k 1 , k 2 ) = E x∈X ρ(k 1 , k 2 ; x). Note that estimating this value requires training numerous f ∈ F(k) for each k ∈ K. Therefore, we carry out the experiments over K ′ = {j2 i : j = 5i = 1, 2, 3, 4, 5} = {10, 20, 40, 80, 160} ⊂ K. For each k 1 , k 2 ∈ K ′ , 100 models are sampled respectively to empirically estimate the expectation over F(k 1 ), F(k 2 ). As observed in Figure 3 , ρ(k 1 , k 2 ) has an increasing trend w.r.t. both k 1 , k 2 . Compared with Figure 3 , the average similarity over the model families is similar to the individual cosine similarity for the same k values. As a result, studying the similarity of two randomly sampled models instead of the expectation over Fs can significantly alleviate the computational burden. This is further discussed in detail in Section 3.2. Besides, It trivially follows that∀k 1 > k 2 , ρ(k 1 , k 1 ) > ρ(k 1 , k 2 ) > ρ(k 2 , k 2 ), which suggests that larger models tend to resemble smaller models even more than smaller models themselves. -Even if the two smaller models only differ in the random seeds during training. Please refer to Appendix A for the results of more datasets, where such increasing trends still exist. 
the spherical distribution of the salience: 
 Since the cosine similarity can be written as the inner product between ∇xf (1) (x) ∥∇xf (1) (x)∥ and ∇xf (2) (x) ∥∇xf (2) (x)∥ , which are high-dimensional unit vectors, we explore the properties and potential distributions of F through the perspective of spherical statistics. Given an input x ∈ X , we denote by G k (x) the set of all possible gradient directions of input x regarding the models f in F(k). Formally, letG k (x) = u = ∇ x f (x) ∥∇ x f (x)∥ ∀f ∈ F(k) , ∀x ∈ X (2)Then the similarity is re-written as the inner product ρ(k 1 , k2 ; x) = E u1∈G k 1 (x),u2∈G k 2 (x) [u T 1 u 2 ].The Intra-Family vs. Cross-Family Paradox. An interesting paradox is raised as ρ(•, •) increases with both inputs. Naturally, one would reasonably deduce that two models f (1) , f (2) ∈ F(k 1 ) should resemble each other since they are from the same family (i.e. having the exact same structure and only differ in training seeds). However, since ρ(k 2 , k 1 ) > ρ(k 1 , k 1 ), the cross-model family similarity becomes greater than the intra-model family similarity. To uncover the mystery of the observations, we present the intuitive understanding and the rigorously analyzed hypotheses as follows.The Intra-Family Hypothesis. Note that for intra-family scenario, u, v ∈ G k (x) are i.i.d., the similarity can be written as ρ(k, k;x) = (E G k (x) [u]) T (E G k (x) [v]) = ∥E G k (x) [u]∥ 2 2, which denotes the square of the population mean resultant length (Mardia et al., 2000) of G k (x). The population mean resultant length ρ(k, k; x) quantifies the degree of dispersion of G k (x), where a larger length suggests a more concentrated distribution. In directional statistics, the degree of dispersion is usually quantified by the spherical variance 2(1 -ρ(k, k; x)) or the total variation 1 -ρ(k, k; x). Since ρ(k, k; x) also increases w.r.t. k, this suggests an increasing concentration of input salience of models as the width k of the model increases. In conclusion, the larger the models are, the smaller the spherical variance of the salience is. Formally, we propose the following hypothesis.• Hypothesis I (H1): Let k denote the width (capacity) of the model and [u]∥ denotes the unit mean direction of G k (x).G k (x) = u = ∇xf (x) ∥∇xf (x)∥ ∀f ∈ F(k) denote the set of input gradient directions regarding x. Then E G k (x) [u] = ρ(k, k; x)µ(k; x) and ρ(k, k; x) increases with k. Here µ(k, x) = E G k (x) [u] ∥E G k (x)Note that H1 also holds for the change of model depths, which is positively related to the dispersion degree of the distribution. However, the change in model depth inevitably affects model width. Thus, we only provide empirical verification in Section 3 but do not include it as a part of H2.The Cross-Family Hypothesis. Unlike the intra-family similarity, the increasing cross-family similarity is where the phenomenon becomes counter-intuitive. Then due to the increasing intra-family similarities, when k 1 , k 2 increase, u 1 ∈ G k1 (x) becomes closer to µ(k 1 ; x), u 2 ∈ G k2 (x) becomes closer to µ(k 2 ; x). However, the cross-familty similarities suggest that as u 1 , u 2 approach their respective mean directions, their similarity increases as well. This indicates that the mean directions µ(k 1 ; x), µ(k 2 ; x) are similar, too. Formally, this intuition is considered as follows. For k 1 > k 2 , when µ(k 1 ; x) and µ(k 2 ; x) are sufficiently similar, µ(kEQUATIONwhich is monotonic w.r.t. both k 1 , k 2 . Formally, this is summarized as• Hypothesis II (H2): Let G k1 (x), G k2 (x) denote the input gradient directions of two model families where k 1 ̸ = k 2 . Then µ(k 1 ; x) ≈ µ(k 2 ; x) regardless of k 1 , k 2 .The two hypotheses are both empirically verified. For a smoother flow of the presentation, we defer the detailed experiments to Section 3. The basic ideas of H1 and H2 are illustrated in Figure 4 (a).Given the analysis and hypothesis above, one can have an overview of the models' internal mechanisms. As the model capacity increases, models are distributed in a more concentrated manner, while the mean direction stays almost invariant. To better understand the models' behavior with the stochasticity, we delve into the distribution of G k (x) and present a semi-parametric analysis with experimental verification. A general form of centralized symmetric distribution over hypersphere is known as the Saw distribution (Fisher et al., 1993) p(u; µ) = ψ(u T µ), where µ is the mean direction with ∥µ∥ = 1, ψ ∈ C([-1, 1]), and Z = S d-1 ψ(u T µ)du is the normalization term for distributions. Due to the symmetry assumption, the shape of the distribution is solely determined by ψ. For example, a monotonically increasing ψ suggests that u is distributed more densely near the mean direction and sparsely distant from the mean direction. Considering the concentration trend of gradients, we hypothesize that ψ k (•) of G k (x) not only monotonically increases with the input, but also increases faster with greater k values.Marginalization. For ∀u ∈ G k (x), it can be decomposed to u = t • µ(x) + √ 1 -t 2 • µ(x) ⊥ , where µ(x) ⊥ is a unit tangent to S d-1 at µ. Then t = u T µ(x). This is shown in Figure 4 (b) . Note that µ(x) ⊥ is independent from t, then the distribution of t is the marginal distribution over the intersection between S d-1 and the hyperplane spanned by µ(x) ⊥ , which is a (d -2)-dimensional hypersphere. According to the symmetry assumption of Saw distribution, conditioned on a fixed similarity t, the distribution of u|t over the dashed S d-2 does not affect ψ. Therefore, we focus on the marginalized distribution of t. Note that the radius of the intersectionS d-2 is √ 1 -t 2 , we thus have du = 2π (d-1)/2 (1-t 2 ) (d-3)/2 Γ((d-1)/2)dt, where the density of t is observed by the integral over the corresponding (d -2)-hypersphere. As a result, the marginal distribution of t has the PDFp k (t; x) = ψ k (t; x)(1 -t 2 ) (d-3)/2 /Z ′ (4)where Z ′ is a constant normalization term. Note that (1 -t 2 ) (d-3)/2 is a symmetric bell curve centered at t = 0. Equation (4) can thus be viewed as using ψ k (t; x) to reweight its PDF p origin (t) = (1 -t 2 ) (d-3)/2  
empirical verification of hypotheses: 
 In this section, we provide comprehensive experimental results to verify the aforementioned hypotheses. First, we introduce the detailed setups of our experiments. They are carried out on Intel(R) Core(TM) i9-9960X CPU @ 3.10GHz with Quadro RTX 6000 GPUs.Datasets & Models. Due to the massive size of experiments, here we mainly follow the setups of the benign overfitting (Nakkiran et al., 2021) , which also present a comprehensive study of optimized DNNs through CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009) . Besides, we also include TinyImagenet-200 (Le and Yang, 2015) as a compromise between the computational efficiency and the dataset complexity. As for models, we include CNNs and ResNets as in (Nakkiran et al., 2021) . These two models represent the two typical architectures -the direction connection and the skip connection. We also notice a striking capacity gap between them in the original implementation. Therefore, we term them CNNSmall (CS) and ResNetLarge (RL), respectively, and include CNNLarge (CL), and ResNetSmall (RS) to fill the gap. The comparison between the small and large families also shows the influence of depths. As for the training process, following Nakkiran et al. (2021) , we use stochastic gradient descent (SGD) as the solver, with a batch size of 128. The input data are normalized, but not augmented. We start with the initial learning rate γ 0 = 0.1 and update it with γ t = γ 0 / √ 1 + t, where t is the epoch. Please refer to Appendix B for more experimental details.H2 can be verified independently from H1 and can provide simplifications and insights to verifying H1. We hence focus on H2 first. As stated in H2, the mean directions of different model families are consistently aligned, i.e., µ(k 1 ; x) ≈ µ(k 2 ; x). For each family F(k) where k ∈ K ′ , M = 100 models with different random seeds are trained. The population mean is then estimated throughEQUATIONThen the similarity of mean directions are naturally ( μ(k 1 ; x)) T μ(k 2 ; x). Note that when k 1 = k 2 , the 100 models are partitioned by the seeds to avoid overlapping. The results of the expectation over X are visualized in Figure 5 . It can be found that the mean directions have extremely high resemblance within each architecture, as proved by the high cosine similarities. It should be noted that with high dimensionality (e.g. d = 3072 for CIFAR), a cosine similarity close to 1 is an extremely significant result. We demonstrate this with the uniform distribution on the hypersphere in Appendix D. The observations verify the hypothesis all G k (x) almost share the same mean direction within the model architecture. This not only hold across different widths determined by k, but also holds across different depths (i.e. CS vs. CL, RS vs. RL). Therefore, the mean direction is mostly related to the certain model architecture instead of any single model f ∈ F, making it an intrinsic For instance, it can be observed that the ResNet architecture admits a closer relation between models of different depths, compared with the CNN architecture.Expectation over F vs. Conditioned on f ∈ F. As previously discussed, the spherical variance of distributions over the hypersphere can be measured by the population mean resultant length ρ(k, k; x), which, unfortunately, requires an estimation of the mean directions. This can be expensive to study for a comprehensive set K of k values. The experiments on a subset K ′ = {10, 20, 40, 80, 160} are already carried out in Figure 3 , shown as the diagonal elements. As k increases, the resultant length increases monotonically, indicating a decreasing spherical variance and a more concentrated distribution around the mean directions.The computational burden of taking the expectation over F can be alleviated by considering randomly picked f . In order to compare ρ and ρ ind , we consider the model-dependent setS(f ) = {ρ ind (f, f ; x) : x ∈ X } for each f ∈ F. Here we compute the expected Wasser- stein distance E f (1) ,f (2) ∈F (k) WD(S(f (1) ), S(f (2))). This is estimated by the M 2 distinct pairs of models. The distances of all 60 (dataset, model family) pairs lie below 0.035. Such observation suggests that after taking the expectation over X , the differences across individual models can be mitigated. Please refer to Table 1 for comprehensive results on all model families and datasets. As a consequence, it suffices to use ρ ind (f, f ) for some f ∈ F(k) to approximate ρ(k, k). This is in fact the diagonal elements of Figure 2 . A comparison between the diagonal elements ρ k,k and ρ ind (f (1) , f (2) ), f (1) , f (2) ∈ F(k) over CIFAR-10 is presented in Figure 6 . Please refer to the appendix for other datasets. ρ ind is evaluated over k ∈ K, while ρ is evaluated over k ∈ K ′ ⊂ K. It can be found that after taking the average over X , even though ρ is a little smoother than ρ ind , they are very consistent. This verifies that the resultant length increases with k ∈ K in a much more comprehensive set of models. Thus H1 is empirically verified. Given G k (x), the marginalized distribution p k (t; x) can be approximated byT k (x) = { t = u T μ(k; x)|u ∈ G k (x)}.In order to obtain the global results over test dataset X , consider the unions of different input samples Verification of the Symmetry. Saw distributions study the marginalized value t = u T µ to directly focus on the degree of concentration of the gradients. This naturally leads to rotationally symmetric distributions since the distribution on the intersection between S d-1 and the hyperplane does not affect the distribution of t. We thus carry out an empirical study of the distribution on the intersection (i.e. conditioned on t). Specifically, we train 1000 CNN models with K = 40 and seeds 1-1000 on CIFAR-10 and compute t regarding each test sample. The distribution of the first sample is visualized in Figure 7 . We partition the range of t into 10 intervals by every 10 percent of the frequency, and inspect the direction of the mean of the gradients in each interval, each direction is estimated by 100 models. If these conditional mean directions are consistent with the population mean direction, then the gradients are symmetrically distributed on each S d-2 hypersphere (R7(right)), thus verifying the rotational symmetry. We investigate the cosine similarities between the conditional and unconditional mean directions on the first 1000 samples. The 10 × 1000 similarity values have a mean and std at approximately 0.970 and 0.013 respectively. Thus the rotational symmetry is empirically verified. After verifying the hypotheses, we explore possible applications and implications of the discovered phenomena. The deep ensemble method makes use of the stochasticity to of models by incorporating the predictions from m members. While deep ensembles have been verified to be effective in improving performance, the source of such capability remains mysterious. Note that ensemble members are i.i.d. optimized models with SGD, which correspond to the population mean of our hypothesis. We thus provide another perspective in understanding the capability of deep ensembles.T k = ∪ x∈X T k (x). This is an approximation to the mixture distri- butions p k (t) = 1 |X | x∈X p k (t; x).For single models, as the model capacity increases, benign overfitting suggests that the testing loss decreases, too. We deduce that this is because the distribution of larger models becomes more concentrated, and combined with H2, the closeness to the aligned population mean is highly related to the models' testing performance. As shown in Figure 9 (a), it can be observed that the expected loss E X [L] and the marginal expectation E X [t] are highly correlated. Similarly, deep ensemble approximates the population mean much more effectively by increasing the number m of members than scaling up a single model by k. We thus scale up the deep ensemble by changing the number of ensemble members. The results are shown in Figure 9 (b). It can be found that (1) the correlation between E X [L] and E X [t] is not only significant, and (2) the correlation pattern is shared between two completely different scaling mechanisms, single model scaling and model ensembles. 
black-box attack via saliency similarity: 
 The understanding of adversarial attacks can benefit from the behaviors of the input salience of optimized models given their close relation to input gradients. We verify the aforementioned similarities through the black-box attacks, where the adversarial samples are generated from the gradients of source models while the gradients of the target models are not available. Let f (1) ∈ F(k 1 ) denote the source model and f (2) ∈ F(k 2 ) denote the target model. We define the attack rate fromEQUATIONwhich is the performance decay of f (2) when attacked by model f (1) . Small α(k 1 , k 2 ) values suggest successful attack from f (1) to f (2) . The results are shown in Figure 10 , where the attack step is set to ϵ = 0.05. In each heatmap figure, the y-axis represents the width of the source models, while the x-axis represents the width of the target models. It can be observed that larger models succeed in attacking smaller models, but the opposite is not true. To attack a large model, the gradient needs to be generated from a model of a comparable level in terms of capacity.Mean Direction Attack. According to the verified hypothesis H2, for any two individual modelsf (1) ∈ F(k 1 ), f (2) ∈ F(k 2 ) the mean directions µ(k; x) is closer to both of them than themselves, regardless of k, k 1 , k 2 .It is then suggested that using the mean gradient can perform more successful black-box attacks. We employ the mean direction μ(160, x) to attack models of different capacities, and compare the results with the attack from the largest single model (red) and the attack from the models of the identical structure (green). The results are shown in Figure 11 , where it can be observed that the mean direction of salience transfers much more successfully than single models. 
conclusions: 
 In this paper, we introduce hypotheses to explain the observations on the input salience convergence w.r.t. the model capacities. Under the same model architecture, stochastic algorithms such as SGD, result in certain distributions of optimized models. We hypothesize and use pointwise methods to verify that such distribution follows a Saw distribution with aligned population means, which is invariant from the model families. Besides, the variance of the distribution decreases as the model capacity increases, suggesting a convergence trend of the models' internal mechanism -the larger the models are, the less variant they tend to be affected by the randomness from the stochastic algorithm during the training phase. Furthermore, since the distributions converge towards the aligned population mean direction, the limiting points can be estimated by the population mean of models. Based on this, we present comprehensive experiments on the properties of the limiting model and demonstrate its capability in various domains, such as the black-box attack transferability, and the explanation of the effectiveness of deep ensembles. However, it is admitted that, due to the high computational burden, although improved from CIFAR-10/100 to TinyImagenet compared to (Nakkiran et al., 2021) , our experiments are limited to rather small datasets.Our introduced hypotheses also lead to various interesting topics. Note that the aligned mean direction stays invariant to the model families, which indicates such population mean is more related to the essence of the dataset itself rather than any single model. Leveraging this property can bring a deeper and more comprehensive understanding of the relation between data distributions and models. 
related work: 
 In terms of the convergence trend of DNNs, existing works focus on the convergence of single models throughout the training process. The parameters of DNNs have been demonstrated to converge to global minima throughout the training progress (Goodfellow et al., 2014b; Li et al., 2018; Allen-Zhu et al., 2019; Liu et al., 2020; Damian et al., 2021; Refinetti et al., 2023; Suh and Cheng, 2024) . Recent years, the studies of benign overfitting also suggest that increasing model capacities can improve the performance instead of exacerbating the overfitting issue (Bartlett et al., 2020; Nakkiran et al., 2021; Cao et al., 2022) . While the studies of input gradients span into an abundant but extremely complicated spectrum. Among them, the area that is the most related to our work is the XAI domain, where the input gradient and its variants are crucial in revealing the models' internal mechanisms (Simonyan et al., 2013; Springenberg et al., 2014; Selvaraju et al., 2017; Sundararajan et al., 2017; Adebayo et al., 2018; Shah et al., 2021) . On the other hand, the studies of the distribution of optimized models have received little attention. Such topics are slightly dipped in the efforts to demystify the source of capability of deep ensembles (Lee et al., 2015; Fort et al., 2019; Allen-Zhu and Li, 2020; Kobayashi et al., 2021; Abe et al., 2022; Ganaie et al., 2022; Theisen et al., 2023) and their implications (Lakshminarayanan et al., 2017; Geiger et al., 2020; Yang et al., 2021; Chen et al., 2023) . Thus, to our knowledge, the studies on the distribution of optimized models remain a novel topic. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Details of the experiments are discussed in both the manuscript and the appendix. The code is also provided in the supplementary materials.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The potential statistical significance lies in the variance among different ensemble members. This is studied in Table 1 .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This work does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This work does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The illustration of CIFAR-100 & TinyImagenet. In addition to the cosine similarity ρ ind (k 1 , k 2 ) shown in Figure 2 , we include more sophisticated datasets such as CIFAR-100 and TinyImagenet in Figure 12 . It is observed that the observed phenomena, where ρ ind (k 1 , k 2 ) tend to increase w.r.t. both k 1 , k 2 , hold across different datasets. It is also worth noticing that compared with the results of CIFAR-10 shown in Figure 2 , the resulting ρ ind (k 1 , k 2 ) of CIFAR-100 and TinyImagnet (Figure 12 ) shows another peak at around k ≈ 10 (see Figure 12 (e)). This is related to the deep double descent phenomenon (Nakkiran et al., 2021) , where when the complexities of the model and the dataset are comparable, the overfitting issue is at peak. For smaller such as CIFAR-10 or larger models such as ResNet, this issue becomes much less significant, as a very small k value is already sufficient for the data distribution. Also, as the complexity of the data distribution increases, the cosine similarity decreases inevitably for the same model complexity. This suggests a less concentrated distribution of the optimized models of the same capacity compared with less complex dataset. Correspondingly, the results of the expectation over the model sets for [10, 20, 40, 80, 160] are shown in Figure 13 . Softmax Activations. Apart from the normalization concern, recent work in (Wang and Wang, 2022) demonstrate the difference between the input salience of the predicted logits and probabilities. As a result, we clarify that, although we define f as the predicted logit (before softmax activations), this choice does not affect the observed increasing trend, no matter when the input salience is generated concerning the logit, probability, or the loss. The results of ρ ind (k 1 , k 2 ), generated from the saliency maps w.r.t. the predicted probability (after softmax activations), are shown in Figure 15 . It can be found that ρ ind (k 1 , k 2 ) still increases with both k 1 and k 2 . It should also be noted that, ideally, CNNSmall and CNNLarge, ResNetSmall and ResNetLarge are considered as the same families due to the same architecture. However, since widths can be adjusted independently of the depth, while the adjustment of depth inevitable affects the width, we split them. But our experiments in Section 3.1 verify that the depths do not affect the population mean of model k ∈ K ′ =This work aims to reveal the convergence trend of the distribution of model behaviors under the stochasticity of the training criterion. This does not limit the conclusion to the specific criterion described above. Distinct training criteria can lead to different distributions of trained models. However, these different distributions of trained models all satisfy the revealed trend. To verify this, we present additional experiments to investigate possible variants such as (1) depths and widths;(2) learning rates;(3) batch sizes; (4) solvers ( 5) initializations, and ( 6) other model architectures. Note that as studied in Figure 6 the manuscript, ρ ind can be a computationally efficient compromise of ρ. Therefore, we studied ρ ind in these additional experiments. Besides, there exists enumerous possible combinations of different variants of different aspects. As a result, here we only vary these settings partially since enumerating the entire grid is infeasible. The results as detailed as follows.Depths and widths. The scale of depths is not as straightforward as the width since modifying depths may change widths as well. Therefore, in the manuscript we study the influence of depth by setting -small and -large variations. Here we present additional results that study the influence of depths continuously, with 1-5 layers, each of which is followed by a max-pooling layer with stride 2. Finally, an adaptive pooling layer is appended at the end. To rule out the influence of widths (channels), all layers have the same width, determined by k. e.g., For the 4-layer scenario, the intermediate layers have widths [k, k, k, k] instead of [k, 2k, 4k, 8k] in the manuscript. The results are shown in Figure 17 . It can be found that (1) Given a fixed depth or width, the influence of the other factor is similar when scaled up.(2) Depths are slightly different from widths. Larger widths lead to higher similarities, while closer structures in depths have higher similarities. For widths (left), the similarity always increases left-to-right and top-to-bottom. But for depths (right), pairs near the diagonal have higher similarities.Batch Sizes. We investigate the influence of batch sizes, varying in {64, 128, 256, 512} The results are shown in Figure 18 . It can be observed that although different batch sizes lead to different performance (e.g. testing accuracy), the convergence trend holds in all scenarios.Learning Rates. We test different learning rates on how they affect the results. We include 1e-1, 1e-2, default, where "default" refers to the criterion used in the manuscript. As shown in Figure 19 , the revealed trend is preserved in all learning rates. It is also worth noticing that learning rates affect ResNets more than CNNs.Solvers. Apart from SGD, we include Adam, AdamW, and SGD w/ momentum. For Adam and AdamW we set the learning rate to 1e-3, while SGD w/ momentum uses a learning rate of 1e-1 with a momentum of 0.9. The results are shown in Figure 20 . Although different solvers lead to models of different performances, they all preserve the same convergence trend. Initializations. Given a training scheme and model family F(k), the training procedure leads to a distribution of trained models p(f ). When the initialization is fixed to θ, the training procedure is essentially sampling from the conditional distribution p(f |θ) instead of the unconditional distribution p(f ). We then studied the difference between the unconditional distribution p(f ) and the conditional distribution p(f |θ 0 ). We focus on two conditional distributions p(f |θ 0 ) and p(f |θ 1 ), where θ 1 represents the initializations under seed=1 and θ 0 represents those under seed=0. Other settings are identical. We thus havef 0 1 , • • • , f 0 100 ∼ p(f |θ 0 ) and f 1 1 , • • • , f 1 100 ∼ p(f |θ 1 ).The superscript indicates the initialization seeds and the subscript indicates the training seeds.First, we notice immediately that the training seeds for both θ 0 and θ 1 are 1 100. This means that ∀i,f 0 i , f 1 i differ only in initializations. We inspect (a) ρ ind (f 0 i , f 1 i ) (100 pairs) to see if they have exceptional similarity compared with (b) ρ ind (f 0 i , f 1 j ), i ̸ = j ( 100 2= 4950 pairs). Besides, within the same condition, all models only differ in terms of the orders of the training batch. We thus also inspect the similarity of all models of the same condition:(c) ρ ind (f 0 i , f 0 j ), i ̸ = j and (d) ρ ind (f 1 i , f 1 j ), i ̸ = j. Each of them has 100 2 = 4950 pairs.As demonstrated in Table 3 , (i) the comparison between (a) and (b) indicates that with different initializations, the same order of batches in the training procedure does not contribute to higher similarities. (ii) The comparison among (b)(c)(d) indicates that the same initialization indeed leads to higher similarity even though the order of batches is distinct. It should be noted that the contributions of batch orders and initializations are also affected by the number of epochs. Intuitively, more training epochs should lead to smaller contributions from the initializations but greater contributions from the batch orders.Vision Transformers. Vision Transformers (ViT) have risen in recent years as another powerful architecture for CV tasks. Here we include a brief study of ViTs to demonstrate that the discovered phenomenon holds across different architectures.Specifically, we train vision transformer (ViT) models on CIFAR-10 with varying capacities controlled by k ∈ {10, 20, 40, 80}. CIFAR-10 has an input size of 32 × 32 pixels, thus the patch size is set as 4 × 4, resulting in 8 × 8 patches. The embedding size is set to 4k, divided by k/2 heads, and we set the depth to 8. The seeds vary in 1-100 and results in 100 trained models of each k. We study the mean of the similarity ρ(k 1 , k 2 ) (i.e. the same experiments as Figure 3 in the manuscript) and the similarity of the population mean (i.e. the same as Figure 5 in the manuscript). The results are shown in Figure 21 . It can be observed that although distinct from convolutional layers, the transformer structure also has the discovered convergence trend. It can also be noted that the degree of dispersion of ViTs is much higher than CNN-based models.In conclusion, although training schemes can affect the resulting distributions of models, the influence of the model capacity stays invariant across different criteria.According to (Muller, 1959) , due to the spherical symmetry property of the zero-mean Gaussian distribution, the cosine similarity between two Gaussian variables are actually uniformly distributed over the hypersphere S d-1 in R d . 3 Therefore, the cosine similarity between two i.i.d. multivariate Gaussian tensors is essentially the inner product between two i.i.d. uniform tensors over the hypersphere. Formally, suppose X, Y to be high-dimensional i.i.d. random variables of dimension d that follow the Gaussian distribution N (0, I d×d ). The cosine similarity is Z = X T Y ∥X∥•∥Y ∥ . WLOG, it suffices to consider the scenario where Y ∥Y ∥ = e 1 = (1, 0, • • • , 0). And it is written asZ = e T i X ∥X∥ = X1 ∥X∥ = X1 √ d i=1 X 2 i = X 2 1 X 2 1 d i=1 X 2 i . Note that X 2 1 ∼ χ 2 (1), d i=2 X 2 i ∼ χ 2 (d-1).As a result,Z 2 = X 2 1 X 2 1 + d i=2 X 2 i follows a beta distribution Z 2 ∼ Beta( 1 2 , d-1 2 ). The pdf is thus f Z 2 (x) = x -1/2 (1-x) (d-3)/2B(1/2,(d-1)/2) . And then when Z > 0,f Z (x) =f Z 2 (x 2 )|2x| (7) = (x 2 ) -1/2 (1 -x 2 ) (d-3)/2 B(1/2, (d -1)/2) |2x| (8) = 2 B( 1 2 , d-1 2 ) (1 -x 2 ) (d-3)/2 (9)According to (Smith et al., 2023) , let u = (1 + x)/2,then x 2 = (2u -1) 2 . Then this can be simplified toEQUATION)This is becauseEQUATIONFinally, we are able to conclude that1+Z 2 ∼ Beta( d-1 2 , d-1 2 ), where Z is the cosine similarity between two i.i.d. d-dimensional Gaussian vectors. This result suggests that the distribution of Z will become very concentrated around 0. And this concentration exacerbates exponentially with the dimension d. Given a cosine similarity level ρ, the probability P(Z > ρ) can be extremely small, and also deceases exponentially with ρ, too. We visualize the relation between the probability P(Z > ρ) and ρ with varying dimensions d in Figure 22 . In the low-dimensional space such as d = 3, the distribution of the cosine similarity is close to uniform as humans' intuition. However, as the dimension increases, the cosine similarity is very unlikely to maintain high, as demonstrated by the curves of d = 3 × 8 × 8 = 48 (orange) and d = 3 × 32 × 32 = 3072 (green).3 This is because of the rotation-invariance. Let O ∈ R d×d be any orthonormal matrix, then after the rotation we obtain OX ∥X∥ = OX ∥OX∥ . Since OX ∼ N (0, I), too, we know that OX ∥X∥ and X ∥X∥ are from the identical distribution. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code is provided as the supplementary material in the submission. The repository will be publicized upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: There is no such risk of this work.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: This work only uses general packages such as Numpy, PyTorch, torchvision, etc.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.Γ(d/2) √ πΓ((d-1)/2) . Note that here p k (t; x) is the distribution of t = u T µ(k; x), which can be empirically estimated, the shape of the function ψ k becomes empirically accessible with varying k values. The empirical studies and verification are provided in Section 3.3.The caps are to illustrate the variance differences. Actual distributions are over the entire hypersphere. 


Decision: This paper examines the the distribution of trained deep neural networks from the perspective of input saliency maps. In particular, paper provides insightful observations regarding how models tend to resemble each other more as their capacity increases and its implications. The reviews for the paper are mostly positive. I recommend acceptance and encourage the authors to address the reviewer's comments in the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper investigates the distribution of trained deep neural networks through the lens of input saliency maps. It empirically shows that as the capacity of either of two stochastically optimized models increases, they tend to resemble each other more. Therefore, the authors hypothesize that within the same architecture (Resnet or CNN), different variants align in their input salience directions, and converge towards a shared population mean as their capacity increases. Furthermore, they propose a semi-parametric model, based on the Saw distribution, to capture this convergence trend. These findings enhance the understanding of various applications such as black-box attacks and deep ensembles.
strength_weakness: strength_weakness: 1. The primary finding is not entirely novel. Previous research has demonstrated that the performance of neural network ensembles converges to that of a single model as the model size increases [1,2]. Similarly, as the size of the models increases, the variance in predictions and disagreement among ensemble members diminishes [1].
2. Several details about the experimental setup are omitted. These include the optimizer used, learning rate, learning rate schedule, batch size, number of iterations, weight decay, and data augmentation techniques.
3. The choice of optimizers and hyperparameters can significantly alter the distribution of trained models. However, the study lacks an analysis of their findings across different optimizers, hyperparameters and sources of stochasticity.
4. The experiments are conducted in small-scale settings, which may limit the generalizability of the results.

[1] Geiger, Mario, et al. "Scaling description of generalization with number of parameters in deep learning." _Journal of Statistical Mechanics: Theory and Experiment_ 2020.2 (2020): 023401.


[2] Kobayashi, Seijin, Johannes von Oswald, and Benjamin F. Grewe. "On the reversed bias-variance tradeoff in deep ensembles." ICML, 2021.
questions: questions: 1. Have the authors conducted experiments to validate their findings across different optimizers and related hyperparameters? 
2. Stochasticity can be introduced in the training of Neural Networks in various ways. Changing the initialization through different seeds is one possibility but other possibilities include for example training on different subsets of the data, large or small batch sizes, large or small learning rates, data augmentation or label noise. Have the authors validated their findings for different sources of stochasticity? 
3. In Figure 8, why does the Large ResNet exhibit a higher test loss than the small CNN? 
4. The explanation for the effectiveness of deep ensembles suggests that they better approximate the population mean. However, this assumes that the mean corresponds to a good generalizing model. Could the authors provide evidence or an explanation to support this assumption?
5. The introduction mentions "input salience" several times without providing a definition. For better clarity and understanding, it would be helpful to include a definition of "input salience" in the introduction.
6. In section 2.1, the authors state, "We focus on $f: \mathbb{R}^d \to \mathbb{R}$ which predicts the logit specifically for the targeted class." Could the authors clarify this statement? Does it mean that the analysis after training considers only the logit corresponding to the true class?
7. In section 2.2, $\mathbf{u}_1$ and $\mathbf{u}_2$ are sometimes used to represent different samples from the same set $\mathcal{G}_k$, and sometimes from different sets.  It would be better to use a different notation to differentiate between the two cases. 
8. In Hypothesis I, line 147, the term $\mu(k;x)$ is only introduced later in the text. 
9. In line 234, could the authors clarify the definition of $\rho_{ind}(f,f;x)$ and explain its relationship with $\rho(k,k;x)$ ?
limitations: limitations: The limitations of the study are correctly addressed in the manuscript.
paper_summary: paper_summary: This paper studies the distribution of input saliency maps of trained neural networks at varying depth and width. The authors observe that as the model capacity increases, these distributions converge towards and become concentrated around a shared mean direction. The authors also state two hypotheses for the behavior of these distributions and provide empirical evidence for them.
strength_weakness: strength_weakness: - The authors do not distinguish between the stochasticity introduced by random initialization and the stochasticity introduced by optimization. In Appendix A, they discuss the similarity scores at initialization, but a more interesting ablation would be to compare fixing the initialization and then randomizing the batches with fixing the ordering of the batches but randomizing the initialization.
- Related to the above point, there is no discussion of the effect of the learning rate or batch size which should greatly affect the final results. This would help disentangle the randomness introduced by initialization vs optimization.
questions: questions: - Have the authors explored the effects of different optimization hyperparameters (e.g. learning rate, batch size) or optimization algorithms (e.g. Adam) on the convergence trends observed in the paper?
- The experiments in this paper are focused on computer vision. Do the authors expect similar conclusions to hold in other domains as well (e.g. NLP)?
limitations: limitations: The authors have adequately addressed the limitations of their work.
paper_summary: paper_summary: This paper studied the uncertainty introduced in stochastically optimized DNNs via the input saliency maps. By empirical evaluations, the authors discovered that 1) within the same model architecture, models with different capacities tend to align in terms of their population mean directions of the input salience, 2) the distributions of the optimized models concentrated more around their shared population mean as the capacity increases. These observations are very interesting and shed some light on the applications of DNNs such as black-box attacks.
strength_weakness: strength_weakness: The main complaint I have on the paper is its presentation. Although the observations quite interesting to me, I found the writing of the paper can be improved.

1. More context is needed for Figure 1.
1.1. How are the dots plotted? I assume you did something like PCA to reduce the gradient dimension to 3 and then plot them on the sphere.
1.2. You said on line 55 that the second hypothesis is illustrated in Figure 1(b), but there is no indication of model capacities in the figure.

2. Even though the authors tried to distinguish between model architecture from model family, the introduce of width and depth complicates the discussion and makes the definition less clear. It would be good to refine the presentation here. For instance, you could give a concrete example like ResNet_depth10_width10_seed0 and ResNet_depth10_width10_seed1 has the same model architecture and belong to the same model family. ResNet_depth10_width10 and ResNet_depth10_width20 has the same model architecture and belong to different model families, etc.

3. It is unclear to me what is defined on line 96.

4. The presentation of the paragraph on line 101 and Figures 2 & 3 can be improved.
4.1. Why do you use the notation $\rho_\text{ind}$? What does ind mean?
4.2. Can you give clear, different names to $\rho_\text{ind}$ and $\rho$, and make both of them display equations?
4.3. In the captions of Figures 2 & 3, can you use the $\rho$ notations and refer to their display equations, respectively?
4.4. The x and y labels in Figures 2 & 3 should be $k_1$ and $k_2$.

5. $\mu(k; x)$ is not defined in Hypothesis I.

6. Some $k_1$ should be $k_2$ on line 158 and in the display equation below. And why do you need $k_1 > k_2$ for this?
questions: questions: Besides the points I made in the Weakness section, I have one more questions. When practitioners scale their models, they usually increase both depth and width simultaneously (especially in the context of scaling law). In your experiments, you mostly studied the effect of increasing width. Could you add results in the flavor of Figures 2 & 3 where the model capacity is parametrized by both the depth and width following some scaling law relationship?
limitations: limitations: NA
paper_summary: paper_summary: This paper studies the resemblance of the functions obtained by training neural networks across various network depth and width. They do so by looking at the gradient field of the learned functions $f_1, f_2$, and defining their similarity by taking the cosine similarity between the gradients, then taking independent average over the inputs of the functions. They empirically observe that increasing the width results in higher similarity between the gradient fields, from which the deduce to properties: (1) the average gradient norm increase with width of the net, and (2) the direction of the average almost does not change when going to networks with larger width, but the variance decrease and the distribution becomes more concentrated. They also calculate the similarity measure and some properties of the gradient distribution assuming the distribution is gaussian or spherically symmetric. The major claim of. this work is summarized in hypothesis H1 and H2 on lines 145 and 160.
strength_weakness: strength_weakness: My major concerns:
(1) The claims are not backed my sufficient reasoning and relevant experiments. Note that the similarity measure is a function of the dot product of the average normalized gradients, so seeing an increasing trend on this quantity can be the result of two things: (*) the angle between the average of the normalized gradients doesn't change or gets smaller, or (**) the size of the average itself is increasing. Note that either of (*) or (**) is sufficient to happen to see an increasing value of cosine similarity that the authors have defined. However, observing an empirical increase in the value of similarity by going to larger nets, the authors conclude both of (*) and (**) at the same time, while it might be the case that (**) happens, ie.e. the size of the average is increasing fast, so that even though the direction of the averages are getting further apart, the overal dot product is still increasing. Therefore, in order to scientifically claim that both (*) , (**), one approach is that authors study the angle between the normalized averages (or dot product of NORMALIZED averages) separately. Specially because these claims are not backed by theoretical arguements.

(2) The authors discuss the 'saw distribution' and modeling the behaviour of gradients with this dsitribution assuming the actuall gradient distribution is spherically symmetric (or close to it), and assumption that seems very strong and probably incorrect for actual neural nets. Is there a particular observation for this assumption? Authors discuss some high level calculation with this distribution on line 246, but I didn't find any calculation in appendix. 

(3) the writing is vague and it does not flow. In particular, the story for the theoretical calculation with gaussian distribution is not explained, and the assumptions are not justified at all. The section regarding saw distribution needs a lot of changes to become mroe clear.

Some minor points:
- the authors claim that the fact that given networks A, B where A has larger width, they claim that the observation that the average similarity of A to B is larger than that of B to itself is 'counter-intuitive' and 'paradoxical'. This is a wrong claim and needs to be omited from the paper; the reason that it is not paradoxical is you are working with 'some measure of similarity' rather than 'distance' or 'metric', for a simple illustrative example, if you define similarity of two numbers a, b to be their product ab, then if a > b > 0, ab > b^2 but it doesn't mean that because a, b are more similar, we should conclude that a is 'closer' to b than b to itself.

- In Equation (5) you are estimating the normalized average of the gradient which is used in calculating your similarity measure, but I think the quantity of interest is the normalized average of the normalized gradients (because of the definition of cosine similarity that you use)?

minor comments:
- at H1 hypothesis in line 145:  the mean vector \mu is not defined at that point of the paper
-typo in line 158 -> should be k_1 and k_2
questions: questions: What is the conclusion for training/optimization/generalization efficiency the authors look into obtain from this observation? To me the average of normalized gradient seems not to be correlated with generalization ability of neural nets. maybe the authors can add more concluding points or remarks about their empirical observation.

- What is the difference between the message of Figure 3 and 5?

- Are you assuming the gradient distribution is gaussian in section C?
limitations: limitations: The claim needs further be backed by more delicate experiments. Furthermore, the assumptions for calculation of the behavior of the similarity measure using certain classes of distributions needs to be justified at least in some regime of training DNNs. For the rest see weaknesses above.

Comments: title: Thanks very much for your response
Response
Thank you for the response
[Continued Response]
Thank you for your valuable comments
[Continued Response]
Thank you for your positive feedback
We are happy to answer more questions if there still exist concerns for our paper.
comment: Thank you for the last clarifications; they helped me gain a better understanding of your work and results. I will update my score accordingly.
We appreciate your comments and suggestions very much! We will revise the manuscript to improve the presentation of the theoretical claim and avoid potential miscommunications. 
Regarding the technical concerns, we believe that these have been adequately addressed in our responses. Therefore, with all due respect, we sincerely hope you will reconsider the assessment of our work.

Kind regards,

Authors
Thank you for your response, I hope that the authors can improve the presentation of their theoretical claims, since currently it seems hard to follow.
I thank the authors for their detailed response. I have no more questions.
**[Different Solvers]**

We clarify that the test performance can be affected by many factors. Such differences are both beyond the scope of our work and do not affect any conclusions in our work. We elaborate on this as follows

- **The Many Factors:**

The testing performance can be affected by batch sizes (e.g. Fig. R2), data augmentations (e.g. Fig. R5), learning rates (e.g. Fig. R3), solvers (e.g. Fig. R4), and also other parameters of the solvers (e.g. weight decay, learning rate decay, momentum, etc.). The choices of these parameters create exponentially many possible combinations, and these combinations can also have different effects for different model architectures. **With different combinations, Adam and SGD can outperform each other respectively.**

- **Results in [1]:**

**[1] aims at studying the effectiveness of different solvers in testing generalizations**. Therefore, [1] pushes the limit of each solver by employing complicated and customized schemes to determine the learning rates and decays for different solvers (Sec. 4.1 & App. D). For Adam, it is also claimed in the reference [1] that

>  we find that tuning the initial learning rate and decay scheme for Adam yields significant improvements over its default settings in all cases.

As for SGD, [1] employs a multistep scheduler where both the initial learning rate $0.5$ and the learning rate decay schemes are studied extensively. It is observed in Fig. 1 of [1] that the loss of SGD decreases at distinct rates in different periods due to the scheduler scheme. If we extend one of the steps along its original slope, a SGD model that underperforms Adam model will be obtained, which verifies our previous claim that the performance comparison greatly depends on the training schemes.
This approach can also create a risk of instability for models of extremely varying capacities. Our work applies a more conservative scheme that scales the learning rate smoothly so that it can be applied to all model architectures and capacities.

Note that from the slow convergence speed in Fig. 1 of [1], we also deduce that data augmentation is applied in [1], which is not included in our experiments. It should be noted that without data augmentation, a learning rate starting at 0.5 can be very risky regarding the overfitting issue. We train CNNSmall models with $k=64$ under SGD w/ the scheme suggested in [1] w/o data augmentation. After 200 epochs, the testing accuracy is only 77.1%. This also verifies the importance of the combinations of parameters regarding the performance.

- **The Unnecessity to Push the Performance Limit:**

Different from [1], in our work, **we study Adam and SGD as two different sources of stochasticity**. Therefore, the very complicated training schemes are beyond the scope of our work. Instead, our experiments have verified that neither the empirical results nor the results are affected by such differences.


[1] Wilson, Ashia C., et al. "The marginal value of adaptive gradient methods in machine learning." Advances in neural information processing systems 30 (2017).
We appreciate the reviewer very much for your comments. This allows us to resolve the reviewer's remaining concerns further.

**[Single Models & Ensembles]**

> The hypothesis is that the population mean is consistent across different capacities for a given architecture and that both ensembling and increasing capacity are methods to approach this population mean

Yes, this understanding regarding the hypothesis about ensembles is correct.

> Could you clarify how your experiments support the conclusion that both methods (ensembling and increasing capacity) converge to the same population mean?

- **Ensemples**:

Note that for homogenuous ensembles, all enesemble members $f^{(1)},\cdots,f^{(M)}\in\mathcal{F}(k)$ are from the same family (i.e. same capacity $k$). Therefore, their mean is by definition the approximation to the population mean of the entire family. Note that the error of such an approximation is inversely controlled by the number $M$ of ensemble members, we train $M=100$ ensemble members to minimize the error as much as possible. Therefore, the following statement

> the population mean is consistent across different capacities

is verified by comparing the approximated population mean of different families $\mathcal{F}(k_1), \mathcal{F}(k_2)$. The results are shown in Fig. 5 of the manuscript. It can be observed that the cosine similarity even reaches 0.9, which is significant given the curse of dimensionality.

- **Single Models**:

Unlike ensembles, single models $f\in\mathcal{F}(k)$ approach the population mean as $k$ increases. This is demonstrated in our experiments. For example, Fig. 3 shows the expected similarity of two randomly picked models of capacity $k_1,k_2$. Larger $k$s lead to higher expected similarity between single models. This suggests a decreasing dispersion as $k$ increases. Therefore, the following statement is verified.

> both ensembling and increasing capacity are methods to approach this population mean

For ensembles, we have explained above that it approaches the population mean by definition. As for single models, they approach the population mean as $k$ increases due to the decreasing dispersion.

- **Performance**:

Now that we have established that *both ensembles and single models can approach the population mean by increasing $M$ and $k$ respectively*, we carry out experiments in Fig. 8 to demonstrate **whether the distance to the population mean is related to the performance**. Here we also purposely align the x,y-axes of the two subfigures. It is observed that even though ensembles and single models approach the population mean differently, the testing performance (y-axis) is related to the dispersion degree (x-axis) in the same pattern.


**[Clarification on Fig. R5]**

We sincerely appreciate the reviewer for pointing this out so that we can clarify the typos of this figure. First, in the right subfigure of Fig. R5, the title should be "Loss" for the left column and "Accuracy" for the right column. As for the accuracy, we clarify that the top row should be the results of **CIFAR-100**, and the bottom row should be the results of **CIFAR-10**. Therefore, the ~50% accuracy is achieved by ResNet trained on CIFAR-100 without data augmentations. As for CIFAR-10 (bottom row), a ResNet trained from scratch with data augmentation achieves an accuracy of ~90%, which is a standard result. We apologize for the typo and the confusion.

[Continued in the next block]
- **Mean Similarity**

Now we study the population mean of $p(f|\theta_0)$ and $p(f|\theta_1)$. We admit that reimplementing the results of Figure 5 for these distributions requires training $200\times|\\{10,20,40,80,160\\}|=1000$ models, which is infeasible given the limited time. We control the number of models used to approximate the population mean by $M_0,M_1\in\\{1,2,5,10,20,50,100\\}$. As the number increases, the approximation becomes closer to the population mean. And we investigate $$CosSim(\nabla\tilde{f}^0_{M_0}(x)),\nabla\tilde{f}^1_{M_1}(x))$$, where $\tilde{f}^l_M(x) = \frac{1}{M}\sum_{i=1}^Mnorm(\nabla f^l_k(x)),l=0,1$ is the average of the normed empirical mean estimated by $M$ models. This leads to a $7\times7$ heatmap, shown below as a table in Tab. R2.
As demonstrated in Fig. 8, increasing $M$ and increasing $k$ change the approximation to the population mean in a very similar way. The following results demonstrate that for models with **distinct initializations but the same batch order**, the conclusion still holds, and the values are almost the same as the similarity between unconditional CS-20 models shown in the top left corner of Fig. 5(a). 

|M0\M1|1|2|5|10|20|50|100|
|-|-|-|-|-|-|-|-|
|1|0.07|0.10|0.15|0.18|0.21|0.23|0.24|
|2|0.10|0.14|0.20|0.24|0.28|0.31|0.33|
|5|0.14|0.19|0.28|0.34|0.39|0.44|0.46|
|10|0.17|0.24|0.34|0.41|0.48|0.53|0.56|
|20|0.19|0.27|0.39|0.48|0.55|0.62|0.65|
|50|0.22|0.30|0.44|0.54|0.62|0.70|0.74|
|100|0.22|0.32|0.46|0.56|0.65|0.73|0.77|

**Table R2:** The similarity between the empirical mean of $p(f|\theta_0),p(f|\theta_1)$. $M_0,M_1$ represents the number of models used to estimate the empirical mean.
For the models with the **same initialization but different batch orders**, we can carry out the same experiments. For $p(f|\theta_0)$, we split the seeds into $1\sim50$ and $51\sim100$ to estimate the population mean, respectively. Note that for $p(f|\theta_1)$, the results are almost identical to $p(f|\theta_0)$ and thus omitted here. The results are shown in Tab. R3, where similarities are higher than Tab. R2. This verifies that initializations have higher contributions to the stochasticity compared with the batch orders.

|M0\M0|1|2|5|10|20|50|
|-|-|-|-|-|-|-|
|1|0.08|0.12|0.17|0.20|0.23|0.26|
|2|0.12|0.16|0.23|0.28|0.32|0.36|
|5|0.16|0.23|0.32|0.39|0.45|0.51|
|10|0.20|0.28|0.39|0.48|0.55|0.62|
|20|0.23|0.32|0.45|0.55|0.64|0.72|
|50|0.26|0.36|0.51|0.62|0.72|0.81|

**Table R3:** The similarity between the empirical mean of $p(f|\theta_0)$, estimated by different groups of models.

We thank the reviewer very much for pointing out this insightful factor. We will update these results in the manuscript with more comprehensive experiments and a more detailed discussion.
We appreciate the reviewer for the timely response to the rebuttal. Regarding the reviewer's remaining questions about the random initializations, we carry out additional experiments accordingly.
We start by formalizing this problem for clarity:

**Problem Setup:** Given a training scheme and model family $\mathcal{F}(k)$, the training procedure leads to a distribution of trained models $p(f)$. As we agreed, when the initialization is fixed to $\theta$, it is the training procedure is essentially sampling from the conditional distribution $p(f|\theta)$ instead of the unconditional distribution $p(f)$.

**Original Approach:** In the original rebuttal, we studied the Wasserstein distance between the unconditional distribution $p(f)$ and the conditional distribution $p(f|\theta_0)$. The latter one is achieved by fixing the initialization with `seed=0` and changing the seed from 1 to 100 in the training procedure. We found out that the distributions of $t$ is distributed very similarly with or without the conditions. 

**Comparison between Conditional Distributions:** As suggested by the reviewer, we focus on two conditional distributions $p(f|\theta_0)$ and $p(f|\theta_1)$, where $\theta_1$ represents the initializations under `seed=1`. Other settings are identical to $p(f|\theta_0)$. We thus have $f^0_1,\cdots,f^0_{100}\sim p(f|\theta_0)$ and $f^1_1,\cdots,f^1_{100}\sim p(f|\theta_1)$. The superscript indicates the initialization seeds and the subscript indicates the training seeds.

- **Individual Similarity**

First, we notice immediately that the training seeds for both $\theta_0$ and $\theta_1$ are 1~100. This means that $\forall i, f^0_i, f^1_i$ differ only in initializations. We inspect (a) $\rho_{ind}(f^0_i, f^1_i)$ (100 pairs) to see if they have exceptional similarity compared with (b) $\rho_{ind}(f^0_i, f^1_j), i\ne j$ ($\binom{100}{2}=4950$ pairs). Besides, within the same condition, all models only differ in terms of the orders of the training batch. We thus also inspect the similarity of all models of the same condition: (c) $\rho_{ind}(f^0_i,f^0_j), i\ne j$ and (d) $\rho_{ind}(f^1_i,f^1_j), i\ne j$. Each of them has $\binom{100}{2}=4950$ pairs.

As demonstrated in Tab. R1, (i) the comparison between (a)(b) indicates that with different inializations, the same order of batches in the training procedure does not contribute to higher similarities. (ii) The comparison among (b)(c)(d) indicates that the same initialization indeed leads to higher similarity even though the order of batches is distinct. This corresponds to the reviewer's intuition. It should be noted that the contributions of batch orders and initializations are also affected by the number of epochs. Intuitively, more training epochs should lead to smaller contributions from the initializations but greater contributions from the batch orders. We will add experiments in the manuscript to explore these factors.

Items|(a) Diff. Init. Same Order|(b) Diff. Init, Diff. Order|(c) Same Init. $\theta_0$ Diff Order|(d) Same Init. $\theta_1$ Diff Order
-|-|-|-|-|
\# of pairs|100|4095|4095|4095|
mean of $\rho_{ind}$|0.0758|0.0753|0.0879|0.0855|
std. of $\rho_{ind}$|0.0038|0.0037|0.0042|0.0048|

**Table R1:** The comparison between the similarities between single models of different criteria.

[Continued in the next block]
Thank you for the provided clarifications and additional experiments. However, I still have the following concerns:

1. Based on your explanation, if I understand correctly, the hypothesis is that the population mean is consistent across different capacities for a given architecture and that both ensembling and increasing capacity are methods to approach this population mean (that is why ensembling works). Could you clarify how your experiments support the conclusion that both methods (ensembling and increasing capacity) converge to the same population mean? Is it only based on the fact that they share similar scaling for the test Loss? Have the authors tried to measure similarities between the ensemble output and single but larger models? 

1. I still have some concerns regarding the experimental setup. For instance, in the newly provided experiments, specifically in Figure R5 (top right, there seems to be a mismatch between the plot title and the y-axis label) it appears that the maximum test accuracy on CIFAR-10 with ResNet, across all widths, is capped at 0.5, which is highly unusual. Similarly, in the experiments comparing Adam and SGD, it seems that Adam achieves better test error compared to SGD, which is generally unexpected see for example [1]. Could you provide further clarifications? 

[1] Wilson, Ashia C., et al. "The marginal value of adaptive gradient methods in machine learning." Advances in neural information processing systems 30 (2017).
I thank the authors for addressing my questions and concerns. I still have a few remaining questions but in response to the rebuttal, I have raised my score.

> It should be noted that this is actually equivalent to uniformly sampling from the original distributions. Therefore, we expect an almost identical distribution compared with the models used in the manuscript, where seeds are determined at the beginning of all steps.

I agree that each individual run has the same distribution, but the runs are now correlated. As an extreme, if I match both the initialization and batch-sampling random seeds, the runs will again also be identically distributed but will be fully correlated. My question was specifically whether the differences between the saliency maps of models at finite size were caused by the stochasticity from initialization, or from the randomness in the batches. For example, it seems a priori possible that for each initialization, the distribution has a *different mean* (depending on the initialization) and a *smaller variance,* but then when you randomize the initialization, these distributions are mixed together creating an overall distribution with a new mean and a *larger variance.*

> [Other Sources of Stochasticity (W2, Q1)]

I thank the reviewer for the additional experiments, and they are a valuable contribution to the overall story. It appears that the batch size makes little to no difference, but the learning rate and optimization algorithm make a fairly large difference (which is to be expected). Indeed, the first observation suggests that the randomness is almost entirely due to the random initialization. I think it would be extremely interesting to include a cosine similarity plot (width x width) where the first distribution uses a fixed initialization (seed=0) and a random ordering of the batches, and the second distribution uses another fixed initialization (seed=1) and a random ordering of the batches. This could also be flipped so that you fix two orderings of the batches and then randomize over the initializations for each distribution. This would help disentangle these two effects.
Dear Reviewers,

Thanks very much for the time and effort that you have dedicated to reviewing our paper. We greatly appreciate your constructive comments and hope that our response adequately addresses your concerns.

Should you have any further questions or confusion, we are more than willing to provide additional clarifications. Thank you again for your valuable insights.

Best regards,

Authors