The Implicit Bias of Gradient Descent toward Collaboration between Layers: A Dynamic Analysis of Multilayer Perceptions
Authors: Zheng Wang
Geyong Min
Wenjie Ruan
Venue: NeurIPS 2024 poster (2024)
Keywords: Implicit Bias
Adversarial Robustness
Dynamic Analysis
Primary area: learning_theory

Abstract: The implicit bias of gradient descent has long been considered the primary mechanism explaining the superior generalization of over-parameterized neural networks without overfitting, even when the training error is zero. However, the implicit bias toward adversarial robustness has rarely been considered in the research community, although it is crucial for the trustworthiness of machine learning models. To fill this gap, in this paper, we explore whether consecutive layers collaborate to strengthen adversarial robustness during gradient descent. By quantifying this collaboration between layers using our proposed concept, co-correlation, we demonstrate a monotonically increasing trend in co-correlation, which implies a decreasing trend in adversarial robustness during gradient descent. Additionally, we observe different behaviours between narrow and wide neural networks during gradient descent. We conducted extensive experiments that verified our proposed theorems.

The Implicit Bias of Gradient Descent toward Collaboration between Layers: A Dynamic Analysis of Multilayer Perceptions
Zheng Wang
Geyong Min
Wenjie Ruan" School of Computer Science USTC rvjieQustc.edu.c en
Department of Computer Science University of Exeter {zw360; M.inn) Min] ac. uk
Abstract
The implicit bias of gradient descent has long been considered the primary mecha- nism explaining the superior generalization of werrparameterized neural networks without overfitting, even when the training error is zero. However, the implicit bias toward adversarial robustness has rarely been considered in the research commu- nity, although is crucial for the trustworthiness of machine learning models. To fill this gap, in this paper, we explore whether layers in neural networks collabo- rate to strengthen adversarial robustness during gradient descent. By quantifying this collaboration between layers using our proposed concept, co-correlation, we demonstrate monotonically increasing trend in co-correlation, which implies a decreasing trend in adversarial robustness during gradient descent. Additionally, we observe different behaviours between narrow and wide neural networks during gradient descent. We conducted extensive experiments that verified our proposed theorems.
Introduction
As Artificial Intelligence (AT) has been widely applied in many industrial sectors, understanding the theoretical properties behind modern machine learning models is important, especially for neural networks due to their black-box nature. One such property is implicit bias, stemming from the phenomenon where over-parameterized neural networks, trained in gradient descent manner, often exhibit great generalization without over-fitting This implicit bias of gradient descent often explained as steering neural networks towards solutions characterized by max-margin 3 23
Another intriguing phenomenon is the existence of adversarial examples - imperceptible perturba- tions of inputs that alter classification results. Apart from existing work on attacks algorithms for generating adversarial examples [5) 39
37 defences against attacks, e.g., adversarial training 25 36)
and distillation (28), and verification 40 34] to identify safe regions guaranteeing the absence of ad- versarial examples, recently some works are aiming at theoretical understanding behind adversarial robustness. However, a principled way to comprehend the core contributors to the vulnerability of neural networks, especially theoretical understanding of their relation to generalization capabilities, remains fragmented. This fragmentation is largely due to the intricate nature of neural networks, where robustness is interconnected with many factors spanning across input data distribution [33 14], sampling complexity [7] optimization techniques (25), weight initialization strategies [41), and model capacity and architectures 32
I& 35 Not to mention that only very few works can address
both generalization and adversarial robustness in uniform framework. One such research by Frei et al. 13 investigates the implicit bias concerning both generalization and adversarial robustness,
*Correponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

asserting that while implicit bias leads to solutions with improved generalization, it results in weaker adversarial robustness. However, this work ignores the architectural factor of neural networks and is hard to generalize to neural networks with more than two layers.
Inspired by this work and to investigate whether neural network layers collaborate against adversarial examples during training, this paper first adopts the novel concept of Dirichlet evergy--oiginatiigg from Partial Differential Equations (PDEs) assess the variability of function [i evaluate the adversarial robustness of neural networks. We then theoretically demonstrate that Dirichlet energy serves as an effective measure of adversarial risk. By decomposing the Dirichlet energy across the entire neural network into its constituent layers, we can quantify the interactions between adjacent layers concerning adversarial robustness. We term this interaction collaboration correlation (co (co-correlation) and find that this metric reflects 'alignments' in feature selection between neigh- bouring layers. Furthermore, we conduct dynamic analysis of co-correlation two-layer MI.Ps, demonstrating with high probability a monotonic increase under gradient descent, which indicates diminishing adversarial robustness. Additionally, our experiments show that two-layer MLPs with small widths tend to enhance their performance through strengthened co-correlation, pattern not observed in wide two-layer MLPs. Our key contributions in this paper can be summarized as follows:
To the best of our knowledge, this work the first to study the implicit bias of interaction between layers. We have quantified the interactions between adjacent layers and theoretically demonstrated that co-correlation between layers strengthens during gradient descent neural networks under mild assumptions, suggesting that it not only fails to collaborate against adversarial perturbations but may even hinder resistance to them during gradient descent.
2. We demonstrate how neural networks with a large width differ in behaviour from the neural network of small width, showing that MLPs with larger widths exhibit more resistance to increased co-correlation and, therefore, are more adversarial robust, which is complementary for the work by Dohmatob and Bietti [8]
3. Extensive experiments have been conducted to validate our proposed framework. By controlling the weight initialization, a perspective also suggested by Zhu et al. [41], we challenge the argument, as proposed by Huang et al. [18) that a wide neural network does not necessarily lead to better adversarial robustness, through the lens of cross-layer collaboration.
2 Related Works
2.1 Implicit Bias of Gradient Descent
The mystery of over-paramen wwerparramteriiee neural network trained with gradient descent manner hardly over-fitting has long been studied. Chizat and Bach (3) study the two-layer neural network with infinite width and homogeneous activations, showing that gradient flow can be characterized as a max-margin classifier on exponentially tailed losses. Lyu et al. [24). Sarussi et al. [29 study the two-layer Leaky ReIU neural network on linearly separable data and claim that networks converge to max-margin linear predictor by gradient descent manner. Frei et al. [12 confirms those claims on high-dimensional nearly orthogonal data.
Lyu and Li [23]. Ji and Telgarsky [19 claims that the homogeneous neural networks with exponentially-taile classification losses converge to KKT point of a maximum-margin problem. Kunin et al. [21] extend these results to a more boarder family of hoaieehomoggeeous neural networks. more recent research [13] considers both generalization and robustness for two-layer ReLU neural networks, arguing that gradient descent is biased towards solutions that generalise well but are more vulnerable against adversarial examples, even the neural network is highly parameterzzd
2.2
Theoretical Investigation of Adversarial Robustness
Since the phenomenon of adversarial examples has been discovered [15] various works have been proposed to understand the theoretical fundamentals behind it, especially for neural networks. Some researchers argue that the source of adversarial vulnerability comes from the input data [33 g 31
30 27.. The more recent researches investigate the fragility of neural networks from an architectural perspective. Simon-Gabriel et al. [32 study the vulnerability of feed-forwant neural
26 14

---

networks measured by Lp norm of the loss function w.r.t. input data, suggesting that the vulnerability increases with input dimension independent of model structures. Daniely and Shacham [ examined the ReI.U neural network characterized by decreasing dimensions at each layer. They asserted that the manifestation of adversarial robustness is intrinsically tied to the network's s architecture, which contrasts with the propositions put forth by Simon-Gabriel et al. [2] Bubeck et al. [2] expanded the findings of Daniely and Shacham's 's work on two-layer neural networks from an "under-complete case" scenario to an "over-complete" one where the number of neurons surpasses the input dimension. They further broadened the conclusions drawn by Daniely and Shacham [6] and Bubeck et al. 2 to encompass Deep ReLU networks, hinting at crucial role played by bottleneck layers in these networks. Zhu et al. [41], instead of merely considering random weights as the standard configuration, conducted comprehensive analysis of the effects of weight initialization adversarial robustness.
Unlike previous studies that focus solely on the overall assessment of neural networks while over- looking layer interactions, our research examines the synergistic involvement between layers within neural networks, taking into account both weight initialization and optimization.
3 Preliminary
3.1 General Setting
We follow the binary classification setting where the input data X S R2, with the label y S {0. 1}. Given data set D = 1= drawn from an unknown probability measure P on X' x 3 and the neural network f X' x x - RR where è denotes the set of parameters, our objective is to optimize f by updating the weights with gradient descent method such that it can predict the label accurately. The prediction result is shown in Equation (I)
sig(S (a)) > 0.5 00 s0g((ff (x)())) S 0.5,
Ypred
(1)
where sig is the sigmoid function, i.e., sig(x) 1/(1+ ")). We use Binary Cross-Entropy (BCE)
loss in Equation (2) as our loss function. For simplicity, we denote u, = f(x,, W), E [n]] as the output of the neural network for input œi, where [n] [k No 1K < n).
L(S.v)
Msig(us), = n
ye (og(si((())) (1 yr) log(1 sig(u))) (2)
3.2 Neural Networks and Adversarial Risk
Our exploration starts from a basic linear model, then to Multilayer Perceptrons (MLP). We provide the proof for both linear and 2-layer MLPs, which can be extended to MLPs with more layers. They are defined as
flneear (z, W) a (Wx)
(3a)
fmip( W) = (o
(3b)
where æ E X is the input data, W E mxd denotes the linear transformation, m is the width of the networks. a denotes the element-wise activation functions.
We follow the initialization setting in [9) where a is randomly initialized and fixed from a binary selection of (- } Additionally, we introduce slightly different setting for the weights W.
which are randomly initialized following the normal distribution [O,, - ) with g O, instead of ((0, However, in our experiment, different settings of q are considered, including the scenario
that q S O.
Generalization ability is one of the most important concepts for machine learning models. Classifiers with better generalization power indicate lower natural risk for unseen data. Given data points (xx v) P and classifier f. the natural risk is defined
R()
(L((()) ([()))))
E (m.n)~
When it comes to 0-1 loss, the natural risk becomes the probability of misclassification for unseen data points.
3

---

Similar to natural risk, the adversarial risk is defined as the probability of misclassification under adversarial perturbations shown in Definition 3.1 Definition 3.1 (Adversarial Risk). Given data points v) ~ P. and a perturbation e within a norm-ball, i.e.,
B, where > O indicates the L2-norm budget for perturbations. The adversarial risk for neural network on loss function L defined as
{|| rr),
(5)
Reob (f.r)
E (æ.y)~
sup L(f(2 + +)) LeEB,
(6)
Since adversarial perturbations are almost invisible to human eyes, we expect to be quite small.
3.3 Dirichlet Energy
The concept of Dirichlet energy, originating from Partial Differential Equations (PDEs), serves tool to assess the variability of a function [11]. However, as argued by Dohmatob and Bietti [8], serves more effective measure of adversarial robustness than the Lipschitz constant. We extend this concept to mappings to make it more suitable for multi-dimensional problems, which is formally defined in Definition 3.2
Definition 3.2 (Dirichlet Energy for Mappings), Let convex set 2 S R and 2g C R=, Given a
differentiable mapping Φ Z1 + 22, the Dirichlet Energy w.r.t. Pa is defined
(Φ)) (IJ
V [| (x))
where || ||2 refers the operator norm and (xx) denotes the Jacobian matrix of Φ w.r.t its input æ.
To be clear, we use || to denote the L2-norm for vectors and operator norm for matrices throughout our analysis. This concept can be extended to layers with ReLU activation by extending the derivative to (x) = [(u2 where 1/20) is the indicator function. With the concepts defined, we now establish the relationship between adversarial risk and Dirichlet energy.
4 Measure Adversarial Risk by Dirichlet Energy
In this section, we establish the relationship between the adversarial risks and Dirichlet energy, showing that Dirichlet Energy is a proper measurement for adversarial risk. Dohmatob and Bietti [8] only compares the Dirichlet energy with the Lipschitz constant. We. instead, illustrate that Dirichlet energy is proper representation of the gap between natural risk and adversarial risk.
Theorem 4.1. Given data points (x, y) P and Pz, the relationship between adversarial risk and Dirichlet energy for classifier f with differentiable loss function L is shown as
(f & +L)++r((I(f).
(8)
where O is the largest perturbation budget and G(I(f)) Vennra [ IT indicating the Dirichlet energy of the classifier on loss L.
As is shown, the Dirichlet energy is a proper representation of the gap between generalization and adversarial risk which indicates the adversarial robustness. The proof relies on the linear approximation of L(f). The detailed proof is shown in Appendix A
We also empirically show that the Dirichlet energy for classifier f. i.e., G(f) instead of G(L(f)), the part that influences the adversarial robustness, as is shown in Figure where we compare the level of Dirichlet energy for ff with the robust accuracy of the classifier attacked by Auto-aftare (5).
Since the Dirichlet energy defined in Equation 7 can used to measure the variability of mappings, it follows that this metric could be employed to evaluate the adversarial robustness of individual layers or modules within neural networks. Consequently, this allows for an assessment of whether there is collaboration between these components in terms of adversarial robustness.

---

4.1 Measure Correlation between Layers
To evaluate the collective impact of neighbouring layers on the adversarial robustness, we regard the neural network as compounded separate mappings, i.e., Φ o y. By calculating the Dirichlet energy for each mapping (6()) and (())) and comparing it with the overall Dirichlet energy for the whole classifier ( v)), we can quantify the collaboration between these two mappings, and we term this quantification co-correlation which defined in 4.2
Definition 4.2 (Co-correlation). Let y 21 Z2 and Φ Z2 23 be successive mappings, where both 2 and 2g are convex. Given input æ ~ Pa. the collaboration correlation (co-correlation)
defined as
Ba plppppp (2)) 11)) 1
A
20.
(En~Pa [1) (x))1))
To avoid any confusion, the output of the function (x))) for a given input æ. Consequently, in (p((p) represents the outputs, while in (2)), it represents the mapping. Since for each Pe, have
to interchangeably denote both the function yp 21 22 and
Jasp (a) (a) || [| Jip (a)]l] < [[ [a){(=)]22
We expect 0 < lo. S 1. The concept of co-correlation can he explained by the feature alignment of layers in neural networks.
Interpretation of co-correlation. Intuitively, co-correlation can be viewed as feature 'alignment' between layers. We illustrate this intuition with 2-layer neural network with linear activation functions, i.e.,
- (({(x)) W2W1æ,
(10)
where W, and W2 weight matrices, and (p(x)) Wi represents the feature selection in layer-1, while (bp)) Wey represents in layer-2. By Definition 4.2 the co-correlation between & and is
|W2Wi1l l|2
2Φ.e
(11)
Wella W1ll2
where - denotes the operator norm. Let S.p arg maxij=]la=1 |||a=||2, representing the set of input features that maximize the Lz-norm of their corresponding outputs. Similarly, let
SΦ arg ma |222|22 Assume lΦ.p 1, there exists Sp such that [<((]]la E Sp- This implies that the maximal output in terms of the Lg-norm at the first layer can lead the output at the second layer to reach its maximum. In other words, in terms of maximizing outputs, layer aligns with layer 2. We can thus regard co-correlation as the degree of aligmment in feature selection between adjacent layers within the context of output maximization. In the non-linear case, the weight matrices are replaced by the Jacohians of the respective layers, which can be equivalently viewed as weight matrices that vary based on their inputs.
<p(x)
Definition 4.3 (Other Related Statistics). Given the same assumptions in Definition 4.2 we define
the linear correlation between (+2) and 1| l|2 as
Em [1) I| (2))) (E (o(())1 (E (x))))111
Pb.u A
(12)
where 60 xp(x) shows that &p follows the distribution of (px) Ps. It obvious that 0 S Pot.yp < 1 and when Pø.v = L, 1o ( and (1))21) are linear correlated.
The mean and variance for [ J| (z: l|2 are defined
HΦ.p 2 EenP (+))l) [1)(()))2)
(13)
and
var Var [1)) (=)22))
(14)
5

---

Uhiignt 8-0 Sye 84 3a0
Epoch
Epoch
Robust Acc.
(b) G())
(c) Linear Model
(d) MLP
For all MLPs considered, lower value of Dirichlet Energy (Figure 1b corresponds to larger robust accuracy on test-set attacked by no-n Auto-attack with e of the co-correlation for linear and MLPs is shown in Figure Ic and [d The architectures of neural networks are defined Equation 3b
Figure 1:
0.5. The dynamics
The parameters to control the weight initialization in
Assumption 5.T 5.1 is set to q 0.25
Remark 4.4 (Linear Correlation for Lo-norm of the Jacobian). We have Pa.p iff there exist t E IR such that
P( = [1 1,
implying that Po.p certainly can be used to assess the linear correlations. It reduces to Pearson correlation coefficient [4] when the mean of both random variables equals zero.
Now we give the theorem that binds them together.
Theorem 4.5 (Robustness Decomposition). Given the same assumption in Definition 4.2 the mea- surement for overall adversarial robustness can be decomposed as
G( c 40) (Rawr P (1
= 20.50 (1+ var var
G(O)6())
(15)
1
The proof is straightforward from the definition. Based on this theorem, we have conducted exper- iments with various linear models and 2-layer MLPs. These experiments demonstrate that, apart from the co-correlation, all other statistics are negligible, as shown in Figure 5 in the Appendix D Consequently, our analysis primarily focuses the co-correlation g.
5
On Dynamics of Co- Co-Correlation
5.1 Dynamics for Linear Model
Before delving into analysis, state assumptions explicitly.
Assumption 5.1. We assume that each element wiyj in the weight matrix (0)) Ruxd at ini- tialization follows the Gaussian distribution N (00, ,f ). with >> 0. Additionally, each element a,,r [m] in a is randomly selected from the set Vml Fin }. and fixed during training.
Assumption 5.2. We assume that for each (æ,, y:) E D, i E [n]. x; is L2 norm bounded such that [ for all i [r]].
Since we only assume bounded inputs and specific weight initialization method, compared to 12, our approach can be easily extended to MLPs with more than two
existing works 23 19
layers.
Now let us focus on the co-correlation defined in Equation and show the dynamics of @ø.c for each step gradient descent. We start from the linear model described Equation (3a). Given binary classification problem and the linear model described. Our first theorem demonstrates that co-correlation lø..pp gradually accumulates throughout gradient descent optimization. Despite the simplicity of the linear model, it effectively exhibits most of the properties under consideration.

---

Since the weights are updated by the gradient descent, the update of the weights at step t N is
n n (ui sig(U;
(t, rjar
where 7) denotes the learning rate. Therefore, the dynamics of the weights be expressed as W(t) @ x² (t),
(16)
where @ denotes the Kronecker product and z(t) is the enror weighted imput such that
(t)
- sig(u,(t)) xi-
(17)
Given the weight updates, we demonstrate that the dynamics of the co-correlation for the linear model, denoted as do, (t), exhibit an increasing trend, particularly during the initial steps of training when most predictions are still essentially random.
Theorem 5.3 (Dynamics of the Co-correlation for Linear Model), Given the linear model defined in
Equation Bap and training dataset D {(r; y:))²= Assume that assumptions) 55. and 5.2 hold for W and a. The gradient descent applied to the weights results in the dynamics of the co-correlation being expressed as:
êa,w = -/)(())a,
(18)
and with high probability,
C(t) T(T) (1- (1)()(())))) +
1 1719
(19)
where the v(t) is the dominate eigenvector for ((t)W(+)?
When m is sufficiently large, and during the initial steps of the optimization process, Ø(T), E [t] are quite similar to each other in terms of cosine similarity, implying an acute angle each other, which leads Σ = (t) 2 0. As result, we can conclude that C(t) 2
The detailed proof can be found in the Appendix B This assertion also corroborated by the results of our experiments shown in Figure Ic Even though Theorem 5.3 is based on linear model, the essential properties are universally applicable and can be summarized as follows:
Property 1.
The co-correlation @ develops during the initial stages of training and becomes saturated as training progresses to later stages.
Property 2. The speed of the accumulation of co-correlation @a.w is inversely related to the operator norm of weights (1
5.1 an increase in network Under the same weight initialization conditions specified Assumption 5.1 width leads to decrease in the Lg- Lg-norm of the weight, consequently causing substantial rise in co-correlation.
5.2 Dynamics for MLP Model
For the non-linear case, we make certain assumptions regarding activation functions. Assumption 5.4. The derivative of the activation function o' o'(x:) in non-linear neural networks is bounded by M. In other words, we have |o [o'(x)] S M.
With Assumption 5.4. Theorem 5.3 can be extended to MLPs, and the two properties still hold. Different from the linear model, the update of weights for non-linear MLP defined in (3b) is
1 Σ (vi sig(u,)) (w, xz)x;
Aw, 1j,
(20)
where d'w,, 2x)) is the derivative of activation function w.r.t. its input. Hence,
AW
T m

---

Max Maxi
Auwmagge pood
Bottlonckk
Botteneck
Buthwrcck
Benteneck
Epach
Epoch
(a) ResNets
(b) A1-A1
(c) B1-B2
Figure 2: The dynamics of co-correlation for ResNet50 and WRN50 under different way of partition.
The way of partition is illustrated in Figure 2a Al-A2 and B1-B2 represent the separations that distinguish the head and separately.
where for each r E [m].
Σ d'w (1) sig(u:))an 2.. n
1
As is shown in Equation 20 a is not the dominant eigenvector for AWAW" due to the difference of ă,, r [m]] from ž. Thus to average out the difference, we define the weighted sum of inputs for the non-linear model as
3. (1)
Σ ==1
(t, sig(u,())))
(21)
where (æ:; yi) E 'D are realized r.v., and x ~ Pm. Given wij t.i.d. N(O, , for each element in (()), we have w00) ~ N(0, 42- Ia). Furthermore, given that W(t) is calculated from W(O), it evident that each row in W (t), denoted as wtt), constitutes random variable. As result, both of (w) T æ) and of (w=(t) TTxa xx)) are bounded random variables that are contingent upon W(0). Hence,
we define (1,(1, x)) as
(t, x) A Ew(o) (t) x)o (w(t)" "a)aaa)))).)))
Since, Pa. (t, x)) is still nv. contingent to and does the H. (t). Now we show the dynamics of co-correlation for two-layer MLP defined Equation (3b).
Theorem 5.5. (Dynamics of the Co-correlation for MLP) Given the MLP defined in Equation with training dataset D {(,,, u)))
X such that Pa- Assume that Assumption 5.1 and
5.2 hold for W and a, and Assumption 5.4 holds for the activation function. we have
Da (t) = (((())), (t).
With high probability,
C(t) T= (1 a (o(T)aT((t)))) v(t)) Ba-p, C (T)
max (0(1)) mm
1 mm
Ex [D())
where
D(t) = diaglo' (w) ... (t) x)),
and v(t) denotes the dominant eigenvector for W (L))(L)), with of is defined in Equation 21 Similar to the Theorem3.33 when m is sufficiently large, and during the initial steps of the optimization where the werohtheighted imputs 2!! (7), [t] do significantly fluctuate, we have that C(t)
The detailed proof is in Appendix In Theorem 5.5 T. serves similar purpose as 2 for the linear model. addition, considers the influence of the activation function. Property still holds for the MLPs, and Property 2 extends to Eg-P, ( = |Joow
(x) [

---

Epoch
Epoch
Epoch
Epoch
(a) width-32
(b) width-512
(c) width-2048
(d) width-8192
Figure 3: The dynamic of co-correlation under different set-up of weight initialization. MLP network
defined in Equation
with ReLU activation function for width 32, 512, 2048 and 8192 are included.
Experiments
To estimate the co-correlation more efficiently and in parallel, we employ the Power Iteration algorithm [10] in conjunction with Functorch [17] The corresponding pscudo-code is presented in the Appendix
We verify our proposed theorem on linear and MLP models on the MNIST dataset 22 The width of hidden layers varied from 2 to 2 with weights initialized via Gaussian distribution N(0, 142 where y was set to values ranging from 0.25 to -0.15. We train both the linear and MLP for 50 epochs with batch size of 512 using the SGD optimizer with learning rate of 0.003. In addition, we also conduct the experiment on more complex ResNet50 [16] and WRN50 38) both models, we opted for default random weight initialization and used the Adam optimizer with learning rate of 0.0005 on CIFAR1O [0)]. To calculate the co-correlation and other statistics, we cover the entire testset. We consider using L2 Auto-attack [5] with 0.5 for all MLPs trained. The experiments were executed on a Nvidia RTX3090 GPU, using Python 3.9.7 and PyToch 1.9.1. The code for the
experiment is available at https: pettp:///gtthut github....... com cow.c///uurreeagg200/oooooooooo correlaaioooooooiiono.o..oo..ooi
6.1 Empirical Evidence for Proposed Theorem
Figure presents a comparison between the robust accuracy and the Dirichlet energy G(f) across all trained MLPs. As observed in Figure Ia and Tb models with lower levels of Dirichlet energy G(f) tend to exhibit higher robust accuracy, suggesting that Dirichlet energy is an effective representation of adversarial robustness. Another noteworthy finding is that wider neural networks, with the same level of weight initialization, demonstrate improved adversarial robustness.
Figure 1 also depicts the dynamic behaviour of shallow neural networks with a weight initialization parameter of 9 0.25. As is shown in Figure Ic and [d] the co-correlation & increases throughout training. Except for narrow widths like 2 and 2 the majority of networks demonstrate an upward trend. This trend, however, flattens for non-lineen models, suggesting potentially stronger adversarial robustness due to the non-linearity of the activation function introduced in MIPs.
Figure 2 shows the dynamics of the co-correlation ResNet50 and Wide ResNet50. Both networks are trained on CIFARIO using the Adam optimizer. We divide them by the pattern of A1-A2 and B1- B2, as shown in Figure 2a The co-correlation outcomes for these divisions displayed in Figure 2b and Figure 2c it shows that even with the Adam optimizer, without specific weight initialization considerations, there is noticeable rise in co-correlation.
6.2 The Impact of Width and Weight Initialization
Figure 3 illustrates the co-correlation dynamics under varying for both linear and MLPs with widths of 32, 512, 2,048, and 891. The figure highlights that proposed theorems' assumption of q > 0 quite tight, as all trajectories with q < 0 remain flat throughout training. We can also observe that the speed of accumulation significantly increases with larger network widths.
Figure [equre/disppe displays the accuracy on testset and co-correlation for both linear and MLPs as heat-maps.
Each cell in the heat-map represents a trained network. From Figure Figurel4d. we observe that the best performance and robustness are shown by the MLPs with the largest widths (width = 8192) and the smallest weight initializations (g -0.15). And when alter the weight initialization to

---

21 3/8 53 AU3
513 21
175
(a) Linear Model/Acc.
(b) Linear Model/g
MLP ReLU/Acc.
(d) MLP
Figure 4: Accuracy and Co-correlation under different initialization and width. Figure 4a and 4b
show the heat map for linear model, and Figure and id for MIP ReLU.
control the co-correlation, making it increase from 0.25 to 0,83, the accuracy declines accordingly from 0.93 to 0.68. On the contrary, for another extreme of models with width of 16, enhanced performance is accompanied by increased co-correlation. Consequently, an interesting conclusion can be drawn about the diverse behaviour of neural networks with small and larger width. Gradient descent tends to enhance the training of neural networks with smaller width by fostering co-correlation among layers, which is intrinsically brittle. However, wide networks are trained with less reliance interlayer correlation, resulting in inherently more robust models.
7 Conclusion and Limitation
Our work investigates the implicit bias of gradient descent toward adversarial robustness from the perspective of collaboration between layers. By adapting Dirichlet energy to estimate the adversarial robustness of neural networks' individual components, characterized the collaboration behaviour between consecutive layers and identified two fundamental properties for dynamics of the co-correlation. The first property shows that the co-correlation for MLPs will build up during gradient descent under mild assumptions for weight initialization. The second property shows that the speed of accumulation for co-correlation is inversely related the operator norm of Jacobian for the corresponding sub-mott modules. In addition, we observed that networks with small widths tend to foster co-correlation among layers improve performance, whereas wide networks' performance improvement does not heavily rely on establishing such co-correlation. Future research can expand upon this by examining the effects of increased network depth and more sophisticated structures on the observed phenomena.
Limitation Our work can be easily extended to multi-layer neural networks since we only assume that the inputs are bounded by the I---norm. However, like many theoretical studies, extending our approach to more complex models is challenging. It remains unknown whether complex models exhibit the same behaviors.
References
[1] R. Bhattacharjee, S. Jha, and K. Chaudhuri. Sample complexity of robust linear classification on separated data. In International Conference on Machine Learning, pages 884-893. PMLR, 2021.
[2] S. Bubeck, Y. Cherapanamjeri, G. Gidel, and R. Tachet des Combes. A single gradient step finds adversarial examples on random two-layers neural networks. Advances in Neural Information Processing Systems, 34:
2021.
[3] L. Chizat and F. Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, pages -338-1338. PMLR, 2020.
[4] Cohen, Huang, J. Chen, J. Benesty, J. Benesty, Chen, Y. Huang, and I. Cohen, Pearson correlation coefficient, Noise reduction in speech processing, pages 1-4, 2009.
10

---

[5] F. Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- free attacks. In International conference on machine learning, pages 2206-2216 PMLR, 2020.
[6] A. Daniely and H. Shacham. Most relu networks suffer from 1 adversarial perturbations. Advances in Neural Information Processing Systems, 33:6629-6636, 2020.
[7] E. Dobriban, H. Hassani, D. Hong, and A. Robey. Provable tradeoffs in adversarially robust classification. arXiv preprint arXiv:2006.05161, 2020.
[8] E. Dohmatob and A. Bietti. On the (non-) robustness of two-layer neural networks in different learning regimes, arXiv preprint arXiv:2203. 11864, 2022.
[9] S. Du, X. Zhai, B. Poczos, and A. Singh. Gradient descent provably optimizes over- parameterized neural networks. arXiv preprint arXiv: 2018.
[10] F. Epperson. An introduction numerical methods and analysis. John Wiley & Sons, 2021.
[11] L. C. Evans. Partial differential equations, volume 19. American Mathematical Society, 2022.
[12] S. Frei, G. Vardi, P. L. Bartlett, N. Srebro, and W. Hu. Implicit bias in leaky relu networks trained on high- ditbeeeinnii data, arXiv preprint arXiv:2210.00082, 2022.
[13] S. Frei, G. Vardi, P. L. Bartlett, and N. Srebro. The double-edged sword of implicit bias: Generalization vs. robustness in relu networks. arXiv preprint arXiv:2303.01456, 2023.
[14] Gilmer, L. Metz, F. Faghri, S. Schoenholz, M. Raghu, M. Wattenberg, and I. Goodfellow. Adversarial spheres. arXiv preprint arXiv:1801.00 2018.
[15] J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and hamessinn adversarial examples. arXiv preprint arXiv: 2014.
[16) K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[17] R. 7. Horace He. functorch: Jax-like composable function transforms for pytorch. https //github. com/pyttorch/Euncorreh 2021.
[18] H. Huang. Y. Wang, S. Erfani, Q. Gu, J. Bailey, and X. Ma. Exploring architectural ingredients of adversarially robust deep neural networks. Advances in Neural Information Processing Systems, 34:5545-5555 2021.
[19] Z. Ji and M. Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems, 33:17176-17186 2020.
[20] A. Krizhevsky. Learning multiple layers of features from tiny images. Master's 's thesis, University of Tront, 2009.
[21] D. Kunin, A. Yamamura, C. Ma, and S. Ganguli. The asymmetric maximum margin bias of quasi-homogeneous neural networks. arXiv preprint arXiv:2210.03820 2022.
[22] Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://aww..ecc..coom/xxx///////
2, 2010.
[23] K. Lyu and J. Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv: 2019.
[24] K. Lyu, Z. Li, R. Wang, and S. Arora. Gradient descent on two-layer nets: Margin maximization and simplicity bias. Advances in Neural Information Processing Systems, 34:12978-12299 2021.
[25] A. Madry, A. Makelov, I.. Schmidt, D. Tsipras, and Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint vrXiv:1706.060S: 2017.

---

[26] S. Mahloujifar, D. I. Diochnos, and M. Mahmoody. The curse of concentration in robust learning: Evasion and poisoning attacks from concentration of measure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4536 4543. Association for the Advancement of Artificial Intelligence (AAAT), 2019.
[27] Y. Min, L. Chen, and A. Karbasi. The curious case adversarially robust models: More data can help, double descend, or hurt generalization. In Uncertainty in Artificial Intelligence, pages 129-139. PMLR, 2021.
[28] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pages 582-597. IEEE, 2016.
[29] R. Sarussi, A. Brutzkus, and A. Globerson. Towards understanding learning in neural networks with linear teachers. International Conference on Machine Learning, pages 9313-9331 PMLR, 2021.
[30] L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and A. Madry. Adversarially robust generaliza- tion requires more data. Advances in neural information processing systems, 31, 2018.
[31] A. Shafahi, W. R. Huang, C. Studer, S. Feizi, and T. Goldstein. Are adversarial examples inevitable? arXiv preprint arXiv: vrXiiv/809.000 004,02104, 2018.
[32] C.-J. Simon-Gabriel, Y. Ollivier, L.. Bottou, B. Schölkopf, and D. Lopez-Paz. First-order adversarial vulnerability of neural networks and input dimension. In International conference on machine learning. pages 5809-5817. PMLR, 2019.
[33] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. arXiv preprint arXiv: /805: 12152, 2018.
[34] F. Wang, P. Xu, W. Ruan, and X. Huang. Towards verifying the geometric robustness of large-scale neural networks. arXiv preprint arXiv:2301.12456, 2023.
[35] Z. Wang and W. Ruan. Understanding adversarial robustness of vision transformers via cauchy problem. In Machine Learning and Knowledge Discovery in Databases: European Conference, PKDD 2022, Grenoble, France, September 19-23, 2022, Proceedings, Part III, pages 562-577. Springer, 2023.
[36] X. Yin and W. Ruan. Boosting adversarial training via fisher-rao norm-based regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24544 24553, 2024.
[37] X. Yin, W. Ruan, and Fieldsend. Dimba: discretely masked black-box attack in single object tracking. Machine Learning. 170(4):105-1172, 2024.
[38] S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint arXtv:1605.07146. 2016.
[39] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472-7482. PMLR, 2019.
[40] T. Zhang, W. Ruan, and J. E. Fieldsend. Proa: A probabilistic robustness assessment against functional perturbations. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 154-170. Springer, 2022.
[41] Z. Zhu, F. Liu, G. Chrysos, and V. Cevher. Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization). Advances in Neural Information Processing Systems, 35:36094-36107, 2022.
12

---

A The Proof for Theorem 4.1
Suppose that the loss function differentiable w.r.t Given data points (x, y) P and x Pa. we have
R (f.r)
E sup ((f(x +), (a.y)~~ cEB,
(22)
sup (z)( + sup L(f(x + +),,)) sup [(f(2,))) eEB,
(23)
(..g)~
LEEB,
sEB,
E [L((()) + (m.
E (a.
sup (L(S(z - ((f v)))
(24)
LrE
E [L(( + V (..)~PP
E
sup
(V, J
(25)
(a. LeEB,
[[ +r
(m.)~p 1 J (x))
(26)
(..p~P
where y L is the gradients of I, w.r.t J. y indicates that it is a row vector. J)(x:) is the Jacobian matrix classifier f.
B The Proof of Theorem
Before formally proving the theorem, provide some lemmas which are useful for our proof.
Lemma B.1 (Dynamic of Weights for Linear Model). Given gradiens descent to optimize the weights, the dynamic of the weights at step is
(tt) a @ (t),
(27)
Proof. Given the step size 77, the update for rth row of the weight matrix is
al af(W, a.)
Aw,
(28)
Ou:
Ow,
i=1
mar
(yi - sig(u;))
(29)
where sig(-) denotes the sigmoid function. Hence, for weight matrix W, have
Awf
AW
(30)
Aw T
m
i= (ys sig(u;) (u:) an?
: 1 Li=1 (us sig(u:)) zFt
(31)
After replacing with
1 Σ
T
(yr sig(us))
(32)
im
13

---

we have
" AW =
(33)
Om x
11a @ "t"
(34)
Since | 1, we also have | ((t), is
1. Therefore, say that the dynamics of weights, ie.
W(t) a 2 at()).
(35)
Lemma B.2 (The Concentration for L2 Norm of Gaussian v..... Let In be identical matrix of size Given Gaussian random vectors 21. 2n such that 2, is.d [(0, ,, >
0, Vi in]. With probability at least ô, we have several conclusions: 1. Average of the norm
+ +0(V
8 8log 8log(2n/0)) (2n/o) ml+ 1+29
(36)
n
Average of the square norm
O
8log(2/6)
8 8log(2/6)
+00 }
+
max
(37)
m2
m +20
1=1
3. Square root of the average of the square
+V8 m+224n
8 (log((20))
12,
me +
(38)
Proof. We first prove the concentration property for the average of the norm. Since Vm i.dd N (0, 1), E [n],r [m]. we have that m 1+22 2 ~ (1). Denote the sub-Exponenticn distribution as SEV22 ai,, we have m 22
1 SE(4, 4). Hence vi E [na].
P
1 Σ
m 12222
2
26 2 5
2exp
me? 5
CE (0.1)
(39)
2exp
me
221
Therefore, have VE > 0
v m 1-12 -1
2 <
|| || /
max max(c,)2))
(40)
m
m
( (- mc²
< 2 exp
(41)
8
Equation 40 is because of the fact that given c fixed, Vo 0 have
I
c 122 - 11 2 maxfc, o²)
(42)
14

---

Hence,
1
n
n1
m
23
2
V
1/2
2
(43)
11
m
PP
Wm
(44)
P
ll 2
+
2c
(45)
2ml
(- mc²
2n exp
(46)
Equivalently, with probability at least 1 - 6,
8
2n
m
|21|12 -1
S
log
(47)
8
2n
log
(48)
π2
m
m2
m
8
To prove the second convergence, we starts from the fact that
m
713
12
m
ml+22
(49)
n
m
24
YUML
(1), Vi E [m], E [m]. similar to Equation (Vml+20zi,)) )² lid
Since
39
we have
m2
2xxp
nm 8
EE (0, 1)
Vml+20 )² 2- 2c 5
(50)
nm
2exp
e 2 1
Equivalently, with probability at least 1 - 6,
log
EE (0, 1)
m
24
|2
mn
(51)
72
log
2
6
mn
8
2
2
n
m229
24
max
mn
log
8*
mn
log
(52)
Now prove the third concentration, with similar trick of the first concentration, we have
m
n
m
ΣΣ
1212222.r7 2
26
SP
V )- 1 2 maxte, e²}
Tum
Tim
(53)
$ 2exp
2
(54)
Therefore,
P
m2
1/ 2
< 2exp
mm 2
(55)
15

---

with probability at least 8, we have
1
n
8
2
m
log
(56)
Y1
mL
8
13
1
8
2
12
log
(57)
172
m
mn
6
Now, we prove the theorem 5.3
Given the linear model and training dataset { yi )
Assume that each Wa, i.d.d N(0, m242)) ),q 0 and is randomly initialized subject the constraint ||a|| = and fixed during training. Hence, with high probability, we have
(()(tt [(11)
(1 (1-(v(t)aa)) + +o
da.W 2 n0a.w
(58)
7729
Proof. We first show that the derivative of co-correlation is
d
ea.w (t) = dt la. (o())t (t)
(59)
d F. [|aT (
(60)
di E. (1)
d ||aTW (t)l2
(61)
di (D]l2
(t) 2
ddlaa d (O1l//d
(I (
(62)
(aTW(1
Ca.w (1)
(dlla=W(D)))
alw (1)BB dt
(63)
W()
dt
by the law of derivatives for inner product of matrices and the eigenvalue of matrix (t)) we have
dl WU)E di
a aT(WWWW( + ((I)(()))
(64)
W ( =(O) + W(O)(()))00)
(65)
dl
where vtt) is the dominant eigenvector for W (t) Equation 60 is because Jw (z) W and
||a|| by assumption. Since Equation (60) do not depend on 4r we can safely drop the Expectation
E,, as is shown in Equation (61). Equation 63 is because ||a'' WI| < ||a||]|[[|3 || || By
Lemma B.I the weight matrix W(I) at training step can approximated
((t) = W(0) + Σ AT.)).
(66)
16

---

then
WOW(0"=a00"0 W(0)T= Σ AW()T)
(67)
= x0 W(O)T +1 Σ a 8 3(T)
(68)
a
W(O)T= naa " Σ a({))
(69)
a (0) +1
a 2 x (t)
a" 2 Σ x(T)
(70)
= () + Σ x(1) aa""
(71)
similarity,
(())((()) = (W()) - Σ AW(T) =
(72)
-(W(0) - na @ Σ x (T) a" 2 x(1)
(73)
= ((O) +1 Σ ;(T)) (()) aa" T.
(74)
Hence
a T W
W(112
(75)
2
dt
dt
( (())(((())
n (aTo(1))
+a'(())(aa'aa (()"W())(()a''(()
(76)
I is the main part of our theorem. And for ②, it can be bounded as
2 = (a aTv(()u(t)) T (())a(t))
(77)
(u sig(ui) a (tt))(()))) (()a,, (((()))))""((())),
(78)
n
Bi sig(u))) (a- (a -1 | | (0)
(79)
71
(0)a.l1
(80)
n
Now since wij ~ N(O, in m ),9 > 0 and each way is independent with each other, for given 2y 1, 1,...,1,, n2 we have
(wi (0) xi
|| 12 1m ) N (0.
(0)a,
(81)
m2I+4
Wm (0)T xz
17

---

where x 1 1 by assumption. By the lemma B.2 with high probability,
[0)) m + +o(
(ogg(2n/))
(82)
1.429
Therefore,
2 O
(83)
172
with high probability, and it comes to our conclusion.
C The Proof of Theorem 5.5
To prove the Theorem 5.3 the following lemma is crucial.
Lemma C.1 (Dynamic of Weights for A Nonlinear Model for Initial Steps). Given gradient descent optimize the weights, the dynamic of the weights at the initial step for the non-linear model is
a1 a (t)
W(t)
(84)
am T (1)
where
n
(t) ar 72
(v= sig(( (w) (t)a;)a; re [m].
(85)
Proof. Similar to the linear case, the update for the rth row for the weight matrix is
1) " Ol Of(W,..)
Aw,
(86)
n Dr
ow.
1
na,
y sig(us) o' (w) xx)
(87)
Hence that the update of the weight matrix W(t) is
Aw?
AW
(88)
Aw?
a1 n ΣR
(y: sig(u,)) d''(('')) xz)x
=1 1
(89)
am n
Ei=1
(yr sig(u=)) of o' x
a 1
1
:
(90)
am T
Hence conclusion.
Lemma C.2 (Concentration for Weighted Sum and Square Root Scalar for Bounded Variables).
Suppose X, X,- 14 E [,, u]. Vi [m] is independent identically distributed with mean O. Given
vector D E Rm | = 1, and a such that a, E {-1,1)). E [mi) we have concentration inequality
m
mc²
X,
T vy 2 < 2exp
911)
2(u - 1)2
18

---

Proof. We first show the moment generation function of the r.v. we would like to estimate.
}
t (u - 1)2 }
E
exp
U
a" -a'u)/)))) vnl
- exp
(92)
V
2m
Now, we prove this inequality.
E
exp
{1
1
ar
1 Xx
a m')
}
E
exp
t
Ur - Σ 0r1
m
m
r=1
(93)
v
Ea,
X, - 11) b
exp
(94)
m
u22
(u 1)2 )
exp
(95)
2m
exp
u
1)²
2
(96)
2m
r=1
exp
(u
1)²
(97)
2m
where 2 O. Hence by Chernoff's bound, we have that
IP
Σ
m
ar Ur X,
_at
a² ma vu
2e)
5 2 inf
exp
2m
12 (u - 1)33
(98)
m
mm
120
exp(te}
= 2 exp {
me²
(99)
2(u - 1)2
Here, prove the theorem 5.5 Given shallow neural network, with high probability we have
oow (t)
(a
a oT)(r)aa o1)) (()))) (()
(100)
E.p
((L)
+ max {ol
(101)
vm
m99
Proof. Because of activation function depends on the input æ, we cannot drop the expectation Eg. Hence, we have
d
ea,w (t)
dt 9a, w (t)
(102)
d Eal (0II)+
(103)
di (
(t)w(t)
EEd a (t))/ E.d Eed|D(t) (t) 11
(104)
2
Ex
la ((O)
E. (L)) (1)
(t.w(t)
alla" D(O) (() di
(105)
2E (())
where D(t) = diag(o' (uo ax), o' (w.T. And for ReLu activation function, o' d'(wTT .)) = fw #203- We assume that the activation does not change at an infinitely small change of f, implying
((t) = 0. Hence, have
19

---

(()((1) = a ( (
dt
+ WWWNWWWWWDDDDDWWWWWWWO
=a" = (DO))OOW()) D()) + DDWOOWODDD))
(T)
= 2 22 D(t) T=1
(1)
Om Tn (8) D(t)a
(m)
01 a1of(1)
D()
W(O))D(1)a.
am am (t)
And similarly,
d ((D(W)(O = o0((0) (DLDDWOOWWOOO DIOWWWO) )
21)
dt
(T)
= =2 (t)(t))
(a)zi(t)
(t)
am T (T)
(t)
+ 22 (1)D(( +20²(1)D((
W(0) D())o()
O. T (1)
I can be
= 7) Σ
t
m
aoo ( (T)" (T) ) ow (t) (t)2))),(())
(Σ
(106)
T=1
0, (w, (T) x0'' m:)] (2r - sig(us))x
(107)
m
La (1) m:)) (y sag(
(108)
Since a, E {- Jm vm } and the derivative activation function o' is bounded by M by assumption,
we have that
Fw(()) (E
m
(w, (w
(w. xx)) (t:2)) 26)
(109)
= -Pw(o)
1 Σ
d' (w, (t)² (t) (w) (t) x.) a;(t;x) 20 2
2314
m
(110)
r=1
20

---

where
0sl1, x²) = Bvrco) (((w(()+)))) x))'(w(())
(111)
Equivalently, with probability at least 1 - 6, we have
m
2M
Lo (ww - (x(t:)) S
log
(112)
m
Let
(1) 71
at(, st)((
(113)
(1))
(yi sig(us(()))
(114)
11
Hence, with probability at least -,
n
a: (T, (T sig((()))))) + +0 [ log (
1j
(115)
m
log 2
(t, sig(u))))))
(116)
1 Σ
W (T) Th t0 ( 1
log 2
2 (1) +
log a(t)
(117)
m
1) Σ
a (7)
T # (1) + 0
(V/ (V11 m
log
2 5
(T(T)= a()()
(118)
+ +0
log
2
2 (T) (7) x(t)
0 (1)
1
log a(1) a(1)T z(t).
2
(119)
m
8
m
Since
4n (T, 2)) sig(ui
| (t)ll2
I M²
(120)
n
(n (T, x) (y) sig(u.(()))
y/1 (()
| <
< 1,
(121)
Un sig(un uun (t))
T
we have that for at least 8,
x (T) 2 (t)))(++)) +0
( m
2 8)
= 17
log
(122)
Similarly,
(1)a, (m) mn)) (us sig sig(u:))ar?
(w,
(123)
(t) ey (t)a,o' (w, x)oo (w (t)² (t)² 2:) (vs sig(u:))zs
(124)
21

---

Let a' = =/ma E 1 and by the conclusion of lemma
C.2
(Σ
m
Pwro))
u
Σ ur (t)a,as(t;x) 2r)
(125)
(1 Vm
m
-Pw(o)
(,,,(t)o'ww,,(t) 2)0 (w, (w) x.) a² ((t)ai(t:z)) 2c)
(126)
mc²
< 22exp
(127)
2M
Equivalently, with probability at least 1 - 8,
2M 2
a
- - (a((())a((;;2))
S
log
(128)
Hence,
x 1 (
log 2
1' a
(129)
m
a T -l +0
- log a(t) )
(130)
a d''(())² (V
log 2 |
a" (("=(()=' (1) + (V log
2 ((t)
(131)
a" d''((Ta" (()))
(132)
1=1
+ +0
(V
- m
log 2 207)) (v log कه (7) 5(1) (7) o 0 log 2
a(t) T ((t).
(133)
We have that for at least 1 - 8.
1000+0011 ( log 2 5
1 = 7 Σ a" (() (()))
(134)
For 2, have
(T)
2 = D()
:
W()) D()ja
(135)
am G (t)
a - (()
(a (0)
am W (0) (t)) l
(136)
< M2 (01 (0)
am (( ||
(137)
< M²
Σ
a [ 12
(138)
22

---

where
1
o' (w, (0)) (v) sig(u:)) :
[ 12
(139)
1 1=1
σ d'(w,(0) (10))())) 21)(11 sig(ui)
(W(0)a1
a())x,))
(140)
72
d' d'( (n) sig(un))
(0)" (0)))))))))))))) 2 sig(4)))
(W(0)x1
(())a,)
(141)
OS...SM2
M [0)a))
(142)
n
Similar to the proof of the linear case, and accords to Lemma B.2 least 1 - 6,
we have that with probability at
8 (lgg(/1)) me + +o(v
[31
(143)
m +-2
n
Therefore, with high probability,
2
S M
I[
(144)
M
[0),,,
(145)
==1
O
(146)
m
Follow the exact same procedure. With high probability, we have
a a (t)
2 = (T)(((())))))))))))))))))))())))
D()))D(())())
(147)
am (t)
$ ||((t² D
(a1 (0)
am (o) 11
(148)
M²
Σ
(149)
O
(150)
We put all the information together, with high probability, we have
oo (t)
--O'+ + 2'+2202 2-22) 2')
ea.oo (1) Z
(151)
E. D(O
oow (t)
Ex D(L) (1)
77
a T o(Taa (7)
(152)
+ max {o
o
(153)
m
m
23

---

1,00000
099660
0.025
std/
10.020
0 0.99986
.99799
0.010
0.99972
0.005
0.99965
Epoch
Epoch
(a) Relative Std
(b) Linear Correlation
Figure 5: Linear Correlation & Relative Std. The linear correlation and the standard deviation over the mean are given for all MLPs with the initialization parameter 0.25.
tot AQa.w tot AQa.a-W
0.3
0.6
0.3
width(2")
Figure 6: The accumulation of co-correlation under different width. X-axis denotes the width of the neural network, and the line different color shows the result under different setting weight initialization. The lower plot is the result for linear model and the result for shallow ReLU is at the top.
D Extra Experiment Results
Figure 5 summarizes the statistics of linear correlation and relative variability, illustrating that co-correlation is the dominant factor in the decomposition outlined in Theorem 4.5
Figure 6 shows the accumulated change in co-correlation during training across varying network widths. Each line, represented by different colours, corresponds to unique weight initialization setting q. A clear inference from the figure the positive relationship between the accumulated co-correlation change and network width under the same q the wider the network, the greater the accumulated co-correlation change. Moreover, as we shift the weight initialization setting from 0.25, the accumulation of co-correlation also increases. These observations align
q 0. 15 with our Theorem 5.3 and 15.5 indicating that the increase co-correlation inversely proportional to [t]l2,

---

E Algorithm
We the Power Iteration algorithm [T]] along with Functorch [1 to compute co-correlation and other relevant statistics. This combination assists in approximating the Lz-norm of the Jacobian for the layers under consideration. The algorithm estimate the related statistics including co- correlation are outlined in Algorithm
The power iteration used to obtain the Lg-norm of the Jacobian summarized in Algorithm
[
Algorithm Estimation of Related Statistics
Input:
Input dataset and modules Φ so for the given blocks in sequential {æ;, = 1,.., NJ
Output: Approximation of co-correlations 2a.ps linear correlation Po.p. mean Po. Mup and variance
oo The of the Jacobian of the module w.r.t. their inputs
1: Computation of Jacobin of || 11p(())))))) (2;) [I)((((();)) for {a,, [n]]} w.r.t their inputs
1. E [[ (x):))))))))) (mn,))la
2: Do.p
3: Po.p
( Ein, [ (+ Ei (=(a)11))**
4: Pi
N ΣE [
5: 0o
<
Algorithm 2 Power Iteration Based Computation
Input: Input dataset and module for the given sequential blocks {æ;; = N..,,N 0
Output: Approximation of co-correlations dd.p. linear correlation Pø.p. mean Maa Mip and variance
Ge Ty of the Jacobian the module w.r.t. their inputs
1: COMPUTATION OF (a,)( FOR {x,, E [n]]
2: while S N₂ do
3: sample randomly u ([o,
4: u < u/|/u|/2
5: generate jacobian-vector product function jep(:;
6: generating vector-jacobian product function ejp(:;; 2;
7:
( <- None
8:
while > 10 do
9:
v <- jup (u; 2z;)
10:
< ujp(v: (U;
11:
- u/ uul2
f- v/ //|//|2
13:
|| (mx) [
[xppp()))))
N None then
(an, (an, 113 (ac,
(mx)))
err
16:
end if
Jg 112dd (.,((x))) new
18: end while
end while
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction have clearly stated the claims made, including the contributions made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: A separate section "Discussion and Limitations" has been designed and can be seen in the end of this paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification: The complete proofs are provided in the appendix. Theorems and Lemmas are properly referenced in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The algorithm and detailed experiment settings are provided.
Guidelines: The code is provided in the supplemental material.
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code is uploaded in github, including detailed README file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "cippscc/puullc//uuu /utee//ggudddeeeedSuuuubmiibii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details can be checked in Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The detailed methods can be found in Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
28

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper introduces the configuration of the computer in Experiment section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::///haarppp..cco/o/a///hhht
Answer: [Yes]
Justification: We have read the NeurlPS Code of Ethics and confirm that the research conforms in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed. This is theoretical paper to understand existing MI methods.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the code package or dataset properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "CgrkhmO7gh", "reviewer": "Reviewer_KqkQ", "summary": "In this work, the authors study the tradeoff between generalization and adversarial robustness from the perspective of collaboration between the layers of a neural network (NN). They adapt the concept of Dirichlet energy to analyze the robustness of different layers in the network. Decomposing this across layers allows them to quantify the alignment between feature selection between consecutive layers as collaboration correlation (or co-correlation). They show that the co-correlation increases while training, which leads to lower adversarial robustness, MLPs with larger widths exhibit more resistance to increased co-correlation and present supporting experimental results.", "questions": "Please see the weaknesses section.", "limitations": "There are no societal negative impacts of this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This work presents a novel *view* on the tradeoff between adversarial robustness and generalization of NNs trained with gradient descent.", "weaknesses": "- While the results are somewhat interesting, it is unclear what the implications of these results are, since many of the observations in this work have been noted in prior work, and are not surprising. It is also not clear if the proposed concept of co-correlation can be used can be used for other applications .\n\n- There are several typos/grammatical errors/formatting issues that should be corrected:\n    - In line 56, ‘adversarial’ should be ‘adversarially’.\n    - The last part of lines 64-65 should be rephrased.\n    - In line 77, ‘digger’ should be ‘dig’.\n    - The heading of section 3 should be ‘Preliminaries’.\n    - Some of the notations (e.g., $sig$) should be corrected (to sig).\n    - In line 127, ‘Slimier’ should be ‘Similar’.\n    - The formatting of Theorem 4.1 (lines 155-156) should be corrected.\n    - In line 179, $J\\mathbf{\\psi}$ should be $J_{\\mathbf{\\psi}}$.\n    - In Ass. 5.2, L-2 should be $L_2$.\n    - In Prop. 1, ‘flattened’ should be ‘saturated’.\n    - A reference is missing in line 275.\n    - Line 297 should be rephrased.\n    - Missing ‘.’ in line 311.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and careful review. We appreciate the opportunity to clarify and expand on our work. Our rebuttals are as follows:\n\n## Theoretical Implications\nAlthough it may not be immediately apparent in the paper, the implications of our work are significant. From a theoretical perspective on implicit bias research, our approach is distinct in assuming only L-2 norm-bounded inputs, Gaussian weight initialization, and bounded derivatives of the activation functions. This differs from most works on implicit bias, such as those by Lyu et al. [23], Frei et al. [12], Kunin et al. [20], and Frei et al. [13], where proofs are typically provided for 2-layer neural networks and rely on strong assumptions on inputs, making it difficult to generalize to deep neural networks. The extension to deeper neural networks in these works may require the same strong assumptions on features between layers, which is unrealistic. However, due to our assumptions on inputs needing only to be $L_2$-norm-bounded, our work can be more easily extended to deep networks. This extension was not included in the paper due to page limitations. However, we have provided experiments with ResNet50 and Wide-ResNet50 on CIFAR10 in this rebuttal and the later version of this paper.\n\nIn the figures shown in the anonymous links: [Illustration of the divide for ResNet50](https://i.imgur.com/1OPWAuO.png), we divided the ResNets in two ways and used the Adam optimizer with a learning rate of 0.003 to track the dynamics of co-correlations. The experiment results are available at [Dynamic of co-correlation of the divide as $A_1$, $A_2$](https://i.imgur.com/icGOSOK.png) and [Dynamic of co-correlation of the divide as $B_1$, $B_2$](https://i.imgur.com/EfTfHx4.png). As is shown in both figures, the co-correlation increases over the training epochs, although the upward trend for the separation of $A_1$ and $A_2$ is not strictly monotonic.\n\nAdditionally, our work offers a possible explanation for the observation that wider neural networks are more resilient to adversarial attacks. This complements recent findings by Frei et al. [13], who suggest that implicit bias leads to non-robust solutions. Our work suggests that these non-robust solutions can be mitigated by increasing the network width.\n\n## Practical Implications\nPractically, our findings imply that designing an isolation layer to decouple the correlation between layers could enhance adversarial robustness. However, due to page limitations, we concentrate on the theoretic analysis of whether there is a collaboration between layers against adversarial examples. This aspect of how to reduce the co-correlation will be explored in future works.\n\nWe greatly appreciate your careful review. All typographical and grammatical errors will be corrected in a subsequent version. And all the suggestions will be considered. Thank you for your understanding and feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ufpGWhDCwL", "reviewer": "Reviewer_G4as", "summary": "This work focuses on studying the implicit bias of correlation between intermittent layers of neural nets and uses this metric to analyze the adversarial robustness of networks in under and over parameterized regimes. The authors further use these findings to suggest that in the under parametrized case, gradient descent enhances accuracy by forcing co-correlation between layers which in turns leads to worse adversarial robustness. However, in the overparameterized case, models depend less on the interlayer correlations and thus result in more robust models. \n\nThe authors approach the problem by using the Dirichlet energy to bound the adversarial risk. They then introduce the co-correlation metric, designed to describe the alignment of layers in a linear 2-layer neural net. Further, the authors manage to bound the Dirichlet energy using the co-correlation metric and additional terms but provide experimental evidence to indicate that main term changing throughout training in under over/under parameterized scenarios is the co-correlation metric. \n\nThe paper then focuses on studying the dynamics of co-correlation change during training on the linear model with emphasis on layer width and initialization, describing how co-correlation increase happens mainly during the early stages of training. Finally, the authors provide experimental results and extend their work to general MLPs.", "questions": "1) To be clear the authors don’t provide any theoretical explanation for why the other terms used for the upper bound in Thm 4.5 are negligible ? And on that note, am I correct in assuming that the point of Fig 4 in appendix is to show how the variance/mean and linear correlation don’t change much with model width ? If so, shouldn't you consider different initializations as well ? \n\n2) On equation 19, the authors make the claim that the first term in the lower bound is large early on ? Wouldn’t this also be true during the late stage of training since the learning rate is smaller and $ \\tilde{x} $ stabilizes ? With enough training epochs, I expect that to be the case right ? \n\n3) Could authors provide further explanation on the potential impact of weight regularization and co-correlation given Property 2 ? \n\n4) The comment made on line 267, wouldn’t it be the opposite that $ \\tilde{x}_* $ fluctuates more during the early stages of training with higher learning rate, as mentioned previously for Thm 4.5 ?", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Overall I found the work to be quite interesting, well written and described. The ideas were well explained, assumptions are mostly reasonable and the paper is easy to understand for me. Additionally I like seeing works done in the interlayer interactions of deepnets and believe such observations and results could have further experimental and theoretical implications. \n\nI am also quite interested in the results regarding implication of initialization and am somewhat surprised by the experimental results describing the importance of initialization.", "weaknesses": "1) While I understand that the focus of the work is on theory, I would like to see 1-2 more experimental results, potentially with CIFAR-10 or larger networks (maybe not MLP) just to confirm the results and give a better perspective. \n\n2) The authors often make claims regarding over-under parametrization vs width of layers. I find it a bit confusing whether some of the claims are considering the number of network parameters or the actual width of layers and would like to see some more distinction. For example in experimental cases, it's possible to try out narrower but deeper networks with the same number of parameters as wider ones to isolate the width effect. I understand that this could be hard in such controlled cases and small MLP’s of course.\n\nI will address some more of my concerns in the question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable feedback and careful review, you indeed checked the details of the paper and pinpointed the flaws. It is highly valued feedback. My rebuttals are as follows: \n\n## More experiments\n### Experiments on ResNet50 and Wide-Resnet50\nWe conducted additional experiments on larger and deeper networks, specifically ResNet50 and Wide-ResNet50 on CIFAR10. These experiments will be included in the paper in a later version. In the figures shown in the anonymous links: [Illustration of the divide for ResNet50](https://i.imgur.com/1OPWAuO.png), we divided the ResNets in two ways and used the Adam optimizer with a learning rate of 0.003 to track the dynamics of co-correlations. The experiment results are available at [Dynamic of co-correlation of the divide as $A_1$, $A_2$](https://i.imgur.com/icGOSOK.png) and [Dynamic of co-correlation of the divide as $B_1$, $B_2$](https://i.imgur.com/EfTfHx4.png). As shown in both figures, the co-correlation increases over the training epochs, although the upward trend for the separation of $A_1$ and $A_2$ is not strictly monotonic.\n\n### About varying depth and width  \nYour suggestion to conduct experiments on networks with varying depth and width, but the same number of total parameters, is quite interesting. However, in this paper, we focus on the theoretical analysis of whether there is collaboration between layers against adversarial examples. How different architectures of neural networks impact the co-correlation, especially with varying depth and width, will be considered in future work. \n\n### Negligible of other terms on different weight initialization\nYes, Fig. 4 in the appendix demonstrates that the other terms are negligible. Additional experiments with different initializations are available at [Relative std](https://i.imgur.com/dLOv5yv.png) and [Linear correlation](https://i.imgur.com/GzsDl3G.png). We consider the MLP with a width of 512 with varying weight initializations. Similar to Fig. 4 in the appendix, the first and second figure show the $\\frac{var^{\\frac{1}{2}}}{\\mu}$ and $\\rho$ in Eq. (15). Empirically, it shows that $\\Big(1 + \\frac{var_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}}{\\mu^2_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}}\\Big)^{\\frac{1}{2}} \\rho_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}$ is quite close to 1, making it negligible in the Eq (15). \n\n## About Eq.(19) and training stage\nAccording to Eq. (19), the first term of Eq. (19) is\n$$\n\\frac{\\sum_{\\tau=1}^{t}\\widetilde{\\boldsymbol{x}}(\\tau)^T\\widetilde{\\boldsymbol{x}}(t)}{\\Vert W(t) \\Vert_2^2}\\cdot \\Big(1 - \\big(\\boldsymbol{v}(t)^T\\boldsymbol{a}\\Big)^2\\Big)\n$$\nThe exact value of this term is complex; however, the sign of this term depends on $\\sum_{\\tau=1}^{t}\\widetilde{\\boldsymbol{x}}(\\tau)^T\\widetilde{\\boldsymbol{x}}(t)$, since other terms are positive, as explained in Thm 5.3. According to Eq. (17), $\\widetilde{\\boldsymbol{x}}$ is a weighted average of all inputs, where the weight is the difference between the ground truth $y_i$ and the predicted likelihood $sig(u_i(t))$.\n\nAt the initial stage with a small learning rate, $\\widetilde{\\boldsymbol{x}}(\\tau_1)$ is similar to $\\widetilde{\\boldsymbol{x}}(\\tau_2)$ for $\\tau_1, \\tau_2 \\in[t]$. More specifically, let $t$ be 5 epochs. Since the model hasn't learned much, the predicted likelihood $sig(u_i(t))$ of $\\tau_1 = 1$ and $\\tau_2 = 4$ are all similar to each other; therefore, $\\widetilde{\\boldsymbol{x}}(\\tau_1)$ and $\\widetilde{\\boldsymbol{x}}(\\tau_2)$ are similar in terms of cosine similarity.\n\nHowever, in the later stages, as the model learns more from the data, most of the predictions become correct, and $y_i - \\sigma(u_i)$ will approach zero, causing $\\sum_{\\tau=1}^{t}\\widetilde{\\boldsymbol{x}}(\\tau)^T\\widetilde{\\boldsymbol{x}}(t)$ to converge. Continued training will also enlarge $\\Vert W(t) \\Vert_2^2$ and make $\\Big(1 - \\big(\\boldsymbol{v}(t)^T\\boldsymbol{a}\\big)^2\\Big)$ approach zero. Considering all the impacts, the co-correlation will increase very fast at the beginning, then the rate of increase will become smaller, and in the final stage, it will approach zero, as shown in Fig. 2 (c) and (d). In Fig. 2 (c) and (d), all the co-correlation of networks flattened at the end. \n\n## About weight regularization and co-correlation on Property 2\nAs shown in Eq. (19), the co-correlation is inversely related to the $L_2$-norm of the weight matrix. This $L_2$-norm is the operator norm, which is different from the norm used in weight regularization, such as L2 regularization, which is the Frobenius norm (F-norm). This implies that merely controlling the F-norm of the weight matrix may not have a direct impact on $\\Vert W \\Vert_2$. However, since all norms are equivalent in finite-dimensional vector spaces, L2 regularization will somehow restrict its increase during training. Additionally, since the value of $C(t)$ also depends on $\\Big(1 - \\big(\\boldsymbol{v}(t)^T\\boldsymbol{a}\\big)^2\\Big)$, where $\\boldsymbol{v}$ is the maximal singular vector of $W(t)$, it is difficult to determine whether $C(t)$ will become larger because of weight regularization. \n\n## About L267\nYes, if the learning rate is too large, whether at the initial stage or at any stage, $\\widetilde{x}^{\\star}$ may fluctuate too much, but it may also prevent the model from converging or result in poor performance. In such cases, it is possible that $C(t)$ could become negative. However, in most machine learning settings, it is required that the learning rate not be too large, such as using 3e-4 with Adam, to ensure convergence and good performance. Therefore, I believe the assumption that the learning rate is not too large is realistic.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jGlPILKzkz", "reviewer": "Reviewer_yLC9", "summary": "This paper investigates the implicit bias of gradient descent in over-parameterized neural networks, particularly focusing on the collaboration between consecutive layers. The study first introduces Dirichlet energy to evaluate the adversarial risk. Then it decomposes Dirichelt energy between layers and measures the alignment of feature selections between layers. The collaboration of Dirichlet energy between layers is called co-correlation. Through theoretical analysis and extensive experiments, the authors demonstrate that over-parameterized networks exhibit stronger resistance to increased co-correlation, thereby enhancing adversarial robustness.", "questions": "1. Why the final co-correlation of MLP is smaller than linear net with same width and initialization?\n2. How will the Dirichlet energy for each mapping $\\phi$ and $\\psi$ changes? Will this also affect the Dirichlet energy and adversarial robustness of the neural network?", "limitations": "The authors have addressed the limitations of their work, acknowledging the challenges in extending the approach to more complex models and the need for broader evaluations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper introduces the novel concept of co-correlation to quantify collaboration between layers, offering a fresh perspective on implicit bias in neural networks. The notion of co-correlation helps with distinguishing the dynamics between over-parameterized and under-parameterized networks. Moreover, this paper provides robust theoretical analysis for the properties of the dynamics of co-correlation.", "weaknesses": "1. The theory part of the paper mainly analyzes the dynamics of co-correlation. However, the paper only explain the relation between Dirichlet energy and adversarial robustness. It does not directly connect adversarial robustness with co-correlation in analysis.Without a valid analysis, co-correlation dynamics cannot lead to the conclusions on adversarial robustness.\n2. In the analysis of co-correlation,  this paper proved $C(t)\\>0$ after training for some time. This result indicates the increeasing trend of co-correlation but not explaining the value and the rate of the convergence, which shows the strength of the implicit bias. Moreover, it could also be better if the authors discuss in the theory part about the effect of initialization shown in experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and careful review. The rebuttals are as follows: \n\n## About the relation between co-correlation and adversarial robustness\nThe relationship between co-correlation and adversarial robustness is bridged by the Dirichlet energy, as depicted in Theorem 4.5 and Eq. (15), shown as \n$$\n  \\mathfrak{S}(\\boldsymbol{\\phi}\\circ \\boldsymbol{\\varphi}) = \\varrho_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}} \\Big(1 + \\frac{var_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}}{\\mu^2_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}}\\Big)^{\\frac{1}{2}} \\rho_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}  \\mathfrak{S}(\\boldsymbol{\\phi}) \\mathfrak{S}(\\boldsymbol{\\varphi}). \n$$\nWhile co-correlation $\\varrho_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}$ is not directly related to adversarial robustness, it is the most significant factor affecting the Dirichlet energy and the Dirichlet energy is directly related to adversarial robustness as shown in Thm. 4.1. Specifically, let $\\phi \\circ \\varphi$ be a 2-layer neural network where $\\varphi$ represents the first layer and $\\phi$ represents the second layer. First, as demonstrated in Eq. (15), the Dirichlet energy of the network $\\mathfrak{S}(\\phi \\circ \\varphi)$ is dominated by the co-correlation $\\varrho$, since the terms $\\Big(1 + \\frac{var_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}}{\\mu^2_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}}\\Big)^{\\frac{1}{2}}$ are negligible as shown in Fig. 4. Seondly, our work concerns more about how the Dirichlet energy of layer 1 and layer 2, i.e., $\\mathfrak{S}(\\boldsymbol{\\phi})$ and $\\mathfrak{S}(\\boldsymbol{\\varphi})$, can be passed to overall  Dirichlet energy of neural networks. And this interaction between layers can be quantified by co-correlation $\\varrho$.  \n\nIt is important to note that this paper primarily aims to understand the interplay between layers concerning adversarial robustness, i.e. $\\varrho$, rather than adversarial robustness itself. Therefore, the direct relationship between co-correlation and adversarial robustness is not our primary focus.\n\n## About the value and the rate of convergence\nAs indicated in Thm. 5.3 and Thm. 5.5, Eq. (17) and Eq. (20), the exact value of $C(t)$ depends on several assumptions about the inputs and the state of the weight matrix $W(t)$. It is possible to provide a more precise evaluation of $C(t)$ based on specific assumptions about the inputs. However, doing so requires particular assumptions, such as Gaussian distribution of the inputs and weight matrix.\n\nBecause our work primarily aims to answer the research question of whether there exists collaboration between layers to improve adversarial robustness, we focused on qualitative analysis of dynamics change rather than quantitative analysis of the convergence speed. This is an interesting problem, but it is impossible to address all issues within a limited article. Future studies may explore this idea.\n\n## About co-correlation of MLP is smaller than linear net\nAs is shown on page 5, the co-correlation represents the feature alignment of layers in neural networks. Introducing an activation function, at least ReLU, as an 'isolation' layer between $W_1$ and $W_2$, is suggested to reduce co-correlation. This concept is experimentally verified in Fig. 1 (c) and (d). While we did not formally prove this idea, and different activation functions other than ReLU may even enhance co-correlation, our conclusion holds as long as the derivative of the activation function is bounded, as detailed in Assumption 5.4.\n\n## The effect of Dirichlet energy for each mapping\nYes, as shown in Eq. (15), the overall Dirichlet energy is also affected by that of individual mappings. However, since our primary focus is on the interplay between layers, this aspect is beyond the scope of our paper. Even if the Dirichlet energy for each mapping, $\\mathfrak{S}(\\boldsymbol{\\phi})$ and $\\mathfrak{S}(\\boldsymbol{\\varphi})$, are large, co-correlation still represents the interplay between $\\phi$ and $\\varphi$. Additionally, since other statistics are negligible shown in Fig. 4, whether the high value of Dirichlet energy for each mapping can be transferred to $\\mathfrak{S}(\\boldsymbol{\\phi} \\circ \\boldsymbol{\\varphi})$ still highly depends on $\\varrho$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PnanC7hM2U", "reviewer": "Reviewer_DyJc", "summary": "In this work, the authors study the difference between underparameterised and overparameterised networks in terms of the collaboration between consecutive layers. They find that under-parameterized networks tend to foster co-correlation among layers to improve performance, whereas performance of over-parameterized networks can improve without relying on this co-correlation. They relate the co-correlation to adversarial robustness.", "questions": "See points raised above. In general, I think this work could be a valuable contribution, but the quality of the text has to be increased to better motivate and explain the approach. In it’s current state, the work is hard to judge. \n\nIf you think a change to the manuscript is warranted, please specify how you would update it.", "limitations": "The limitations are not addressed sufficiently. E.g., not clear what the influence of the strong assumption that the second layer remains fixed has on the generality of the findings. \n\nNo potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper adresses an important open question in deep learning: how should we relate the concepts of generalization, adversarial robustness and the network archictecture and size? The paper adresses this question from an original perspective. \n\nThe literature study is quite extensive and motivates the work (although it remains quite vague/confusing, see next). \n\nGreat care has been taken in a correct mathematical formulation off all definitions and proofs. \n\nThe authors have verified some of their theoretical claims with a series of experiments.", "weaknesses": "1.\tFrom your abstract, introduction and conclusion, it is not clear why you would study  co-correlation between layers in the first place, and/or how this relates to robustness and generalization. E.g., starting from line 40, the reader can infer: Dirichlet energy = measure of adversarial robustness -> Dirichlet energy per layer -> co-correlation -> gradient descent increases co-correlation. But how and why is this co-correlation exactly related to adversarial robustness? Etc. \n\n2.\tThis does not become really clear to me in the other parts of the paper either. I understand that the dirichlet can be used to measure the gap between generalization and adversial risk. But what this means for individual layers and why you would want to look into this remains unclear. E.g., Line 163: “Since the Dirichlet energy defined in Equation (7) can be used to measure the variability of mappings,it follows that this metric could be employed to evaluate the adversarial robustness of individual layers or modules within neural networks. Consequently, this allows for an assessment of whether there is collaboration between these components in terms of adversarial robustness.”: this seems an important point, but I could’t understand what you meant exactly, especially the second sentence.\n\n3.\tL117: why is the second layer random and fixed? If needed for a mathematical analysis, what is the influence of this on the generality of your findings?\n\n4.\tThe claim made in L160-161 cannot be supported by fig. 1 alone. It’s not because Dirichlet energy has a similar (inverse) profile over width and initialization as the adversarial robustness, that one influences the other directly. The same goes for the other figures.  More information/verification seems needed. \n\n5.\tL192-194: what about non-linear activation functions? \n\n6.\tThe experiments are only conducted on the simplified models used for the theoretical derivations, and as such do not show anything about how the theoretical findings relate to actual, practical neural networks (i.e., models with more layers, where no layers are held fixed). \n\nTypos/grammar/symbols: \n7.\tLine 64, 76, 77, 105, 111 (strange sentence), 127, caption figure 1, 275, …\n8.\tEq 1: f_W undefined\n9.\tFig 1: legend and labels too small, caption not clear", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable feedback and careful review. Your detailed examination of the paper and identification of its flaws are greatly appreciated. My rebuttals are as follows:\n## The reason why study co-correlation: \nThe motivation for studying co-correlation actually comes from the research question: \n\n*Can different layers of a neural network collaborate against adversarial examples?* \n\nThis is an important point, especially in an era dominated by complex neural networks. For example, in the main block of a vanilla Vision Transformer, we have Multi-head Self Attention (MSA) followed by an MLP. People may expect these components to have specific functionalities and imagine the forward propagation working like independent robots on a conveyor belt. However, even for a simple MLP, there is hardly any research on whether different layers interact with each other, let alone on the effects on adversarial robustness. Our work aims to fill this gap to some extent.\n## How co-correlation related to adversarial robustness:\nThe relationship between co-correlation and adversarial robustness is bridged by the Dirichlet energy, as depicted in Theorem 4.5 and Eq. (15). While co-correlation $\\varrho_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}$ is not directly related to adversarial robustness, it is the most significant factor affecting the Dirichlet energy as shown in Thm. 4.1. Specifically, let $\\phi \\circ \\varphi$ be a 2-layer neural network where $\\varphi$ represents the first layer and $\\phi$ represents the second layer. Since the terms $\\Big(1 + \\frac{var_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}}{\\mu^2_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}}\\Big)^{\\frac{1}{2}}$ is negligible, the co-correlation $\\varrho$ dominate how the Dirichlet energy of each layer can be passed to that of neural network. \n## About L163:\nThe use of Dirichlet energy to measure adversarial robustness was first considered in the paper by Dohmatob et al. [8], as mentioned in line 137. It was originally proposed to replace the Lipschitz constant as a better measure of adversarial robustness, as shown on page 14, Appendix A, in [8]. In our work, we first demonstrate that Dirichlet energy can measure the gap between generalization and adversarial risk, implying that Dirichlet energy is a proper measurement for adversarial robustness since higher Dirichlet energy indicates a larger gap.\n\nWith simple modifications, we can also calculate the Dirichlet energy for a single layer or blocks within neural networks, as defined in Eq. (7) in Def. 3.2. We use this concept because of the decomposition of the Dirichlet energy for neural networks, as shown in Eq. (15). If we regard the neural networks as compound functions, i.e., $\\theta \\circ \\phi$, the decomposition of the Dirichlet energy for this compound function requires the computation of Dirichlet energy for $\\phi$ and $\\theta$, each of which can be an individual layer inside the neural network. This explains the first sentence that we can evaluate the adversarial robustness of individual layers. I apologize for the misuse of “adversarial robustness,” which will be corrected in a later version. \n\nFor the second sentence, as is shown in Eq. (15), since the term $\\Big(1 + \\frac{var_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}}{\\mu^2_{\\boldsymbol{\\phi}, \\boldsymbol{\\varphi}}}\\Big)^{\\frac{1}{2}}$ is negligible shown in Fig. 4, leave co-correlation $\\varrho$ dominating whether Dirichlet energy for each mapping can be transferred to overall Dirichlet energy. Hence, we say that we can assess the collaboration between layers via $\\varrho$.  \n## About L117\nThe random and fixed second layer is used for simplification of our analysis, following the previous work by Du et al. [9], which was published in ICLR. Since we use similar approaches—directly analyzing the gradient descent—and to make the results easier to understand, we adopted the same setting. We will emphasize this in the paper of a later version.  \n## About the claim in L160\nEach neural network in Fig. 1 (a) and (b) was trained to converge under identical conditions, except for width and weight initialization. Therefore they demonstrate the relationship between Dirichlet energy and adversarial robustness, though not causally. However, as verification of Thm 4.1, it is adequate to show the correctness of the proof. And more experiments will be included in a later version.\n## About L192-L194\nL192-L194 discusses the interpretation of co-correlation in a linear case without activation functions to simplify understanding. Since the activation functions will make the transformation at layers become non-linear, we can replace the weight matrix with the Jacobian. Let $\\theta$ and $\\phi$ be the weight matrices with activation functions, then Eq. (11) becomes\n$$\n\\frac{\\Vert J_{\\theta} J_{\\phi}  \\Vert_2 }{\\Vert J_{\\theta} \\Vert_2 \\Vert J_{\\theta} \\Vert_2 }\n$$\nThe explanation at L190-L194 is still applicable for the Jacobians of both layers. The difference is that the Jacobian varies with different inputs, potentially attect co-correlation, as empirically shown for ReLU in Fig. 1 (c) and (d).\n## Extra experiments \nWe conduct the experiments with ResNet50 and Wide-ResNet50 on CIFAR10. We divided the ResNets in two ways and used the Adam optimizer with a learning rate of 0.003 to track the dynamics of co-correlations. The experiment results are available at the link: [Dynamic of co-correlation of the divide as $A_1$, $A_2$](https://i.imgur.com/icGOSOK.png) and [Dynamic of co-correlation of the divide as $B_1$, $B_2$](https://i.imgur.com/EfTfHx4.png). As shown in both figures, the co-correlation increases over the training epochs, although the upward trend for the separation of $A_1$ and $A_2$ is not strictly monotonic. The figure shown in the anonymous link: [Illustration of the divide for ResNet50](https://i.imgur.com/1OPWAuO.png) shows the way we divide ResNet50.\n\n### We will check all lines and make them more clear in a later version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their careful review.

## Motivation of the paper 
This paper is mainly to address the research problem: 

*Whether there a collaboration between layers against adversarial examples during Gradient Descent?* 

To quantify this collaboration, we introduce a new concept called *Co-correlation*, interpreted as the alignment of feature selections to maximize outputs for each layer and investigate the implicit bias of gradient descent. We theoretically show that Gradient Descent enhances this collaboration between layers. Additionally, we observe different behaviours for under- and over-parameterized neural networks: under-parameterized networks tend to foster co-correlation among layers to improve performance, whereas over-parameterized networks' performance improvement does not heavily rely on establishing such co-correlation.

## How we will adjust in the later version
After reviewing all the feedback, we will adopt all the necessary suggestions. To facilitate understanding of our work, we will add more intuition behind our theorems and better clarify our arguments. Because this is theoretical work, it inevitably has some limitations in the experimental aspects. However, more real-world experiments will be included in a later version. The code for these experiments will also be made publicly available.

Meta-review: This paper explores the tradeoff between generalization and adversarial robustness, particularly focusing on the collaboration between layers. Specifically, the concept of "Co-correlation" is introduced to quantify the alignment of feature selections across layers. By decomposing the Dirichlet energy and analyzing interactions between layers, the authors reveal that gradient descent optimization enhances collaboration correlation, affecting adversarial robustness. Through theoretical analyses and experiments, this paper demonstrates how larger widths in MLPs can enhance adversarial robustness by resisting increased co-correlation.

This paper received divergent scores before the rebuttal and the reviewers raised concerns about insufficient experimental results, unclear claims, weak presentation, and limited generality. Reviewers and AC appreciate the hard work authors have put into responses during the rebuttal, including additional results and detailed clarification. The discussion led to several score increases and all the reviewers came to a consensus of acceptance. AC concurs with the evaluations given by the reviewers.

Note that half of the reviewers mentioned that the presentation needs improvement. The authors are strongly encouraged to improve the final version according to the feedback from all the reviewers as promised in the discussion.
Decision: Accept (poster)