Residual Alignment: Uncovering the Mechanisms of Residual Networks
Authors: Jianing Li
Vardan Papyan
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Learning
Residual Networks
Neural Networks
Generalization
Spectral Analysis

Abstract: The ResNet architecture has been widely adopted in deep learning due to its significant boost to performance through the use of simple skip connections, yet the underlying mechanisms leading to its success remain largely unknown. In this paper, we conduct a thorough empirical study of the ResNet architecture in classification tasks by linearizing its constituent residual blocks using Residual Jacobians and measuring their singular value decompositions. Our measurements ([code](https://colab.research.google.com/drive/1yKjEg2yF616tnZFAfuN0aQ-E9v3JmyjN?usp=sharing)) reveal a process called Residual Alignment (RA) characterized by four properties:
- **(RA1):** intermediate representations of a given input are *equispaced* on a *line*, embedded in high dimensional space, as observed by Gai and Zhang [2021];
- **(RA2):** top left and right singular vectors of Residual Jacobians align with each other and across different depths;
- **(RA3):** Residual Jacobians are at most rank $C$ for fully-connected ResNets, where $C$ is the number of classes; and
- **(RA4):** top singular values of Residual Jacobians scale inversely with depth.

RA consistently occurs in models that generalize well, in both fully-connected and convolutional architectures, across various depths and widths, for varying numbers of classes, on all tested benchmark datasets, but ceases to occur once the skip connections are removed. It also provably occurs in a novel mathematical model we propose. This phenomenon reveals a strong alignment between residual branches of a ResNet (RA2+4), imparting a highly rigid geometric structure to the intermediate representations as they progress *linearly* through the network (RA1) up to the final layer, where they undergo Neural Collapse.

Residual Alignment:
Uncovering the Mechanisms of Residual Networks
Jianing University of Toronto jrobert. 11 @mail. utoronto.ca
Vardan Papyan University of Toronto vardan hat.laaapppyaauutooootoooooo ca
Abstract
The ResNet architecture has been widely adopted in deep learning due to its significant boost to performance through the use of simple skip connections, yet the underlying mechanisms leading to its success remain largely unknown. In this paper, conduct thorough empirical study of the ResNet architecture in classification tasks by linearizing constituent residual blocks using Residual Jacobians and measuring their singular value decompositions, Our measurements (code) reveal a process called Residual Alignment (RA) characterized by four properties:
(RA1) intermediate representations of a given input are equispaced on line, embedded in high dimensional space, as observed by Gai and Zhang [202T]:
(RA2) top left and right singular vectors of Residual Jacobians align with each other and across different depths;
(RA3) Residual Jacobians are at most rank for fully-connected ResNets, where
C' is the number of classes; and
(RA4) top singular values of Residual Jacobians scale inversely with depth.
RA consistently occurs in models that generalize well, in both fully-commected and convolutional architectures, across various depths and widths, for varying numbers of classes, all tested benchmark datasets, but ceases to occur once the skip connections are removed. It also provably occurs in a novel mathematical model propose. This phenomenon reveals strong alignment between residual branches of ResNet (RA2+4), imparting highly rigid geometric structure to the intermediate representations as they progress linearly through the network (RA1) to the final layer, where they undergo Neural Collapse.
Introduction
1.1 Background
The Residual Network (ResNet) architecture [He et al. 2016aa special case of Highway Networks (Srivastava al. 2015), has taken the field of deep learning by storm, since its proposal in 2015. The incredibly simple architectural modification of adding a skip connection, spanning a set of layers, has become a go-to architectural choice for deep learning researchers and practitioners alike. Initially applied in computer vision, has since been integrated as crucial component in biomedical imaging and generative models via the U-Net [Ronebberger et al., 2015), natural language processing through the Transformer /Vaswani et al. 2017], and reinforcement learning as seen in the case of AlphaCio Zero (Silver et al.) 2017
The ResNet architecture first passes an input x through initial layers I (comprising a sequence of convolution, batch normalization, and ReLU operations), depending on trainable parameters Wo.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Figure I: Visualization of Residual A Alignment. Intermediate representations of a ResNer34 trained on CIFARIO, are projected onto two random vectors. Representations of each individual image are color-coded based on its true label and connected form trajectory, so as to showcase their progression throughout the network. Notice the linear arrangement of intermediate representations along with equidistant spacing between representations corresponding consecutive layers (RA1), Our work shows, this phenomenon results from the alignment of top singular vectors of Residual Jacobians (RA2) and the inverse scaling of top singular values with depth (RA4). It also noteworthy that the magnitudes of class means significantly increase with depth compared to the within-class variability, indicating the representations undergo layer-wise Neural Collapse [Papyan 2020 Galanti
et al. 2022
He and Su 2022 Li et al. 2023
generate initial representation
h1 [[; E R
This is then processed through sequence of residual blocks
h441 oh, + F(,,; W.)), ISiSL,
each refining the previous layer's representation, h4 Rd through simple residual branch F(h;; Wa) R -RR,,,
which depends on trainable parameters W; and an wise non-linearity, o, which is simply an identity mapping, in the case of pre-activation ResNets (He et al. 2016b]. The final block's output, hL+1, is fed to a classifier,
((h++1:W++) R RC.
depending on trainable parameters WL+1-
In the original ResNet architecture, F and C are compositions of linear transformations, element- nonlinearities, and normalization layers. The WideResNet architecture. by(Zagoruyko and Komodakis [0116]. further incorporates dropout layers into F. Yet more changes are made in ResNext (Xie et al. 2017]. where is computed from the summation parallel computational branches.
Training a ResNet for classification involves minimizing a loss on a training dataset, (I J consisting of inputs In and labels yo. plus a weight decay term
X 11
minimize
vn)
(WJ
where f f( W) are the outputs of the classifier C, also called logits, and W are all the parameters.
1.2 Problem Statement
The significant improvement in performance, achieved through the simple addition of a skip con- nection, has generated interest in understanding the underlying mechanisms of ResNets. Despite this, to this date, no definitive theory or explanation has been widely accepted by the deep learning community for the success of ResNets.
"ResNe334 has (31 2)/2 16 residual blocks since each residual block consists of two weight matrices and two layers are subtracted one due to initial convolution and the other the final classification layer. Therefore, we would expect to see at most 16 dots. In the plot, we can count around 10 dots but the rest are too cluttered around the center. This is in agreement with Figure C.4Jin in the Appendix showing the inverse scaling of the top singular value occurs for roughly the last blocks, which correspond to the visible dots.

---

1.3 Method Overview
In this paper, we conduct thorough empirical study of the ResNet architecture and its constituent residual blocks with the aim of investigating the characteristics of an individual residual block and the relationship between any pair of them. As the residual blocks are nonlinear functions of their inputs, we examine their linearizations through the Residual Jacobian matrices
o' d' + F(h;; W.)) 0F W.) E ERD
Oh,
Following Equation [1.Ip, these correspond to the derivative of the residual block with respect to its input, excluding the contribution from the skip connection, o' + F(2;; W.)) E RDxD, In the case of pre-activation ResNet, these simply equal to the derivative of the residual branch with respect to its input, i.e., 0.F(h:: Wa) Since the Residual Jacobians are high-dimensional matrices, and likely contain meaningful information only in some of their subspaces, measure their singular value decomposition (SVD), given by J; = U.S.V,", where U; and Vi are the respective left and right singular vectors, and S; is the singular value matrix.
1.4 Contributions
We discover a phenomenon called Residual Alignment (RA), consistently occurring in ResNet models that generalize well, characterized by four propertie
(RA1) Intermediate representations of a given input are equispaced on a line, embedded in high dimensional space, shown in Figure [ [and and observed by Gai and Zhang] [2021);
(RA2) Top left and right singular vectors of Residual Jacobians align with each other and across different depths, as observed in Figure 2
(RA3) Residual Jacobians are most rank for fully-connected ResNets, where is the number of classes, as illustrated in Figures and 4 and
(RA4) Top singular values of Residual Jacobians scale inversely proportional with depth, as depicted in Figure
The properties are interrelated and, in fact, (RA1) can be logically derived from the other properties, demonstrated in Section 3
As a further contribution, section B of the Appendix we prove theoretically the emergence of RA under the setting of binary classification with cross-entropy loss. Our proof relies on novel mathematical abstraction called the Unconstrained Jacobians Model, in which the Residual Jacobians optimized directly, parameters of the residual branches. This mathematical abstraction is motivated by recent thcoretical works Neural Collapse, discussed in Section
minimize the loss, and not constrained by the architecture and
1.5 Results Summary
Our empirical investigation, presented the main text and Appendix, consistently identifies RA across an extensive range of:
Architectures: standard ResNets (convolutional layers with progressively increasing channels, interspersed with downsampling layers) as well simpler designs (fully-connected layers);
Datasets: MNIST, FashionMNIST, CIFAR10, CIFAR100, and ImageNette [Howard]: ;; and
Hyperparameters: network depth and width.
In addition to our main findings, include an experiment showing the co-occurrence of (RA) and Neural Collapse in Figure 7 We also performed three counterfactual experiments that show:
FFor Type 1 model, D = 512 where 512 is the width of the network. For Type 2 model, D 16 x 16 64 16384 because the representations have 64 channels and spatial dimension resolution of 16. 'These properties hold once the Jacobians are evaluated on the training data. evaluated on other data, different phenomena might emerge. For example, we observed that the growth rate in (RA4) becomes exponential once the Jacobians evaluated the classification boundary, where the logits close to

---

1. If the number of classes in the dataset is increased, the singular vector alignment occurs in higher dimensional subspace (Figure 3
2. If stochastic depth [Huang et al. 2016 is incorporated, the singular vector alignment is amplified (Figure 5): and
3. If the skip connections are removed, RA does not occur (Figure 6
(a)
(b)
Figure 2: (RA2): Top singular vectors of Residual Jacobians align. Subfigure 2a and Subfigure 2b present the alignment of the first 8 blocks and the last 7 blocks, respectively, for ResNet34 trained on CIFAR (Type model Section 2.I forwarding a single randomly sampled input. Each subplot (i.j) illustrates the matrix U j,30 j.30. where 3j.30 and Vj,30 denote the top-30 left and right singular vectors the Residual Jacobian Jj. respectively, and i,j are the indices of the residual blocks, i.e., their depth. A distinct diagonal line of intense pixels is apparent in almost every subplot, signifying that the top singular vectors J, diagonalize J,. In simpler terms, this means that the top singular vectors J; and Jj align and (RA2) holds. This pattern persists when J,U,,, plotted, instead of / V5,30. further confirming that the top left and right singular vectors align accordance with (RA2) Additional visualizations of both matrices, across various models and datasets, are available in subsections C22 and C.3 of the Appendix. It is crucial to highlight that no alignment exists between the Jacobians at initialization, and the alignment emerges during training.
(a) classes
(b) classes
(c) 8 classes
(d) classes
(e) classes
Figure 3: (RA3): Singular vector alignment occurs in subspace of rank C. The figure presents a sequence of subplots that illustrate the matrix UTa.10 V16. Here, Jg represents the 9-th Residual Jacobian, while U16.10 and V16,10 correspond the leading 10 left and right singular vectors, respectively, of the 16-th Residual Jacobian, J16- These calculations are based on ResNet34 models (Type model in Section p. These models have been trained specific subsets of the CIFARI0 dataset, comprising of 4, 6, and 8 classes, as well as the complete CIFAR 10 and CIFAR 100 datasets. Each result is presented in the corresponding Subfigures 3a
3b
3c
3d
and 3e As the number of
classes increases, the alignment of singular vectors occurs in an increasingl ly higher-dimensional subspace.

---

2.0
10
1.5
1 Singular Value
8
Singular Value
6
1.0
0.5
2
0.0
5
10
15
5
10
15
Depth
Depth
(a) (RA3): : Residual Jacobians are at most rank C. (b) (RA4): : Top singular value s 1/depth, Post block
The top-10 singular values (orange) noticeably surpass 5, the reciprocal of the top singular value scales linearly the next 10 (blue), indicating that the Residual Jaco- with depth and with slope 0.992, indicating singular bians rank less than C.
value close to 1/1 the i-th block after RA begins.
Figure 4: Depiction of Residual Jacobian singular values for ResNet34 trained on CIFARIO (Type 1 model in Section 2. Subfigure 4a shows the top 20 singular values of Residual Jacobians, while Subfigure illustrras the inverse scaling of the top values. More singular value plots, from diverse models and datasets, are available in subsection C.4 of the Appendix.
(a) V3,30 regular Type model.
(b) U3,30. V,30. Type model with stochastic depth.
Figure 5: Stochastic depth amplifies singular vector alignment. A comparison of (RA2) for two Type models trained on CIFARIO over 50 epochs: one employing the stochastic depth technique (with drop probability of 0.3 for skipping residual blocks during training) and the other without it.
Table 1: (RA) occurs in models that generalize well. The table displays the accuracies of models trained to study (RA) Our reported accuracies closely align with those presented in Table of Papyan al. [2020], indicating that (RA) observed in extensively trained models that exhibit strong performance in terms of test accuracy.
Model
MNIST
FashionMNIST
CIFARIO
CIFARI00
ImageNette
Type
98.9
90.9
58.3
30.4
42,6
Type
99.5
92.0
88.5
54.6
67.9
Type
99.6
92.8
87.3
62.6
64.2
5

---

(b)
Figure 6: Skip connections cause Residual Alignment. The experiment depicted in Figure is replicated using two additional models. The first is ResNet18 trained on CIFARIO (Type 4a model in Section 2.1). with results showcased in Subfigure 6a The second is ResNet18 without skip connections (Type 4b model in Section 2.1 with results displayed in Subfigure [bb] When the skip connections are removed, the top singular vectors no longer align. Alignment is visible in the diagonal subplots of Subfigure 6b each Residual Jacobian is diagonalized by its own singular vectors.
10F
10
10th
M
102
200
300
200
100
200
(a) NCI: activation col- (b) NC2: equinorm class (c) NC2: equinorm classi- fiers.
(d) NC2: maximal equian- gularity of class means.
lapse.
means.
10-4
139
10-1
30
10-²
10
10
10"
100
200
300
100
200
300
300
200
(c) NC2: maximal equian- gularity of classifiers.
NC3: self duality.
(g)
NC4: convergence to nearest class center.
(h) U3.30 V9.30.
Figure 7: Co-oceurrence of Residual Alignment and Neural Collapse. The sub-figures display the progression of Neural Collapse metrics for Type 4a model throughout 350 training epochs on the MNIST dataset as well as the emergence of Residual Alignment at the end of the training process.
2 Methods
2.1 Networks
We train types models:
Type models consist of 16 basic residual blocks, each containing two fully-connected layers of dimension D=512;

---

Type 2 models consist of 16 basic residual blocks, each containing two convolutional layers with 64 channels, operating on tensor with a 16x16 spatial resolution;
Type 3 models are Type 2 models, but the last 8 basic residual blocks have 128 channels operating on 8x8 spatial resolution instead;
Type 4 models are Type models with 8 instead of 16 residual blocks. Type 4a models have skip connections and Type 4b models do not. These models serve the purpose of verifying that the residual connection causes RA, and RA does not occur once they are removed. F
Type 5 models are obtained from Type 2 models in the same way that Type models are obtained from Type I models. We again have Type 5a and Type 5b, denoting models with and without skip connections, respectively.
In our approach, we have deliberately chosen to use two spatial resolutions instead of the more common three. This decision allows to have greater number of residual blocks with the same spatial dimension.
2.2 Datasets
We train these models the MNIST, FashionMNIST. CIFARIO. CIFARIO0, and ImageNette [Howard] datasets. To explore the relationship between RA and the number of classes, we have trained additional models subsets of these datasets, specifically with 2, 4, 6, and 8 classes.
2.3 Optimization
We train for 500 epochs, using the SGD optimizer, with batch size 128, an initial learning rate of 0.1, the cosine learning rate scheduler [Loshchilov and Hutter [016]. and weight decay of 1c-1 for convolutional models and lc-2 or 5e for fully-connected models, A large weight decay does not negatively impact the generalization performance, seen in Table [ but it does strengthen RA.
2.4 Randomized SVD
The Residual Jacobian matrix of convolutional residual block with 64 channels operating on a spatial dimension of 16x is of size 16384x 16384. Given that SVI) scales cubically with the matrix side length, becomes essential to leverage more efficient algorithm to expedite computation and mitigate memory consumption. Furthermore, it is worth noting that we do not require the full SVD of the matrix. Our objective is to obtain the top singular values to confirm the trend in (RA3) and (RA4) and the top singular vectors validate the alignment in (RA2).
We leverage Algorithm 971 (Li et al. 2017], a randomized SVD algorithm, to accurately compute the desired top singular values and vectors. This method involves series of matrix multiplications and QR decompositions, requiring a runtime of OT K + TK3) instead of O O(D*), where T is the number of algorithm iterations, D is the matrix side length, and K is the number of top singular vectors. In our implementation, the T is set to 20, and the K ranges from 10 to 30, depending on the specific experiment. Through extensive examination, we have verified that this algorithm reliably provides singular values and vectors of sufficient accuracy.
Results
3.1 Empirical Results
The empirical results supporting (RA) are found in Table I and Figures and 4 The counterfactual experiments are in Figures 3 and 6 We also include an experiment showing the co-occurrence
of (RA) and Neural Collapse in Figure Experimental descriptions and discussions are within the captions. Additional supporting data can be found in the Appendix.
"The 9th residual block downsamples representation through projection shortcut He al. 2016a
'ee use blocks because the optimization fails converge when using 16 blocks and no skip connections
7

---

3.2 (RA2+3+4) Imply (RA1)
As mentioned the introduction, the properties of (RA) are interconnected, and this relationship demonstrated through the following theorem:
Theorem 3.1. For binary classification, in pre-activation ResNet, assuming the Jacobian lineariza- tions are exact and satisfy (RA2+3+4), then (RA1) 1) holds for the intermediate representations.
The proof of this Theorem is deferred to section A of the Appendix.
3.3 Unconstrained Jacobians Model Leads to RA
We propose the following abstraction of the optimization problem in Equation 1 Definition 3.2 (Unconstrained Jacobians Model). Given fixed imput AS E RP and its label y E (+1,-1) find matrices J; E RDD, S S I, and vector w E TR that
minimize (0.
L(w II J.)
Jell
1wl
In the problem described above, J4 again represents the Residual Jacobian of the i-th residual branch and wo represents the classifier Jacobian,
ac WL+1)
Hhg,-1
It is referred to the "Unconstrained Jacobians Model" because the Jacobians are not restricted to any specific form and are not required to be realizable by a set of layers. In the Unconstrained Jacobians Model, the Jacobians are regularized through simple functions. However, in reality, weight decay, dropout, normalization layers, and parallel branches regularize the Jacobians intricate ways that are hard capture mathematically.
prove section the Appendix the following theorem:
Theorem 3.3. For binary classification, there exists a global optimum the the Unconstrained Jacobians Model where the top Jacobian singular vectors are aligned (RA2). all Jacobians are rank one, analogous to (RA3), and the top Jacobian singular values are equal, analogous to (RA4).
Here, the top singular values are equal and not decaying predicted by (RA4), because the Ja- cobians are evaluated on the classification boundary instead of on training examples. Therefore, the intermediate representations are not equispaced on line, as predicted by (RA1) but are rather exponentially-speen on a line.
4 Discussion
In this section, we pose research questions that emerge from the discovery of RA.
The discovery of RA offers a vivid analogy for comprehending generalization in deep learning. According RA, we can envision a ResNet as system of conveyor belts, where each conveyor carries representations of a specific class in a unified direction at constant speed. Misclassification occurs when the representation mistakenly steps onto the wrong conveyor belt in the initial layers of the network. This perspective leads us to hypothesize that the first few layers of the network play vital role in generalization, surpassing the significance of subsequent layers. Consequently, more research should be dedicated towards studying the dynamics of the initial layers.
Generalization
Neural Collapse As mentioned in Figure I there is an intriguing pattern where models exhibit RA concurrently with layer-wise Neural Collapse Papyan 2020 Galanti et al. 2022 et al. 2023). The concurrent manifestation of these two distinct events could possibly be more than mere coincidence. would interesting to explore if RA could shed light on phenomena related layer-wise Neural Collapse such the "Law of Data Separation" [He and Su 2022

---

RA in Transformers We have empirically substantiated the occurrence of RA in ResNets. However, our study has not yet extended to Transformers, which also incorporate residual connections. An intriguing line of inquiry for future work would be to probe whether RA, or a phenomenon akin to it, manifests within these architectures.
Recurrent Architectures Exhibiting RA Recurrent neural networks, Neural ODEs (Chen et al. 2018], and deep equilibrium models [Bai el. 2019] iteratively apply layer within deep neural network. This leads to the following questions:
Do the intermediate representations of these models exhibit RA? If not, can propose novel architecture that recurrently applies computation but does exhibit RA?
Model Compression In our work, we demonstrate that the network can converge to a model that iteratively applies computation, even without imposing explicit constraints, like the architectures in the previous subsection. This leads us to yet another question:
Can replicate the original network's performance by distilling the aligning layers into a single layer and deratively applying it?
Existing regularization techniques, including layer permutation Liaw jt al. 202T] and structured dropout (Fan et al. 2019) should strengthen RA. A question arises:
Regularization Techniques
Could RA explain the success of these methods and do they indeed amplify RA?
Related Work
Linearization of Residual Jacobians Prior to our work, Rothauge et al. (0199 proposed the linearization of residual Jacobians and investigated the distribution of their singular values. Our work, however, concerns the discovery of the alignment between the Residual Jacobians, as well as its theoretical understanding.
Generalization A thorough empirical investigation by Novak et al. [0188 found that the Frobenius norm of the input-output Jacobian of network correlates well with generalization. Our research complements theirs by conducting both empirical and theoretical investigations on the Residual Jacobians, which form the input-output Jacobian of ResNet.
ResNets at Initialization Previous works have contributed significantly the understanding of the properties of intermediate representations of randomly initialized ResNets, with studies such as |Hayou| [2022] exploring the infinite- width and finite-width regime and [L et al. [2021] 2022] investigating the infinite-depth and infinite-wite width regime. Our current research diverges from these previous works by specifically focusing on studying the Residual Jacobians of fully trained ResNets.
Optimization Landscape Li et al. (016) analyzed the Hessian of the loss function for a ResNet initialized with zero parameters. Additionally, [Lu et al. [2020) use a mean-field analysis of ResNets to demonstrate convergence to a global minimum. Their analysis builds upon the observation that a shallow network ensemble, first noted by Veit al. 20166)
ResNet is similar
Rather than focusing on the optimization convergence properties of SGD, our goal is to examine the properties of the intermediate representations and Residual Jacobians that emerge during training.
Neural Collapse Recent theoretical work by
Mixon et al. [0200] Lu and Steinerberger 20200
E and Wojtowytschi [020], Poggio and Liao 2020]. Zhu et al. [2021]. Han et al. [2021] Tirer
and Bruna 2022], Wang | Kothapalli et al. [2022] analyzed the Neural Collapse phenomenon,
discovered by Papyan. Han, and Donoho [2020] through the unconstrained features model and the
layer-peeled model. In these, the assumption made that the last- representations, which are fed to classifier, have the freedom to move independently and are not constrained to be the output of deep network. Our Unconstrained Jacobians Model takes inspiration from these mathematical models by abstracting away the complex Residual Jacobians and assuming they can be optimized directly.

---

Unrolled Iterative Algorithm Perspective Greff et al. (016), Papyan et al. [017]] and Ebski et al. 2018) recognized ResNets as unrolled iterative algorithms performing iterative inference. We build upon this understanding by delving deeper into the empirical and theoretical relationships between the Residual Jacobians.
Neural ODE Chen et al. 20188 introduced the Neural ODE: a continuous-depth. tied-weights
ResNet. Initially, it may seem unlikely that the iterations or Jacobians of a traditional ResNet would possess any characteristics associated with Neural ODEs. However, Sander et al. [2022] proved that assuming the initial loss is small and the initial parameters are smooth with depth, linear ResNets converge to Neural ODE as the number of layers increases. Still. it is unclear if these findings extend to nonlinear ResNets trained on real data.
Our work, however, demonstrates that even in such cases the Residual Jacobians align in their top subspaces and, result, simple ODEs emerge within these subspaces. We also provide theoretical justification for this claim through the Unconstrained Jacobians Model.
Optimal Transport Gai and Zhang 202 view ResNets aiming transport input distribution to a label distribution, through a geodesic curve in the Wasserstein space, induced by the optimal transport map. They provide empirical evidence support their claim, showing that intermediate representations are equidistant on straight line induced by the optimal transport map, and comment: "Though ResNet approximates the geodesic curve better than plain network, it may not be a perfect implement in high-dimensional space due to its layer-wise heterogeneity."
Our research demonstrates that intermediate representations lie line result of the Jacobian singular vectors aligning (RA2) and the Jacobian singular values scaling inversely with depth (RA4) and, in fact, due to the absence of purported "layer-wise heterogencity."
Analysis of Deep Linear Networks Mulayoff and Michaeli (2020) proved that training deep linear network, with quadratic loss, and no weight decay, necessarily converges to the flattest of all minima. At this optimum point, the spectral norm of the input-feature Jacobian increases exponentially with depth, and the singular vectors of consecutive weight matrices align.
Our theoretical study considers ResNets, trained with binary cross-entropy loss, and weight decay. Instead of assuming that training has reached a flat minimum, we analyze the Unconstrained Jacobians Model, on the classification boundary in the input space, and prove phenomena that we have observed through experiments on nonlinear ResNets.
Analysis of Weights Cohen et al. [021] studied the scaling behavior of trained weights in deep residual networks. Our study complements theirs by focusing on examining the Residual Jacobians of ResNet architectures.
6 Conclusion
In this paper, we offer a detailed empirical examination of the ResNet architecture, a model that has seen extensive application across broad spectrum of deep learning domains. Our primary aim was to demystify the mechanisms that contribute to ResNet's 's remarkable success, a topic that, to date, remains an enigma within the deep learning community.
Our investigation has led to the discovery of a consistent phenomenon, which we have termed RA. The characteristics of RA were observed across wide array of benchmark datasets, canonical architectures, and hyperparameters, demonstrating its general applicability. Moreover, we conducted counterfactual studies that underscored the critical role of skip connections in the emergence of RA and the effect of the number of classes. an attempt to theoretically ground the emergence of RA, we proposed the use of an innovative mathematical abstraction the Unconstrained Jacobians Model, specifically in the context of binary classification with cross-entropy loss.
Our exploration has not only shed light on the intricate mechanisms driving ResNets' performance but also points to connections with the recent phenomenon of layer-wise Neural Collapse. Further- more, our findings pave the way for future research in the understanding of existing regularization methods, the design of novel architectures, the development of model compression techniques, and the theoretical investigation of generalization.
10

---

Acknowledgments and Disclosure of Funding
We thank Kirill Serkh for inspiring discussions and feedback on the manuscript. We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), [funding reference number 512236 and 512265]. This research was enabled in part by support provided by Com- pute Ontario (ttp:///www.ccompuutnnnnrcr.cc and Compute Canada (ttp://www..oomputeaaddaaaa
References
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Alain-Sam Cohen, Rama Cont, Alain Rossier, and Renyuan Xu. Scaling properties of deep residual networks. In International Conference on Machine Learning. pages 2039-2048. PMLR, 2021.
Weinan E and Stephan Wojtowytsch. On the emergence of tetrahedral symmetry in the final and penultimate layers neural network classifiers. arXiv preprint arXiv:2012.05440,, 2020.
Stanislaw Jastrz Ebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio.
Residual connections encourage iterative inference. In International Conference on Learning Representations, 2018
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. arXiv preprint arXiv: 1556, 2019.
Kuo Gai and Shihua Zhang. mathematical principle of deep learning: Learn the geodesic curve in the wasserstein space. arXiv preprint arXiv:2102.09225,, 2021.
Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. On the implicit bias towards minimal depth of deep neural networks, 2022.
Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.0777.. 2016.
XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity and dynamics on the central path. In International Conference on Learning Representations, 2021.
Soufiane Hayou. On the infinite-depth limit of finite-width neural networks. arXiv preprint arXiv:2210. 006888 2022.
Hangfeng He and Weijie J. Su. law of data separation in deep learning, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645. Springer, 2016b.
Jeremy Howard. GitHub fastai/imagenette: smaller subset of 10 easily classified classes from Imagenet, and a little more French - github.com. setps://gitain com comm/fastaaa/mmageetta. [Accessed 14-May-2023).
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth, 2016.
Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani. Neural collapse: review on modelling principles and generalization. arXiv preprint arXiv:2206.04040 2022.
11

---

Huamin Li, George C. Linderman, Arthur Szlam, Kelly P. Stanton, Yuval Kluger, and Mark Tygert. Algorithm 971: An implementation of randomized algorithm for principal component analysis. ACM Trans. Math. Softw., 43(3), jan 2017. ISSN 0098-3500. doi: 10. 10.1145/3004053. URL
https://dott org/10 10. 30044033
Mufan Li, Mihai Nica, and Dan Roy. The future is log- gaussian: Resnets and their infinite-depth-anre and width limit at initialization. Advances in Neural Information Processing Systems, 34:7852-7866 2021.
Mufan Bill Li, Mihai Nica, and Daniel M Roy. The neural covariance sde: Shaped infinite depth-and- width networks at initialization. arXiv preprint arXiv:2206.02768. 2022.
Sihan Li, Jiantao Jiao, Yanjun Han, and Tsachy Weissman. Demystifying resnet. arXiv preprint arXiv:1611.01188 2016.
Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, and Qing Qu. Principled and efficient transfer learning of deep models via neural collapse, 2023.
Andrew Liaw, Jia- Hao Hsu, and Chung- Hsinnn Wu. Ensemble of one model: Creating model variations for transformer with layer permutation. In 2021 Asia-Pacijs Signal and Information Processing Association Annual Summil and Conference (APSIPA ASC), pages 1026-1030. IEEE, 2021.
Hya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608. 03983, 2016.
Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint arXiv:2012.08465; 2020.
Yiping Lu, Chao Ma, Yulong Lu, Jianfeng Lu, and Lexing Ying. A mean field analysis of deep resnet and beyond: Towards provably optimization via overparameterization from depth. In International Conference Machine Learning, pages
PMLR, 2020.
Hector Miranda and Robert Thompson. A trace inequality with a subtracted term. Linear algebra and its applications, 185: 185-1651, 1993.
Leon Mirsky, A trace inequality of john von neumann. Monatshefte für mathematik, 79(4)3303-306 1975.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11613 2020.
Rotem Mulayoff and Tomer Michaeli. Unique properties of flat minima in deep networks. In International Conference on Machine Learning, pages 7108-7118. PMLR, 2020.
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study, arXiv preprint arXiv:1802.08760, 2018.
Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra, 2020.
Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via convolutional sparse coding. The Journal of Machine Learning Research, 18(1):2887-2938, 2017.
Vardan Papyan, X. Y. Han, and David I.. Donoho. Prevalence of Neural Collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXiv:2101.0000 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pages 234-241. Springer, 2015.
12

---

Kai Rothauge. Zhewei Yao, Zixi Hu, and Michael W. Mahoney. Residual networks as nonlinear systems: Stability analysis using linearization. 2019.
Michael E Sander, Pierre Ablin, and Gabriel Peyré. Do residual neural networks discretize neural ordinary differential equations? arXiv preprint arXiv:2205. 14612, 2022.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CoRR, 0000/5505000887,,. 2015. URL http::////aaxii.oraaaaaa///// 100000
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In international conference on machine learning (ICMI., 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances neural information processing systems, 29, 2016.
Peng Wang, Huikang Liu, Can Yaras, Laura Balzano, and Qing Qu. Linear convergence analysis of
neural collapse with unconstrained features. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop).
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1492-1500, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-200 298344, 2021.
13

---

A (RA2+3+4) Imply (RA1)
Theorem 3.1. For binary classification, a pre-activation ResNet, assuming the Jacobian lineariza- tions are exact and satisfy [RA2+3+4), then (RA1) holds for the intermediate representations.
Proof. In a pre-activation ResNet,
h++1 h4 + F(hi W.).
Since Jacobian linearizations are exact, we have:
h = (T+ Ji)hs
Recall, the singular value decomposition of J, is given by
J; = U.S.V,
where U2 and V; are the respective left and right singular vector matrices, and S is the singular value matrix. Invoking (RA2) and (RA3), for some matrix Up and for any block i,
J; = UUUUU
and therefore
h4+1 = (1+ U,S,U ).
Applying recursively the above equality leads
k-1 [I
1-1 II Si)
((T+UUSSU))
Un
Up hn-
(2)
For binary classification, (RA3) implies the Jacobians are rank and therefore
k-1 II + (1a+) Uoinn + (I Uo.
h4 = U0,1
where Jo.j is the jth column of Ug. According to (RA4),
St1 =
and
k-1
1-1 (1+ II (1+ k.
II
1=1
Substituting the above into Equation 2 we obtain
h = ,U, hy + (I Uo, (k - 1)U. Vo, + h1.
This proves that the intermediate representations of a given input are equispaced on a line embedded in high dimensional space, i.e., (RA1).
B Unconstrained Jacobians Model Leads to RA
We start by providing motivation for the unconstrained Jacobians problem introduced in the main text. Assume a training example, x, is situated next to a point on the classification boundary, denoted by Tmid, satisfying {Timia W) 0. Performing a Taylor expansion of the logits of x around Tmid yields
0f(;; W)
f(x; W) f(rmia; W)
A + h(x, Fmia)
Ox
Os(x;W)
Ar + 10(2,, Xmii);;
(3)
dx
14

---

where A, = Zmiid and (2, Tmid) accounts for the approximation error, which is O(1IA=11) and assumed be negligible in our analysis.
Recall the loss associated with training a ResNet:
())(aa X IW11
minimize (Wa
Σ
Substituting Equation (3) into the above, neglecting the approximation error, and considering only the objective associated with the training sample and its label y, we get
(0ff @f(lz: W)
L
dx
2
By using the chain rule, we can then obtain the following
( ac(hr+1; WL+1) L
[1(++ 0F(h;; i=1
A ==1
mill + X 1 (5)
A
ShL+1
Oh,
where all the Jacobians are evaluated at the point 2mid- This naturally gives rise the following definition, as introduced in the main text.
Definition 3.2 (Unconstrained Jacobians Model). Given a fixed input As RD and its label E (+1, }1). find matrices J; E RRDxD iii I, and vector w E TRD that
I. L(w
minimize W.
II + J.)
J
In the main text, we stated the following theorem.
Theorem 3.3. For binary classificution, There exists globul optimum of the Unconstrained Jaco- bians Model where the top Jucobian singular vectors are aligned (RA2), all Jacobians are rank one, analogous to (RA3), and the top Jacobian singular values are equal, analogous to (RA4).
Proof. Throughout the proof, we assume, without loss of generality, that the label is y = 1. Using the cyclic property of the trace, the logit, w I + Jz) equals
tr Aew w II J.) }
Denoting the power set of all natural numbers between and L by P(L), the above can be expressed as follows:
Σ SEP(L)
tr A
J4
Each element in the above summation can be upper bounded through the following theorem (a generalization of Von Neumann's trace inequality [Mirsky 1975) to the product of more than two real matrices).
Theorem 2 (Miranda and Thompson 1993].. Let Ai,..., Am be matrices with real entries! Take the singular values of A to be $ (A₂) ... 2 sn(A)) > 0, for = 1,..., m, and denote Sj diag(s (Aj).... s, (A₂)). Then, the matrices PI...., P., range over all possible rotations, ie., the special orthogona group SO(n),
sup Pi..... (SO(a)
Lr(A, Am Pm))
1-1 m
EI ==
s,(A,) + Isign deg And II s (A,).
j=1
'oor simplicity, we ignore the input transformation that maps input A., to first representation /t1 _ by simply assuming A. E R..
Assume the convention that the singular values positive.
15

---

Moreover, assuming sign det A1 ..) = 1,
sup tr(A1 Pm) tr S Pl Pm ESO(n)
We will continue our proof using contradiction. Suppose all existing global optima of the uncon- strained Jacobians problem consist of Jacobians that do not have aligning singular vectors, or do not have equal singular values, singular value decomposition, we have
T. and w. Using the not rank 1. Then take any solution (J.)
Jj = U.S.V for i= 1......4 L,
and
A UL+IS
Then Theorem implies
Σ sEP(L)
lr
II iEa
Ji
Σ NEP(L)
Ir S II S
EES
= lr
S1.t1
[I + Si) [=1
For all E P(L). the inequality becomes equality the singular vectors of all the Jacobians align with those of T. and once the vector is chosen to be proportional to A, (so that the matrix A,w" is symmetric). The implication of the steps thus is that one can increase, or least keep constant the logit, and consequently reduce, or at least keep constant the loss by simply aligning the singular vectors the Jacobians. addition, since the regularization term || = tr (S)) this change of Jacobians does not affect the regularization terms.
Notice that Azw' a rank one matrix and so S has single non-zero diagonal entry. Furthermore, the matrices S,, for < < L, are all diagonal. As such, we can zero out all their other diagonal entries and leave single non-zero entry at the location that SL+1 has one, which does not affect the logits but reduces the regularization terms.
Using the inequality of arithmetic and geometric means this only non-zero entry 8; of every diagonal matrix S; gives
8 II + st) S 8111
+
The implication of the above inequality is that, once the singular vectors of the Jacobians are aligned, one can further increase the logits and reduce the loss by averaging all the top singular values, 8; for < S L, and forcing them be equal. Furthermore, since || tr 5 s is convex, by Jensen's 's inequality, averaging the singular values only decreases the value of the Jacobian regularization.
All in all, we obtain higher, or at least no lower logit, and lower, or at least no higher loss when all singular vectors aligned, all top singular values are equal and all other singular values zero, which contradicts the statement that no global optima of the unconstrained Jacobians problem satisfies all of these conditions.
16

---

C
Additional Empirical Evidence
C.1 (RA1)
20
15
10
5
-15
-10
-
0
Figure Fully-connected ResNet34 (Type 1 model) trained on MNIST.
10
5
0
-5
-10
-15
-10.0
-7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
Figure 9: Fully-connected ResNet34 (Type model) trained on FashionMNIST:
17

---

5
-5
-10
-15
-20
-25
-20
-15
-10
-5
10
15
Figure 10: Fully-connected ResNet34 (Type model) trained CIFAR10.
50
-50
-100
-150
-100
-80
-60
-40
-20
0
20
40
60
Figure 11: Convolutional ResNet34 (Type model) trained on MNIST.
18

---

50
-50
-100
-100
-50
0
50
100
Figure 12: Convolutional ResNet34 (Type 2 model) trained on FashionMNIST.
125
100
75
50
25
0
-25
-50
-75
-150
-100
-50
50
100
Figure 13: Convolutional ResNet34 (Type 2 model) trained on CIFAR10.
19

---

60
40
20
0
-20
-40
-60
-40
-20
0
20
40
Figure 14: Convolutional ResNet34 with downsampling (Type model) trained MNIST.
60
40
20
-20
60
-40
-20
20
Figure 15: Convolutional ResNet34 with downsampling (Type model) trained on FashionMNIST..
20

---

20
10
0
-10
-20
60
400
-20
20
40
60
Figure 16: Convolutional ResNet34 with downsampling (Type 3 model) trained on CIFAR10.
21

---

C.2
(RA2): : U.KKK.V
Figure 17: Fully-commected ResNet34 (Type model) trained on MNIST.
22

---

Figure 18: Fully-connected ResNet34 (Type model) trained MNIST.
23

---

Figure 19: Fully-connected ResNet34 (Type model) trained on CIFAR10.
24

---

Figure 20: Fully-connected ResNet34 (Type model) trained CIFAR100.
25

---

Figure 21: Convolutional ResNet34 (Type 2 model) trained on MNIST.
26

---

Figure 22: Convolutional ResNet34 (Type 2 model) trained on MNShionMNIST..
27

---

Figure 23: Convolutional ResNet34 (Type 2 model) trained on CIFAR10.
28

---

Figure 24: Convolutional ResNet34 (Type 2 model) trained on CIFAR100.
29

---

Figure 25: Convolutional ResNet34 (Type 2 model) trained on ImageNette
30

---

Figure 26: Convolutional ResNet34 with downsampling (Type model) trained on MNIST.
31

---

Figure 27: Convolutional ResNet34 with downsampling (Type 3 model) trained on FaShionMNIST..
32

---

Figure 28: Convolutional ResNet34 with downsampling (Type 3 model) trained on CIFAR10.
33

---

Figure 29: Convolutional ResNet34 with downsampling (Type 3 model) trained on CIFAR100.
34

---

Figure 30: Convolutional ResNet34 with downsampling (Type 3 model) trained on ImageNette.
35

---

C.3
(RA2): : J.U
Figure 31: Fully-commected ResNet34 (Type model) trained on MNIST.
36

---

Figure 32: Fully-connected ResNet34 (Type model) trained MNIST.
37

---

Figure 33: Fully-connected ResNet34 (Type model) trained on CIFAR10.
38

---

Figure 34: Fully-connected ResNet34 (Type model) trained CIFAR100.
39

---

Figure 35: Convolutional ResNet34 (Type 2 model) trained on MNIST.
40

---

Figure 36: Convolutional ResNet34 (Type 2 model) trained on MaShionMNIST.
41

---

Figure 37: Convolutional ResNet34 (Type 2 model) trained on CIFAR10.
42

---

Figure 38: Convolutional ResNet34 (Type 2 model) trained on CIFAR100.
43

---

Figure 39: Convolutional ResNet34 (Type 2 model) trained on Image
44

---

Figure 40: Convolutional ResNet34 with downsampling (Type model) trained on MNIST.
45

---

Figure 41: Convolutional ResNet34 with downsampling (Type 3 model) trained on FNShionMNIST..
46

---

Figure 42: Convolutional ResNet34 with downsampling (Type 3 model) trained on CIFAR10.
47

---

Figure 43: Convolutional ResNet34 with downsampling (Type 3 model) trained on CIFAR100.
48

---

Figure 44: Convolutional ResNet34 with downsampling (Type 3 model) trained on ImageNette.
49

---

C.4
(RA3) and (RA4)
2.5
15.0
2.0
1 Singular Value
12.5
Singular Value
10.0
7.5
1.0
5.0
0.5
2.5
0.0
Depth
Depth
Figure 45: Fully-connected ResNet34 (Type model) trained on MNIST.
15.0
2.0
1/ Singular Value
12.5
Singular Value
10.0
7.5
1.0
5.0
0.5
2.5
0.0
0.0
10
15
10
15
Depth
Depth
Figure 46: Fully-connected ResNet34 (Type model) trained on FashionMNIST.
Singular Value
8
Singular Value
1.0
0.5
0.0
10
15
10
15
Depth
Depth
Figure 47: Fully-connected ResNet34 (Type model) trained on CIFAR10.
50

---

3.5
3.0
/ Singular Value
Singular Value
2.5
2.0
1.5
1.0
0.5
0.0
10
15
10
15
Depth
Depth
Figure 48: Fully-connected ResNet34 (Type model) trained CIFAR100.
1 Singular Value
10
15
Depth
Figure 49: Convolutional ResNet34 (Type model) trained on MNIST.
1/ Singular Value
3.
Depth
Figure 50: Convolutional ResNet34 (Type model) trained on FaShionMNIST.
51

---

5
1/ Singular Value
4
3
10
15
Depth
Figure 51: Convolutional ResNet34 (Type 2 model) trained on CIFAR1O.
1/ Singular Value
4
3
2
1
Depth
Figure 52: Convolutional ResNet34 (Type 2 model) trained on CIFAR100.
1/ Lingullrr Value
10
15
Depth
Figure 53: Convolutional ResNet34 (Type model) trained on Imate
52

---

D Broader Impacts
This work presents foundational research and do not foresee any potential negative societal impacts.
53


Reviews:
[{"review_id": "5JZUVfZLFw", "reviewer": "Reviewer_EjRc", "summary": "This paper investigates the ResNet architecture, a popular deep-learning model known for its improved performance through skip connections. The authors aim to uncover the underlying mechanisms behind its success. They conduct an empirical study by linearizing the residual blocks of ResNet using Residual Jacobians and analyzing their singular value decompositions. The measurements and analysis conducted in the study reveal a phenomenon called Residual Alignment (RA), which is characterized by four fundamental properties. First, intermediate representations of a given input are evenly distributed on an embedded line in the space. Second, the top left and right singular vectors of Residual Jacobians align with each other and across different depths. Third, Residual Jacobians are at most rank C for fully-connected ResNets, where C represents the number of classes. Lastly, the top singular values of Residual Jacobians decrease inversely with depth. The study consistently demonstrates the occurrence of RA in ResNet models that generalize well. This phenomenon holds for both fully-connected and convolutional architectures across various depths and widths and different numbers of classes on benchmark datasets. However, RA ceases to occur when skip connections are removed. The authors also propose a novel mathematical model where RA is present. The findings suggest a strong alignment between residual branches in ResNet, imparting a rigid geometric structure to the intermediate representations as they progress linearly through the network until they reach the final layer, where Neural Collapse occurs.", "questions": "* It would have been great to demonstrate the relationship between RA and Neural Collapse a little bit more through experiments.\n\n* Exploring the behavior of the singular values’ distribution can provide more insights into RA.\n", "limitations": "The authors do not provide potential limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: Using the idea of linearization of the Residual Jacobian is commonly known for exploring the behavior of Residual Networks during the training and even pre-training. The authors use the idea of the Unconstrained Jacobian Model, which is interesting but also limited to a binary classification task. Still, it is novel to the best of my knowledge. \n\nQuality: The motivation and idea are clearly defined, and experiments support the RA phenomena.\n\nClarity: The paper would benefit from improved organization, such as relocating the related work section from section 5 to section 3 or 2. Additionally, the appendix appears disorganized, with figures located randomly. Other than these, it is written very well and easy to follow.\n\nSignificance: The study's results demonstrate the presence of the Residual Alignment (RA) phenomenon in the singular value decomposition (SVD) of the linearized Residual Jacobian. The mathematical analysis further confirms the occurrence of RA in binary classification. However, the paper lacks a deeper exploration of RA (2-4) and other potential insights.\n", "weaknesses": "Linearization of the Residual Jacobian and analysis of its SVD decomposition: Despite being a valid idea, is not novel. The authors could also potentially investigate the distribution of singular values with the help of random matrix theory. These singular values depend on the input feature vectors (or hidden representations), but this dependency is not explored.\n\nLimited applicability to binary classification: The use of Unconstrained Jacobian Models, while interesting, is deemed limited in the context of binary classification tasks. The paper does not explore its potential beyond this specific task.\n\nOrganization and clarity: The paper would benefit from improved organization, particularly in the placement of the related work section, which could be better situated in an earlier section. The appendix also lacks clear structure and organization, with figures placed randomly throughout.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude to the reviewer for the time and effort they put into the assessment of our paper as well as for their insightful comments and critiques. In what follows, we address each of the points raised.\n\n> Linearization of the Residual Jacobian and analysis of its SVD decomposition: Despite being a valid idea, is not novel.\n\nThank you for bringing this to our attention. Indeed, [1] had previously proposed the linearization of residual Jacobians, and regrettably, we failed to recognize their contribution. We have now duly referenced their work in our manuscript.\n\nWe would also like to emphasize that our novelty does not lie in the linearization of the residual Jacobians, as [1] had already explored this aspect. Instead, our original contribution centers around the discovery of the alignment between the residual layers, as well as its theoretical understanding.\n\n[1] Rothauge, K., Yao, Z., Hu, Z., & Mahoney, M. W. (2019). Residual networks as nonlinear systems: Stability analysis using linearization.\n\n\n> The authors could also potentially investigate the distribution of singular values with the help of random matrix theory. These singular values depend on the input feature vectors (or hidden representations), but this dependency is not explored.\n\nThe authors of [1] have greatly advanced our understanding by thoroughly investigating the distribution of singular values in linearized residual Jacobians. Establishing a link between their findings and Residual Alignment represents an intriguing prospect that warrants a very careful and prolonged exploration. Moreover, Random Matrix Theory (RMT) stands out as a powerful tool capable of providing a formal framework to explore such connections. However, this endeavor extends considerably beyond the scope of our current work. We, therefore, leave it for future work.\n\n\n> Significance: The study's results demonstrate the presence of the Residual Alignment (RA) phenomenon in the singular value decomposition (SVD) of the linearized Residual Jacobian. The mathematical analysis further confirms the occurrence of RA in binary classification. However, the paper lacks a deeper exploration of RA (2-4) and other potential insights.\n\nWe have enriched our manuscript with a more comprehensive exploration of RA by intertwining its attributes with established research findings. Specifically, we've connected RA to: \n1. The stability analysis discussed in [1].\n2. The concepts of Neural Collapse and layer-wise Neural Collapse.\n3. The Law of Data Separation principle.\n\nFurthermore, the attached PDF includes two novel experiments that offer deeper insights into the RA phenomenon: \n1. An experiment that showcases the simultaneous occurrence of Neural Collapse and Residual Alignment. \n2. An investigation revealing that the introduction of stochastic depth amplifies the effects of RA. \n\n\n> Limited applicability to binary classification: The use of Unconstrained Jacobian Models, while interesting, is deemed limited in the context of binary classification tasks. The paper does not explore its potential beyond this specific task.\n\nOur goal was to develop a reasonable mathematical model that effectively demonstrates the emergence of Residual Alignment. We concur that expanding this theoretical framework to incorporate multi-class classification and other tasks presents an intriguing opportunity. Nonetheless, we defer a more rigorous, comprehensive exploration of these theoretical aspects to subsequent research.\n\n\n> Organization and clarity: The paper would benefit from improved organization, particularly in the placement of the related work section, which could be better situated in an earlier section. The appendix also lacks clear structure and organization, with figures placed randomly throughout.\n\nThank you for your suggestion. We have now relocated the \"Related Works\" section from its previous position in Section 5 to its new place in Section 3. Furthermore, we have undertaken considerable efforts to refine the organization of the figures within the Appendix, ensuring they are arranged in a more meticulous and systematic manner.\n\n\n> It would have been great to demonstrate the relationship between RA and Neural Collapse a little bit more through experiments.\n\nWe appreciate your wonderful recommendation. The enclosed PDF presents an experiment demonstrating the co-occurrence of Neural Collapse and Residual Alignment.\n\n>\n\nWe again reiterate our appreciation to the reviewer for their detailed recommendations which led to revisions that improved the quality and clarity of our paper. In light of these refinements, should you find it fitting, we would be most grateful for any potential reconsideration of our score."}, {"review_id": "Y5H6WtNOsP", "reviewer": "Reviewer_7wWU", "summary": "The paper tries to analyze the remarkable performance of ResNet architecture and they find the residual alignment phenomenon. The phenomenon is general and they also proposed a mathematical model call the Unconstrained Jacobian Models to theoretically analyze it.", "questions": "1. Is such residual alignment phenomenon a general case in different models with skip connection, like ViT or ResGCN?\n2. Are there any techniques or architectures can cause residual alignment and then lead to better performance besides skip connection?", "limitations": "None", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors find an interesting phenomenon for residual networks called residual alignments. I think it can guide future exploration for module designing, model compression, and regularization techniques.", "weaknesses": "I think the weaknesses of this paper have they only discussed such a phenomenon. It will be great if they can utilize such phenomenon to design some useful techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude to the reviewer for their inspiring questions and careful review of our paper. Below, we respond to each point that was raised.\n\n\n> I think the weaknesses of this paper have they only discussed such a phenomenon. It will be great if they can utilize such phenomenon to design some useful techniques.\n\nWe agree that building upon the identified phenomenon to devise practical techniques is a promising avenue for research. The primary focus of this paper was to shed light on the phenomenon and analyze it theoretically. While we see immense potential in its application, introducing and rigorously testing any new methodologies against established baselines would be beyond the scope of this study. Nonetheless, we look forward to future exploration and research on this front.\n\n\n> Is such residual alignment phenomenon a general case in different models with skip connection, like ViT or ResGCN?\n\nThis is a wonderful question. We asked the exact same question ourselves as well. Addressing this question requires a precise definition of what Residual Jacobians are for such networks. For example, for a ViT, a Residual Jacobian could be defined for a single token or the whole sequence. Depending on the definition, one might or might not observe Residual Alignment. The same question applies to graph neural networks. We hope future work will shed light on this question since it would be extremely interesting to understand the pervasiveness of our observations.\n\n\n> Are there any techniques or architectures can cause residual alignment and then lead to better performance besides skip connection?\n\nThank you for this thought-provoking question. Our experiments consistently revealed that architectures with skip connections were the only ones exhibiting residual alignment. Moving forward, we hope future research investigates whether Residual Alignment manifests in alternative networks. Such investigations are sure to contribute significantly to gaining a deeper understanding of this intriguing phenomenon and evaluating its prevalence. Moreover, we postulate existing regularization techniques, including layer permutation and structured dropout, should strengthen RA. \n\nTo support this claim, the attached PDF showcases an additional experiment we conducted that illustrates how the incorporation of stochastic depth further boosts RA.\n\n\n>\n\nWe would like to express our sincere appreciation to the reviewer for their insightful questions, which resulted in revisions that enhanced the quality of our work. In light of these refinements, should you find it fitting, we would be most grateful for any potential reconsideration of our score.\n\n"}, {"review_id": "CPuQ4oUNiq", "reviewer": "Reviewer_Tohn", "summary": "The paper \"Residual Alignment: Uncovering the Mechanisms of Residual Networks\" explores the underlying mechanisms and success factors of the ResNet architecture, which has gained significant popularity in deep learning. The authors conduct an empirical study by linearizing the residual blocks of ResNet using Residual Jacobians and measuring their singular value decompositions. \n\nThey introduce the concept of Residual Alignment (RA) characterized by four properties: equispaced intermediate representations, alignment of singular vectors, low rank of Residual Jacobians, and inverse scaling of singular values. The phenomenon of RA is observed in well-generalizing models and is absent when skip connections are removed. The authors also propose a mathematical model that demonstrates the occurrence of RA. \n\n*Contribution & significance:* The paper identifies an interesting and seemingly novel phenomenon which they term residual alignment. They show that this observation is on solid theoretical grounds by studying the binary classification problem with cross entropy loss.  This sheds new light on an the role of residual connections in neural nets and will contribute to our overall understanding of this importance architectural component. \n\n*Major writing issues* The main drawback of this paper is its writing quality which is very low at the moment. While I grasped the main ideas by going through the appendix, several of critical pieces of information, such as definition of residual alignment Jacobians ($J_i$'s), are missing from the main text. This will make it very difficult to understand this paper just by reading the main text (which should be the default assumption for a reader). This is quite unfortunate given that the contributions are technically strong & very interesting. Thus, a significant overhaul of the writing seems to be necessary to make this work publishable.\n\n*Clarity about novelty* It would be helpful if authors clarify the new insights and novelty of this work, in contrast to the paper that they numerously cite \"A Mathematical Principle of Deep Learning: Learn the Geodesic Curve in the Wasserstein Space\". Also adding more related literature will also be helpful to the readers.", "questions": "One of my main questions concerns the background of this work. The relevant (recent) literature seems to rather scarce. The only directly related literature currently is \"A Mathematical Principle of Deep Learning: Learn the Geodesic Curve in the Wasserstein Space\". So my question is two folds:\n- Do the authors believe there is no other related literature on this phenomenon of \"emerging singular vector alignment\"? Given the importance of this topic and its relevant to modern models, it seems to me that there should be more \"adjacent\" studies on the topic. \n- Given that the aforementioned paper \"A mathematical ... \" does have quite a bit of overlap with the present work, can the authors make it clear how are their contributions different from this paper? In particular, the residual alignment phenomenon has been extensively discussed in the other paper with largely overlapping settings. It would be helpful for readers to understand what is the additional insights, ideas, proofs, etc, in this present paper. ", "limitations": "yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 1, "contribution": 4, "strengths": "- Identification of Residual Alignment: The paper identifies and characterizes an interesting and seemingly novel phenomenon of Residual Alignment (RA), and highlights its four key properties. The properties of RA are logically interrelated and consistently observed across various architectures. The authors present evidence to support the existence of RA.\n\n- The authors conducted a comprehensive empirical study of the ResNet architecture, linearizing residual blocks using Residual Jacobians and analyzing their singular value decompositions. This approach provides valuable insights into the underlying mechanisms of ResNet models.\n\n- Theoretical Proofs and Abstraction: One of the strongest points of this paper seems to be the theoretical proofs for the emergence of RA in the setting of binary classification with cross-entropy loss. The introduction of the Unconstrained Jacobians Model as a mathematical abstraction adds further depth to the analysis and strengthens the paper's theoretical foundations.\n\n- The proof the main theorem in the appendix is interesting in its own right. They authors first linearize the loss function and then approximate it as a product of $(I+J_i)$ terms where $J_i$'s are the Jacobians. They go on to argue that the loss of a general configurtion (non-aligned singular vectors) is surprisingly bounded by a term that is equivalent to the aligned Jacobians. This is mainly done by invoking an interesting mathematical theorem. While I did not go into full detail of the proof and there might be details/flaws that I missed, the overall proof strategy makes sense and is very interesting, as it breaks down a very intricate problem to a tractable one. (As a side point, I recommend authors to conduct a few more detailed experimentation on the details of these steps, namely by directly. testing the inequalities they arrive at). \n", "weaknesses": "\n# Major issues with writing & presentation of results\n\nOne of the main drawbacks of the current manuscript is its presentation. There are major problems with the flow and writing which hinder my understanding of some details. Another major issues is the lack of enough related works and connection to the existing machine learning & statistics literature. Overall, I would recommend authors to do a major revision of this manuscript to make it publishable and readable. \n\n I tried to compile a list of minor issues with the writing to help with this.\n\n- Fig 1 caption: “s true label and connected to form a trajectory” every connected line goes from input to output? Shouldn’t it have 34 layers (it looks like fewer dots than 34) \n\n- Equation line 34-35: what is $I$? \n- Equation between 57 & 58, what’s the dimensionality of $ \\sigma’(…) $ in this equation? \n- Non capitalised sentences (examples 68, 70, 74, 92, 93, several more ) \n- Jacobians are input-dependent, how are the Figures 2 & 3 made? Do the figures correspond to a single sample input? or averaged across multiple inputs? \n- What does index $i$ in $J_i$ stand for?  This seems to be defined in the appendix but in the main text?\n- line 59 “excluding the contribution from the skip connection.” What does this mean? \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to extend our appreciation to the reviewer for their diligent and thorough evaluation of our paper. In what follows, we address carefully each raised point.\n\n\n> Fig 1 caption: “s true label and connected to form a trajectory” every connected line goes from input to output? Shouldn’t it have 34 layers (it looks like fewer dots than 34) \n\nThank you for this question. Indeed, every connected line goes from the input to the output. Following the convention set by the original ResNet paper, a ResNet34 has $(34-2)/2=16$ residual blocks. The reason for this is that each residual block consists of two weight matrices and two layers are subtracted—one due to the initial convolution and the other at the final classification layer. Therefore, we would expect to see at most $16$ dots instead of $34$. In the plot, we can count around $10$ dots but the rest are too cluttered around the center. This is in agreement with Figure 40 in the Appendix showing the inverse scaling of the top singular value occurs for roughly the last $10$ blocks, which correspond to the $10$ visible dots. We will add a footnote to the manuscript explaining the above.\n\n\n> Equation line 34-35: what is I?\n\nWe apologize for this oversight. In the revised manuscript, we clarify that the function I denotes the Initial layers (comprising a sequence of convolution, batch normalization, and ReLU operations) that occur prior to the first residual block within a ResNet architecture.\n\n\n> Equation between 57 & 58, what’s the dimensionality of in this equation?\n\nThank you for this question. In the revised manuscript we clarified that the Residual Jacobians are square matrices of size $d \\times d$, where $d$ is the number of elements in a ResNet intermediate representation. For instance, for a type 1 model, $d=512$ where $512$ is the width of the network. For a type 2 model, $d=64 \\cdot 16 \\cdot16=16384$ because the representations have $64$ channels and a spatial dimension resolution of $16 \\times 16$. \n\n\n> Non capitalised sentences (examples 68, 70, 74, 92, 93, several more )\n\nWe apologize for the oversight. We have rectified these errors, and all the sentences are now correctly capitalized.\n\n\n> Jacobians are input-dependent, how are the Figures 2 & 3 made? Do the figures correspond to a single sample input? or averaged across multiple inputs?\n\nThe Figures correspond to a single randomly sampled input. We have incorporated this information into the Figures' caption.\n\n\n> What does index $i$ in $J_i$ stand for? This seems to be defined in the appendix but in the main text?\n\nWe apologize for this omission. The index $i$ stands for the index of the residual block, i.e., its depth. We will clarify this in the main text.\n\n\n> line 59 “excluding the contribution from the skip connection.” What does this mean?\n\nFollowing Equation 35, recall a residual block evaluates\n$$h_{i+1} = \\sigma (h_i + \\mathcal{F} (h_i; \\mathcal{W}_i)).$$\n\nNotice that the derivative of the residual block with respect to its input is given by\n$$\\sigma’(h_i + \\mathcal{F} (h_i; \\mathcal{W}_i)) \\left( I + \\frac{\\partial \\mathcal{F} (h_i; \\mathcal{W}_i)}{\\partial h_i} \\right),$$ where $I$ is an identity matrix.\n\nIn line 59, we exclude the contribution from the skip connection, i.e., the term \n$$\\sigma’(h_i + \\mathcal{F} (h_i; \\mathcal{W}_i)) I.$$\n\nWe have added the above clarification to the revised manuscript.\n\n\n> Do the authors believe there is no other related literature on this phenomenon of \"emerging singular vector alignment\"? Given the importance of this topic and its relevant to modern models, it seems to me that there should be more \"adjacent\" studies on the topic.\n\nWe appreciate the reviewer's recognition of the importance and relevance of our work in the context of modern models. In Section 5, we diligently acknowledge and reference all empirical and theoretical research that explores the fundamental mechanisms contributing to the remarkable success of ResNets. As far as we are aware, no one has previously mentioned the alignment of singular vectors of linearized residual blocks.\n\n\n> Given that the aforementioned paper \"A mathematical ... \" does have quite a bit of overlap with the present work, can the authors make it clear how are their contributions different from this paper? In particular, the residual alignment phenomenon has been extensively discussed in the other paper with largely overlapping settings. It would be helpful for readers to understand what is the additional insights, ideas, proofs, etc, in this present paper.\n\nThe related work showed empirically that intermediate representations are equidistant on a straight line, i.e., RA1. However, they did not measure or even mention the remaining RA properties, i.e., RA2-4. In fact, they explicitly say in their work that they believe there is \"layer-wise heterogeneity\" in ResNets, i.e., they expect misalignment between residual blocks.\n\nOur research demonstrates empirically and theoretically, that such equidistant intermediate representations are the result of the Jacobian singular vectors aligning (RA2) and the top Jacobian singular values scaling inversely with depth (RA3-4) and, in fact, due to the absence of purported \"layer-wise heterogeneity.\"\n\nWe have incorporated the above ideas into our revised manuscript.\n\n> \n\nWe are genuinely thankful to the reviewer for their queries. These led to changes that greatly improved the quality of our work. In light of these refinements, should you find it fitting, we would be most grateful for any potential reconsideration of our score.\n"}, {"review_id": "WR3yewXi5u", "reviewer": "Reviewer_jJfi", "summary": "This work discovers the phenomenon of Residual Alignment (RA) in ResNets, whereby the top left and right singular vectors of residual Jacobians align with each other and in between different residual blocks. Through extensive experimental verification as well as novel theoretical frameworks and derivations, the paper shows that RA naturally emerges in ResNets but does not emerge in non-residual networks. By directly linking RA to ResNets, this work sheds new light on the broad success of these architectures.", "questions": "I would be keen to have the authors' opinion on the following points:\n- What happens outside the context of classification (e.g. in regression, object detection, etc)?\n- Could you expand on the connection between RA and layer-wise Neural Collapse? It seems layer-wise Neural Collapse was originally found to be present in both ResNets and non-residual networks (notably VGGs). Given your findings concerning the absence of RA in non-residual networks, does that mean layer-wise Neural Collapse can occur without RA?\n- Could you confirm that RA is still in agreement with the experimental findings in Veit et al., 2016? If so, should Veit et al., 2016 be reinterpreted with a new light?\n- Could you express your view on the connection between RA and stochastic depth? If all residual blocks have aligned singular vectors, shouldn't the effect of stochastic depth be a plain random multiplicative factor of the logits?", "limitations": "If the authors' findings are restricted to the context of classification, perhaps this restriction should be more clearly stressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I believe this paper can have a significant impact.\n- *Significance*: This work provides novel insights that have wide ramifications in a broad range of topics (deep learning theory, regularization, architecture design, model compression). Indeed, the precise mechanisms underpinning the success of ResNets remains a hot topic of research with high stakes. I believe this work makes a rare breakthrough in that direction.\n- *Novelty*: To my knowledge, the discovery of RA and its precise mechanisms is novel. There is also novelty in both the experimental protocols and the theoretical frameworks. These protocols and frameworks might prove useful for the community.\n- *Quality*: The analysis is highly convincing. I feel the experimental and theoretical evidences supporting the analysis are undebatable.\n   - The experimental protocols are innovative and strongly convey the paper's claim, demonstrating both the presence of RA in residual networks and its absence in non-residual networks. \n   - The theoretical analysis is equally innovative and excellent.\n- *Contextualization*: The context of the work is appropriately provided (aside from minor points detailed below), with connections made to previous works, including to the latest developments on layer-wise Neural Collapse. The paper also details potential ramifications of the discovery of RA.\n- *Clarity*: The paper is well-structured, concise, clear. The writing is good. The plots and figures are striking, providing the required evidence to support the claims.\n- *Soundness*: The theoretical proofs are sound (aside from very minor points detailed below).\n- *Reproducibility*: The authors released their code for reproducibility. The code is neat and clean, with high quality standards.", "weaknesses": "I see *no real weaknesses*, only minor points that can be easily addressed.\n\nMinor points:\n- Restriction to the context of classification. RA1 and RA3 are specific to classification, and all the tasks considered fall within such a context. If the findings are restricted to classification, I think this restriction should be clearly stated.\n- I understand that the class-wise equi-spacing on a line of intermediate representations is a consequence of RA (in particular the scaling of top singular values inversely with depth), combined with the increased magnitude of $h_i$ due to the aggregated summation in ResNets. If this reasoning is correct, perhaps it would be worthwhile detailing it?\n- The authors missed some previous works that studied ResNets vs non-residual networks at initialization, notably [1], [2] and [3].\n- There seem to be typos on lines 160, 239, 243. Also shouldn't \"these values\" be replaced by \"the top-1 singular values\" in the caption of Figure 5?\n- It seems Theorem 3.1 is only proved in the context of binary classification. Perhaps this should be stated. Also, I believe there is a term missing in the Equation following line 11 in the Appendix. That term relates to $h_1 - U_{k,1} U_{k,1}^T h_1$ (as can be seen e.g. in the case where all $S_i$ would be equal to zero).\n- Perhaps Theorem 2 in the Appendix should state the convention of positive singular values, meaning that inequality on the $\\sup$ of the trace would always be guaranteed, with equality obtained if the determinant's sign equals 1. Also shouldn't $L+1$ be replaced by $L$ on line 68?\n\nReferences:\n\n[1] The Shattered Gradients Problem: If ResNets are the answer, then what is the question?, D. Balduzzi et al., ICML 2017 \n\n[2] Gradients Explode - Deep Networks Are Shallow - ResNet Explained , G. Philipp et al., ICLR Workshop 2018\n\n[3] Characterizing Well-Behaved vs. Pathological Deep Neural Networks, A. Labatie, ICML 2019", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the time and effort they must have dedicated to providing such a thorough and constructive review. Below, we respond to each point that was raised.\n\n> RA1 and RA3 are specific to classification, and all the tasks considered fall within such a context. If the findings are restricted to classification, I think this restriction should be clearly stated. What happens outside the context of classification (e.g. in regression, object detection, etc)?\n\nIndeed, our measurements of RA are restricted to classification problems. Since we can not presume RA extends to other settings, we will explicitly mention in the manuscript that we are considering multiclass classification problems. \n\nAt this point, we have no intuition as to whether RA would occur for regression or object detection. It would be very interesting to explore the question empirically or even theoretically in future work.\n\n\n> I understand that the class-wise equi-spacing on a line of intermediate representations is a consequence of RA (in particular the scaling of top singular values inversely with depth), combined with the increased magnitude of h_i due to the aggregated summation in ResNets. If this reasoning is correct, perhaps it would be worthwhile detailing it?\n\nThe reasoning is precisely correct! In fact, this is a crucial step in the proof of Theorem 3.1 (second to last equation on page 1 of the Appendix). Following your suggestion, we will move this portion of the proof into the manuscript so as to provide a stronger intuition regarding RA.\n\n\n> The authors missed some previous works that studied ResNets vs non-residual networks at initialization.\n\nThank you. We now cite these relevant works.\n\n\n> There seem to be typos on lines 160, 239, 243. Also shouldn't \"these values\" be replaced by \"the top-1 singular values\" in the caption of Figure 5?\n\nThank you. We have fixed these issues.\n\n\n> It seems Theorem 3.1 is only proved in the context of binary classification. Perhaps this should be stated.\n\nWe apologize for this omission. We now explicitly state this in the Theorem.\n\n\n> Also, I believe there is a term missing in the Equation following line 11 in the Appendix. That term relates to $h_1 - U_{k,1} U_{k,1}^T h_1$.\n\nThank you for catching this mistake. We corrected the proof by adding the missing term. The final conclusion from the theorem does not change as the intermediate representations are still situated on a line, except the line is now shifted by a constant vector.\n\n\n> Perhaps Theorem 2 in the Appendix should state the convention of positive singular values.\n\nThanks for highlighting this. We will add to Theorem 2 the convention of positive singular values.\n\n\n> Also shouldn't L+1 be replaced by L on line 68?\n\nThank you for pointing out this mistake.\n\n\n> Could you expand on the connection between RA and layer-wise Neural Collapse? It seems layer-wise Neural Collapse was originally found to be present in both ResNets and non-residual networks (notably VGGs). Given your findings concerning the absence of RA in non-residual networks, does that mean layer-wise Neural Collapse can occur without RA?\n\nA fascinating question. Indeed, current empirical evidence suggests that layer-wise Neural Collapse can occur without RA. From a different perspective, RA can be thought of as a more structured or regularized version of layer-wise Neural Collapse where Residual Jacobians are forced to align. At this point, it is unclear what benefits such structure or regularization confers to generalization or robustness. We hope future work investigates this intriguing question.\n\nOn a related note, the enclosed PDF presents an experiment demonstrating the co-occurrence of Neural Collapse and Residual Alignment.\n\n\n> Could you confirm that RA is still in agreement with the experimental findings in Veit et al., 2016? If so, should Veit et al., 2016 be reinterpreted with a new light?\n\nRA is indeed in agreement with Veit et al. Consider a simple linear ODE:\n$$x' = Ax.$$\n\nIts solution is given by\n$$x(t) = \\exp(t A) x(0).$$\n\nRecalling the following identity\n$$\\exp(y) = \\lim_{L \\to \\infty} (1+y/L)^L,$$\n\nwe can write\n$$x(t) = \\lim_{L \\to \\infty} (I + t A/L)^L x(0),$$\n\nor equally\n$$x(t) = \\lim_{L \\to \\infty} (I + t A/L) (I + t A/L) \\dots (I + t A/L) x(0),$$\n\nwhere the product is repeated $L$ times. Expanding the brackets shows that $x(t)$ can be thought of as a collection of many paths of various lengths, due to the binomial identity.\n\nVeit et al. state:\n\"...residual networks can be viewed as a collection of many paths.\"\nIgnoring constants, they view ResNets as\n$$x(t) = (I + t A_1/L) (I + t A_2/L) \\dots (I + t A_L/L) x(0).$$\n\nNotice how they do not make any claims about the alignment of the various $A_i$ matrices. We view ResNets as implementing\n$$x(t) = (I + t A/L) (I + t A/L) \\dots (I + t A/L) x(0),$$\n\nwhere all the $A_i$ matrices are aligned. One benefit of this interpretation is that, as $L \\to \\infty$, we can write $x(t)$ in closed form, namely $x(t) = \\exp(t A) x(0)$. \n\nThe above intuition is imprecise since the singular values should scale inversely proportionally with depth. Still, we believe it explains the difference in perspective between Veit et al. and our work.\n\n\n> Could you express your view on the connection between RA and stochastic depth? If all residual blocks have aligned singular vectors, shouldn't the effect of stochastic depth be a plain random multiplicative factor of the logits?\n\nWe agree. However, in practice, RA falls short of perfection. The enclosed PDF presents an experiment demonstrating that introducing stochastic depth enhances RA even further.\n\n> \n\nIn closing, we'd like to express again our gratitude to the reviewer for their feedback which not only improved the current state of our manuscript but also set the stage for intriguing future investigations. In light of these refinements, should you find it fitting, we would be most grateful for any potential increase in our score.\n"}]

Author rebuttal:
Enclosed, please find a page detailing further experiments we carried out in response to the reviewers' inquiries.

Meta-review: The paper makes a significant and clear contribution toward understanding the role of residual connections in ResNets. As shown by well-designed experiments, they perform a specific form of iterative reasoning in which hidden representations are pushed along low-dimensional “conveyor belts”. Reviewers are in agreement that the paper should be accepted. It is my pleasure to also recommend acceptance.
Decision: Accept (poster)