Title: Residual Alignment: Uncovering the Mechanisms of Residual Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Learning

Residual Networks

Neural Networks

Generalization

Spectral Analysis
Abstract: The ResNet architecture has been widely adopted in deep learning due to its significant boost to performance through the use of simple skip connections, yet the underlying mechanisms leading to its success remain largely unknown. In this paper, we conduct a thorough empirical study of the ResNet architecture in classification tasks by linearizing its constituent residual blocks using Residual Jacobians and measuring their singular value decompositions. Our measurements ([code](https://colab.research.google.com/drive/1yKjEg2yF616tnZFAfuN0aQ-E9v3JmyjN?usp=sharing)) reveal a process called Residual Alignment (RA) characterized by four properties:
- **(RA1):** intermediate representations of a given input are *equispaced* on a *line*, embedded in high dimensional space, as observed by Gai and Zhang [2021];
- **(RA2):** top left and right singular vectors of Residual Jacobians align with each other and across different depths;
- **(RA3):** Residual Jacobians are at most rank $C$ for fully-connected ResNets, where $C$ is the number of classes; and
- **(RA4):** top singular values of Residual Jacobians scale inversely with depth.

RA consistently occurs in models that generalize well, in both fully-connected and convolutional architectures, across various depths and widths, for varying numbers of classes, on all tested benchmark datasets, but ceases to occur once the skip connections are removed. It also provably occurs in a novel mathematical model we propose. This phenomenon reveals a strong alignment between residual branches of a ResNet (RA2+4), imparting a highly rigid geometric structure to the intermediate representations as they progress *linearly* through the network (RA1) up to the final layer, where they undergo Neural Collapse.

Residual Alignment:

Abstract

The ResNet architecture has been widely adopted in deep learning due to its significant boost to performance through the use of simple skip connections, yet the underlying mechanisms leading to its success remain largely unknown. In this paper, we conduct a thorough empirical study of the ResNet architecture in classification tasks by linearizing its constituent residual blocks using Residual Jacobians and measuring their singular value decompositions. Our measurements (code) reveal a process called Residual Alignment (RA) characterized by four properties:

( RA1 ) intermediate representations of a given input are equispaced on a line , embedded in high dimensional space, as observed by Gai and Zhang [2021];

( RA2 ) top left and right singular vectors of Residual Jacobians align with each other and across different depths;

( RA3 ) Residual Jacobians are at most rank C for fully-connected ResNets, where

C is the number of classes; and

( RA4 ) top singular values of Residual Jacobians scale inversely with depth.

RA consistently occurs in models that generalize well, in both fully-connected and convolutional architectures, across various depths and widths, for varying numbers of classes, on all tested benchmark datasets, but ceases to occur once the skip connections are removed. It also provably occurs in a novel mathematical model we propose. This phenomenon reveals a strong alignment between residual branches of a ResNet ( RA2+4 ), imparting a highly rigid geometric structure to the intermediate representations as they progress linearly through the network ( RA1 ) up to the final layer, where they undergo Neural Collapse.

1 Introduction

1.1 Background

The Residual Network (ResNet) architecture [He et al., 2016a], a special case of Highway Networks [Srivastava et al., 2015], has taken the field of deep learning by storm, since its proposal in 2015. The incredibly simple architectural modification of adding a skip connection, spanning a set of layers, has become a go-to architectural choice for deep learning researchers and practitioners alike. Initially applied in computer vision, it has since been integrated as a crucial component in biomedical imaging and generative models via the U-Net [Ronneberger et al., 2015], natural language processing through the Transformer [Vaswani et al., 2017], and reinforcement learning as seen in the case of AlphaGo Zero [Silver et al., 2017].

The ResNet architecture first passes an input x through initial layers I (comprising a sequence of convolution, batch normalization, and ReLU operations), depending on trainable parameters W 0 , to

Vardan Papyan

generate an initial representation

This is then processed through a sequence of residual blocks

each refining the previous layer's representation, h i ∈ R d , through a simple residual branch

which depends on trainable parameters W i and an element-wise non-linearity, σ , which is simply an identity mapping, in the case of pre-activation ResNets [He et al., 2016b]. The final block's output, h L +1 , is fed to a classifier,

depending on trainable parameters W L +1 .

In the original ResNet architecture, F and C are compositions of linear transformations, element-wise nonlinearities, and normalization layers. The WideResNet architecture, by Zagoruyko and Komodakis [2016], further incorporates dropout layers into F . Yet more changes are made in ResNext [Xie et al., 2017], where F is computed from the summation of parallel computational branches.

Training a ResNet for classification involves minimizing a loss on a training dataset, { x n , y n } N n =1 , consisting of inputs x n and labels y n plus a weight decay term

where f ( x n ; W ) are the outputs of the classifier C , also called logits, and W are all the parameters.

1.2 Problem Statement

The significant improvement in performance, achieved through the simple addition of a skip connection, has generated interest in understanding the underlying mechanisms of ResNets. Despite this, to this date, no definitive theory or explanation has been widely accepted by the deep learning community for the success of ResNets.

1.3 Method Overview

In this paper, we conduct a thorough empirical study of the ResNet architecture and its constituent residual blocks with the aim of investigating the characteristics of an individual residual block and the relationship between any pair of them. As the residual blocks are nonlinear functions of their inputs, we examine their linearizations through the Residual Jacobian matrices 2 :

Following Equation (1.1), these correspond to the derivative of the residual block with respect to its input, excluding the contribution from the skip connection, σ ′ ( h i + F ( h i ; W i )) ∈ R D × D . In the case of a pre-activation ResNet, these are simply equal to the derivative of the residual branch with respect to its input, i.e., ∂ F ( h i ; W i ) /∂h i . Since the Residual Jacobians are high-dimensional matrices, and likely contain meaningful information only in some of their subspaces, we measure their singular value decomposition (SVD), given by J i = U i S i V ⊤ i , where U i and V i are the respective left and right singular vectors, and S i is the singular value matrix.

1.4 Contributions

We discover a phenomenon called Residual Alignment (RA) , consistently occurring in ResNet models that generalize well, characterized by four properties 3 :

( RA1 ) Intermediate representations of a given input are equispaced on a line , embedded in high dimensional space, as shown in Figure 1 and observed by Gai and Zhang [2021];

( RA2 ) Top left and right singular vectors of Residual Jacobians align with each other and across different depths, as observed in Figure 2;

( RA3 ) Residual Jacobians are at most rank C for fully-connected ResNets, where C is the number of classes, as illustrated in Figures 3 and 4; and

( RA4 ) Top singular values of Residual Jacobians scale inversely proportional with depth, as depicted in Figure 4.

The properties are interrelated and, in fact, ( RA1 ) can be logically derived from the other properties, as demonstrated in Section 3.

As a further contribution, in section B of the Appendix we prove theoretically the emergence of RA under the setting of binary classification with cross-entropy loss. Our proof relies on a novel mathematical abstraction called the Unconstrained Jacobians Model , in which the Residual Jacobians are optimized directly, so as to minimize the loss, and are not constrained by the architecture and parameters of the residual branches. This mathematical abstraction is motivated by recent theoretical works on Neural Collapse, as discussed in Section 5.

1.5 Results Summary

Our empirical investigation, presented in the main text and the Appendix, consistently identifies RA across an extensive range of:

Architectures: standard ResNets (convolutional layers with progressively increasing channels, interspersed with downsampling layers) as well as simpler designs (fully-connected layers);

Datasets: MNIST, FashionMNIST, CIFAR10, CIFAR100, and ImageNette [Howard]; and

Hyperparameters: network depth and width.

In addition to our main findings, we include an experiment showing the co-occurrence of ( RA ) and Neural Collapse in Figure 7. We also performed three counterfactual experiments that show:

If the number of classes in the dataset is increased, the singular vector alignment occurs in a higher dimensional subspace (Figure 3);

If stochastic depth [Huang et al., 2016] is incorporated, the singular vector alignment is amplified (Figure 5); and

If the skip connections are removed, RA does not occur (Figure 6);

(a)

(b)

L

(a) 4 classes

(b) 6 classes

(c) 8 classes

(d) 10 classes

(e) 100 classes

2.0

2

10

4

5

Value

1.5

8

6

7

alue

8

9

Val

11

10

Singular

6

lar

1.0

12

13

Singula

14

15

16

4

17

0.5

18

19

20

1

2

0.0

0

5

10

15

5

10

15

Depth

Depth

(a)

U

⊤

j,

30

J

i

V

j,

30

, regular Type 3 model.

(b)

U

⊤

j,

30

J

i

V

j,

30

, Type 3 model with stochastic depth.

(a)

(b)

(a) NC1:

activation

col-

lapse.

(b) NC2:

equinorm class

means.

(c) NC2: equinorm classi-

fiers.

(d) NC2: maximal equian-

gularity of class means.

(e) NC2: maximal equian-

gularity of classifiers.

(f) NC3: self duality.

(g) NC4:

convergence to

nearest class center.

(h)

U

⊤

j,

30

J

i

V

j,

30

.

101

2 × 10-2

10-1

100

>

10-1

10-2

10-1

0

100

200

300

0

100

200

300

0

100

200

300

0

100

200

300

10-1

100

10-1

10-2

10-1

10-3

10-2

10-2

10-4

D

100

200

300

0

100

200

300

0

100

200

300

2 Methods

2.1 Networks

We train 5 types of models:

Type 2 models consist of 16 basic residual blocks, each containing two convolutional layers with 64 channels, operating on a tensor with a 16 × 16 spatial resolution;

Type 3 models are Type 2 models, but the last 8 basic residual blocks 4 have 128 channels operating on an 8 × 8 spatial resolution instead;

Type 4 models are Type 1 models with 8 instead of 16 residual blocks. Type 4a models have skip connections and Type 4b models do not. These models serve the purpose of verifying that the residual connection causes RA, and RA does not occur once they are removed. 5 ;

Type 5 models are obtained from Type 2 models in the same way that Type 4 models are obtained from Type 1 models. We again have Type 5a and Type 5b, denoting models with and without skip connections, respectively.

In our approach, we have deliberately chosen to use two spatial resolutions instead of the more common three. This decision allows us to have a greater number of residual blocks with the same spatial dimension.

2.2 Datasets

We train these models on the MNIST, FashionMNIST, CIFAR10, CIFAR100, and ImageNette [Howard] datasets. To explore the relationship between RA and the number of classes, we have trained additional models on subsets of these datasets, specifically with 2 , 4 , 6 , and 8 classes.

2.3 Optimization

We train for 500 epochs, using the SGD optimizer, with a batch size of 128 , an initial learning rate of 0 . 1 , the cosine learning rate scheduler [Loshchilov and Hutter, 2016], and a weight decay of 1e -1 for convolutional models and 1e -2 or 5e -2 for fully-connected models. A large weight decay does not negatively impact the generalization performance, as seen in Table 1, but it does strengthen RA.

2.4 Randomized SVD

The Residual Jacobian matrix of a convolutional residual block with 64 channels operating on a spatial dimension of 16 × 16 is of size 16384 × 16384 . Given that SVD scales cubically with the matrix side length, it becomes essential to leverage a more efficient algorithm to expedite computation and mitigate memory consumption. Furthermore, it is worth noting that we do not require the full SVD of the matrix. Our objective is to obtain the top singular values to confirm the trend in ( RA3 ) and ( RA4 ) and the top singular vectors to validate the alignment in ( RA2 ) .

We leverage Algorithm 971 [Li et al., 2017], a randomized SVD algorithm, to accurately compute the desired top singular values and vectors. This method involves a series of matrix multiplications and QR decompositions, requiring a runtime of O ( TD 2 K + TK 3 ) instead of O ( D 3 ) , where T is the number of algorithm iterations, D is the matrix side length, and K is the number of top singular vectors. In our implementation, the T is set to 20 , and the K ranges from 10 to 30, depending on the specific experiment. Through extensive examination, we have verified that this algorithm reliably provides singular values and vectors of sufficient accuracy.

3 Results

3.1 Empirical Results

The empirical results supporting ( RA ) are found in Table 1 and Figures 1, 2, and 4. The counterfactual experiments are in Figures 3, 5, and 6. We also include an experiment showing the co-occurrence of ( RA ) and Neural Collapse in Figure 7. Experimental descriptions and discussions are within the captions. Additional supporting data can be found in the Appendix.

3.2 ( RA2+3+4 ) Imply ( RA1 )

As mentioned in the introduction, the properties of ( RA ) are interconnected, and this relationship is demonstrated through the following theorem:

Theorem 3.1. For binary classification, in a pre-activation ResNet, assuming the Jacobian linearizations are exact and satisfy ( RA2+3+4 ), then ( RA1 ) holds for the intermediate representations.

The proof of this Theorem is deferred to section A of the Appendix.

3.3 Unconstrained Jacobians Model Leads to RA

We propose the following abstraction of the optimization problem in Equation (1).

Definition 3.2 ( Unconstrained Jacobians Model ) . Given a fixed input ∆ x ∈ R D and its label y ∈ { +1 , -1 } , find matrices J i ∈ R D × D , 1 ≤ i ≤ L , and vector w ∈ R D that

In the problem described above, J i again represents the Residual Jacobian of the i -th residual branch and w represents the classifier Jacobian,

It is referred to as the 'Unconstrained Jacobians Model' because the Jacobians are not restricted to any specific form and are not required to be realizable by a set of layers. In the Unconstrained Jacobians Model, the Jacobians are regularized through simple functions. However, in reality, weight decay, dropout, normalization layers, and parallel branches regularize the Jacobians in intricate ways that are hard to capture mathematically.

We prove in section B the Appendix the following theorem:

Theorem 3.3. For binary classification, there exists a global optimum of the Unconstrained Jacobians Model where the top Jacobian singular vectors are aligned ( RA2 ) , all Jacobians are rank one, analogous to ( RA3 ) , and the top Jacobian singular values are equal, analogous to ( RA4 ) .

Here, the top singular values are equal and not decaying as predicted by ( RA4 ) , because the Jacobians are evaluated on the classification boundary instead of on training examples. Therefore, the intermediate representations are not equispaced on a line, as predicted by ( RA1 ) but are rather exponentially-spaced on a line.

4 Discussion

In this section, we pose research questions that emerge from the discovery of RA.

Generalization The discovery of RA offers a vivid analogy for comprehending generalization in deep learning. According to RA, we can envision a ResNet as a system of conveyor belts, where each conveyor carries representations of a specific class in a unified direction at a constant speed. Misclassification occurs when the representation mistakenly steps onto the wrong conveyor belt in the initial layers of the network. This perspective leads us to hypothesize that the first few layers of the network play a vital role in generalization, surpassing the significance of subsequent layers. Consequently, more research should be dedicated towards studying the dynamics of the initial layers.

Neural Collapse As mentioned in Figure 1, there is an intriguing pattern where models exhibit RA concurrently with layer-wise Neural Collapse [Papyan, 2020, Galanti et al., 2022, Li et al., 2023]. The concurrent manifestation of these two distinct events could possibly be more than mere coincidence. It would be interesting to explore if RA could shed light on phenomena related to layer-wise Neural Collapse such as the 'Law of Data Separation' [He and Su, 2022].

RAin Transformers Wehave empirically substantiated the occurrence of RA in ResNets. However, our study has not yet extended to Transformers, which also incorporate residual connections. An intriguing line of inquiry for future work would be to probe whether RA, or a phenomenon akin to it, manifests within these architectures.

Recurrent Architectures Exhibiting RA Recurrent neural networks, Neural ODEs [Chen et al., 2018], and deep equilibrium models [Bai et al., 2019] iteratively apply a layer within a deep neural network. This leads to the following questions:

Do the intermediate representations of these models exhibit RA? If not, can we propose a novel architecture that recurrently applies a computation but does exhibit RA?

Model Compression In our work, we demonstrate that the network can converge to a model that iteratively applies a computation, even without imposing explicit constraints, like the architectures in the previous subsection. This leads us to yet another question:

Can we replicate the original network's performance by distilling the aligning layers into a single layer and iteratively applying it?

Regularization Techniques Existing regularization techniques, including layer permutation [Liaw et al., 2021] and structured dropout [Fan et al., 2019], should strengthen RA. A question arises:

Could RA explain the success of these methods and do they indeed amplify RA?

5 Related Work

Linearization of Residual Jacobians Prior to our work, Rothauge et al. [2019] proposed the linearization of residual Jacobians and investigated the distribution of their singular values. Our work, however, concerns the discovery of the alignment between the Residual Jacobians, as well as its theoretical understanding.

Generalization A thorough empirical investigation by Novak et al. [2018] found that the Frobenius norm of the input-output Jacobian of a network correlates well with generalization. Our research complements theirs by conducting both empirical and theoretical investigations on the Residual Jacobians , which form the input-output Jacobian of a ResNet.

ResNets at Initialization Previous works have contributed significantly to the understanding of the properties of intermediate representations of randomly initialized ResNets, with studies such as Hayou [2022] exploring the infinite-width and finite-width regime and Li et al. [2021, 2022] investigating the infinite-depth and infinite-width regime. Our current research diverges from these previous works by specifically focusing on studying the Residual Jacobians of fully trained ResNets.

Optimization Landscape Li et al. [2016] analyzed the Hessian of the loss function for a ResNet initialized with zero parameters. Additionally, Lu et al. [2020] use a mean-field analysis of ResNets to demonstrate convergence to a global minimum. Their analysis builds upon the observation that a ResNet is similar to a shallow network ensemble, first noted by Veit et al. [2016].

Rather than focusing on the optimization convergence properties of SGD, our goal is to examine the properties of the intermediate representations and Residual Jacobians that emerge during training.

Neural Collapse Recent theoretical work by Mixon et al. [2020], Lu and Steinerberger [2020], E and Wojtowytsch [2020], Poggio and Liao [2020], Zhu et al. [2021], Han et al. [2021], Tirer and Bruna [2022], Wang et al., Kothapalli et al. [2022] analyzed the Neural Collapse phenomenon, discovered by Papyan, Han, and Donoho [2020], through the unconstrained features model and the layer-peeled model. In these, the assumption is made that the last-layer representations, which are fed to a classifier, have the freedom to move independently and are not constrained to be the output of a deep network. Our Unconstrained Jacobians Model takes inspiration from these mathematical models by abstracting away the complex Residual Jacobians and assuming they can be optimized directly.

Unrolled Iterative Algorithm Perspective Greff et al. [2016], Papyan et al. [2017], and Ebski et al. [2018] recognized ResNets as unrolled iterative algorithms performing iterative inference. We build upon this understanding by delving deeper into the empirical and theoretical relationships between the Residual Jacobians.

Neural ODE Chen et al. [2018] introduced the Neural ODE: a continuous-depth, tied-weights ResNet. Initially, it may seem unlikely that the iterations or Jacobians of a traditional ResNet would possess any characteristics associated with Neural ODEs. However, Sander et al. [2022] proved that assuming the initial loss is small and the initial parameters are smooth with depth, linear ResNets converge to a Neural ODE as the number of layers increases. Still, it is unclear if these findings extend to nonlinear ResNets trained on real data.

Our work, however, demonstrates that even in such cases the Residual Jacobians align in their top subspaces and, as a result, simple ODEs emerge within these subspaces. We also provide theoretical justification for this claim through the Unconstrained Jacobians Model.

Optimal Transport Gai and Zhang [2021] view ResNets as aiming to transport an input distribution to a label distribution, through a geodesic curve in the Wasserstein space, induced by the optimal transport map. They provide empirical evidence to support their claim, showing that intermediate representations are equidistant on a straight line induced by the optimal transport map, and comment: 'Though ResNet approximates the geodesic curve better than plain network, it may not be a perfect implement in high-dimensional space due to its layer-wise heterogeneity. '

Our research demonstrates that intermediate representations lie on a line as a result of the Jacobian singular vectors aligning ( RA2 ) and the Jacobian singular values scaling inversely with depth ( RA4 ) and, in fact, due to the absence of purported 'layer-wise heterogeneity.'

Analysis of Deep Linear Networks Mulayoff and Michaeli [2020] proved that training a deep linear network , with a quadratic loss , and no weight decay , necessarily converges to the flattest of all minima. At this optimum point, the spectral norm of the input-feature Jacobian increases exponentially with depth, and the singular vectors of consecutive weight matrices align.

Our theoretical study considers ResNets , trained with a binary cross-entropy loss , and weight decay . Instead of assuming that training has reached a flat minimum, we analyze the Unconstrained Jacobians Model, on the classification boundary in the input space, and prove phenomena that we have observed through experiments on nonlinear ResNets.

Analysis of Weights Cohen et al. [2021] studied the scaling behavior of trained weights in deep residual networks. Our study complements theirs by focusing on examining the Residual Jacobians of ResNet architectures.

6 Conclusion

In this paper, we offer a detailed empirical examination of the ResNet architecture, a model that has seen extensive application across a broad spectrum of deep learning domains. Our primary aim was to demystify the mechanisms that contribute to ResNet's remarkable success, a topic that, to date, remains an enigma within the deep learning community.

Our investigation has led to the discovery of a consistent phenomenon, which we have termed RA. The characteristics of RA were observed across a wide array of benchmark datasets, canonical architectures, and hyperparameters, demonstrating its general applicability. Moreover, we conducted counterfactual studies that underscored the critical role of skip connections in the emergence of RA and the effect of the number of classes. In an attempt to theoretically ground the emergence of RA, we proposed the use of an innovative mathematical abstraction - the Unconstrained Jacobians Model, specifically in the context of binary classification with cross-entropy loss.

Our exploration has not only shed light on the intricate mechanisms driving ResNets' performance but also points to connections with the recent phenomenon of layer-wise Neural Collapse. Furthermore, our findings pave the way for future research in the understanding of existing regularization methods, the design of novel architectures, the development of model compression techniques, and the theoretical investigation of generalization.

Acknowledgments and Disclosure of Funding

We thank Kirill Serkh for inspiring discussions and feedback on the manuscript. We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), [funding reference number 512236 and 512265]. This research was enabled in part by support provided by Compute Ontario (http://www.computeontario.ca) and Compute Canada (http://www.computecanada.ca).

References

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems , 32, 2019.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Alain-Sam Cohen, Rama Cont, Alain Rossier, and Renyuan Xu. Scaling properties of deep residual networks. In International Conference on Machine Learning , pages 2039-2048. PMLR, 2021.

Weinan E and Stephan Wojtowytsch. On the emergence of tetrahedral symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint arXiv:2012.05420 , 2020.

Stanisław Jastrz Ebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations , 2018.

Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909.11556 , 2019.

Kuo Gai and Shihua Zhang. A mathematical principle of deep learning: Learn the geodesic curve in the wasserstein space. arXiv preprint arXiv:2102.09235 , 2021.

Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. On the implicit bias towards minimal depth of deep neural networks, 2022.

Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.07771 , 2016.

XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations , 2021.

Soufiane Hayou. On the infinite-depth limit of finite-width neural networks. arXiv preprint arXiv:2210.00688 , 2022.

Hangfeng He and Weijie J. Su. A law of data separation in deep learning, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016a.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision , pages 630-645. Springer, 2016b.

Jeremy Howard. GitHub - fastai/imagenette: A smaller subset of 10 easily classified classes from Imagenet, and a little more French - github.com. https://github.com/fastai/imagenette . [Accessed 14-May-2023].

Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth, 2016.

Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.04041 , 2022.

Huamin Li, George C. Linderman, Arthur Szlam, Kelly P. Stanton, Yuval Kluger, and Mark Tygert. Algorithm 971: An implementation of a randomized algorithm for principal component analysis. ACM Trans. Math. Softw. , 43(3), jan 2017. ISSN 0098-3500. doi: 10.1145/3004053. URL https://doi.org/10.1145/3004053 .

Mufan Li, Mihai Nica, and Dan Roy. The future is log-gaussian: Resnets and their infinite-depth-andwidth limit at initialization. Advances in Neural Information Processing Systems , 34:7852-7864, 2021.

Mufan Bill Li, Mihai Nica, and Daniel M Roy. The neural covariance sde: Shaped infinite depth-andwidth networks at initialization. arXiv preprint arXiv:2206.02768 , 2022.

Sihan Li, Jiantao Jiao, Yanjun Han, and Tsachy Weissman. Demystifying resnet. arXiv preprint arXiv:1611.01186 , 2016.

Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, and Qing Qu. Principled and efficient transfer learning of deep models via neural collapse, 2023.

Andrew Liaw, Jia-Hao Hsu, and Chung-Hsien Wu. Ensemble of one model: Creating model variations for transformer with layer permutation. In 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) , pages 1026-1030. IEEE, 2021.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016.

Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint arXiv:2012.08465 , 2020.

Yiping Lu, Chao Ma, Yulong Lu, Jianfeng Lu, and Lexing Ying. A mean field analysis of deep resnet and beyond: Towards provably optimization via overparameterization from depth. In International Conference on Machine Learning , pages 6426-6436. PMLR, 2020.

Hector Miranda and Robert C Thompson. A trace inequality with a subtracted term. Linear algebra and its applications , 185:165-172, 1993.

Leon Mirsky. A trace inequality of john von neumann. Monatshefte für mathematik , 79(4):303-306, 1975.

Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11619 , 2020.

Rotem Mulayoff and Tomer Michaeli. Unique properties of flat minima in deep networks. In International Conference on Machine Learning , pages 7108-7118. PMLR, 2020.

Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha SohlDickstein. Sensitivity and generalization in neural networks: an empirical study. arXiv preprint arXiv:1802.08760 , 2018.

Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra, 2020.

Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via convolutional sparse coding. The Journal of Machine Learning Research , 18(1):2887-2938, 2017.

Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of Neural Collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40): 24652-24663, 2020.

Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXiv:2101.00072 , 2020.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computerassisted intervention , pages 234-241. Springer, 2015.

Kai Rothauge, Zhewei Yao, Zixi Hu, and Michael W. Mahoney. Residual networks as nonlinear systems: Stability analysis using linearization. 2019.

Michael E Sander, Pierre Ablin, and Gabriel Peyré. Do residual neural networks discretize neural ordinary differential equations? arXiv preprint arXiv:2205.14612 , 2022.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CoRR , abs/1505.00387, 2015. URL http://arxiv.org/abs/1505.00387 .

Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In international conference on machine learning (ICML) , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems , 29, 2016.

Peng Wang, Huikang Liu, Can Yaras, Laura Balzano, and Qing Qu. Linear convergence analysis of neural collapse with unconstrained features. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop) .

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1492-1500, 2017.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016 . British Machine Vision Association, 2016.

Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems , 34:29820-29834, 2021.

A ( RA2+3+4 ) Imply ( RA1 )

Theorem 3.1. For binary classification, in a pre-activation ResNet, assuming the Jacobian linearizations are exact and satisfy ( RA2+3+4 ), then ( RA1 ) holds for the intermediate representations.

Proof. In a pre-activation ResNet, and therefore

Since Jacobian linearizations are exact, we have:

Recall, the singular value decomposition of J i is given by

where U i and V i are the respective left and right singular vector matrices, and S i is the singular value matrix. Invoking ( RA2 ) and ( RA3 ) , for some matrix U 0 and for any block i ,

Applying recursively the above equality leads to

For binary classification, ( RA3 ) implies the Jacobians are rank 1 and therefore

where U 0 ,j is the j th column of U 0 . According to ( RA4 ) ,

Substituting the above into Equation (2), we obtain

This proves that the intermediate representations of a given input are equispaced on a line embedded in high dimensional space, i.e., ( RA1 ) .

B Unconstrained Jacobians Model Leads to RA

We start by providing motivation for the unconstrained Jacobians problem introduced in the main text. Assume a training example, x , is situated next to a point on the classification boundary, denoted by x mid, satisfying f ( x mid ; W ) = 0 . Performing a Taylor expansion of the logits of x around x mid yields

and

where ∆ x = x -x mid and h ( x, x mid ) accounts for the approximation error, which is O ( ∥ ∆ x ∥ 2 2 ) and assumed to be negligible in our analysis.

Recall the loss associated with training a ResNet:

Substituting Equation (3) into the above, neglecting the approximation error, and considering only the objective associated with the training sample x and its label y , we get

By using the chain rule, we can then obtain the following: 6

where all the Jacobians are evaluated at the point x mid. This naturally gives rise to the following definition, as introduced in the main text.

Definition 3.2 ( Unconstrained Jacobians Model ) . Given a fixed input ∆ x ∈ R D and its label y ∈ { +1 , -1 } , find matrices J i ∈ R D × D , 1 ≤ i ≤ L , and vector w ∈ R D that

In the main text, we stated the following theorem.

Theorem 3.3. For binary classification, There exists a global optimum of the Unconstrained Jacobians Model where the top Jacobian singular vectors are aligned ( RA2 ) , all Jacobians are rank one, analogous to ( RA3 ) , and the top Jacobian singular values are equal, analogous to ( RA4 ) .

Proof. Throughout the proof, we assume, without loss of generality, that the label is y = 1 . Using the cyclic property of the trace, the logit, w ⊤ ∏ L i =1 ( I + J i )∆ x , equals

Denoting the power set of all natural numbers between 1 and L by P ( L ) , the above can be expressed as follows:

Each element in the above summation can be upper bounded through the following theorem (a generalization of Von Neumann's trace inequality [Mirsky, 1975] to the product of more than two real matrices).

Theorem 2 ([Miranda and Thompson, 1993]) . Let A 1 , . . . , A m be matrices with real entries. 7 Take the singular values of A j to be s 1 ( A j ) ≥ · · · ≥ s n ( A j ) > 0 , for j = 1 , . . . , m , and denote S j = diag( s 1 ( A j ) , . . . , s n ( A j )) . Then, as the matrices P 1 , . . . , P m range over all possible rotations, i.e., the special orthogonal group SO( n ) ,

Moreover, assuming sign det( A 1 . . . A m ) = 1 ,

We will continue our proof using contradiction. Suppose all existing global optima of the unconstrained Jacobians problem consist of Jacobians that do not have aligning singular vectors, or do not have equal singular values, or are not rank 1 . Then take any solution { J i } L i =1 and w . Using the singular value decomposition, we have

and

Then Theorem 2 implies

For all s ∈ P ( L ) , the inequality becomes equality once the singular vectors of all the Jacobians align with those of ∆ x w ⊤ and once the vector w is chosen to be proportional to ∆ x (so that the matrix ∆ x w ⊤ is symmetric). The implication of the steps thus far is that one can increase, or at least keep constant the logit, and consequently reduce, or at least keep constant the loss by simply aligning the singular vectors of the Jacobians. In addition, since the regularization term ∥ J i ∥ 2 F = tr { S 2 i } , this change of Jacobians does not affect the regularization terms.

Notice that ∆ x w ⊤ is a rank one matrix and so S L +1 has a single non-zero diagonal entry. Furthermore, the matrices S i , for 1 ≤ i ≤ L , are all diagonal. As such, we can zero out all their other diagonal entries and leave a single non-zero entry at the location that S L +1 has one, which does not affect the logits but reduces the regularization terms.

Using the inequality of arithmetic and geometric means on this only non-zero entry s i of every diagonal matrix S i gives

The implication of the above inequality is that, once the singular vectors of the Jacobians are aligned, one can further increase the logits and reduce the loss by averaging all the top singular values, s i for 1 ≤ i ≤ L , and forcing them to be equal. Furthermore, since ∥ J i ∥ 2 F = tr { S 2 i } = s 2 i is convex, by Jensen's inequality, averaging the singular values only decreases the value of the Jacobian regularization.

All in all, we obtain higher, or at least no lower logit, and lower, or at least no higher loss when all singular vectors are aligned, all top singular values are equal and all other singular values are zero, which contradicts the statement that no global optima of the unconstrained Jacobians problem satisfies all of these conditions.

C Additional Empirical Evidence

C.1 ( RA1 )

20

15

10

5

0

-15

-10

-5

5

10

5

0

-5

-10

-15

-10.0

-7.5

-5.0

-2.5

0.0

2.5

5.0

7.5

5

0

-5

-10

-15

-20

-25

-20

-15

-10

-5

0

5

10

15

50

0

-50

-100

-150

-100

-80

-60

-40

-20

0

20

40

60

50

0

-50

-100

-100

-50

0

50

100

125

100

75

50

25

0

-25

-50

-75

-150

-100

-50

0

50

100

60

40

20

0

-20

-40

-60

-40

-20

0

20

40

60

40

20

0

-20

-60

-40

-20

0

20

20

10

0

-10

-20

-60

-40

-20

0

20

40

60

C.2 ( RA2 ) : U ⊤ j,K J i V j,K

8

15

821

198159

C.3 ( RA2 ) : V ⊤ j,K J i U j,K

C.4 ( RA3 ) and ( RA4 )

2.5

15.0

2.0

Value

e

12.5

6

7

8

6

10.0

Val

1.5

10

gular

11

ngular

12

13

Sing

g

7.5

1.0

14

15

Sing

5.0

1

0.5

2.5

0.0

0.0

5

10

15

5

10

15

Depth

Depth

15.0

2.0

lar Value

12.5

Value

1.5

8

6

10.0

11

10

Singular

12

13

Sing

6

7.5

1.0

14

15

5.0

0.5

2.5

0.0

0.0

5

10

15

5

10

15

Depth

Depth

2.0

10

Singular Value

8

Value

1.5

6

8

6

10

6

1.0

11

12

13

Singula

14

4

15

0.5

1

2

0

0.0

5

10

15

5

10

15

Depth

Depth

3.5

8

3.0

Singular Value

6

6

Value

e

2.5

8

6

10

2.0

11

Singular

12

13

4

1.5

14

15

S

1.0

0.5

0.0

5

10

15

5

10

15

Depth

Depth

6

Singular Value

4

2

1

0

5

10

15

Depth

5

anjen ieinbuis

4

3

2

1

1

5

10

15

Depth

5

anjen jeinbu!s i

4

3

２

1

1

5

10

15

Depth

5

Singular Value

4

3

2

1

0

5

10

15

Depth

Singular Value

3

2

L

1

0

5

10

15

Depth

D Broader Impacts

This work presents foundational research and we do not foresee any potential negative societal impacts.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work discovers the phenomenon of Residual Alignment (RA) in ResNets, whereby the top left and right singular vectors of residual Jacobians align with each other and in between different residual blocks. Through extensive experimental verification as well as novel theoretical frameworks and derivations, the paper shows that RA naturally emerges in ResNets but does not emerge in non-residual networks. By directly linking RA to ResNets, this work sheds new light on the broad success of these architectures.\n\nSTRENGTHS:\nI believe this paper can have a significant impact.\n- *Significance*: This work provides novel insights that have wide ramifications in a broad range of topics (deep learning theory, regularization, architecture design, model compression). Indeed, the precise mechanisms underpinning the success of ResNets remains a hot topic of research with high stakes. I believe this work makes a rare breakthrough in that direction.\n- *Novelty*: To my knowledge, the discovery of RA and its precise mechanisms is novel. There is also novelty in both the experimental protocols and the theoretical frameworks. These protocols and frameworks might prove useful for the community.\n- *Quality*: The analysis is highly convincing. I feel the experimental and theoretical evidences supporting the analysis are undebatable.\n   - The experimental protocols are innovative and strongly convey the paper's claim, demonstrating both the presence of RA in residual networks and its absence in non-residual networks. \n   - The theoretical analysis is equally innovative and excellent.\n- *Contextualization*: The context of the work is appropriately provided (aside from minor points detailed below), with connections made to previous works, including to the latest developments on layer-wise Neural Collapse. The paper also details potential ramifications of the discovery of RA.\n- *Clarity*: The paper is well-structured, concise, clear. The writing is good. The plots and figures are striking, providing the required evidence to support the claims.\n- *Soundness*: The theoretical proofs are sound (aside from very minor points detailed below).\n- *Reproducibility*: The authors released their code for reproducibility. The code is neat and clean, with high quality standards.\n\nWEAKNESSES:\nI see *no real weaknesses*, only minor points that can be easily addressed.\n\nMinor points:\n- Restriction to the context of classification. RA1 and RA3 are specific to classification, and all the tasks considered fall within such a context. If the findings are restricted to classification, I think this restriction should be clearly stated.\n- I understand that the class-wise equi-spacing on a line of intermediate representations is a consequence of RA (in particular the scaling of top singular values inversely with depth), combined with the increased magnitude of $h_i$ due to the aggregated summation in ResNets. If this reasoning is correct, perhaps it would be worthwhile detailing it?\n- The authors missed some previous works that studied ResNets vs non-residual networks at initialization, notably [1], [2] and [3].\n- There seem to be typos on lines 160, 239, 243. Also shouldn't \"these values\" be replaced by \"the top-1 singular values\" in the caption of Figure 5?\n- It seems Theorem 3.1 is only proved in the context of binary classification. Perhaps this should be stated. Also, I believe there is a term missing in the Equation following line 11 in the Appendix. That term relates to $h_1 - U_{k,1} U_{k,1}^T h_1$ (as can be seen e.g. in the case where all $S_i$ would be equal to zero).\n- Perhaps Theorem 2 in the Appendix should state the convention of positive singular values, meaning that inequality on the $\\sup$ of the trace would always be guaranteed, with equality obtained if the determinant's sign equals 1. Also shouldn't $L+1$ be replaced by $L$ on line 68?\n\nReferences:\n\n[1] The Shattered Gradients Problem: If ResNets are the answer, then what is the question?, D. Balduzzi et al., ICML 2017 \n\n[2] Gradients Explode - Deep Networks Are Shallow - ResNet Explained , G. Philipp et al., ICLR Workshop 2018\n\n[3] Characterizing Well-Behaved vs. Pathological Deep Neural Networks, A. Labatie, ICML 2019\n\nQUESTIONS:\nI would be keen to have the authors' opinion on the following points:\n- What happens outside the context of classification (e.g. in regression, object detection, etc)?\n- Could you expand on the connection between RA and layer-wise Neural Collapse? It seems layer-wise Neural Collapse was originally found to be present in both ResNets and non-residual networks (notably VGGs). Given your findings concerning the absence of RA in non-residual networks, does that mean layer-wise Neural Collapse can occur without RA?\n- Could you confirm that RA is still in agreement with the experimental findings in Veit et al., 2016? If so, should Veit et al., 2016 be reinterpreted with a new light?\n- Could you express your view on the connection between RA and stochastic depth? If all residual blocks have aligned singular vectors, shouldn't the effect of stochastic depth be a plain random multiplicative factor of the logits?\n\nLIMITATIONS:\nIf the authors' findings are restricted to the context of classification, perhaps this restriction should be more clearly stressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Residual Alignment: Uncovering the Mechanisms of Residual Networks\" explores the underlying mechanisms and success factors of the ResNet architecture, which has gained significant popularity in deep learning. The authors conduct an empirical study by linearizing the residual blocks of ResNet using Residual Jacobians and measuring their singular value decompositions. \n\nThey introduce the concept of Residual Alignment (RA) characterized by four properties: equispaced intermediate representations, alignment of singular vectors, low rank of Residual Jacobians, and inverse scaling of singular values. The phenomenon of RA is observed in well-generalizing models and is absent when skip connections are removed. The authors also propose a mathematical model that demonstrates the occurrence of RA. \n\n*Contribution & significance:* The paper identifies an interesting and seemingly novel phenomenon which they term residual alignment. They show that this observation is on solid theoretical grounds by studying the binary classification problem with cross entropy loss.  This sheds new light on an the role of residual connections in neural nets and will contribute to our overall understanding of this importance architectural component. \n\n*Major writing issues* The main drawback of this paper is its writing quality which is very low at the moment. While I grasped the main ideas by going through the appendix, several of critical pieces of information, such as definition of residual alignment Jacobians ($J_i$'s), are missing from the main text. This will make it very difficult to understand this paper just by reading the main text (which should be the default assumption for a reader). This is quite unfortunate given that the contributions are technically strong & very interesting. Thus, a significant overhaul of the writing seems to be necessary to make this work publishable.\n\n*Clarity about novelty* It would be helpful if authors clarify the new insights and novelty of this work, in contrast to the paper that they numerously cite \"A Mathematical Principle of Deep Learning: Learn the Geodesic Curve in the Wasserstein Space\". Also adding more related literature will also be helpful to the readers.\n\nSTRENGTHS:\n- Identification of Residual Alignment: The paper identifies and characterizes an interesting and seemingly novel phenomenon of Residual Alignment (RA), and highlights its four key properties. The properties of RA are logically interrelated and consistently observed across various architectures. The authors present evidence to support the existence of RA.\n\n- The authors conducted a comprehensive empirical study of the ResNet architecture, linearizing residual blocks using Residual Jacobians and analyzing their singular value decompositions. This approach provides valuable insights into the underlying mechanisms of ResNet models.\n\n- Theoretical Proofs and Abstraction: One of the strongest points of this paper seems to be the theoretical proofs for the emergence of RA in the setting of binary classification with cross-entropy loss. The introduction of the Unconstrained Jacobians Model as a mathematical abstraction adds further depth to the analysis and strengthens the paper's theoretical foundations.\n\n- The proof the main theorem in the appendix is interesting in its own right. They authors first linearize the loss function and then approximate it as a product of $(I+J_i)$ terms where $J_i$'s are the Jacobians. They go on to argue that the loss of a general configurtion (non-aligned singular vectors) is surprisingly bounded by a term that is equivalent to the aligned Jacobians. This is mainly done by invoking an interesting mathematical theorem. While I did not go into full detail of the proof and there might be details/flaws that I missed, the overall proof strategy makes sense and is very interesting, as it breaks down a very intricate problem to a tractable one. (As a side point, I recommend authors to conduct a few more detailed experimentation on the details of these steps, namely by directly. testing the inequalities they arrive at).\n\nWEAKNESSES:\n# Major issues with writing & presentation of results\n\nOne of the main drawbacks of the current manuscript is its presentation. There are major problems with the flow and writing which hinder my understanding of some details. Another major issues is the lack of enough related works and connection to the existing machine learning & statistics literature. Overall, I would recommend authors to do a major revision of this manuscript to make it publishable and readable. \n\n I tried to compile a list of minor issues with the writing to help with this.\n\n- Fig 1 caption: “s true label and connected to form a trajectory” every connected line goes from input to output? Shouldn’t it have 34 layers (it looks like fewer dots than 34) \n\n- Equation line 34-35: what is $I$? \n- Equation between 57 & 58, what’s the dimensionality of $ \\sigma’(…) $ in this equation? \n- Non capitalised sentences (examples 68, 70, 74, 92, 93, several more ) \n- Jacobians are input-dependent, how are the Figures 2 & 3 made? Do the figures correspond to a single sample input? or averaged across multiple inputs? \n- What does index $i$ in $J_i$ stand for?  This seems to be defined in the appendix but in the main text?\n- line 59 “excluding the contribution from the skip connection.” What does this mean?\n\nQUESTIONS:\nOne of my main questions concerns the background of this work. The relevant (recent) literature seems to rather scarce. The only directly related literature currently is \"A Mathematical Principle of Deep Learning: Learn the Geodesic Curve in the Wasserstein Space\". So my question is two folds:\n- Do the authors believe there is no other related literature on this phenomenon of \"emerging singular vector alignment\"? Given the importance of this topic and its relevant to modern models, it seems to me that there should be more \"adjacent\" studies on the topic. \n- Given that the aforementioned paper \"A mathematical ... \" does have quite a bit of overlap with the present work, can the authors make it clear how are their contributions different from this paper? In particular, the residual alignment phenomenon has been extensively discussed in the other paper with largely overlapping settings. It would be helpful for readers to understand what is the additional insights, ideas, proofs, etc, in this present paper.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper tries to analyze the remarkable performance of ResNet architecture and they find the residual alignment phenomenon. The phenomenon is general and they also proposed a mathematical model call the Unconstrained Jacobian Models to theoretically analyze it.

STRENGTHS:
The authors find an interesting phenomenon for residual networks called residual alignments. I think it can guide future exploration for module designing, model compression, and regularization techniques.

WEAKNESSES:
I think the weaknesses of this paper have they only discussed such a phenomenon. It will be great if they can utilize such phenomenon to design some useful techniques.

QUESTIONS:
1. Is such residual alignment phenomenon a general case in different models with skip connection, like ViT or ResGCN?
2. Are there any techniques or architectures can cause residual alignment and then lead to better performance besides skip connection?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the ResNet architecture, a popular deep-learning model known for its improved performance through skip connections. The authors aim to uncover the underlying mechanisms behind its success. They conduct an empirical study by linearizing the residual blocks of ResNet using Residual Jacobians and analyzing their singular value decompositions. The measurements and analysis conducted in the study reveal a phenomenon called Residual Alignment (RA), which is characterized by four fundamental properties. First, intermediate representations of a given input are evenly distributed on an embedded line in the space. Second, the top left and right singular vectors of Residual Jacobians align with each other and across different depths. Third, Residual Jacobians are at most rank C for fully-connected ResNets, where C represents the number of classes. Lastly, the top singular values of Residual Jacobians decrease inversely with depth. The study consistently demonstrates the occurrence of RA in ResNet models that generalize well. This phenomenon holds for both fully-connected and convolutional architectures across various depths and widths and different numbers of classes on benchmark datasets. However, RA ceases to occur when skip connections are removed. The authors also propose a novel mathematical model where RA is present. The findings suggest a strong alignment between residual branches in ResNet, imparting a rigid geometric structure to the intermediate representations as they progress linearly through the network until they reach the final layer, where Neural Collapse occurs.\n\nSTRENGTHS:\nOriginality: Using the idea of linearization of the Residual Jacobian is commonly known for exploring the behavior of Residual Networks during the training and even pre-training. The authors use the idea of the Unconstrained Jacobian Model, which is interesting but also limited to a binary classification task. Still, it is novel to the best of my knowledge. \n\nQuality: The motivation and idea are clearly defined, and experiments support the RA phenomena.\n\nClarity: The paper would benefit from improved organization, such as relocating the related work section from section 5 to section 3 or 2. Additionally, the appendix appears disorganized, with figures located randomly. Other than these, it is written very well and easy to follow.\n\nSignificance: The study's results demonstrate the presence of the Residual Alignment (RA) phenomenon in the singular value decomposition (SVD) of the linearized Residual Jacobian. The mathematical analysis further confirms the occurrence of RA in binary classification. However, the paper lacks a deeper exploration of RA (2-4) and other potential insights.\n\nWEAKNESSES:\nLinearization of the Residual Jacobian and analysis of its SVD decomposition: Despite being a valid idea, is not novel. The authors could also potentially investigate the distribution of singular values with the help of random matrix theory. These singular values depend on the input feature vectors (or hidden representations), but this dependency is not explored.\n\nLimited applicability to binary classification: The use of Unconstrained Jacobian Models, while interesting, is deemed limited in the context of binary classification tasks. The paper does not explore its potential beyond this specific task.\n\nOrganization and clarity: The paper would benefit from improved organization, particularly in the placement of the related work section, which could be better situated in an earlier section. The appendix also lacks clear structure and organization, with figures placed randomly throughout.\n\nQUESTIONS:\n* It would have been great to demonstrate the relationship between RA and Neural Collapse a little bit more through experiments.\n\n* Exploring the behavior of the singular values’ distribution can provide more insights into RA.\n\nLIMITATIONS:\nThe authors do not provide potential limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}