SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jianing Li
jrobert.li@mail.utoronto.ca
Vardan Papyan
vardan.papyan@utoronto.ca
The ResNet architecture has been widely adopted in deep learning due to its significant boost to performance through the use of simple skip connections, yet the underlying mechanisms leading to its success remain largely unknown. In this paper, we conduct a thorough empirical study of the ResNet architecture in classification tasks by linearizing its constituent residual blocks using Residual Jacobians and measuring their singular value decompositions. Our measurements (code) reveal a process called Residual Alignment (RA) characterized by four properties:
(RA1) intermediate representations of a given input are equispaced on a line, embedded in high dimensional space, as observed by Gai and Zhang [2021];
(RA2) top left and right singular vectors of Residual Jacobians align with each other and across different depths;
(RA3) Residual Jacobians are at most rank  $C$  for fully-connected ResNets, where  $C$  is the number of classes; and
(RA4) top singular values of Residual Jacobians scale inversely with depth.
RA consistently occurs in models that generalize well, in both fully-connected and convolutional architectures, across various depths and widths, for varying numbers of classes, on all tested benchmark datasets, but ceases to occur once the skip connections are removed. It also provably occurs in a novel mathematical model we propose. This phenomenon reveals a strong alignment between residual branches of a ResNet  $(\mathrm{RA2} + 4)$ , imparting a highly rigid geometric structure to the intermediate representations as they progress linearly through the network (RA1) up to the final layer, where they undergo Neural Collapse.
The Residual Network (ResNet) architecture [He et al., 2016a], a special case of Highway Networks [Srivastava et al., 2015], has taken the field of deep learning by storm, since its proposal in 2015. The incredibly simple architectural modification of adding a skip connection, spanning a set of layers, has become a go-to architectural choice for deep learning researchers and practitioners alike. Initially applied in computer vision, it has since been integrated as a crucial component in biomedical imaging and generative models via the U-Net [Ronneberger et al., 2015], natural language processing through the Transformer [Vaswani et al., 2017], and reinforcement learning as seen in the case of AlphaGo Zero [Silver et al., 2017].
The ResNet architecture first passes an input  $x$  through initial layers  $\mathcal{I}$  (comprising a sequence of convolution, batch normalization, and ReLU operations), depending on trainable parameters  $\mathcal{W}_0$ , to
Figure 1: Visualization of Residual Alignment. Intermediate representations of a ResNet34 $^{1}$ , trained on CIFAR10, are projected onto two random vectors. Representations of each individual image are color-coded based on its true label and connected to form a trajectory, so as to showcase their progression throughout the network. Notice the linear arrangement of intermediate representations along with equidistant spacing between representations corresponding to consecutive layers (RA1). Our work shows, this phenomenon results from the alignment of top singular vectors of Residual Jacobians (RA2) and the inverse scaling of top singular values with depth (RA4). It is also noteworthy that the magnitudes of class means significantly increase with depth compared to the within-class variability, indicating the representations undergo layer-wise Neural Collapse [Papyan, 2020, Galanti et al., 2022, He and Su, 2022, Li et al., 2023].
generate an initial representation
$$
h _ {1} = \mathcal {I} (x; \mathcal {W} _ {0}) \in \mathbb {R} ^ {D}.
$$
This is then processed through a sequence of residual blocks
$$
h _ {i + 1} = \sigma \left(h _ {i} + \mathcal {F} \left(h _ {i}; \mathcal {W} _ {i}\right)\right), \quad 1 \leq i \leq L,
$$
each refining the previous layer's representation,  $h_i \in \mathbb{R}^d$ , through a simple residual branch
$$
\mathcal {F} \left(h _ {i}; \mathcal {W} _ {i}\right): \mathbb {R} ^ {D} \rightarrow \mathbb {R} ^ {D},
$$
which depends on trainable parameters  $\mathcal{W}_i$  and an element-wise non-linearity,  $\sigma$ , which is simply an identity mapping, in the case of pre-activation ResNets [He et al., 2016b]. The final block's output,  $h_{L + 1}$ , is fed to a classifier,
$$
\mathcal {C} \left(h _ {L + 1}; \mathcal {W} _ {L + 1}\right): \mathbb {R} ^ {D} \rightarrow \mathbb {R} ^ {C},
$$
depending on trainable parameters  $\mathcal{W}_{L + 1}$
In the original ResNet architecture,  $\mathcal{F}$  and  $\mathcal{C}$  are compositions of linear transformations, element-wise nonlinearities, and normalization layers. The WideResNet architecture, by Zagoruyko and Komodakis [2016], further incorporates dropout layers into  $\mathcal{F}$ . Yet more changes are made in ResNext [Xie et al., 2017], where  $\mathcal{F}$  is computed from the summation of parallel computational branches.
Training a ResNet for classification involves minimizing a loss on a training dataset,  $\{x_{n},y_{n}\}_{n = 1}^{N}$  consisting of inputs  $x_{n}$  and labels  $y_{n}$  plus a weight decay term
$$
\underset {\{\mathcal {W} _ {i} \} _ {i = 1} ^ {L + 1}} {\text {m i n i m i z e}} \frac {1}{N} \sum_ {n = 1} ^ {N} \mathcal {L} \left(f \left(x _ {n}; \mathcal {W}\right), y _ {n}\right) + \frac {\lambda}{2} \| \mathcal {W} \| _ {2} ^ {2}, \tag {1}
$$
where  $f(x_{n};\mathcal{W})$  are the outputs of the classifier  $\mathcal{C}$ , also called logits, and  $\mathcal{W}$  are all the parameters.
The significant improvement in performance, achieved through the simple addition of a skip connection, has generated interest in understanding the underlying mechanisms of ResNets. Despite this, to this date, no definitive theory or explanation has been widely accepted by the deep learning community for the success of ResNets.
$^{1}$ ResNet34 has  $(34 - 2) / 2 = 16$  residual blocks since each residual block consists of two weight matrices and two layers are subtracted—one due to the initial convolution and the other at the final classification layer. Therefore, we would expect to see at most 16 dots. In the plot, we can count around 10 dots but the rest are too cluttered around the center. This is in agreement with Figure C.4 in the Appendix showing the inverse scaling of the top singular value occurs for roughly the last blocks, which correspond to the visible dots.
1.3 Method Overview
In this paper, we conduct a thorough empirical study of the ResNet architecture and its constituent residual blocks with the aim of investigating the characteristics of an individual residual block and the relationship between any pair of them. As the residual blocks are nonlinear functions of their inputs, we examine their linearizations through the Residual Jacobian matrices<sup>2</sup>:
$$
\sigma^ {\prime} (h _ {i} + \mathcal {F} (h _ {i}; \mathcal {W} _ {i})) \frac {\partial \mathcal {F} (h _ {i} ; \mathcal {W} _ {i})}{\partial h _ {i}} \in \mathbb {R} ^ {D \times D}.
$$
Following Equation (1.1), these correspond to the derivative of the residual block with respect to its input, excluding the contribution from the skip connection,  $\sigma^{\prime}(h_i + \mathcal{F}(h_i;\mathcal{W}_i))\in \mathbb{R}^{D\times D}$ . In the case of a pre-activation ResNet, these are simply equal to the derivative of the residual branch with respect to its input, i.e.,  $\partial \mathcal{F}(h_i;\mathcal{W}_i) / \partial h_i$ . Since the Residual Jacobians are high-dimensional matrices, and likely contain meaningful information only in some of their subspaces, we measure their singular value decomposition (SVD), given by  $J_{i} = U_{i}S_{i}V_{i}^{\top}$ , where  $U_{i}$  and  $V_{i}$  are the respective left and right singular vectors, and  $S_{i}$  is the singular value matrix.
1.4 Contributions
We discover a phenomenon called Residual Alignment (RA), consistently occurring in ResNet models that generalize well, characterized by four properties<sup>3</sup>:
(RA1) Intermediate representations of a given input are equispaced on a line, embedded in high dimensional space, as shown in Figure 1 and observed by Gai and Zhang [2021];
(RA2) Top left and right singular vectors of Residual Jacobians align with each other and across different depths, as observed in Figure 2;
(RA3) Residual Jacobians are at most rank  $C$  for fully-connected ResNets, where  $C$  is the number of classes, as illustrated in Figures 3 and 4; and
(RA4) Top singular values of Residual Jacobians scale inversely proportional with depth, as depicted in Figure 4.
The properties are interrelated and, in fact, (RA1) can be logically derived from the other properties, as demonstrated in Section 3.
As a further contribution, in section B of the Appendix we prove theoretically the emergence of RA under the setting of binary classification with cross-entropy loss. Our proof relies on a novel mathematical abstraction called the Unconstrained Jacobians Model, in which the Residual Jacobians are optimized directly, so as to minimize the loss, and are not constrained by the architecture and parameters of the residual branches. This mathematical abstraction is motivated by recent theoretical works on Neural Collapse, as discussed in Section 5.
1.5 Results Summary
Our empirical investigation, presented in the main text and the Appendix, consistently identifies RA across an extensive range of:
Architectures: standard ResNets (convolutional layers with progressively increasing channels, interspersed with downsampling layers) as well as simpler designs (fully-connected layers);
Datasets: MNIST, FashionMNIST, CIFAR10, CIFAR100, and ImageNette [Howard]; and
Hyperparameters: network depth and width.
In addition to our main findings, we include an experiment showing the co-occurrence of (RA) and Neural Collapse in Figure 7. We also performed three counterfactual experiments that show:
For a Type 1 model,  $D = 512$  where 512 is the width of the network. For a Type 2 model,  $D = 16 \times 16 \times 64 = 16384$  because the representations have 64 channels and a spatial dimension resolution of 16.
These properties hold once the Jacobians are evaluated on the training data. If evaluated on other data, different phenomena might emerge. For example, we observed that the growth rate in (RA4) becomes exponential once the Jacobians are evaluated on the classification boundary, where the logits are close to zero.
1. If the number of classes in the dataset is increased, the singular vector alignment occurs in a higher dimensional subspace (Figure 3);
2. If stochastic depth [Huang et al., 2016] is incorporated, the singular vector alignment is amplified (Figure 5); and
3. If the skip connections are removed, RA does not occur (Figure 6);
(b)
Figure 2: (RA2): Top singular vectors of Residual Jacobians align. Subfigure 2a and Subfigure 2b present the alignment of the first 8 blocks and the last 7 blocks, respectively, for a ResNet34 trained on CIFAR100 (Type 3 model in Section 2.1) forwarding a single randomly sampled input. Each subplot  $(i,j)$  illustrates the matrix  $U_{j,30}^{\top}J_{i}V_{j,30}$ , where  $U_{j,30}$  and  $V_{j,30}$  denote the top-30 left and right singular vectors of the Residual Jacobian  $J_{j}$ , respectively, and  $i,j$  are the indices of the residual blocks, i.e., their depth. A distinct diagonal line of intense pixels is apparent in almost every subplot, signifying that the top singular vectors of  $J_{j}$  diagonalize  $J_{i}$ . In simpler terms, this means that the top singular vectors of  $J_{i}$  and  $J_{j}$  align and (RA2) holds. This pattern persists when  $V_{j,30}^{\top}J_{i}U_{j,30}$  is plotted, instead of  $U_{j,30}^{\top}J_{i}V_{j,30}$ , further confirming that the top left and right singular vectors align in accordance with (RA2). Additional visualizations of both matrices, across various models and datasets, are available in subsections C.2 and C.3 of the Appendix. It is crucial to highlight that no alignment exists between the Jacobians at initialization, and the alignment emerges during training.
(a) 4 classes
(b) 6 classes
(c) 8 classes
(d) 10 classes
Figure 3: (RA3): Singular vector alignment occurs in subspace of rank  $\leq \mathbf{C}$ . The figure presents a sequence of subplots that illustrate the matrix  $U_{16,10}^{\top}J_{9}V_{16,10}$ . Here,  $J_{9}$  represents the 9-th Residual Jacobian, while  $U_{16,10}$  and  $V_{16,10}$  correspond to the leading 10 left and right singular vectors, respectively, of the 16-th Residual Jacobian,  $J_{16}$ . These calculations are based on ResNet34 models (Type 1 model in Section 2.1). These models have been trained on specific subsets of the CIFAR10 dataset, comprising of 4, 6, and 8 classes, as well as the complete CIFAR10 and CIFAR100 datasets. Each result is presented in the corresponding Subfigures 3a, 3b, 3c, 3d, and 3e. As the number of classes increases, the alignment of singular vectors occurs in an increasingly higher-dimensional subspace.
(e) 100 classes
(a) (RA3): Residual Jacobians are at most rank  $C$ . The top-10 singular values (orange) noticeably surpass the next 10 (blue), indicating that the Residual Jacobians rank is less than  $C$ .
(b) (RA4): Top singular value  $\approx 1/$  depth. Post block 5, the reciprocal of the top singular value scales linearly with depth and with slope 0.992, indicating a singular value close to  $1/i$  for the  $i$ -th block after RA begins.
Figure 4: Depiction of Residual Jacobian singular values for ResNet34 trained on CIFAR10 (Type 1 model in Section 2.1). Subfigure 4a shows the top 20 singular values of Residual Jacobians, while Subfigure 4b illustrates the inverse scaling of the top 1 values. More singular value plots, from diverse models and datasets, are available in subsection C.4 of the Appendix.
(a)  $U_{j,30}^{\top}J_{i}V_{j,30}$ , regular Type 3 model.
Figure 5: Stochastic depth amplifies singular vector alignment. A comparison of (RA2) for two Type 3 models trained on CIFAR10 over 50 epochs: one employing the stochastic depth technique (with a drop probability of 0.3 for skipping residual blocks during training) and the other without it.
(b)  $U_{j,30}^{\top}J_{i}V_{j,30}$ , Type 3 model with stochastic depth.
Table 1: (RA) occurs in models that generalize well. The table displays the test accuracies of models trained to study (RA). Our reported accuracies closely align with those presented in Table 1 of Papyan et al. [2020], indicating that (RA) is observed in extensively trained models that exhibit strong performance in terms of test accuracy.
Model	MNIST	FashionMNIST	CIFAR10	CIFAR100	ImageNette
Type 1	98.9	90.9	58.3	30.4	42.6
Type 2	99.5	92.0	88.5	54.6	67.9
Type 3	99.6	92.8	87.3	62.6	64.2
Figure 6: Skip connections cause Residual Alignment. The experiment depicted in Figure 2 is replicated using two additional models. The first is a ResNet18 trained on CIFAR10 (Type 4a model in Section 2.1), with results showcased in Subfigure 6a. The second is a ResNet18 without skip connections (Type 4b model in Section 2.1), with results displayed in Subfigure 6b. When the skip connections are removed, the top singular vectors no longer align. Alignment is visible in the diagonal subplots of Subfigure 6b, as each Residual Jacobian is diagonalized by its own singular vectors.
(a) NC1: activation collapse.
(b) NC2: equinorm class means.
(e) NC2: maximal equiangularity of classifiers.
(f) NC3: self duality.
(g) NC4: convergence to nearest class center.
Figure 7: Co-occurrence of Residual Alignment and Neural Collapse. The sub-figures display the progression of Neural Collapse metrics for a Type 4a model throughout 350 training epochs on the MNIST dataset as well as the emergence of Residual Alignment at the end of the training process.
(c) NC2: equinorm classifiers.
(d) NC2: maximal equiangularity of class means.
(h)  $U_{j,30}^{\top}J_{i}V_{j,30}$
2.1 Networks
We train 5 types of models:
Type 1 models consist of 16 basic residual blocks, each containing two fully-connected layers of dimension  $D = 512$ ;
Type 2 models consist of 16 basic residual blocks, each containing two convolutional layers with 64 channels, operating on a tensor with a  $16 \times 16$  spatial resolution;
Type 3 models are Type 2 models, but the last 8 basic residual blocks have 128 channels operating on an  $8 \times 8$  spatial resolution instead;
Type 4 models are Type 1 models with 8 instead of 16 residual blocks. Type 4a models have skip connections and Type 4b models do not. These models serve the purpose of verifying that the residual connection causes RA, and RA does not occur once they are removed.  $^{5}$ ;
Type 5 models are obtained from Type 2 models in the same way that Type 4 models are obtained from Type 1 models. We again have Type 5a and Type 5b, denoting models with and without skip connections, respectively.
In our approach, we have deliberately chosen to use two spatial resolutions instead of the more common three. This decision allows us to have a greater number of residual blocks with the same spatial dimension.
2.2 Datasets
We train these models on the MNIST, FashionMNIST, CIFAR10, CIFAR100, and ImageNette [Howard] datasets. To explore the relationship between RA and the number of classes, we have trained additional models on subsets of these datasets, specifically with 2, 4, 6, and 8 classes.
We train for 500 epochs, using the SGD optimizer, with a batch size of 128, an initial learning rate of 0.1, the cosine learning rate scheduler [Loshchilov and Hutter, 2016], and a weight decay of  $1\mathrm{e} - 1$  for convolutional models and  $1\mathrm{e} - 2$  or  $5\mathrm{e} - 2$  for fully-connected models. A large weight decay does not negatively impact the generalization performance, as seen in Table 1, but it does strengthen RA.
2.4 Randomized SVD
The Residual Jacobian matrix of a convolutional residual block with 64 channels operating on a spatial dimension of  $16 \times 16$  is of size  $16384 \times 16384$ . Given that SVD scales cubically with the matrix side length, it becomes essential to leverage a more efficient algorithm to expedite computation and mitigate memory consumption. Furthermore, it is worth noting that we do not require the full SVD of the matrix. Our objective is to obtain the top singular values to confirm the trend in (RA3) and (RA4) and the top singular vectors to validate the alignment in (RA2).
We leverage Algorithm 971 [Li et al., 2017], a randomized SVD algorithm, to accurately compute the desired top singular values and vectors. This method involves a series of matrix multiplications and QR decompositions, requiring a runtime of  $\mathcal{O}(TD^2 K + TK^3)$  instead of  $\mathcal{O}(D^3)$ , where  $T$  is the number of algorithm iterations,  $D$  is the matrix side length, and  $K$  is the number of top singular vectors. In our implementation, the  $T$  is set to 20, and the  $K$  ranges from 10 to 30, depending on the specific experiment. Through extensive examination, we have verified that this algorithm reliably provides singular values and vectors of sufficient accuracy.
3.1 Empirical Results
The empirical results supporting (RA) are found in Table 1 and Figures 1, 2, and 4. The counterfactual experiments are in Figures 3, 5, and 6. We also include an experiment showing the co-occurrence of (RA) and Neural Collapse in Figure 7. Experimental descriptions and discussions are within the captions. Additional supporting data can be found in the Appendix.
4The 9th residual block downsamples the representation through a projection shortcut [He et al., 2016a].
We use 8 blocks because the optimization fails to converge when using 16 blocks and no skip connections
3.2 (RA2+3+4) Imply (RA1)
As mentioned in the introduction, the properties of (RA) are interconnected, and this relationship is demonstrated through the following theorem:
Theorem 3.1. For binary classification, in a pre-activation ResNet, assuming the Jacobian linearizations are exact and satisfy  $(RA2 + 3 + 4)$ , then (RA1) holds for the intermediate representations.
The proof of this Theorem is deferred to section A of the Appendix.
3.3 Unconstrained Jacobians Model Leads to RA
We propose the following abstraction of the optimization problem in Equation (1).
Definition 3.2 (Unconstrained Jacobians Model). Given a fixed input  $\Delta_x \in \mathbb{R}^D$  and its label  $y \in \{+1, -1\}$ , find matrices  $J_i \in \mathbb{R}^{D \times D}$ ,  $1 \leq i \leq L$ , and vector  $w \in \mathbb{R}^D$  that
$$
\underset {w, \{J _ {i} \} _ {i = 1} ^ {L}} {\text {m i n i m i z e}} \quad \mathcal {L} (w ^ {\top} \prod_ {i = 1} ^ {L} (I + J _ {i}) \Delta_ {x}, y) + \frac {\lambda}{2} \sum_ {i = 1} ^ {L} \| J _ {i} \| _ {F} ^ {2} + \frac {\lambda}{2} \| w \| _ {2} ^ {2}.
$$
In the problem described above,  $J_{i}$  again represents the Residual Jacobian of the  $i$ -th residual branch and  $w$  represents the classifier Jacobian,
$$
\frac {\partial \mathcal {C} (h _ {L + 1} ; \mathcal {W} _ {L + 1})}{\partial h _ {L + 1}}.
$$
It is referred to as the "Unconstrained Jacobians Model" because the Jacobians are not restricted to any specific form and are not required to be realizable by a set of layers. In the Unconstrained Jacobians Model, the Jacobians are regularized through simple functions. However, in reality, weight decay, dropout, normalization layers, and parallel branches regularize the Jacobians in intricate ways that are hard to capture mathematically.
We prove in section B the Appendix the following theorem:
Theorem 3.3. For binary classification, there exists a global optimum of the Unconstrained Jacobians Model where the top Jacobian singular vectors are aligned (RA2), all Jacobians are rank one, analogous to (RA3), and the top Jacobian singular values are equal, analogous to (RA4).
Here, the top singular values are equal and not decaying as predicted by (RA4), because the Jacobians are evaluated on the classification boundary instead of on training examples. Therefore, the intermediate representations are not equispaced on a line, as predicted by (RA1) but are rather exponentially-spaced on a line.
In this section, we pose research questions that emerge from the discovery of RA.
Generalization The discovery of RA offers a vivid analogy for comprehending generalization in deep learning. According to RA, we can envision a ResNet as a system of conveyor belts, where each conveyor carries representations of a specific class in a unified direction at a constant speed. Misclassification occurs when the representation mistakenly steps onto the wrong conveyor belt in the initial layers of the network. This perspective leads us to hypothesize that the first few layers of the network play a vital role in generalization, surpassing the significance of subsequent layers. Consequently, more research should be dedicated towards studying the dynamics of the initial layers.
Neural Collapse As mentioned in Figure 1, there is an intriguing pattern where models exhibit RA concurrently with layer-wise Neural Collapse [Papyan, 2020, Galanti et al., 2022, Li et al., 2023]. The concurrent manifestation of these two distinct events could possibly be more than mere coincidence. It would be interesting to explore if RA could shed light on phenomena related to layer-wise Neural Collapse such as the "Law of Data Separation" [He and Su, 2022].
RA in Transformers We have empirically substantiated the occurrence of RA in ResNets. However, our study has not yet extended to Transformers, which also incorporate residual connections. An intriguing line of inquiry for future work would be to probe whether RA, or a phenomenon akin to it, manifests within these architectures.
Recurrent Architectures Exhibiting RA Recurrent neural networks, Neural ODEs [Chen et al., 2018], and deep equilibrium models [Bai et al., 2019] iteratively apply a layer within a deep neural network. This leads to the following questions:
Do the intermediate representations of these models exhibit RA? If not, can we propose a novel architecture that recurrently applies a computation but does exhibit RA?