SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model Compression In our work, we demonstrate that the network can converge to a model that iteratively applies a computation, even without imposing explicit constraints, like the architectures in the previous subsection. This leads us to yet another question:
Can we replicate the original network's performance by distilling the aligning layers into a single layer and iteratively applying it?
Regularization Techniques Existing regularization techniques, including layer permutation [Liaw et al., 2021] and structured dropout [Fan et al., 2019], should strengthen RA. A question arises:
Could RA explain the success of these methods and do they indeed amplify RA?
Linearization of Residual Jacobians Prior to our work, Rothauge et al. [2019] proposed the linearization of residual Jacobians and investigated the distribution of their singular values. Our work, however, concerns the discovery of the alignment between the Residual Jacobians, as well as its theoretical understanding.
Generalization A thorough empirical investigation by Novak et al. [2018] found that the Frobenius norm of the input-output Jacobian of a network correlates well with generalization. Our research complements theirs by conducting both empirical and theoretical investigations on the Residual Jacobians, which form the input-output Jacobian of a ResNet.
ResNets at Initialization Previous works have contributed significantly to the understanding of the properties of intermediate representations of randomly initialized ResNets, with studies such as Hayou [2022] exploring the infinite-width and finite-width regime and Li et al. [2021, 2022] investigating the infinite-depth and infinite-width regime. Our current research diverges from these previous works by specifically focusing on studying the Residual Jacobians of fully trained ResNets.
Optimization Landscape Li et al. [2016] analyzed the Hessian of the loss function for a ResNet initialized with zero parameters. Additionally, Lu et al. [2020] use a mean-field analysis of ResNets to demonstrate convergence to a global minimum. Their analysis builds upon the observation that a ResNet is similar to a shallow network ensemble, first noted by Veit et al. [2016].
Rather than focusing on the optimization convergence properties of SGD, our goal is to examine the properties of the intermediate representations and Residual Jacobians that emerge during training.
Neural Collapse Recent theoretical work by Mixon et al. [2020], Lu and Steinerberger [2020], E and Wojtowitsch [2020], Poggio and Liao [2020], Zhu et al. [2021], Han et al. [2021], Tirer and Bruna [2022], Wang et al., Kothapalli et al. [2022] analyzed the Neural Collapse phenomenon, discovered by Papyan, Han, and Donoho [2020], through the unconstrained features model and the layer-peeled model. In these, the assumption is made that the last-layer representations, which are fed to a classifier, have the freedom to move independently and are not constrained to be the output of a deep network. Our Unconstrained Jacobians Model takes inspiration from these mathematical models by abstracting away the complex Residual Jacobians and assuming they can be optimized directly.
Unrolled Iterative Algorithm Perspective Greff et al. [2016], Papyan et al. [2017], and Ebski et al. [2018] recognized ResNets as unrolled iterative algorithms performing iterative inference. We build upon this understanding by delving deeper into the empirical and theoretical relationships between the Residual Jacobians.
Neural ODE Chen et al. [2018] introduced the Neural ODE: a continuous-depth, tied-weights ResNet. Initially, it may seem unlikely that the iterations or Jacobians of a traditional ResNet would possess any characteristics associated with Neural ODEs. However, Sander et al. [2022] proved that assuming the initial loss is small and the initial parameters are smooth with depth, linear ResNets converge to a Neural ODE as the number of layers increases. Still, it is unclear if these findings extend to nonlinear ResNets trained on real data.
Our work, however, demonstrates that even in such cases the Residual Jacobians align in their top subspaces and, as a result, simple ODEs emerge within these subspaces. We also provide theoretical justification for this claim through the Unconstrained Jacobians Model.
Optimal Transport Gai and Zhang [2021] view ResNets as aiming to transport an input distribution to a label distribution, through a geodesic curve in the Wasserstein space, induced by the optimal transport map. They provide empirical evidence to support their claim, showing that intermediate representations are equidistant on a straight line induced by the optimal transport map, and comment: "Though ResNet approximates the geodesic curve better than plain network, it may not be a perfect implement in high-dimensional space due to its layer-wise heterogeneity."
Our research demonstrates that intermediate representations lie on a line as a result of the Jacobian singular vectors aligning (RA2) and the Jacobian singular values scaling inversely with depth (RA4) and, in fact, due to the absence of purported "layer-wise heterogeneity."
Analysis of Deep Linear Networks Mulayoff and Michaeli [2020] proved that training a deep linear network, with a quadratic loss, and no weight decay, necessarily converges to the flattest of all minima. At this optimum point, the spectral norm of the input-feature Jacobian increases exponentially with depth, and the singular vectors of consecutive weight matrices align.
Our theoretical study considers ResNets, trained with a binary cross-entropy loss, and weight decay. Instead of assuming that training has reached a flat minimum, we analyze the Unconstrained Jacobians Model, on the classification boundary in the input space, and prove phenomena that we have observed through experiments on nonlinear ResNets.
Analysis of Weights Cohen et al. [2021] studied the scaling behavior of trained weights in deep residual networks. Our study complements theirs by focusing on examining the Residual Jacobians of ResNet architectures.
In this paper, we offer a detailed empirical examination of the ResNet architecture, a model that has seen extensive application across a broad spectrum of deep learning domains. Our primary aim was to demystify the mechanisms that contribute to ResNet's remarkable success, a topic that, to date, remains an enigma within the deep learning community.
Our investigation has led to the discovery of a consistent phenomenon, which we have termed RA. The characteristics of RA were observed across a wide array of benchmark datasets, canonical architectures, and hyperparameters, demonstrating its general applicability. Moreover, we conducted counterfactual studies that underscored the critical role of skip connections in the emergence of RA and the effect of the number of classes. In an attempt to theoretically ground the emergence of RA, we proposed the use of an innovative mathematical abstraction – the Unconstrained Jacobians Model, specifically in the context of binary classification with cross-entropy loss.
Our exploration has not only shed light on the intricate mechanisms driving ResNets' performance but also points to connections with the recent phenomenon of layer-wise Neural Collapse. Furthermore, our findings pave the way for future research in the understanding of existing regularization methods, the design of novel architectures, the development of model compression techniques, and the theoretical investigation of generalization.
We thank Kirill Serkh for inspiring discussions and feedback on the manuscript. We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), [funding reference number 512236 and 512265]. This research was enabled in part by support provided by Compute Ontario (http://www.computeontario.ca) and Compute Canada (http://www.computecanada.ca).
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Alain-Sam Cohen, Rama Cont, Alain Rossier, and Renyuan Xu. Scaling properties of deep residual networks. In International Conference on Machine Learning, pages 2039-2048. PMLR, 2021.
Weinan E and Stephan Wojtowitsch. On the emergence of tetrahedral symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint arXiv:2012.05420, 2020.
Stanisław Jastrz Ebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations, 2018.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909.11556, 2019.
Kuo Gai and Shihua Zhang. A mathematical principle of deep learning: Learn the geodesic curve in the wasserstein space. arXiv preprint arXiv:2102.09235, 2021.
Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. On the implicit bias towards minimal depth of deep neural networks, 2022.
Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.07771, 2016.
XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations, 2021.
Soufiane Hayou. On the infinite-depth limit of finite-width neural networks. arXiv preprint arXiv:2210.00688, 2022.
Hangfeng He and Weijie J. Su. A law of data separation in deep learning, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645. Springer, 2016b.
Jeremy Howard. GitHub - fastai/imagenette: A smaller subset of 10 easily classified classes from Imagenet, and a little more French — github.com. https://github.com/fastai/imagenette. [Accessed 14-May-2023].
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth, 2016.
Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.04041, 2022.
Kai Rothauge, Zhewei Yao, Zixi Hu, and Michael W. Mahoney. Residual networks as nonlinear systems: Stability analysis using linearization. 2019.
Michael E Sander, Pierre Ablin, and Gabriel Peyré. Do residual neural networks discretize neural ordinary differential equations? arXiv preprint arXiv:2205.14612, 2022.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CoRR, abs/1505.00387, 2015. URL http://arxiv.org/abs/1505.00387.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In international conference on machine learning (ICML), 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016.
Peng Wang, Huikang Liu, Can Yaras, Laura Balzano, and Qing Qu. Linear convergence analysis of neural collapse with unconstrained features. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop).
Saining Xie, Ross Girshick, Piotr Dolkar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In *British Machine Vision Conference* 2016. British Machine Vision Association, 2016.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-29834, 2021.
A (RA2+3+4) Imply (RA1)
Proof. In a pre-activation ResNet,
$$
h _ {i + 1} = h _ {i} + \mathcal {F} (h _ {i}; \mathcal {W} _ {i}).
$$
Since Jacobian linearizations are exact, we have:
$$
h _ {i + 1} = (I + J _ {i}) h _ {i}.
$$
Recall, the singular value decomposition of  $J_{i}$  is given by
$$
J _ {i} = U _ {i} S _ {i} V _ {i} ^ {\top},
$$
where  $U_{i}$  and  $V_{i}$  are the respective left and right singular vector matrices, and  $S_{i}$  is the singular value matrix. Invoking (RA2) and (RA3), for some matrix  $U_{0}$  and for any block  $i$ ,
$$
J _ {i} = U _ {0} S _ {i} U _ {0} ^ {\top},
$$
$$
h _ {i + 1} = \left(I + U _ {0} S _ {i} U _ {0} ^ {\top}\right) h _ {i}.
$$
Applying recursively the above equality leads to
$$
h _ {k} = \left(\prod_ {i = 1} ^ {k - 1} \left(I + U _ {0} S _ {i} U _ {0} ^ {\top}\right)\right) h _ {1} = U _ {0} \left(\prod_ {i = 1} ^ {k - 1} \left(I + S _ {i}\right)\right) U _ {0} ^ {\top} h _ {1}. \tag {2}
$$
For binary classification, (RA3) implies the Jacobians are rank 1 and therefore
$$
h _ {k} = U _ {0, 1} \left(\prod_ {i = 1} ^ {k - 1} \left(1 + S _ {i, 1}\right)\right) U _ {0, 1} ^ {\top} h _ {1} + \left(I - U _ {0, 1} U _ {0, 1} ^ {\top}\right) h _ {1},
$$
where  $U_{0,j}$  is the  $j$ th column of  $U_0$ . According to (RA4),
$$
S _ {i, 1} = \frac {1}{i}
$$
$$
\prod_ {i = 1} ^ {k - 1} \left(1 + S _ {i, 1}\right) = \prod_ {i = 1} ^ {k - 1} \left(1 + 1 / i\right) = k.
$$
Substituting the above into Equation (2), we obtain
$$
h _ {k} = k U _ {0, 1} U _ {0, 1} ^ {\top} h _ {1} + (I - U _ {0, 1} U _ {0, 1} ^ {\top}) h _ {1} = (k - 1) U _ {0, 1} U _ {0, 1} ^ {\top} h _ {1} + h _ {1}.
$$
This proves that the intermediate representations of a given input are equispaced on a line embedded in high dimensional space, i.e., (RA1).
B Unconstrained Jacobians Model Leads to RA
We start by providing motivation for the unconstrained Jacobians problem introduced in the main text. Assume a training example,  $x$ , is situated next to a point on the classification boundary, denoted by  $x_{\mathrm{mid}}$ , satisfying  $f(x_{\mathrm{mid}}; \mathcal{W}) = 0$ . Performing a Taylor expansion of the logits of  $x$  around  $x_{\mathrm{mid}}$  yields
$$
\begin{array}{l} f (x; \mathcal {W}) = f (x _ {\mathrm {m i d}}; \mathcal {W}) + \left. \frac {\partial f (x ; \mathcal {W})}{\partial x} \right| _ {x _ {\mathrm {m i d}}} \Delta_ {x} + h (x, x _ {\mathrm {m i d}}) \\ = \left. \frac {\partial f (x ; \mathcal {W})}{\partial x} \right| _ {x _ {\mathrm {m i d}}} \Delta_ {x} + h (x, x _ {\mathrm {m i d}}), \tag {3} \\ \end{array}
$$
where  $\Delta_x = x - x_{\mathrm{mid}}$  and  $h(x,x_{\mathrm{mid}})$  accounts for the approximation error, which is  $\mathcal{O}(\| \Delta_x\| _2^2)$  and assumed to be negligible in our analysis.
Recall the loss associated with training a ResNet:
$$
\underset {\{\mathcal {W} _ {i} \} _ {i = 1} ^ {L + 1}} {\text {m i n i m i z e}} \frac {1}{N} \sum_ {n = 1} ^ {N} \mathcal {L} \left(f \left(x _ {n}; \mathcal {W}\right), y _ {n}\right) + \frac {\lambda}{2} \| \mathcal {W} \| _ {2} ^ {2}. \tag {4}
$$
Substituting Equation (3) into the above, neglecting the approximation error, and considering only the objective associated with the training sample  $x$  and its label  $y$ , we get
$$
\mathcal{L}\left(\frac{\partial f(x;\mathcal{W})}{\partial x}\bigg|_{x_{\text{mid}}}\Delta_{x},y\right) + \frac{\lambda}{2}\| \mathcal{W}\|_{2}^{2}.
$$
By using the chain rule, we can then obtain the following:6
$$
\mathcal {L} \left(\frac {\partial \mathcal {C} \left(h _ {L + 1} ; \mathcal {W} _ {L + 1}\right)}{\partial h _ {L + 1}} \prod_ {i = 1} ^ {L} \left(I + \frac {\partial \mathcal {F} \left(h _ {i} ; \mathcal {W} _ {i}\right)}{\partial h _ {i}}\right) \Delta_ {x}, y\right) + \frac {\lambda}{2} \sum_ {i = 1} ^ {L} \| \mathcal {W} _ {i} \| _ {F} ^ {2} + \frac {\lambda}{2} \| \mathcal {W} _ {L + 1} \| _ {F} ^ {2}, \tag {5}
$$
where all the Jacobians are evaluated at the point  $x_{\mathrm{mid}}$ . This naturally gives rise to the following definition, as introduced in the main text.
In the main text, we stated the following theorem.
Theorem 3.3. For binary classification, There exists a global optimum of the Unconstrained Jacobians Model where the top Jacobian singular vectors are aligned (RA2), all Jacobians are rank one, analogous to (RA3), and the top Jacobian singular values are equal, analogous to (RA4).
Proof. Throughout the proof, we assume, without loss of generality, that the label is  $y = 1$ . Using the cyclic property of the trace, the logit,  $w^{\top} \prod_{i=1}^{L} (I + J_i) \Delta_x$ , equals
$$
\operatorname {t r} \left\{\Delta_ {x} w ^ {\top} \prod_ {i = 1} ^ {L} (I + J _ {i}) \right\}.
$$
Denoting the power set of all natural numbers between 1 and  $L$  by  $\mathcal{P}(L)$ , the above can be expressed as follows:
$$
\sum_ {s \in \mathcal {P} (L)} \operatorname {t r} \left\{\Delta_ {x} w ^ {\top} \prod_ {i \in s} J _ {i} \right\}.
$$
Each element in the above summation can be upper bounded through the following theorem (a generalization of Von Neumann's trace inequality [Mirsky, 1975] to the product of more than two real matrices).
Theorem 2 ([Miranda and Thompson, 1993]). Let  $A_1, \ldots, A_m$  be matrices with real entries. Take the singular values of  $A_j$  to be  $s_1(A_j) \geq \dots \geq s_n(A_j) > 0$ , for  $j = 1, \ldots, m$ , and denote  $S_j = \mathrm{diag}(s_1(A_j), \ldots, s_n(A_j))$ . Then, as the matrices  $P_1, \ldots, P_m$  range over all possible rotations, i.e., the special orthogonal group  $\mathrm{SO}(n)$ ,
$$
\sup_{P_{1},\ldots ,P_{m}\in \mathrm{SO}(n)}\operatorname {tr}(A_{1}P_{1}\ldots A_{m}P_{m})
$$
$$
= \sum_ {i = 1} ^ {n - 1} \prod_ {j = 1} ^ {m} s _ {i} (A _ {j}) + [ \operatorname {s i g n} \det  (A _ {1} \dots A _ {m}) ] \prod_ {j = 1} ^ {m} s _ {n} (A _ {j}).
$$
For simplicity, we ignore the input transformation that maps the input  $\Delta_x$  to the first representation  $h_1 \in \mathbb{R}^D$  by simply assuming  $\Delta_x \in \mathbb{R}^D$ .
<sup>7</sup>Assume the convention that the singular values are positive.
Moreover, assuming  $\operatorname{sign} \det(A_1 \ldots A_m) = 1$ ,
$$
\sup _ {P _ {1}, \ldots , P _ {m} \in \mathrm {S O} (n)} \operatorname {t r} (A _ {1} P _ {1} \ldots A _ {m} P _ {m}) = \operatorname {t r} \left\{\prod_ {i = 1} ^ {m} S _ {i} \right\}.
$$
We will continue our proof using contradiction. Suppose all existing global optima of the unconstrained Jacobians problem consist of Jacobians that do not have aligning singular vectors, or do not have equal singular values, or are not rank 1. Then take any solution  $\{J_i\}_{i=1}^L$  and  $w$ . Using the singular value decomposition, we have
$$
J _ {i} = U _ {i} S _ {i} V _ {i} ^ {\top}, \quad \mathrm {f o r} i = 1, \ldots , L,
$$
$$
\Delta_ {x} w ^ {\top} = U _ {L + 1} S _ {L + 1} V _ {L + 1} ^ {\top}.
$$
Then Theorem 2 implies
$$
\begin{array}{l} \sum_ {s \in \mathcal {P} (L)} \operatorname {t r} \left\{\Delta_ {x} w ^ {\top} \prod_ {i \in s} J _ {i} \right\} \leq \sum_ {s \in \mathcal {P} (L)} \operatorname {t r} \left\{S _ {L + 1} \prod_ {i \in s} S _ {i} \right\} \\ = \operatorname {t r} \left\{S _ {L + 1} \prod_ {i = 1} ^ {L} \left(I + S _ {i}\right) \right\}. \\ \end{array}
$$
For all  $s \in \mathcal{P}(L)$ , the inequality becomes equality once the singular vectors of all the Jacobians align with those of  $\Delta_x w^\top$  and once the vector  $w$  is chosen to be proportional to  $\Delta_x$  (so that the matrix  $\Delta_x w^\top$  is symmetric). The implication of the steps thus far is that one can increase, or at least keep constant the logit, and consequently reduce, or at least keep constant the loss by simply aligning the singular vectors of the Jacobians. In addition, since the regularization term  $\| J_i \|_F^2 = \mathrm{tr}\{S_i^2\}$ , this change of Jacobians does not affect the regularization terms.
Notice that  $\Delta_x w^\top$  is a rank one matrix and so  $S_{L + 1}$  has a single non-zero diagonal entry. Furthermore, the matrices  $S_{i}$ , for  $1\leq i\leq L$ , are all diagonal. As such, we can zero out all their other diagonal entries and leave a single non-zero entry at the location that  $S_{L + 1}$  has one, which does not affect the logits but reduces the regularization terms.
Using the inequality of arithmetic and geometric means on this only non-zero entry  $s_i$  of every diagonal matrix  $S_i$  gives
$$
s _ {L + 1} \prod_ {i = 1} ^ {L} (1 + s _ {i}) \leq s _ {L + 1} \left(1 + \frac {1}{L} \sum_ {i = 1} ^ {L} s _ {i}\right) ^ {L}.
$$
The implication of the above inequality is that, once the singular vectors of the Jacobians are aligned, one can further increase the logits and reduce the loss by averaging all the top singular values,  $s_i$  for  $1 \leq i \leq L$ , and forcing them to be equal. Furthermore, since  $\| J_i\|_F^2 = \mathrm{tr}\bigl \{\bar{S}_i^2\bigr \} = s_i^2$  is convex, by Jensen's inequality, averaging the singular values only decreases the value of the Jacobian regularization.
All in all, we obtain higher, or at least no lower logit, and lower, or at least no higher loss when all singular vectors are aligned, all top singular values are equal and all other singular values are zero, which contradicts the statement that no global optima of the unconstrained Jacobians problem satisfies all of these conditions.
C Additional Empirical Evidence
C.1 (RA1)
Figure 8: Fully-connected ResNet34 (Type 1 model) trained on MNIST.
Figure 9: Fully-connected ResNet34 (Type 1 model) trained on FashionMNIST.
Figure 10: Fully-connected ResNet34 (Type 1 model) trained on CIFAR10.
Figure 11: Convolutional ResNet34 (Type 2 model) trained on MNIST.
Figure 12: Convolutional ResNet34 (Type 2 model) trained on FashionMNIST.
Figure 13: Convolutional ResNet34 (Type 2 model) trained on CIFAR10.
Figure 14: Convolutional ResNet34 with downsampling (Type 3 model) trained on MNIST.
Figure 15: Convolutional ResNet34 with downsampling (Type 3 model) trained on FashionMNIST.
Figure 16: Convolutional ResNet34 with downsampling (Type 3 model) trained on CIFAR10.
C.2 (RA2):  $U_{j,K}^{\top}J_{i}V_{j,K}$
Figure 17: Fully-connected ResNet34 (Type 1 model) trained on MNIST.
Figure 18: Fully-connected ResNet34 (Type 1 model) trained on FashionMNIST.
Figure 19: Fully-connected ResNet34 (Type 1 model) trained on CIFAR10.
Figure 20: Fully-connected ResNet34 (Type 1 model) trained on CIFAR100.
Figure 21: Convolutional ResNet34 (Type 2 model) trained on MNIST.
Figure 22: Convolutional ResNet34 (Type 2 model) trained on FashionMNIST.
Figure 23: Convolutional ResNet34 (Type 2 model) trained on CIFAR10.
Figure 24: Convolutional ResNet34 (Type 2 model) trained on CIFAR100.
Figure 25: Convolutional ResNet34 (Type 2 model) trained on ImageNette.
Figure 26: Convolutional ResNet34 with downsampling (Type 3 model) trained on MNIST.
Figure 27: Convolutional ResNet34 with downsampling (Type 3 model) trained on FashionMNIST.
Figure 28: Convolutional ResNet34 with downsampling (Type 3 model) trained on CIFAR10.
Figure 29: Convolutional ResNet34 with downsampling (Type 3 model) trained on CIFAR100.
Figure 30: Convolutional ResNet34 with downsampling (Type 3 model) trained on ImageNette.
C.3 (RA2):  $V_{j,K}^{\top} J_i U_{j,K}$
Figure 31: Fully-connected ResNet34 (Type 1 model) trained on MNIST.
Figure 32: Fully-connected ResNet34 (Type 1 model) trained on FashionMNIST.
Figure 33: Fully-connected ResNet34 (Type 1 model) trained on CIFAR10.
Figure 34: Fully-connected ResNet34 (Type 1 model) trained on CIFAR100.
Figure 35: Convolutional ResNet34 (Type 2 model) trained on MNIST.
Figure 36: Convolutional ResNet34 (Type 2 model) trained on FashionMNIST.