SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wojciech Masarczyk $^{1,*}$
Mateusz Ostaszewski
Ehsan Imani²
Razvan Pascanu<sup>3</sup>
Piotr Miłos<sup>4,5</sup>
Tomasz Trzciński $^{1,4,6}$
Deep neural networks are widely known for their remarkable effectiveness across various tasks, with the consensus that deeper networks implicitly learn more complex data representations. This paper shows that sufficiently deep networks trained for supervised image classification split into two distinct parts that contribute to the resulting data representations differently. The initial layers create linearly-separable representations, while the subsequent layers, which we refer to as the tunnel, compress these representations and have a minimal impact on the overall performance. We explore the tunnel's behavior through comprehensive empirical studies, highlighting that it emerges early in the training process. Its depth depends on the relation between the network's capacity and task complexity. Furthermore, we show that the tunnel degrades out-of-distribution generalization and discuss its implications for continual learning.
Neural networks have been the powerhouse of machine learning in the last decade. A significant effort has been put into understanding the mechanisms underlying their effectiveness. One example is the analysis of building representations in neural networks applied to image processing [1]. The consensus is that networks learn to use layers in the hierarchy by extracting more complex features than the layers before [2, 3], meaning that each layer contributes to the final network performance.
Extensive research has shown that increasing network depth exponentially enhances capacity, measured as the number of lines that deep and overparameterized neural net
Figure 1: The tunnel effect for VGG19 trained on the CIFAR-10. In the tunnel (shaded area), the performance of linear probes attached to each layer saturates (blue line), and the representations rank is steeply reduced (red dashed line).
1Warsaw University of Technology, Poland 2University of Alberta, Canada 3University College London, UK 4IDEAS NCBR, Poland 5Institute of Mathematics, Polish Academy of 6Tooploo, Poland \*Corresponding author: wojciech.masarczyk
depth [7, 8]. This phenomenon arises because, despite their large capacity, these networks strive to reduce dimensionality and focus on discriminative patterns during supervised training [7-10]. Motivated by these findings, we aim to investigate this phenomenon further and formulate the following research question:
How do representations depend on the depth of a layer?
Our investigation focuses on severely overparameterized neural networks through the prism of their representations as the core components for studying neural network behavior [11, 12].
We extend the commonly held intuition that deeper layers are responsible for capturing more complex and task-specific features [13, 14] by showing that neural networks after learning low and high-level features use the remaining layers for compressing obtained representations.
Specifically, we demonstrate that deep neural networks split into two parts exhibiting distinct behavior. The first part, which we call the extractor, builds representations, while the other, dubbed the tunnel, propagates the representations further to the model's output, compressing them significantly. As we show, this behavior has important implications for generalization, transfer learning, and continual learning. To investigate the tunnel effect, we conduct multiple experiments that support our findings and shed some light on the potential source of this behavior. Our findings can be summarized as follows:
- We conceptualize and extensively examine the tunnel effect, namely, deep networks naturally split into the extractor responsible for building representations and the compressing tunnel, which minimally contributes to the final performance. The extractor-tunnel split emerges early in training and persists later on.
- We show that the tunnel deteriorates the generalization ability on out-of-distribution data.
- We show that the tunnel exhibits task-agnostic behavior in a continual learning scenario. Simultaneously it leads to higher catastrophic forgetting of the model.
2 The tunnel effect
The paper introduces and studies the dynamics of representation building in overparameterized deep neural networks called the tunnel effect. The following section validates the tunnel effect hypothesis in a number of settings. Through an in-depth examination in Section 3.1, we reveal that the tunnel effect is present from the initial stages and persists throughout the training process. Section 3.2 focuses on the out-of-distribution generalization and representations compression. Section 3.3 hints at important factors that impact the depth of the tunnel. Finally, in Section 4, we confront an auxiliary question: How does the tunnel's existence impact a model's adaptability to changing tasks and its vulnerability to catastrophic forgetting? To answer these questions we formulate our main claim as:
The tunnel effect hypothesis: Sufficiently large * neural networks develop a configuration in which network layers split into two distinct groups. The first one which we call the extractor, builds linearly-separable representations. The second one, the tunnel, compresses these representations, hindering the model's out-of-distribution generalization.
2.1 Experimental setup
To examine the phenomenon, we designed the setup to include the most common architectures and datasets, and use several different metrics to validate the observations.
Architectures We use three different families of architectures: MLP, VGGs, and ResNets. We vary the number of layers and width of networks to test the generalizability of results. See details in Appendix A.1.
Tasks We use three image classification tasks to study the tunnel effect: CIFAR-10, CIFAR-100, and CINIC-10. The datasets vary in the number of classes: 10 for CIFAR-10 and CINIC-10 and 100 for CIFAR-100, and the number of samples: 50000 for CIFAR-10 and CIFAR-100 and 250000 for CINIC-10). See details in Appendix A.2.
*We note that 'sufficiently large' covers most modern neural architectures, which tend to be heavily overparameterized.
We probe the effects using: the average accuracy of linear probing, spectral analysis of representations, and the CKA similarity between representations. Unless stated otherwise, we report the average of 3 runs.
Accuracy of linear probing: a linear classification layer is attached to a given layer  $\ell$  of the neural network. We train this layer on the classification task and report the average accuracy. This metric measures to what extent  $\ell$ 's representations are linearly separable.
Numerical rank of representations: we compute singular values of the sample covariance matrix for a given layer  $\ell$  of the neural network. Using the spectrum, we estimate the numerical rank of the given representations matrix as the number of singular values above a certain threshold  $\sigma$ . The threshold  $\sigma$  is set to  $\sigma_{1} * 1e - 3$ , where  $\sigma_{1}$  is the highest singular value of the particular matrix. The numerical rank of the representations matrix can be interpreted as the measure of the degeneracy of the matrix.
CKA similarity: is a metric computing similarity between two representations matrices. Using this normalized index, we can identify the blocks of similar representations within the network. The definition and more details can be found in Appendix E.
Inter and Intra class variance: inter-class variance refers to the measure of dispersion or dissimilarity between different classes or groups in a dataset, indicating how distinct they are from each other. Intra-class variance, on the other hand, measures the variability within a single class or group, reflecting the homogeneity or similarity of data points within that class. The exact formula for computing these values can be found in Appendix F
2.2 The main result
Table 1 presents our main result. Namely, we report the network layer at which the tunnel begins which we define as the point at which the network reaches  $95\%$  (or  $98\%$ ) of its final accuracy. We found that all tested architectures exhibit the extractor-tunnel structure across all datasets used in the evaluation, but the relative length of the tunnel varies between architectures.
Architecture	# layers	Dataset	>0.95	>0.98
MLP	13	CIFAR-10	4 (31%)	5 (38%)
VGG	19	CIFAR-10	7 (36%)	7 (36%)
CIFAR-100	8 (42%)	8 (42%)
CINIC-10	7 (36%)	7 (36%)
ResNet	34	CIFAR-10	20 (58%)	29 (85%)
CIFAR-100	29 (85%)	30 (88%)
CINIC-10	17 (50%)	17 (50%)
Table 1: The tunnel of various lengths is present in all tested configurations. For each architecture and dataset, we report the layer for which the average linear probing accuracy is above 0.95 and 0.98 of the final performance. The values in the brackets describe the part of the network utilized for building representations with the extractor.
We now discuss the tunnel effect using MLP-12, VGG-19, and ResNet-34 on CIFAR-10 as an example. The remaining experiments (for other architectures, datasets combinations) are available in Appendix B. As shown in Figure 1 and Figure 2, the early layers of the networks, around five for MLP and eight for VGG, are responsible for building linearly-separable representations. Linear probes attached to these layers achieve most of the network's final performance. These layers mark the transition between the extractor and the tunnel part (shaded area). In the case of ResNets, the transition takes place in deeper stages of the network at the  $19^{th}$  layer.
(a) MLP 12
(b) ResNet 34
Figure 2: The tunnel effect for networks trained on CIFAR-10. The blue line depicts the linear probing accuracy, and the shaded area depicts the tunnel. The red dashed line is the numerical rank of representations. The spike in the ResNet-34 representations rank coincides with the end of the penultimate residual stage.
While the linear probe performance nearly saturates in the tunnel part, the representations are further refined. Figure 2 shows that the numerical rank of the representations (red dashed line) is reduced to approximately the number of CIFAR-10 classes, which is similar to the neural collapse phenomenon observed in [10]. For ResNets, the numerical rank is more dynamic, exhibiting a spike at  $29^{th}$  layer, which coincides with the end of the penultimate residual block. Additionally, the rank is higher than in the case of MLPs and VGGs.
Figure 3 reveals that for VGG-19 the inter-class representations variation decreases throughout the tunnel, meaning that representations clusters contract towards their centers. At the same time, the average distance between the centers of the clusters grows (inter-class variance). This view aligns with the observation from Figure 2, where the rank of the representations drops to values close to the number of classes. Figure 3 (right) presents an intuitive explanation of the behavior with UMAP [15] plots of the representations before and after the tunnel.
Figure 3: The tunnel compresses the representations discarding indiscriminative features. Left: The evolution and inter and intra-class variance of representations within the VGG-19 network. Right: UMAP plot of representations before  $(7^{th}$  layer) and after  $(18^{th}$  layer) the tunnel.
To complement this analysis, we studied the similarity of MLPs representations using the CKA index and the L1 norm of representations differences between the layers. Figure 4 shows that the representations change significantly in early layers and remain similar in the tunnel part when measured with the CKA index (left). The L1 norm of representations differences between the layers is computed on the right side of Figure 4.
Figure 4: Representations within the tunnel are similar to each other for MLP with 12 hidden layers trained on CIFAR-10. Comparison of representations with CKA index (left) and average L1 norm of representations differences.
3 Tunnel effect analysis
This section provides empirical evidence contributing to our understanding of the tunnel effect. We hope that these observations will eventually lead to explanations of this phenomenon. In particular, we show that a) the tunnel develops early during training time, b) it compresses the representations and hinders OOD generalization, and c) its size is correlated with network capacity and dataset complexity.
3.1 Tunnel development
Motivation In this section, we investigate tunnel development during training. Specifically, we try to understand whether the tunnel is a phenomenon exclusively related to the representations and which part of the training is crucial for tunnel fi
Experiments We train a VGG-19 on CIFAR-10 and save intermediate checkpoints every 10 epochs of training. We use these checkpoints to compute the layer-wise weight change during training (Figure 5) and the evolution of numerical rank throughout the training (Figure 6).
Results Figure 5 shows that the split between the extractor and the tunnel is also visible in the parameters space. It could be perceived already at the early stages, and after that, its length stays roughly constant. Tunnel layers change significantly less than layers from the extractor. This result raises the question of whether the weight change affects the network's final output. Inspired by [16], we reset the weights of these layers to the state before optimization. However, the performance of the model deteriorated significantly. This suggests that although the change within the tunnel's parameters is relatively small, it plays an important role in the model's performance. Figure 6 shows that this apparent paradox can be better understood by looking at the evolution of representations' numerical rank during the very first gradient updates of the model. Throughout these steps, the rank collapses to values near-the-number of classes. It stays in this regime until the end of the training, meaning that the representations of the model evolve within a low-dimensional subspace. It remains to be understood if (and why) low-rank representations and changing weights coincide with forming linearly-separable representations.
Takeaway Tunnel formation is observable in the representation and parameter space.
It emerges early in training and persists throughout the whole optimization. The collapse in the numerical rank of deeper layers suggest that they preserve only the necessary information required for the task.
Figure 5: Early in training, tunnel layers stabilize. Color-coded: weight difference norm between consecutive checkpoints for each layer. Norm calculated as  $\frac{1}{\sqrt{nm}}\|\theta_d^{\tau_1} - \theta_d^{\tau_2}\|_2$ , where  $\theta_d^\tau \in \mathbb{R}^{nm}$  is flattened weight matrix at layer  $d$ , checkpoint  $\tau$ . Values capped at 0.02 for clarity. The learning rate decayed (by  $10^{-1}$ ) at epochs 80 and 120, and the scale adapted accordingly. Experiment: VGG-19 on CIFAR-10.
Figure 6: The representations rank for deeper layers collapse early in training. The curves present the evolution of representations' numerical rank over the first 75 training steps for all layers of the VGG-19 trained on CIFAR-10. We present a more detailed tunnel development analysis in Appendix G.
3.2 Compression and out-of-distribution generalization
Motivation Practitioners observe intermediate layers to perform better than the penultimate ones for transfer learning [17-19]. However, the reason behind their effectiveness remains unclear [20]. In this section, we investigate whether the tunnel and, specifically, the collapse of numerical rank within the tunnel impacts the performance on out-of-distribution (OOD) data.
Experiments We train neural networks (MLPs, VGG-19, ResNet-34) on a source task (CIFAR-10) and evaluate it with linear probes on the OOD task, in this case, a subset of 10 classes from CIFAR-100. We report the accuracy of linear probing and the numerical rank of the representations.
(b) VGG-19
Figure 7: The tunnel degrades the out-of-distribution performance correlated with the representations' numerical rank. The accuracy of linear probes (blue) was trained on the out-of-distribution data subset of 10 classes from CIFAR-100. The backbone was trained on CIFAR-10. The shaded area depicts the tunnel, and the red dashed line depicts the numerical rank of representations.
(c) ResNet 34
Results Our results presented in Figure 7 reveal that the tunnel is responsible for the degradation of out-of-distribution performance. In most of our experiments, the last layer before the tunnel is the optimal choice for training a linear classifier on external data. Interestingly, we find that the OOD performance is tightly coupled with the numerical rank of the representations, which significantly decreases throughout the tunnel.
To assess the generalization of our findings we extend the proposed experimentation setup to additional dataset. To that end, we train a model on different subsets of CIFAR-100 while evaluating it with linear probes on CIFAR-10. The results presented in Figure 8 are consistent with our initial findings. We include detailed analysis with reverse experiment (CIFAR-10  $\rightarrow$  CIFAR-100), additional architectures and datasets in the Appendix C.
In all tested scenarios, we observe a consistent relationship between the start of the tunnel and the drop in OOD performance. An increasing number of classes in the source task result in a shorter tunnel and a later drop in OOD performance. In the fixed source task experiment (Appendix C), the drop in performance occurs around the  $7^{th}$  layer of the network for all tested target tasks, which matches the start of the tunnel. This observation aligns with our earlier
Figure 8: Fewer classes in the source task create a longer tunnel, resulting in worse OOD performance. The network is trained on subsets of CIFAR-100 with different classes, and linear probes are trained on CIFAR-10. Shaded areas depict respective tunnels.
findings suggesting that the tunnel is a prevalent characteristic of the model rather than an artifact of a particular training or dataset setup.
Moreover, we connect the coupling of the numerical rank of the representations with OOD performance, to a potential tension between the objective of supervised learning and the generalization of OOD setup. Analogous tension was observed in [21] where adversarial robustness is at odds with model's accuracy. The results in Figure 7 align with the findings presented in Figure 3, demonstrating how the tunnel compresses clusters of class-wise representations. In work [22], the authors show that reducing the variation within each class leads to lower model transferability. Our experiments support this observation and identify the tunnel as the primary contributor to this effect.
Takeaway Compression of representations happening in the tunnel severely degrades the OOD performance of the model which is tightly coupled with the drop of representations rank.
3.3 Network capacity and dataset complexity
Motivation In this section, we explore what factors contribute to the tunnel's emergence. Based on the results from the previous section we explore the impact of dataset complexity, network's depth, and width on tunnel emergence.
Experiments First, we examine the impact of networks' depth and width on the tunnel using MLPs (Figure 9), VGGs, and ResNets (Table 2) trained on CIFAR-10. Next, we train VGG-19 and ResNet34 on CIFAR-{10,100} and CINIC-10 dataset investigating the role of dataset complexity on the tunnel.
Results Figure 9 shows that the depth of the MLP network has no impact on the length of the extractor part. Therefore increasing the network's depth contributes only to the tunnel's length. Both extractor section and numerical rank remain relatively consistent regardless of the network's depth, starting the tunnel at the same layer. This finding suggests that overparameterized neural networks allocate a fixed capacity for a given task independent of the overall capacity of the model.
Figure 9: Networks allocate a fixed capacity for the task, leading to longer tunnels in deeper networks. The extractor is consistent across all scenarios, with the tunnel commencing at the 4th layer.
1/4	1	2
VGG-16	8 (50%)	7 (44%)	7 (44%)
VGG-19	8 (42%)	7 (37%)	7 (37%)
ResNet18	15 (83%)	13 (72%)	13 (72%)
ResNet34	24 (68%)	20 (59%)	24 (68%)
Results in Table 2 indicate that the tunnel length increases as the width of the network grows, implying that representations are formed using fewer layers. However, this trend does not hold for ResNet34, as the longest tunnel is observed with the base width of the network. In the case of VGGs, the number of layers in the network does not affect the number of layers required to form representations. This aligns with the results in Figure 9.
The results presented above were obtained from a dataset with a consistent level of complexity. The data in Table 3 demonstrates that the number of classes in the dataset directly affects the length of the tunnel. Specifically, even though the CINIC-10 training dataset is three times larger than CIFAR-10, the tunnel length remains the same for both datasets. This suggests that the number of samples in the dataset does not impact the length of the tunnel. In contrast, when examining CIFAR-100 subsets, the tun
Table 2: Widening networks layers results in a longer tunnel and shorter extractor. Column headings describe the factor in which we scale each model's base number of channels. The models were trained on the CIFAR-10 to the full convergence. We use the  $95\%$  threshold of probing accuracy to estimate the tunnel beginning.
model	dataset	30%	50%	100%
VGG-19	CIFAR-10	6 (32%)	7 (37%)	7 (37%)
CIFAR-100	8 (42%)	8 (42%)	9 (47%)
CINIC-10	6 (32%)	7 (37%)	7 (37%)
ResNet34	CIFAR-10	19 (56%)	19 (56%)	21 (61%)
CIFAR-100	30 (88%)	30 (88%)	31 (91%)
CINIC-10	9 (27%)	9 (27%)	17 (50%)
Table 3: Networks trained on tasks with fewer classes utilize fewer resources for building representations and exhibit longer tunnels. Column headings describe the size of the class subset used in training. Within the (architecture, dataset) pair, the number of gradient steps during training in all cases was the same. We use the  $95\%$  threshold of probing accuracy to estimate the tunnel beginning.
nel length for both VGGs and ResNets increase. This indicates a clear relationship between the dataset's number of classes and the tunnel's length.
Takeaway Deeper or wider networks result in longer tunnels. Networks trained on datasets with fewer classes have longer tunnels.
4 The tunnel effect under data distribution shift
Based on the findings from the previous section and the tunnel's negative impact on transfer learning, we investigate the dynamics of the tunnel in continual learning scenarios, where large models are often used on smaller tasks typically containing only a few classes. We focus on understanding the impact of the tunnel effect on transfer learning and catastrophic forgetting [23]. Specifically, we examine how the tunnel and extractor are altered after training on a new task.
4.1 Exploring the effects of task incremental learning on extractor and tunnel
Motivation In this section, we aim to understand the tunnel and extractor dynamics in continual learning. Specifically, we examine whether the extractor and the tunnel are equally prone to catastrophic forgetting.
Experiments We train a VGG-19 on two tasks from CIFAR-10. Each task consists of 5 classes from the dataset. We subsequently train on the first and second tasks and save the corresponding extractors  $E_{t}$  and tunnels  $T_{t}$ , where  $t \in \{1,2\}$  is the task number. We also save a separate classifying head for trained on each task, that we use during evaluation.