SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Results As presented in Table 4, in any combination changing  $T_{1}$  to  $T_{2}$  or vice versa have a marginal impact on the performance. This is quite remarkable, and suggests that the tunnel is not specific to the training task. It seems that it compresses the representations in a task-agnostic way. The extractor part, on the other hand, is task-specific and prone to forgetting as visible in the first four rows of Table 4. In the last two rows, we present two experiments that investigate how the existence of a tunnel affects the possibility of recovering from this catastrophic forgetting. In the first one, referred to as  $(E_{2} + T_{1}(FT))$ , we use original data from Task 1 to retrain a classifying head attached on top of extractor  $E_{2}$  and the tunnel  $T_{1}$ . As visible, it has minimal effect on the accuracy of the first
First Task	Second Task
E1+T1	92.04%	56.8%
E1+T2	92.5%	58.04 %
E2+T2	50.84 %	93.94 %
E2+T1	50.66 %	93.72 %
E2+T1(FT)	56.1%	-
E2(FT)	74.4%	-
Table 4: The tunnel part is task-agnostic and can be freely mixed with different extractors retaining the original performance. We test the model's performance on the first or second task using a combination of extractor  $E_{t}$  and tunnel  $T_{t}$  from tasks  $t \in \{1,2\}$ . The last two rows  $(FT)$  show how much performance can be recovered by retraining the linear probe attached to the penultimate layer  $E_{1} + T_{1}$  or the last layer of the  $E_{2}$ .
task. In the second experiment, we attach a linear probe directly to the extractor representations  $(E_2(FT))$ . This difference hints at a detrimental effect of the tunnel on representations' usability in continual learning.
In Appendix D.1 we study this effect further by training a tunnels on two tasks with a different number of classes, where  $n_1 > n_2$ . In this scenario, we observe that tunnel trained with more classes  $(T_1)$  maintains the performance on both tasks, contrary to the tunnel  $(T_2)$  that performs poorly on Task 1. This is in line with our previous observations in Section 2.2, that the tunnel compresses to the effective number of classes.
These results present a novel perspective in the ongoing debate regarding the layers responsible for causing forgetting. However, they do not align with the observations made in the previous study [24]. In Appendix D, we delve into the origin of this discrepancy and provide a comprehensive analysis of the changes in representations with a setup introduced with this experiment and the CKA similarity.
Takeaway The tunnel's task-agnostic compression of representations provides immunity against catastrophic forgetting when the number of classes is equal. These findings offer fresh perspectives on studying catastrophic forgetting at specific layers, broadening the current understanding in the literature.
4.2 Reducing catastrophic forgetting by adjusting network depth
Motivation Experiments from this section verify whether it is possible to retain the performance of the original model by training a shorter version of the network. A shallower model should also exhibit less forgetting in sequential training.
Experiments We train VGG-19 networks with different numbers of convolutional layers. Each network is trained on two tasks from CIFAR-10. Each task consists of 5 classes from the dataset.
Results: The results shown in Figure 10 indicate that training shorter networks yields similar performance compared to the original model. However, performance differences become apparent when the network becomes shorter than the extractor part in the original model. This observation aligns with previous findings suggesting that the model requires a certain capacity to perform the task effectively. Additionally, the shorter models exhibit significantly less forgetting, which corroborates the conclusions drawn in previous works [25, 26] on the importance of network depth and architecture in relation to forgetting.
Takeaway It is possible to train shallower networks that retain the performance of the original networks and experience significantly less forgetting. However, the shorter networks need to have at least the same capacity as the extractor part of the original network.
Figure 10: Training shorter networks from scratch gives a similar performance to the longer counterparts (top) and results in significantly lower forgetting (bottom). The horizontal lines denote the original model's performance. Top image: blue depicts accuracy on first task, orange depicts accuracy on the second task.
This paper empirically investigates the tunnel effect, opening the door for future theoretical research on tunnel dynamics. Further exploration could involve mitigating the tunnel effect through techniques like adjusting learning rates for specific layers. One limitation of our work is its validation within a specific scenario (image classification), while further studies on unsupervised or self-supervised methods with other modalities would shed more light and verify the pertinence of the tunnel elsewhere.
In the experiments, we observed that ResNet-based networks exhibited shorter tunnels than plain MLPs or VGGs. This finding raises the question of whether the presence of skip connections plays a role in tunnel formation. In Appendix H, we take the first step toward a deeper understanding of this relationship by examining the emergence of tunnels in ResNets without skip connections.
The analysis of representations in neural network training is an established field [27-29]. Previous studies have explored training dynamics and the impact of model width [30-35], but there is still a gap in understanding training dynamics [36, 24, 27, 37]. Works have investigated different architectures' impact on continual learning [38, 26] and linear models' behavior [39-42]. Our work builds upon studies examining specific layers' role in model performance [16, 11, 31, 36, 20, 43] and sheds light on the origins of observed behaviors [10, 44-46].
Previous works have explored the role of specific layers in model performance [16, 11, 31, 36, 20, 43]. While some studies have observed a block structure in neural network representations, their analysis was limited to ResNet architectures and did not consider continual learning scenarios. In our work, we investigate a similar phenomenon, expanding the range of experiments and gaining deeper insights into its origins. On the other hand, visualization of layer representations indicates that higher layers capture intricate and meaningful features, often formed through combinations of lower-layer features [13]. This phenomenon potentially accounts for the extension of feature extractors for complex tasks. Work [47] builds a theoretical picture that stacked sequence models tend to converge to a fixed state with infinite depth and proposes a method to compute the finite equivalent of such networks. The framework of [47] encompasses previous empirical findings of [48-50]. Independently, research on pruning methods has highlighted a greater neuron count in pruned final layers than in initial layers [51], which aligns with the tunnel's existence. Furthermore, in [52, 53], authors showed that training neural networks may lead to compressing information contained in consecutive hidden layers.
Yet another work [16] offers a different perspective, where the authors distinguish between critical and robust layers, highlighting the importance of the former for model performance, while individual layers from the latter can be reset without impacting the final performance. Our analysis builds upon this finding and further categorizes these layers into the extractor and tunnel, providing insights into their origins and their effects on model performance and generalization ability.
Our findings are also related to the Neural Collapse (NC) phenomenon [10], which has gained recent attention [44-46]. Several recent works [54-56] have extended the observation of NC and explored its impact on different layers, with a notable emphasis on deeper layers. [55] establishes a link between collapsed features and transferability. In our experiments, we delve into tunnel creation, analyzing weight changes and model behavior in a continual learning scenario, revealing the task-agnostic nature of the tunnel layers.
This work presents new insights into the behavior of deep neural networks during training. We discover the tunnel effect, an intriguing phenomenon in modern deep networks where they split into two distinct parts - the extractor and the tunnel. The extractor part builds representations, and the tunnel part compresses these representations to a minimum rank without contributing to the model's performance. This behavior is prevalent across multiple architectures and is positively correlated with overparameterization, i.e., it can be induced by increasing the model's size or decreasing the complexity of the task.
We emphasise that our motivation for investigating this phenomenon aimed at building a coherent picture encompassing both our experiments and evidence in the literature. Specifically, we aim to understand better how the neural networks handle the representation-building process in the context of depth.
Additionally, we discuss potential sources of the tunnel and highlight the unintuitive behavior of neural networks during the initial training phase. This novel finding has significant implications for improving the performance and robustness of deep neural networks. Moreover, we demonstrate that the tunnel hinders out-of-distribution generalization and can be detrimental in continual learning settings.
Overall, our work offers new insights into the mechanisms underlying deep neural networks. Building on consequences of the tunnel effect we derive a list of recommendations for practitioners interested in applying deep neural networks to downstream tasks. In particular, focusing on the tunnel entry features is promising when dealing with distribution shift due to its strong performance with OOD data. For continual learning, regularizing the extractor should be enough, as the tunnel part exhibits task-agnostic behavior. Skipping feature replays in deeper layers or opting for a compact model without a tunnel can combat forgetting and enhance knowledge retention. For efficient inference, excluding tunnel layers during prediction substantially cuts computation time while preserving model accuracy, offering a practical solution for resource-constrained situations.
We would like to thank the reviewers for their work and insightful remarks that made this paper more complete. We especially thank Kamil Deja and other colleagues for valuable feedback and discussions during the project and writing the manuscript.
This research was funded by National Science Centre, Poland grant no 2020/39/B/ST6/01511, grant no 2022/45/B/ST6/02817, grant no 2019/35/O/ST6/03464, and grant no 2022/45/N/ST6/04098. The research was funded by Warsaw University of Technology within the Excellence Initiative: Research University (IDUB) programme. The authors have applied a CC BY license to any Author Accepted Manuscript (AAM) version arising from this submission, in accordance with the grants' open access conditions.
We gratefully acknowledge Polish high-performance computing infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2023/016321
[58] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016. 15
[59] G. Bebis and M. Georgiopoulos, “Feed-forward neural networks,” *Ieee Potentials*, vol. 13, no. 4, pp. 27–31, 1994. 15
[60] Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, “Rethinking the value of network pruning,” 2019. 15
[61] A. Krizhevsky, “Learning multiple layers of features from tiny images,” 2009. 15, 16
[62] L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey, "Cinic-10 is not imagenet or CIFar-10," arXiv preprint arXiv:1810.03505, 2018. 16
[63] L. Bossard, M. Guillaumeun, and L. Van Gool, "Food-101 – mining discriminative components with random forests," in European Conference on Computer Vision, 2014. 16
[64] M.-E. "Nilback and A. Zisserman, "'automated flower classification over a large number of classes'," in "Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing", "2008". 16
[65] O. M. "Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar, "'cats and dogs'," in "IEEE Conference on Computer Vision and Pattern Recognition", "2012". 16
[66] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. 16
[67] A. Coates, A. Ng, and H. Lee, “An Analysis of Single Layer Networks in Unsupervised Feature Learning,” in AISTATS, 2011. https://cs.stanford.edu/~acoates/papers/coatesleeng_aistats_2011.pdf. 16
[68] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, "Reading digits in natural images with unsupervised feature learning," NeurIPS, 2011. 16
[69] S. Ebrahimi, F. Meier, R. Calandra, T. Darrell, and M. Rohrbach, "Adversarial continual learning," 2020. 29
[70] L. Pellegrini, G. Graffieti, V. Lomonaco, and D. Maltoni, “Latent replay for real-time continual learning,” 2020. 29
[71] S. Balakrishnama and A. Ganapathiraju, “Linear discriminant analysis-a brief tutorial,” 1998. 33
[72] A. Islam, C.-F. R. Chen, R. Panda, L. Karlinsky, R. Radke, and R. Feris, “A broad study on the transferability of visual representations with contrastive learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8845–8855, 2021. 33
[73] N. Zhao, Z. Wu, R. W. Lau, and S. Lin, “What makes instance discrimination good for transfer learning?” 2020. 33
A Experimental setup.
A.1 Architectures and hyperparameters
In this section, we detail the model architectures examined in the experiments and list all hyperparameters used in the experiments.
VGG [57] In the main text use two types of VGG networks, namely VGG-19 and VGG-16. Both architectures consist of five stages, each consisting of a combination of convolutional layers with ReLU activation and max pooling layers. The VGG-19 has 19 layers, including 16 convolutional layers and three fully connected layers. The first two fully connected layers are followed by ReLU activation. On the other hand, VGG-16 has a total of 16 layers, including 13 convolutional layers and three fully connected layers. In additional experiments, we extend our analysis by VGG-11 and VGG-16. The base number of channels in consecutive stages for VGG architectures equals 64, 128, 256, 512, and 512.
ResNet [58] In experiments, we utilize two variants of the ResNet family of architectures, i.e., ResNet-18 and ResNet-34. ResNet- $N$  is a five-staged network characterized by depth, with a total of  $N$  layers. The initial stage consists of a single convolutional layer - with kernel size  $7 \times 7$  and 64 channels and ReLU activation, followed by max pooling  $2 \times 2$ , which reduces the spatial dimensions. The subsequent stages are composed of residual blocks. Each residual block typically contains two convolutional layers and introduces a shortcut connection that skips one or more layers. Each convolutional layer in the residual block is followed by batch normalization and ReLU activation. The remaining four stages in ResNet-18 and ResNet-34 architectures consist of 3x3 convolutions with the following number of channels: 64, 128, 256, and 512.
MLP [59] An MLP (Multi-Layer Perceptron) network is a feedforward neural network architecture type. It consists of multiple layers of artificial neurons – in our experiments, we consider MLPs with 6,8,10,12 layers with ReLU activations (except last layer, which has linear activation). In our experiments, the underlying architecture has 1024 neurons per layer.
In VGGs, MLPs, and ResNets without skips, we use the  $98\%$  threshold to estimate the tunnel for the plots. In the case of ResNets, we use the  $95\%$  threshold. In the case of ResNets, we report the results for the 'conv2' layers. Due to computational constraints, we randomly choose a subset of 8000 features to compute the numerical rank.
Hyperparameters Hyperparameters used for neural network training are presented in the leftmost Table A.1. Each column shows the values of the hyperparameters corresponding to a different architecture. The presented hyperparameters are recommended for the best performance of these models on the CIFAR-10 dataset [60]. However, in experiments focused on continual learning scenario (Section 4.2), we refrain from decaying the learning rate and shorten the network's training to 30 epochs to mimic the actual settings used in continual learning settings.
Hyperparameters used for training linear probes in our experiment are presented in the rightmost table. Linear probes were trained with Adam optimizer instead of SGD.
Parameter	VGG	ResNet	MLP
Learning rate (LR)	0.1	0.1	0.05
SGD momentum	0.9	0.9	0.0
Weight decay	10-4	10-4	0
Number of epochs	160	164	1000
Mini-batch size	128	128	128
LR-decay-milestones	80, 120	82, 123	-
LR-decay-gamma	0.1	0.1	0.0
Parameter	Value
Learning rate	0.001
Weight decay	0
Number of epochs	30
Mini-batch size	512
In this article, we present the results of experiments conducted on following datasets: CIFAR-10 [61] CIFAR-10 is a widely used benchmark dataset in the field of computer vision. It consists of 60,000 color images in 10 different classes, with each class containing 6,000 images. The dataset is divided into 50,000 training images and 10,000 test images. The images in CIFAR-10 have a resolution of  $32 \times 32$  pixels.
CIFAR-100 [61] CIFAR-100 is a dataset commonly used for image classification tasks in computer vision. It contains 60,000 color images, with 100 different classes, each containing 600 images. The dataset is split into 50,000 training images and 10,000 test images. The images in CIFAR-100 have a resolution of  $32 \times 32$  pixels. Unlike CIFAR-10, CIFAR-100 offers a higher level of granularity, with more fine-grained categories such as flowers, insects, household items, and various types of animals and vehicles.
CINIC-10 [62] CINIC-10 is a dataset that stands as a 'bridge' between CIFAR-10 and ImageNet for image classification tasks. It combines 60,000 images of CIFAR-10, and 210,000 downsampled images of ImageNet. The images in CINIC-10 have a resolution of  $32 \times 32$  pixels.
Food-101 [63] The Food-101 dataset is a collection of food images commonly used for image classification tasks. It contains 101 categories of food, with each category consisting of 1,000 images. The dataset covers a wide range of food items from various cuisines, including fruits, vegetables, desserts, and main dishes.
102-Flower [64] The 102-Flower dataset is a collection of images representing 102 different categories of flowers. Each image in the dataset has a fixed resolution of 256 pixels in both width and height. The dataset provides a diverse set of flower images.
The Oxford-IIIT Pet Dataset [65] The Oxford-IIIT Pet Dataset is a collection of images of cats and dogs belonging to 37 different breeds. The dataset includes a total of 7,349 images.
Places-365 [66] The Places-365 dataset is a large-scale dataset consisting of 365 different scene categories. It contains over 1.8 million images, each depicting a specific scene or environment. The images in the dataset have 256x256 pixels.
STL-10 [67] STL-10 dataset is a benchmark image dataset consisting of 10 different classes, including various animals, vehicles, and household objects. It contains a total of 5,000 training images and 8,000 test images, each with a resolution of 96 pixels by 96 pixels. The STL-10 dataset is derived from the larger ImageNet dataset but is specifically designed for low-resolution image classification tasks.
SVHN [68] The SVHN (Street View House Numbers) dataset is a large-scale dataset for digit recognition from real-world images. It consists of labeled images of house numbers captured from Google Street View. The dataset includes over 600,000 images for training and 26,032 images for testing. Each image is RGB and has a resolution of 32 pixels by 32 pixels.
We preprocess all datasets with standardization, additionally we rescale each image to  $32px \times 32px$ .
A.3 Compute
We conducted approximately 300 experiments to finalize our work, each taking about three wall-clock hours on a single NVIDIA A5000 GPU. We had access to a server with eight NVIDIA A5000 GPUs, enabling us to parallelize our experiments and reduce total computation time. We estimate to perform over 2000 experiments (including failed ones) during the development phase of the project.
B Full results
B.1 MLPs
In this section, we present the results of the tunnel effect for MLP architectures with different depths. All models are trained on CIFAR-10, and their OOD properties are evaluated on ten randomly selected classes of CIFAR-100.
(a) In distribution
(b) Out of distribution
Figure 11: In and out of distribution linear probing performance for MLP-6 trained on CIFAR-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance is computed with random 10 class subsets of CIFAR-100.
(b) Out of distribution
Figure 12: In and out of distribution linear probing performance for MLP-8 trained on CIFAR-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance is computed with random 10 class subsets of CIFAR-100.
(a) In distribution
Figure 13: In and out of distribution linear probing performance for MLP-10 trained on CIFAR-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance is computed with random 10 class subsets of CIFAR-100.
(b) Out of distribution
(b) Out of distribution
Figure 14: In and out of distribution linear probing performance for MLP-12 trained on CIFAR-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance is computed with random 10 class subsets of CIFAR-100.
B.2 ResNet-34
In this section, we present the results of the tunnel effect for ResNet architectures with different depths. All models are trained on datasets CIFAR-10, CIFAR-100, and CINIC-10.
(a) In distribution
Figure 15: In and out of distribution linear probing performance for ResNet-34 trained on CIFAR-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance is computed with random 10 class subsets of CIFAR-100.
(a) In distribution
Figure 16: In and out of distribution linear probing performance for ResNet-34 trained on CIFAR-100. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance is computed on CIFAR-10.
(b) Out of distribution
Figure 17: In and out of distribution linear probing performance for ResNet-34 trained on CINIC-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance is computed on subset of ten classes from CIFAR-100.
B.2.1 VGG-19
In this section, we present the results of the tunnel effect for VGG architectures with different depths. All models are trained on datasets CIFAR-10, CIFAR-100, and CINIC-10.