The Tunnel Effect: Building Data Representations in Deep Neural Networks
Authors: Wojciech Masarczyk
Mateusz Ostaszewski
Ehsan Imani
Razvan Pascanu
Piotr Miłoś
Tomasz Trzcinski
Venue: NeurIPS 2023 poster (2023)
Keywords: representation learning
continual learning
training dynamics

Abstract: Deep neural networks are widely known for their remarkable effectiveness across various tasks, with the consensus that deeper networks implicitly learn more complex data representations. This paper shows that sufficiently deep networks trained for supervised image classification split into two distinct parts that contribute to the resulting data representations differently. The initial layers create linearly-separable representations, while the subsequent layers, which we refer to as \textit{the tunnel}, compress these representations and have a minimal impact on the overall performance. We explore the tunnel's behavior through comprehensive empirical studies, highlighting that it emerges early in the training process. Its depth depends on the relation between the network's capacity and task complexity. Furthermore, we show that the tunnel degrades out-of-distribution generalization and discuss its implications for continual learning.

The Tunnel Effect: Building Data Representations in Deep Neural Networks
Wojciech Masarezyk!'
Mateusz Ostaszewski!
Ehsan Imani
Razvan Pascanu
Piotr Mitog
Tomasz Trzciński146
Abstract
Deep neural networks are widely known for their remarkable effectiveness across various tasks, with the consensus that deeper networks implicitly learn more complex data representations. This paper shows that sufficiently deep networks trained for supervised image classification split into two distinct parts that contribute to resulting data representations differently. The initial layers create linearly- separable representations, while the subsequent layers, which we refer to the tunnel, compress these representations and have a minimal impact on the overall performance. We explore the tunnel's 's behavior through comprchensive empirical studies, highlighting that it emerges early in the training process. Its depth depends the relation between the network's 's capacity and task complexity. Furthermore, we show that the tunnel degrades -iittii-ittiitttionn generalization and discuss its implications for continual learning.
1 Introduction
Neural networks have been the powerhouse of machine learning in the last decade. A significant effort has been put into under- standing the mechanisms underlying their effectiveness. One example is the analysis of building representations in neural net- works applied to image processing [1. The consensus is that networks learn ers in the hierarchy by extracting more com- plex features than the layers before 3),
6000
0.9
Numerical rank
0.8
4000
0.7
Numerical
Linear probing ACC
2000
0.4
lay-
0.3
Layer
meaning that each layer contributes to the final network performance.
Figure 1:
The tunnel effect for VGG19 trained on the CIFAR- 10. In the tunnel (shaded area), the performance of linear probes attached each layer saturates (blue line), and the representations rank is steeply reduced (red dashed line).
Extensive research has shown that increas- ing network depth exponentially enhances capacity, measured as the number of linear regions that deep and overparameterized neural networks tend simplify representations with increasing
4 However, practical scenarios reveal
Warsaw University of Technology, Poland University of Alberta, Canada
3 "University College London, UK
*IDEAS NCBR, Poland
'Institute of Mathematics, Polish Academy Sciences, Poland
'Tooploox, Poland *Conesponding author: vojciech. mwsarczykkkmaill.ccomm
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

depth 17.8. This phenomenon arises because, despite their large capacity, these networks strive to reduce dimensionality and focus on discriminative patterns during supervised training [7HIO]. Moti- vated by these findings, we aim to investigate this phenomenon further and formulate the following research question:
How do representations depend the depth of a layer?
Our investigation focuses severely overparameterized neural networks through the prism of their representations as the core components for studying neural network behavior [2].
We extend the commonly held intuition that deeper layers are responsible for capturing more complex and task-specific features L3 14 by showing that neural networks after learning low and high-level features use the remaining layers for compressing obtained representations.
Specifically, demonstrate that deep neural networks split into two parts exhibiting distinct behavior. The first part, which we call the extractor, builds representations, while the other, dubbed the tunnel, propagates the representations further to the model's output, compressing them significantly. As show, this behavior has important implications for generalization, transfer learning, and continual learning. To investigate the tunnel effect, we conduct multiple experiments that support our findings and shed some light on the potential source of this behavior. Our findings can be summarized as follows:
We conceptualize and extensively examine the tunnel effect, namely, deep networks naturally split into the extractor responsible for building representations and the compressing tunnel, which minimally contributes to the final performance. The extractor-tunnel split emerges early in training and persists later on.
We show that the tunnel deteriorates the generalization ability on -fitt--diittiuution data.
We show that the tunnel exhibits task-agnostie behavior in a continual learning scenario. Simultancously it leads to higher catastrophic forgetting of the model.
2 The tunnel effect
The paper introduces and studies the dynamics of representation building in overparameterized deep neural networks called the tunnel effect. The following section validates the tunnel effect hypothesis in number of settings. Through an in-depth examination in Section [..]] we reveal that the tunnel effect is present from the initial stages and persists throughout the training process. Section 3.2 focuses on the diiti--distiibution generalization and representations compression. Section [33 hints important factors that impact the depth of the tunnel. Finally, in Section [ confront an auxiliary question: How does the tunnel's existence impact a model's adaptability to changing tasks and its vulnerability to catastrophic forgetting? To answer these questions we formulate our main claim as:
The tunnel effect hypothesis: Sufficiently large Largean neural networks develop a configuration in which network layers split into two distinct groups. The first one which we call the extractor, builds linearly- separable representations. The second one, the tunnel, compresses these representations, hindering the model's out-of-distribution generalization.
2.1 Experimental setup
To examine the phenomenon, we designed the setup to include the most common architectures and datasets, and use several different metrics to validate the observations.
Architectures We use three different families of architectures: MIP, VGGs, and ResNets. We vary the number of layers and width of networks to test the generalizability of results. See details in Appendix A.
Tasks We use three image classification tasks to study the tunnel effect: CIFAR-IO, CIFAR-100, and CINIC-10. The datasets vary in the number of classes: 10 for CIFAR-10 and CINIC-10 and 100 for CIFAR-100, and the number of samples: 50000 for CIFAR- 10 and CIFAR- 100 and 250000 for CINIC-10). See details in Appendix A.2
"We note that 'sufficiently large' covers most modern neural architectures, which tend to be heavily overpa- rameterized.

---

We probe the effects using: the average accuracy of linear probing, spectral analysis of represen- tations, and the CKA similarity between representations. Unless stated otherwise, we report the average of 3 runs.
Accuracy of linear probing: linear classification layer is attached to given layer & of the neural network. We train this layer on the classification task and report the average accuracy. This metric measures what extent ('s representations are linearly separable.
Numerical rank of representations: we compute singular values the sample covariance matrix for given layer l of the neural network. Using the spectrum, estimate the numerical rank of the given representations matrix as the number of singular values above certain threshold a. The threshold is set to 01 1e- 3, where Ø1 is the highest singular value of the particular matrix. The numerical rank of the representations matrix can be interpreted as the measure of the degeneracy of the matrix.
CKA similarity: is metric computing similarity between two representations matrices. Using this normalized index, we can identify the blocks of similar representations within the network. The definition and more details can be found in Appendix B
Inter and Intra class variance: inter-class variance refers the measure of dispersion or dissim- ilarity between different classes groups in dataset, indicating how distinct they are from each other. Intra-class variance, on the other hand, measures the variability within a single class or group, reflecting the homogeneity or similarity of data points within that class. The exact formula for computing these values can be found in Appendix
2.2 The main result
Table I presents our main result. Namely, we report the network layer at which the tunnel begins which we define as the point which the network reaches 95% (or 98%) of its final accuracy. We found that all tested architectures exhibit the extractor-tunne structure across all datasets used in the evaluation, but the relative length of the tunnel varies between architectures.
Architecture
# layers
Dataset
> 0.95
0.98
MLP
13
CIFAR-10
4 (31%)
(38%)
CIFAR-10
(36%)
(36%)
VGG
19
CIFAR-100
8 (42%)
8 (42%)
CINIC-10
7 (36%)
(36%)
CIFAR-10
20 (58%)
29 (85%)
ResNet
34
CIFAR-100 100
29 (85%)
30 (88%)
CINIC-10
17 (50%)
17 (50%)
Table The tunnel of various lengths is present in all tested configurations. For each architecture and datasct, we report the layer for which the average linear probing accuracy is above 0.95 and 0.98 of the final performance. The values in the brackets describe the part of the network utilized for building representations with the extractor.
We discuss the tunnel effect using MLP-12, VGG-19, and ResNet-34 CIFAR-10 example. The remaining experiments (for other architectures, datasets combinations) are available in Appendix B As shown in Figure and Figure 2] the early layers of the networks, around five for MI.P and eight for VGG, are responsible for building iinearly-separable representations. Linear probes attached to these layers achieve most of the network's final performance. These layers mark the transition between the extractor and the tunnel part (shaded area). the case ResNets, the transition takes place in deeper stages of the network the 19th layer.
3

---

7000
0.55
Numerical
rane
0.30
AcCura
5000
4000
AccurOC
probing
0.7
200
0.40
2000
Jmmrrcal rank Uncor probing ACC
0.
0.35
10
12
15 17 21 Layer
Layer
(a) MLP 12
ResNet
The tunnel effect for networks trained on CIFAR-10. The blue line depicts the linear probing accuracy, and the shaded area depicts the tunnel. The red dashed line is the numerical rank of representations. The spike the ResNet-34 representations rank coincides with the end of the penultimate residual stage.
Figure 2:
While the linear probe performance nearly saturates in the tunnel part, the representations are further refined. Figure shows that the numerical rank of the representations (red dashed line) is reduced to approximately the number of CIFAR-10 classes, which similar the neural collapse phenomenon observed in [1O]. For ResNets, the numerical rank is more dynamic, exhibiting a spike at 29th layer, which coincides with the end of the penultimate residual block. Additionally, the rank is higher than in the case of MLPs and VGGs.
Figure reveals that for VGG-19 the inter-class representations variation decreases throughout the tunnel, meaning that representations clusters contract towards their centers. At the same time, the average distance between the centers of the clusters grows (inter-class variance). This view aligns with the observation from Figure - where the rank of the representations drops to values close to the number of classes. Figure (right) presents an intuitive explanation of the behavior with UMAP plots of the representations before and after the tunnel.
Intra class variance
12
1.
Tunnel
02
10
Figure 3: The tunnel compresses the representations discarding indiscriminative features. Left: The evolution and inter and intra-class variance of representations within the VGG-19 network. Right: UMAP plot of representations before (7th layer) and after (18th layer) the tunnel.
To complement this analysis, we stud- ied the similarity of MLPs representa- tions using the CKA index and the LI norm of representations differences between the layers. Figure shows that the representations change signif- icantly in early layers and remain sim- ilar in the tunnel part when measured with CKA index (left). The norm of representations differences between the layers computed on the right side of Figure
20075
30030
Figure Representations within the tunnel are similar to each other for MLP with 12 hidden layers trained CIFAR- Comparison of representations with CKA index (left) and average LI norm of representations differences.

---

Tunnel effect analysis
This section provides empirical evidence contributing our understanding the tunnel effect. We hope that these observations will eventually lead to explanations of this phenomenon. In particular, we show that a) the tunnel develops early during training time, compresses the representations and hinders OOD generalization, and c) its size is correlated with network capacity and dataset complexity.
3.1 Tunnel development
Motivation In this section, we investigate tunnel development during training. Specifically, we try to understand whether the tunnel is a phenomenon exclusively related the representations and which part of the training is crucial for tunnel
Experiments We train VGG-19 on CIFAR-10 and save intermediate check- points every 10 epochs of training. We use these checkpoints to compute the
30-40
441
layer-wise weight change during training Figure ( and the evolution of numerical rank throughout the training (Figure
50.60
60-70
Epochs
3.00
Results Figure 5 shows that the split between the extractor and the tunnel is also visible in the parameters space. It could be perceived already at the early stages, and after that, its length stays roughly constant. Tunnel layers change significantly less than layers from the extractor. This result raises the question of whether the weight change affects the network's ' final output. Inspired by [16]. reset the weights of these layers to the state before optimization. However, the performance of the model de- teriorated significantly. This suggests that although the change within the tunnel's s parameters is relatively small, it plays an important role in the model's performance. showss that this apparent paradox can be better understood by looking at the evolution of representations' numerical rank during the very first gradient updates of the model. Throughout these steps, the rank collapses
20-100
130:220
130:40
1000
150 10:000
Layers
Figure 5: Early in training, tunnel layers stabilize. Color- coded: weight difference norm between consecutive check- points for each layer. Norm calculated as yaoi |0" 6 1 where 07 E R= is flattened weight matrix at layer d, check- point т. Values capped at 0.02 for clarity. The learning rate decayed (by 10 at epochs 80 and 120, and the scale adapted accordingly. Experiment: VGG-19 19 CIFAR-
70
Numerical
3000
90
ning steps
2000
values near-the-ne the-number of classes. It stays in this regime until the end of the training, meaning that the representations of the model evolve within a low-dimensional subspace. It remains to be understood if (and why) low-rank representations and changing weights coincide with forming linearly-separaal representations.
1000
11 Layer
32
Figure 6: The representations rank for deeper layers collapse early in training. The curves present the evolution of repre- sentations' numerical rank the first 75 training steps for layers the VGG-19 trained CIFAR-10 10. We present more detailed tunnel development analysis in Appendix G
Takeaway Tunnel formation is observable the representation and parameter space. It emerges early in training and persists throughout the whole optimization. The collapse in the numerical rank of deeper layers suggest that they preserve only the necessary information required for the task.
5

---

3.2 Compression and out-of-distribution generalization
Motivation Practitioners observe intermediate layers to perform better than the penultimate ones for transfer learning [17-19] However, the reason behind their effectiveness remains unclear [20]. In this section, we investigate whether the tunnel and, specifically, the collapse of numerical rank within the tunnel impacts the performance on out-of-distribution (OOD) data.
Experiments We train neural networks (MLPs, VGG-19, ResNet-34) on source task (CIFAR- 10) and evaluate it with linear probes on the OOD task, in this case, a subset of 10 classes from CIFAR-100. We report the accuracy of linear probing and the numerical rank of the representations.
LOWe pIOTT ACE
4000
prakiy ACC
570
LayaT
(a) MLP 12
(b) VGG-19
(c) ResNet 34
Figure 7:
The tunnel degrades the out- distriiiiitiitttionn performance correlated with the representations' numerical rank. The accuracy of linear probes (blue) was trained on the out-of-distribution data subset of 10 classes from CIFAR- 100. The backbone was trained on CIFAR-10. The shaded area depicts the tunnel, and the red dashed line depicts the numerical rank representations.
Results Our results presented in Figure reveal that the tunnel is responsible for the degradation of out-of-distribution performance. In most of our experiments, the last layer before the tunnel the optimal choice for training a linear classifier on external data. Interestingly, we find that the OOD performance is tightly coupled with the numerical rank of the representations, which significantly decreases throughout the tunnel.
To assess the generalization of our findings extend the proposed experimentation setup to additional dataset. To that end, train model different subsets of CIFAR- 100 while evaluating with linear probes on CIFAR-10. The results presented in Figure are consistent with our initial findings. We include detailed analysis with reverse experiment (CIFAR-10 - CIFAR-100), additional archi- tectures and datasets in the Appendix C
0.5
In all tested scenarios, observe consistent relationship between the start of the tunnel and the drop in OOD perfor- mance. An increasing number of classes in the source task result in a shorter tunnel and a later drop in OOD perfor- mance. In the fixed source task experiment (Appendix C the drop in performance occurs around the 7th layer of the network for all tested target tasks, which matches the start of the tunnel. This observation aligns with our earlier
Figure 8: Fewer classes in the source task create a longer tunnel, resulting in worse OOD performance. The network is trained on subsets of CIFAR-100 with different classes, and linear probes trained CIFAR- 10. Shaded areas depict respective tunnels.
findings suggesting that the tunnel is a prevalent characteristic of the model rather than artifact of a particular training or dataset setup.
Moreover, connect the coupling of the numerical rank of the representations with OOD perfor- mance, to a potential tension between the objective of supervised learning and the generalization of OOD setup. Analogous tension was observed in [1]] where adversarial robustness is at odds with model's 'ss accuracy. The results in Figure 7 align with the findings presented in Figure 3 demonstrating how the tunnel compresses clusters of class-wise representations. In work [22]. the authors show
that reducing the variation within each class leads to lower model transferability. Our experiments support this observation and identify the tunnel the primary contributor this effect.
Takeaway Compression of representations happening in the tunnel severely degrades the OOD performance of the model which is tightly coupled with the drop of representations rank.

---

3.3 Network capacity and dataset complexity
Motivation In this section, we explore what factors contribute to the tunnel's emergence. Based on the results from the previous section we explore the impact of dataset complexity, network's depth, and width tunnel emergence.
Experiments First, we examine the impact of networks' depth and width on the tunnel using MLPs (Figure 2. VGGs, and ResNets (Ta trained on CIFAR-10. Next, we train VGG-19 and ResNer34 on CIFAR- (IFAR-(10,100) (10,100) and CINIC-10 dataset investigating the role of dataset complexity on tunnel.
Results TigureDssowe that the depth of the MI.P network has no impact on the length of the extractor part. Therefore increasing the network's depth contributes only to the tunnel's length. Both extractor section and numerical rank remain relatively consistent regardless of the network's 's depth, starting the tunnel at the same layer. This finding suggests that overparameterized neural networks allocate fixed capacity for given task independent of the overall capacity of the model.
1/1
2
Numerical rank rank
VGG-16
8 (50%)
7 (445))
7 (41%)
600
byyers
0.50
Accuracy
VGG-19
8 (42%)
7 (37%)
7 (37%)
400
lapers
0.45
ResNet18
15 (83%)
13 (72%)
13 (72%)
layers
ResNet34
24 (68%)
20 (59%)
24 (08%)
200
Table 2:
Widening networks layers results in longer tunnel and shorter extractor. Column headings describe the factor in which we scale each model's 's base number of channels. The models were trained on the CIFAR-10 to the full convergence, We use the 95% threshold of probing accuracy estimate the tunnel beginning.
0.35
10
11
12
13
Figure 9: Networks allocate fixed capacity for the task, leading to longer tunnels in deeper networks. The extractor is consistent across all scenarios, with the tunnel commencing the 4th layer.
Results in Table 2 indicate that the tunnel length increases the width of the network grows, implying that representations are formed using fewer layers. However, this trend does not hold for ResNet34, as the longest tunnel is observed with the base width of the network. In the case of VGGs, the number of layers in the network does not affect the number of layers required to form representations. This aligns with the results in Figure 9
The results presented above were obtained from dataset with a consistent level of complexity. The data in demontaeesssssstttes that the number of classes in the dataset directly affects the length of the tunnel. Specifi cally, even though the CINIC- 10 training dataset three times larger than CIFAR-10 10, the tunnel length remains the same for both datasets. This suggests that the number of samples in the dataset does not impact the length of the tunnel. contrast, when examin- ing CIFAR-100 subsets, the tun- nel length for both VGGs and ResNets increase. This indicates a clear relationship between the dataset's s number of classes and the tunnel's length.
model
dataset
30%
50%
100%
CIFAR-10
6 (32%)
7 (37%)
7 (37%)
VGG-19
CIFAR-100
(42%)
(42%)
9 (47%)
CINIC-10
6 (32%)
(37%)
(37%)
CIFAR-10
19 (56%)
19 (56%)
21 (61%)
ResNet34
CIFAR-100
30 (88%)
30 (88%)
31 (91%)
CINIC-10
(27%)
(27%)
17 (50%)
Table 3: Networks trained on tasks with fewer classes utilize fewer re- sources for building representations and exhibit longer tunnels. Column headings describe the size of the class subset used in training. Within the (architecture, dataset) pair, the number of gradient steps during training in all cases was the same. We use the 95% threshold of probing accuracy to estimate the tunnel beginning.
Takeaway Deeper or wider networks result in longer tunnels. Networks trained on datasets with fewer classes have longer tunnels.
7

---

4 The tunnel effect under data distribution shift
Based on the findings from the previous section and the tunnel's negative impact on transfer learning. we investigate the dynamics of the tunnel in continual learning scenarios, where large models are often used on smaller tasks typically containing only few classes We focus on understanding the impact of the tunnel effect on transfer learning and catastrophic forgetting (23). Specifically, we examine how the tunnel and extractor are altered after training on a new task.
4.1 Exploring the effects of task incremental learning on extractor and tunnel
Motivation In this section, we aim to understand the tunnel and extractor dynamics in continual learn- ing. Specifically, we examine whether the extractor and the tunnel are equally prone to catastrophic forgetting.
Experiments We train VGG-19 two tasks from CIFAR-10. 10. Each task consists of 5 classes from the dataset. We subsequently train on the first and second tasks and save the corresponding extractors E, and tunnels T2, where i {1, 2] is the task number. We also save separate classifying head for trained on each task, that we use during evaluation.
Results As presented in Table 4 in any com- bination changing Ti to T/2 or vice versa have marginal impact the performance. This
First Task
Second Task
E1 + Ti
92.04%
56.8%
is quite remarkable, and suggests that the tun- nel is not specifie the training task. It seems that it compresses the representations in a task- agnostic way. The extractor part, on the other hand, is task-specific and prone to forgetting as visible in the first four rows of Table 4 the last two rows, we present two experiments that investigate how the existence of a tunnel affects the possibility of recovering from this catastrophic forgetting. In the first one, referred to as (F22 + (FT))), we use original data from Task 1 to retrain a classifying head attached on top of extractor Eg and the tunnel T1. As visible, it has minimal effect on the accuracy of the first task. In the second experiment, we attach a linear probe directly to the extractor representations (E2(FT)) This difference hints at a detrimental effect of the tunnel on representations' usability in continual learning.
Ej + T2
92.5%
58.04 %
Eg T2
50.84 %
93.94 %
E2 + Ti
50.66 %
93.72 %
En + (TFT)
56. 1%
(FT)
74.4%
In
Table 4: The tunnel part is task-agnostic and can be freely mixed with different extractors retaining the orig- inal performance. We test the model's performance on the first second task using combination of extrac- tor E, and tunnel T; from tasks t E [1,2). The last two rows (F"I") show how much performance can be recovered by retraining the linear probe attached to the penultimate layer Es Ti the last layer of the E'e
In Appendix D.I we study this effect further by training a tunnels on two tasks with different number of classes, where n1 > n2. In this scenario, we observe that tunnel trained with more classes (Ti) maintains the performance on both tasks, contrary to the tunnel (T²) that performs poorly on
Task 1. This is in line with our previous observations in Section 2.2 that the tunnel compresses to the effective number of classes.
These results present a novel perspective in the ongoing debate regarding the layers responsible for causing forgetting. However, they do not align with the observations made in the previous study (24). In Appendix D we delve into the origin of this discrepancy and provide a comprehensive analysis of the changes in representations with a setup introduced with this experiment and the CKA similarity.
Takeaway The tunnel's task-agnostic compression of representations provides immunity against catastrophic forgetting when the number of classes is equal. These findings offer fresh perspectives on studying catastrophic forgetting at specific layers, broadening the current understanding in the literature.
4.2 Reducing catastrophic forgetting by adjusting network depth
Motivation Experiments from this section verify whether is possible retain the performance of the original model by training a shorter version of the network. A shallower model should also exhibit less forgetting in sequential training.
8

---

Experiments We train VGG-19 networks with different numbers of convolutional layers. Each network is trained on two tasks from CIFAR-10. Each task consists of 5 classes from the dataset.
0.95
Accurac)
0.0
0.85
0.0
The results shown in Figure 10findicate that train- ing shorter networks yields similar performance compared to the original model. However, performance differences become apparent when the network becomes shorter than
Results:
0.75
layers in the network
Forgetting
the extractor part in the original model. This observation aligns with previous findings suggesting that the model requires certain capacity to perform the task effectively, Additionally, the shorter models exhibit significantly less forgetting, which corroborates the conclusions drawn in previous works [25 26 on the importance of network depth and architecture in relation to forgetting.
0.4
0.3
13 15
# of conv layers in the network
Training shorter networks from scratch gives a similar performance the longer counterparts (top) and results significantly lower forgetting (bottom). The horizontal lines denote the original model's 's performance. Top image: blue depicts accuracy first task, orange depicts accuracy the second task.
Figure 10:
Takeaway It is possible to train shallower networks that retain the performance of the original networks and expe- rience significantly less forgetting. However, the shorter networks need to have at least the same capacity as the extractor part of the original network.
Limitations and future work
This paper empirically investigates the tunnel effect, opening the door for future theoretical research tunnel dynamics. Further exploration could involve mitigating the tunnel effect through techniques like adjusting learning rates for specific layers. One limitation of our work is its validation within specific scenario (image classification), while further studies unsupervised self-supervised methods with other modalities would shed more light and verify the pertinence of the tunnel elsewhere.
In the experiments, MLPs or VGGs. This finding raises the question of whether the presence of skip connections plays a role in tunnel formation. In Appendix H we take the first step toward deeper understanding of this relationship by examining the emergence of tunnels in ResNets without skip connections.
observed that ResNet-based networks exhibited shorter tunnels than plain
6 Related work
The analysis of representations in neural network training is an established field (7-29 Previous studies have explored training dynamics and the impact of model width (50 but there is still a gap in understanding training dynamics 36
24 27 37). Works have investigated different
architectures' impact on continual learning [38
26 and linear models' behavior 39
Our work
builds upon studies examining specific layers' role in model performance [16
36
20
43
and sheds light on the origins observed behaviors
46).
Previous works have explored the role of specific layers in model performance [6
31
36
20
43
While some studies have observed block structure in neural network representations, their analysis was limited to ResNet architectures and did not consider continual learning scenarios. In our work, we investigate similar phenomenon, expanding the range of experiments and gaining deeper insights into its origins. On the other hand, visualization of layer representations indicates that higher layers capture intricate and meaningful features, often formed through combinations of lower-layer features [I3]. This phenomenon potentially accounts for the extension of feature extractors for complex tasks. Work [47] builds theoretical picture that stacked sequence models tend converge to fixed state with infinite depth and proposes a method to compute the finite equivalent of such networks. The framework of [47 encompasses previous empirical findings [48 50 Independently, research on pruning methods has highlighted greater neuron count in pruned final layers than in initial layers [5I), which aligns with the tunnel's 's existence. Furthermore, 52 5x authors showed that
training neural networks may lead to compressing information contained in consecutive hidden layers.

---

Yet another work [16] offers a different perspective, where the authors distinguish between critical and robust layers. highlighting the importance of the former for model performance, while individual layers from the latter can be reset without impacting the final performance. Our analysis builds upon this finding and further categorizes these layers into the extractor and tunnel, providing insights into their origins and their effects on model performance and generalization ability.
Our findings are also related the Neural Collapse (NC) phenomenon [00], which has gained recent attention 46 Several recent works 55 have extended the observation of NC and explored its impact on different layers, with a notable emphasis on deeper layers. [55] establishes link between collapsed features and transferabilityy. In our experiments, we delve into tunnel creation, analyzing
weight changes and model behavior in continual learning scenario, revealing the task-agnostic nature of the tunnel layers.
7 Conclusions
This work presents new insights into the behavior of deep neural networks during training. We discover the tunnel effect, an intriguing phenomenon in modern deep networks where they split into two distinct parts the extractor and the tunnel. The extractor part builds representations, and the tunnel part compresses these representations to a minimum rank without contributing to the model's s performance. This behavior is prevalent across multiple architectures and positively correlated with overparameterization, i.e., it can be induced by increasing the model's size decreasing the complexity of the task.
We emphasise that our motivation for investigating this phenomenon aimed at building a coherent picture encompassing both our experiments and evidence in the literature. Specifically, we aim to understand better how the neural networks handle the petresennttion-uuuddinn process in the context of depth.
Additionally, we discuss potential sources of the tunnel and highlight the unintuitive behavior of neural networks during the initial training phase. This novel finding has significant implications for improving the performance and robustness of deep neural networks. Moreover, we demonstrate that the tunnel hinders out-of-distribution generalization and can be detrimental in continual learning settings.
Overall, work offers new insights into the mechanisms underlying deep neural networks. Building on consequences of the tunnel effect we derive a list of recommendations for practitioners interested in applying deep neural networks to downstream tasks. In particular, focusing the tunnel entry features is promising when dealing with distribution shift due to its strong performance with OOD data. For continual learning, regularizing the extractor should be enough, the tunnel part exhibits task-agnostic behavior. Skipping feature replays in deeper layers or opting for a compact model without a tunnel can combat forgetting and enhance knowledge retention. For efficient inference, excluding tunnel layers during prediction substantially cuts computation time while preserving model accuracy, offering a practical solution for resource-constrained situations.
Acknowledgments and Disclosure of Funding
We would like to thank the reviewers for their work and insightful remarks that made this paper more complete. We especially thank Kamil Deja and other colleagues for valuable feedback and discussions during the project and writing the manuscript.
This research was funded by National Science Centre, Poland grant no 2020/39/B/ST6/01551 grant no 2022/45/B/STT/002117 17, grant no 2019/35/0//76//34464, and grant no 2022/45/N/ST6/4409888 The research was funded by Warsaw University of Technology within the Excellence Initiative: Research University (IDUB) programme. The authors have applied a CC BY license to any Author Accepted Manuscript (AAM) version arising from this submission, in accordance with the grants' open access conditions.
We gratefully acknowledge Polish high-performance computing infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2023/016321
10

---

References
[I] M. Klabunde, T. Schumacher, M. Strohmaier, and F. Lemmerich, "Similarity of neural network models: A survey of functional and representational measures," 2023.
[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," Advances in Neural Information Processing Systems 25, Curran Associates, Inc., 2012. !
[3] C. Olah, A. Satyanarayan, 1. Johnson, S. Carter, L. Schubert, K. Ye, and A. Mordvintsev, "The building blocks interpretabilityy" Distill, 2018. [
[4] M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein, "On the expressive power of deep neural networks," in Proceedings of the 34th International Conference Machine Learning, PMLR, 2017.
[5] G. Montúfar, R. Pascanu, Cho, and Y. Bengio, "On the number of linear regions of deep neural networks," in Proceedings the 27th International Conference Neural Information Processing Systems- Volume 2, 2014.
[6] M. Telgarsky, "Representation benefits of deep feedforward networks," 2015.
[7] G. Valle-Perez, C. Q. Camargo, and A. A. Louis, "Deep learning generalizes because the parameter-function map is biased towards simple functions," 2018. 2
[8] B. Hanin and D. Rolnick, "Complexity of linear regions in deep networks," 2019. 2
[9] M. Huh, H. Mobahi, R. Zhang, B. Cheung, P. Agrawal, and P. Isola, "The low-rank simplicity bias deep networks," 2021.
[10] V. Papyan, X. Han, and D. L. Donoho, "Prevalence of neural collapse during the terminal phase of deep learning training," Proceedings of the National Academy of Sciences, 2020. 2 TO
[11] T. Nguyen, M. Raghu, and S. Kornblith, "Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth," in International
Conference Learning Representations, 2020. 2 9 32
[12] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, "Similarity of neural network representations revisited," in Proceedings of the 36th International Conference on Machine Learning, PMLR, 2019. 2
[13] C. Olah, A. Mordvintsev, and L. Schubert, "Feature visualization," Distill, 2017. visualiiztiiiiaaiiiio....m 2
[14] G. Yang and H. Salman, "A fine-grained spectral perspective on neural networks," 2019.
2
[15] L. McInnes, Healy, and Melville, "Umap: Uniform manifold approximation and projection for dimension reduction," 2020. 4
[16] C. Zhang, S. Bengio, and Y. Singer, "Are all layers created equal?," 2022. 5
9
L0
[17] K. Choi, G. Fazekas, M. Sandler, and K. Cho, "Transfer learning for music classification and regression tasks," 2017. 6
[18] Lee and Nam, "Multi-level and multi-scale feature aggregation using pretrained convolu- tional neural networks for music auto-tagging." IEEE Signal Processing Letters, 2017.
[19] J. Shor, A. Jansen, R. Maor, O. Lang, O. Tuval, F. de Chaumont Quitry, M. Tagliasacchi, Shavitt, D. Emanuel, and Haviv, "Towards learning a universal non-semantic representation of speech," 2020. 6
[20] U. Evci, V. Dumoulin, H. Larochelle, and M. C. Mozer, "Head2toe: Utilizing intermediate representations for better transfer learning," 2022. 6
[21] D. Tsipras, S. Santurkar, Engstrom, Turner, and A. Madry, "Robustness may be at odds with accuracy," in International Conference on Learning Representations, 2019.
11

---

[22] Y. Wang, S. Tang, F. Zhu, L. Bai, R. Zhao, D. Qi. and W. Ouyang, "Revisiting the transferability of supervised pretraining: an mlp perspective." " in 2022 IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), 2022. G
[23] R. M. French, "Catastrophic forgetting in connectionist networks." Trends in cognitive sciences, 1999. g
[24] V. V. Ramasesh, E. Dyer, and M. Raghu, "Anatomy of catastrophic forgetting: Hidden represen-
tations and task semantics," in International Conference on Learning Representations, 2020. 8
9
[2 22
[25] S. I. Mirzadeh, A. Chaudhry, D. Yin, H. Hu, R. Pascanu, D. Gorur, and M. Farajtabar, "Wide neu- ral networks forget less catastrophically." Proceedings of the 39th International Conference on Machine Learning, 2022. 9
[26] S. I. Mirzadeh, A. Chaudhry, D. Yin, T. Nguyen, R. Pascanu, D. Gorur, and M. Farajtabar, "Architecture matters in continual learning." " 2022. 9
[27] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, "How transferable are features in deep neural networks?," Advances in neural information processing systems, 2014. 9
[28] A. H. Williams, E. Kunz, S. Kornblith, and S. Linderman, "Generalized shape metrics on neural representations," Advances in Neural Information Processing Systems, 2021.
[29] Y. Lou, C. E. Mingard, and S. Hayou, "Feature learning and signal propagation in deep neural networks," in International Conference on Machine Learning. 2022. 9
[30] N. Maheswaranathan, A. Williams, M. Golub, S. Ganguli, and D. Sussillo, "Universality and individuality in neural dynamics across large populations of recurrent networks," Advances in neural information processing systems, vol. 32, 2019. 9
[31] M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein, "Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretabilityy" 2017. 9
[32] A. Thompson, Y. Bengio, and M. Schönwiesner, "The effect of task and training on inter- mediate representations in convolutional neural networks revealed with modified rv similarity analysis," " 2019.
[33] A. Jacot, F. Gabriel, and C. Hongler, "Neural tangent kernel: Convergence and generalization in neural networks," Advances in neural information processing systems, 2018.
[34] A. Lewkowycz, Y. Bahri, E. Dyer, J. Sohl-Dickstein, and G. Gur-Ari, "The large learning rate phase of deep learning: the catapult mechanism." arXiv preprint arXiv:2003.022 02218, 2020.
[35] C. Wei, J. D. Lee, Liu, and T. Ma, "Regularization matters: Generalization and optimization of neural nets vs their induced kernel," Advances in Neural Information Processing Systems, vol. 32, 2019. 9
[36] Y. Chen, A. Yuille, and 7.. Zhou, "Which layer is learning faster? a systematic exploration of layer-wise convergence rate for deep neural networks," in The Eleventh International Conference on Learning Representations, 2023. 9
[37] B. Neyshabur, H. Sedghi, and C. Zhang, "What is being transferred in transfer learning?." in Advances in Neural Information Processing Systems (H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, eds.), 2020. 9
[38] S. I. Mirzadeh, A. Chaudhry, D. Yin, H. Hu, R. Pascanu, D. Gorur, and M. Farajtabar, "Wide neural networks forget less catastrophically." in International Conference on Machine Learning. 2022.
[39] Lee, S. S. Mannelli, C. Clopath, S. Goldt, and A. M. Saxe, "Maslow's 's hammer in catastrophic forgetting: Node re-use vs. node activation," in International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, Proceedings of Machine Learning
Research, PMLR, 2022. 9
12

---

[40] S. Lee, S. Goldt, and A. Saxe, "Continual learning in the teacher-student setup: Impact of task similarity," Proceedings of the 38th International Conference on Machine Learning, PMLR, 2021.
[41] Evron, E. Moroshko, R. Ward, N. Srebro, and Soudry, "How catastrophic can catastrophic forgetting be in linear regression?," in Proceedings of Thirty Fifth Conference on Learning Theory, 2022.
[42] W. J. Maddex. S. Tang, P. G. Moreno, A. G. Wilson, and A. Damianou, "Fast adaptation with linearized neural networks," 2021.
[43] T. Nguyen, M. Raghu, and S. Kornblith, "On the origins of the block structure phenomenon in neural network representations," Transactions on Machine Learning Research, 2022. g
[44] Z. Zhu, T. Ding. J. Zhou, X. Li. C. You, Sulam, and Q. Qu, "A geometric analysis of neural collapse with unconstrained features," Advances in Neural Information Processing Systems, vol. 34, 2021.
9 T0
[45] T. Galanti, A. György, and M. Hutter, "On the role of neural collapse in transfer learning," " in International Conference on Learning Representations, 2022.
[46] L. Hui, M. Belkin, and P. Nakkiran, "Limitations of neural collapse for understanding general- ization in deep learning," arXiv preprint arXiv:2202.08888, 2022. 9 I0
[47] S. Bai, J. Z. Kolter, and V. Koltun, "Deep equilibrium models," Advances in Neural Information Processing Systems, vol. 32, 2019.
[48] R. Dabre and A. Fujita, "Recurrent stacking of layers for compact neural machine translation models," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 6292- 6299, 2019. 9
[49] S. Bai, Z. Kolter, and V. Koltun, "Trellis networks for sequence modeling," arXiv preprint 1rXiv:810000668. 2018.
[50] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser, "Universal transformers," arXiv preprint arXiv: 1807:03807 2018.
[51] A. Morcos, H. Yu, M. Paganini, and Y. Tian, "One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers," Advances in neural information processing systems, vol. 32, 2019. 9
[52] A. M. Saxe, Y. Bansal, J. Dapello, M. Advani, A. Kolchinsky, B. D. Tracey, and D. D. Cox, "On the information bottleneck theory of deep learning," Journal of Statistical Mechanics: Theory and Experiment, vol. 2019, no. 12, p. 124020, 2019.
[53] R. Shwartz-Ziv and N. Tishby, "Opening the black box of deep neural networks via information," " arXiv preprini arXiv: 1oXiv:/703..0 0000000800 2017. 9
[54] A. Ansuini, A. Laio, J. H. Macke, and D. Zoccolan, "Intrinsic dimension of data representations in deep neural networks," Advances in Neural Information Processing Systems, vol. 32, 2019.
10
[55] X. Li. S. Liu, J. Zhou, X. Lu, C. Fernandez-Granda Z. Zhu, and Q. Qu, "Principled and efficient transfer learning of deep models via neural collapse," arXiv preprint arXiv:2212.12206, 2022.
10
[56] A. Rangamani, M. Lindegaard, T. Galanti, and T. A. Poggio, "Feature learning in deep classifiers through intermediate neural collapse," in International Conference on Machine Learning, pp. 28745, PMLR, 2023. 10
[57] K. Simonyan and Zisserman, "Very deep convolutional networks for large-scale image recognition," in International Conference on Learning Representations, 2015. 15
13

---

[58] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778,
2016. 15
[59]
G. Bebis and M. Georgiopoulos, "Feed-forward neural networks," leee Potentials, vol. 13, no. 4,
pp. 27-31, 1994. 15
[60] Z. Liu, M. Sun, T. Zhou, G. Huang, and Darrell, "Rethinking the value of network pruning." 2019. [5
[61] A. Krizhevsky, "Learning multiple layers of features from tiny images," 2009. 15 16
[62] L. N. Darlow, Crowley, A. Antoniou, and Storkey, "Cinic-10 not imagenet or cifar-10," arXiv preprint arXiv:
2018. T6
[63] L. Bossard, M. Guillaumin, and L. Van Gool, "Food-101 - miinigg discriminative components with random forests," in European Conference on Computer Vision, 2014. T6
[64] M.-E. "Nilsback and A. Zisserman, ""automated flower classification over a large number of classes"." in "Proceedings of the Indian Conference Computer Vision, Graphics and Image Processing". "2008". 16
[65] O. M. "Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar, ""cats and dogs"." in "IEEE Conference on Computer Vision and Pattern Recognition", "2012". 16
[66] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. "Places: 10 million image datahase for scene recognition," JEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2017. [6
[67] A. Coates, A. Ng, and H. Lee, "An Analysis of Single Layer Networks in Unsupervised Feature Learning," in AISTATS, 2011. https://cs. stanf edu/ wtt.aaffrr..ddaaaaaaattt//aaaaap
ottps://cs.staafford..
coatesleeng_aaie wwtgiggassttt__eeee 2011. pdf [6
[68] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, "Reading digits in natural images with unsupervised feature learning," NeurIPS, 2011. 16
[69] S. Ebrahimi, F. Meier, Calandra, Darrell, and M. Rohrbach, "Adversarial continual learning." 2020.
29
[70] L. Pellegrini, G. Graffieti, V. Lomonaco, and D. Maltoni, "Latent replay for real-time continual learning." " 2020. 29
[71] Balakrishnama and A. Ganapathiraju, "Linear discriminant analysis-a brief tutorial," 1998.
33
[72] A. Islam, C.-F. Chen, R. Panda, L. Karlinsky, R. Radke, and Feris, "A broad study
the transferability of visual representations with contrastive learning," in Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 8845-8855, 2021. 33
[73] N. Zhao, Z. Wu, Lau, and S. Lin, "What makes instance discrimination good for transfer learning?," 2020. 33
14

---

A Experimental setup.
A.1 Architectures and hyperparameters
In this section, we detail the model architectures examined in the experiments and list all hyperpa- rameters used in the experiments.
VGG [57] In the main text use two types VGG networks, namely VGG-19 and VGG-16. Both architectures consist of five stages, each consisting of combination of convolutional layers with ReLU activation and max pooling layers. The VGG-19 has 19 layers, including 16 convolutional layers and three fully connected layers. The first two fully connected layers are followed by ReLU activation. On the other hand, VGG- 16 has a total of 16 layers, including 13 convolutional layers and three fully connected layers. In additional experiments, VGG-16. The base number of channels in consecutive stages for VGG architectures equals 64, 128, 256, 512, and 512.
extend our analysis by VGG-11 11 and
ResNet (58) experiments, we utilize two variants of the ResNet family of architectures, i.e., ResNet-18 and ResNet-34. ResNet-N is five-staged network characterized by depth, with total of N layers. The initial stage consists single convolutional layer with kernel size x 7 and 64 channels and ReLU activation, followed by max pooling x 2, which reduces the spatial dimensions. The subsequent stages are composed of residual blocks. Each residual block typically contains two convolutional layers and introduces shorteut connection that skips one more layers. Each convolutional layer in the residual block is followed by batch normalization and ReLU activation. The remaining four stages in ResNet-18 18 and ResNet-34 architectures consist of 3x3 convolutions with the following number of channels: 64, 128, 256, and 512.
MLP [59] An MLP (Multi-Layer Perceptron) network is a feedforward neural network architecture type. consists of multiple layers of artificial neurons - in our experiments, we consider MLPs with 6,8, 10,12 layers with ReLU activations (except last layer, which has linear activation). In our experiments, the underlying architecture has 1024 neurons per layer.
In VGGs, MLPs, and ResNets without skips. we use the 98% threshold to estimate the tunnel for the plots. In the case of ResNets, we use the 95% threshold. In the case of ResNets, we report the results for the 'conv2' layers. Due to computational constraints, we randomly choose a subset of 8000 features to compute the numerical rank.
Hyperparameters Hyperparameters used for neural network training are presented in the leftmost Table A.I Each column shows the values of the hyperparameters corresponding to different architecture. The presented hyperparameters are recommended for the best performance of these models on the CIFAR-10 dataset (60). However, in experiments focused continual learning scenario (Section 4.2), we refrain from decaying the learning rate and shorten the network's 's training 30 epochs to mimic the actual settings used continual learning settings.
Hyperparameters used for training linear probes in our experiment are presented in the rightmost table. Linear probes were trained with Adam optimizer instead of SGD.
Parameter
VGG
ResNet
MLP
Learning rate (LR)
0.1
0.1
0.05
Parameter
Value
SGD momentum
0.9
0.9
0.0
Learning rate
0.001
Weight decay
10
10 4
0
Weight decay
0
Number of epochs
160
164
1000
Number of epochs
30
Mini-batch size
128
128
128
Mini-bated size
512
LR- Re-decay-milesstee
80, 120
82, 123
LR J.R-eccy-gaamm
0.1
0.1
0.0
A.2 Datasets
In this article, we present the results of experiments conducted on following datasets: CIFAR-10 61 CIFAR-10 is widely used benchmark dataset the field of computer vision. It consists 60,000 color images in 10 different classes, with each class containing 6,000 images. The dataset is divided into 50,000 training images and 10,000 test images. The images in CIFAR-10 have a resolution of 32 32 pixels.
15

---

CIFAR-100 100 is dataset commonly used for image classification tasks in computer vision. It contains 60,000 color images, with 100 different classes, each containing 600 images. The dataset is split into 50,000 training images and 10,000 test images. The images in CIFAR- 100 have resolution of 32 32 pixels. Unlike CIFAR-IO, CIFAR- offers a higher level of granularity, with more fine-grained categories such as flowers, insects, household items, and various types of animals and vehicles.
CIFAR-100 61
CINIC-10 [62] CINIC-10 is dataset that stands as a "bridge' between CIFAR-10 and ImageNet for image classification tasks. combines 60,000 images of CIFAR-10, and 210,000 downsampled images of ImageNet. The images in CINIC-10 have resolution of 32 x 32 pixels.
Food-101 633 The Food-101 101 dataset is collection of food images commonly used for image classification tasks. It contains 101 categories of food, with each category consisting of 1,000 images. The dataset covers wide range of food items from various cuisines, including fruits, vegetables, desserts, and main dishes.
102-Flower [64] The 102-Flower dataset is collection of images representing 102 different categories of flowers. Each image in the dataset has fixed resolution of 256 pixels in both width and height. The dataset provides diverse set of flower images.
The Oxford-IIT Pet Dataset 65 The Oxford IITT Pet Dataset is collection of images of cats and dogs belonging to 37 different breeds. The dataset includes a total of 7,349 images.
Places-365 666] The Places-365 dataset is large-scale dataset consisting of 365 different scene categories. It contains over 1.8 million images, each depicting a specific scene or environment. The images in the dataset have 256x256 pixels.
STL-10 67 STL-10 dataset is benchmark image dataset consisting of 10 different classes, including various animals, vehicles, and household objects. It contains a total of 5,000 training images and 8,000 test images, each with a resolution of 96 pixels by 96 pixels. The STL-10 dataset is derived from the larger ImageNet dataset but is specifically designed for low-resolution image classification tasks.
SVHN 68 The SVHN (Street View House Numbers) dataset is a large-scale dataset for digit recognition from real-world images. consists of labeled images of house numbers captured from Google Street View. The dataset includes over 600,000 images for training and 26,032 images for testing. Each image is RGB and has resolution of 32 pixels by 32 pixels.
We preprocess all datasets with standardization, additionally we rescale each image to 32px x 32pr.
A.3 Compute
We conducted approximately 300 experiments finalize work, each taking about three wall-clock hours on a single NVIDIA A5000 GPU. We had access to server with eight NVIDIA A5000 GPUs, enabling to parallelize our experiments and reduce total computation time. We estimate perform over 2000 experiments (including failed ones) during the development phase of the project.
16

---

B
Full results
B.1 MIPs
In this section, we present the results of the tunnel effect for MLP architectures with different depths. All models are trained on CIFAR-10, 10, and their OOD properties are evaluated on ten randomly selected classes of CIFAR-100.
600
0.30
600
0.45
Accuracy
400
0.45
400
0.40
0.40
0.35
probing
0.35
Layer
Layer
(a) In distribution
(b) Out distribution
Figure 11: In and of distribution linear probing performance for MLP-6 trained on CIFAR-10 The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance is computed with random 10 class subsets of CIFAR-100.
Numerical mink
rank
0.55
Lincrr probing ACC
0.50
600
800
0.50
0.45
0.45
3.10
200
Numerica
0.40
200
0.35
0.35
0.30
8 9
(a) distribution
(b) Out distribution
Figure 12: In and out of distribution linear probing performance for MLP-8 trained on CIFAR- 10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed with random 10 class subsets of CIFAR-100.
0.55
ronk probing
600
600-
0.50
3,5
Numertcal
400
0.45
400-
0.40
2
3
8
10 11
2
3
8
9
10
11
(a) distribution
(b) Out distribution
Figure 13: In and out of distribution linear probing performance for MLP-10 trained on CIFAR- 10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed with random 10 class subsets of CIFAR-100.
17

---

Numerical
0.55
3.50
0.30
0.43
Numerical
0.45
Accura
Numerical
0.40
0.38
0.40
Numericaa rens Uncar probing ACC
0.35
12
30
12
Layer
Layer
In distribution
(b) Out of distribution
Figure 14: In and out of distribution linear probing performance for MLP-12 trained on CIFAR- 10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed with random 10 class subsets of CIFAR-100.
B.2 ResNet-34
In this section, we present the results of the tunnel effect for ResNet architectures with different depths. All models are trained on datasets CIFAR-10, CIFAR-100, and CINIC-10.
1000
7000
0.75
0,70
3000
3.8
Accirac
5000
Acourad
4000
4000
0.7
0.50
0.55
2000
Numercaa tank
0.6
2000
Namer ranic
0.50
1000
Lnner probing ACE
1000
Unser prob ACC
0.45
19
21 Layer
25
29
33
31
13 15
Layer
(a) In distribution
(b) Out of distribution
Figure 15: In and out of distribution linear probing performance for ResNer-34 trained on CIFAR-10 10. The shaded depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed with random 10 class subsets of CIFAR-100.
7000
Numerca of man
077
7000
Numetra mank
Uneor prab ng ACC
Uncor prob ACC
5000
Accurag
5000
Accurag
4000
4000
2000
-.2
2000
17
Layer
Layer
(a) In distribution
(b) Out distribution
Figure 16: In and out of distribution linear probing performance for ResNet-34 trained on CIFAR-I00. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and of distribution) respectively. Out-of-distribution performance is computed on CIFAR-10.
18

---

1500
1500
Numoficll
1250
0:63
3250
1000
Accursic
2000
Numerical
Numtericall ank
0.5
probing
500
500
0.5
250
0.1
10
36
18 Layer
20
22
30
32
10 12
18 Layer
16
20
22
26
28
30
In distribution
(b) Out of distribution
Figure 17: In and out of distribution linear probing performance for ResNet-34 trained on CINIC- -10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed on subset of classes from CIFAR-100.
B.2.1 VGG-19
In this section, present the results of the tunnel effect for VGG architectures with different depths. All models are trained on datasets CIFAR-10 10, CIFAR-100, and CINIC-10.
0.9
Numerial
Liner omb ng
Numerical
4003
Accuracy
4002
0.6
2000
2002
Numertaal mink
3
10
12
10 12 14
16
(a) distribution
(b) Out distribution
Figure 18: In and out of distribution linear probing performance for VGG-19 trained on CIFAR- 10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed with random 10 class subsets of CIFAR-100.
6000
0.7
5000
07
2
Numertcal
0.8
2000
-033
2000
0.5
Kumerical 1/1k Unerr wwebing ACC
Numorica ank
Unear prob ACC
10
15 16
18
10 13 Layer
Layer
In distribution
(b) Out of distribution
Figure 19: In and out of distribution linear probing performance for VGG-19 trained on SIFAR-100. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distributioo performance computed on CIFAR-10.
19

---

Numerica rakk
Unear ng ACC
Numencall
Accurag
00.
0.5
Numericl
-044
0.3
2
10
17
10 11 Layer
13
14
16
18
(a) distribution
(b) Out distribution
Figure 20: In and out of distribution linear probing performance for VGG-19 trained CINIC-10 10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed on subset of ten classes from CIFAR-100.
20

---

B.3 Dataset complexity experiments
B.3.1
ResNet-34
3000
Numefial mirk
2500
2500
Unee smbing
9.7
Numerical
2000
0.90
2000
0.85
ACCUFaL
1000
1000
04
500
0.75
500
0.70
11
15
17
23
25
27
29
31
31
13
15
17
19
21
23
25
27
29
Layer
(a) distribution
(b) Out distribution
Figure 21: In and out of distribution linear probing performance for ResNet-34 trained 3-class subset of CIFAR-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribae performance is computed with random 10 class subsets of CIFAR-100.
0.90
3000
ACcuraC
3000
Accursicy
Numerical
0.25
0.75
0.5
1000
1000
11
13
15
17
25
29
31
13
15
25
29
31
Layer
Layer
In distribution
(b) Out of distribution
Figure 22: In and out of distribution linear probing performance for ResNet-34 trained on a 5-class subset of CIFAR-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed with random 10 class subsets of CIFAR-100.
7000
Numor OI
7000
Numafici FanK
0.75
Liner ACE
Linear probing ACE
0.70
3000
Accuracy
Accuracy
4000
4000
0.7
0.00
0.58
2000
1000
1000
0.50
Layer
Layer
(a) In distribution
(b) Out of distribution
Figure 23: In and out distribution linear probing performance for ResNet-34 trained on CIFAR- 10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance is computed with random 10 class subsets of CIFAR-100.
21

---

Numar ca fank
Numatia rank
2500
Lineer probing ACE
0.7
2500
Linear probing ACE
Accuracy
Accuraa
0,6
1500
0.5
3500
1000
0.3 1000
17 19
13
27
29
Layer
Layer
(a) In distribution
(b) Out of distribution
Figure 24: In and out of distribution linear probing performance for ResNet-34 trained on 30-class subset of CIFAR-100. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. performance is computed on CIFAR-10.
4003
0.8
4000
prob
Liner probing
3000
3002
Accuracy
0.5
0.6
0.2
11
13
15
17
19
21
23
25
27
29
31
7
9
11
13
15
17
23
25
27
29
31
Layer
(a) distribution
(b) Out distribution
Figure 25: In and out of distribution linear probing performance for ResNet-34 trained 50-class subset of CIFAR-100 The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distributte performance is computed on CIFAR-10.
7000
7000
9.7
6000
5000
5000
-0.5
Accurac
0.7
Accuracy
4000
0.3
0.5
17
Layer
Layer
(a) In distribution
(b) Out of distribution
Figure 26: In and out of distribution linear probing performance for ResNet-34 trained on CIFAR-100 The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed CIFAR-10.
22

---

0.78
Numedra mark
1000
Unear pr0b ACC
Numencall
800
800
Numerica
0.74
0.5
0.77
200
0.70
200
02
10
26
18 Layer
24 28
Layer
(a) In distribution
(b) Out of distribution
Figure 27: In and out of distribution linear probing performance for ResNer-34 trained on a 3-class subset of CINIC-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. performance is computed on CIFAR-10.
1200
0.50
1200
1000
0.78
3000
srobing
0.7
800
Numer cal ranc
Unear probing ACC
0.74
0.6
Accuract cu
0.5
400
400
0.70
0.50
10
12
14
16
22
24
26
28
30
10
12
14
16
18 Layer
20
22
24
26
28
30
Layer
(a) distribution
(b) Out distribution
Figure 28: In and out of distribution linear probing performance for ResNet-34 trained 5-class subset of CINIC-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribun performance is computed on CIFAR-10.
1300
1300
Numerial
smbing
2.65
0.7
Numertcal
1000
Accuracy
3000
Accuracy
0.0
probing
500
500
0.5
250
250
0.4
Layer
Layer
(a) In distribution
(b) Out of distribution
Figure 29: In and out of distribution linear probing performance for ResNet-34 trained on CINIC- 10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed CIFAR-10.
23

---

B.3.2 VGG-19
2500
2500
Numerica ark
Uneer smbi
2000
2000
Accuracy
1900
1500
05
1000
1000
0.4
500
Numeniaa rane Linar probing ACC
500
Layer
Layer
(a) In distribution
(b) Out of distribution
Figure 30: and out of distribution linear probing performance for VGG-19 trained on a 3-class subset of CIFAR-10 10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribtioo performance is computed with random 10 class subsets of CIFAR-100.
Numzrical tark
Lineer smuh ng ACC
Accurao BCy
2500
merertaal
0.5
1000
FARK
probing
0.2
18
Layer
Layer
(a) In distribution
(b) Out of distribution
Figure 31: and out of distribution linear probing performance for VGG-19 19 trained on 5-class subset of CIFAR-10 10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribute performance is computed with random 10 class subsets of CIFAR-100.
6000
0.
60.00
= Linear srob ng ACC
Numefial mark
Numerical
4009
Accurad
Numerfical
4000
0.6
2000
Numericl
2
10 12
16
10 Layer
15
(a) distribution
(b) Out distribution
Figure 32: In and out of distribution linear probing performance for VGG-19 trained on CIFAR 10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed with random 10 class subsets of CIFAR-100.
24

---

Numelra: mark
2500
Unear p0 ACC
0.7
Numencall
2000
9.6
Accuracy
2000
Accuracy
1509
1500
Numerical
pmoting
16
14
26
Layer
Layer
(a) In distribution
(b) Out of distribution
Figure 33: and out distribution linear probing performance for VGG- 19 trained on a 30-class subset of CIFAR-100. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. performance is computed on CIFAR-10.
0.7
3000
3000
Numerfcal
Accuracy
2000
2000
2000
Numerical
Numorical
0.4
-0.2
2
10
10 12 Layer
15
16
(a) distribution
(b) Out distribution
Figure 34: In and out of distribution linear probing performance for VGG-19 19 trained on 50-class subset of CIFAR-100. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribute performance is computed on CIFAR-10.
5000
-06
5000
4000
Accurac)
4000
Accuracy
2000
03
0.5
1000
Numerical Pank
1000
Numerical rank
Userr pmaing ACC
Uneer sooh ng ACC
10
10 Layer
Layer
(a) In distribution
(b) Out of distribution
Figure 35: In and out of distribution linear probing performance for VGG-19 trained on CIFAR- The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed CIFAR-10.
25

---

6005
6000
Numzfiaa ank
Unner
Numencall
4000
Accuracy
04
0.3
Fank probing
2617
Layer
Layer
(a) In distribution
(b) Out of distribution
Figure 36: and out of distribution linear probing performance for VGG-19 trained on 3-class subset of 100. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. performance is computed on CIFAR-10.
6000
5000
Unner srob ng
0.81
Accura BCy
Numerfcal
4000
Numerfcal
4000
2000
2000
Numerical
0.3
2
10 16
5
10 Layer
14
(a) distribution
(b) Out distribution
Figure 37: In and out of distribution linear probing performance for VGG-19 19 trained 5-class subset of CINIC-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribuute performance is computed on CIFAR-10.
Numerical rank
6000
60.00
ng
0.7
4000
Accuracy
Numerical
4000
005
Accuracy
2000
2000
Numerical Pank
Unerr pmaing ACC
10
Layer
Layer
(a) In distribution
(b) Out of distribution
Figure 38: In and out of distribution linear probing performance for VGG-19 trained on CINIC- 10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank, and the blue curve depicts linear probing accuracy (in and out of distribution) respectively. Out-of-distribution performance computed CIFAR-10.
26

---

C Out of distribution generalization extended results
In this experiment, we aim determine if the tunnel consistently decreases the performance of models on out-of-distribution (OOD) datasets. To achieve this, we trained YGG-19 and ResNet34 models on CIFAR-10 and conducted linear probing on various OOD datasets. The results, depicted in Figure 39 are consistent across both the tested models and the datasets used. Notably, in all cases except for training dataset (CIFA--1O, we observe a decline in performance starting from beginning of the tunnel and continuing degrade further. In the case of ResNet-34, there is a spike in performance at the 29th layer, which aligns with the findings the main paper. Interestingly, the dataset that exhibits the least deterioration is STL-10. This dataset consists of 10 classes, 9 of which overlap with classes found in CIFAR-10 10. However, the images STL-10 are sampled from the ImageNet dataset. These results suggest that models can generalize well to OOD data that share semantic similarities with the in-distribution data. Note that the linear probing performance was normalized for better presentation.
20
10
Normaieed
chunto (Fontos Rix roaft
(19r102
paboT precmoos
0.
pEaST
piccioes
VGG-19
(b) ResNet-34
Figure 39: Out of distribution normalized linear probing performance for different datasets. The shaded area depicts the tunnel, different colors depict the linear probing performance on given dataset. Note that all the results are normalized for clarity of presentation.
The following experiment complements the analysis presented in the main paper, aiming to further explore the degradation of -iittiidiitiittonnn performance caused by the tunnel effect. In this particular setup, the network is trained using CIFAR- 10, and linear probes are trained and evaluated us- ing subsets of CIFAR-100 with varying numbers of classes. The results, depicted Figure440 consistently demonstrate that regardless of the number of classes used to train the linear probes, the tunnel effect consistently leads formance. These findings confirm our ob- servations from the main paper, indicating that the tunnel effect is prevalent charac- teristic of the model rather than peculiar artifact of the dataset or training setup.
1.0
2
200
Accuracy
0.6
0.4
decline in their per-
10
12
19
Layer
Figure 40: Source task with fixed number of classes results a tunnel consistently degrading the OOD per- formance for different number of classes. VGG-19 is trained on CIFAR- 10 and linear probes are trained on different subsets of CIFAR-100 with different numbers of classes. The tunnel is marked with shaded color.
27

---

D
Exploring the effects of task incremental learning on extractor and tunnel extended results
Task
Task
1.0
bottom up top down
0.8
Accuracy
0.6
0.4
5
10
15
5
10
15
# of layers reset
# of layers reset
Figure 41: Substituting layer experiment. VGG-19 trained on the sequence two tasks on split- CIFAR1O. First task 3 class, second task 7 classes.
Task 1
Task
1.0
0.9
0.8
Accuracy
0.7
0.6
0.5
bottom up top down
0.4
10
15
10
15
# of layers reset
layers reset
Figure 42: Substituting layer experiment. VGG-19 trained the sequence of two tasks split- CIFARIO. First task class, second task 5 classes.
Task
Task
1.0
0.8
Accuracy
0.6
0.4
bottom up
top down
15
15
# of layers reset
# of layers reset
Figure 43: Substituting layer experiment. VGG-19 trained on the sequence of two tasks on split- CIFAR10. First task 7 class, second task 3 classes.
First, we focus on examining reset layers experiment case sequence tasks with different number of classes in section D.1
In this section we discuss in greater details experiment from section 4.1
Next, we discuss the the discrepancies between results and results presented in work 224
28

---

D.1 Different number of classes in source and target tasks.
In this experiment, our aim is gain a better understanding of tunnel immunity to catastrophic forgetting. Specifically, we are interested in exploring scenarios where the number of classes differs in each task. To analyze this scenario, we conducted three experiments using the VGG-19 19 network. We trained the network on sequences of two tasks, each composed of CIFAR-10 classes with different splits: (3, 7), (5, 5), and (7,
During training, we saved the model after completing the first and second tasks, denoted as M1 and My respectively. When we refer to Mi + MM we mean that the network consists the first layers with parameters from after completing the first task, combined with the remaining layers from the network after completing the second task.
Here, instead of a table we present the results using plots, see Figure 41 for the reference. The y-axis values represent the accuracy of the model when substituting certain number of layers, denoted as The blue plot represents the situation where we substitute layers starting from the bottom (M2-- + MM+1""), while the orange plot represents the opposite scenario (M2 + MT+1") Please note the change in subscripts.
In Figure we observe that when the tasks have an equal number of classes, the tunnel is preserved perfectly. Specifically, substituting 10 layers from the top down does not affect the performance on the second task, and substituting more than 8 layers does not yield any improvement on the first task.
Conversely, in Figure 41] substituting more than 7 layers from the bottom up does not lead to any improvement in the second task. Additionally, substituting any layers from the top down actually harms the performance on the second task. This suggests that while the network encountered more classes in the second task, it built upon the existing tunnel, maintaining its performance on the first task.
In the opposite scenario, where the second task involves fewer classes, a reverse situation is observed. Substituting any layers from the top down negatively impacts the performance on the first task, while substituting 10 layers from the top down does not affect the performance on the second task. This suggests that the network successfully reused a portion of the tunnel from the first task while discarding the unnecessary part.
D.2
On the primary source of catastrophic forgetting on split-CIFARIC task.
There is an ongoing discussion surrounding the layers responsible for driving the phenomenon of forgetting. In study (24). authors claim that "Higher layers are the primary source of catastrophic forgetting on split CIFAR- 10 task." However, our findings present a different perspective compared to the conclusions drawn in that research. Specifically, the results presented in Section 4.1 and Section D.1 indicate that there exist continual learning scenarios where the deeper layers do not contribute to catastrophic forgetting. Instead, we show that in certain scenarios the earlier layers are responsible for performance degradation, while the deeper layers remain unaffected due to their task-agnostic nature. This insight is particular significance because many studies have built upon the assumption that mainly deeper layers are responsible for catastrophic forgetting, potentially leading to inadequate or inefficient continual learning mechanisms (69 70).
It is important to note that the tunnel hypothesis effect holds for overparameterized networks. In contrast, the authors (24] evaluated their claims using the VGG-13 network, with the width of the layers reduced by factor of four. This discrepancy plays crucial role in tunnel formation, as it reduces the model's capacity. Figures 44 49 illustrate the disparity between these models in the reset experiment.
From this comparison, the main conclusion emerges that the question of "which layers are the primary source of catastrophic forgetting?" is nuanced and contingent upon multiple factors.
29

---

Task 1
Task
0.9
bottom up top down
0.8
Accuracy
0.7
0.6
0.5
2
10
12
0
2
4
10 12
# of layers reset
# of layers reset
Figure 44: Substituting layer experiment. VGG-13, width factor = 0.25, trained on the sequence of Two tasks split-CIFARIO.
Task
Task
bottom top down
0.8
Accuracy
0.7
0.6
0.5
0.4
10
# of layers reset
# of layers reset
Figure 45: Substituting layer experiment. VGG-13, width factor 1, trained the sequence of two tasks on split-CIFAR10.
Task 1
Task
0.9
bottom
top
0.8
Accuracy
0.7
0.6
0.5
0.4
10
4
10
# of layers reset
layers reset
Figure 46: Substituting layer experiment. VGG-13, width factor 2, trained the sequence of two tasks on split-CIFARIO.
30

---

Task 1
Task
bottom up top down
0.8
Accuracy
0.6
5
10
15
0
5
10
15
# of layers reset
# of layers reset
Figure 47: Substituting layer experiment. VGG-19, width factor = 0.25, trained on the sequence of Two tasks split-CIFARIO.
Task
Task
bottom top down
0.9
0.8
Accuracy
0.7
0.6
15
# of layers reset
# of layers reset
Figure 48: Substituting layer experiment. VGG-19, width factor trained the sequence of two tasks on split-CIFAR10.
Task 1
Task
bottom top
Accuracy
0.6
0.4
10
15
15
# of layers reset
layers reset
Figure 49: Substituting layer experiment. VGG-19, width factor 2, trained the sequence of two tasks on split-CIFARIO.
31

---

E CKA similarity
The Centered Kernel Alignment (CKA) similarity is measure commonly used in machine learning and neuroscience to quantify the similarity between two representations or feature spaces. It provides a way to assess the similarity of representations learned by different models layers, even when the representations may have different dimensionalities. CKA is invariant to orthogonal transforma- tions, such as isotropic scaling, permutations, reflections and rotations. This invariance property is particularly valuable when comparing representations that have undergone different preprocessing or normalization steps. By accounting for the underlying relationships between representations while being insensitive to orthogonal transformations, CKA enables a more meaningful and reliable assessment of similarity, aiding in tasks such as model comparison, representation learning, and understanding the neural code. For computational reasons, given by the following formula.
modification of CKA index
CKA similarity: Let X; E R Y; E RmxXP be the representations matrices from 8dh minibatch of two layers of m samples and pi and p2 number features respectively. Similarly to [I. we estimate CKA index by averaging over k mini-batches:
E HSIC X Y,Y,)
sCKA
(I)
E HSIC (X.X) X,X, E HSIC (Y,Y7, Y.Y.)
where HSIC is an unbiased estimate or of the HSIC score
1 K11 L1
HSIC(K, L) nln - 3) tr(KL)+ + (n 1)(n 2)
n - - KL1
(2)
where L (L).
CKA is normalized similarity index, hence value means that representations matrices are identical.
In Figure 50 we present the plots of representations similarity for ResNet-18 and VGG-19 networks.
ResNet-34
VGG-19
1
7
12
23
25
27
29
31
Layers
Layers
(a) ResNet-34
(b) VGG-19
Figure 50: CKA-similarity across network's layers for ResNet-34 and VGG-19.
F Inter and Intra class variance
Understanding the concepts inter-class and intra-class variance is particularly important in the context of deep neural network representations analysis for classification tasks. In this context,
32

---

inter-class variance refers to the variability between different classes categories of data. On the one hand, they capture the representation of the linear separability of a given task (7T]. On the other hand, intra-class variance is an indicator of representations transferabilityy 72 73
Let Xj
R4 xx be the representations matrix for samples from 5th class. (X;)l²) is the intra-class variance, where is representa- tion of sample (Xj) is the mean representation of representations matrix Xj. and C is the number classes. Then C(C-1) E De E ll (X,) f (Xx) I|? the inter-class variance.
t E &2 EmEx, |
G Tunnel development
In this section, we provide a more detailed analysis of the evolution of numerical rank in VGG-19 dataset. In this experiment, we save the checkpoint of the network
7000
150
140
6000
130
120
Numerical rank
110
100
every epoch and caleulate its numerical rank. The re- sults are depicted in Fig- 51
4000
3000
E
2000
50
Initially, during the early epochs, the rank collapses primarily in the deeper lay-
40
1000
30
20
10
ers. Throughout the train- distinct
ing process, patterns can be observed. Firstly, the numerical rank of representations from the earlier layers tends in- crease. Secondly, the nu- merical rank of representations from the deeper layers decreases. Interestingly, the place of this transition aligns with the beginning of the tunnel in the network. Once the numerical rank in deeper layers collapsed in the first gradient steps, shown in Figure it remained collapsed throughout the whole training.
2
5
8
10 Layer
9
11
12
13
14
17
Figure 51: Evolution of numerical rank of VGG-19 representations throughout the training on CIFAR-10.
H ResNets without skip connections
In this section, we delve into the impact of skip connections on the formation the tunnel effect. To investigate this relationship, we trained ResNet models (ResNet-18 18 and ResNet-34) without skip connections on CIFAR-10 and conducted the same analysis used the main paper. Specifically, we examined the linear probing performance for both in-distribution and out-of-distributio data and estimated the representations' numerical rank. The results, depicted in Figure 52 and Figure highlight the significance of skip connections in the formation of the tunnel effect. Firstly, in the absence of skip connections (Plainnets), the tunnel effect is slightly more pronounced, with the model's performance saturating layers earlier than standard ResNet networks. Secondly, the rank of the representations exhibits a more predictable pattern without the spike at 29th layer. This suggests that the spike in the numerical rank and in OOD performance is related the skip connections. Interestingly, the numerical rank in both networks is higher than in the case of VGGs. The reason for this difference needs a further investigation. Lastly, the presence or absence of skip connections performance. However, in the absence of skip
does not alter the degradation of connections, the deterioration is more severe, aligning with the observation that it correlates with the numerical rank of the representations.
33

---

7000
oal
7000
Nimetica rant.
Unew prob ng ACC
Linar probing ACC
5000
Accuracy
5000
0.55
Accurac)
0.00
1000
3002
0.55
2000
0.50
5
11
13
15
17
5
9
11
13
15
17
Layer
Layer
(a) In distribution
(b) Out distribution
Figure 52: In and out of distribution linear probing performance for ResNet-1e 18 without skip con- nections trained on CIFAR-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respec- tively. Out-of-distribution performance is computed with random 10 class subsets of CIFAR-100.
7000
0.90
7000
0.70
rank
0.85
0.55
5000
0.50
5000
0.00
Accuracy
4000
4000
0.70
0.53
2000
Numerica
0.65
2000
0.45
1000
1000
11
13
15
17
21
23
25
27
29
7
11
13
19
25
27
29
31
(a) distribution
(b) Out distribution
Figure 53: In and out of distribution linear probing performance for ResNet-34 without skip con- nections trained on CIFAR-10. The shaded area depicts the tunnel, the red dashed line depicts the numerical rank and the blue curve depicts linear probing accuracy (in and out of distribution) respec- tively. diitr---titiitttion performance is computed with random 10 class subsets of CIFAR-100.
34


Reviews:
[{"review_id": "KKmNR9v9iA", "reviewer": "Reviewer_xLcw", "summary": "This paper shows an effect of deep neural networks when trained for classification tasks — the initial layers create linearly separable features, and the later layers collapse the features for the final prediction. This phenomenon is explored with extensive experiments. ", "questions": "- Can the tunnel effect, or the numerical rank provide practical guidance on which feature we should use for different problems, e.g. out-of-distribution data, continual learning? \n\n- Following up on the previous question, if this tunnel effect can provide guidance, how can we use this tunnal effect while model training and/or model inference. \n\n\n", "limitations": "The limitations of the paper is relatively thoroughly discussed. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper explored a very interesting phenomenon of how the features are learned dynamically through layers.\n\n- The paper performed extensive experiments to show how this \"tunnel effect\" affects the model performance under different settings. \n\n- The experiments are well-designed, and the results are demonstrated well. \n\n\n", "weaknesses": "-  This is not the first/only paper that discovered some similar effects under a similar setting, therefore, a more comprehensive comparison with them in the related work section as well as a clarification on the contribution should be added. e.g. [1] examines the feature \"intrinsic dimension\", [2]  analyzes the generalization effects of feature neural collapse on in-domain and out-of-domain data.\n\n- The authors mentioned the network can be split into the extractor and the tunnel which compress the features, however, it can be difficult to systematically split the network, it seems the author also did not provide a systematically split of the network based on the numerical rank. \n\n\n[1] Ansuini et. al. Intrinsic dimension of data representations in deep neural networks.\n[2] Li et. al. Principled and Efficient Transfer Learning of Deep Models via Neural Collapse.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments.\n\n**Considering similar findings in other papers, enhance the related work section with comparisons and clarify contributions.**\n\nWe have added the references [1] and [2] to the related works section:\n\n>Several recent works~\\cite{ansuini2019intrinsic,li2022principled,rangamani2023feature} have extended the observation of NC and explored its impact on different layers, with a notable emphasis on deeper layers. \\cite{li2022principled} establishes a link between collapsed features and transferability. In our experiments, we delve into tunnel creation, analyzing weight changes and model behavior in a continual learning scenario, revealing the task-agnostic nature of the tunnel layers.\n\n**Systematically dividing the network is challenging, and the authors did not offer a numerical rank-based network split.**\n\nWe are still determining if we understand this concern correctly (if the following answer is off, please let us know). Our experiments observed relatively straightforward patterns, where the accuracy flattens (e.g.,> 90% of the final value) coincides with the point where the rank starts to drop. This gives a rather sharp boundary between the extractor and the tunnel.\n\n\n**Can the tunnel effect, or the numerical rank provide practical guidance on which feature we should use for different problems, e.g. out-of-distribution data, continual learning?**\n\nWe have compiled a paragraph of recommendations based on new insights from the tunnel hypothesis (see below). We include these recommendations in the conclusions section. \n\n> Our tunnel hypothesis has multiple consequences relevant to downstream tasks. \n>\n> In particular, focusing on the tunnel entry features is promising when dealing with distribution shift due to its strong performance with OOD data. \nFor continual learning, regularizing the extractor should be enough, as the tunnel part exhibits task-agnostic behavior. Skipping feature replays in deeper layers or opting for a compact model without a tunnel can combat forgetting and enhance knowledge retention. \n>\n>For efficient inference, excluding tunnel layers during prediction substantially cuts computation time while preserving model accuracy, offering a practical solution for resource-constrained situations.\n\n\n**How can we use this tunnal effect while model training and/or model inference?**\n\nOne practical approach is to conduct model inference without the tunnel layers. As these layers do not contribute to the final performance. In continual learning, if tasks have the same class count, the tunnel remains unchanged; thus, not updating or removing the tunnel layers can be considered for new tasks.\n\n\nIf our responses have adequately addressed your concerns, we kindly request your support and considerating of improving your score. If you have any further concerns or additional points to raise, we are eager to address them. Your insights are valuable in enhancing the quality and impact of our research."}, {"review_id": "MbWPwdoakP", "reviewer": "Reviewer_dmw5", "summary": "This paper proposes the Tunnel Hypothesis: Training of deep networks splits layers into two distinct phases: (1) extractor phase and (2) tunnel phase. Extractor phase learns the linearly separable features whereas the tunnel phase compresses the representations. The authors provide evidence towards degrading effects of this tunnel effect on OOD (out-of-distribution) samples. Further, better understanding of continual learning may be possible due to the proposed hypothesis.", "questions": "Please see above.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper has following strengths: \n\n1. The OOD representation section was very interesting.\n\n2. Applications to continual learning based on knowledge gained from the detailed analysis from this paper can also be useful.\n\n3. Overall, a lot of work has gone into this paper (many experiments).\n", "weaknesses": "The paper has following weaknesses:\n\n1. Many of the observations are not particularly surprising. I think a lot depends on capacity of the network and the difficulty of the task at hand. It is not surprising that for a given task and a type of network, representations get learned up to a certain layer and then the remaining layers simply make the representations more compact. Indeed, if the task becomes more complex or more difficult, the tunnel length would reduce (since more layers would be spent trying to learn more complex features). This is clear from Table 1 where for ResNet-34, the tunnel length significantly reduces when going from CIFAR-10 to CIFAR-100 (from 20 to 29 layers). Thus, when the task became more difficult, more layers started getting used to learn better features. Similarly, for simple MLPs, we know that beyond a certain depth, depth does not help (this comes from many other studies, e.g., that analyze gradient properties, etc.: beyond a certain depth and without skip connections, adding more layers does not help due to vanishing gradients). The insight that “many later layers do not contribute significantly to accuracy” is also known and is precisely why “deep network pruning” literature is not able to prune later layers too significantly.\n\n2. In the introduction section, the authors claim that “they challenge the commonly held intuition that deeper layers are responsible for capturing more complex and task-specific features”. I do not see any evidence that they changed this commonly held view. In fact, many of their experiments reinforce exactly the common viewpoint. Specifically, the authors show that later layers hurt the OOD performance. This indicates that the later layers got specialized towards the within-distribution task which is why they hurt the OOD task. Hence, if the commonly held view is being reinforced with the evidence provided by the authors, there is nothing particularly surprising about the findings.\n\n3. I think the proposed work can have significant value in the field of continual learning and also multi-task/multi-modal learning if the observed insights can be used to guide novel architectural designs and/or loss functions. Unfortunately, the current work (despite a lot of hard work) only plays around with toy datasets in that problem space. If the authors can build further on the new insights and create new models/losses for the aforementioned areas, this can be an impactful work.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their valuable feedback. \n\nWe acknowledge some presentation issues, which we discuss in the general answer. Please let us know if you find it satisfactory. We discuss the other issues below.\n\n\n**Increasing depth does not help because representations are learned up to a certain layer and subsequent layers make them compact. (Weakness #1)**\n\nWe have thoroughly researched existing literature but have yet to find any work that describes this particular behavior. However, we are open to any references or sources you might be aware of in this context, and we would be grateful for any insights you can provide.\n\nFurthermore, credible alternatives, without a tunnel, could be easily imagined. For example, considering the perspective of gradient flow, one would anticipate a contrary pattern. As the gradient norm diminishes when we move away from the output layer, one could conjecture that the upper layers should \"learn more\" than the bottom ones.\n\nLast, the contribution of this work lies in conceptualizing the tunnel hypothesis underpinning several observations made in referenced literature and extensively examining its implications in transfer learning and continual learning settings.\n\n\n\n**Lack of support for challenging the view that deep layers learn task-specific features. (Weakness #2)**\n\nThe scattered presentation could have harmed clarity and readability. Thus, we provide a concise summary of these results in one place:\n\n1. The tunnel layers, which demonstrate a decrease in out-of-distribution (OOD) performance, do **not** contribute significantly to the in-distribution (ID) performance (Figure 1, 2, 7).\n2. We want to emphasize that the tunnel layers remain unchanged when optimizing the network for a novel task (see Table 4). This transferability across tasks (without tunnel retraining) falsifies the task-specific view of the deeper layers. \n3. Measured with CKA, representations at the tunnel's beginning and end are roughly the same. This suggests that no significant transformation was applied to them in the tunnel layers (Figure 4).\n\nWe are not aware of any such findings being comprehensively reported in the literature. Thus, our research provides a new, structured, and arguably valuable perspective. We'd be happy to engage in further discussion or amend our work with prior work if the reviewer offers any specific references.\n\n \n**If the authors can build further on the new insights and create new models/losses for the aforementioned areas, this can be an impactful work. (Weakness #3)**\n\nWe agree this is an exciting area for further research. We have compiled a paragraph of recommendations based on new insights from the tunnel hypothesis (see below). We include these recommendations in the conclusions section. \n\n> Our tunnel hypothesis has multiple consequences relevant to downstream tasks. \n>\n> In particular, focusing on the tunnel entry features is promising when dealing with distribution shift due to its strong performance with OOD data. \nFor continual learning, regularizing the extractor should be enough, as the tunnel part exhibits task-agnostic behavior. Skipping feature replays in deeper layers or opting for a compact model without a tunnel can combat forgetting and enhance knowledge retention. \n>\n>For efficient inference, excluding tunnel layers during prediction substantially cuts computation time while preserving model accuracy, offering a practical solution for resource-constrained situations.\n\n \n**Indeed, if the task becomes more complex or more difficult, the tunnel length would reduce (since more layers would be spent trying to learn more complex features).** \n\nThis observation is indeed compelling and is further explored in section 3.3 (Table 3), where we confirm the intuition that training on more complex datasets results in shorter tunnels. However, the complexity of the dataset is understood here as the number of classes. Note that more complex datasets regarding the number of samples do not impact the tunnel's length (compare CIFAR-10 vs CINIC-10 in Table 1). Formalizing this relationship with more rigorous mathematical treatment is an exciting direction for future work.\n \n \n**The insight that “many later layers do not contribute significantly to accuracy” is also known and is precisely why “deep network pruning” literature is not able to prune later layers too significantly.**\n\nYour comment suggests a link between layer depth and pruning potential, but we're unsure how this aligns with our findings and whether we should reference any works that confirm those observations in our paper. We'd be happy to see a clarification of this remark and amend our submission accordingly.\n \n \nWe again thank the reviewer for raising important issues. We hope that our answers are satisfactory. If not, we'd be happy to provide more details. Otherwise, we'd appreciate if the reviewer reconsidered the final score of our submission.\n "}, {"review_id": "cBG1y0aPGH", "reviewer": "Reviewer_EeeK", "summary": "The tunnel effect is described for deep overparameterized networks, whereby early layers form a linearly separable representation while later layers form a \"tunnel\" which passes this representation to the output without substantial change, other than compression (reducing its rank, a.k.a. discarding information). A large number of experiments show that this effect occurs in a variety of different models, datasets, and training times, and impacts out-of-distribution and continual learning settings. The effect implies that the capacity of a given architecture/dataset is fixed.", "questions": "In figure 6, why is the numerical rank for higher layers already less than that of lower layers at initialization? Given that random matrices tend to be near full-rank, this points to an issue with how numerical rank is computed (e.g. the threshold for determining rank should be layer-dependent, since different layers have different weight scales).", "limitations": "ImageNet-scale networks and Transformer architectures are out of scope of this work. But given the importance of such networks to transfer learning applications, the obvious next step is to investigate the tunnel effect in these networks. Unlike the CIFAR/CINIC-scale models in this work, it is unclear what degree of overparameterization is occurring in ImageNet-scale networks and Transformers.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper provides a clear and comprehensive argument for both the existence of its hypothesis and its effects on several relevant subdomains of DNN research. The experiments are thorough, well organized, and address many potential concerns (such as consistency of the observations across different models and datasets). Some subtle details are handled correctly (e.g. use of unbiased CKA estimator in appendix E). The effect is convincing and could point to significant improvements in how transfer and continual learning is handled.", "weaknesses": "The procedure for computing the numerical rank should be given fully (e.g. the value of the threshold $\\sigma$ is not explicitly stated). In particular, figure 6 indicates some issues (see Questions below). Also, figure 6 is missing shading for the tunnel region after training, which would help readers reference other figures.\n\nThe difference in weights in figure 5 is not fully explained (e.g. $\\tau$ should be defined). Also, while the difference is normalized relative to $\\theta^0$ (the initial weight norm?, or is it the number of weights?), a better comparison would be relative to the norm of the mean change iterations/epochs, since the learning rate changes over training. As a result, the experiment cannot rule out the possibility that the tunnel is changing relative to other layers after the learning rate is reduced.\n\nOverall the evidence from CKA is lacking (figure 4) as it is only shown for a MLP. However, this is not a central piece of evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the effort you put into reviewing our work. We find your feedback valuable and helpful in improving the quality of the paper!\n\n**The procedure for computing the numerical rank should be given fully.**\n\nWe added the following clarification in the text:\n> The threshold $\\sigma$ is set to $\\sigma_{1}*1e-3$, where $\\sigma_{1}$ is the highest singular value of the particular matrix.\n\n\n**Also, Figure 6 is missing shading for the tunnel region after training, which would help readers reference other figures.**\n\nWe added shading in Figure 6.\n\n**The difference in weights in figure 5 is not fully explained (e.g. should be defined).**\n\nWe fixed the notation in the caption of Figure 5 and explained the formula in more detail.\n\n>The tunnel layers stabilize early during training. Color depicts the norm of weights difference between subsequent checkpoints (rows) for a given layer (columns). Norm is computed with $\\frac{1}{\\sqrt{n m}}\\left\\|\\theta_d^{\\tau_1}-\\theta_d^{\\tau_2}\\right\\|_{2^{\\prime}}$ where $\\theta_d^\\tau \\in \\mathbb{R}^{n m}$ is flattened matrix of weights from $d$-th layer at checkpoint $\\tau$. The values are clipped at $0.02$ for better presentation. At epochs 80 and 120, the learning rate is decayed. VGG-19, CIFAR-10.\n\n**The experiment cannot rule out the possibility that the tunnel is changing relative to other layers after the learning rate is reduced.**\n\nThank you for pointing out that possibility. To check whether tunnel formation is reversed after decaying the learning rate, we split Figure 5 into three parts, with the scale adjusted by the decay factor. The results in Figure 1 in the attached PDF align with the previously presented material. Despite the decaying learning rate, the model's behavior is consistent throughout the training, and the split between the extractor/tunnel part is visible in all phases. We will update the text and Figure 5 accordingly in the paper.\n\n**The CKA evidence is limited to an MLP in Figure 4, which is not a pivotal part of the argument.**\n\nIn Figure 3 in the attached PDF, we present the plots for CKA similarities for VGG-19 and ResNet-34 (we also added the Figure to the paper). VGG-19 exhibits similar representations for deeper layers (starting around layer 8). In the case of ResNets, the pattern is less visible. We suspect this can be caused by skip-connections and their impact on the final representations, as mentioned in the last paragraph of the Limitations section.\n\n**In Figure 6, why is the numerical rank for higher layers already less than that of lower layers at initialization?**\n\nAs we explained earlier, the threshold $\\sigma$ used for estimating the numerical rank is computed separately for each layer based on their top-singular value making it layer-dependent. Also, please note that the dimensionalities of the lower-layer representations are higher. Due to computational constraints, we subsample these layers to have at most 8000 features (e.g. [1]). Yet, the effect of decreased rank in upper layers is also present in MLPs where the dimensionality of the representation is constant across layers, and we do not subsample representations.\n\n[1] G. Alain, Y. Bengio - Understanding intermediate layers using linear classifier probes\n\nIf your concerns have been sufficiently addressed in our responses, we humbly seek your support for the paper.\n\n"}, {"review_id": "FWEOTMH1Pk", "reviewer": "Reviewer_Kpac", "summary": "The paper offers an empirical study of deep neural networks. The focus is on the role of intermediate layers in building a representation that is linearly separable and can eventually solve the task. The work highlights the fact that this linearly separable, low-rank representation emerges at a depth that is a fraction of the total depth (usually 1/3 to 2/3 for common image classification setups). Layers before such point are named \"the extractor\", and layers afterward \"the tunnel\". The authors further investigate the role of \"the tunnel effect\" on transfer learning and catastrophic forgetting. ", "questions": "- Section 3.1 put emphasis on the fact that the tunneling effect emerges \"early\" during training. How do you assess what early is? Is the effect happening before/same time/after train and test losses start decreasing? It would be nice to have a comparison of these different timescales to better assess the claim of the section. \n- In the introduction, the authors claim to challenge the idea that deeper layer capture more complex and task-specific features. What are the results that question this idea? For example, the observation that the transfer learning performance deteriorates after neural collapse (i.e. in the tunnel part) is consistent with the idea that the representation in these layers becomes task-specific. The fact that more complex tasks require a longer \"extractor\" is consistent with the idea that more complex tasks require more complex representations to be built and more complex representations are built in a hierarchical manner through more layers.\n- I find the results of Figure 9 exemplificative of the fact that the network uses always the first layers to solve the task, independently of its total depth. It would be nice to have the same Figure for a CNN.\n- Minor: What is the difference between blue and orange dots in Figure 10 (top).", "limitations": "The authors clearly highlight that their work focuses only on image classification and that a study for other data modalities would be interesting. I think this is a fair limitation and does not impact my evaluation of their work.\nAlso, at first sight, the role of skip connection would be very relevant for the effect studied in this work, and possibly it would deserve to be discussed more in the main text. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The main strength of the paper is the thoroughness of the empirical study. Although only pertaining to image classification (as mentioned by the authors), the study includes a wide range of benchmark datasets and architectures, and shows results with plots that are easy to parse.\nConnecting the tunneling effect to practical recommendations for transfer learning is another interesting point. ", "weaknesses": "Although the authors put a strong emphasis on the novelty of their results, the fact that a linearly separable representation emerges well before the final layer is not completely novel. See e.g. [1] for an analysis using similar methods.\nAlso, the reasons behind the so-called paradox that motivates the work are not completely clear. It's not evident how the fact that capacity increases with depth, is at stake with the fact that there is an \"inductive bias\" toward simple solutions. \nIn my view, the motivation paragraph in the introduction would need to be clarified, and the conclusion should clearly state how this paradox has been addressed by the paper.\n\n[1] Feature learning in deep classifiers through Intermediate Neural Collapse, Rangamani, Akshay; Lindegaard, Marius; Galanti, Tomer; Poggio, Tomaso.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. \n\n**The fact that a linearly separable representation emerges well before the final layer is not completely novel.**\n\nWe do agree that this observation has been made in multiple works, including those we cite. We consider the primary value of our work to be *conceptualization and providing a coherent perspective of how representations are formed*. In particular, we do not mean to claim that all observations are new. We instead attempt to build standalone empirical evidence to support our tunnel hypothesis and explore its consequences in transfer learning or continual learning scenarios. We have added the reference to the Related Works section:\n\n>Several recent works~\\cite{ansuini2019intrinsic,li2022principled,rangamani2023feature} have extended the observation of NC and explored its impact on different layers, with a notable emphasis on deeper layers. \\cite{li2022principled} establishes a link between collapsed features and transferability. In our experiments, we delve into tunnel creation, analyzing weight changes and model behavior in a continual learning scenario, revealing the task-agnostic nature of the tunnel layers.\n\n**Clarification of the so-called paradox.**\n\nWe acknowledge that using the term “paradox” may have caused confusion, and we have revised the introduction and conclusion to eliminate any ambiguity. The revised paragraph in the introduction now reads as follows:\n\nThe revised paragraph in the introduction now reads as follows:\n\n> Extensive research has ..... to simplify representations with increasing depth [38,7]. This phenomenon arises because, despite their large capacity, these networks strive to compress hidden representations by focusing on discriminative patterns during supervised training [38, 7, 8, 30]. Motivated by these findings, we aim to investigate this phenomenon further and build a comprehensive picture to answer the following research question: \n>\n> \"How do representations depend on the depth of a layer?\"\n\n\nWe add the following paragraph to the conclusions:\n\n> We emphasize that our motivation for investigating this (i.e., tunnel) phenomenon is to build a coherent picture encompassing our experiments and evidence in the literature. Specifically, we aim to understand better how the neural networks handle the representation-building process in the context of depth.\n\n**In the introduction, the authors claim to challenge the idea that deeper layer capture more complex and task-specific features. What are the results that question this idea?**\n\n\nWe recall the following facts from our study\n\n1. Experiments show that the tunnel layers, with decreased out-of-distribution (OOD) performance, have minimal contribution to in-domain (ID) performance (Figure 1, 2, 7). [roughly speaking the representations 'do not change' in the tunnel as it comes to the performance]\n2. The tunnel layers' representations are identical (measured via CKA), indicating a lack of meaningful task-specific features (Figure 4) [roughly speking, the representations do not change in the tunnel when measure with CKA]\n3. Experiments demonstrate that the tunnel remains unchanged when optimizing the network for a new task, allowing for transferability across tasks without retraining (Table 4). [i.e. the tunnel is task-agnostic]\n\nPerhaps the current phrasing is somewhat misleading. Would you find the following better:\n> The deeper layers (i.e. the tunnel) concentrates on compressing the representations instead of building more complex ones.\n\n\n\n**The role of skip connection would be very relevant and deserve to be discussed more in the main text.**\n\nWe agree that skip-connections seem to play a very relevant role in this phenomenon. In the updated version of the paper, we included the study about the role of skip connections in the main text. \n\n\n\n**The early emergence of the tunnel**\n\nThe tunnel effect develops gradually, making it difficult to precisely identify the specific step or epoch when it becomes noticeable from our standpoint. However, regarding rank, we can observe a sudden decrease within the initial training steps, as depicted in Figure 6. To gain further insights, we conducted additional experiments using linear probes attached to the network at various training stages. The results in Figure 4 of the attached PDF indicate that the accuracy saturates in deeper layers as early as the 10th epoch despite an overall gradual improvement in accuracy in later epochs. These findings suggest tunnel formation occurs within 5% of the training process.\n\n\n\n**I find the results of Figure 9 exemplificative of the fact that the network uses always the first layers to solve the task (...). It would be nice to have the same Figure for a CNN.**\n\nWe provide the analogous Figure for the VGG-based networks of various depths -- see Figure 2 in the attached PDF file.\n\n\n**Minor: What is the difference between blue and orange dots in Figure 10 (top).**\n\nIn this scenario each network (of various depth) was trained on a sequence of two tasks. The blue and orange colors depict the performance of the models after training on the first task and second task respectively. We'll update the Figure to include this information in the legend and caption. \n\n\n\n\nIf your concerns have been sufficiently addressed in our responses, we humbly seek your support for the paper and ask you to consider improving your score. Should you have any further concerns or additional points to raise, we are eager to address them."}]

Author rebuttal:
## General Response
Dear reviewers,

Many thanks for providing valuable feedback in your reviews, both positive and negative. We are delighted to note that all the reviewers prized the scale of our experimentation and found our results very interesting (xLcw, dmw5), with xLcw reporting that they are 'well-designed' and EeeK agreeing on their thoroughness and organization. Moreover, dmw5, Eeek, and Kpac indicated that the presented analysis could have implications for downstream tasks like OOD or continual learning. The reviewers expressed interest in further pursuing this line of research in other modalities (Kpac).

### Scope and novelty
On the negative side, reviewers dmw5 and Kpac raised concerns regarding the novelty of our findings and, consequently, the contribution of our work. We consider the primary value of our work to be *conceptualization and providing a coherent perspective of how representations are formed*. In particular, we do not mean to claim that all observations are new. We instead attempt to build a standalone empirical evidence to support our tunnel hypothesis. Moreover, we acknowledge that we missed discussing more thoroughly [1,2,3], which will be added. 


To be clear, we apply the following amendments to the paper:
- we add the papers (see above) to the related work section along with the discussion
- we update the motivation and conclusion paragraphs (see below)
- we change the first point of the contribution to "We *conceptualize* and extensively examine the tunnel effect, namely, deep networks naturally split into the extractor responsible for building."


**The revised paragraph in the introduction now reads as follows:**

> Extensive research has ..... to simplify representations with increasing depth [38,7]. This phenomenon arises because, despite their large capacity, these networks strive to compress hidden representations by focusing on discriminative patterns during supervised training [38, 7, 8, 30]. Motivated by these findings, we aim to investigate this phenomenon further and build a comprehensive picture to answer the following research question: 
>
> "How do representations depend on the depth of a layer?"


**We add the following paragraph to the conclusions:**

> We emphasize that our motivation for investigating this (i.e., tunnel) phenomenon is to build a coherent picture encompassing our experiments and evidence in the literature. Specifically, we aim to understand better how the neural networks handle the representation-building process in the context of depth.

[1] Ansuini et. al. Intrinsic dimension of data representations in deep neural networks. 

[2] Li et. al. Principled and Efficient Transfer Learning of Deep Models via Neural Collapse.

[3] Feature learning in deep classifiers through Intermediate Neural Collapse, Rangamani, Akshay; Lindegaard, Marius; Galanti, Tomer; Poggio, Tomaso.



Meta-review: The paper received mixed recommendations after the rebuttal. The majority of reviewers (3 out of 4) find the paper has value in conceptualizing and experimentally validating the "tunnel effect". On the other hand, reviewer dmw5 maintains their rejection rating, concerning novelty and the additional value given certain evidences from related research works. The AC checked all the materials and sides with the majority vote, especially given the OOD study (which dmw5 also finds interesting) and the continued effort that improves the positioning and presentation of the paper. Please incorporate necessary changes in the final version.
Decision: Accept (poster)