Auto-Compressing Networks
Authors: Vaggelis Dorovatas
Georgios Paraskevopoulos
Alexandros Potamianos
Venue: NeurIPS 2025 oral ()
Keywords: deep learning architectures
representation learning
residual connections
continual learning
Primary area: deep_learning

Abstract: Deep neural networks with short residual connections have demonstrated remarkable success across domains, but increasing depth often introduces computational redundancy without corresponding improvements in representation quality. We introduce Auto-Compressing Networks (ACNs), an architectural variant where additive long feedforward connections from each layer to the output replace traditional short residual connections. By analyzing the distinct dynamics induced by this modification, we reveal a unique property we coin as *auto-compression*—the ability of a network to organically compress information during training with gradient descent, through architectural design alone. Through auto-compression, information is dynamically "pushed" into early layers during training, enhancing their representational quality and revealing potential redundancy in deeper ones. We theoretically show that this property emerges from layer-wise training patterns found only in ACNs, where layers are dynamically utilized during training based on task requirements. We also find that ACNs exhibit enhanced noise robustness compared to residual networks, superior performance in low-data settings, improved transfer learning capabilities, and  mitigate catastrophic forgetting suggesting that they learn representations that generalize better despite using fewer parameters. Our results demonstrate up to 18\% reduction in catastrophic forgetting and 30-80\% architectural compression while maintaining accuracy across vision transformers, MLP-mixers, and BERT architectures. These findings establish ACNs as a practical approach to developing efficient neural architectures that automatically adapt their computational footprint to task complexity, while learning robust representations suitable for noisy real-world tasks and continual learning scenarios.

Auto-Compressing Networks
Vaggelis Dorovatas" 1.2
Georgios Paraskevopoulos . .pprraskerpooloos@eeeeerrr..
vdorovatasChotmail.cc
Alexandros Potamianos 1.2
potam@central .ntua.ua. gr
National Technical University of Athens 2 Archimedes RU, Athena RC Institute of Language and Speech Processing, Athena RC
Abstract
Deep neural networks with short residual connections have demonstrated remark- able success across domains, but increasing depth often introduces computational redundancy without corresponding improvements in representation quality. We introduce Cooto-Conrrssiinggg Networks (ACNs), an architectural variant where additive long feedforward connections from each layer the output replace tradi- tional short residual connections. By analyzing the distinct dynamics induced by this modification, ability of a network to organically compress information during training with gradient descent, through architectural design alone. Through auto-compression, information is dynamically "pushed" into early layers during training, enhancing their representational quality and revealing potential redundancy in deeper ones. We theoretically show that this property emerges from layer-wise training patterns present ACNs, where layers are dynamically utilized during training based on task requirements. We also find that ACNs exhibit enhanced noise robustness com- pared residual networks, superior performance low-data settings, improved transfer learning capabilities, and mitigate catastrophic forgetting suggesting that they learn representations that generalize better despite using fewer parameters. Our results demonstrate up to 18% reduction in catastrophic forgetting and 30-80% architectural compression while maintaining accuracy across vision transformers, MLP-mixers, and BERT architectures. These findings establish ACNs as practical approach to developing efficient neural architectures that automatically adapt their computational footprint to task complexity, while learning robust representations suitable for noisy real-world tasks and continual learning scenarios.
reveal unique property coin cotto-compressionn-to the
Introduction
Deep learning has achieved significant breakthroughs across diverse tasks and domains 228 30 6 however, still lacks the flexibility, robustness, and efficiency of biological networks. Modern models rely deep architectures with billions of parameters, leading to high computational, storage, and energy costs, Architecturally, these large models are primarily characterized by short residual connections [20] design initially developed to enable robust training of deep neural networks via backpropagation by providing stable gradient flow through these exceptionally deep networks. These skip connections establish network topology where multiple information pathways are created (53], resulting in an ensemble-lie behavior that delivers more efficient training and superior generalization compared to traditional feedforward networks.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Historically, since the emergence of Highway Networks 477, which first proposed additive skip con- nections researchers have explored numerous architectural variations. Residual Networks (ResNets) 20 removed learned gating functions and adopted direct identity skip connections becoming the industry standard. DenseNets 22 utilized feature concatenation instead of addition, while Fractal- Nets 29 introduced recursive tree-like architecture combining subnetworks of multiple depths to further enrich feature fusion. More recent works include learned weighted averaging across layer outputs [37, application of attention mechanisms across block outputs [14] and denser connectivity patterns between network nodes [63] Other works have explored adding scalars to either the residual or block stream to improve performance, training stability and representation learning 443 3 62 16].
In neural machine translation, researchers have drawn inspiration from both vision and language do- mains to combine information from different layers, enabling richer semantic and spatial propagation throughout the network
59
Connectivity
Forward Propugation
(Gradient) Propagation
FEN
yF 11 30.200
dye 3at
II
We
II Wm
Zo
backward tera
jinward
ResNet
UR 11
II baskharrd
II m=
(1 mn
funward
ACN
Na (1+ I1i=
dya Dat
1 +
II Wk
II W0
Jireard
Table I: Connectivity (2D case), Forward and Backward Propagation (1D linear case) for FFN, ResNet, and ACN architectures.
While Residual Networks, as discussed, offer efficient and effective training especially in very deep architectures. there are various works that explore behaviors of these architectures that may harm generalization. In [53] 48) it is shown that these architectures exhibit a notable resilience to layer dropping and permutation, which could indicate potential redundancy of some layers (i.e. removing a layer does not affect the network). In [23]. it was further observed that dropping subsets of layers during training can reduce overfitting and improve generalization. In a related study [], the authors showed that introducing skip connections between layers can lead to parts of the network being effectively bypassed and trdiee. In 662, they show that unscaled residual connections can harm generalization capabilities in generative representation learning. Finally, more recently, research has revealed substantial parameter redundancy and inefficient parameter usage large-scale foundation models, particularly within their deeper layers [88 (7]. All these observations can be unified under the perspective that, although residual architectures facilitate training via multiple signal pathways, these same pathways to be either underutilized or prone to sometimes act shorteuts that cause certain components uuliimttlll limiting effective generalization.
Inspired by these observations, in this work we explore whether we can design alternttie architectures that preserve the key advantages Residual Networks surc as multiple signal pathways and efficient gradient flow-while addressing issues like redundancy and shortcut overuse, ultimately enabling more efficient parameter utilization and improved representation learning. To this end, we propose an architectural variant where additíve long feedforward connections from each layer to the output replace traditional short residual connections as shown in Table introducing Compressin Networks (ACNs). ACNs showcase unique property coin as auto-compression the ability

---

of a network to organically compress information during training with gradient descent, through architectural design alone, dynamically pushing information bottom layers, enhancing their repre- sentational quality, and naturally revealing redundant in deeper layers. We theoretically investigate the emergence of this property by analyzing the gradient dynamics of networks with different connec- tivity patterns. As illustrated in Figure ACNs demonstrate layer-wise training patterns in which early layers receive significantly stronger gradients during the initial stages of training, in contrast to the more uniform gradient distribution observed in Residual Networks. Next, empirically demonstrate a broad range of advantages that ACN-learned representations offer compared to residual feedforward architectures, including: enhanced information compression, superior generalization, reduced catastrophic forgetting, and efficient transferabilityy Our contributions can be summarized as:
We introduce Juto-Compressing Networks (ACNs), an architecture that organically com- presses information into a subset of the full network's 's layers, through architectural design alone.
We provide a detailed analysis of the gradient dynamics of ACNs, along with residual and feedforward networks, shedding light on their distinct behaviors and arguing that different connectivity patterns result in unique training regimes that drive distinct learned representations.
We implement ACNs in fully connected and ceaeffommer-based architectures, finding that they match or outperform residual baselines, while 80% of top layers effectively become redundant as information concentrates in the lower layers. Notably, ACNs are hardware- friendly and require no specialized software.
We show that ACNs learn representations that better in low-data regimes compared to residual architectures.
robust against noise and generalize
ACNs reduce catastrophic forgetting by up to 18% compared to residual networks in continual learning by preserving capacity for new unseen tasks.
ACNs outperform regularization-bas approaches at transfer learning without requiring hyperparameter tuning.
Auto-Compressing Networks
In ACNs residual short connections are replaced with long feedforward connections, as described Eq. for network of depth L, and shown in Table
24 = f((+-11,,
y
In ACNs the output of each layer directly connected to the output of the network, and thus is directly optimized by the objective function during gradient descent training. Furthermore, the number of possible shortcuts is equal to the number layers, L. We find this simplification maintains the improved signal flow that shortcut connections provide, while also introducing the ability detect potential parameter redundancy
1 the architecturee) We note that ACNs differ structurally from other models employing long connections, such as DenseNets 22 and DenseFormer [37] which residual networks variants, as previously discussed. These models connect each layer to all preceding layers whereas ACNs connect each layer only to the output, leading An overview of these models along with an extended literature review is provided in Appendix A For completeness, we also provide direct, fair comparison against other residual variants, such as DenseFormers and DenseNets, in Appendix
distinct structural design.
!Code for the paper is available hcre
"The connections shown for ResNet (red) are implicit, resulting from the residual summation. A detailed example is provided in Appendix [D]
Note that long connections strict subset of the 2 shortcut connections in residual networks.

---

2.1 Gradient Propagation Across Network Architectures
In this section, we examine and compare the forward and backward pass (gradient flow) dynamics of three architectures: traditional feedforward networks (FFN), residual networks (ResNet), and the proposed cuto-comprssiing networks (ACN), based on the equations of [ See Appendix a detailed derivation of the gradient equations for ID linear neural networks.
Normalized d Gradient Norm across layers during training
Incremental Layer Performance Contribution during training
Residual Networks
Auto Compressing Networks
Residual Networks
Auto-Compressing Networks
31
6000
630
-610
4.5
6.10
636
10 20 40 50 Epocto
60 100
10
20
GU 8U
50
100
1b 3/o
100
150
100
300
Epocia
Figure (left) ACNs vs Residual Networks gradient flow across layers during training for MLP- Mixer architecture [50] on CIFAR- 10 [27]. showcasing implicit layer-wise training and information concentration on the bottom layers for ACNs. (right) ACNs vs Residual Networks incremental performance contribution across layers for ViT architecture [T0] on ImageNet- K, revealing auto- compression by gradual layer-wise wise training in ACNs.
2.2 Emergent Gradient Paths
Ov. in the The forward and backward components: As shown in Table each gradient w; (see Dw, 3rd column of the respective table) decomposes into forward and backward terms. The forward term determines gradient and forward propagation stability (whether the signal vanishes or explodes), while the backward term influences learning. For backward paths, ID FFNS contain a single path, while ID ACNs have L, - i + 1: one direct path using the layer's own long connection to the output, plus L - additional paths where gradient flows from each subsequent layer's long connection and back through the network. ID ResNets have 2 paths since at each layer there are two options: flow through the network or follow the residual connection. It is also worth observing that ACNs feature a forward term identical to FFNs for intermediate layers (single path), while their backward components is closer to ResNets since consists of multiple paths.
The Backward component (Full Gradient - FG) can be further decomposed into Network-meeiated Gradient (NG) component that is scaled by network weights and backpropagates information through (a subset of) the network and Direct Gradient (DG) component that directly connects from the output to each layer, shown as the term "1" in the backpropagation equations of ACNs and ResNets in Table This direct path acts as an information super highway, especially early in training where weights are typically initialized close to zero, informing each layer directly how to contribute towards lowering the optimization objective. Finally, the DG contribution is more significant for ACNs compared to ResNets, due to ACNs' linear (rather than exponential) total gradient path count.
Unlike the symmetric forward and backward terms of ResNets and FFNs, ACNs gradients, as argued, consist of single forward path and multiple backward paths. This design creates an implicit layer- wise training dynamic, where deeper layers are trained at a slower rate compared to earlier layers, since they have weaker forward component (assuming close-to-zers initialization) and a smaller number of backward paths. Further, when compared to ResNets, ACNs have stronger contribution during backpropagations from the DG path (vs. NG) and this effect becomes more pronounced for deeper networks and for the early layers. For example, when training the second layer of a ID T. = 12 layer network, DG is one of 11 ACN backward paths, while for ResNets the DG is competing with another 127 paths (of the NG term). This further accelerates training of the early layers.
Thus, we postulate that: 1) a strong DG component coupled with a weaker feed-forward signal leads implicitly to efficient layer-wise training, and 2) architecturallyinddee induced layer-wise training results
'This backward path has been previously explored as an alternative to traditional backpropagation and is typically referred to Direct Feedback Alignment (DFA) in the literature [35 40).

---

inadvertently in a form of structural learning where information is naturally pushed to early layers, i.., later layers will become redundant (effectively identity mappings) if the earlier layers can already solve for the task. We refer to this new class of networks as auto-compressors since they naturally "shed" their redundant layers during backpropagation simply via architectural design. These claims are experimentally validated in the rest of the paper.
ACNs in Practice: Information Compression and Gradient Flow
Next, we implement auto-compressing networks top of state-of-the-ar neural architectures across diverse tasks and datasets. We implement ACNs using variants of the Transformer [52] for language and vision tasks and MLP-Mixer [50] for vision tasks. This allows evaluate our approach
diverse benchmarks including image classification (CIFAR- 10 [27]. ImageNet- IK 41 sentiment
analysis, and language understanding (BERT 9 GLUE (54
In ACNs, for each input token, the final output vector y² the sum all intermediate layer represen- tations plus the input embedding (Eq.I). For classification, we apply pooling for images the [CLS] token for text. For a network of depth L, predictions using intermediate layers compute y (sum of representations up layer k) for each token, passed single shared classification head. This yields L+1 sub-networks, from input-only to nell-network. For residual network baselines, at depth k we take the output V of layer k as our i + subnetwork's 's output.
Our experiments begin by empirically validating the main claim established in the previous section, i.e., the presence of strong DG component coupled with implicit layer-wise training dynamics drives auto-compression, a property that resembles form structural (layer-wise) learning.
Direct Gradient (DG) to total gradient ratio during training
Residual Networks
Auto-Compressing Networks,
Accurees
0.4
ACN
Residual Network
AGN only DG
1914
10
40 60 Epochs
80
100
20
80 Epociaa
00
Prediction Layer
Figure 2: Results for the CIFAR-10 10 task using the MLM-Mixer base architecture: (left) ACN variants achieve auto-compression needing only few layers to achieve good performance. (right) The ratio of direct gradient DG to the total gradient FG significantly higher in early layers for ACNs.
To this end, We train feedfoward (FFN), residual and auto- compressing variants incorporated the MLP-Mixer architec- ture CIFAR- 10 dataset for 100 epochs. To emphasize the role of the DG gradient nuto-compression, we also train an ACN variant receiving gradients only from the long connections (ACN only DG component). In Figure leett, we show clas- sification accuracy plotted against network depth (layer probing) and observe that among ACNs, FFNs, and Residual Networks, only ACNs exhibit auto-compression. Moreover, ACNs uti- lizing only the direct gradient (DG) still achieve significant auto-compression, highlighting the importance of strong DG component to achieve this behavior and explaining why FFNs do not exhibit auto-compression, term (Equation 4. In the case of Residual Networks, we pre- viously argued that the exponential number of gradient paths substantially diminishes the influence of the direct gradient (DG)
Best Best Annueny
Prodiction Layer
Figure 3: Performance of inter- mediate layers of AC Resid- ual Vision Transformers trained on Imagenet-1K.
they lack a direct gradient
ACNs with only the DG component under-pertonn perform, underpinning the importance of the NG component for maximizing performance.
5

---

on the overall gradient, a component crucial for auto compressrn.s.... To further illustrate this, Figure rigght) presents the ratio of DG the full gradient FG across layers during training for both AC and Residual variants. The results indicate a significantly higher DG to FG ratio in ACNs, confirming the increased contribution of direct gradients in the early layers of coo-oompessiing architectures compared to residual networks and explaining the auto-compression property. Furthermore, from Figure [ we observe that ACNs demonstrate a concentrated gradient pattern with stronger signals in early layers and stronger patterns of layer-wise learning. Residual Networks exhibit more "uniform layer learning" " pattern, whereas deeper layers show increasing gradient contribution in later epochs, suggesting task-specific adaptation training progresses. Interestingly, the pattern observed in Residual Networks indicates that high gradient norms are primarily concentrated in the early and deep layers, while middle layers receive significantly lower gradients, suggesting potential redundancy.
4
ACNs compress more
4.1 Auto-Compressing Vision Transformers and MLP-Mixers
Next, we evaluate ACNs in the context of transformer architectures by implementing auto- compressing variant of Vision Transformer (ViT) [10). We train a Vision Transformer (ViT) with long connections (AC-ViT) from scratch on the ILSVRC-2012 ImageNet-IK, following the training setup in the original paper. For both models we use 256 batch size due to memory constraints. AC-ViT converges at 700 epochs, while the Residual ViT converges at 300 epochs. As shown in Fig. 3 AC-ViT reaches top performance at only 6 layers while the vanilla ViT needs all 12 layers to reach similar performance. effectively suggesting that ACNs can improve inference time and memory consumption without sacrificing performance. To gain more intuition about the training dynamics and task learning of the two variants, in Figure Tright) we plot the incremental layer performance contribution (difference in accuracy of subnetwork ++ 1 to subnetwork i) to track the behavior of intermediate layers throughout training. The key observation that ACNs are trained in a layer-wise fashion, while the residual variant performs task-learning in the last 2-3 layers, effectively utilizing the full network to achieve top performance.
The effect of Task Difficulty Intuitively, overparameter- ized networks trained on easier tasks should demonstrate higher levels of redundancy. Therefore, ACNs should con- verge to utilizing fewer layers as task difficulty decreases. To verify this, we the number of classes as proxy for task difficulty for image classification on the CIFAR-10
Solid: AC.M Dushed: Ras Mixer
0.8
Accuracy
0.6
dataset (27). Specifically,, we create subsets of 2, 5, and 10 classes, the assumption being that binary classification should be easier than 10-class classification. For this exper- iment we utilize MLP-Mixer (50) and train two variants, the original MLP-Mixx with residual connections and the modified MLP-Mixer with long connections (AC-Mixer), Results are presented in Fig 4 We observe that indeed AC- Mixer converges to solutions with larger effective depth, as the task "difficulty" increases. Specifically, in this ex- periment, ACN needs 8, 10 and 12 layers for the 2, 5 and 10-class classification problem, respectively. In contrast, the Residual Mixer converges to solutions where the full depth of the network is utilized, irrespective the task difficulty
0.4
C- 10/13
5-10
21000
0.2
C 10/19
G 510
2-1
10
14
Prediction Layer
Figure 4: Performance of the intermedi-
ate layers as the number of classes (and examples) in the CIFAR-10 dataset in- creases from 2, to 5 to 10 classes: (a) Residual Mixer vs (b) AC-mixer (C de- notes the number of classes in the sub- set).
4.2 Auto-Compressing Encoder Architectures for Language Modeling
In this section, we conduct preliminary study on the effectiveness of the ACN architecture in general pre-training (masked language modeling with a BERT architecture) followed by fine-tuning. The
'The Residual Mixer was trained for 300 epochs, while AC-Mixer for 420 epochs to reach the performance of its residual counterpart.

---

results show that ACNs learn compact representations that: 1) achieve ma-par performance with the residual architecture on transfer learning tasks, while utiliziig significantly fewer parameters, and 2) complement post-training pruning techniques, enhancing their effectiveness.
4.2.1 Masked Language Modeling and Transfer Learning with ACNs
We compare the ACN and residual architectures in the standard BERT pre-training and fine-tuning paradigm. Using the original BERT pretraining corpus CoooksCorpus [64] and English Wikipedia), we train both architectures to equivalent loss values; the AC-BERT variant requires two epochs vs one epoch for the residual baseline. Following pre-training, we fine-tune both models on three GLUE benchmark datasets [54]: SST-2 sentiment analysis [45], QQP paraphrasing, and QNLI question answering (38]. Figure left) demonstrates key advantage of the ACN architecture: it naturally converges to using significantly fewer layers (approximately 75% less layers) while maintaining performance comparable to the full residual network. These results suggest promising applications for ACNs in large language models, where pre-training could be performed with long connections, allowing downstream tasks adaptively utilize only the necessary subset of layers during fine-tuning. To further enhance compression, ACNs can be combined with standard pruning techniques preliminary investigation of this approach is provided Appendix G
5 ACNs generalize better
0.85
While ACNs demonstrate effective parameter reduction through architectural compression, key question re- mains: do these compressed representations offer ad- ditional benefits beyond parameter efficiency? In this section, we investigate whether the concentrated infor- mation ACNs' early layers leads to improved gener- alization capabilities compared to traditional residual ar- chitectures. Specifically, we explore robustness to input noise proxy for generalization. Moreover, pare the inherent cotpo-compessionn of through architectural design-to recent regularization- based methods that rely on externally imposed interme- diate losses.
0.80
0.70
0.65
QNLI AC BERT
QNLI BuskBual BERT
0.60
SST22 BERT
0.55
98T-22, Resithual
QQR AC-BERT
0.50
DOP, Resiical NERT
10
12
Prediction Layer
Figure 5: Downstream performance of AC- BERT vs residual BERT on three GLUE tasks.
5.1 Robustness to Input Noise
Next, we present results assessing the robustness of ACNs versus residual transformer architectures to input noise. The experiments are performed with the ViT and residual VIT architectures trained on ImageNet- K. In this experiment, we inject increasing levels of additive Gaussian noise with standard deviation a 0.1, 0.2, 0.4, and salt-and-pepper noise with percentage of altered pixels y 1%, 2%, 10%. Results (average accuracy) are shown in Table (a) for Gaussian and (b) for salt- and-pepper noise. We observe that ACNs display improved robustness to noise, and the performance gap with the residual transformer increases as the noise levels increase. These results align with the findings of (58). who showed that architectures with forward passes closer to feedforward networks (like our ACNs) exhibit enhanced noise robustness. In residual architectures, short connections allow noise to propagate and accumulate throughout the network, whereas the long-connection design of ACNs helps mitigate this amplification effect.
Baseline
Gaussian Noise
Salt and Pepper Noise
Model
w/o noise
0 = 0.1
o 0.2
a= 0.4
p n= 0.01
p = 0.05
p = 0.1
Residual ViT
70.74
67.68
62.80
45.46
56.80
27.48
10.34
AC-VIT
70.76
69.50
64.54
51.89
59,80
36.35
19.98
Table 2: Robustness (average accuracy %) of ViT with long connections (AC- ViT) and with residual connections (Residual ViT) to additive Gaussian noise and calt-and-pepper noise on ImageNet-18
test set.
7

---

5.2 Robustness to Data Sparsity
Next, we experimentally compare the performance of residual and long connections architectures in low-data scenarios. For this purpose, we create random subset of CIFAR-10 [27] by retaining only 100 samples per class, resulting total of 1000 examples. Using the same training settings and models as described in Section 4.1 (MLP-Mixer on CIFAR- 10), we train both architectures for 150 epochs to assess how fast the training and test loss decrease, as a proxy for the generalization capabilities of each architecture. Results shown Fig. [ reveal that ACNs achieve lower training and test loss in fewer epochs compared to residual networks. This faster convergence in loss metrics is a strong indication that auto-comprest compressnn networks can be effectively utilized in scenarios with limited data.
2.4
Mixer: Tram Loss
Mixer
2.1
Sliner Truin Loke
Mlnner Lon
2.2
1.8
340
Epochs
Figure 6: Train and Test Loss of AC- Mixer and Residual Mixer on CIFAR- 10 (100 samples per class).
6 ACNs forget less
Continual learning involves training models on a sequence of tasks without access to past data, aiming to retain performance on previous tasks while learning new ones | 551. A central challenge in CL is catastrophic forgetting -te tendency of neural networks overwrite old knowledge when updated with new data. Common approaches include data replay methods
[3 33 and
regularization techniques that penalize changes to important parameters 26 61 2]. We've already demonstrated that ACNs, through implicit layer-wise training, dynamically allocate parameters based on task demands while preserving redundant parameters for future tasks. Conversely, Residual Networks optimized for efficient task learning risk overfitting and suboptimal parameter usage in these sequential learning settings. To test our claims, we evaluate both architectures on the split CIFAR-100 continual learning benehmark, comprising 20 sequential disjoint 5-class classification tasks, focusing on mask-incremental learning [51] where task identity is known. We utilize MLP- Mixer architectures (hyperparameters in Appendix B) and we test two continual learning algorithms trained for 10 epochs for each task: naive fine-tuning (Naive FT) and Synaptic Intelligence (SI) [6], which adds gradient-based regularizer to each parameter depending on how changes in it affect the total loss in a task over the training trajectory. Across experiments, we report Average Forgetting, defined the mean difference between a task's best performance (right after it is learned) and its final performance after all tasks are learned, and Average Accuracy, defined as the mean accuracy over all tasks at the end of training. We expect gradient-based regularization methods perform particularly well with ACNs since unused, redundant parameters receive small gradients, making their detection easier compared Residual Networks where gradients are more uniformly distributed (see gradient heatmaps, Fig. [lett). Results in Table confirm our intuition: ACNs consistently exhibit significantly less forgetting (up to 18% improvement) compared to Residual Networks. Notably, with SI, increasing ACN depth decreases forgetting an ideal behavior for CL systems where increasing network capacity reduces forgetting- while Residual Networks show the opposite pattern, indicating potential overfitting. ACNs also achieve better average accuracy across all tasks, further establishing them as a more suitable architecture for continual learning.
Avg. Accuracy (%) 1
Avg. Forgetting
Method
Arch
=
L= 10
= 15
L 5
L=
L
Naive FT
AC Mixer
32.97 d: 2.4
32.94 5.3 28.16
31.61 + 2.2
46.55 1 2.2
45.46 1: 5.8
46.91 4: 2.4
ResMixer
31.77 +: 1.8
26.14 2.3
62.76 4 2.3
54.89 + 1.6
54.49 +: 2.2
AC-Mixer
14.5 41.5 2.2
46.1 # 1.3
46.2 + 0.8
35.7 + 2.1
33.8 0..8
32 1.8
S1
ResMixer
43.47 3.1
36.1
5
32.1 + 0.8
42.4
44.6
50
2.1
Table 3: Average accuracy and forgetting across layers, methods, and architectures on the Split CIFAR- continual learning benchmark. Models are trained for 10 epochs per task, where each task consists of classifying 5 out of 100 classes presented sequentially. L. denotes the number of layers in the architecture, ACNs consistently forget less and they also do not waste capacity.
8

---

7 ACNs transfer better
Parameter redundancy, and specifically potential layer redundancy, in residual architectures is Recent works 13 have attempted to address this through regularization-bait based layer-wise compression techniques during training, specifically by adding losses to all intermediate layers of the network and using a weighted sum of them as the total loss, a technique formally introduced in [31 for improved training. Such loss- based regularization methods rely heavily on precise tuning of intermediate loss weights, creating practical challenges. If early-layer loss weights are set too high, the network risks overfitting and poor generalization; if set too low, performance improves gradually across layers with no clear cutoff point, reaching optimal results only at the final layer. This sensitivity to hyperparameter selection makes it difficult to reliably identify an optimal depth for inference using loss-based regularization. ACNs address this challenge through architectural design rather than regularization, naturally compressing information without requiring complex hyperparameter tuning.
phenomenon that has been well documented 53
To evaluate whether different layer compression ap- proaches learn generalizable representations, we con- duct transfer learning experiment from CIFAR-100 to CIFAR-10 27 using MLP-Mixer architectures (for hyperparameter choices we refer to Appendix B). This setup allows us to assess how well each model's learned representations transfer to similar task. In regularization-ban based layer compression meth- ods, explicitly training all layers to directly minimize a task loss through intermediate supervision can lead to overfitting on the base task, which can further re- sult in weaker transfer capabilities on downstream tasks. In contrast, ACNs' implicit compression mechanism naturally balances generalizability and task performance without imposing external constraints. To ensure fair comparison, we train all models to achieve comparable performance on the CIFAR- 100 pre-training task, enabling direct assessment of their transfer capabilities to CIFAR- 10. The results in Table 4 confirm our analysis: ACNs demonstrate superior performance on the downstream CIFAR-10 task compared regularization-ben CIFAR- 100 task performance is similar. This confirms our claim that the representations learned by ACNs are more generalizable and thus exhibit greater transferability.
Method
Accuracy (%)
AC-Mixer
85.38 1.0.7
Aligned
82.9 100.9
LayerSkip
79 :
Table 4: C-100 to C-10 transfer learning per- formance of ACNs vs recent regularization- based layer copression approaches.
methods, even when upstream
Conclusion
In this work, we introduced Auto-Compe Networks (ACNs), an architectural design that organically compresses information into early layers of a neural network during training via long skip connections from each layer to the output, a property we coined as cotoo-oomreessoon Unlike residual networks, ACNs do not require explicit compression objectives regularization; instead, they leverage architectural design and gradient- based optimization to induce implicit layer-wise training dynamics that drive nuto-compression.
Our theoretical and empirical analyses demonstrate that ACNs alter gradient flow, imposing implicit layer-wise training dynamics and resulting in distinct representations compared to feedforward and residual architectures. In practice, this leads to 30-80% of upper layers becoming effectively redun- dant, enabling faster inference and reduced memory usage without sacrificing accuracy. Experiments across diverse modalities (vision, language) and architectures (ViTs, Mixers, BERT) further show that ACNs match outperform residual baselines, while offering greater robustness to noise and low-data regimes, excelling in transfer learning, and reducing catastrophic forgetting by up to 18%-all all without specialized tuning, overall suggesting that they learn better representations despite using fewer parameters. A summary of the main results is shown in T Another practical advantage of the comprssompessioon property of ACNs is its potential to enhance complement other compression and efficient inference techniques. Preliminary results demonstrating this synergy are presented in Appendices Cand
Concluding, ACNs highlight the potential of architectural design as implicit regularization and pave the way toward self-adapting neural networks that allocate depth and capacity dynamically. Future

---

work may extend ACNs to self-supervised, multi-task, and generative settings, as well as explore per-sample adaptive inference and other short long architectural variants. Some initial
Limitations and Broader Impact
Due to resource constraints, our evaluation of ACNs was limited to relatively small-scale tasks, though the architecture consistently performed well across modalities, datasets, and statc-of-the-an baselines. Broader language or multimodal models) essential to fully understand its capabilities and boundaries. notable limitation is the increased training time compared to residual architectures. While this may contribute to stronger representations, optimizing training efficiency remains open challenge, warranting further exploration of scheduling and initialization strategies.
large-scale, self-supervised, and multi-task settings (e.g..
By enabling implicit layer pruning, ACNs aim to reduce inference-time resource use, supporting more sustainable AI. Long-term, this could inspire adaptive architectures with System System behavior (25). dynamically adjusting depth based on task complexity. However, like all efficiency advances, broader deployment may introduce ethical concerns such as misuse in low-regulation environments or rrisiiv-ensitive applications applications-highliill highlighting the need for responsible development and oversight.
10 Acknowledgments and funding disclosure
This work has been partially supported by project MIS-5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the EU under the NextGenerationEU Program. We acknowledge EuroHPC JU project ID CINECA, Italy.
for use of the supercomputer LEONARDO@
G. Paraskevopoulos was funded by the European Union's Horizon Europe research and innovation programme under the AIXPERT project (Grant Agreement No. 101214389).
References
[1] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 2016.
[2] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139-154, 2018
[3] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pages 1352-1361. PMLR, 2021.
[4] Yu Bai, J. Zico Kolter, and Vladlen Koltun. Understanding the low-rank bias of deep neural networks. In International Conference on Learning Representations, 2021.
[5] David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? International conference on machine learning, pages 342-350. PMLR, 2017.
[6] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
[7] Róbert Csordás, Christopher D Manning, and Christopher Potts. Do language models use their depth efficiently? arXiv preprint arXiv:2505.1388 13898, 2025.
10

---

[8] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEFE transactions on pattern analysis and machine intelligence, 44(7):3366- 3385, 2021.
[9] Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguisties: human language lechnologies, volume I (long and shori papers), pages 4171-41888 2019.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[11] Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang. Exploiting deep representations for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4262, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
[12] Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. Layerskip: Enabling early exit inference and self self- speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page 36622-12622 Association for Computational Linguistics, 2024.
[13] Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speculative decoding. arXiv preprint 24ii::2404.16110, 2024.
[14] Muhammad EINokrashy, Badr AlKhamissi, and Mona Diab. Depth wise attention (dwatt): layer fusion method for data-efficient classification. arXiv preprint 2rXiv:2209.5516; 2022.
[15] Scott Fahlman and Christian Lebiere. The cascade-correlation learning architecture. Advances in neural information processing systems, 2, 1989.
[16] Kirsten Fischer, David Dahmen, and Moritz Helia.. Field theory for optimal signal propagation in resnets. arXiv preprint arXiv:2305.07715, 2023.
[17] Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint drXiv:1603.08983, 2016.
[18] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Dan Roberts. The unreasonable ineffectiveness of the deeper layers. In The Thirteenth International Conference on Learning Representations, 2025.
[19] Suriya Gunasekar, Jason Lee, Tong Zhang, and Behnam Neyshabur. Implicit bias of gradient descent on wide deep networks. In Advances in Neural Information Processing Systems, 2018.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q. Wein- berger. Multi-scale dense networks for resource efficient image classification. In International Conference on Learning Representations, 2017.
[22] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[23] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 646-661. 1. Springer, 2016.
11

---

[24] Jiachen Jiang, Jinxin Zhou, and Zhihui Zhu. Tracing representation progression: Analyzing and enhancing layer-wiss wise similarity. In The Thirteenth International Conference on Learning Representations, 2025.
[25] Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, 2011.
[26] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Baabska-Barrinkka, et al. Overcoming catastrophic forgetting in neural networks. Peevorrs..rreeeeeee Proceedings of the national academy of sciences, 1 11((1)):521-3356 2017.
[27] Alex Krizhevsky. Learning multiple layers of features from tiny images, pages 32-33, 2009.
[28] Alex Krizhevsky, lly Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[29] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In International Conference on Learning Representations, 2017.
2017.
[30] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444 2015.
[31] Chen- Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. In Artificial intelligence and statistics, pages 562-570. Pmlr, 2015.
[32] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[33] David Lopez-Pat Papez-Pazzeee and Marc' Aurelio Ranzato. Gradient episodic memory for continual lcarning. Advances in neural information processing systems, 30, 2017.
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[35] Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advances in neurul information processing
processsinnnnn systems, 29, 2016.
[36] Emin Orhan and Xaq Pitkow. Skip connections eliminate singularities. In International Conference on Learning Representations, 2018
[37] Matteo Pagliardini, Amirkeivan Mohtashami, François Fleuret, and Martin Jaggi. Denseformer: Enhancing information flow in transformers via depth weighted averaging. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[38] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pages 2383-222 2392, Austin, Texas, November 2016. Association for Computational Linguistics.
[39] Sylvestre-Alvise Rebulfi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[40] Maria Refinetti, Stéphane d' Ascoli, Ruben Ohana, and Sebastian Goldt. Align, then memorise: the dynamics of learning with feedback alignment. In International Conference on Machine Learning, pages 8925 8935. PMLR, 2021.
[41] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
[42] Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in neural information processing systems, 33:20378-2000 203899, 2020.
12

---

[43] Pedro HP Savarese, Leonardo O Mazza, and Daniel R Figueiredo. Learning identity mappings with residual gates. arXiv preprint arXiv: 1611.0120 2016.
[44] Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling, 2022.
[45] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors, Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA, October 2013. A Association for Computational Linguistics.
[46] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 5(56):192991999 2014.
[47] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint 1rXXiv:505.00335 2015.
[48] Qi Sun, Mare Pickett, Aakash Kumar Nain, and Llion Jones. Transformer layers as painters. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25219-25227, 2025.
[49] Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference pattern recognition (ICPR), pages 2464-2469. IEBE, 2016.
[50] Ily O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, al. Mlp- mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:24261-24272, 2021.
[51] Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXfv:190.00773, 2019.
[52] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[53] Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016.
[54] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi, editors. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics.
[55] Liyun Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[56) Xin Wang, Fisher Yu, 7i-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In The European Conference on Computer Vision (ECCV), September 2018
[57] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[58] Zonghan Yang, Yang Liu, Chenglong Bao, and Zuoqiang Shi. Interpolation between residual and non-residual networks. In International Conference on Machine Learning, pages 10736-10745. PMLR, 2020.
13

---

[59] Fisher Yu, Dequan Wang, Evan Shelhamer. and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2403- 2412, 2018.
[60] Alireza Zaeemzadeh, Nazanin Rahnavard, and Mubarak Shah. Norm-preservation: Why residual networks can become extremely deep? IEEE transactions on pattern analysis and machine intelligence, 43(1 ((111)):390-3999 3990, 2020.
[61] Friedemann Zenke, Ben Poole, and Surya Ganguli.. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987-3995, PMLR, 2017.
[62] Xiao Zhang, Ruoxi Jiang, William Gao, Rebecca Willett, and Michael Maire. Residual connec- tions harm generative representation learning. arXiv preprint arXiv:2404. 10947, 2024.
[63] Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, and Xun Zhou. dypper-conecction. In The Thirteenth International Conference on Learning Representations, 2025.
[64] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27, 2015.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main scope/contribution of the paper that is stated in the abstract and intro- duction is the introduction of an alternative to residual networks architecture, theoretically and empirically analyzed throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We provide " "Limitations" Section in the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
14

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: In our theoretical results we provide assumptions (aka 1d linear analysis of gradient equations) and analytical derivations can be found in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We clearly describe experiments and evaluation setup and further provide all experimental details in the Appendix. Moreover, we provide an anonymous link where code for the paper will be released.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may
15

---

be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide an anonymous link where code for the paper will be released. Datasets used in the paper are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
jublic/guidess httol//c/uiide//ddeeebbbisssssss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 7/nips. cc/ httcc//bbllcggguuueeeeeee wwwpc//ggueeeeeeee wewoodeSumisssionll
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
16

---

Justification: In the main paper we discuss core details regarding experiments and in the Appendix we report detailed choices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Yes]
We report average accuracy where applicable (c.g., noise experiments), and include mean metrics with standard deviation in continual learning settings where variability is significant.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Our experiments are small-scale but consistently report training epochs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
17

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics Sttn.hhttppps//arrr.....///pl//t
Question:
Answer:| Yes]
We conform to the NeurIPS Code of Ethics. We clearly describe our methods, include details to reproduce them and plan release our code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include "Broader Impact" section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer:[Yes)
Justification:
We do not release new data and train small-scale discriminative models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
18

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We check the license and cite models and datasets we use in the experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi paperssithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We train small reproducible models with public data and explain in detail our setups. We not introduce major assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: not invloved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: not involved
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are only used for grammar and formatting.
Guidelines:
The answer NA means that the method development in this research does involve LLMSs as any important, original, or non-standard components.
Please refer to our LLM policy /attps://neur// cc/ /oonferencces/2255/ for what should or should not described.
20

---

Technical Appendices and Supplementary Material
A Extended Literature Review
We present here an extended literature review across four key areas relevant to our work: 1) residual connections and their role in training stability, 2) architectural variants with longer/denser residual connections, 3) methods that employ intermediate layer losses to learn better representations and exit early, and 4) neural architectures that induce regularization and better representations. Our work is more closely related to research area 4, but it is important to note that training stability, efficiency, performance, and representation learning are related goals, which can be achieved either through architectural choices (1. 2, 4) and / or through additional optimization criteria (3).
Residual Connections and Training Stability: Training deep neural networks with gradient descent becomes increasingly difficult as network depth increases. Multipath network architectures date back the 1980s, with early work exploring cascade structures in fully connected networks trained layer by layer to improve training stability [5]. Highway networks [47] introduced gated bypass paths that allowed for effective training of networks with hundreds of layers. (20] found that deeper convolutional neural networks (CNNs) not only suffer from a decrease in generalization performance, often due overfitting, but also experience decrease in training performance. To address this, they introduced residual connections (or identity mappings), proposing that learning residual functions relative to identity mappings simplifies optimization. These skip connections improve the training process and often improve performance (15). [3). [60], (32). (53) further argued that a residual network with n layers can be viewed as collection of 2n paths of varying length. At each layer, the signal either skips the layer or passes through it, creating 2" possible paths. Despite sharing weights, these paths function as an ensemble of networks, as confirmed by experiments. In contrast, a traditional deep feedforward network has only one path, so removing any random layer significantly degrades performance. Additionally, the authors showed that these paths are typically shallow, with backward gradients often vanishing after passing through only small fraction of the total layers.
Following the success of residual networks various architectural modifications were proposed to improve efficiency and performance. In DenseNets [22]. each layer is connected to all subsequent layers enabling for more efficient feature reuse; fusion achieved through concatena- tion rather than addition. More recently, DenseFormer (37) introduced learned weighted averaging across layer outputs, while Depth-Wis Wise Attention applies attention mechanisms across block outputs.
Residual Variants:
Intermediate Supervision and Early Exit: In deeply supervised nets [11. complementary objec- tives are added to all intermediate layers to encourage hidden layers to learn more discriminative representations. In this approach, each intermediate objective is loss function that captures the classification error of an SVM trained on the output features of layer i. The overall loss is the
sum of the intermediate and final objectives. This idea evolved in several directions: Graves [17] proposed adaptive computation time, while more recent work like MSDNet [2I] and CALM [44] introduced dedicated prediction heads. Other approaches employ trainable routing mechanisms
[5|57] 57 to determine layer usage. Concurrent to our work, LayerSkip [2] proposes an architecture similar to ACNs, focusing primarily on inference acceleration through layer dropout and early exit mechanisms. Additionally, (24] also incorporates intermediate losses with a common head and a linearly increasing weight curriculum, justifying it through the lens of representational similarity between intermediate layers. While these approaches rely on explicit auxiliary objectives or dedicated components, our work achieves similar benefits through architectural design alone, enabling natural depth determination through gradient-based optimization.
Frchitecturally-induee regularization and representation learning: Stochastic regularization methods like Dropout [46] and its variants demonstrated that randomly dropping connections during training can lead to more robust feature learning. This insight was extended to other structural approaches like Stochastic Depth 23 where randomly dropping entire layers improved generalization. Residual connections initially proposed address the vanishing gradient problem [20] have been shown to contribute to smoother loss landscapes and improved generalization (3). These findings align with theoretical work showing that architectural choices impose implicit biases that influence the solutions found during training [19] Recent work on transformers shows that architectural choices like attention patterns and layer normalization can also induce implicit regularization effects, e.g., the combination of skip connections and layer normalization can bias the model toward low-rank solutions
21

---

4 The proposed long connection approach builds on these insights, using architectural design naturally encourage the learning robust representation while enabling automatic information compression allowing for early exit.
B Experimental Details and Setup
CIFAR-10 MLP Mixer: The MLP Mixers have 16 layers with hidden size of 128. The patch size is 4 (the input is 32x32, 3 channels). The MLP dimension De is 512, while Ds is 64. We are using the A optimizer (34 with maximum learning rate of 0.001 and Cosine Scheduler with Warmup. The batch size is 64.
For Magnitude pruning, we consider the setting where the pruning happens after fine-tuning on the downstream task. For Movement pruning, we follow a gradual fine-tune and prune curriculum, where in setting (I): 20% of the parameters are pruned after each epoch, whereas in setting (II): we prune 40% of the parameters after an epoch.
BERT post-training pruning:
Continual Learning Experiments: We using the same
setup with the CIFAR- 10 experiment (see above). We train for 10 epochs in each task, using AdamW with learning rate of 0.001 and batch size of 64. For Synaptic Intelligence
coefficient X 1.
Transfer Learning experiment Hyperparameter Choices: our main paper experiment we compare: 1) our proposed AC-Mixer, 2) a Residual Mixer with the setup of [24] (Aligned) and 3) a Residual Mixer with the setup of [13] (LayerSkip), with the rotational early exit curriculum with
Pmoz 0.1, escale 0.2.
C Gradient Propagation equations derivation
Notation: x, the output of layer i, wy is the weight of layer (the weight used to construct æ;),, To is the input (after potential initial embedding operation) and yr, VR. MA is the output for each architecture.
FFN forward pass
(r x2)
II ==1
WFFo
(2)
FFN backward pass for weight
Oyr
Oyr da:
(3)
Ow;
Or: Ow;
OyF
II
II Wm
(4)
Ow:
kmit1
backward term
forward term
ResNet forward pass
2; W;F;-11 + 2;-1 (1 wx)x;-1
(5)
yn I (1 ws)ao
(6)
ResNet backward pass for weight i
Oyn
Oyn Oz: Ør; Ow:
( (1 (()+w))) wm) 20
(1 wx)
0w;
k=i+1
22

---

L II kmi+1
Oyn Ow:
14
Wk
Σ
wkwy
Wk
II
Wmm
I
kmi+1
HISkejEL
backward term
forwaad term
equivalently:
Tyn
L-i+1
Σ sum of T ( k-tuples
i-1 II wmm) m==1
1 +
ID
(9)
Ow;
h
backward term
forward
ACN forward pass
DA 20
34
T0
wyz0
(10)
1=1
ACN backward pass for weight i
OyA dz.
Ook
1
Ri
(11)
Ow;
dr, Ow;
dr:
Ami+1
i-1 II m=1
1+
Σ
II
Wk
Wm
20
(12)
Ow;
#=1+1 Ami+1
backwardd term
forward term
D Analysis of the Connectivity across architectures
Here, we further clarify the shown connections on Table The red connections shown for ResNet connectivity, show the implicit direct connections (paths) that are formed in the architecture through the residual summation. Specifically, denote:
z To
2 1: input layer (for i 0), which is multiplied with w; to get:
s; = Wi 2 output of layer i before residual summation
= 2i-1 + æœ: output of layer i after residual summation, or equivalently, input to layer
++1
our case, we want to express the input to the final layer 22, which is:
20
21 = Ti + 20 = σι + 20
22 + 22 21 +
Thus, we see that the residual summation 2, 2 + xj effectively results in each layer receiving as input (z:) a cumulative sum of previous layer outputs (x.)). In standard feedforward networks (FFNs), each layer directly receives input only from the immediate previous layer. In contrast, in ACNs, intermediate layers behave like those in FFNs, but the final layer receives as input the outputs of all preceding layers.
23

---

E
Comparisons with Other Residual Architectures
To address concerns overfitting and ensure fair comparisons in terms of training time, we hypoth- esize that ACNs tend to train longer due to reduced number of information pathways compared residual architectures (which scale linearly vs, exponentially in terms of connectivity). However, rather than overfitting, we argue that ACNs utilize this extended training period to learn more robust and generalizable representations.
To verify this, we conducted a controlled experiment using various architectures in the MLP-Mixer setting CIFAR-10, training all models for 700 epochs ensure fair comparison. The evaluated models include:
AC-Mixer: An auto-compressing (ACN-style) Mixer,
Res-Mixer: vanilla residual Mixer,
DenseNet-Mixer: DenseNet-style variant 22 where each layer receives input the concatenation of all previous layers' outputs, projected back to the hidden dimension via learnable projection matrix. In order to have similar parameter count with the other architectures (since this architecture introduces additional parameters growing with depth), we reduce the number of layers 10 (compared 16 in the other models), and
A DenseFormer-style variant [37], where each layer receives learnable linear combination of all previous layers' outputs, with one scalar weight per previous layer (O()2) extra parameters).
MenseFormer-Minn
We report: (1) Accuracy, (2) Best Acc. Epoch the epoch at which this peak accuracy was reached; and (3) Cutoff Layer - the earliest layer whose output reaches comparable performance to the final output, measure auto-compression.
Model
Accuracy
% Network used
Best Acc. Epoch
Res-Mixer
90.3 + 0.09
100%
615
DenseNet-Mixer
90.3 + 0.10
100%
600
Denne
90.4 + 0.09
100%
637
AC-Mixer
92.0 + 0.08
75%
695
Table 5: Performance comparison across AC and residual variants on the ML.P-Mixe/CIFARR-I0 experiment. AC-Mixer converges slower but achieves the highest accuracy (improved generalization) with an earlier cutoff layer, showcasing stronger auto-compression.
The results in show that AC-Mixer converges more slowly than residual variants but leverages the extended training to learn more compact and generalizable representations. Unlike residual and densely connected variants, which reach peak performance earlier, AC-Mixer achieves higher accuracy with fewer effective layers (earlier cutoff) and without signs of overfitting but rather stronger generalization, supporting our hypothesis.
Summary of Results
Table 6 presents the main results comparing residual (Res-) and auto-compressing (AC-) variants across both vision and language tasks. We observe that ACN variants consistently achieve comparable or superior accuracy while requiring significantly fewer parameters, lower computational cost inference, and reduced storage requirements. These findings highlight the efficiency and scalability of the AC design, making it strong alternative traditional residual architectures.
G Post-Training Pruning with AC-Encoders
ACN's primary advantage lies in its inherent compression capabilities during training, suggesting that when combined with pruning techniques, it should significantly outperform traditional residual architectures. To provide validation for this hypothesis, we conducted experiments using magnitude

---

Model
Accuracy
#Params
GFLOPs
Size (MB)
Res-VIT on ImageNet
70.74 0.09
86M
33.7
330
AC-VIT on ImageNet
70.76 = 0.12
51M
19.7
195
Res-BERT on SST-2
86.63 0.09
110M
21.72
418
AC-BERT on SST-2
86.68 + 0.06
46M
5.44
174
Res-BERT QNLI
83.14 + 0.07
110M
21.72
418
AC-BERT on QNLI
83.07 0.10
46M
5.44
174
Res-BERT on QQP
87.20 0.09
110M
21.72
418
AC-BERT QQP
87.30 0.07
46M
5.44
174
Table Comparison residual (Res-) and auto-compressing (AC-) variants across both vision and language tasks.
and movement pruning [42, two commonly employed baseline pruning techniques. Results are shown when fine-tuning of SST-2 dataset sentiment analysis task. refer to Appendix for details regarding the experimental setup. Figure rright) confirms our hypothesis: ACNs consistently demonstrate superior
trade-offs compared to standard architectures, with their advantage becoming more pronounced at higher compression rates. This indicates that ACNs' architectural design naturally leads to more efficient parameter utilization, creating representations that are inherently more amenable to further pruning. While these preliminary results validate approach to addressing parameter redundancy, they also point toward promising future directions. We anticipate that combining pre-trained ACN architectures with state-of-the-ar pruning methods will result in extremely efficient, high-performing models, though rigorous validation of this hypothesis requires further investigation.
pearressioo-perffrmaaa
85
80
Accuracy
75
70
MagPrun AC-BERT
MagPrun Residual BERT
65
MovPrun (I) AC-BERT
MovPrun (I) Residual BERT
60
MovPrun (II) AC-BERT
MovPrun (II) Residual BERT
55
40
50
60
70
80 90
100
110
# #Parms (M)
Figure 7: Accuracy vs model size of AC-BERT and Residual BERT on SST-2 when pruned with Magnitude and Movement Pruning (with two different settings, refer to Appendix B for details).
H Relation to Early Exit (EE) Methods
In this section, we explore the relationship between ACNs and Early Exit (EE) methods. While both aim to improve computational efficiency, ACNs are fundamentally different in their approach. Traditional EE methods are training-time strategies that rely on explicit intermediate supervision, whereas ACNs are architectural designs that achieve similar benefits implicitly, without auxiliary
25

---

losses or supervision. Importantly, these approaches are complementary: ACNs can be combined with EE techniques further enhance both efficiency and flexibility.
Overview of EE Methods. Early Exit methods (e.g., BranchyNet 49 typically:
1. Introduce explicit losses at intermediate layers to encourage early discriminative features.
2. Train using a weighted combination intermediate and final losses.
3. Use confidence-based criteria (e.g., entropy) at inference to decide whether to exit early.
This line work, originating from Deep Supervision BT, introduces several hyperparameters- loss weights-ther that require careful tuning. This tuning is crucial for the final behavior of the network since overemphasizing early-layer losses can lead to overfitting, as early layers are pushed to achieve low training loss.
In contrast, ACNs do not use any intermediate losses or auxiliary heads. They are trained end-to- end like standard feedforward or residual networks. Nevertheless, ACNs naturally produce more informative intermediate representations due to their implicit layer-wise compression dynamics. As demonstrated in Section adding intermediate supervision to residual networks requires deli- cate balancing to avoid overfitting, while ACNs generalize more robustly Souussly-patiiulaarlyy in transfer settings-withow any such tuning. thanks their architectural regularization.
ACNs complement EE methods. Given that ACNs naturally concentrate discriminative information earlier layers, they are well-suited for integration with EE mechanisms. To validate this, we conducted experiments CIFAR-10 using MLP-Mixer variants, comparing six configurations:
Res-Mixer: Standard residual MLP-Mixer,
AC-Mixer: auto-compressing MLP-Mixer.
Res-Mixer w/ Branches: Explicit early-exit branches layers 4, 8, and 12, each with dedicated MLP classifier,
AC-Mixer w/ Branches: Same branching structure applied to the AC-Mixer,
Res-Mixer w/ Shared EE Head: All layers connect to shared classifier head, following
recent works 24 13, allowing exit decisions without additional parameters, and
AC-Mixer w/ Shared EE Head: Same thared-head mechanism applied to the AC-Mixer.
For the Branches method, use fixed weights [0.8, 0.6, 0.4 for the three intermediate branches. In contrast, the Shared EE Head employs a linearly increasing layer weighting scheme, as (24). defined by:
2(1 + 1). Wt T(T+
for 1 0, 1, 1
During inference, we compute the entropy of the logits at each potential exit point. If the entropy falls below pre-defined threshold (r = 0.7 for results below), the model exits at that layer. Inference speedup is measured in terms of FLOPs, normalized to the baseline Res-Mixer.
Model
Speedup (FLOPs)
Res-Mixer
1.0x
AC-Mixer (global cutoff at L 12)
1.5x
Res-Mixer w/ Branches
2.2x
AC-Mixer w/ Branches
2.6x
Res-Mixer w/ Shared EE Head
2.4x
AC-Mixer w/ Shared EE Head
3.3x
Table 7: Inference speedup (measured in FLOPs) of various Early Exit configurations CIFAR-10 using MiP-Mixer variants. All values are normalized to the FLOPPs of the full Res-Mixer.
The results (T confirm that ACNs substantially improve the effectiveness of EE methods. When combined with shared early-exit head, the AC-Mixer achieves the highest speedup more than
26

---

3 be-efiiii from both implicit compression and confident early exits. This synergy highlights the strength of ACNs as a foundation for efficient inference.
We also draw a parallel between this behavior and the compatibility of ACNs with pruning methods (see Appendix in both cases, the compact representations of ACNs, through auto-compression, facilitate downstream efficiency techniques. Developing new early-exit strategies tailored to ACNs is promising direction for future work.
27


Reviews:
[{"review_id": "UkLpDZrwn0", "reviewer": "Reviewer_DqLE", "summary": "This work proposes a new network architecture that includes long skip connections from each layer to the output. The authors demonstrate that this design outperforms traditional skip-connection networks across diverse modalities and training settings, including supervised learning, low-data regimes, and task-incremental learning.", "questions": "* Why do ACNs show better generalization? Are there any theoretical explanations or underlying intuitions that support this behavior?\n* How many runs were performed for each experiment? Please report mean and variance if applicable.\n* Line 183: \"AC-ViT converges at 700 epochs, while the Residual ViT converges at 300 epochs.\" Does this imply that Residual ViT converges faster than AC-ViT? If so, can you clarify the implications for training efficiency? Please include learning curves if possible. Additionally, do all residual networks converge faster than ACNs across tasks?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* Well-designed experiments are provided, offering detailed analysis that directly supports the authors' claims.\n* Experimentally, the proposed method outperforms residual networks in reducing forgetting, improving generalization, increasing robustness to input noise, and achieving better capacity utilization.\n\n### Weaknesses\n\n* The equations and figures in Table 1 appear to be incorrect. For example, in the ResNet architecture, there should not be a connection between the first and last layers. Additionally, activation functions are missing from the equations. In the case of FFN, the final output $y_F$ cannot be obtained as currently formulated unless only linear activations are assumed. This omission likely affects all derivations in Appendix C.\n* In most experiments, the proposed method is compared only against ResNet. Ideally, additional baselines—such as those mentioned in the related work section—should be included for a more comprehensive evaluation.\n* Table 4 lacks error bars, which limits the interpretability and reliability of the reported results.\n\nI am open to increasing the score once these limitations are addressed. If corrected, this paper has the potential to make a meaningful contribution to the community.", "was_revised": true, "final_justification": "The rebuttal addressed all my concerns. Thus, I incline to accept this work.", "rebuttal": "##  **Response to Reviewer**\n\nWe thank the reviewer for the detailed technical feedback on our mathematical formulations, experimental setup, and theoretical foundations. Your questions helped us clarify key aspects of ACNs and strengthen our experimental validation.\n\n### **Key highlights of our response** \n- *Mathematical clarification*: We've clarified that the equations in Table 1 refer to an 1D linear network, while the connectivity diagram shows the 2D linear case. \n- *Expanded baselines*: New comprehensive comparison including DenseNet and DenseFormer variants on CIFAR-10, showing ACNs achieve the best performance with auto-compression.\n- *Error bars added*: Updated Table 4 with proper statistical reporting (mean ± variance).\n- *Theoretical insights*: Detailed explanation of why ACNs show better generalization through more effective depth utilization and robust layer-wise training dynamics.\n\n### **Detailed response** \n\n**Regarding Table 1:**  The mathematical analysis for clarity and simplicity is conducted for 1D linear neural networks of depth L as mentioned in line 98 and in the caption of Table 1. However, the connectivity is shown for a 2D linear case. We apologize if this was not clear for the connectivity and we will modify the caption of Table 1 to state it explicitly (2D case -> 2D linear case for connectivity diagram vs 1D linear case for the equations). \n\nTo clarify a bit further: with the red connections in the case of ResNet connectivity we show the implicit direct connections (paths) that are formed in the architecture through the residual summation.  We denote: \n- $z_0 = x_0$\n- $z_{i-1}$: input to layer $i$ (  for $i>0$), that is multiplied with $w_i$ to get:\n- $x_i = w_i \\cdot z_{i-1}$: output of layer $i$ (BEFORE residual sum)\n- $z_{i} = z_{i-1} + x_i$: output of layer $i$ AFTER residual sum, or equivalently input to layer $i+1$\n\n\nSo in our case, we want to express the input to the final layer $z_2$, which is:\n\n- $z_0 = x_0$ \n- $z_1 = x_1 + z_0 = x_1 + x_0$ \n- $z_2 = x_2 + z_1 = x_2 + x_1 + x_0$\n\n\nWe will add this detailed explanation in the updated Appendix.\n\n**Regarding comparison with discussed related works:** We agree that comparison with discussed related works on different architectural ideas should be included. For this purpose, we conducted an experiment with the setup of MLP-Mixer on CIFAR-10 including: \n- an ACN style Mixer (AC-Mixer) \n- a Residual Mixer \n- a DenseNet(https://arxiv.org/abs/1608.06993) style Mixer, where each layer takes as input all previous layers' outputs, concatenates them, and projects them to *hidden_dim* with a learnable projection matrix per layer. For this variant, to keep the number of parameters the same as the others (because these projection matrices introduce extra parameters for each layer that also grow with depth), ensuring fair comparison, we set the number  of layers to 10 (instead of 16).\n - a DenseFormer(https://arxiv.org/abs/2402.02622) style Mixer, where each layer takes as input a linear combination of all previous layers' outputs with learnable scalars (each layer has one learnable scalar for each previous layer) \n\nTo allow fair comparison, since ACNs are typically trained slower, we run all models for 700 epochs and report best accuracy (Acc.) and the (average across runs) epoch at which each model achieved the best accuracy (Best Acc. Epoch). Furthermore, we report Cutoff Layer, which is the earliest layer that reaches top performance (showcasing auto-compression):\n\n| Models | Accuracy | Cutoff Layer | Best Acc. Epoch |\n|--------|----------|--------------|-----------------|\n| Res-Mixer | 90.3±0.09 | 16 | 615 |\n| **AC-Mixer** | **92±0.08** | **12** | **695**|\n| DenseNet-Mixer | 90.3±0.1 | 16 | 600 |\n| DenseFormer-Mixer | 90.4±0.09 | 16 | 637 |\n\nFrom the results, we see that the AC-Mixer achieves the best performance (even better than the numbers reported in the main paper, due to increased training time), showcasing improved generalization despite the longer convergence time. Moreover, AC-Mixer is the only variant that exhibits Auto-Compression. \nWe plan to include these results in the updated Appendix. \n\n**Regarding Table 4:** We present below the updated table with the error bars and we plan to also update the Table in the updated version of the paper. \n\n| Models | Acc.|\n|--------|---------|\n| **AC-Mixer** | **85.38±0.7** |\n|Aligned | 82.9±0.9|\n| LayerSkip | 79±1.2 |\n\n**ACN vs ResNet generalization capabilities:** While Residual Networks were introduced to offer efficient and effective training especially in very deep architectures, there are various works that explore behaviors of these architectures that may harm generalization:\n- In [1, 2] it is shown that these architectures exhibit a notable resilience to layer dropping and permutation, which could indicate potential redundancy of some layers (i.e. removing a layer does not affect the network). \n- In [3], it was further observed that dropping subsets of layers during training can reduce overfitting and improve generalization. \n- In a related study [4], the authors showed that introducing skip connections between layers can lead to parts of the network being effectively bypassed and under-trained.\n- In [5], they show that unscaled residual connections can harm generalization capabilities in generative representation learning.\n- More recently, research has revealed substantial parameter redundancy and inefficient parameter usage in large-scale foundation models, particularly within their deeper layers [6] and  also [7].\n\nAll these observations can be unified under the perspective that, although residual architectures facilitate training via multiple signal pathways, these same pathways\ncan sometimes act as shortcuts that cause certain components to be either underutilized or prone to overfitting—ultimately limiting effective generalization. That means that vanilla residual architectures may not utilize their depth that effectively. ACNs have the potential of achieving better representations through direct derivative flow to each layer and the removal of skip connections not allowing layers to be bypassed during training. In this work, we experimentally show that ACNs  can match Residual Networks task performance while utilizing significantly fewer layers, while the representations learned can exhibit extra benefits such as noise robustness.  A detailed analysis of mechanistic interpretability of each architecture is left as future work.\n\n[1] https://arxiv.org/abs/1605.06431, [2] https://arxiv.org/abs/2407.09298, [3] https://arxiv.org/abs/1603.09382, [4] https://arxiv.org/abs/1610.01644, [5] https://arxiv.org/abs/2404.10947, [6] https://arxiv.org/abs/2403.17887, [7] https://arxiv.org/abs/2505.13898\n\n**Detailed summary of results:** We present in the table below mean and variance for the main experiments (along with parameter counts, flops, storage size)\n\n| Models | Accuracy ↑ | #Params ↓ | GFLOPs ↓ | Size in MB ↓ |\n|--------|------------|-----------|----------|--------------|\n| Res-ViT on ImageNet | 70.74±0.09 | 86M | 33.7 | 330 |\n| **AC-ViT on ImageNet** | **70.76±0.12** | **51M** | **19.7** | **195** |\n| | | | | |\n| Res-BERT on SST-2 | 86.63±0.09 | 110M | 21.72 | 418 |\n| **AC-BERT on SST-2** | **86.68±0.06** | **46M** | **5.44** | **174** |\n| Res-BERT on QNLI | 83.14±0.07 | 110M | 21.72 | 418 |\n| **AC-BERT on QNLI** | 83.07±0.1 | **46M** | **5.44** | **174** |\n| Res-BERT on QQP | 87.2±0.09 | 110M | 21.72 | 418 |\n| **AC-BERT on QQP** | **87.3±0.07** | **46M** | **5.44** | **174** |\n\nWe will add this table to the updated Appendix.\n\n**Discussion on Training time:** We found that in most cases ACNs do converge slower than residual networks (sometimes needing double the number of steps). We believe that this might be due to the direct derivative flow to the attention modules making inter-layer coordination harder to achieve, as well as a smaller number of information pathways that facilitate information flow. We view the increased training time as a limitation of the architecture that is outweighed by better and more compact representations.   An active research direction that we are exploring is how to combine both architectures to achieve improved training efficiency while enjoying the benefits of Auto-Compression. Moreover, since training represents a one-time upfront cost while inference occurs repeatedly throughout a model's deployment lifecycle, the substantial inference speedups make this trade-off favorable for real-world applications.\n\nWe are happy to provide any additional analysis upon the reviewer’s request."}, {"review_id": "VB5groQNKB", "reviewer": "Reviewer_PUR5", "summary": "The proposed method, Auto-Compressing Networks, introduces a new architectural variant for neural networks where long feed-forward connections from each layer to the output replace shorter residual connections. This architectural modification is shown to organically compress information during training, with early layers capable of highlighting features that are sufficient for high performance, as well as number of other benefits like enhanced robustness to noise, superior performance in low-data settings, improved transfer learning capabilities and lower catastrophic forgetting.", "questions": "- Could the authors provide an in-depth comparison of the parameter count + FLOPs for the example of a single block (within any of the architectures used in the experiments) in addition to the overall training time required (for a chosen experiment)? A comprehensive comparison could help identify scenarios where the selection of the proposed architecture and its improved performance outweighs the cost.\n- Could the authors provide notations on the graph of Table 1 that correspond to the notations used for forward and backward propagation?\nThis would help the reader quickly parse the table's information.\n- Could the authors discuss the impact of hyper-parameter optimization on the experiments conducted in the paper (a subset would suffice)? The intention is to isolate the cause for improved performance and ensure the main reason is the change in architecture as opposed to sub-optimal hyper-parameter choice.\n- I would recommend the authors provide the definition of \"incremental layer performance\" early on in the manuscript to ensure correct understanding of Fig. 1.\n- Could the authors clarify how the gradient norm is normalized in Fig. 1?\n- Could the authors double check MLM-Mixer in Fig. 2's caption? In addition, could the authors also double check the last 2 sentences in Section 3 (L. 171-176) and confirm whether the statements are correct?\n- In Section 4.1, the manuscript mentions \"ACNs can improve inference time and memory consumption without sacrificing performance\". Are these exclusively measured using the layer vs. accuracy graphs? If so, from a practical standpoint, could the authors provide a params + FLOPs reduction count for the optimal layer setting as well? (Is this layer setting sufficient across all experiments?)\n- Could the authors also discuss the potential of over-fitting when compared to Residual networks? (POV: Longer training time and smaller-scale datasets)\n- Could the authors also provide a comparison where the residual architecture is trained for number of epochs = convergence epochs of ACN variant? In addition, could you comment on a potential causes + solutions to speed up convergence?\n- Interestingly, in low-data settings, the convergence pattern is reversed. Could the authors discuss potential reasons for this?\n- Due to the nature of the architectural variation, could the authors discuss and compare against early exit networks? (training strategy, stability, performance ?)\n- To develop an even stronger appreciation for the changes affected by ACNs, could the authors discuss, using retrieval metrics or other similarity metrics, the feature representations learned by ACNs and Residual-variants?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality\n\nThe technical content and claims made in the paper are straightforward and well supported by experimental evidence. For individual comparisons, while the baselines are relevant they are not the current SoTA in all cases, slightly weakening the impact of the proposed model. Providing a more rounded discussion of the increase in size and training time and scenarios where ACNs might have drawbacks would serve to make this a very well rounded paper.\n\nClarity\n\nThe manuscript is extremely well written and clear, with sections dedicated to specific claims and supporting experimental results.\n\nSignificance\n\nWith the potential inclusion of FLOPs + Parameter counts and training time comparisons, the simplicity and implicit training dynamics  of ACNs could be extremely beneficial to the community at large, with the added bonus of offering larger compression and real-time deployment options.\n\nOriginality\n\nThe discussion about connection points with existing architectural variants and how ACN-specific changes affect training underline relevant hypotheses, novelty and how ACNs distinguish themselves.", "was_revised": true, "final_justification": "The authors have provided comprehensive and detailed responses to all queries from my side in addition to questions from other reviewers as well. With the current state of responses in mind most of the remaining questions with relative short responses are more exploratory in nature. \nWith all of these in mind, my recommendation would be to accept the paper (with the caveat that all the manuscript level modifications are complete).", "rebuttal": "##  **Response to Reviewer**\n\nThank you for your constructive feedback on computational efficiency, practical deployment considerations, and the need for more comprehensive analysis. Your suggestions have helped us provide a more complete picture of ACNs' practical benefits.\n\n### **Key highlights of our response**\n\n- *Comprehensive efficiency analysis*: Detailed FLOPs, parameter counts, and storage comparisons showing up to 4× reduction in inference costs\n- *Training vs. inference trade-offs*: Thorough analysis of roughly doubled training time against significant deployment benefits\n- *Fair comparison experiments*: All architectures trained for equal epochs (700) on CIFAR-10, confirming ACNs achieve better generalization (rather than overfitting) despite slower convergence\n- *Practical deployment focus*: Emphasis on one-time training cost vs. repeated inference savings for real-world applications\n\n### **Detailed response**\n\n**Regarding FLOPs, Parameter counts and Training Time:** Below we present an analytical Table showing performance across runs, inference FLOPs, number of Parameters (after potential layer removal in ACNs) and Storage Size in MB for the ImageNet and BERT experiments:\n\n| Models | Accuracy ↑ | #Params ↓ | GFLOPs ↓ | Size in MB ↓ |\n|--------|------------|-----------|----------|--------------|\n| Res-ViT on ImageNet | 70.74±0.09 | 86M | 33.7 | 330 |\n| **AC-ViT on ImageNet** | **70.76±0.12** | **51M** | **19.7** | **195** |\n| | | | | |\n| Res-BERT on SST-2 | 86.63±0.09 | 110M | 21.72 | 418 |\n| **AC-BERT on SST-2** | **86.68±0.06** | **46M** | **5.44** | **174** |\n| Res-BERT on QNLI | 83.14±0.07 | 110M | 21.72 | 418 |\n| **AC-BERT on QNLI** | 83.07±0.1 | **46M** | **5.44** | **174** |\n| Res-BERT on QQP | 87.2±0.09 | 110M | 21.72 | 418 |\n| **AC-BERT on QQP** | **87.3±0.07** | **46M** | **5.44** | **174** |\n\nWe see that consistently ACNs showcase significantly reduced GFLOPs (up to 4 times less) and reduced storage requirements without performance degradation, making them very well-suited for resource constrained real-world deployment.\n\nWe will add this table to the updated Appendix.\n\nA block of an ACN is similar to a ResNet block in terms of computation, both having a residual summation:\n\n- ResNet block: y = f(x); x = y + x\n\n- ACN block: y = f(x); accum = accum + y\n\nIn terms of training time, it's true that ACNs typically require more steps to converge, roughly double the steps in both the discussed experiments. We believe that this is a Limitation (as also mentioned in the related section of the paper and discussed in the response to R1) that requires further investigation. One interesting question is whether there exists an actual trade-off between training time and generalization capabilities (strength of learned representations), since it seems that ACNs utilize the increased training time to learn more robust representations often with improved generalization (see also experiment below). An interesting direction that we are currently exploring is combining the two architectures in order to improve training efficiency while keeping the auto-compression property that drives strong representation learning. All in all, we believe that since training is a one-time process, the significant inference benefits of ACNs along with the representational benefits (noise robustness, generalization) can potentially balance the increased cost of the training process. \n\n**Regarding potential overfitting and fair comparison in terms of training epochs:** Relevant to the previous discussion, we conducted an experiment with various architectures with the setup of MLP-Mixer on CIFAR-10, training all models for 700 epochs to ensure fair comparison (as requested also by R1). The involved models are: \n- an ACN style Mixer (AC-Mixer) \n- a Residual Mixer \n-  a DenseNet(https://arxiv.org/abs/1608.06993) style Mixer, where each layer takes as input all previous layers' outputs, concatenates them, and projects them to *hidden_dim* with a learnable projection matrix per layer. For this variant, to keep the number of parameters the same as the others (because these projection matrices introduce extra parameters for each layer that also grows with depth), ensuring fair comparison, we set the number  of layers to 10 (instead of 16).\n - DenseFormer(https://arxiv.org/abs/2402.02622) style Mixer, where each layer takes as input a linear combination of all previous layers' outputs with learnable scalars (each layer has one learnable scalar for each previous layer) \n\nWe report best accuracy (Acc.) and the epoch (averaged over runs) at which each model achieved it (Best Acc. Epoch). Furthermore, we report Cutoff Layer, which is the earliest layer that reaches top performance (compared to last):\n| Models | Accuracy | Cutoff Layer | Best Acc. Epoch |\n|--------|----------|--------------|-----------------|\n| Res-Mixer | 90.3±0.09 | 16 | 615 |\n| **AC-Mixer** | **92±0.08** | **12** | **695** |\n| DenseNet-Mixer | 90.3±0.1 | 16 | 600 |\n| DenseFormer-Mixer | 90.4±0.09 | 16 | 637 |\n\nThe results show that  ACNs converge slower than all residual variants but utilize this increased training time to learn representations that generalize better and are more compact, rather than overfit to the dataset reaching better performance.\n\n**Regarding Early Exit methods** We provide a detailed  discussion of the relations between ACNs and the EE method in our response to R3; because of word limits in this response we cannot  copy the analysis here, we refer the reviewer to our response to R3 (Reviewer PWfu).   \n\n**Comments on the Manuscript:**\n\n- For the notations of Table 1, we presented in more detail in the response of R1 what the dashed lines represent in the forward pass (direct connections). In the backward propagation, for ACNs and ResNets, for a layer i, the dashed lines (direct connections) denote the Direct Gradient path, where signals coming from subsequent layers form the Network-mediated gradient term. We presented the table this way for simplicity but we will consider your suggestions.\n\n- Thanks for the recommendation regarding the term \"incremental layer performance\" in Figure 1, we will add the definition to the caption of the Figure in the updated version. With  \"incremental layer performance\" of layer i, in ACNs we mean adding the output of layer i to the current sum (taking a partial sum from input embedding to this layer), passing it to the classifier (common head) and measuring performance. For ResNets, we get the output of layer i, add it to the residual stream, pass the result to the common classifier and get performance. So this heatmap essentially shows how the new computation induced by a layer increases the overall performance of the network (up to this layer).\n\n- In Figure 1 we take the L2 norm of the gradient of a layer. We will add this to the caption of the Figure. \n\n- In Figure 2 MLM-Mixer is a typo, the correct term is MLP-Mixer, thanks for noticing, we will update accordingly. Regarding (L. 171-176) there is an error in  the caption of  Figure 1  - we apologize for that. Specifically, the LEFT Figure (16 layers - 100epochs) is the MLP-Mixer on Cifar-10 experiment described in section 3 and the RIGHT figure is the AC-ViT on ImageNet experiment (12 layers - 300epochs). We will correct the caption and we apologize for the confusion. With this correction, in (L. 171-176) we note that as we see in Figure 1 (Left) the gradients of the residual network seem much more uniform compared to ACNs, with higher norms concentrating in the early and deeper layers.\n\n**Low Data settings** \nThis was an interesting behavior that we observed. We hypothesize that this may be related to the distinct training dynamics of ACNs compared to ResNets. While ResNets are designed to utilize all parameters uniformly, ACNs may be more adaptive to data availability through their implicit layer-wise training dynamics. This is partly supported by our Task Difficulty experiment (Section 3), where ACNs appear to dynamically adjust capacity usage based on task complexity, while ResNets utilize all layers regardless. However, the exact mechanisms underlying this behavior are still under investigation.\n\n**Hyper-parameter optimization** We did not experiment extensively with hyperparameters, but rather used the default hyperparameters:   ViT on ImageNet from the original paper, and BERT pretraining and fine-tuning).  While experimenting further with hyperparameters  might improve ACN performance or convergence speed, we show that using off-the-self vanilla resnet hyperparameter values provide good results. One observation worth noting is that in some cases, ACNs benefit from higher warmup and/or lower learning rates compared to Residual Networks. This more conservative setup likely helps facilitate the implicit layer-wise dynamics found in ACNs.\n\n**Representations in ACNs vs ResNets:** We have an active research thread on mechanistic interpretability for ACNs and hope to be able to report to the community soon our findings. We plan to include a more detailed analysis of feature representations using similarity metrics and retrieval methods in our revised manuscript. Preliminary analysis suggests ACNs learn more compressed yet discriminative representations in early layers.\n\nWe are happy to provide any additional analysis and discuss further upon the reviewer’s request."}, {"review_id": "pdSsC9GDsK", "reviewer": "Reviewer_PWfu", "summary": "This paper introduces Auto-Compressing Networks (ACNs), which replace traditional short residual connections with long feedforward connections from each layer directly to the output. The authors claim this architectural change leads to \"auto-compression\"—a property where networks organically concentrate information in early layers during training, making later layers redundant. They provide theoretical analysis of gradient dynamics and demonstrate that ACNs can achieve 30-80% layer reduction while maintaining performance across vision and language tasks, with some other benefits in noise robustness, continual learning, and transfer learning.", "questions": "1. can you give some head-to-head comparisons with MSDNet, BranchyNet, and EENet on identical datasets? Given the overlap, direct comparisons seem essential.\n2. Can you show ACN results on full ImageNet and larger language models? The current small-scale experiments limit the generalisability claims.\n3. From what I understand, training ACNs takes roughly twice as long as standard networks. When you factor in both the training overhead and inference costs, how do ACNs compare overall? I'm curious about the total computational budget, not just the inference savings - granted this is a one time cost.\n4. There's already quite a bit of research on early exit mechanisms in neural networks. Beyond the specific way ACNs are architected, what makes them fundamentally different from existing early exit approaches? I'd like to understand the unique contribution here.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths\n\n+ The gradient flow analysis (Table 1, Section 2.2) provides good intuition for why ACNs exhibit different training dynamics than ResNets, with the decomposition into Direct Gradient and Network-mediated Gradient components being interesting to read about.\n+ The core auto-compression is demonstrated consistently across multiple architectures (ViT, MLP-Mixer, BERT) and datasets, showing the effect is not architecture-specific.\n+ It's nice to see the authors also evaluate noise robustness, continual learning, and transfer learning, providing a broader picture of ACN behavior.\n+ The architectural change is minimal and hardware-friendly, requiring no specialised software or complex hyperparameter tuning.\n\n Weaknesses\n\n- The core contribution—networks with multiple prediction points that can exit early—has been extensively studied in the early exit literature (BranchyNet 2017, MSDNet 2017, EENet 2023). The paper fails to adequately compare with or even cite many key works in this space. MSDNet for example, already achieves \"6% higher accuracy with 2-3× less computation time\" on ImageNet\n- Most experiments are on small-scale tasks (CIFAR-10/100, ImageNet with 256 batch size). For an architecture claiming broad applicability, this is insufficient. Early exit methods have been demonstrated at full ImageNet scale with superior results.\n- ACNs seem to require significantly more training time (AC-ViT: 700 vs 300 epochs for ResNet). This negates claimed efficiency benefits and suggests the method may not scale to larger models where training cost matters more - unless I'm missing something?\n- The paper claims \"groundbreaking\" results but often just matches baseline performance. Early exit methods like BranchyNet show 2-6× speedups, while ACN improvements are much more modest.\n- No direct comparison to established early exit methods (MSDNet, BranchyNet, EENet) on the same tasks. This is a significant oversight given the clear conceptual overlap.\n- Different convergence times between ACN and baseline variants make fair comparison difficult.\n- Although it's great to see some transfer learning eval comparisons -they are pretty narrow with only CIFAR-100→CIFAR-10 transfer, which are very similar datasets.", "was_revised": true, "final_justification": "A thorough rebuttal demonstrating overall solid work. The paper makes a meaningful architectural contribution that complements existing efficiency methods.", "rebuttal": "##  **Response to Reviewer**\n\nWe thank the reviewer for highlighting the early exit literature and pushing us to clarify ACNs' unique contribution. Your feedback helped us articulate that ACNs represent an architectural insight about connectivity patterns and  how these affect information flow and training dynamics, rather than another early exit method. We plan to include this discussion in the updated manuscript.\n\n### **Our core argument** \n\nOur work reveals a fundamental architectural principle: direct-to-output connections coupled with reduced connectivity between layers (not \"all-to-all\" as in the case of residual networks) naturally compress information into early layers while maintaining performance—a property we term auto-compression. This insight transcends existing residual architecture variants by showing how connection topology intrinsically shapes representation learning. ACNs demonstrate that we can achieve ResNet-level performance with significantly fewer active layers and more robust representations (suitable for noisy and limited data tasks, as well as continual learning settings), opening new possibilities for hybrid architectures that adaptively combine short and long connections for optimal training dynamics, improved continual learning and deployment efficiency, particularly crucial for large language models.\n\n### **Key highlights of our response** \n\n- *Architectural innovation vs. training strategy*: ACNs introduce a connectivity pattern that fundamentally changes gradient flow, complementing (not competing with) early exit (EE) training strategies\n- *Empirical validation of the insight*: ACN+EE outperforms ResNet+EE, proving that our connection pattern enhances existing methods\n- *Foundation for future work*: Our architecture opens new research directions in hybrid long/short connection patterns for large-scale models\n- *Training efficiency context*: Detailed analysis of training time trade-offs and their relationship to improved generalization\n- *Transfer learning validation*: Additional ImageNet→CIFAR-10 experiments confirming ACNs' representational advantages across domains\n\n### **Detailed response**\n\n**Key Insight of our work:** The key insight of our work is not about early exit per se, but about how different connectivity patterns affect information flow, compression and ultimately representations during training - an insight that could help improve how large-scale architectures are designed.\n\n**Relation to Early Exit (EE) Methods:** ACNs are fundamentally different from Early Exit (EE) methods. While EE methods are training strategies that add explicit intermediate losses to existing architectures, ACNs are architectural backbones that achieve similar benefits through implicit design alone. Importantly, ACNs can be combined with EE methods to amplify their effectiveness.\nTypically  EE methods (e.g., see BranchyNet) methods follow a common pattern: (1) they add explicit intermediate losses at multiple layers to improve early feature discriminability, (2) these losses are weighted and combined with the final loss during training, and (3) confidence metrics like entropy are used at inference to decide where each sample should exit. This approach, pioneered by Deep Supervision (https://arxiv.org/abs/1409.5185), requires careful hyperparameter tuning of the loss weights.\n\nACNs do not have any explicit regularizer in terms of extra intermediate losses or other regularizers, but instead they are trained \"naturally,\" like residual or feedforward networks, and exhibit---through architectural design alone---improved intermediate layer performance (similar to intermediate losses) through implicit layer-wise dynamics. \n\nIn Section 7, we show that coupling residual networks with these external intermediate losses requires careful tuning of the loss weights or it can result in potential overfitting (e.g., setting early losses to high weights can \"overpush\" early layers to achieve good predictions and results in overfitting on the training dataset). In practice, when compared with some recent works on intermediate losses, we showed that ACNs generalize better (as seen from improved transfer learning in the respective section) without requiring any extra hyperparameter tuning but instead relying on implicit architectural regularization.\n\nNonetheless, ACN’s auto-compression property (pushing information naturally to earlier layers)  can be exploited by EE methods and magnify their effect. To empirically show this, we conducted an experiment on MLP-Mixer on CIFAR-10 where we compare: \n- a Residual Mixer \n- an AC-Mixer\n -  a Residual Mixer with Branches (discussed below) \n- an AC-Mixer with Branches \n- a Residual Mixer with common EE head (discussed below) \n- an AC-Mixer with common EE head (discussed below) \n\n*Branches*: The idea is similar to BranchyNet. Specifically, we incorporate branches after layers 4, 8, and 12 with an MLP classifier to get intermediate branch losses that we use during training. After each of these layers, the outputs are passed to a dedicated classifier of the branch to get logits and calculate a loss that is combined with the common full network loss. At inference, we calculate the entropy of the logits of the branch classifiers and if it is below a certain threshold, we choose for this input to exit at the partlicular branch.\n\n *Common EE head*: Based on more recent works [Aligned, LayerSkip - see paper], we get intermediate losses for each layer by passing its output to the common classification head and getting the logits. This has the advantage that no extra parameters are introduced, while fine-grained multiple exiting is achieved (since there is an exit at each layer---each layer is a branch). Again at inference, we calculate the entropy of the logits of an intermediate classifier and if it is below a threshold, then we exit at this layer.\nThe results are shown below---the speedup is counted in terms of FLOPs relative to the FLOPs of the full Residual Network (16 layers):\n\n| Models | Speedup |\n|--------|---------|\n| Res-Mixer | 1x |\n| AC-Mixer | 1.5x |\n| Res-Mixer with Branches | 2.2x |\n| AC-Mixer with Branches | 2.6x |\n| Res-Mixer with shared EE head | 2.4x |\n| AC-Mixer with shared EE head | 3.3x |\n\nWith 1$\\times$ we have the Residual Network without EE exits, essentially utilizing the full network irrespective of the input. Then we see that ACN with Branches outperforms the residual variant with Branches, confirming our intuition that auto-compression can boost EE methods. Moreover, we see that coupling ACN with the common EE head method, which allows more fine-grained early exiting, provides a significantly larger speedup boost (and larger gap between ACNs and Residual Networks), showcasing the ability of ACNs to utilize different layers for different inputs and build strong EE schemes with  their internal classification head.\n\nFinally, we note that we view the behavior of ACNs with EE methods in a similar manner as ACNs coupled with pruning techniques that we showed in Appendix D: the inherent ability of ACNs to represent information better and more efficiently into fewer layers acts complementarily to pruning and EE  techniques and boosts their effect. Exploring new EE methods tailored for ACNs for even better efficiency is an exciting research direction that we are currently exploring.\n\n**Longer Training Times:**  We refer the reviewer to the related discussion with R1(**Discussion on Training time:**) and R2 (**Regarding potential overfitting and fair comparison in terms of training epochs**). Unfortunately, we cannot copy the responses here due to word limits. \n\n**Transfer Learning:** For testing the generalizability of the learned ACN representations we tested Cifar-100 to Cifar-10 transfer learning as you mentioned. We  also tested the BERT pre-training to fine-tuning (section 4.2) where we showed that ACNs achieve on-par performance with residual variants by utilizing 1/4th of the full network (3 vs 12 layers). Finally we present below results on transfer learning from imagenet (ViT) to Cifar-10:\n\n| Models | Acc.| Cutoff Layer|\n|--------|---------|---------|\n| Res-ViT | 94.09 |12 |\n| AC-ViT | 94.08 |6 |\n\nWe see that again, ACNs achieve comparable performance with residuals while utilizing half of the network.\n\n**Scaling up:**\nWhile our current experiments are at moderate scale, the architectural insights about connection patterns become even more important for large models where both training and inference efficiency are critical. We are currently scaling these experiments to larger models and datasets, including full ImageNet training and large language models. Early results suggest that the auto-compression property becomes more pronounced at scale, making ACNs particularly promising for efficiency-critical large model deployment.\n\nWe are happy to provide any additional analysis and discuss further upon the reviewer’s request."}, {"review_id": "uO7Q4DwT7I", "reviewer": "Reviewer_ndrj", "summary": "This paper proposes a new type of residual connection method, which makes the final network output directly combine all the previous layers' outputs. This network has the ability called \"auto-compressing\", which can identify the redundancy of layers in the neural networks.", "questions": "See weakness above.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper writing is fluent and easy to read. The motivation is clear and easy to follow. The figures and tables are also clear.\n2. The method is simple but effective, showing interesting \"auto-compressing\" properties of the network, which shows great potential in network compression.\n3. The performance is good compared to the resnet counterpart.\n4. The experiments are highly sufficient, showing their great potential in performance experiments, compression experiments, robust experiments, continual learning, and transfer learning.\n\nWeaknesses:\n1. Why Direct Gradient is so important in model training? It seems that its Direct Gradient makes the ACNs have the ability of the \"auto-compressing\". Do you have some hints of why?\n2. It will be more convincing if you compare your method with some other variants of residual connection, like DenseNet.\n3. Have you tried to further reduce the ACNs residual connections of some rear layers to further enhance the importance of the Direct Gradient to early layers and enhance the compression rate? Which I think can further verify your claims.", "was_revised": true, "final_justification": "This is a solid and interesting work, and the authors' responses have addressed most of my concerns. Thus, I maintain my accept score.", "rebuttal": "##  **Response to Reviewer**\n\nThank you for taking time reading our manuscript and providing your insightful questions about the theoretical foundations of ACNs, particularly the role of Direct Gradient in driving auto-compression.\n\n### **Key highlights of our response** \n\n- *Direct Gradient mechanism*: More detailed explanation of how direct gradient signals improve intermediate layer performance and drive information concentration\n- *Architectural comparisons*: Comprehensive comparison with DenseNet and other connectivity patterns as requested\n- *Gradient component analysis*: Experimental validation showing Direct Gradient drives auto-compression while Network-mediated Gradient remains important for performance\n\n### **Detailed response** \n\n**The role of Direct Gradient:** Indeed , Direct Gradient (DG) is an important component for training, but it has to be  combined with the reduction of derivative flow through the residual paths for auto-compression to emerge (which is what happens in ACNs). We note also here that there is a line of work that investigates a similar idea (taking gradient signal directly from the output) as an alternative to backpropagation (see Direct Feedback Alignment).\n\nIn our case, the significant contribution of the Direct Gradient (due to the linear number of paths) plays a crucial role in implicitly improving intermediate layer performance and driving the emergence of auto-compression. If we take as an example a classification task where $x_i$ is the output of layer $i$, the full output of the network is $y = x_L + \\ldots + x_0$ (assuming $L$ layers), which is passed to a classification head to get a prediction. For a layer $i$, the information contained in the Direct Gradient dictates how this layer's weights need to change so that $x_i$, when summed to produce $y$, directly increases the performance. In this way, the partial sum up to layer $i$, $y_{\\text{partial}} = x_i + \\ldots + x_0$, is also directly improved (in terms of classification performance), effectively enhancing intermediate layer performance and driving information concentration in early layers. Another way to see this is by writing ACNs as $y = 1+w_1(1+w_2(1+w_3\\ldots))$. This asymmetric form induced by ACNs, reveals that early layers are critical for the entire network---for example, a small $w_1$ makes all subsequent layers largely irrelevant. The DG signal facilitates training of these crucial early layers. In contrast, ResNets have the form $y = (1+w_1)(1+w_2)\\ldots$, where the multiplicative structure with the +1 terms makes the network more robust to small weights in any individual layer.\n\nThus, the Direct Gradient also facilitates early layers' learning at early stages of training, since the Network-mediated Gradient (NG) term consists of untrained weights and thus provides a noisy training signal, whereas the Direct Gradient, coming directly from the output, provides a more robust and reliable training signal early in training.\n\n**Regarding comparison with discussed related works:** We copy here for convenience the response to R1 (since word limit allows it), where we conducted an experiment including related works for comparison:\n\nWe agree that comparison with discussed related works on different architectural ideas should be included. For this purpose, we conducted an experiment with the setup of MLP-Mixer on CIFAR-10 including: \n\n- an ACN style Mixer (AC-Mixer) \n- a Residual Mixer \n- a DenseNet(https://arxiv.org/abs/1608.06993) style Mixer, where each layer takes as input all previous layers' outputs, concatenates them, and projects them to *hidden_dim* with a learnable projection matrix per layer. For this variant, to keep the number of parameters the same as the others (because these projection matrices introduce extra parameters for each layer that also grow with depth), ensuring fair comparison, we set the number  of layers to 10 (instead of 16).\n - DenseFormer(https://arxiv.org/abs/2402.02622) style Mixer, where each layer takes as input a linear combination of all previous layers' outputs with learnable scalars (each layer has one learnable scalar for each previous layer) \n\nTo allow fair comparison, since ACNs are typically trained slower, we run all models for 700 epochs and report best accuracy (Acc.) and the (average across runs) epoch at which each model achieved it (Best Acc. Epoch). Furthermore, we report Cutoff Layer, which is the earliest layer that reaches top performance (compared to last):\n| Models | Accuracy | Cutoff Layer | Best Acc. Epoch |\n|--------|----------|--------------|-----------------|\n| Res-Mixer | 90.3±0.09 | 16 | 615 |\n| **AC-Mixer** | **92±0.08** | **12** | **695** |\n| DenseNet-Mixer | 90.3±0.1 | 16 | 600 |\n| DenseFormer-Mixer | 90.4±0.09 | 16 | 637 |\n\nFrom the results, we see that the AC-Mixer achieves the best performance (even better than the numbers reported in the main paper, due to increased training time), showcasing improved generalization despite the longer convergence time. Moreover, AC-Mixer is the only variant that exhibits Auto-Compression. \nWe plan to include these results in the updated Appendix. \n\n**Reducing residual connections / Boosting Direct Gradient:** This is a valid and insightful point. In ACNs, each layer is connected directly to the next one and to the last one. The extra connections (non-direct), which are parts of the Full Gradient, are implicit and it is not straightforward to remove them. However, following the same line of thought, the general idea—which is intriguing—is that for a layer, one can essentially weight the backward signals, for example to enhance the Direct Gradient signal, effectively breaking the symmetry of backpropagation. While we did not test this approach, which would require careful implementation, we believe it is an interesting way to investigate the behavior of the different gradient signals.\nHowever, an experiment along the same lines is presented in Section 3, where we trained a variant of AC-Mixer that receives only the Direct Gradient as a backward signal (this setting represents the extreme case of what you are suggesting—removing all but the Direct Gradient signal). What we observed is that this variant exhibits strong Auto-Compression, highlighting the importance of that term in the emergence of this property. However, the model fails to match the performance of the other models, underscoring at the same time the importance of the NG component in task learning.\n\nWe are happy to provide any additional analysis upon the reviewer’s request."}]

Author rebuttal:

Meta-review: This paper introduces auto-compressing networks, with feedforward connections from each layer to the final layer. These networks are capable of generalizing better than typical residual networks, and have enhanced transfer learning capabilities. The reviews for this paper are excellent. Any concerns reviewers expressed were thoroughly addressed through the rebuttal process. A new method that out-performs residual networks will have wide ranging impacts for the community.
Decision: Accept (oral)