Title: Auto-Compressing Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: deep learning architectures

representation learning

residual connections

continual learning
Abstract: Deep neural networks with short residual connections have demonstrated remarkable success across domains, but increasing depth often introduces computational redundancy without corresponding improvements in representation quality. We introduce Auto-Compressing Networks (ACNs), an architectural variant where additive long feedforward connections from each layer to the output replace traditional short residual connections. By analyzing the distinct dynamics induced by this modification, we reveal a unique property we coin as *auto-compression*—the ability of a network to organically compress information during training with gradient descent, through architectural design alone. Through auto-compression, information is dynamically "pushed" into early layers during training, enhancing their representational quality and revealing potential redundancy in deeper ones. We theoretically show that this property emerges from layer-wise training patterns found only in ACNs, where layers are dynamically utilized during training based on task requirements. We also find that ACNs exhibit enhanced noise robustness compared to residual networks, superior performance in low-data settings, improved transfer learning capabilities, and  mitigate catastrophic forgetting suggesting that they learn representations that generalize better despite using fewer parameters. Our results demonstrate up to 18\% reduction in catastrophic forgetting and 30-80\% architectural compression while maintaining accuracy across vision transformers, MLP-mixers, and BERT architectures. These findings establish ACNs as a practical approach to developing efficient neural architectures that automatically adapt their computational footprint to task complexity, while learning robust representations suitable for noisy real-world tasks and continual learning scenarios.


Peer reviews:
comments: SUMMARY:
This paper proposes a new type of residual connection method, which makes the final network output directly combine all the previous layers' outputs. This network has the ability called "auto-compressing", which can identify the redundancy of layers in the neural networks.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper writing is fluent and easy to read. The motivation is clear and easy to follow. The figures and tables are also clear.
2. The method is simple but effective, showing interesting "auto-compressing" properties of the network, which shows great potential in network compression.
3. The performance is good compared to the resnet counterpart.
4. The experiments are highly sufficient, showing their great potential in performance experiments, compression experiments, robust experiments, continual learning, and transfer learning.

Weaknesses:
1. Why Direct Gradient is so important in model training? It seems that its Direct Gradient makes the ACNs have the ability of the "auto-compressing". Do you have some hints of why?
2. It will be more convincing if you compare your method with some other variants of residual connection, like DenseNet.
3. Have you tried to further reduce the ACNs residual connections of some rear layers to further enhance the importance of the Direct Gradient to early layers and enhance the compression rate? Which I think can further verify your claims.

QUESTIONS:
See weakness above.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Auto-Compressing Networks (ACNs), which replace traditional short residual connections with long feedforward connections from each layer directly to the output. The authors claim this architectural change leads to \"auto-compression\"—a property where networks organically concentrate information in early layers during training, making later layers redundant. They provide theoretical analysis of gradient dynamics and demonstrate that ACNs can achieve 30-80% layer reduction while maintaining performance across vision and language tasks, with some other benefits in noise robustness, continual learning, and transfer learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n+ The gradient flow analysis (Table 1, Section 2.2) provides good intuition for why ACNs exhibit different training dynamics than ResNets, with the decomposition into Direct Gradient and Network-mediated Gradient components being interesting to read about.\n+ The core auto-compression is demonstrated consistently across multiple architectures (ViT, MLP-Mixer, BERT) and datasets, showing the effect is not architecture-specific.\n+ It's nice to see the authors also evaluate noise robustness, continual learning, and transfer learning, providing a broader picture of ACN behavior.\n+ The architectural change is minimal and hardware-friendly, requiring no specialised software or complex hyperparameter tuning.\n\n Weaknesses\n\n- The core contribution—networks with multiple prediction points that can exit early—has been extensively studied in the early exit literature (BranchyNet 2017, MSDNet 2017, EENet 2023). The paper fails to adequately compare with or even cite many key works in this space. MSDNet for example, already achieves \"6% higher accuracy with 2-3× less computation time\" on ImageNet\n- Most experiments are on small-scale tasks (CIFAR-10/100, ImageNet with 256 batch size). For an architecture claiming broad applicability, this is insufficient. Early exit methods have been demonstrated at full ImageNet scale with superior results.\n- ACNs seem to require significantly more training time (AC-ViT: 700 vs 300 epochs for ResNet). This negates claimed efficiency benefits and suggests the method may not scale to larger models where training cost matters more - unless I'm missing something?\n- The paper claims \"groundbreaking\" results but often just matches baseline performance. Early exit methods like BranchyNet show 2-6× speedups, while ACN improvements are much more modest.\n- No direct comparison to established early exit methods (MSDNet, BranchyNet, EENet) on the same tasks. This is a significant oversight given the clear conceptual overlap.\n- Different convergence times between ACN and baseline variants make fair comparison difficult.\n- Although it's great to see some transfer learning eval comparisons -they are pretty narrow with only CIFAR-100→CIFAR-10 transfer, which are very similar datasets.\n\nQUESTIONS:\n1. can you give some head-to-head comparisons with MSDNet, BranchyNet, and EENet on identical datasets? Given the overlap, direct comparisons seem essential.\n2. Can you show ACN results on full ImageNet and larger language models? The current small-scale experiments limit the generalisability claims.\n3. From what I understand, training ACNs takes roughly twice as long as standard networks. When you factor in both the training overhead and inference costs, how do ACNs compare overall? I'm curious about the total computational budget, not just the inference savings - granted this is a one time cost.\n4. There's already quite a bit of research on early exit mechanisms in neural networks. Beyond the specific way ACNs are architected, what makes them fundamentally different from existing early exit approaches? I'd like to understand the unique contribution here.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed method, Auto-Compressing Networks, introduces a new architectural variant for neural networks where long feed-forward connections from each layer to the output replace shorter residual connections. This architectural modification is shown to organically compress information during training, with early layers capable of highlighting features that are sufficient for high performance, as well as number of other benefits like enhanced robustness to noise, superior performance in low-data settings, improved transfer learning capabilities and lower catastrophic forgetting.\n\nSTRENGTHS AND WEAKNESSES:\nQuality\n\nThe technical content and claims made in the paper are straightforward and well supported by experimental evidence. For individual comparisons, while the baselines are relevant they are not the current SoTA in all cases, slightly weakening the impact of the proposed model. Providing a more rounded discussion of the increase in size and training time and scenarios where ACNs might have drawbacks would serve to make this a very well rounded paper.\n\nClarity\n\nThe manuscript is extremely well written and clear, with sections dedicated to specific claims and supporting experimental results.\n\nSignificance\n\nWith the potential inclusion of FLOPs + Parameter counts and training time comparisons, the simplicity and implicit training dynamics  of ACNs could be extremely beneficial to the community at large, with the added bonus of offering larger compression and real-time deployment options.\n\nOriginality\n\nThe discussion about connection points with existing architectural variants and how ACN-specific changes affect training underline relevant hypotheses, novelty and how ACNs distinguish themselves.\n\nQUESTIONS:\n- Could the authors provide an in-depth comparison of the parameter count + FLOPs for the example of a single block (within any of the architectures used in the experiments) in addition to the overall training time required (for a chosen experiment)? A comprehensive comparison could help identify scenarios where the selection of the proposed architecture and its improved performance outweighs the cost.\n- Could the authors provide notations on the graph of Table 1 that correspond to the notations used for forward and backward propagation?\nThis would help the reader quickly parse the table's information.\n- Could the authors discuss the impact of hyper-parameter optimization on the experiments conducted in the paper (a subset would suffice)? The intention is to isolate the cause for improved performance and ensure the main reason is the change in architecture as opposed to sub-optimal hyper-parameter choice.\n- I would recommend the authors provide the definition of \"incremental layer performance\" early on in the manuscript to ensure correct understanding of Fig. 1.\n- Could the authors clarify how the gradient norm is normalized in Fig. 1?\n- Could the authors double check MLM-Mixer in Fig. 2's caption? In addition, could the authors also double check the last 2 sentences in Section 3 (L. 171-176) and confirm whether the statements are correct?\n- In Section 4.1, the manuscript mentions \"ACNs can improve inference time and memory consumption without sacrificing performance\". Are these exclusively measured using the layer vs. accuracy graphs? If so, from a practical standpoint, could the authors provide a params + FLOPs reduction count for the optimal layer setting as well? (Is this layer setting sufficient across all experiments?)\n- Could the authors also discuss the potential of over-fitting when compared to Residual networks? (POV: Longer training time and smaller-scale datasets)\n- Could the authors also provide a comparison where the residual architecture is trained for number of epochs = convergence epochs of ACN variant? In addition, could you comment on a potential causes + solutions to speed up convergence?\n- Interestingly, in low-data settings, the convergence pattern is reversed. Could the authors discuss potential reasons for this?\n- Due to the nature of the architectural variation, could the authors discuss and compare against early exit networks? (training strategy, stability, performance ?)\n- To develop an even stronger appreciation for the changes affected by ACNs, could the authors discuss, using retrieval metrics or other similarity metrics, the feature representations learned by ACNs and Residual-variants?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new network architecture that includes long skip connections from each layer to the output. The authors demonstrate that this design outperforms traditional skip-connection networks across diverse modalities and training settings, including supervised learning, low-data regimes, and task-incremental learning.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* Well-designed experiments are provided, offering detailed analysis that directly supports the authors' claims.\n* Experimentally, the proposed method outperforms residual networks in reducing forgetting, improving generalization, increasing robustness to input noise, and achieving better capacity utilization.\n\n### Weaknesses\n\n* The equations and figures in Table 1 appear to be incorrect. For example, in the ResNet architecture, there should not be a connection between the first and last layers. Additionally, activation functions are missing from the equations. In the case of FFN, the final output $y_F$ cannot be obtained as currently formulated unless only linear activations are assumed. This omission likely affects all derivations in Appendix C.\n* In most experiments, the proposed method is compared only against ResNet. Ideally, additional baselines—such as those mentioned in the related work section—should be included for a more comprehensive evaluation.\n* Table 4 lacks error bars, which limits the interpretability and reliability of the reported results.\n\nI am open to increasing the score once these limitations are addressed. If corrected, this paper has the potential to make a meaningful contribution to the community.\n\nQUESTIONS:\n* Why do ACNs show better generalization? Are there any theoretical explanations or underlying intuitions that support this behavior?\n* How many runs were performed for each experiment? Please report mean and variance if applicable.\n* Line 183: \"AC-ViT converges at 700 epochs, while the Residual ViT converges at 300 epochs.\" Does this imply that Residual ViT converges faster than AC-ViT? If so, can you clarify the implications for training efficiency? Please include learning curves if possible. Additionally, do all residual networks converge faster than ACNs across tasks?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}