Hyper-Connections

Authors: Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, Xun Zhou
Primary area: foundation or frontier models, including LLMs
Keywords: Network Architecture, Residual Connections, LLMs, Pre-training

Abstract:
We present hyper-connections, a simple yet effective method that can serve as an alternative to residual connections. This approach specifically addresses common drawbacks observed in residual connection variants, such as the seesaw effect between gradient vanishing and representation collapse. Theoretically, hyper-connections allow the network to adjust the strength of connections between features at different depths and dynamically rearrange layers. We conduct experiments focusing on the pre-training of large language models, including dense and sparse models, where hyper-connections show significant performance improvements over residual connections. Additional experiments conducted on vision tasks also demonstrate similar improvements. We anticipate that this method will be broadly applicable and beneficial across a wide range of AI problems.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1730597178126

thread_type:
official_review

<thread_object>
creator: reviewer_fvae

<content>
summary:
The paper introduces hyper-connections, a approach that aims to address the limitations of residual connections of transformers architectures.
The hyper-connections approach introduce depth-connections and width-connections, allowing a more customizable interaction between layers. Depth-connections are weighted connections across different layers, while width-connections facilitate two layers at the same depths. More importantly the hyper-connections are learned, so it improves model adaptability.
Models with hyper-connections have significant performance improvements over the original architectures.

strengths:
The experimental results seems good enough to claim the hyper-connections is better than the original architectures.

weaknesses:
1. The presentation is horrible, it is really hard to understand what is model is doing. Probably a better figure or even pseudocode should be provided to explain the methods.
2. Although the experimental results are good, there is no information on how much extra computation is need to achieve such good results. I suspected that the extra performance is highly due to the extra parameters or extra computation it needed. If you make your model have the same floats, probably you could see that the performance is really similar to the original transformers.
3. The ordering of the paper is horrible, there is almost no explanation why you do each thing in the paper.

questions:
1. In the introduction, you stated that "Post-Norm applies normalization operations after the output of each residual block, weakening the
"strength" of residuals." I don't think it is correct, since post-norm applies normalization after the summation of the skip and residual branch, so it shouldn't weaken the strength of the residuals.

2. Can you also should the flops count for your model and the original model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645844177

thread_type:
official_review

<thread_object>
creator: reviewer_vjdj

<content>
summary:
The paper introduced hyperconnection - width and depth connections with learnable strengths. Hyperconnections are proposed to find a balance between the seesaw effect noticed between vanishing gradients and representation collapse. The input is split into n copies for different connections, and these different copies are added together before layer computation.  The paper also introduced dynamic hyperconnections that dynamically update the strength of connections based on inputs. The paper performs comprehensive analysis of effect of hyperconnections on OLMO and OLMOE and image generation and classifcation problems and investigates its effect with prominent residual connections.

strengths:
1. The results on LLM benchmarks and losses suggest a better balance between vanishing gradients and representation collapse.

2. Section 4.5 discusses the effect of hyperconnections, which displays that hyperconnections eliminate input embeddings from the output, form parallel blocks which have less reliance on each other increasing chances for unique representations. 

3.Parallel block formation is particularly important as similar layers in a transformer block tends to learn similar representations. The ability to parallely form blocks dynamically based on input reduces the chances for similar representations enabling better models.

4. Assessing the hyperconnections can reveal some internal logic of the neural network. For instance, we may find that a set of classes follow similar paths compared to other classes.

5. The paper is well detailed and mathematically sound, proofs, hyperparameters and other implementation details are discussed appropriately.

weaknesses:
1. The main drawback is when creating $n$ copies,  it leads to a considerable amount of increase in memory, though the burden can be reduced through engineering, the impact is yet to be known.

2. If the goal of creating multiple copies is just to make sure multiple depth connections can be modelled parallelly, is creating such copies actually necessary? Can't a single copy be used with different residual strengths? The only difference would be in gradient computation , were additional terms for each depth connection would be added to the singular copy. Can an ablation be conducted between DHC ($n=4$ and $n=1$) and SHC ($n=4$ an $n=1$) to show the importance and need for additional copies. What I am particularly interested is the advantages with updating the gradients into different copies and then adding them before every layer computation verses  updating all the gradients with a single copy. If there is any other specific need for expansion please free to refute this point.

3. Figure 2 caption improvement. The figure can discuss the diagram better, whats $\beta$, and need to add $\alpha_{1,0}$ and $\alpha_{2,0}$ be explained in the diagram itself. This is important as Figure 2 is the central figure that tries to encompass hyperconnections therefore adding these information would make it more clearer for readers.

questions:
Some questions that may add more value to the paper:
1. With Dynamic Hyperconnections, there is a possibility for redundant connections (when strength becomes 0) how does these cases affect the seesaw effect of gradient vanishing and representation collapse.
2. Do DHC connections behave similar for images belonging to the same class in case of image classification?

Things I expect from the rebuttal are actions on points 2 and 3.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666264904

thread_type:
official_review

<thread_object>
creator: reviewer_dubh

<content>
summary:
This paper presents hyper-connexions, a new neural network architectural improvement which consists in dynamically adjusting the residual connections between layers, effectively managing the trade-off between vanishing gradients and feature collapse. Many experiments with LLMs and vision models demonstrate the effectiveness of hyper-connections to improve stability of training and downstream performance. Various hyper-connections patterns are also studied in depth, with thorough ablations and visualizations.

strengths:
- There is a clear signal that incorporating hyper-connections in LLMs architectures, without any other modification, improves the training loss for a given number of tokens, and boosts performance on downstream metrics. This result is validated for both dense and MOE architectures.

- Hyper-connexions help reduce training instabilities. This is clear by looking at Figure 5 and 6, the training curves of the models with hyper-connections are smoother and do not have spikes, which is a major advantage for training large models.

- The author did a thorough analysis of the learned connections patterns, with nice visualizations.

- The results generalizes to the vision modalities with experiments on image generation and classification. Hyper-connections seem to be a general improvement for the transformer architecture.

weaknesses:
- The main concern I have with this paper is the computational impact of replicating the activations of the network $n$ times for hyper-connections. There is no study on the computational impact both in terms of running time and memory usage. The authors mention Line 394 that “Both methods expand the hidden size by n times with negligible computational overhead” but it is not shown with a proper experiment on the throughput, overall running time, and peak memory usage. Also, it seems that n=1 performs worse than no hyper-connection, so if n>=2 is necessary, and the memory usage is high, it is necessary to study the trade-off between downstream performance, stability and computational cost.

- Although the signal is promising, a full experiment with scaling the number of tokens beyond 500B will be necessary to fully validate the approach. Not asking for this experiment, but current best LLMs are trained on many more tokens and exhibit much better performance than the number reported. I would be curious to see if hyper-connections are useful for training state-of-the-art LLMs.

- In section 3.2 several non-learnable patterns are presented but are not tried in practice. It is not clear whether learning the hyper-connection patterns is really better that having simple fixed patterns, and an analysis on that would be interesting.

questions:
- Why is expansion rate = 1 worse than no hyper-connection, do you have an intuition ?

- Do these findings generalize to other types of architectures such as ResNets ?

- Line 345 typo: “Static”
</content>
</thread_object>

</thread>
<thread>
creation date:
1731281922923

thread_type:
official_review

<thread_object>
creator: reviewer_hhyn

<content>
summary:
The paper introduces Hyper-Connections, a novel extension to residual connections that dynamically adjusts the strength of connections between layers in deep neural networks. This method addresses the limitations of traditional residual connections, such as gradient vanishing and representation collapse, by introducing Depth-Connections and Width-Connections thus enabling both cross-layer and intra-layer interactions. A dynamic variant of residual connection named Dynamic Hyper-Connections (DHC), further adapts connection strengths based on input. The approach is evaluated extensively across pretraining of large language models (LLMs), Mixture-of-Experts (MoE) models on vision tasks. Experimental results demonstrate significant improvements in training stability, convergence speed, and model performance on various benchmarks, highlighting Hyper-Connections as a general-purpose enhancement for neural architectures.

strengths:
- The paper provides a clear and systematic extension to residual connections named Dynamic Hyper-Connections (DHC), where residual could be consider a static hyperconnection, addressing the trade-off between gradient vanishing and representation collapse.
- Experimenttal results demonstrated effectiveness of DHC across diverse domains, including LLM pretraining and vision tasks.

weaknesses:
- There seem a lack of comparsion to a fullt enabled depth-connections and width-connections (DenseNet style) where all of the connection in Figure 2 are enabled and learnable.
- The main result focus on LLM and downstream peformance on langugage tasks, the result on Vision task in the appendix seem to demonstrate less gain compare to langugage tasks, can the author elebrate more on this?

questions:
- The author mentioned other baselines such as Altup and ResiDual had gains in the early stages of training, Can the author show the full loss curve of OLMo-1B-ResiDual and OLMo-1B-Altup×2 in Table4?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732193819358

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We appreciate the reviewers for dedicating their time to thoroughly evaluate our manuscript and offering constructive feedback. We are gratified to see the universal acknowledgment of the significance of our work. Incorporating your recommendations, we have meticulously revised the document, with all modifications denoted in red within the updated version. The key changes are as follows:
1. Update Description (in response to reviewers `VJDJ`, `DUBh`) . **We revised the introduction, redrew Figure 2, updated its caption, and added the complete network architecture in Appendix A.** This provides a better introduction to our method.
2. **Efficiency Analysis** (in response to reviewers `fvae`, `VJDJ`, `DUBh`). We added the parameter and computational costs to the 7B experiment table and provided the parameter and computational costs for all experiments in Appendix B. **This clearly demonstrates that the parameter, computational overhead and memory footprint introduced by our method is negligible.**
3. Update Appendix L to encompass the loss curves for altup and ResiDual (in response to reviewer `HHYn`). 
4. Update Appendix L to address the concern regarding scalability for results beyond 500B tokens (in response to reviewer `DUBh`). For some 1B models, we extended the training trajectory to even 1T tokens, where the gains from hyper-connections were consistently maintained. Furthermore, based on the magnitude of the reduction in training/validation loss and our prior experience in training production-level LLMs, we are confident in the efficacy of this approach, even for model scales with hundreds of billions of parameters.
5. Update Appendix E to include the training curves of ViT (in response to reviewer `HHYn`),  which explains why the gains in vision tasks are not as significant as those in LLMs.
6. Update Appendix F to explain why the performance is suboptimal when n=1 (in response to reviewers  `VJDJ`, `DUBh`). 

We believe that these revisions have significantly strengthened the quality and impact of our work, and we hope that the reviewers will find the manuscript now suitable for publication. We welcome any additional feedback or clarification that the reviewers may have, and we remain committed to addressing their concerns to the best of our ability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733055530695

thread_type:
public_comment

<thread_object>
creator: ~harvie_zhang1

<content>
title:
Questions for Hyper-Connections

comment:
Dear Authors,

I have a few questions regarding your proposed Hyper-Connections.

1. **Motivation of the Paper:** The primary goal of residual connections is to minimize information loss at each layer, as referenced in  https://arxiv.org/pdf/2401.17948.pdf. Your proposed framework is similar in that it increases the model's width by copying the hidden features $n$ times.

2. Hyper-connections effectively transform the input using a small number of training parameters. This aligns with the concept of existing Hyper Interaction, as introduced on page 6 of the above paper.

3. The results presented in the paper indicate that only DHC-based models are effective, and their performance is solely enhanced by increasing the model's width. Please refer to my first question.

I look forward to your response if there are any misunderstandings.

Best regards,

Harvie
</content>
</thread_object>

</thread>
<thread>
creation date:
1733299747754

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Author General Response

comment:
We thank all reviewers for their thoughtful feedback and constructive suggestions, which have greatly helped us improve the quality of this work. Below, we summarize the main contributions of our paper, highlight the points of agreement from reviewers, and address key questions and concerns raised during the review process.

---

### **Summary of Contributions**

This paper introduces **Hyper-Connections**, a novel and simple alternative to residual connections that addresses fundamental issues such as **gradient vanishing** and **representation collapse**. The key idea is to dynamically adjust the strength of connections between features at different depths and enable layer rearrangements, improving overall gradient flow and feature representation. 

Hyper-Connections achieve notable performance improvements in **LLM pretraining (7B models)**, for both dense and sparse models, as well as in **vision tasks** (including image classification and generation). For the **7B MoE model**, Hyper-Connections achieve a remarkable **1.8x** convergence speedup, and the performance gain remains consistent throughout training. Despite their flexibility, Hyper-Connections introduce negligible computational overhead compared to standard residual connections.

---

### **Points of Agreement from Reviewers**

We are grateful that reviewers acknowledged the following strengths of our work:  

1. **Empirical Performance, Generalization, and Stability (`HHYn`, `DUBh`, `VJDJ`, `fvae`)**:      
Reviewers widely acknowledged the strong empirical results, particularly the significant improvements in large language model pretraining. Hyper-Connections also demonstrated robust generalization to vision tasks, with experiments on image generation and classification (e.g., ImageNet) showing consistent performance gains. Additionally, Hyper-Connections help reduce training instabilities, as shown in **Figure 5** and **Figure 6**, where the training curves are smoother and free of spikes—an important advantage for training large models.




2. **Visualization and Analysis (`DUBh`, `VJDJ`)**:      
Reviewers appreciated the detailed analysis of the learned connection patterns, supported by insightful visualizations. These visualizations reveal the internal logic of the neural network, such as how certain classes follow similar paths compared to others. This provides a deeper understanding of how Hyper-Connections function and their role in addressing issues like gradient vanishing and representation collapse.  




---

### **Addressed Reviewer Concerns**

We carefully addressed the primary concerns raised by reviewers, as summarized below:


1. **Parameters, Computational Costs, and Memory Footprint (`DUBh`, `VJDJ`, `fvae`)**:      Concerns were raised regarding whether the flexibility of Hyper-Connections incurs significant **parameters**, **computational costs**, and **memory footprint**. To address this, we supplemented specific numbers for each in our response. The analysis demonstrated that the additional parameters and computational costs are negligible, while the increase in memory footprint is well within a reasonable range. Furthermore, we emphasized that the memory footprint can be further optimized using recomputation techniques. Reviewers appreciated this detailed clarification and acknowledged the efficiency of the proposed method.

2. **Generalization to Vision Tasks (`HHYn`)**:  
   One of reviewers questioned the result on Vision task in the appendix seem to demonstrate less gain compare to langugage tasks. We clarified that Hyper-Connections achieve significant performance gains in vision tasks as well. While the gains are relatively smaller compared to LLMs, they remain notable, as shown in **Table 7** and **Table 8**. This highlights the versatility of Hyper-Connections across domains. Furthermore, we offered an intuitive explanation for why the gains in vision tasks are less pronounced, pointing to differences in dataset size and reasoning requirements (e.g., [Physics of Language Models: Part 2.1](https://arxiv.org/abs/2407.20311)).  
   
3. **Conceptual Clarity**:
Reviewers' interpretations of certain concepts in the paper differed slightly from our intended explanation. To clarify these points, we refined our descriptions in the introduction and figure 2 with red highlights to better explain these concepts. These updates provided additional clarity and helped align our presentation more closely with the reviewers' perspectives.
</content>
</thread_object>

</thread>
<thread>
creation date:
1739091919393

thread_type:
public_comment

<thread_object>
creator: ~harvie_zhang1

<content>
title:
Response to authors & Public comments

comment:
Thanks to the authors for their previous responses. Since I didn't receive any reminders, I am now adding more detailed comments.

First, regarding your question about the similarity between my work and GoogleNet (https://arxiv.org/pdf/1409.4842) and CBAM (https://arxiv.org/pdf/1807.06521), I cited and discussed them in my paper. I would also like to clarify that while our approaches are not exactly the same, they share the same underlying principle, although you present it in a different context.

Next, I will outline the similarities between your method and several previous works. **Additionally, you did not discuss the related works mentioned by the Area Chair (FractalNet and DenseNet) in the latest version.**

1. **The use of "Hyper" in neural networks.** As we all know, Hypernetworks (https://arxiv.org/pdf/1609.09106) utilize implicit small networks to generate the weights of the main network, whereas your work employs layer connections as learnable parameters in implicit matrices.

2. **Implicit connections.** DiracNet (https://arxiv.org/pdf/1706.00388) parameterizes network weights as a residual of the Dirac function to eliminate residual connections, and in my Terminator architecture (https://arxiv.org/pdf/2401.17948), this concept is extended to model outputs. Your method can be viewed as a further extension of this idea. Furthermore, DiracNet, as an important variant of residual connections, shares similarities with the initialization of implicit matrices in Hyper-Connections, but you did not discuss this.

2. **Dynamic model weights.** You emphasized the importance of dynamic connections that depend on the input (Section 2.2), which are also referred to as fast weights. Reference links: https://arxiv.org/pdf/2401.17948, https://people.idsia.ch//~juergen/fast-weight-programmer-1991-transformer.html.

3. **Single branch vs. multi-branch.** My work highlights the significance of multi-branch architectures (https://arxiv.org/pdf/2401.17948, visualization results https://github.com/hyperevolnet/Terminator/blob/main/assets/plain_resnet.png) to reduce information loss between model layers. Additionally, multi-branch networks can be traced back to LSTM, and your approach bears similarities to this.

Finally, I have some questions regarding your work.

1. **Motivation:** You use pre-norm and post-norm as motivation, asserting that residual connections cannot effectively address gradient vanishing and representation collapse. For the former, you only cited a paper discussing the existence of gradient vanishing in RNNs (https://ieeexplore.ieee.org/document/279181), while transformers differ in this regard. Moreover, the performance degradation mentioned in the ResNet paper (https://arxiv.org/pdf/1512.03385) is not equivalent to representation collapse (see the third point).

2. **Experiments:** You claim that the proposed Hyper-connections can replace residual connections, but you **only** provide results based on the transformer architecture.

3. **Representation collapse and visualization:** Its definition can be found at https://arxiv.org/pdf/2411.02344, https://arxiv.org/pdf/2406.04267 and https://arxiv.org/pdf/2206.04041, which raises concerns about your visualization result (Fig. 3) and its conclusion.


4. **Ablation study and theoretical analysis:** You did not provide any visualization results or formula derivations demonstrating that Hyper-Connections can alleviate gradient vanishing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1740913256445

thread_type:
public_comment

<thread_object>
creator: ~harvie_zhang1

<content>
title:
Academic Misconduct and Plagiarism

comment:
First, the authors still did not make improvements based on the meta-review in the camera-ready version. In addition, the multiple concepts mentioned in your paper did not discuss the inspiration from which work, which is **academic misconduct**.

**Plagiarism**: 

1) The first author claims that the Hyper-Connections work took a year, but someone told me that it was not. This false claim is only intended to show that they started before my work was published. Can the authors provide a timeline for the Hyper-Connections proposal? 

2) Why did the authors choose evolutionary algorithms as a subject in the arXiv? Your paper does not mention any discussion of this. In contrast, our HyperEvol AI Lab name does mention it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1752504512744

thread_type:
public_comment

<thread_object>
creator: ~harvie_zhang1

<content>
title:
Evidence of Plagiarism & Academic Misconduct & Factual Errors & Structural Incoherence

comment:
Plagiarism evidence document can be found below:

https://github.com/plagiarism-ai-papers/Plagiarism-hyper-connections/blob/main/Plagiarism%20Hyper-Connections.pdf
</content>
</thread_object>

</thread>
<thread>
creation date:
1770658419308

thread_type:
public_comment

<thread_object>
creator: ~harvie_zhang1

<content>
title:
Plagiarism HC authors’ WRONG claim and dishonest behavior

comment:
1. Authors’ wrong rebuttal:

 “Similar experiments and conclusions can be found in Table 1 of https://arxiv.org/pdf/1603.05027: namely, that a shortcut in the **residual connection with weights predicted by a network does not perform better than the standard residual connection.**”

Refer to Weighted Skip Connections are Not Harmful for Deep Nets (https://rupeshks.cc/blog/skip.html#:~:text=The%20paper%20Identity%20Mappings%20in,networks%20with%20gated%20skip%20connections.)

2. Dishonest behavior: 

“In fact, in our subsequent research, we have found that it is possible to achieve gains without using n copies, although the gains are smaller. **The core idea here is not to copy n times but to split the hidden state into n parts. We will disclose the relevant results of this research in our future work.**”

The above response shows that the author had developed Frac-connections (https://arxiv.org/pdf/2503.14125) before the rebuttal. However, in order to package the paper, they **deliberately** did not discuss the previous works related to multi-branch networks (inception models and FracNet). 

*Three may keep a secret if two of them are dead.*
</content>
</thread_object>

</thread>
<thread>
creation date:
1772182350454

thread_type:
public_comment

<thread_object>
creator: ~harvie_zhang1

<content>
title:
New evidence of academic misconduct

comment:
All researchers in the AI ​​field know who **first** discovered the vanishing gradient problem. 
However, the authors of Plagiarism-HC did not cite it. Instead, they only cited Bengio et al., 1994 and Glorot & Bengio, 2010.

Furthermore, they deliberately ignored the Area Chair's suggestion not to cite Highway Network and FracNet. 
**Why did they do this?**

The plagiarism and academic misconduct are quite clear!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The proposed hyper-connections (HCs) are a form of learned architecture that can be optimized to connect different representations across depth and width by summation. In this way they are an extension of residual connections. In addition, HCs can be conditioned on the input to vary these connections during inference for dynamic hyper-connections (DHCs). The main paper concerns the applications of HCs to LLMs, working with open-source OLMoE models, and shows improvements to the loss, data efficiency, and computationally efficiency relative to other recent approaches to expand architectures. In the appendix additional results on vision are shown, where there is still gain, but the effect is more marginal.

Strengths:

- experiments show empirical improvement for pre-training and downstream tasks on text and visual data
- the cost in computation time, memory, and parameters is measured and reasonable (following the rebuttal and revision)
- the idea and the implementation are clear (and have been clarified in the rebuttal phase)

Weaknesses:

- The related work is limited and the scholarship is shallow in its inclusion of only the most recent papers. As a project on connecting across depth and width, there is a well-developed body of work including DenseNets, most practically, and other models such as FractalNets and HighwayNets.
- The work claims similar improvement across text and vision, but more precisely there is a larger improvement for LLMs than current vision models. This is tempered by equally good or still improved results on vision however.
- The organization of the presentation of the paper challenged comprehension, as evidenced by shared questions across reviewers and multiple comments on these points. 

Missing: Most critically, the submission is missing discussion (and ideally experiments) for certain related works like DenseNet and it is inadequately organized for ease of comprehension. The issues with exposition however have begun to be addressed in the rebuttal and revision, and could be dealt with in the final revision. Additional discussion is likewise feasible, and would be an acceptable resolution of the related work.

Decision: This work is borderline. The four expert reviewers vote for acceptance (VJDJ: 8, HHYn: 6, DUBh: 6, ) and rejection (fvae: 5). The meta-reviewer sides with acceptance because of the strength of the empirical results for task performance and computation, the satisfaction of the majority of the reviewers and the points addressed in the rebuttal, and the generality of the results across multiple tasks, datasets, and modalities. However, the meta-reviewer cautions that the lack of discussion and experimentation on related work about more sophisticated handling of depth (DenseNets, FractalNets, ...) remains a negative. For broader impact and improved reception of the proposed technique it may be advisable to incorporate more material about these prior works, as has likewise been suggested by the reviewers.

Note: The meta-reviewer acknowledges the confidential comment from the authors. The decision reflects the paper, reviews, rebuttal, and the full discussion.