Hyper-Connections

## Abstract

Abstract We present hyper-connections, a simple yet effective method that can serve as an alternative to residual connections. This approach specifically addresses common drawbacks observed in residual connection variants, such as the seesaw effect between gradient vanishing and representation collapse. Theoretically, hyper-connections allow the network to adjust the strength of connections between features at different depths and dynamically rearrange layers. We conduct experiments focusing on the pre-training of large language models, including dense and sparse models, where hyper-connections show significant performance improvements over residual connections. Additional experiments conducted on vision tasks also demonstrate similar improvements. We anticipate that this method will be broadly applicable and beneficial across a wide range of AI problems.

## 1 Introduction

Figure: Figure 1: The performance of the baseline model OLMoE-1B-7B and the model with hyper-connections, OLMoE-1B-7B-DHC$\times$4. (1) and (2) show the training loss (0.99 EMA smoothed) and the C4-en validation loss, respectively. Our method converges 1.8 times faster compared to the baseline and maintains a significant advantage at the 500B tokens. (3) and (4) show the accuracy curves on HellaSwag and ARC-Challenge, demonstrating the superior performance of the OLMoE-1B-7B-DHC$\times$4 model.
Refer to caption: x1.png

Deep learning has achieved tremendous success across various domains, where residual connections (He et al., ) have been instrumental in contemporary neural network architectures, including transformers and CNNs. Residual connections help mitigate the problem of gradient vanishing, enabling the effective training of very deep networks. However, it is important to acknowledge that residual connections are not infallible solutions and still present limitations that remain unresolved.

The two main variants of residual connections, Pre-Norm and Post-Norm, each make distinct trade-offs between gradient vanishing and representation collapse. Pre-Norm applies normalization operations to the input before each residual block, effectively addressing the problem of gradient vanishing (Bengio et al., ; Glorot & Bengio, ). However, it can also lead to the issue of collapse in deep representations (Liu et al., ), where hidden features in deeper layers become highly similar, diminishing the contribution of additional layers as their number increases.
In contrast, Post-Norm applies normalization operations after the output of each residual block, weakening the "strength" of residuals. This approach can alleviate the issue of representation collapse but also reintroduces the problem of vanishing gradients. The vanishing gradient and the representation collapse are like two ends of a seesaw, with these two variants making respective trade-offs between these issues. The key issue is that residual connections, including both Pre-Norm and Post-Norm variants, predefine the strength of connections between the output and input within a layer.

Figure: Figure 2: Hyper-connections with expansion rate $n=2$. (a) The highlighted green and blue connections compose depth-connections between the output of the layer and the blue hidden vector. (b) The highlighted blue and yellow connections represent a part of width-connections between the hidden vectors $h_{1}$ (blue) and $h_{2}$ (yellow). (c) The matrix representation of hyper-connections.
Refer to caption: x5.png

Figure: Figure 3: Cosine similarity between the input of the current layer and the previous layers for the OLMo-1B Model (Groeneveld et al., ). The curve represents the median of similarity, while the shaded area indicates the range between the 5th and 95th percentiles. The red curve shows the model with Pre-Norm, and the blue curve shows that with hyper-connections.

Driven by the limitations of residual connections, an important question arises: Can neural networks autonomously learn the optimal strength of connections to improve performance? To address this, we propose hyper-connections (HC), which lead to significantly improved performance with a negligible increase in computation and parameters. We will show that both Post-Norm and Pre-Norm variants can be expressed as specific non-trainable forms of hyper-connections, as discussed in § 3.1.

The core idea of hyper-connections is to propose learnable depth-connections and width-connections, as depicted in Fig. 2. Depth-connections can be considered as a generalized form of residual connections, assigning weights to the connections between the inputs and outputs of each layer. To enable the network to model different depth-connections simultaneously, we expand the network’s input into $n$ copies, each having its own depth connection, as shown in Fig. 2 (a).
Moreover, we establish width connections between the $n$ hidden vectors, allowing information exchange between hidden vectors within the same layer, as shown in Fig. 2 (b). Finally, we can formalize the hyper-connections into a matrix, as shown in Fig. 2 (c).
We find that hyper-connections can not only learn to adjust the strength of residuals but also learn to rearrange the layers (sequential or parallel), as discussed in § 3.2.
Furthermore, we introduce dynamic hyper-connections (DHC), enabling the network to adjust connection weights according to the input.
Notably, although hyper-connections seem to increase the network’s width by $n$ times, the additional parameters and computational cost are almost negligible.

Our research, primarily centered on large language models (LLMs) pre-training, also extends to visual generation and classification tasks. Using Pre-Norm as a baseline, we demonstrate the significant benefits of hyper-connections, including their effectiveness in 1B and 7B dense models as well as MoE models with 7B parameters, as detailed in § 5. The benefits are particularly prominent for OLMoE (Muennighoff et al., ) as presented in Fig.1. The model utilizing DHC converges 1.8 times faster and shows an improvement of approximately 6 points on ARC-Challenge compared to the baseline trained with 500 B tokens. According to our visual analysis, as shown in Fig.3, the baseline model tends toward representation collapse, characterized by high similarity between features of adjacent layers. In contrast, models with hyper-connections exhibit significantly lower similarity between features across adjacent layers and a wider range of similarities. This suggests that hyper-connections enhance the impact of each layer. Further discussion is provided in §4.5 and in Appendix D.
These compelling pieces of evidence demonstrate the generality of the hyper-connections principle, and we anticipate their applicability in numerous other AI challenges.

## 2 Method

### 2.1 Static Hyper-Connections

Consider the hidden vector $\mathbf{h}^{k-1}\in\mathbb{R}^{d}$ (or $\mathbf{h}^{k-1}\in\mathbb{R}^{d\times 1}$) as the input to the $k$-th layer, with the initial input $\mathbf{h}^{0}$ to the network. Initially, $\mathbf{h}^{0}\in\mathbb{R}^{d}$ is replicated $n$ times to form the initial hyper hidden matrix
$\mathbf{H}^{0}=\begin{pmatrix}\mathbf{h}^{0}&\mathbf{h}^{0}&\dots&\mathbf{h}^{0}\end{pmatrix}^{\intercal}\in\mathbb{R}^{n\times d}$. Here,
$n$ is called the expansion rate.
For the $k$-th layer, the input consists of the hyper hidden matrix from the previous layer $\mathbf{H}^{k-1}=\begin{pmatrix}\mathbf{h}^{k-1}_{1}&\mathbf{h}^{k-1}_{2}&\dots&\mathbf{h}^{k-1}_{n}\end{pmatrix}^{\intercal}\in\mathbb{R}^{n\times d}$. Finally, we sum the last hyper hidden matrix row-wise to obtain the required hidden vector, which is then passed through a final projector to produce the final output of the network (i.e., a normalization layer and an unembedding layer in transformers). To simplify the notation in subsequent analysis, we omit the layer index and simply denote the hyper-hidden matrix as $\mathbf{H}=\begin{pmatrix}\mathbf{h}_{1}&\mathbf{h}_{2}&\dots&\mathbf{h}_{n}\end{pmatrix}^{\intercal}$.

The hyper-connections can be represented by a matrix $\mathcal{HC}$, where each element defines the connection weight. The matrix is structured as follows:

$$ $\mathcal{HC}=\begin{pmatrix}\mathbf{0}_{1\times 1}&\mathbf{B}\\ \mathbf{A_{m}}&\mathbf{A_{r}}\end{pmatrix}=\begin{pmatrix}0&\beta_{1}&\beta_{2}&\cdots&\beta_{n}\\ \alpha_{1,0}&\alpha_{1,1}&\alpha_{1,2}&\cdots&\alpha_{1,n}\\ \alpha_{2,0}&\alpha_{2,1}&\alpha_{2,2}&\cdots&\alpha_{2,n}\\ \vdots&\vdots&\vdots&\ddots&\vdots\\ \alpha_{n,0}&\alpha_{n,1}&\alpha_{n,2}&\cdots&\alpha_{n,n}\end{pmatrix}\in\mathbb{R}^{(n+1)\times(n+1)}.$ (1) $$

Consider a network layer, denoted by $\mathcal{T}$, it integrates self-attention layers and feed-forward networks within transformers. The output of the hyper-connections, denoted by $\mathbf{\hat{H}}$, can be simply formulated as follows:

$$ $\mathbf{\hat{H}}=\mathcal{HC}(\mathcal{T},\mathbf{H})\\ =\mathbf{B}^{\intercal}\mathcal{T}(\mathbf{H}^{\intercal}\mathbf{A_{m}})^{\intercal}+\mathbf{A_{r}}^{\intercal}\mathbf{H}.$ (2) $$

We use $\mathbf{A_{m}}$ as weights to perform a weighted sum on the input $\mathbf{H}=\begin{pmatrix}\mathbf{h}_{1}&\mathbf{h}_{2}&\dots&\mathbf{h}_{n}\end{pmatrix}^{\intercal}$ to obtain the input $\mathbf{h}_{0}$ of the current layer $\mathcal{T}$, which is given by:

$$ $\mathbf{h}_{0}^{\intercal}=\mathbf{A_{m}}^{\intercal}\mathbf{H},$ (3) $$

While $\mathbf{A_{r}}$ is used to connect $\mathbf{H}$ and map it into a residual hyper hidden matrix $\mathbf{H^{\prime}}$. It is represented as follows:

$$ $\mathbf{H^{\prime}}=\mathbf{A_{r}}^{\intercal}\mathbf{H}.$ (4) $$

Subsequently, the output is given by:

$$ $\mathbf{\hat{H}}=\mathbf{B}^{\intercal}(\mathcal{T}\mathbf{h}_{0})^{\intercal}+\mathbf{H^{\prime}}.$ (5) $$

The depth-connections can be decoupled as the following matrix, which is shown at Fig 2 (a):

$$ $\mathcal{DC}=\begin{pmatrix}\mathbf{B}\\ \text{diag}(\mathbf{A_{r}})\end{pmatrix}=\begin{pmatrix}\beta_{1}&\beta_{2}&\cdots&\beta_{n}\\ \alpha_{1,1}&\alpha_{2,2}&\cdots&\alpha_{n,n}\end{pmatrix}\in\mathbb{R}^{2\times n},$ (6) $$

where the first row $\mathbf{B}$ represents the weights of the output of the current layer $\mathcal{T}$, and the last row $\text{diag}(\mathbf{A_{r}})$ represents the weights of the input. We use $\text{diag}(\mathbf{A_{r}})$ to represent the flatten vector of the diagonal entries of $\mathbf{A_{r}}$.

The width-connections matrix can be defined as follows, which is shown at Fig 2 (b):

$$ $\mathcal{WC}=\begin{pmatrix}\mathbf{A_{m}}&\mathbf{A_{r}}\end{pmatrix}\in\mathbb{R}^{n\times(n+1)}.$ (7) $$

The algorithm that employs hyper-connections is presented in Algorithm 1.

### 2.2 Dynamic Hyper-Connections

The entries of $\mathcal{HC}$ can dynamically depend on the input $\mathbf{H}$, which the matrix representation of dynamic hyper-connections (DHC) is defined as follows:

$$ $\mathcal{HC}(\mathbf{H})=\begin{pmatrix}\mathbf{0}_{1\times 1}&\mathcal{B}(\mathbf{H})\\ \mathcal{A}_{m}(\mathbf{H})&\mathcal{A}_{r}(\mathbf{H})\end{pmatrix}$ (8) $$

Similarly, given a layer $\mathcal{T}$ and input $\mathbf{H}$, we obtain the output of the DHC as follows:

$$ $\mathbf{\hat{H}}=\mathcal{HC}(\mathbf{H})(\mathcal{T},\mathbf{H}).$ (9) $$

In practice, we combine the dynamic and static matrices to achieve DHC. The dynamic parameters are obtained through a linear transformation. To stabilize the training process, we introduce normalization before the linear transformation and apply the tanh activation function after it, scaling it by a small initial learnable factor.
The following equations detail how these dynamic parameters are computed:

$$ $\displaystyle\overline{\mathbf{H}}$ $\displaystyle=\texttt{norm}(\mathbf{H})$ (10) $\displaystyle\mathcal{B}(\mathbf{H})$ $\displaystyle=s_{\beta}\circ\texttt{tanh}(\overline{\mathbf{H}}\mathbf{W}_{\beta})^{\intercal}+\mathbf{B}\in\mathbb{R}^{1\times n}$ (11) $\displaystyle\mathcal{A}_{m}(\mathbf{H})$ $\displaystyle=s_{\alpha}\circ\texttt{tanh}(\overline{\mathbf{H}}\mathbf{W}_{m})+\mathbf{A}_{m}\in\mathbb{R}^{n\times 1}$ (12) $\displaystyle\mathcal{A}_{r}(\mathbf{H})$ $\displaystyle=s_{\alpha}\circ\texttt{tanh}(\overline{\mathbf{H}}\mathbf{W}_{r})+\mathbf{A}_{r}\in\mathbb{R}^{n\times n}$ (13) $$

Our experiments in § 5 demonstrate that dynamic hyper-connections outperform static hyper-connections in language modeling tasks.
The PyTorch implementations for both the static and dynamic variants of hyper-connections are detailed in Algorithm 2.

### 2.3 Initialization

In order to make the initialization of the hyper-connections equivalent to the Pre-Norm residual connections, we adopt the following initialization strategy. The dynamic parameters $\mathbf{W}_{\beta}$, $\mathbf{W}_{m}$, and $\mathbf{W}_{r}$ in Eqs. 11, 12, and 13 are initialized to 0, while the static matrices are initialized as follows:

$$ $\begin{pmatrix}\mathbf{0}_{1\times 1}&\mathbf{B}^{k}\\ \mathbf{A_{m}}^{k}&\mathbf{A_{r}}^{k}\end{pmatrix}=\begin{pmatrix}\mathbf{0}_{1\times 1}&\mathbf{1}_{1\times n}\\ \mathbf{e}_{k\bmod n}&\mathbf{e}_{n\times n}\end{pmatrix},$ (14) $$

where $k$ is the index of the layer. $\bmod$ denotes the modulo operation.

## 3 Why Hyper-Connections

In this section, we elucidate the rationale behind hyper-connections. We explore how variants of residual connections, namely Pre-Norm and Post-Norm, can be viewed as non-trainable hyper-connections, and introduce the concept of sequential-parallel duality, demonstrating how hyper-connections can dynamically optimize layer arrangements to enhance network performance. A visulize analysis of hyper-connections through an unfolded view is discussed in § 4.5.

### 3.1 Residual Connections as Non-trainable Hyper-Connections

The Pre-Norm and Post-Norm residual connections can be represented as the following hyper-connections matrices with an expansion rate $n=1$:

$$ $\mathcal{HC}_{PreNorm}=\begin{pmatrix}0&1\\ 1&1\\ \end{pmatrix},$ (15) $$

$$ $\mathcal{HC}_{PostNorm}=\begin{pmatrix}0&\frac{1}{\sqrt{\sigma_{i}^{2}+\sigma_{o}^{2}+2\sigma_{io}}}\\ 1&\frac{1}{\sqrt{\sigma_{i}^{2}+\sigma_{o}^{2}+2\sigma_{io}}}\\ \end{pmatrix},$ (16) $$

where $\sigma_{i}$ and $\sigma_{o}$ denote the standard deviations of the input and output of the neural network layer, respectively, and $\sigma_{io}$ is the covariance between them.

For Pre-Norm, its hyper-connection matrix is a $2\times 2$ matrix where the bottom right triangular part is filled with $1$ and the rest is a placeholder 0. For Post-Norm, the weights depend on the variances and covariance of the input and output, forming a $2\times 2$ matrix. Therefore, their hyper-connection matrices are non-trainable. In this work, we propose hyper-connections that can be $(n+1)\times(n+1)$ matrices, with weights that are trainable or even predicted based on the input. The complete derivation is provided in Appendix E.

### 3.2 Sequential-Parallel Duality

Given a series of neural network modules, we have the option to arrange them either sequentially or in parallel. However, hyper-connections offer an approach that learns to rearrange these layers in a configuration blending both sequential and parallel arrangements.

Figure: Figure 4: Sequential and parallel arrangements of hyper-connections with $n=2$.
Refer to caption: x6.png

Without loss of generality, we set the expansion rate to $n=2$.
If the hyper-connections are learned as the following matrix, the neural network will be arranged sequentially:

$$ $\mathcal{HC}=\begin{pmatrix}0&1&1\\ 1&1&0\\ 0&0&1\end{pmatrix}.$ (17) $$

In this case, the depth connection degenerates into a residual connection, as shown in Fig. 4 (a).

When the hyper-connections for odd and even layers (with layer numbering starting from 1) are defined by the following matrices, the neural network will be arranged in parallel every two consecutive layers, similar to the arrangement of parallel transformer blocks in transformers (Wang, ), as shown in Fig. 4 (b). The general and complete derivation is provided in Appendix F.

$$ $\mathcal{HC}_{odd}=\begin{pmatrix}0&1&0\\ 1&1&1\\ 1&1&1\end{pmatrix},$ (18) $$

$$ $\mathcal{HC}_{even}=\begin{pmatrix}0&0&1\\ 0&1&0\\ 1&0&1\end{pmatrix}.$ (19) $$

Thus, learning the hyper-connection matrix in various forms can create layer arrangements that surpass traditional sequential and parallel configurations, resulting in a soft-mixture or even dynamic arrangement. For static hyper-connections, the layer arrangement within the network remains fixed after training. In contrast, dynamic hyper-connections allow the arrangement to adapt dynamically for each token.

## 4 Results

Figure: Figure 5: Comparison of training loss curves for different expansion rate. The left subfigure includes models with dynamic hyper-connections (DHC) at various expansion rates, while the right subfigure shows the effect of omitting the tanh function. Both subfigures illustrate how increasing the expansion rate leads to improved training loss performance over $500$B tokens. Results are smoothed using an exponential moving average with a coefficient of 0.99.
Refer to caption: x7.png

**Table 1: Performance comparison of different expansion rates on 500 B tokens.**
| Methods | V2 Eval<br>Loss $\downarrow$ | V2 Eval<br>PPL $\downarrow$ | V3 Eval<br>Loss $\downarrow$ | V3 Eval<br>PPL $\downarrow$ | Down Stream<br>Avg, Acc. $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| OLMo-1B | 2.811 | 18.023 | 2.544 | 14.229 | 62.5 |
| OLMo-1B-DHC$\times$1 W/O tanh | 2.822 | 18.270 | 2.556 | 14.428 | 62.3 |
| OLMo-1B-DHC$\times$2 W/O tanh | 2.792 | 17.663 | 2.537 | 14.033 | 63.8 |
| OLMo-1B-DHC$\times$4 W/O tanh | 2.779 | 17.451 | 2.516 | 13.844 | 64.4 |
| OLMo-1B-DHC$\times$8 W/O tanh | 2.777 | 17.425 | 2.514 | 13.819 | 63.8 |
| OLMo-1B-DHC$\times$1 | 2.819 | 18.125 | 2.556 | 14.418 | 62.3 |
| OLMo-1B-DHC$\times$2 | 2.802 | 17.950 | 2.534 | 14.114 | 63.0 |
| OLMo-1B-DHC$\times$4 | 2.781 | 17.509 | 2.514 | 13.826 | 63.8 |
| OLMo-1B-DHC$\times$8 | 2.778 | 17.445 | 2.516 | 13.843 | 62.8 |

We primarily conduct experiments on pre-training of large language model, including dense and
Mixture-of-Experts (MoE) (Shazeer et al., ) models, and extend to visual generation and classification tasks. Due to space constraints, we include the vision experiments in the Appendix C.

Experiment Settings. We employ the experimental setup outlined by OLMo (Groeneveld et al., ) for dense models and by OLMoE (Muennighoff et al., ) for MoE models. For dense models, we use dolmap-v1.5-sample (Soldaini et al., ) as our training dataset. We conduct ablation studies on $1$B models and assess the effectiveness of our method at the $7$B model scale. For MoE models, we train the OLMoE-1B-7B model, both with and without hyper-connections, on the OLMOE-MIX dataset. These models activate $1.3$B out of a total of $7$B parameters. All experiments are trained on $500$B tokens.

Implementation. We maintain the training configuration of the baseline model, replacing the residual connections with hyper-connections. The static component in Eqs. 1, 11, 12, 13 does not utilize weight decay, whereas the dynamic component does. Since the hyper hidden vectors of the final transformer block are ultimately summed, we ensure that the standard deviation (std) of the output (before the final layernorm and unembedding layers) remains consistent with the original. At initialization, we scale the std of the weights of the output module at all layers, including those of the second linear layer of the feedforward network and the output projector of the attention module, by a factor of $\sqrt{n}$, where $n$ represents the expansion rate.

Metrics. In accordance with the methodology of OLMo (Groeneveld et al., ), we report the average perplexities (PPL) and losses on both the V2 and V3 validation sets, along with the average metrics for zero-shot evaluation on downstream benchmarks (refer to Table 10). We observe significant volatility in the zero-shot performance indicators for the datasets (highlighted in grey in Table 10), with fluctuations exceeding 20% across neighboring checkpoints. For more reliable and consistent results, we excludes these volatile datasets from our analysis. For the MoE models, in line with OLMoE, we also present losses on V3 validation sets, and accuracies on downstream benchmarks (refer to Table 11).

### 4.1 Ablation Study

We use the dynamic hyperconnections with an expansion rate of $n=4$ and include the tanh function as the default method, marked with the suffix -DHC, while -SHC denotes static hyper-connections.

The evaluation results are presented in Table 1, and the training loss curves are depicted in Fig. 5. We observe that with an expansion rate of $n=1$, the performance of DHC is inferior to the baseline. However, for $n>1$, DHC significantly outperforms the baseline, achieving superior results at $n=4$, with the increase to $n=8$ providing minimal additional benefits. Notably, OLMo-1B-DHC$\times$8 W/O tanh excels on both V2 and V3 validation sets, with a reduction in V2 Eval Loss by 0.034 and V3 Eval Loss by 0.029 compared to the baseline. Furthermore, the decline rate of training losses for DHC ($n\geq 2$) is steeper than that of the baseline, and DHC demonstrates greater stability, with no spikes observed in any DHC experiments.

Staitc and dynamic hyper-connections.
Table 2 presents an ablation study comparing SHC and DHC. All hyper-connection (HC) variants significantly outperform the baseline. At an expansion rate of 2, the improvements of DHC and SHC are similar. However, at an expansion rate of 4, DHC performs notably better than SHC.

**Table 2: Ablation Study on Static and Dynamic Hyper-Connections with Training on 500 B Tokens**
| Methods | V2 Eval<br>Loss $\downarrow$ | V2 Eval<br>PPL $\downarrow$ | V3 Eval<br>Loss $\downarrow$ | V3 Eval<br>PPL $\downarrow$ | Down Stream<br>Avg, Acc. $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| OLMo-1B | 2.811 | 18.023 | 2.544 | 14.229 | 62.5 |
| OLMo-1B-SHC$\times$2 | 2.799 | 17.778 | 2.538 | 14.152 | 63.4 |
| OLMo-1B-DHC$\times$2 | 2.802 | 17.950 | 2.534 | 14.114 | 63.0 |
| OLMo-1B-DHC$\times$2 W/O tanh | 2.792 | 17.663 | 2.529 | 14.033 | 63.8 |
| OLMo-1B-SHC$\times$4 | 2.791 | 17.671 | 2.528 | 14.025 | 63.6 |
| OLMo-1B-DHC$\times$4 | 2.781 | 17.509 | 2.515 | 13.826 | 63.8 |
| OLMo-1B-DHC$\times$4 W/O tanh | 2.779 | 17.451 | 2.516 | 13.844 | 64.4 |

The importance of $\mathbf{B}$ and $\mathcal{WC}$. As shown in Table 3, not training $\mathcal{WC}$ leads to significant performance declines, with the V2 loss increasing by 0.021 and the V3 loss by 0.017, as seen when comparing the 4th and 6th lines of Table 3. In contrast, the impact is less pronounced when $\mathbf{B}$ is not trained. Therefore, ensuring the trainability of both $\mathcal{WC}$ and $\mathbf{B}$ is crucial.

**Table 3: Ablation study on OLMo-1B-DHC$\times$4. In the $\mathbf{B}$ or $\mathcal{WC}$ column, the symbol "✗" denotes parameters that are not trainable from initialization.**
| $\mathcal{WC}$ | $\mathbf{B}$ | Tanh | V2 Eval<br>Loss $\downarrow$ | V2 Eval<br>PPL $\downarrow$ | V3 Eval<br>Loss $\downarrow$ | V3 Eval<br>PPL $\downarrow$ | Down Stream<br>Avg, Acc. $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ✗ | ✓ | ✗ | 2.804 | 17.912 | 2.537 | 14.145 | 62.5 |
| ✓ | ✗ | ✗ | 2.781 | 17.493 | 2.518 | 13.874 | 63.6 |
| ✓ | ✓ | ✗ | 2.779 | 17.773 | 2.516 | 13.823 | 64.4 |
| ✗ | ✓ | ✓ | 2.802 | 17.914 | 2.532 | 14.072 | 63.4 |
| ✓ | ✗ | ✓ | 2.783 | 17.504 | 2.520 | 13.906 | 63.4 |
| ✓ | ✓ | ✓ | 2.781 | 17.835 | 2.515 | 13.807 | 63.8 |

### 4.2 Comparison with related works

**Table 4: Performance of Related Methods on OLMo-1B.**
| Methods | V2 Eval<br>Loss $\downarrow$ | V2 Eval<br>PPL $\downarrow$ | V3 Eval<br>Loss $\downarrow$ | V3 Eval<br>PPL $\downarrow$ | Down Stream<br>Avg, Acc. $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| OLMo-1B | 2.811 | 18.023 | 2.544 | 14.229 | 62.5 |
| OLMo-1B-ResiDual | 2.825 | 18.375 | 2.551 | 14.346 | 62.0 |
| OLMo-1B-Altup$\times$2 | 2.827 | 18.268 | 2.558 | 14.454 | 62.4 |
| OLMo-1B-DHC$\times$2 | 2.802 | 17.950 | 2.534 | 14.114 | 63.0 |
| OLMo-1B-DHC$\times$2 W/O tanh | 2.792 | 17.663 | 2.529 | 14.033 | 63.8 |

We implemented the Altup (Baykal et al., ) and ResiDual (Xie et al., ) methods in OLMo. Altup is motivated to widen the hidden dimension while maintaining low computation cost by passing only a part of hidden state to transformer blocks. By contrast, ResiDual is proposed to combine both Pre- and Post-Norm in a two-stream style. Both methods expand the hidden size by $n$ times with negligible computational overhead, with ResiDual expanding it exactly $2$ times. For a fair comparison, we set $n=2$ in our experiments. Unfortunately, these methods show gains in the early stages of training but are gradually surpassed by the baseline, as shown in Table 4.

### 4.3 7B Models

Figure: Figure 6: (1) and (2) Training loss (0.99 EMA smoothed) and C4-en validation loss for OLMo-7B and OLMo-7B-DHC$\times$4 models. (3) and (4) Accuracy curves on hellaswag and sciq, demonstrating the superior performance of the OLMo-7B-DHC$\times$4 model.
Refer to caption: x9.png

We evaluate the effectiveness of hyper-connections on the 7B model, training a model with DHCs with an expansion rate of 4, denoted as OLMo-7B-DHC$\times$4. According to Table 5, OLMo-7B-DHC$\times$4 significantly outperforms the baseline OLMo-7B model in all average metrics. In the V2 evaluation, OLMo-7B-DHC$\times$4 shows improvements of 0.022 for loss and 0.293 for PPL. Furthermore, the average score of downstream benchmarks 0.710 surpasses the baseline 0.701, with the results of specific tasks are shown in Fig. 9.

**Table 5: Evaluation Metrics for 7B Models**
| Methods | V2 Eval<br>Loss $\downarrow$ | V2 Eval<br>PPL $\downarrow$ | V3 Eval<br>Loss $\downarrow$ | V3 Eval<br>PPL $\downarrow$ | Down Stream<br>Avg, Acc. $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| OLMo-7B | 2.581 | 14.316 | 2.322 | 11.324 | 70.1 |
| OLMo-7B-DHC$\times$4 | 2.559 | 14.023 | 2.304 | 11.120 | 71.0 |

Based on Fig 6, the OLMo-7B-DHC$\times$4 model consistently shows better metrics compared to baseline, including training and validation loss and accuracy in downstream benchmarks. Notably, after 400 B tokens, the model maintains its improvement without the gains diminishing. This indicates that the OLMo-7B-DHC$\times$4 model continues to provide consistent benefits in reducing loss, even at higher token counts. Furthermore, according to Fig. 6, the baseline model exhibits frequent spikes, while our model with DHCs shows no spikes throughout the training. This shows that our approach not only achieves better loss but also ensures more stable training.

### 4.4 MoE Models

We evaluate the effectiveness of hyper-connections on the Mixture-of-Experts (MoE) model. We retrain the original OLMoE-1B-7B model as the baseline and train a model that applies Dynamic Hyper-Connections (DHC) with $n=4$, replacing the residual connections. The full results are shown in Fig. 8, which illustrates that hyper-connections outperform residual connections in almost all metrics. In many metrics, our method requires only half of the training tokens to achieve the same performance as the baseline. Fig. 1 and Table 6 highlight some of the results, such as a reduction in training loss of approximately 0.027, a reduction in loss on the C4-en validation set of 0.028, an improvement of 6 points on the ARC-Challengeand an improvement of 1.2 points on MMLU Var.

**Table 6: Downstream evaluations for MoE models training with 500B tokens under the OLMoE evaluation setting. ARC-C stands for ARC-Challenge, and ARC-E for ARC-Easy. MMLU Var is a modified version of MMLU that includes varying few-shot examples, providing stable feedback during early training, as outlined in the OLMoE setting (Muennighoff et al., ).**
| Methods | MMLU<br>Var | Hella-<br>Swag | ARC-C | ARC-E | PIQA | Wino-<br>Grande | BoolQ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| OLMoE-1B-7B | 38.5 | 69.5 | 41.8 | 72.8 | 77.6 | 64.4 | 65.4 |
| OLMoE-1B-7B-DHC$\times$4 | 39.7 | 70.2 | 47.8 | 76.7 | 78.2 | 64.6 | 68.5 |

### 4.5 Visualization Analysis

Figure: Figure 7: Visualization of connection matrices for hyper-connections and various related baseline methods. The attention layers, which have odd ids, are marked with green tick marks.

In this section, we investigate the learned hyper-connection weights and show how the output of the former layer contributes to the latter ones. To this end, we convert hyper-connections to dense connections cross layers.
Consider the input hidden vectors $\mathbf{h}_{0}^{k}$ in $k$-th layer, it can be unfolded as a weighted summation over previous layer outputs:

$$ $\displaystyle\mathbf{h}_{0}^{k}=\sum_{j=0}^{k-1}{c_{kj}^{(0)}\mathcal{T}^{j}}(\mathbf{h}_{0}^{j}),$ (20) $$

where $c_{kj}^{(0)}$ describes how much layer-$j$ ($\mathcal{T}^{j}$) contributes to layer-$k$’s input $\mathbf{h}_{0}^{k}$. Then, $\mathbf{C}^{(0)}$ denotes a dense connection weight matrix. In particular, let layer-0 be the word embedding and $\mathcal{T}^{0}$ be an identity mapping, layer-$L+1$ be the hidden state before the unembedding layer, which is a summation over the last hidden vectors, i.e., $\mathbf{h}_{0}^{L+1}=\sum_{j}\mathbf{h}_{j}^{L}$.

OLMo-1B-DHC$\times$4 model is adopted for visualization. We take the checkpoint at 500B tokens and forward random validation text to obtain dynamic hyper-connection weights. In addition, we show connection patterns for some related baseline methods. Finally, the visualization is illustrated in Fig. 10. We present the following findings, with more detailed discussions provided in Appendix D.

Connection patterns for baseline methods. For Pre-Norm baseline, the connection matrix is simply a lower triangular matrix with diagonal elements erased, because each transformer layer joins the residual equally. In the Pre-Norm parallel transformer block (PTB) baseline, the connection matrix appears jagged because the input to the FFN layer does not depend on the output of the previous attention layer. For Post-Norm baseline, the connection only holds for adjacent layers, as the weight for bottom layers decays every time the residual passes a post-norm layer. For the two-hop residual baseline (Ma et al., ), the outputs of attention layers are not added to residual and only contributes to the next one FFN layer, resulting in a vertical strip pattern in the connection matrix.

$\Lambda$-shaped connection pattern. In the connection matrix for hyper-connections, a long-term decay pattern can be observed, where layers are generally preferred to rely on a few adjacent layer outputs. Moreover, the bottom layers (e.g. layer 0,2) are observed frequently used in most of subsequent layers. Therefore, the two patterns together form a $\Lambda$-shaped connection pattern. Note that the long-term decay pattern is a Post-Norm style pattern, while the frequently accessed pattern is Pre-Norm style, indicating that the hyper-connection introduces a free mixture of Pre- and Post-Norm architecture.

Input word embedding is eliminated from model output. As per the first column in the connection matrix for layer inputs, the input word embedding contributes to most of the layers except for the final one. This last layer, which products the model’s output, is used for next token prediction. In most cases, keeping a component of input embedding in model output is harmful to next token prediction, especially when using a tied word embedding such as that employed by OLMo-1B. Similar results are found in previous works (Ma et al., ).

Parallel transformer blocks are observed. As discussed in § 3.2, parallel transformer block, which performs attention and FFN in parallel, is a special case for hyper-connection. In practice, PTB-like patterns, which can be identified by the local jagged pattern, are surprisingly observed to be learned by hyper-connections. For instance, layer 11 has a minimal contribution to the input of layer 12 (refer to row 12 in the hyper-connection connection matrix). This suggests that layers 11 and 12 can operate in parallel, thereby forming a PTB module.

Attention layers tend to have fewer long-term connections. It is observed that attention layers at the bottom barely have long-term contribution, a trend that persists until layer 17. Upon examining the connection matrix for hyper hiddens (refer to Fig. 10 in the appendix), it’s evident that the outputs of the FFN layers have significantly greater magnitudes than those of the attention layers. This pattern resembles a two-hop residual connection design, wherein the attention output contributes to the input of the following FFN layer, but doesn’t join the main residual path.

## 5 Related Work

Transformers (Vaswani et al., ) have revolutionized various fields, particularly natural language processing and computer vision. They rely heavily on residual connections to facilitate the training of deep models. Our hyper-connections approach can replace residual connections, providing stable training and consistent improvements in both natural language processing and computer vision.

The issues of gradient vanishing and representation collapse (Bengio et al., ; Glorot & Bengio, ; Liu et al., ) have been extensively studied. The combinations of normalization techniques (Ioffe & Szegedy, ; Ba et al., ) and residual connections (He et al., ), like Pre-Norm and Post-Norm, actually reflects different emphases in solving these two issues. However, despite these advancements, the fundamental trade-off between gradient vanishing and representation collapse in deep networks remains a critical challenge. Building on these findings, our work introduces a novel approach that enables neural networks to autonomously learn the optimal strength of connections, potentially improving both gradient stability and representation quality.

## 6 Conclusion

In conclusion, we have introduced hyper-connections as an effective alternative to residual connections in transformers. Our analysis reveals that hyper-connections not only overcome the limitations of residuals but also enable dynamic adjustments in network architecture. Experimental results confirm their promising benefits across various tasks, including pre-training of large language model, image generation, and image classification.

## Acknowledgements

This research was conducted at ByteDance Inc. We are grateful for the suggestions and assistance provided by Yaowei Zheng, Yuyu Zhang, Yunshui Li, Xiang Li, and Bairen Yi.

## Appendix A MoE 1B/7B Model Experiments

Figure: Figure 8: Loss curves in V3 validation set and accuracy curves on downstream tasks for OLMoE-1B7B and OLMoE-1B7B-DHC$\times{4}$ models.
Refer to caption: x13.png

## Appendix B 7B Model Experiments

Figure: Figure 9: Loss curves in V3 validation set and accuracy curves on downstream tasks for OLMo-7B and OLMo-7B-DHC$\times$4 models.
Refer to caption: x14.png

## Appendix C Vision Experiments

Datasets. We use the ILSVRC-2012 ImageNet dataset (Deng et al., ) with 1k classes and 1.3M images (see ImageNet in the following) for image generation and classification.

### C.1 Image Generation

To investigate the generalizability of hyper-connections in image generation, our experiments are conducted using the DiT framework (Peebles & Xie, ) training the models for 1400 epochs. In order to save experimental costs, we use FP16 precision, introduce flash-attention to speed up training, and introduce QK-Norm (Wortsman et al., ) to stabilize training.

**Table 7: Benchmarking class-conditional image generation on ImageNet 256$\times$256, with cfg=1.50. NP, P, and R are short for Numerical Precision, Precision, and Recall, respectively.**
| Method | NP | QK-Norm | Size (M) | FID$\downarrow$ | sFID$\downarrow$ | IS$\uparrow$ | P$\uparrow$ | R$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DiT-XL/2 | FP32 | ✗ | 675 | 2.27 | 4.60 | 278.24 | 0.83 | 0.57 |
| DiT-XL/2 | FP16 | ✓ | 675 | 2.36 | 4.54 | 269.46 | 0.83 | 0.58 |
| DiT-1B/2 | FP16 | ✓ | 983 | 2.13 | 4.50 | 288.69 | 0.82 | 0.59 |
| DiT-XL/2-SHC$\times$2 | FP16 | ✓ | 675 | 2.18 | 4.52 | 287.24 | 0.82 | 0.60 |

Our experimental results demonstrate that DiT models incorporating hyper-connections exhibit comparable performance metrics to DiT models with 50% more parameters. This finding underscores the efficiency and efficacy of hyper-connections in enhancing model performance without increasing model size.

### C.2 Image Classification

For the image classification experiments, we train ViT/16-Base and ViT/16-Large models with images at a resolution of $224\times 224$ for 300 epochs, following the experimental setup used by (Dosovitskiy et al., ).To speed up the training process, we use bfloat16 numerical precision. The training configuration is detailed in Table 9. Within this configuration, we replace the residual connections with static and dynamic hyper-connections, referred to as SHC and DHC, respectively, using an expansion rate of $n=2$. The top-1 accuracy results are presented in Table 8.

For the Base model (85M), our re-implemented ViT/16 achieves 76.38% accuracy on $224\times 224$ images. The SHC and DHC enhance performance to 77.60% and 77.26%, respectively. representing relative increases of 1.22% and 0.88%. For the Large model (307M parameters), ViT/16 achieves 77.25% accuracy. The SHC and DHC configurations further enhance accuracy to 78.38% and 79.94%, respectively. This corresponds to relative improvements of 1.13% and 2.69%, with DHC showing the highest performance. These results demonstrate that hyper-connections (SHC and DHC) significantly improve accuracy, especially in the Large model scale.

**Table 8: Accuracy on ImageNet. ViT*/16 refers to the results reported by (Dosovitskiy et al., ), whereas ViT/16 denotes our re-implemented baseline. SHC and DHC indicate that residual connections are replaced with static and dynamic hyper-connections, respectively.**
| Model Scales | Params (M) | ViT*/16 | ViT/16 | ViT/16-SHC$\times 2$ | ViT/16-DHC$\times 2$ |
| --- | --- | --- | --- | --- | --- |
| $384\times 384$ | $224\times 224$ |  |  |  |  |
| Base | 85 | 77.91 | 76.38 | 77.60 | 77.26 |
| Large | 307 | 76.53 | 77.25 | 78.38 | 79.94 |

**Table 9: Training hyperparameters for ViT.**
| Hyperparameter | Value |
| --- | --- |
| Learning Rate (lr) | 0.003 |
| Batch Size | 4096 |
| Scheduler | Cosine Annealing with Linear Warmup (10k steps) |
| Data Augmentation | Mixup ($\alpha=0.2$) |
| Epochs | 300 |
| Optimizer | AdamW ($\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=1e-8$) |
| Gradient Clipping | 1.0 |
| Weight Decay | 0.3 |
| Dropout | 0.1 |
| Precision | bf16 |

## Appendix D More Visualization and Analysis

Figure: (a) Connection matrix for DHC model.

#### Unfolding hyper-connections.

We first introduce how to determine the connection matrix $\mathbf{C}^{(0)}$ for hyper-connections. To simplify writing, the layer output $\mathcal{T}^{k}(\mathbf{h}_{0}^{k})$ is denoted by $\mathcal{T}^{k}$ for short. The recurrent form of hyper connection in Eq. 2 is expanded as follows:

$$ $\displaystyle{\mathbf{h_{0}}^{k}}=$ $\displaystyle{\mathbf{H}^{k}}^{\intercal}\mathbf{A_{m}}^{k}=(\mathcal{T}^{k-1}\mathbf{B}^{k-1}+{\mathbf{H}^{k-1}}^{\intercal}\mathbf{A_{r}}^{k-1})\mathbf{A_{m}}^{k}$ $\displaystyle=$ $\displaystyle\sum_{j=0}^{k-1}\mathcal{T}^{j}\mathbf{B}^{j}(\mathbf{A_{r}}^{j+1}\mathbf{A_{r}}^{j+2}...\mathbf{A_{r}}^{k-1})\mathbf{A_{m}}^{k}$ $\displaystyle=$ $\displaystyle\sum_{j=0}^{k-1}\mathcal{T}^{j}\mathbf{B}^{j}(\prod_{t=j+1}^{k-1}{\mathbf{A_{r}}^{t}})\mathbf{A_{m}}^{k}.$ (21) $$

Therefore, we obtain connection matrix $c^{(0)}_{kj}=\mathbf{B}^{j}(\prod_{t=j+1}^{k-1}{\mathbf{A_{r}}^{t}})\mathbf{A_{m}}^{k}$. Similarly, the connection matrix $\mathbf{C}^{(i)}$ for the $i$-th hyper hidden from $k$-th layer can be computed by substituting the last ${\mathbf{A_{m}}^{k}}$ with $\mathbf{A_{r}}^{k}$ in Eq. 21, i.e.,

$$ $\displaystyle{\mathbf{H^{\prime}}^{k}}={\mathbf{A_{r}}^{k}}^{\intercal}\mathbf{H}^{k}=\sum_{j=0}^{k-1}(\prod_{t=j+1}^{k}{\mathbf{A_{r}}^{t}})^{\intercal}{\mathbf{B}^{j}}^{\intercal}{\mathcal{T}^{j}}^{\intercal}$ (22) $\displaystyle c^{(i)}_{kj}=\left({(\prod_{t=j+1}^{k}{\mathbf{A_{r}}^{t}})}^{\intercal}{\mathbf{B}^{j}}^{\intercal}\right)_{i}.$ (23) $$

#### Visualization for hyper hidden.

We visualize connection matrices for hyper hiddens in Fig. 10 to reveal how hyper-connection maintains intermediate layer outputs.
First of all, the four hyper hiddens are dissimilar and show completely different connection patterns. Then, we can see outputs from FFN layers are preserved long-termly in hyper hiddens, while attention layers are reserved less. It is also observed that the long-term connections are usually stored in pairs of hyper hiddens, where the connection is positive in one hyper hidden but negative in the other, for example, column 0 and 2 in $\mathbf{C}^{(1)},\mathbf{C}^{(3)}$. With such strategy, these connections can be easily eliminated in the sum-pooling operation before the unembedding layer.

#### SHC shares similar connection pattern with DHC.

We show the connection matrices for OLMo-1B-SHC$\times$4 model in Fig. 10(b). Comparing to DHC, as shown in Fig. 10(a), SHC shares exactly the same connection patterns. Moreover, we observe many more PTB-like blocks in SHC, e.g., layers from 13 to 18. Note that the connection relation for SHC is token independent, and such PTB-like blocks can be physically reorganized to be parallelly computed.

## Appendix E Derivation of Non-Trainable Hyper-Connection Matrix for Residual Connections

### E.1 Pre-Norm Residual Connection

In the Pre-Norm residual connection, the input to a layer is first normalized before being passed through the layer. The output of the layer is then added to the original input. This can be represented as:

$$ $\mathbf{\hat{h}}=\mathcal{T}(\texttt{Norm}(\mathbf{h}))+\mathbf{h}.$ (24) $$

By incorporating the normalization operator into the layer, $\mathcal{T}\coloneqq\mathcal{T}\circ\texttt{Norm}$, we can express the entire process as:

$$ $\mathbf{\hat{h}}=\mathcal{T}(\mathbf{h})+\mathbf{h}.$ (25) $$

To express this using hyper-connections, the matrix for Pre-Norm can be structured as follows:

$$ $\mathcal{HC}_{PreNorm}=\begin{pmatrix}0&1\\ 1&1\\ \end{pmatrix}$ (26) $$

Given hyper hidden matrix $\mathbf{H}=\mathbf{h}^{\intercal}$, we prove that the output of $\mathcal{HC}_{\text{PreNorm}}$ $\mathbf{\hat{H}}=\mathbf{\hat{h}}^{\intercal}$.

###### Proof.

$$ $\displaystyle\mathbf{\hat{H}}$ $\displaystyle=\mathcal{HC}(\mathcal{T},\mathbf{H})$ (27) $\displaystyle=\mathbf{B}^{\intercal}\mathcal{T}(\mathbf{H}^{\intercal}\mathbf{A_{m}})^{\intercal}+\mathbf{A_{r}}^{\intercal}\mathbf{H}$ $\displaystyle=\mathcal{T}(\mathbf{h})^{\intercal}+\mathbf{h}^{\intercal}$ $\displaystyle=\mathbf{\hat{h}}^{\intercal}.$ $$

∎

###### Proof.

### E.2 Post-Norm Residual Connection

In the Post-Norm residual connection, the input to a layer is passed through the layer first, and then the output is normalized after being added to the original input. In matrix form, this can be represented as:

$$ $\mathbf{h}^{\prime}=\mathcal{T}(\mathbf{h})$ (28) $$

The summation of the input and the normalized output of the layer is:

$$ $\mathbf{\hat{h}}=\texttt{Norm}(\mathbf{h}+\mathbf{h}^{\prime})$ (29) $$

We consider Norm to be LayerNorm (Zhang & Sennrich, ). The analysis process for RMSNorm is almost identical. In fact, the affine transformation can be incorporated into the subsequent layer, while the mean subtraction operation can be integrated into the current layer.

$$ $\mathcal{T}=\mathcal{C}\circ\mathcal{T}\circ\mathcal{A},$ (30) $$

where $\mathcal{A}$ is the affine transformation, and $\mathcal{C}$ is the re-centering operator. Thus, the mean of the output of $\mathcal{T}$ is 0.

To express this using hyper-connections with an expansion rate $n=1$, we need a hyper-connection matrix $\mathcal{HC}$ that encapsulates this operation:

$$ $\mathcal{HC}_{PostNorm}=\begin{pmatrix}0&\frac{1}{\sqrt{\sigma_{\mathbf{h}}^{2}+\sigma_{\mathbf{h}^{\prime}}^{2}+2\sigma_{\mathbf{h}\mathbf{h}^{\prime}}}}\\ 1&\frac{1}{\sqrt{\sigma_{\mathbf{h}}^{2}+\sigma_{\mathbf{h}^{\prime}}^{2}+2\sigma_{\mathbf{h}\mathbf{h}^{\prime}}}}\\ \end{pmatrix}=\begin{pmatrix}0&\mathbf{B}\\ \mathbf{A}_{m}&\mathbf{A}_{r}\\ \end{pmatrix}.$ (31) $$

Similar to the previous proof, we prove that the output of $\mathcal{HC}_{\text{PostNorm}}$ is equivalent to the transpose of the output of the Post-Norm residual connection:

$$ $\mathbf{\hat{H}}=\mathbf{\hat{h}}^{\intercal}.$ (32) $$

###### Proof.

Note that

$$ $\sigma_{\mathbf{h}+\mathbf{h}^{\prime}}=\sqrt{\sigma_{\mathbf{h}}^{2}+\sigma_{\mathbf{h}^{\prime}}^{2}+2\sigma_{\mathbf{h}\mathbf{h}^{\prime}}}.$ (33) $$

Given this fact, we can derive the Post-Norm:

$$ $\displaystyle\mathbf{\hat{h}}$ $\displaystyle=\text{Norm}(\mathbf{h}^{\prime}+\mathbf{h})$ (34) $\displaystyle=\frac{\mathbf{h}^{\prime}+\mathbf{h}-\mu_{\mathbf{h}^{\prime}+\mathbf{h}}}{\sigma_{\mathbf{h}+\mathbf{h}^{\prime}}}$ $\displaystyle=\frac{1}{\sigma_{\mathbf{h}^{\prime}+\mathbf{h}}}(\mathbf{h}^{\prime}+\mathbf{h})$ $\displaystyle=\frac{1}{\sqrt{\sigma_{\mathbf{h}}^{2}+\sigma_{\mathbf{h}^{\prime}}^{2}+2\sigma_{\mathbf{h}\mathbf{h}^{\prime}}}}(\mathbf{h}^{\prime}+\mathbf{h})$ $$

For hyper-connections side, we have:

$$ $\displaystyle\mathbf{\hat{H}}$ $\displaystyle=\mathbf{B}^{\intercal}\mathbf{h}^{\prime\intercal}+\mathbf{H}^{\prime}$ (35) $\displaystyle=\mathbf{B}^{\intercal}\mathbf{h}^{\prime\intercal}+\mathbf{A}_{r}\mathbf{H}$ $\displaystyle=\mathbf{B}^{\intercal}\mathbf{h}^{\prime\intercal}+\mathbf{A}_{r}\mathbf{h}^{\intercal}$ $\displaystyle=\frac{1}{\sqrt{\sigma_{\mathbf{h}}^{2}+\sigma_{\mathbf{h}^{\prime}}^{2}+2\sigma_{\mathbf{h}\mathbf{h}^{\prime}}}}\mathbf{h}^{\prime\intercal}+\frac{1}{\sqrt{\sigma_{\mathbf{h}}^{2}+\sigma_{\mathbf{h}^{\prime}}^{2}+2\sigma_{\mathbf{h}\mathbf{h}^{\prime}}}}\mathbf{h}^{\intercal}$ $\displaystyle=\mathbf{\hat{h}}^{\intercal}.$ $$

∎

###### Proof.

## Appendix F Sequential-Parallel Duality

### F.1 Hyper-Connection Matrix of Sequential Arrangement

In this section, we demonstrate that the following hyper-connection matrix will produce $n$ identical networks arranged sequentially with residual connections between them:

$$ $\mathcal{HC}=\begin{pmatrix}\mathbf{0}_{1\times 1}&\mathbf{1}_{1\times n}\\ \mathbf{e}_{1}&\mathbf{e}_{n\times n}\end{pmatrix},$ (36) $$

where $\mathbf{e}_{n\times n}$ denotes an $n\times n$ identity matrix, $\mathbf{e}_{i}\in\mathbb{R}^{n\times 1}$ represents the $i$-th column of $\mathbf{e}_{n\times n}$, and $\mathbf{1}_{1\times n}$ signifies a $1\times n$ matrix of ones.

We will use mathematical induction to prove that $\mathbf{h}_{i}^{k}=\mathbf{h}_{j}^{k}$ and $\mathbf{h}_{i}^{k+1}=\mathcal{T}^{k}(\mathbf{h}_{i}^{k})+\mathbf{h}_{i}^{k}$, $\forall i,j\in\{0,1,\ldots,n\}$, $\forall k\in\{0,1,\ldots,L\}$, where $L$ is the number of layers.

###### Proof.

###### Proof.

### Base Case

For $k=0$, we have the initial condition $\mathbf{h}_{i}^{0}=\mathbf{h}_{j}^{0}$, $\forall i,j\in\{0,1,\ldots,n\}$, as we define $\mathbf{H}^{0}=\begin{pmatrix}\mathbf{h}^{0}&\mathbf{h}^{0}&\dots&\mathbf{h}^{0}\end{pmatrix}^{\intercal}\in\mathbb{R}^{n\times d}$.

### Induction Hypothesis

Assume that for some $k\in\{1,\ldots,L-1\}$, we have $\mathbf{h}_{i}^{k}=\mathbf{h}_{j}^{k}$ and $\mathbf{h}_{i}^{k}=\mathcal{T}^{k}(\mathbf{h}_{i}^{k-1})+\mathbf{h}_{i}^{k-1}$, $\forall i,j\in\{0,1,\ldots,n\}$.

### Induction Step

We have

$$ $\displaystyle\mathbf{H}^{k+1}$ $\displaystyle=\mathcal{HC}(\mathcal{T}^{k},\mathbf{H}^{k})$ (37) $\displaystyle=\mathbf{B}^{\intercal}(\mathbf{h}^{\prime k}_{0})^{\intercal}+\mathbf{H^{\prime}}^{k}$ (38) $\displaystyle=\mathbf{B}^{\intercal}\mathbf{A_{m}}^{\intercal}\mathbf{H}^{k}+\mathbf{A_{r}}^{\intercal}\mathbf{H}^{k}$ (39) $\displaystyle=\mathbf{1}_{n\times 1}\mathcal{T}^{k}(\mathbf{e}_{1}^{\intercal}\mathbf{H}^{k})+\mathbf{e}_{n\times n}\mathbf{H}^{k}$ (40) $\displaystyle=\begin{pmatrix}\mathcal{T}^{k}(\mathbf{h}_{1}^{k})&\mathcal{T}^{k}(\mathbf{h}_{1}^{k})&\ldots&\mathcal{T}^{k}(\mathbf{h}_{1}^{k})\end{pmatrix}^{\intercal}+\begin{pmatrix}\mathbf{h}_{1}^{k}&\mathbf{h}_{2}^{k}&\dots&\mathbf{h}_{n}^{k}\end{pmatrix}^{\intercal}$ (41) $\displaystyle=\begin{pmatrix}\mathcal{T}^{k}(\mathbf{h}_{1}^{k})+\mathbf{h}_{1}^{k}&\mathcal{T}^{k}(\mathbf{h}_{1}^{k})+\mathbf{h}_{2}^{k}&\ldots&\mathcal{T}^{k}(\mathbf{h}_{1}^{k})+\mathbf{h}_{n}^{k}\end{pmatrix}^{\intercal}$ (42) $\displaystyle=\begin{pmatrix}\mathbf{h}_{1}^{k+1}&\mathbf{h}_{2}^{k+1}&\dots&\mathbf{h}_{n}^{k+1}\end{pmatrix}^{\intercal}$ (43) $$

Since $\mathbf{h}_{i}^{k}=\mathbf{h}_{j}^{k}$, $\forall i,j\in\{0,1,\ldots,n\}$, it follows that $\mathcal{T}^{k}(\mathbf{h}_{1}^{k})+\mathbf{h}_{i}^{k}=\mathcal{T}^{k}(\mathbf{h}_{1}^{k})+\mathbf{h}_{j}^{k}$. Thus, we have

$$ $\mathbf{h}_{i}^{k+1}=\mathbf{h}_{j}^{k+1}$ (44) $$

Since $\mathbf{h}_{i}^{k}=\mathbf{h}_{j}^{k}$, $\forall i,j\in\{0,1,\ldots,n\}$, it follows that $\mathbf{h}_{1}^{k}=\mathbf{h}_{i}^{k}$, $\forall i\in\{0,1,\ldots,n\}$. Thus, we have

$$ $\displaystyle\mathbf{h}_{i}^{k+1}$ $\displaystyle=\mathcal{T}^{k}(\mathbf{h}_{1}^{k})+\mathbf{h}_{i}^{k}$ (45) $\displaystyle=\mathcal{T}^{k}(\mathbf{h}_{i}^{k})+\mathbf{h}_{i}^{k}$ (46) $$

∎

### F.2 Hyper-Connection Matrix of Parallel Arrangement

In this section, we demonstrate that the following hyper-connection matrix will produce a network where every $n$ adjacent layers are arranged in parallel, with each layer incorporating residual connections.
We define a parallel-arranged network such that $n$ adjacent layers form a group, with layers within a group being parallel and groups arranged sequentially. The output of $k$-th group is given by:

$$ $\mathbf{h}^{k+1}=\sum_{i=1}^{n}(\mathcal{T}^{k\times n+i}(\mathbf{h}^{k})+\mathbf{h}^{k}).$ (47) $$

It can be proved that this arrangement can be described by the following hyper-connection matrices.

First, for $k$ where $k-1\equiv 0\pmod{n}$:

$$ $\mathcal{HC}^{\{k\mid k-1\equiv 0\pmod{n}\}}=\begin{pmatrix}\mathbf{0}_{1\times 1}&\mathbf{e}_{1}^{\intercal}\\ \mathbf{1}_{n\times 1}&\mathbf{1}_{n\times n},\end{pmatrix}$ (48) $$

where the $\mathcal{HC}$ matrix can be decomposed into two operations: 1) sum up all the outputs of the previous group and use it as the input of the current layer and as the residual of the subsequent layers; 2) sum up the output and input saving to the first hidden vector slot.

Next, for $k$ where $k-1\equiv i\pmod{n}$ and $i\neq 0$:

$$ $\mathcal{HC}^{\{k\mid k-1\equiv i\pmod{n},i\neq 0\}}=\begin{pmatrix}\mathbf{0}_{1\times 1}&\mathbf{e}_{i}^{\intercal}\\ \mathbf{e}_{i}&\mathbf{e}_{n\times n},\end{pmatrix}.$ (49) $$

where the $\mathcal{HC}$ matrix selects the $i$-th hidden vector as the input of the current layer, and sums up the output and input, saving to the $i$-th hidden vector slot.

This means:

$$ $\displaystyle\mathbf{h}^{k+1}=$ $\displaystyle\mathcal{HC}^{(k+1)\times n}(\mathcal{T}^{(k+1)\times n},$ (50) $\displaystyle\mathcal{HC}^{(k+1)\times n-1}(\mathcal{T}^{(k+1)\times n-1},$ (51) $\displaystyle\cdots$ (52) $\displaystyle\mathcal{HC}^{k\times n+1}(\mathcal{T}^{k\times n+1},\mathbf{h}^{k})))$ (53) $$

This can also be proved by mathematical induction; however, the conclusion is quite obvious through drawing, and the proof process is very tedious. Therefore, we don’t repeat the similar proof here.

## Appendix G Pseudocode of Hyper-connections

Figure: Algorithm 1 Network with Hyper-Connections

## Appendix H PyTorch Implementation of Hyper-connections

Figure: Algorithm 2 Pseudocode of Hyper-connections in a PyTorch-like style.

## Appendix I Validation Sets and Downstream Tasks

**Table 10: OLMo’s default configuration was evaluated using multiple metrics. Perplexity (PPL) and loss were used for the V2 and V3 Validation Sets, while zero-shot testing was applied to the Downstream Benchmarks. However, the grey benchmarks were excluded from our analysis due to the instability of their performance indicators.**
| V2 Validation Sets |
| --- |
| v2-small-4chan-validation |
| v2-small-c4_100_domains-validation |
| v2-small-c4_en-validation |
| v2-small-gab-validation |
| v2-small-ice-validation |
| v2-small-m2d2_s2orc-validation |
| v2-small-m2d2_wiki-validation |
| v2-small-manosphere-validation |
| v2-small-mc4_en-validation |
| v2-small-pile-validation |
| v2-small-ptb-validation |
| v2-small-twitterAEE-validation |
| v2-small-wikitext_103-validation |
| V3 Validation Sets |
| v3-small-c4_en-validation |
| v3-small-dolma_books-validation |
| v3-small-dolma_common-crawl-validation |
| v3-small-dolma_pes2o-validation |
| v3-small-dolma_reddit-validation |
| v3-small-dolma_stack-validation |
| v3-small-dolma_wiki-validation |
| v3-small-ice-validation |
| v3-small-m2d2_s2orc-validation |
| v3-small-pile-validation |
| v3-small-wikitext_103-validation |
| Downstream Benchmarks |
| piqa (Bisk et al., ) |
| hellaswag (Zellers et al., ) |
| winogrande (Sakaguchi et al., ) |
| openbook_qa (Mihaylov et al., ) |
| sciq (Johannes Welbl, ) |
| arc_easy (Clark et al., ) |
| copa (Roemmele et al., ) |
| commitment_bank (De Marneffe et al., ) |
| mrpc (Dolan & Brockett, ) |
| rte (Dagan et al., ) |
| sst2 (Socher et al., ) |

**Table 11: Downstream Benchmarks for OLMoE.**
| Downstream Benchmarks for OLMoE |
| --- |
| piqa (Bisk et al., ) |
| hellaswag (Zellers et al., ) |
| winogrande (Sakaguchi et al., ) |
| openbook_qa (Mihaylov et al., ) |
| sciq (Johannes Welbl, ) |
| arc_easy (Clark et al., ) |
| arc_challenage (Clark et al., ) |
| copa (Roemmele et al., ) |
| boolq (Clark et al., ) |
| commonsense_qa (Talmor et al., ) |
| social_iqa (Sap et al., ) |
| mmlu (Hendrycks et al., ) |

## Appendix J 1B Model Experiments

**Table 12: Results on Downstream Benchmarks for 1B Model.**
| Method | arc_easy | copa | hellaswag | openbook_qa | piqa | sciq | winogrande | avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OLMo-1B | 56.8 | 76.0 | 56.1 | 33.8 | 74.4 | 85.1 | 55.6 | 62.5 |
| Scaling n in DHC W/O tanh |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx1 W/O tanh | 56.8 | 75.0 | 55.3 | 33.4 | 72.9 | 85.4 | 57.1 | 62.3 |
| OLMo-1B-DHCx2 W/O tanh | 63.0 | 74.0 | 57.1 | 34.6 | 73.5 | 86.0 | 58.2 | 63.8 |
| OLMo-1B-DHCx4 W/O tanh | 61.2 | 80.0 | 57.5 | 33.6 | 75.5 | 85.8 | 56.9 | 64.4 |
| OLMo-1B-DHCx8 W/O tanh | 61.1 | 75.0 | 57.6 | 35.4 | 73.8 | 85.2 | 58.5 | 63.8 |
| Scaling n in DHC |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx1 | 59.7 | 74.0 | 55.5 | 33.6 | 73.5 | 85.4 | 54.5 | 62.3 |
| OLMo-1B-DHCx2 | 59.7 | 73.0 | 56.7 | 34.0 | 74.7 | 85.2 | 57.9 | 63.0 |
| OLMo-1B-DHCx4 | 59.8 | 79.0 | 58.1 | 32.4 | 74.3 | 86.1 | 57.1 | 63.8 |
| OLMo-1B-DHCx8 | 56.8 | 75.0 | 58.0 | 34.4 | 73.8 | 84.2 | 57.3 | 62.8 |
| Scaling n in SHC |  |  |  |  |  |  |  |  |
| OLMo-1B-SHCx2 | 59.1 | 77.0 | 56.6 | 35.4 | 74.2 | 85.3 | 56.4 | 63.4 |
| OLMo-1B-SHCx4 | 59.3 | 77.0 | 56.7 | 34.0 | 74.3 | 86.6 | 57.1 | 63.6 |
| Non-trainable $\mathcal{WC}$ |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx4 | 60.5 | 78.0 | 56.2 | 34.0 | 73.5 | 86.0 | 55.8 | 63.4 |
| OLMo-1B-DHCx4 W/O tanh | 59.1 | 72.0 | 56.8 | 35.0 | 73.3 | 86.0 | 55.5 | 62.5 |
| Non-trainable $\mathbf{B}$ |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx4 | 59.5 | 77.0 | 57.9 | 33.8 | 73.3 | 85.6 | 56.6 | 63.4 |
| OLMo-1B-DHCx4 W/O tanh | 60.4 | 74.0 | 57.6 | 34.0 | 74.9 | 86.7 | 57.5 | 63.6 |

**Table 13: Losses of V2 validation sets for 1B Model.**
| Method | 4chan | c4_100_domains | c4_en | gab | ice | m2d2_s2orc | m2d2_wiki | manosphere | mc4_en | pile | ptb | twitterAAE | wikitext_103 | avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OLMo-1B | 2.319 | 2.615 | 2.762 | 3.364 | 2.719 | 3.085 | 2.594 | 3.028 | 2.522 | 2.250 | 2.953 | 3.672 | 2.657 | 2.811 |
| Scaling n in DHC W/O tanh |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx1 W/O tanh | 2.320 | 2.626 | 2.773 | 3.379 | 2.725 | 3.102 | 2.609 | 3.036 | 2.531 | 2.264 | 2.948 | 3.703 | 2.672 | 2.822 |
| OLMo-1B-DHCx2 W/O tanh | 2.311 | 2.600 | 2.749 | 3.362 | 2.700 | 3.069 | 2.583 | 3.015 | 2.503 | 2.231 | 2.908 | 3.635 | 2.625 | 2.792 |
| OLMo-1B-DHCx4 W/O tanh | 2.295 | 2.591 | 2.735 | 3.344 | 2.686 | 3.056 | 2.562 | 3.005 | 2.492 | 2.221 | 2.898 | 3.632 | 2.610 | 2.779 |
| OLMo-1B-DHCx8 W/O tanh | 2.292 | 2.589 | 2.734 | 3.350 | 2.685 | 3.060 | 2.562 | 3.006 | 2.492 | 2.218 | 2.878 | 3.628 | 2.609 | 2.777 |
| Scaling n in DHC |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx1 | 2.323 | 2.625 | 2.775 | 3.376 | 2.728 | 3.090 | 2.606 | 3.037 | 2.533 | 2.262 | 2.961 | 3.652 | 2.678 | 2.819 |
| OLMo-1B-DHCx2 | 2.309 | 2.608 | 2.754 | 3.367 | 2.703 | 3.061 | 2.587 | 3.022 | 2.509 | 2.237 | 2.930 | 3.704 | 2.636 | 2.802 |
| OLMo-1B-DHCx4 | 2.290 | 2.591 | 2.738 | 3.354 | 2.683 | 3.064 | 2.564 | 3.005 | 2.492 | 2.218 | 2.890 | 3.641 | 2.611 | 2.781 |
| OLMo-1B-DHCx8 | 2.295 | 2.591 | 2.739 | 3.353 | 2.684 | 3.054 | 2.567 | 3.008 | 2.493 | 2.219 | 2.876 | 3.631 | 2.608 | 2.778 |
| Scaling n in SHC |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-SHCx2 | 2.307 | 2.610 | 2.757 | 3.360 | 2.703 | 3.063 | 2.587 | 3.023 | 2.511 | 2.238 | 2.933 | 3.643 | 2.643 | 2.799 |
| OLMo-1B-SHCx4 | 2.300 | 2.603 | 2.751 | 3.357 | 2.692 | 3.062 | 2.580 | 3.018 | 2.504 | 2.232 | 2.899 | 3.653 | 2.627 | 2.791 |
| Non-trainable WC |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx4 | 2.312 | 2.608 | 2.752 | 3.357 | 2.700 | 3.077 | 2.583 | 3.024 | 2.508 | 2.238 | 2.959 | 3.678 | 2.636 | 2.802 |
| OLMo-1B-DHCx4 W/O tanh | 2.308 | 2.609 | 2.755 | 3.357 | 2.710 | 3.100 | 2.585 | 3.025 | 2.510 | 2.240 | 2.945 | 3.663 | 2.644 | 2.804 |
| Non-trainable Beta |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx4 | 2.296 | 2.594 | 2.742 | 3.348 | 2.684 | 3.051 | 2.569 | 3.008 | 2.497 | 2.221 | 2.917 | 3.627 | 2.622 | 2.783 |
| OLMo-1B-DHCx4 W/O tanh | 2.295 | 2.592 | 2.739 | 3.347 | 2.689 | 3.066 | 2.567 | 3.005 | 2.496 | 2.222 | 2.887 | 3.638 | 2.606 | 2.781 |

**Table 14: Perplexities of V2 validation sets for 1B Model.**
| Method | 4chan | c4_100_domains | c4_en | gab | ice | m2d2_s2orc | m2d2_wiki | manosphere | mc4_en | pile | ptb | twitterAAE | wikitext_103 | avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OLMo-1B | 10.167 | 13.666 | 15.829 | 28.901 | 15.166 | 21.860 | 13.377 | 20.651 | 12.453 | 9.488 | 19.161 | 39.328 | 14.251 | 18.023 |
| Scaling n in DHC W/O tanh |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx1 W/O tanh | 10.174 | 13.815 | 16.004 | 29.328 | 15.259 | 22.231 | 13.587 | 20.823 | 12.562 | 9.620 | 19.071 | 40.580 | 14.462 | 18.270 |
| OLMo-1B-DHCx2 W/O tanh | 9.920 | 13.340 | 15.412 | 28.340 | 14.676 | 21.243 | 12.965 | 20.181 | 12.079 | 9.219 | 18.129 | 37.768 | 13.594 | 17.451 |
| OLMo-1B-DHCx4 W/O tanh | 10.082 | 13.470 | 15.625 | 28.848 | 14.882 | 21.521 | 13.234 | 20.392 | 12.217 | 9.312 | 18.321 | 37.905 | 13.806 | 17.663 |
| OLMo-1B-DHCx8 W/O tanh | 9.897 | 13.313 | 15.387 | 28.488 | 14.658 | 21.337 | 12.960 | 20.200 | 12.084 | 9.185 | 17.782 | 37.650 | 13.592 | 17.425 |
| Scaling n in DHC |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx1 | 10.210 | 13.810 | 16.031 | 29.265 | 15.302 | 21.986 | 13.539 | 20.847 | 12.584 | 9.606 | 19.326 | 38.564 | 14.555 | 18.125 |
| OLMo-1B-DHCx2 | 10.061 | 13.568 | 15.710 | 29.002 | 14.925 | 21.349 | 13.284 | 20.524 | 12.294 | 9.362 | 18.727 | 40.592 | 13.957 | 17.950 |
| OLMo-1B-DHCx4 | 9.877 | 13.344 | 15.430 | 28.624 | 14.633 | 21.410 | 13.006 | 20.186 | 12.080 | 9.189 | 18.102 | 38.136 | 13.606 | 17.509 |
| OLMo-1B-DHCx8 | 9.922 | 13.346 | 15.467 | 28.591 | 14.640 | 21.198 | 13.025 | 20.240 | 12.097 | 9.196 | 17.749 | 37.743 | 13.570 | 17.445 |
| Scaling n in SHC |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-SHCx2 | 10.046 | 13.601 | 15.753 | 28.782 | 14.931 | 21.391 | 13.294 | 20.562 | 12.319 | 9.374 | 18.791 | 38.212 | 14.060 | 17.778 |
| OLMo-1B-SHCx4 | 9.977 | 13.507 | 15.655 | 28.691 | 14.766 | 21.372 | 13.194 | 20.457 | 12.234 | 9.315 | 18.149 | 38.569 | 13.836 | 17.671 |
| Non-trainable WC |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx4 | 10.054 | 13.587 | 15.721 | 28.689 | 15.023 | 22.186 | 13.263 | 20.594 | 12.310 | 9.390 | 19.016 | 38.959 | 14.070 | 17.912 |
| OLMo-1B-DHCx4 W/O tanh | 10.092 | 13.566 | 15.666 | 28.704 | 14.873 | 21.696 | 13.242 | 20.579 | 12.276 | 9.377 | 19.272 | 39.570 | 13.963 | 17.914 |
| Non-trainable Beta |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx4 | 9.927 | 13.354 | 15.475 | 28.417 | 14.722 | 21.454 | 13.021 | 20.185 | 12.135 | 9.228 | 17.932 | 38.005 | 13.553 | 17.493 |
| OLMo-1B-DHCx4 W/O tanh | 9.932 | 13.386 | 15.510 | 28.436 | 14.641 | 21.130 | 13.051 | 20.253 | 12.142 | 9.220 | 18.478 | 37.610 | 13.766 | 17.504 |

**Table 15: Losses of V3 validation sets for 1B Model.**
| Method | c4_en | dolma_books | dolma_common-crawl | dolma_pes2o | dolma_reddit | dolma_stack | dolma_wiki | ice | m2d2_s2orc | pile | wikitext_103 | avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OLMo-1B | 2.702 | 2.906 | 2.722 | 2.333 | 2.980 | 1.041 | 2.487 | 2.715 | 3.199 | 2.232 | 2.663 | 2.544 |
| Scaling n in DHC W/O tanh |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx1 W/O tanh | 2.712 | 2.928 | 2.732 | 2.349 | 2.991 | 1.045 | 2.499 | 2.721 | 3.219 | 2.246 | 2.677 | 2.556 |
| OLMo-1B-DHCx2 W/O tanh | 2.676 | 2.880 | 2.698 | 2.306 | 2.961 | 1.024 | 2.456 | 2.682 | 3.174 | 2.204 | 2.617 | 2.516 |
| OLMo-1B-DHCx4 W/O tanh | 2.689 | 2.890 | 2.706 | 2.317 | 2.969 | 1.030 | 2.471 | 2.697 | 3.200 | 2.213 | 2.633 | 2.529 |
| OLMo-1B-DHCx8 W/O tanh | 2.674 | 2.876 | 2.695 | 2.303 | 2.960 | 1.022 | 2.454 | 2.680 | 3.176 | 2.200 | 2.616 | 2.514 |
| Scaling n in DHC |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx1 | 2.714 | 2.927 | 2.732 | 2.346 | 2.991 | 1.045 | 2.499 | 2.723 | 3.211 | 2.245 | 2.683 | 2.556 |
| OLMo-1B-DHCx2 | 2.694 | 2.901 | 2.712 | 2.321 | 2.976 | 1.032 | 2.478 | 2.699 | 3.202 | 2.218 | 2.642 | 2.534 |
| OLMo-1B-DHCx4 | 2.675 | 2.876 | 2.697 | 2.301 | 2.962 | 1.021 | 2.455 | 2.679 | 3.176 | 2.200 | 2.617 | 2.515 |
| OLMo-1B-DHCx8 | 2.677 | 2.880 | 2.701 | 2.304 | 2.964 | 1.022 | 2.456 | 2.680 | 3.177 | 2.201 | 2.614 | 2.516 |
| Scaling n in SHC |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-SHCx2 | 2.698 | 2.907 | 2.718 | 2.325 | 2.980 | 1.032 | 2.479 | 2.700 | 3.198 | 2.221 | 2.650 | 2.537 |
| OLMo-1B-SHCx4 | 2.689 | 2.892 | 2.711 | 2.315 | 2.973 | 1.028 | 2.472 | 2.688 | 3.195 | 2.214 | 2.633 | 2.528 |
| Non-trainable WC |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx4 | 2.695 | 2.903 | 2.716 | 2.324 | 2.978 | 1.035 | 2.477 | 2.705 | 3.201 | 2.221 | 2.649 | 2.537 |
| OLMo-1B-DHCx4 W/O tanh | 2.692 | 2.899 | 2.714 | 2.321 | 2.976 | 1.032 | 2.474 | 2.695 | 3.189 | 2.219 | 2.641 | 2.532 |
| Non-trainable Beta |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx4 | 2.679 | 2.880 | 2.697 | 2.306 | 2.961 | 1.025 | 2.458 | 2.684 | 3.188 | 2.204 | 2.612 | 2.518 |
| OLMo-1B-DHCx4 W/O tanh | 2.681 | 2.886 | 2.702 | 2.306 | 2.966 | 1.024 | 2.462 | 2.680 | 3.183 | 2.204 | 2.628 | 2.520 |

**Table 16: PPL of V3 validation sets for 1B Model.**
| Method | c4_en | dolma_books | dolma_common-crawl | dolma_pes2o | dolma_reddit | dolma_stack | dolma_wiki | ice | m2d2_s2orc | pile | wikitext_103 | avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OLMo-1B | 14.908 | 18.289 | 15.216 | 10.305 | 19.686 | 2.832 | 12.026 | 15.098 | 24.503 | 9.319 | 14.334 | 14.229 |
| Scaling n in DHC W/O tanh |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx1 W/O tanh | 15.064 | 18.699 | 15.356 | 10.473 | 19.909 | 2.843 | 12.167 | 15.191 | 25.013 | 9.451 | 14.540 | 14.428 |
| OLMo-1B-DHCx2 W/O tanh | 14.531 | 17.817 | 14.857 | 10.038 | 19.323 | 2.783 | 11.662 | 14.608 | 23.906 | 9.061 | 13.694 | 13.844 |
| OLMo-1B-DHCx4 W/O tanh | 14.711 | 17.996 | 14.975 | 10.146 | 19.479 | 2.800 | 11.830 | 14.839 | 24.524 | 9.146 | 13.917 | 14.033 |
| OLMo-1B-DHCx8 W/O tanh | 14.494 | 17.749 | 14.813 | 10.000 | 19.306 | 2.779 | 11.630 | 14.587 | 23.948 | 9.021 | 13.684 | 13.819 |
| Scaling n in DHC |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx1 | 15.093 | 18.675 | 15.360 | 10.442 | 19.909 | 2.845 | 12.174 | 15.225 | 24.810 | 9.436 | 14.632 | 14.418 |
| OLMo-1B-DHCx2 | 14.794 | 18.190 | 15.061 | 10.191 | 19.612 | 2.806 | 11.915 | 14.870 | 24.589 | 9.187 | 14.043 | 14.114 |
| OLMo-1B-DHCx4 | 14.514 | 17.743 | 14.829 | 9.989 | 19.343 | 2.776 | 11.650 | 14.573 | 23.948 | 9.028 | 13.689 | 13.826 |
| OLMo-1B-DHCx8 | 14.546 | 17.807 | 14.889 | 10.011 | 19.366 | 2.779 | 11.653 | 14.579 | 23.964 | 9.030 | 13.653 | 13.843 |
| Scaling n in SHC |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-SHCx2 | 14.854 | 18.293 | 15.150 | 10.230 | 19.689 | 2.807 | 11.934 | 14.876 | 24.478 | 9.214 | 14.150 | 14.152 |
| OLMo-1B-SHCx4 | 14.717 | 18.028 | 15.049 | 10.121 | 19.550 | 2.796 | 11.846 | 14.699 | 24.407 | 9.155 | 13.912 | 14.025 |
| Non-trainable WC |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx4 | 14.810 | 18.224 | 15.120 | 10.215 | 19.650 | 2.816 | 11.902 | 14.954 | 24.552 | 9.220 | 14.135 | 14.145 |
| OLMo-1B-DHCx4 W/O tanh | 14.756 | 18.160 | 15.095 | 10.191 | 19.613 | 2.806 | 11.868 | 14.807 | 24.273 | 9.203 | 14.021 | 14.072 |
| Non-trainable Beta |  |  |  |  |  |  |  |  |  |  |  |  |
| OLMo-1B-DHCx4 | 14.574 | 17.820 | 14.840 | 10.038 | 19.320 | 2.787 | 11.677 | 14.647 | 24.233 | 9.059 | 13.621 | 13.874 |
| OLMo-1B-DHCx4 W/O tanh | 14.593 | 17.926 | 14.904 | 10.032 | 19.405 | 2.785 | 11.724 | 14.588 | 24.108 | 9.060 | 13.839 | 13.906 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: - The author mentioned other baselines such as Altup and ResiDual had gains in the early stages of training, Can the author show the full loss curve of OLMo-1B-ResiDual and OLMo-1B-Altup×2 in Table4?
soundness: 3
strengths: - The paper provides a clear and systematic extension to residual connections named Dynamic Hyper-Connections (DHC), where residual could be consider a static hyperconnection, addressing the trade-off between gradient vanishing and representation collapse.
- Experimenttal results demonstrated effectiveness of DHC across diverse domains, including LLM pretraining and vision tasks.
summary: The paper introduces Hyper-Connections, a novel extension to residual connections that dynamically adjusts the strength of connections between layers in deep neural networks. This method addresses the limitations of traditional residual connections, such as gradient vanishing and representation collapse, by introducing Depth-Connections and Width-Connections thus enabling both cross-layer and intra-layer interactions. A dynamic variant of residual connection named Dynamic Hyper-Connections (DHC), further adapts connection strengths based on input. The approach is evaluated extensively across pretraining of large language models (LLMs), Mixture-of-Experts (MoE) models on vision tasks. Experimental results demonstrate significant improvements in training stability, convergence speed, and model performance on various benchmarks, highlighting Hyper-Connections as a general-purpose enhancement for neural architectures.
weaknesses: - There seem a lack of comparsion to a fullt enabled depth-connections and width-connections (DenseNet style) where all of the connection in Figure 2 are enabled and learnable.
- The main result focus on LLM and downstream peformance on langugage tasks, the result on Vision task in the appendix seem to demonstrate less gain compare to langugage tasks, can the author elebrate more on this?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Why is expansion rate = 1 worse than no hyper-connection, do you have an intuition ?

- Do these findings generalize to other types of architectures such as ResNets ?

- Line 345 typo: “Static”
soundness: 3
strengths: - There is a clear signal that incorporating hyper-connections in LLMs architectures, without any other modification, improves the training loss for a given number of tokens, and boosts performance on downstream metrics. This result is validated for both dense and MOE architectures.

- Hyper-connexions help reduce training instabilities. This is clear by looking at Figure 5 and 6, the training curves of the models with hyper-connections are smoother and do not have spikes, which is a major advantage for training large models.

- The author did a thorough analysis of the learned connections patterns, with nice visualizations.

- The results generalizes to the vision modalities with experiments on image generation and classification. Hyper-connections seem to be a general improvement for the transformer architecture.
summary: This paper presents hyper-connexions, a new neural network architectural improvement which consists in dynamically adjusting the residual connections between layers, effectively managing the trade-off between vanishing gradients and feature collapse. Many experiments with LLMs and vision models demonstrate the effectiveness of hyper-connections to improve stability of training and downstream performance. Various hyper-connections patterns are also studied in depth, with thorough ablations and visualizations.
weaknesses: - The main concern I have with this paper is the computational impact of replicating the activations of the network $n$ times for hyper-connections. There is no study on the computational impact both in terms of running time and memory usage. The authors mention Line 394 that “Both methods expand the hidden size by n times with negligible computational overhead” but it is not shown with a proper experiment on the throughput, overall running time, and peak memory usage. Also, it seems that n=1 performs worse than no hyper-connection, so if n>=2 is necessary, and the memory usage is high, it is necessary to study the trade-off between downstream performance, stability and computational cost.

- Although the signal is promising, a full experiment with scaling the number of tokens beyond 500B will be necessary to fully validate the approach. Not asking for this experiment, but current best LLMs are trained on many more tokens and exhibit much better performance than the number reported. I would be curious to see if hyper-connections are useful for training state-of-the-art LLMs.

- In section 3.2 several non-learnable patterns are presented but are not tried in practice. It is not clear whether learning the hyper-connection patterns is really better that having simple fixed patterns, and an analysis on that would be interesting.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Some questions that may add more value to the paper:
1. With Dynamic Hyperconnections, there is a possibility for redundant connections (when strength becomes 0) how does these cases affect the seesaw effect of gradient vanishing and representation collapse.
2. Do DHC connections behave similar for images belonging to the same class in case of image classification?

Things I expect from the rebuttal are actions on points 2 and 3.
soundness: 4
strengths: 1. The results on LLM benchmarks and losses suggest a better balance between vanishing gradients and representation collapse.

2. Section 4.5 discusses the effect of hyperconnections, which displays that hyperconnections eliminate input embeddings from the output, form parallel blocks which have less reliance on each other increasing chances for unique representations. 

3.Parallel block formation is particularly important as similar layers in a transformer block tends to learn similar representations. The ability to parallely form blocks dynamically based on input reduces the chances for similar representations enabling better models.

4. Assessing the hyperconnections can reveal some internal logic of the neural network. For instance, we may find that a set of classes follow similar paths compared to other classes.

5. The paper is well detailed and mathematically sound, proofs, hyperparameters and other implementation details are discussed appropriately.
summary: The paper introduced hyperconnection - width and depth connections with learnable strengths. Hyperconnections are proposed to find a balance between the seesaw effect noticed between vanishing gradients and representation collapse. The input is split into n copies for different connections, and these different copies are added together before layer computation.  The paper also introduced dynamic hyperconnections that dynamically update the strength of connections based on inputs. The paper performs comprehensive analysis of effect of hyperconnections on OLMO and OLMOE and image generation and classifcation problems and investigates its effect with prominent residual connections.
weaknesses: 1. The main drawback is when creating $n$ copies,  it leads to a considerable amount of increase in memory, though the burden can be reduced through engineering, the impact is yet to be known.

2. If the goal of creating multiple copies is just to make sure multiple depth connections can be modelled parallelly, is creating such copies actually necessary? Can't a single copy be used with different residual strengths? The only difference would be in gradient computation , were additional terms for each depth connection would be added to the singular copy. Can an ablation be conducted between DHC ($n=4$ and $n=1$) and SHC ($n=4$ an $n=1$) to show the importance and need for additional copies. What I am particularly interested is the advantages with updating the gradients into different copies and then adding them before every layer computation verses  updating all the gradients with a single copy. If there is any other specific need for expansion please free to refute this point.

3. Figure 2 caption improvement. The figure can discuss the diagram better, whats $\beta$, and need to add $\alpha_{1,0}$ and $\alpha_{2,0}$ be explained in the diagram itself. This is important as Figure 2 is the central figure that tries to encompass hyperconnections therefore adding these information would make it more clearer for readers.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In the introduction, you stated that "Post-Norm applies normalization operations after the output of each residual block, weakening the
"strength" of residuals." I don't think it is correct, since post-norm applies normalization after the summation of the skip and residual branch, so it shouldn't weaken the strength of the residuals.

2. Can you also should the flops count for your model and the original model?
soundness: 2
strengths: The experimental results seems good enough to claim the hyper-connections is better than the original architectures.
summary: The paper introduces hyper-connections, a approach that aims to address the limitations of residual connections of transformers architectures.
The hyper-connections approach introduce depth-connections and width-connections, allowing a more customizable interaction between layers. Depth-connections are weighted connections across different layers, while width-connections facilitate two layers at the same depths. More importantly the hyper-connections are learned, so it improves model adaptability.
Models with hyper-connections have significant performance improvements over the original architectures.
weaknesses: 1. The presentation is horrible, it is really hard to understand what is model is doing. Probably a better figure or even pseudocode should be provided to explain the methods.
2. Although the experimental results are good, there is no information on how much extra computation is need to achieve such good results. I suspected that the extra performance is highly due to the extra parameters or extra computation it needed. If you make your model have the same floats, probably you could see that the performance is really similar to the original transformers.
3. The ordering of the paper is horrible, there is almost no explanation why you do each thing in the paper.

Meta-review: The proposed hyper-connections (HCs) are a form of learned architecture that can be optimized to connect different representations across depth and width by summation. In this way they are an extension of residual connections. In addition, HCs can be conditioned on the input to vary these connections during inference for dynamic hyper-connections (DHCs). The main paper concerns the applications of HCs to LLMs, working with open-source OLMoE models, and shows improvements to the loss, data efficiency, and computationally efficiency relative to other recent approaches to expand architectures. In the appendix additional results on vision are shown, where there is still gain, but the effect is more marginal.

Strengths:

- experiments show empirical improvement for pre-training and downstream tasks on text and visual data
- the cost in computation time, memory, and parameters is measured and reasonable (following the rebuttal and revision)
- the idea and the implementation are clear (and have been clarified in the rebuttal phase)

Weaknesses:

- The related work is limited and the scholarship is shallow in its inclusion of only the most recent papers. As a project on connecting across depth and width, there is a well-developed body of work including DenseNets, most practically, and other models such as FractalNets and HighwayNets.
- The work claims similar improvement across text and vision, but more precisely there is a larger improvement for LLMs than current vision models. This is tempered by equally good or still improved results on vision however.
- The organization of the presentation of the paper challenged comprehension, as evidenced by shared questions across reviewers and multiple comments on these points. 

Missing: Most critically, the submission is missing discussion (and ideally experiments) for certain related works like DenseNet and it is inadequately organized for ease of comprehension. The issues with exposition however have begun to be addressed in the rebuttal and revision, and could be dealt with in the final revision. Additional discussion is likewise feasible, and would be an acceptable resolution of the related work.

Decision: This work is borderline. The four expert reviewers vote for acceptance (VJDJ: 8, HHYn: 6, DUBh: 6, ) and rejection (fvae: 5). The meta-reviewer sides with acceptance because of the strength of the empirical results for task performance and computation, the satisfaction of the majority of the reviewers and the points addressed in the rebuttal, and the generality of the results across multiple tasks, datasets, and modalities. However, the meta-reviewer cautions that the lack of discussion and experimentation on related work about more sophisticated handling of depth (DenseNets, FractalNets, ...) remains a negative. For broader impact and improved reception of the proposed technique it may be advisable to incorporate more material about these prior works, as has likewise been suggested by the reviewers.

Note: The meta-reviewer acknowledges the confidential comment from the authors. The decision reflects the paper, reviews, rebuttal, and the full discussion.

Decision: Accept (Poster)