Hyper-Connections
Authors: Defa Zhu
Hongzhi Huang
Zihao Huang
Yutao Zeng
Yunyao Mao
Banggu Wu
Qiyang Min
Xun Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Network Architecture
Residual Connections
LLMs
Pre-training
Primary area: foundation or frontier models, including LLMs

Abstract: We present hyper-connections, a simple yet effective method that can serve as an alternative to residual connections. This approach specifically addresses common drawbacks observed in residual connection variants, such as the seesaw effect between gradient vanishing and representation collapse. Theoretically, hyper-connections allow the network to adjust the strength of connections between features at different depths and dynamically rearrange layers. We conduct experiments focusing on the pre-training of large language models, including dense and sparse models, where hyper-connections show significant performance improvements over residual connections. Additional experiments conducted on vision tasks also demonstrate similar improvements. We anticipate that this method will be broadly applicable and beneficial across a wide range of AI problems.

Published as a conference paper at ICLR 2025
HYPER-CONNECTIONS
Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, Xun Zhou
Geed-Foundation-Mouu Team, ByteDance
1zhudefa, huanghongzhi. 51, huangzihao notabot, yut zeng, maoyunyao. myy, wubanggu, yangiyang, zhouxun| @bytedance com
ABSTRACT
We present hyper- soper-connections, simple yet effective method that can serve an alternative residual connections. This approach specifically addresses common drawbacks observed in residual connection variants, such the seesaw effect between gradient vanishing and representation collapse. Theoretically, hyper- connections allow the network to adjust the strength of connections between fea- tures at different depths and dynamically rearrange layers. We conduct experiments focusing the pre-training of large language models, including dense and sparse models, where residual connections. Additional experiments conducted vision tasks also demonstrate similar improvements. We anticipate that this method will be broadly applicable and beneficial across wide range of Al problems.
show significant performance improvements
INTRODUCTION
Training Loss
C4 en val. loss
HellaSwag Acc.
ARC Challenge Acc.
2,400
2375
OLINE 13 DHCK4
OLANWE118-78
2.85
2.80
OLMuE-18-11
12-78
Accuracy (96)
67.3
70.0
Accuracy (96)
45.0
12.5
OLMoE-18-75
2350
36.0
2.325
2.70
62.5
37.3
0:007
60.0
OLMAC-10-70
35.0
2273
2.63
0.
57.3
10-78-DHCH4
32.5
1000
310
500
Tokens (Billions)
Takens (Bil (ollions)
Takens
Takens (Billions)
Figure 1: The performance of the baseline model 7LMoE-1B-77 and the model with hyper- connections,
(1) and (2) show the training loss (0.99 EMA smoothed) and the C4-en validation loss, respectively. Our method converges 1.8 times faster compared to the baseline and maintains a significant advantage the 500B tokens. (3) and show the accuracy curves on Hella and ARC-Challenge, demonstrating the superior performance of the LLMMEE-BB-7B-HOCE model.
Deep learning has achieved tremendous success across various domains, where residual connections (He al. 2016) have been instrumental contemporary neural network architectures, including transformers and CNNs. Residual connections help mitigate the problem of gradient vanishing, enabling the effective training of very deep networks. However, is important acknowledge that residual connections are not infallible solutions and still present limitations that remain unresolved.
The two main variants of residual connections, Pre-Norm and Post-Norm, each make distinet trade-offs between gradient vanishing and representation collapse. Pre-Norm applies normalization operations to the input before each residual block, effectively addressing the problem of gradient vanishing (Bengio al. 1994: Glorot & Bengio 2010). However, it can also lead to the issue of collapse in deep representations (Liu et al.) 20200). where hidden features in deeper layers become highly similar, diminishing the contribution of additional layers their number increases. In contrast, Post-Norm applies normalization after the output of each residual block, reducing the influence of a hidden state subsequent layers. This approach can alleviate the issue of representation collapse but

---

Published as a conference paper at ICLR 2025
rotales, weights of Iypreccomatiion
chidden sectors
laypr
caltention ITN
Tayer
layer
layer
layer
Rypercoomentiooss
Depth-cocarction
Figure 2: Hyper-connections (HC) with an expansion rate of n = 2. (a) Residual connections.
(b) Hyper-comnections: 31. 82, og,u, Q0,1 01,0, 01,1, Qg,1, and 022, are learnable scalars or scalars predicted by the network, depending on the specific HC version. These connections enable lateral information exchange and vertical integration of features across depths. The Transformer with HC is shown in Fig. 7 They can be decoupled into depth--onnections and width-connections. (c) Depth-connections perform a weighted sum between the layer output and the hidden vector h1. (d) Width-connection. allow information exchange between the hidden vectors hy and hg.
also reintroduces the problem of vanishing gradients. The vanishing gradient and the representation collapse are like two ends of seesaw, with these two variants making respective trade-offs between these issues. The key issue is that residual connections, including both Pre-Norm and Post-Norm variants, predefine the strength of connections between the output and input within layer.
Driven by the limitations of residual connections, important question arises: Can neural networks autonomously learn the optimal strength of connections to improve performance? To address this, we propose dyper-connectionn (HC), which lead to significantly improved performance with negligible increase in computation and parameters. We will show that both Post- Norm and Pre-Norm variants can be expressed specific non- trainable forms of hyper-connections, as discussed in 3.1
cos(ba h
The core idea of hyper-connections (HC) is propose learnable depth-connections and width-connections, as depicted in Fig (b). These connections flexibly integrate features vertically across depths, compared to the residual connections shown in general-
Pre-Norm Hyper- Connectio
(ig (a). Depthhconnecttons can be considered ized residual connections, assigning weights to the connections between the inputs and outputs of each layer. To enable the network to model different depth-conmections simultaneously, we expand the network's 's input into n copies, each having its depth connection, shown in Fig. 2 (b). This design allows multiple hidden vectors to reserve multiple patterns con-
10 20
25 30
Layer Index
Figure 3: Cosine similarity be- tween the input of the current and the previous layers for the OLMo 1B models (Groeneveld et al. 2024). The curve represents the median of similarity, while the shaded area indicates the range be- tween the 5th and 95th percentiles. The red curve shows the model with Pre-Norm, and the blue curve shows that with hyper-comnections.
necting preceding layers, shown in $ 4.5 Moreover, establish width connections between the n. hidden vectors, al- lowing information exchange between hidden vectors within the same layer, as shown in Fig. (b). We argue that (> 1) hidden states necessary. As analyzed in Appendix F the seesaw effect persists when 1, and experiments show that it does not improve performance, shown in Fig. [ In contrast, when n > hyper-connections can not only learn to adjust the
strength of residuals but also rearrange layers, either sequentially or in parallel, as discussed in 3.2 To further enhance flexibility, introduce dynamic hyper-connections (DHC), enabling the network to adjust connection weights according to the input. Notably, although HC seem to increase the network's width by times, the additional parameters and computational cost are almost negligible, as analyzed Appendix B The Transformer with HC is shown Fig. 17
Our research, primarily centered large language models (LLMs) pre-training, also extends to visual generation and classification tasks. Using Pre-Norm as a baseline, we demonstrate the significant benefits of hyper-connections, including 1B and 7B dense models well as 7B MoE models, as
2

---

Published as a conference paper at ICLR 2025
detailed in $ The benefits are particularly prominent for OLMoE (Muennighoff al. 2024 as presented in Fig I The model utilizing DHC converges 1.8 times faster and shows an improvement of 6 points ARC-Challenge compared to the bascline trained with 500 B tokens. According to visualization analysis, as shown in Fig 3 the baseline model tends toward representation collapse, characterized by high similarity between features of adjacent layers. In contrast, models with HC exhibit significantly lower similarity between features across adjacent layers and wider range of similarities. This suggests that HC enhance the impact of each layer. Further discussion is provided in $4.5 and in Appendix[P [F] These compelling pieces of evidence demonstrate the generality of the hyper-connections principle, and we anticipate their applicability in numerous other Al challenges.
METHOD
2.1
STATIC HYPEE-CONNECTTION
Consider the hidden vector h* E Rd (or h E IRd11 as the input to the k-th layer, with the initial imput h" to the network. Initially, he E R is replicated times form the initial hyper hidden matrix H (ho h
h²) Raxd Here, is the expansion rate. For the k-th layer, the input consists of the hyper hidden matrix from the previous layer H the last hyper hidden matrix row-wise obtain the required hidden vector, which is then passed through final projector to produce the final output of the network (i.e., normalization layer and an unembedding layer in transformers). To simplify the notation in subsequent analysis, we omit the layer index and simply denote the h.)
(h/-1 h
h Raxd, Finally,
hyper-hidden matrix as (hı h2
The hyper-comee conpectonnsss (HC) can be represented by matrix HC, where each element defines the connection weight. The matrix is structured as follows:
0
B
32
On
011,0
01,1
01,2
011
HCC
01xl Am
B
022,0
Q221
0222
022,
E R
(+++))((++))
(1)
A,
On,0
0n,1
0nn2
On,a
Consider a network layer T, integrates self-attention layers and feed-forward networks within
transformers. The output of the HC, denoted by H, can be simply formulated as follows:
A HC(T, H) = BT A A.
We use Am as weights to perform a weighted sum on the input = (h1 h2 the input ha of the current layer T, which is given by:
h to obtain
hj Am TH
While A, is used to connect H and map it to hyper hidden matrix H', as shown below:
H' - H.
(4)
Subsequently, the output is given by:
A TT=B"Thha' H'.
(5)
The depthh-onnections can be decoupled as the following matrix, which shown at Fig (a):
DC
B diag(A,))
B1
82
3n
2xn
a1,D
02
On,
where the first row B represents the weights of the output of the current layer T. and the last row diag(A,) represents the weights of the input. We use diag(A,) to represent the flatten vector of the diagonal entries of Ar-
3

---

Published as a conference paper at ICLR 2025
The width-connections matrix can be defined as follows, which is shown at Fig 2 (b):
WC (Am Ar) E Rax(++))
The algorithm that employs hyper-comeection is presented in Algorithm
2.2 DYNAMIC HYPER-CONNECHI CONNECTIONS
The entries of HC can dynamically depend on the input H, which the matrix representation of dynamic hyper-comnections (DHC) is defined as follows:
0 01x1 Am (H)
B A, (H)
HC(H)-
(8)
Similarly, given layer T and input H, we obtain the output of the DHC follows:
A HC(H) H).
In practice, combine the dynamic and static matrices achieve DHC. The dynamic parameters obtained through a linear transformation. To stabilize the training process, we introduce normalization before the linear transformation and apply the tanh activation function after it, scaling by small initial learnable factor. The following equations detail how these dynamic parameters are computed:
H norm H)
(10)
B(H) 89 tanh HW,,) 3) + B RIxx
(11)
Am (H) Sa tanh HW m) Am Rax1
(12)
(H Sa o ttanh(HW, + A, Rax
(13)
Our experiments in $ demonstrate that dynamic hyper-connections outperform static hyper- connections in language modeling tasks. The PyTorrch implementations for both the static and dynamic variants of hyper-comnections are detailed in Algorithm 2 and
2.3 INITIALIZATION
In order to make the initialization of the hyper-connections equivalent to the Pre-Norm residual connections, we adopt the following initialization strategy. The dynamic parameters W W m- and W, in Eqs. II 12 and 13 are initialized to O, while the static matrices are initialized as follows:
O1xl Am
)
B
01x1
11x8
(14)
A."
ex mod
Caxn
where k is the index of the layer. mod denotes the modulo operation.
3 WHY
In this section, we elucidate the rationale behind hyper-connections. We explore how variants of residual comnections, namely Pre-Norm and Post-Norm, can be viewed as non-trainable hyper- connections, and introduce the concept of eqquential-paralle duality, demonstrating how hyper- connections can dynamically optimize layer arrangements to enhance network performance. visulize analysis of hyper-connections through an unfolded view is discussed in 4.5
3.1 RESIDUAL CONNECTIONS AS NON-TRAINABLE TYYPE-COONECCTION
The Pre-Norm and Post-Norm residual connections can be represented as the following hyper- connections matrices with expansion rate
4

---

Published as a conference paper at ICLR 2025
0
Votto3+20..
(i 1).
(15) HC PostNorm
(16)
HC PreNorm
Vot+o?+20..
where and 0a denote the standard deviations of the input and output of the neural network layer, respectively, and Gio is the covariance between them.
For Pre-Norm, its hyper-connection matrix is 2 matrix where the bottom right triangular part filled with and the rest is placeholder O. For Post-Norm, the weights depend on the variances 2 matrix. Therefore, their hyper-connection
and covariance of the input and output, forming matrices are non-trainable. this work, we propose hyper-connections that can be (n + 1) x (n + 1) matrices, with weights that are trainable even predicted based on the input. The complete derivation is provided in Appendix
3.2
DUALITY
Given series of neural network modules, we have the option arrange them either sequentially or in parallel. However, hyper-connections offer an approach that learns to rearrange these layers in configuration blending both sequential and parallel arrangements.
layer
layer
layer
layer
(a) Sequential Arrangement
(b) Parallel Arrangement
Figure 4: Sequential and parallel arrangements of hyper-connections with 2.
Without loss of generality, we set the expansion rate to n = 2. If the hyper-connections are learned as the following matrix, the neural network will be arranged sequentially:
0 HC
0
(17)
0
0
In this case, the depth connection degenerates into residual connection, as shown in Fig. (a).
When the hyper-connections for odd and even layers (with layer numbering starting from 1) are de- fined by the following matrices, the neural network will be arranged in parallel every two consecutive
layers, similar to the arrangement of parallel transformer blocks transformers (Wang [022]) as shown in Fig. (b). The general and complete derivation is provided in Appendix
0
0
1
HCodd =
1
(18)
HCeven even
0
(19)
1
Thus, learning the hyper-connection matrix in various forms can create layer arrangements that surpass traditional sequential and parallel configurations, resulting in soft-mixture even dynamic

---

Published as a conference paper at ICLR 2025
arrangement. For static hyper-connections, the layer arrangement within the network remains fixed after training. contrast, dynamic hyper-commections allow the arrangement to adapt dynamically for each token.
RESULTS
Training Loss Tokens
Training Loss Tokens
2.40
CUMo DHCX1
OLMo DH- WC tanh
2.50
GUNo-18-DHCK2
OLMo-1B-Dee Cx WD tanh
Training Loss.
Training Loss
OUMo-18-DH
CUMo 18 DHCX8
OLMo DK Wo tanh
2.50
2..00
2.45
2.46
155
202
250
932
302
100
450 502
102
350
700
252
150
Takens (Billions)
Tokens
(Billions)
Figure 5: Comparison of training loss curves for different expansion rate. The left subfigure includes models with dynamic hyper-connections (DHC) at various expansion rates, while the right subfigure shows the effect of omitting the tanh function. Both subfigures illustrate how increasing the expansion rate leads to improved training loss performance over 500B tokens. Results are smoothed using an exponential moving average with coefficient of 0.99.
Table 1: Ablation study on expansion rates n with training on 500 B tokens.
V2 Eval
V2 Eval
V3 Eval Loss |
V3 Eval
Down Stream
Methods
Loss
PPL +
PPL +
Avg, Acc. 1
OLMo- IB
2.811
18.023
2.544
14.229
62.5
OLMo- 10---1B--CCXI W/O tanh
2.822
18.270
2.556
14.428
62.3
DLMo-1B-DHCX2 W/O tanh
2.792
17.663
2.537
14.033
63.8
IL-Do-B- W/O tanh
2.779
17.451
2.516
13.844
64.4
OLMo-IBB-DHCX8 W/O tanh
2.777
17.425
2,514
13.819
63.8
2.819
18.125
2.556
14.418
62.3
IL.Mo-B-DHCCX OLMo IB-DHCXI
2.802
17.950
2.534
14.114
63.0
2.781
17.509
2.514
13,826
63.8
2.778
17.445
2.516
13.843
62.8
We primarily conduct experiments on pre-training of large language model, including dense and Mixture-of-Expere (MoE) (Shazeer al.) (017) models, and extend visual generation and classification tasks. Due to space constraints, we include the vision experiments in the Appendix
Experiment Settings. We employ the experimental setup outlined by OLMo (Groeneveld al. 2024) for dense models and by OLMoE (Muennighoff et al. (024) for MoE models. For dense models, use dolmap
sample (Soldaini al. [2024) our training dataset. We conduct ablation studies on 1B models and assess the effectiveness of our method at the 7B model scale. For MoE models, we train the OLMoE-1B-1 model, both with and without hyper-comecctonns, on the OIMOE.-MTX dataset. These models activate 1.3B out of total of 7B parameters. All experiments are trained on 500B tokens.
Implementation. We maintain the training configuration of the baseline model, replacing the residual connections with hyper-connections. The static component in Eqs. | [ decay, whereas the dynamic component does. Since the hyper hidden vectors of the final transformer block are ultimately summed, we ensure that the standard deviation (st of the output (before the final layernorm and unembedding layers) remains consistent with the original. At initialization, scale the at d of the weights the output module at all layers, including those the second linear layer of the feedforward network and the output projector of the attention module, by factor of /ni
12 13 does not utilize weight

---

Published as a conference paper at ICLR 2025
where represents the expansion rate. The parameters and computational overhead introduced by hyper-connections is negligible, see Table. and g
Metrics. In accordance with the methodology of OLMo (Groeneveld et al. 2024 report the average perplexities (PPL) and losses on both the V2 and V3 validation sets, along with the average metries for zero-shot evaluation downstream benchmarks (refer to Table T3). We observe significant volatility in the zero-shot performance indicators for the datasets (highlighted in Table 13), with fluctuations exceeding 20% across neighboring checkpoints. For more reliable and consistent results, we excludes these volatile datasets from our analysis. For the MoE models, in line with OLMoE, we also present losses on V3 validation sets, and accuracies on downstream benchmarks (refer to Table [4]
grey
4.1 ABLATION STUDY
We use the dynamic hyperconnections with an expansion rate of and include the tanh function as the default method, marked with the suffix -DHC, while -SHC denotes static hyper-connections.
The evaluation results are presented Table D and the training loss curves are depicted in Fig. [ We observe that with an expansion rate of = I, the performance of DHC is inferior to the baseline. However, for > 1, DHC significantly outperforms the baseline, achieving superior results at n with the increase to tanh excels on both V2 and V3 validation sets, with a reduction v2 Eval Loss by 0.034 and V3 Eval Loss by 0.029 compared to the baseline. Furthermore, the decline rate of training losses for DHC (n 2 2) is steeper than that of the baseline, and DHC demonstrates greater stability, with no spikes observed in any DHC experiments.
8 providing minimal additional benefits. Notably, OLMo Biffo--B-DHEE B-DHCX8 w/O
Static and dynamic hyper-connections. Table 2 presents an ablation study comparing SHC and DHC. All typer-connection (HC) variants significantly outperform the baseline. At expansion rate 2, the improvements of DHC and SHC are similar. However, at an expansion rate of 4, DHC performs notably better than SHC.
Table 2: Ablation study on static and dynamic hyper-connections with training on 500 B tokens.
V2 Eval
V2 Eval
V3 Eval V3 Eval
Down Stream
Methods
Loss
P
Loss 4
PPL 4
Avg, Acc. t
OLMo-1B IB
2.811
18.023
2.544
14.229
62.5
OLMo 1BMo-IB-SHCX2
2.799
17.778
2.538
14.152
63.4
OLMo IB-DHI22
2.802
17.950
2.534
14.114
63.0
OLMo--B--HHC W/O tanh
2.792
17.663
2.529
14.033
63.8
ILMM-IB-SHCX4
2.791
17.671
2.528
14.025
63.6
OI.Mo-IB-DHCxx IB-DHCX4
2.781
17.509
2,515
13.826
63.8
ILMo-IB-DHCCS W/O tanh
2.779
17.451
2.516
13.844
64.4
The importance of B and WC. As shown in Table 3 not training WC leads to significant per- formance declines, with the V2 loss increasing by 0.021 and the V3 loss by 0.017, as seen when comparing the 4th and 6th lines of T contrast, the impact is less pronounced when B not trained. Therefore, ensuring the trainability of both WC and B is crucial.
4.2 COMPARISON WITH RELATED WORKS
We implemented the Altup Baykal et al. 2024) and Resi (Xie et al., 2023) methods in OLMo. Altup is motivated to widen the hidden dimension while maintaining low computation cost by passing only a part of hidden state to transformer blocks. By contrast, ResiDual is proposed to combine both Pre- and Post-Norm in two-stream style. Both methods expand the hidden size by times with negligible computational overhead, with ResiDual expanding it exactly 2 times. For a fair comparison, set 2 in experiments. Unfortunately, although these methods show gains in the early stages of training, they are gradually surpassed by the baseline, as demonstrated by the results in Table and the training loss curves in Fig. 15
7

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study on OLMo- Mito--B-DHCx44 In the B WCC column, the symbol "x" denotes parameters that are not trainable from initialization.
V2 Eval
V2 Eval PPL +
V3 Eval Loss +
V3 Eval
Down Stream
WC
B
Tanh
Loss
PPL 4
Avg, Acc. t
d
x
2.804
17.912
2.537
14.145
62.5
X
x
2.781
17.493
2.518
13.874
63.6
/
x
2.779
17.773
2.516
13.823
64.4
X
V
d
2.802
17.914
2.532
14.072
63.4
X
2.783
17.504
2.520
13.906
63.4
V
2.781
17.835
2.515
13.807
63.8
Table 4: Performance of related methods
OLMo- 1B models.
V2 Eval
V2 Eval
V3 Eval Loss +
V3 Eval
Down Stream
Methods
Loss +
PPL +
PPL
Avg, Acc. f
OLMo-1B OLMo- -B-eesiDua
2.811
18.023
2.544
14.229
62.5
2.825
18.375
2.551
14.346
62.0
OLMo-IB-ANl IB-AupB-
2.827
18.268
2.558
14.454
62.4
OLMo-IB-DHCHE
2.802
17.950
2.534
14.114
63.0
OLMo-IB-DHCX2 W/O tanh
2.792
17.663
2.529
14.033
63.8
4.3
7B MODELS
Training
C4-en Loss
HallaSwag
SciO Acc.
2.4
2.33
2.8
Accuracy
ka
2.3
GLMo-78 MLLMo-7B-Daee
GLA
7B-DHCM
OLMo-7B DHCa
GLMS-1B
100
200
300
402
300
130
403 Tokens lons)
200
300
500
300
200
300
abs
500
100
200
300
abs s00
Tokens (Billions)
Tokens (Bi lions)
Tokens (Bil lions)
Figure 6: (1) and (2) Training loss (0.99 EMA smoothed) and C4-en validation loss for OLMo- and FLLMo-7B-DHCE models. (3) and (4) Accuracy curves on he lawaa and sciq. demonstrating the superior performance of the OLMo 7B-DHCX 4 model.
We evaluate the effectiveness of hyper-connections on the 7B model, training a model with DHCs with an expansion rate of 4, denoted CLMo 700------e 7B-DDCX 4. According to Table OLMo 7B-DHCX significantly outperforms
baseline OLMo 7B model in average metrics. In the V2 evaluation, OLMo 73 4 shows improvements of 0.022 for loss and 0.293 for PPL. Furthermore. the average score of downstream benchmarks 0.710 surpasses the bascline 0.701, with the results of specific tasks shown in Fig. 10
Based on Fig the OLMo OLLMo-7B-DHCE 7B-DHCX model consistently shows better metrics compared to baseline, including training and validation loss and accuracy in downstream benchmarks. Notably, after 400 B tokens, the model maintains its improvement without the gains diminishing. This indicates that the OLMo-73 73 - 4 model continues provide consistent benefits in reducing loss, even at higher token counts. Furthermore, according to Fig. 6 the baseline model exhibits frequent spikes, while our model with DHCs shows no spikes throughout the training. This shows that our approach not only achieves better loss but also ensures more stable training.
4.4
MoE MODELS
We evaluate the effectiveness of hyper-connectionn on the Mixture-of -xpeers (MoE) model. We retrain the original OLMoE-1 1B model as the baseline and train model that applies Dynamic
8

---

Published as a conference paper at ICLR 2025
Table Performance of 7B models. FLOPs refers to the computation per token in the forward pass.
Params (B)
FLOPs
V2
V2
V3
V3
Tasks Avg.
Methods
(G)
Loss |
PPL +
Loss
PPL |
Acc.
OLMo-7B LLMM--B--DHC-.
6.9
13.36
2.581
14.316
2.322
11.324
70.1
6.9
13.38
2.559
14.023
2.304
11.120
71.0
Conpe-Connections (DHC) with replacing the residual connections. The full results shown in Fig. 2 which illustrates that hyper-connections outperform residual connections in almost all metrics. In many metrics, our method requires only half of the training tokens achieve the same performance as the baseline. Fig. and Table 6 highlight some of the results, such as reduction training loss of approximately 0.027, reduction in loss on the C4-en validation of 0.028, an improvement of 6 points the ARC-Challengeand an improvement of 1.2 points on MMLU Var.
Table 6: Downstream evaluations for MoE models training with 500B tokens under the OLMoE evaluation setting. ARC-C stands for ARC-Challenge, and ARC-E for ARC-Easy. MMLU Var is modified version of MMLU that includes varying few-shot examples, providing stable feedback during early training, as outlined the OLMoE setting (Muennighoff et al. 2024
MMLU
Hella- Swag
Wino- Grande
Methods
ARC-C
ARC-E
PIQA
BoolQ
Var
OLMoE-IB-7B
38.5
69.5
41.8
72.8
77.6
64.4
65.4
OLMoE-IB-77--HHCX
39.7
70.2
47.8
76.7
78.2
64.6
68.5
4.5 VISUALIZATION ANALYSIS
R. 12 20
12
28
0-
3.0
83
8-
8-
95
12
PTB
125
12
20
20
245
24
24
24
32
Hyper-Connection
Post-Norm
Pre-Norm
Pre-Norm PTB
Two-hop Residual
Figure 7: Visualization of connection matrices for hyper-connections and various related baseline methods. The attention layers, which have odd ids, are marked with green tick marks.
In this section, we investigate the learned hyper-connection weights and show how the output of the former layer contributes to the latter ones. To this end, convert hyper-connections to dense connections cross layers. Consider the input hidden vectors h k-th layer, can be unfolded as weighted summation over previous layer outputs:
h
Σ T'b j=0
(20)
where kj describes how much layer-j (T*) contributes to layer-k's input hy. Then, (1)) denotes dense connection weight matrix. In particular, let layer-0 be the word embedding and 7o be identity mapping, layer-L L+yer--+11 1 be the hidden state before the unembedding layer, which is a summation over the last hidden vectors, i.e., h [+++ Σ, h/-
(0)
OLMo-13-DHCC 4 model is adopted for visualization. We take the checkpoint at 500B tokens and forward random validation text to obtain dynamic hyper-connection weights. In addition, show connection patterns for some related baseline methods. Finally, the visualization illustrated in We present the following findings, with more detailed discussions provided in Appendix
Fig. 13

---

Published as a conference paper at ICLR 2025
Connection patterns for baseline methods. For Pre-Norm baseline, the connection matrix is simply a lower triangular matrix with diagonal elements erased, because each transformer layer joins the residual equally. In the Pre-Norm parallel transformer block (PTB) baseline, the connection matrix appears jagged because the input to the FFN layer does not depend on the output of the previous attention layer. For Post-Norm baseline, the connection only holds for adjacent layers, as the weight for bottom layers decays every time the residual passes a post-norm layer. For the two-hop residual baseline (Ma al. 2024), the outputs of attention layers are not added residual and only contributes to the next one FFN layer, resulting in a vertical strip pattern in the connection matrix.
A-shaped connection pattern. In the connection matrix for hyper-comnections, a long-tern -erm-termm decay pattern can be observed, where layers are generally preferred to rely on a few adjacent layer outputs. Moreover, the bottom layers (e.g. layer 0,2) are observed frequently used in most of subsequent layers. Therefore, the two patterns together form A-shaped connection pattern. Note that the long-term decay pattern Post-Norm style pattern, while the frequently accessed pattern is Pre-Norm style, indicating that the hyper-connection introduces free mixture of Pre- and Post-Norm architecture.
Input word embedding is eliminated from model output. As per the first column in the connection matrix for layer inputs, the input word embedding contributes to most of the layers except for the final one. This last layer, which products the model's s output, is used for next token prediction. In most cases, keeping component of input embedding in model output is harmful to next token prediction, especially when using a tied word emhedding such as that employed by OLMo- B. Similar results found in previous works (Ma et al. 2023
Parallel transformer blocks are observed. As discussed in $ 3.2 parallel transformer block, which performs attention and FFN in parallel, is special case for hyper-connection. In practice, PTB-like patterns, which can be identified by the local jagged pattern, are surprisingly observed to be learned by hyper-connections. For instance, layer 11 has a minimal contribution to the input of layer 12 (refer to row 12 in the hyper-connection connection matrix). This suggests that layers 11 and 12 can operate in parallel, thereby forming PTB module.
Attention layers tend to have fewer long-term connections. It is observed that attention layers at the bottom barely have long-tew ten-termm contribution, trend that persists until layer 17. Upon examining the connection matrix for hyper hiddens (refer to Fig. [3 in the appendix), it's evident that the outputs of the FFN layers have significantly greater magnitudes than those of the attention layers. This pattern resembles two-hop residual connection design, wherein the attention output contributes to the input of the following FFN layer, but doesn't 't join the main residual path.
RELATED WORK
Transformers (Vaswani 2017) have revolutionized various fields, particularly natural language processing and computer vision. They rely heavily on residual connections to facilitate the training of deep models. Our hyper-connections approach can replace residual connections, providing stable training and consistent improvements in both natural language processing and computer vision.
The issues of gradient vanishing and representation collapse (Bengio et al.] 1994 Glorot & Bengio 2010: Liu et al.] (2020) have been extensively studied. The combinations of normalization techniques (loffe Szegedy] 2015; [Ba et al., 2016) and residual connections (He et al., (016)), like Pre-Norm and Post-Norm, actually reflects different emphases in solving these two issues. However, despite these ad advancements, the fundamental trade-off between gradient vanishing and representation collapse in deep networks remains critical challenge. Building on these findings, work introduces a novel approach that enables neural networks to autonomously learn the optimal strength of connections, potentially improving both gradient stability and representation quality.
CONCLUSION
In conclusion, we have introduced hyper-connections as an effective alternative to residual connections in transformers. Our analysis reveals that hyper-connection not only overcome the limitations of residuals but also enable dynamic adjustments in network architecture. Experimental results confirm their promising benefits across various tasks, including pre-training of large language model, image generation, and image classification.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This research was conducted at ByteDance Inc. We are grateful for the suggestions and assistance provided by Yaowei Zheng, Yuyu Zhang, Yunshui Li, Xiang Li, Bairen Yi, Zhenyi Lu and Xintian Han.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 07Xiv:6607006450. 2016.
Cenk Baykal, Dylan Cutler, Nishanth Dikkala, Nikhil Ghosh, Rina Panigrahy, and Xin Wang. Alternating updates for efficient transformers. Advances in Neural Information Processing Systems, 36, 2024.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions neural networks, 5(2), 1994.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence, volume 34, 2020.
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 1905. 10044, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 1003: (00.00055571 2018.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop. Springer, 2005.
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung. volume 23, 2019.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. leee, 2009.
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005). 2005.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 1rXiv:2010.11929 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerat- ing the science of language models. arXiv preprint arXiv.2402.00818. 2024.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference Learning Representations (ICLR), 2021.
Sergey loffe and Christian Szegedy. Batch normalization: reducing internal covariate shift. In International conference on machine learning. PMLR, 2015.
Accelerating deep network training by
Matt Gardner Johannes Welbl, Nelson F, Liu. Crowdsourcing multiple choice science questions. 2017.
Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. arXiv preprint arXiv:2205.05198 2022.
Liyua Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. arXiv preprint arXivv2004.08224 082449, 2020.
Haoyan Ma, Xiang Li, Xia Yuan, and Chunxia Zhao. Denseformer: A dense transformer framework for person re-identification. IET Computer Vision, 17(5), 2023.
Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, and Chunting Zhou. Megalodon: Efficient Ilm pretraining and inference with unlimited context length. arXiv preprint (xXiv:2404.08801 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct clectricity? a new dataset for open book question answering. In EMNLP, 2018.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. Olmoe: Open mixture-of-experts language models, 2024. URL www.i. abs 2409. 02060
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. 2011 AAAI spring symposium series, 2011.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9), 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv: 09728, 2019.
N Shazeer, A Mirhoseini, K Maziarz, A Davis, Q Le, G Hinton, and Dean. The sparsely-gated mixture-of-experts layer. Outrageously large neural networks, 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, 2013.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00155, 2024.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ:: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
12

---

Published as a conference paper at ICLR 2025
Ben Wang. Mesh- TrssTTranffrmmr-JAAX Model-PPrallel Implementation of Transformer Language Model with JAX.
jax
May 2021.
Mitchell Wortsman, Peter Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322, 2023.
Shufang Xie, Huishuai Zhang, Junliang Guo, Xu Tan, Jiang Bian, Hany Hassan Awadalla, Arul Menezes, Tao Qin, and Rui Yan. Residual: Transformer with dual residual connections. arXiv preprint arXiv:2304. 14802, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07886. 2019.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
13

---

Published as a conference paper at ICLR 2025
A TRANSFORMER WITH
P1
P2
FFN
FFN
P
Attention
Attention
15
FFN
FFN
15
15
Attention
Attention
Repeat R
(b) Transformer with Residual Connections
(b) Transformer with Hyper-Connections
Figure 8: Comparison between transformers with hyper-connections and that with residual connec- tions.
14

---

Published as a conference paper at ICLR 2025
B PARAMETERS, COMPUTATION AND MEMORY FOOTPRINT ANALYSIS
Static Hyper-Connections. All learnable parameters are included in the hyper-connection matrix HC in Eq. The number of parameters in one HC is given by:
Sacc
[0g] + AAl n + n (n 1)
2).
(21)
where n. is the expansion rate, [0B] is the number of parameters in B in SHC, and |0^] is the number of parameters A. Each layer contains two hyper-commection modules (one for the self attention and one for the feedforward network). Thus, the number of extra parameters is:
Pextra = [0cc| x 2 x L,
222)
where L the number layers. For example, in OLMo 1B SSCCX4 Pextra =X 4x4(2)) 768.
Dynamic Hyper- Copper-Connection. The parameters DHC are defined Eqs. the number of parameters is given by:
and
and
1uueel
n0rml
+ |s + 1, + [0B] + |sal + +=10 AA... + 10w. + 10A.1
(23)
norml + 1 dmodel + + + + dmode1 + + dmodel X n +
(24)
Oneral + daadel
(n + 2) + n (n + 2) + 2,
(25)
where dmade: is the dimension of the hidden states in the transformer, and 1n depends on the type of normalization module. In OLMo models, there are no parameters for normalization, so [narml 0. In OLMoE, @norml daodel. Similar to the static hyper-connections, the number of extra parameters is:
Poxtra 00 2 x T,
(26)
For example, for ELMo-1B-DDHO P (0 2048 (4 (4+2) + 4 (4+ +2) 2 16 394, 048.
The number of parameters for DHC and SHC used in the experiments is detailed in Table while
their corresponding FLOPs comparisons are provided Table X Regardless whether SHC or DHC is used, the additional parameters and computational overhead introduced minimal and can be considered negligible.
Table 7: Comparison of number of parameters.
Method
HC Params(B)
Total Params(B)
Total Params A rate (%)
OLMo-IB
1.17676442
OLMo-IB-SHC. x2
0.0000026
1.17676467
+0.00002 %
ILMo-IB-SHC
0.0000077
1.17676518
+0.00007 %
OLMo-IB-DHCxx
0.0002625
1.17702688
+0.02230%
OLMo-IB-DHHCx4
0.0003940
1.17715846
+0.03349 %
OLMo-7B
6.88809574
OLMo-7B-DHCX.
0.0013124
6.88967027
+0.02286%
OLMoE-IB 1B-7B
6.91909427
DOMMOE-IB-BB-HCCEE
0.0003940
6.91948832
+0.00570% %
Computation Analysis. The main computational cost of SHC and DHC lies in line of Algorithm
where the complexity is O(dmodel x x (n + 1)). The computational cost of the FFN is O X dmudel X dnn), and that of the projection part of attention O(4 dinadel dmmose). Since O ((dymdee X
(n + 1)) < ( ((4 x dmostel X (inotee) < O x demateel X dua), the computational cost of HC is
negligible compared to the cost of both FFN and the attention projection part. Here, dn, is the inner
dimension of the FFN, The detailed computation cost statistics are presented in Table
15

---

Published as a conference paper at ICLR 2025
Table 8: FLOPs per token in forward pass.
Method
HC FLOPs (G)
Total FLOPs (G)
Total FLOPs A rate (%)
OLMo-IB
2.3536
MMMo-BBSHCX2
0.0010
2.3545
+0.038 %
0.0031
2.3566
+0.127 %
OLMo-IB-DHCx2
0.0020
2.3554
+0.076
DLMo-1B-
0.0049
2.3583
+0.200 %
OLMo-7B
13.3647
4
0.0197
13.3844
+0.147
OLMoF- 1B-7B
2.3580
0.0049
2.3629
+0.208
Memory Footprint. The introduction of HC results in minor increase in activation memory usage during training. For transformer model with L layers, model dimension of dmmoeets batch size b, sequence length and number of attention heads the activation memory is calculated as sbdmmad + 5as/dmode dmodel), as outlined Korthikanti et al. (2022). Incorporating HC with an expansion rate of n adds extra memory overhead of 2nd L. For = 2, this contributes less than 15% to the total memory usage of standard transformer. Notably, the memory consumption is mostly driven by the weight parameters, which experience only slight increase with HC. Additionally, given HC's low computational cost, the hidden states generated by HC can be discarded post forward pass and recomputed during backpropagation to further optimize memory usage. With this approach, the additional memory requirement is reduced to nosddmode)- During inference, the memory usage for activations is largely determined by the Key-Value cache, which not impacted by the extra activations brought by HC. Moreover, the hidden states from earlier layers can be released as soon as the next layer's computations start, significantly lowering memory requirements. The actual memory footprint is empirically measured on 8 GPUs, as shown in Table
Table 9: Measured Memory Footprint on GPUs.
Micro Batch Size (tokens per GPU)
Method
Memory (GB)
Memory Rate (%)
OLMo-1B
41.11
16,384
OLMo-1BBSHCC>.
47.55
+15.7% %
16,384
DLMo-IB-SHCX:
51,85
+26.0 %
16,384
OLMo-IB-DHCX2
47.56
+15.7% %
16,384
OLMo-IB-DHCC4
51.86
+26.1 %
16,384
OLMo-7B
26.27
2,048
OLMo-7B-DHOC 7B-DHCX4
33.70
+28.28%
2,048
OLMoE- DLMOE-IB-7B
31.59
4,096
OLMoE- 1--7B-B-CCC
34.65
+9.7
4,096
16

---

Published as a conference paper at ICLR 2025
C MOE 1B/7B MODEL EXPERIMENTS
training
loss otfiee
val,
GLMee 13-22
Di divdrd 16 16 Med
o divv it 38
26
2.05
2.
2.50
2.70
Tokens (B)
Tokenss (9)
Tokens (B)
Tokens (B)
dolma pe:2o val. loss
dolma reddltt val. less
dolma stack val. loss
dolma wiki val. loss
2.35
3.22
2.4
2.15
2.90
2.30
2.55
120
700
Y00
100
100
120
215
110
100
120 400 100
275
100
100 256 100
GLMtel 13:33
CLMEE
2.40
Ciiv 16 36
0iire la 78 altre Dacst
0LMel
3.05
3:40
2.10
2.5
2.23
3.5
3.00
Tokers (B)
lokenss (8)
Tokent (B)
Tokens (B)
MMLU stem Var Acc. (%)
MMLU hum. Var Acc. (% (N)MMLLU soc, scl. Var Acc. (%/JMILU other Var Acc. (%)
37.3
24
CLM1E-22-10
35.0
CLPe61B-10
OLPED
40
126
100 Y00
T00
215
110
420 300
120 275 100
4to 100
100 276 100 400
500
avg.
Acc.
Challenge
42-
outte La 2a
34
CLFGE-L5-75
CLMEE-LE-75-
CLP 100---1------
CLIG5-22-11
Tokens (B)
lokenss (8)
Tokens (B)
Tokens (8)
ARC easy Acc. (%)
PIDA Acc. (%)
WinnGraadde Acc. (%)
Openbook QA Acc. (%)
C2.
72
CJR-618-18
57.5
120
200
100
100
100
215 100 420
300
120
275
300
420
100
120
276
100
400 50
Acc. (%)
Acc. (%)
Commonsense Acc. (%)
GuMbS 18
s0.
475
82.5
45.
4G
CLVeE--5578 CUMeE-LB-73-2HcC
45
oJhcb 28 C3CC
Tokerss (8)
Tokens (8)
Tokens (B)
Tokees (8)
Figure 9: Loss curves in V3 validation sets and accuracy curves on downstream tasks for OLMoE-1B78 and LLLMOE-1B7B-DeR models.
17

---

Published as a conference paper at ICLR 2025
D 7B MODEL EXPERIMENTS
val. loss
dolma books val. loss
dolma val. loss
2.7
OLMo-7B.
CLMc-7B
OLMo-78
OLNo-7B.DHC44
DINo-78-DHCM4
Mo-78-0HCas
2.6
2.7-
2.5
2.6
kene
lakens (B)
Takens (B)
dolma pes2o val. loss
dolma reddit val. loss
dolma stack val. loss
OL 7B-
3.0
CLKc 7B
OLMo 78
OLNo-7B-DHCx4
OLNo-78-DMCC4
1.00
2.2
0.95
2.1
100
200
300
400
500
100
200
300
400
500
100
200
300
400 500
To kens (B)
Takens (B)
Tokens (B)
val.
m2d2-s2orc
2.5
CLMo-7B
2.8
OLMc-7B.
3.2
OLMo-78:DHCS4
OLMo-7B-DHCx4
OL 7L---
2.7-
3.1
300
200
Takens
(B)
pile val. loss
wikitext 103 val. loss
HellaSwag Acc. (%)
OLMc-7B-
OLMc-7B-
70
2.2
2.6
2.5
60
2.4
OLMo CHC
OLMo 78
100
200
300
400
500
100
200
300
400
200
300
400
500
16 kakks (8)
Tokenns (B)
Tokens (B)
SclO Acc. (%)
COPA Acc. (%)
Openbook QA Acc. (%)
90.0
40.0
83.0
OLMc-7B.
CLMc-7B.
CLMo-7B- MB-1B-
82.5
O1 D-DD-7
OIMo-78-DHCa4
32.5
1010
7000
300
400
500
100
700
300
400
500
100
200
300
400
500
Takens (B)
Takens (B)
Takens (B)
PIOA Acc.
WinoGrande
ARC-Easy Acc.
78
70
65-
74
OLMo 7B
CLNc 7B
60
OLMo 7B
72
CLNe-7B-DHDkd
OLMc-7B DHCH4
OL M0--H--HCK
keks
lokens (B)
Takens
Figure 10: Loss curves in V3 validation set and accuracy curves on downstream tasks for OLMo-7B and OLMo 7B 7B-DHCX models.
18

---

Published as a conference paper at ICLR 2025
E VISION EXPERIMENTS
Datasets. We use the ILSVRC-2012 ImageNet dataset (Deng et al. 2009) with 1 classes and 1.3M images (see ImageNet in the following) for image generation and classification.
E.1 IMAGE GENERATION
To investigate the generalizability of hyper-connections in image generation, our experiments are conducted using the DiT framework (Peebles & Xie (022) training the models for 1400 epochs. In order to save experimental costs, use FPI6 precision, introduce flash-attention to speed up training, and introduce QK-Norm (Wortsman et al. 2023) to stabilize training.
Table 10: Benchmarking class-con image generation on ImageNet x with cfg=1.50. NP, P. and R are short for Numerical Precision, Precision, and Recall, respectively.
Method
NP
QK-Norm
Size (M)
FID
sFID
ISt
Pt
R1
DiT-XL/2
FP32
X
675
2.27
4.60
278.24
0.83
0.57
DiT-XL/2
FP16
/
675
2.36
4.54
269.46
0.83
0.58
DiT-1B/2
FP16
/
983
2.13
4.50
288.69
0.82
0.59
DiT-XL/2-SHCX2
FP16
/
675
2.18
4.52
287.24
0.82
0.60
Our experimental results demonstrate that DiT models incorporating hyper-connections exhibit comparable performance metrics to DiT models with 50% more parameters. This finding underscores the efficiency and efficacy of hyper-comeections in enhancing model performance without increasing model size.
E.2 IMAGE CLASSIFICATION
For the image classification experiments, we train ViT/16-Basd Basee and ViT/16-Large models with images resolution of 224 224 for 300 epochs, following the experimental setup used by (Dosovitskiy et al. 2020).TD To speed up the training process, we use bfloat16 numerical precision. The training configuration detailed in Table T2 Within this configuration, with static and dynamic hyper-connections, referred to as SHC and DHC, respectively, using an expansion rate of 2. The top-1 accuracy results are presented in Table II and the training loss curves for ViT/16-Large and ViT/16-Large with DHCx2 are shown in Fig.
replace the residual connections
For the Base model (85M), our re-implemented ViT/16 achieves 76.38% accuracy on 224 224 images. The SHC and DHC enhance performance to 77.60% and 77.26%, respectively. representing relative increases of 1.22% % and 0.88% %. For the Large model (307M parameters), ViT/16 achieves 77.25% accuracy. The SHC and DHC configuration further enhance accuracy to 78.38% % and 79.94%, respectively. This corresponds relative improvements of 1.13% and 2.69% %, with DHC showing the highest performance. These results demonstrate that hyper-connections (SHC and DHC) significantly improve accuracy, especially the Large model scale.
Table 11: Accuracy on ImageNet. VIT refers to the results reported by (Dosovitskiy et al. 2020). whereas VTT/16 denotes re- ie-mllmmeeeed baseline. SHC and DHC indicate that residual connections are replaced with static and dynamic hyper-comeections, respectively.
VIT
VIT/16
ViT/16-SHCX2
ViT/16-DHCx2
Model Scales
Params (M)
384 x 384
224 x 224
Base
85
77.91
76.38
77.60
77.26
Large
307
76.53
77.25
78.38
79.94
19

---

Published as a conference paper at ICLR 2025
Training Loss
VIT/16-Lagre
VI7/16-Lagre-HHC2
1.6
1.4
1.2
Loss
1.0
0.8
0.4
0.2
30000
40000
50000
60000
70000
80000
90000
Steps
Figure 11: Training loss curves of ViT/16-Large and ViT/16-Large-HHXX2. smoothed using Exponential Moving Average (EMA) with a decay rate of 0.999. The gain from Hyper-Connectionn decreases training progresses, likely due pass over the same dataset across many epochs, resulting in diminishing returns from the additional capacity provided by Hyper-Connecttins.
E.3 VISULIZATION OF DHC
We randomly select three categories from the ImageNet dataset and sample the corresponding examples from the validation set. These samples are fed into the ViT-Base/I6-DHCx2 model to compute the dynamic connection weights of the DHC in the final layer. As shown in Fig. [2 we visualize the distribution of these weights. We observe that the intra-class distribution highly concentrated, indicating that samples within the same category tend to have similar beta values. In contrast, the distribution of alpha less concentrated, but the differences between the distributions of different categories more pronounced, as exemplified by 02,0.
beta
Table 12: Training hyperparameters for VIT.
Hyperparameter
Value
Learning Rate (Ir)
0.003
Batch Size
4096
Scheduler
Cosine Annealing with Linear Warmup (10k steps)
Data Augmentation
Mixup 0.2)
Epochs
300
Optimizer
AdamW (01 = 0.9, B2 = 0.999, le -)
Gradient Clipping
1.0
Weight Decay
0.3
Dropout
0.1
Precision
bf16
20

---

Published as a conference paper at ICLR 2025
33lleggerhead turtle
958icapitulum
575cchloll bus
33 el turtle
SS8:capitulum
779:school
0.5
1.50
1.09
1.10
1.15
1.25
885
1.50
1.0
1.35
1.21
9.5
1.55
1.10
1.35
3.20
3l:loggerhead turtle
$08icapitulum
779sschool bus
-0.00
0,3
23 33:loggerhead turtle
998:capitulum
779tschool bus
1.5
60
1.2
1.
1.5
1.
3.3
14
1.3
33lloggerhcad
998:capitulum
779sschool
33 logggerheaal turtle
998:capitulum
779:school bus
2.0
2,3
218
2.15
3.20
2.5
2.30
2.35
2.0
2.08
2008
218
2.15
2,28
225
2.30
2.35
2.40
2.50
2.05
3.17
2.15
2.20
2.25
2.30
2.40
998:capitulum
0.0
02
-2
-01
-0.30
0.10
00
%a
Figure 12: Distribution of weights of last DHC in ViT-Basc/16-DHCSS model.
F MORE VISUALIZATION AND ANALYSIS
Unfolding hyper-connections. We first introduce how to determine the connection matrix ((0) for hyper-connections. To simplify writing, the layer output T (*(ht) is denoted by T for short. The
21

---

Published as a conference paper at ICLR 2025
29
24
((0)
C
(
C(3)
((4)
125
16
16
16
16-
24
24
24
24
28
28
(a) Connection matrix for DHC model.
12 20 28
12
22
2425
12 20
26.28
((0)
C
(
C
C(4)
12
12
12-
125
12-
20
203
24
24
32
32
32
(b) Connection matrix for SHC model.
Figure 13: Visualization of unfolded connection matrix. Matrices from left to right are (Connection for { (2), C (Connections for (h') //2) for E f1,2,3, 4). The at- tention layers, which have odd ids, are marked with green tick marks.
recurrent form of hyper connection Eq. expanded as follows:
ho
HH Am &
B + H 17 A. R- A &
(A, 141 1t2 Ar.AA
k- Am k
k-1
k-1
Σ
T 5=0
II A, Am
(27)
ビ=+1
Therefore, we obtain connection matrix 4 = B'( Am)
Similarly, the connection matrix (10) for the i-th hyper hidden from k-th layer be computed by substituting the last Am with A, & in Eq. 27 i.e.
k-1
H A
H
El
ATT
(28)
j=0
t=j+1
II (=j+1
A
B
(29)
Visualization for hyper hidden. We visualize connection matrices for hyper hiddens in Fig. 13 to reveal how hyper-connection maintains intermediate layer outputs. First of all, the four hyper hiddens dissimilar and show completely different connection patterns. Then, FFN layers are preserved long-termly in hyper hiddens, while attention layers are reserved less. also observed that the long-term connections the connection is positive in one hyper hidden but negative in the other, for example, column 0 and 2 in (()), ( With such strategy, these connections can be easily eliminated in the sum-pooling operation before the unembedding layer.
outputs from
usually stored pairs of hyper hiddens, where
SHC shares similar connection pattern with DHC. We show the connection matrices for OLMo-13-SHCX4 4 model in Fig. 13b Comparing to DHC, as shown in Fig. 13a SHC shares exactly the same connection patterns. Moreover, observe many more PTB-like blocks in SHC, e.g., layers from 13 to 18. Note that the connection relation for SHC is token independent, and such PTB-like blocks be physically reorganized be parallelly computed.
22

---

Published as a conference paper at ICLR 2025
20
0.75
4-
0.75
0.75
0.50
82
0.50
0.50
124
12-
12
162
wasted
16.
16
20
-0.25
20
-025
20
24
-0.50
-0.50
-0.50
-0.75
-0.75
-0.75
(a) 1MMM-11-DHCx1
(h) OLMO- B-DDCX2
(c) OL 18-DHCx4
Figure 14: Comparison of unfolded connection matrices for 01Mo- O1Mo- and O0MM--15-DHCC 4 model.
How HCx1 fails. The CLMo-1BXI model is observed perform worse than baseline in experiments. Its connection matrix is visualized in Fig. 14 show how it fails, Above all, we observe that layer 17 is wasted, who has connection subsequent layers all. Secondly, compared to HCx2 and HCx4 models, the shaped pattern does not appear. Note that HCx1 does not support the pattern A in its mathematical formulation, where the connections previous layers must be weakened or strengthened simultancously. Thus, the lack of connection from the early layers to the final layers may suffer from gradient vanishing, like post-norm style transformers, which leads to performance degeneration.
23

---

Published as a conference paper at ICLR 2025
G DERIVATION OF NON-TRAINABLE HYPER-CONNERR MATRIX FOR RESIDUAL CONNECTIONS
G.1 PRE-NORM RESIDUAL CONNECTION
In the Pre-Norm residual connection, the input to a layer is first normalized before being passed through the layer. The output of the layer is then added the original input. This can be represented as:
h T (Norm((h) + h.
(30)
By incorporating the normalization operator into the layer, T = T Norm, express the entire process as:
h T(h) + h.
(31)
To express this using hyper-connections, the matrix for Pre-Norm can be structured as follows:
( 1 1
HC PreNorm
(32)
Given hyper hidden matrix H hT, we prove that the output of PCerooo A hT.
Proof.
Å HC(T, .H) (H Am) T + +
(33)
T(h)T +hT
= GT.
G.2 POST-NORM RESIDUAL CONNECTION
In the Post-Norm residual connection, the input to a layer passed through the layer first, and then the output is normalized after being added the original input. In matrix form, this can be represented as:
h = T(h)
(34)
The summation of the input and the normalized output of the layer is:
h = Norm h - h')
(35)
We consider Norm to be LayerNorm (Zhang & Sennrich 2019). The analysis process for RMSNorm is almost identical. In fact, the affine transformation can be incorporated into the subsequent layer, while the mean subtraction operation can be integrated into the current layer.
T Coooo A, where A is the affine transformation, and C is the re-centering operator. Thus, the mean of the output of T is
(36)
To express this using dyper-connections with an expansion rate 1, we need hyper-connection matrix HC that encapsulates this operation:
HC PostNorm
Vah of
20
-(A
-00
B A,
(37)
Am
Vof- - +20
24

---

Published as a conference paper at ICLR 2025
Similar to the previous proof, prove that the output of PostNoom is equivalent to the transpose of the output the Post-Norm residual connection:
= h
(38)
Proof. Note that
Oh-hh Voh + oh + n0
(39)
Given this fact, derive the Post-Norm:
= Norm(h's h
h + h ph
Thth
(40)
(h' + h)
Oh
(h' + h)
Voh + +o 2h
For hyper-connections side, have:
A BTh' + HH'
BTh' A,H
=ThhT + A,h
(41)
hT
h = -h.
Vol + o + 20
Vo + oh + 20
25

---

Published as a conference paper at ICLR 2025
H SEQUENTIAL-PRRALLLL DUALITY
H.1 HYPER-CONNECTION MATRIX SEQUENTIAL ARRANGEMENT
In this section, we demonstrate that the following hyper-connection matrix will produce identical networks arranged sequentially with residual connections between them:
0 e1
1ixn
HC
(42)
enxn
where Gnxn denotes
identity matrix, 6; E RRxxl represents the i-th column of 6nxns and
lxx signifies a x n matrix of ones.
We will use mathematical induction to prove that h h and h = (* + h Vi.j E {0, 1, 1............... vk E {0, 10.1,,...,., L.J, where L is the number of layers.
Proof. BASE CASE
For h = O, we have the initial condition hς hg. Vi,j E {0, 1,. ,n], as we define H
(ho h h²) E gaxd
INDUCTION HYPOTHESIS
Assume that for some l E {L,...,L 1}, we have h h and h T*( + h Vi,j (0.1,...,,1,
INDUCTION STEP
We have
H #+1 HC(T* H
(43)
BT(h(()T + H
(44)
BTA T + T
(45)
=InxIT H + enxa H
(46)
(T*(hf) (hitt)
T*(bt)) + (hf h T*(h)) h()²
h
(47)
(T*(h)) + of T*(b)) hy
(48)
(hi+1 ba k+1 b
(49)
Since by = h/j Vi E (0. n,....n), it follows that T* + hō = T**(h)) bby. Thus, we have
h h -h htt1
(50)
Since hā = hhy. Vi {0, 1, .......,,.. n). it follows that h h&, Vi {0, 1, ...,, n). Thus, we have
h h (bfi + h
(51)
= -T²(h)) + h
(52)
26

---

Published as a conference paper at ICLR 2025
H.2 HYPER- EIPR-COONECTIONN MATRIX OF PARALLEL ARRANGEMENT
In this section, we demonstrate that the following hyper-connection matrix will produce a network where every adjacent layers are arranged in parallel, with each layer incorporating residual connec- tions. We define a parallel-arranged network such that n adjacent layers form a group, with layers within a group being parallel and groups arranged sequentially. The output of k-th group is given by:
7
h h+1
Σ (Tkxnti (h*) h*).
(53)
be proved that this arrangement can be described by the following hyper-connection matrices. First, for where k - = 0 (mod n)::
HClhkk-11 (mod n)}
x1x1
ef Iaxns
(54)
1 1xx1
where the HC matrix can be decomposed into two operations: 1) sum up all the outputs of the previous group and use as the input of the current layer and as the residual of the subsequent layers; 2) sum up the output and input saving to the first hidden vector slot.
Next, for where is- 1 = (mod n) and + 0:
(01x1 ei
HC (mod n),(200
(55)
enxu:
where the HC matrix selects the i-th hidden vector as the input of the current layer, and sums up the output and input, saving the i-th hidden vector slot.
This means:
++1 =HC
(56)
(
(57)
(58)
Jcckn+I(Tkxnnt h²)))
(59)
This can also be proved by mathematical induction; however, the conclusion is quite obvious through drawing, and the proof process is very tedious. Therefore, don't repeat the similar proof here.
27

---

Published as a conference paper at ICLR 2025
I PSEUDOCODE OF HYPER-CONNECTIONS
Algorithm Network with Hyper-Connections
Require: Initial hidden vector h
Rd
Require:
Expansion rate n
Ensure: Final output
Initialize:
2: H <- (h" h
T E Ruxd
3: for
do
For each layer
4:
H H
5:
(ho -
>> Width Connections
6:
hb 6- (*bbo)
Layer Computation
7:
H+ TTBBthh +
b Depth Connections
8:
H* H
9: end for
10: Final Output:
h sum rows H
12: h +- Normalization Layer(h*)
13: y + Output "
14: return
28

---

Published as a conference paper at ICLR 2025
J PYTORCH IMPLEMENTATION OF HYPER-CONNECTIONS
Algorithm 2 Pseudocode hyper-commectioos in a Py Tyrrch-like style.
hyper hidden matrix (BxLxxiXD)
claas HyperConneettion (nn. Moa.Medllel:
def inii _ini__ (self, dim, rate, layer_id, dynamic, devicenNone))
super (wpperyonneeottiioooo
1n1_init___
seif.rate zate sel self.layer_id layer_id
se.1.dynamic cynamic
self.sratic_beta an.Parometer (torch. ones (zate,), devico=devicc))
init_alphal = zeras 1), devigesdevine
init_alpha [layerr_d rate,
sssf.ststic_aaphe an.Peremeter ftorch. cat ([init (iinit_slphs), torch.eye ((rate), device devicel 1, dim=l))
15 self.dynamic:
self. wyt.srcymmmccapaaaaooom nn. Parameer zeros (dim, rate+1), denivendev
nn.Farameter (toren. devi devicesdevice) 0.01) Paraamette:: serooc..zeoos (dim, ), device=device))
s.l..dyyamic_bete_fl
self, self.dynamice detddaammeetteseaaaaeeeeeeeeeeee scats an. Parameter (torch. ones (1, devicemdavice) 0.011
sel self.layer 1a.e-aayy_noomm TayerMorm (dim)
det soacch_connectoonneeelll
31:
alpha and bete
self.dyramie:
rorm_h self.layer_normth
self.dyramic:
wc weight nor 8 hal..ddnamiccaphhofff we_weight F. ww__eigh_
dynamic_elpha we_weight self. www.aaaceelasaaa...eoeoeaa
zlpha = dynamic_alphs self.ctatin__ slptiic__pppppeee (Nona, None, ...]
else:
sipha alif.
....
if selt. dyyaamiem de_woight nors_h 0 galffdynamic_heta de de_weight F.tanh Ido_ weight)
dynamic de_ weigtt self. dynamdi beaa_ beta dynamic_ self stff.sttti__hcaa (None, elae:
beta self.static (None, None, ...]
connection
mix_ - alpha. awp.a..taaaaaee....... transposea 2, 2)
return mix_h, beta
def depth__onneetion mix_h, bLo, bets): torch. ennum ("blh, >blah", h_o, betal mix.hl.... 12,
return h
Algorithm 3 Pseudocode of transformer with hyper-connections in PyTorch-like style.
ha hyper hidden matrix (BxIxNxD)
hot_yrrprcooottiinnn won__yypr__ooreetloee ayper-conreoslon modules atta_nozm, Σin_norm: nozmellzation modules
Actentioo Blook
mix_h, bet coo.eeyeeecoonttiioonn wtwprrooooottttttddooooeeeeeeee comeetcoo
h = stta__erm tit
selt_attention
h hww.llyypeccomecctoooddeeehc.cco
Imxx_,, dropout (h), betal
4 PEN Block
h, beta Sasper_connettoo www.twtcccttttttiiioom
h (mixmmiix_h(.... 3,
h IIn
compessioonnnioon (mix_h, drepout (h),
29

---

Published as a conference paper at ICLR 2025
K VALIDATION SETS AND DOWNSTREAM TASKS
Table 13: OLMo's default configuration was evaluated using multiple metrics. Perplexity (PPL) and loss used for the V2 and V3 Validation Sets, while zero-shot testing was applied the Downstream Benchmarks. However, the grey benchmarks were excluded from our analysis due to the instability of their performance indicators.
V2 Validation Sets
v2 val.iaaiionaaliiaaioo
v2 =2-smmalj-cee_en--vaallaatt.n
V2-omall--gaa-vaaiidtttoo
12-amall1icce-vaaiiaaioo
v2 -
v2 o
v2 22=
v2 02-=maall-piiaa-va1ldd....
v2 12--mmall-pabb-vaaiiatttto
v2 =
V3 Validation
v2-2mmallccc__e--vallaaaato
23=omaall-ccce-vvaliiattoo
v3- v
v3 smalmmlllammm [i
Downstream Benchmarks
piga (Bisk al. 2020)
hellaswag (7eller et al. 2019
winogrande (Sakaguchi et 2021
openbook_ag (Mihaylov et al. 2018
sciq (Johannes Welbl 2017
arc_easy (Clark et al.
2018
copa (Roemmele et al.
2011)
comm Ement bank De Märneffe al.
2019
mrpc Dolan & Brockett
2005
rte agan al. 2005
sst2
Socher et al
30

---

Published as a conference paper at ICLR 2025
Table 14: Downstream Benchmarks for OLMoE.
Downstream Benchmarks for OLMoE
piga (Bisk et al. 2020)
hellaswaq Lel et al., 2019
winogrande (Sakaguchi et al.
2021
openbook_se (Mrhaylov al. 2018
sciq (Johannes Welbl (017)
arc_ (Clark et al. 2018
arc_challenage (Clark et al. 2018
copa (Roemmele et al. 2011)
boolq (Clark al. 2019))
comnonsense __aa________ (Talmor et al. 2018)
social_iga (Sap et 2019
mmlu (Hendrycks et al. 2021)
L IB MODEL EXPERIMENTS
Training Loss
2.9
OLMo-1B
OLMo-1B-ResiDup
2.8
LLMo-1B-Altupae
OLMo-1B
1MMD-1B-DMCX2 DHCX2
OLMo-1B DHCx2 W/O tanh
2.7
Loss
2.6
2.5
2.4
100
200
300
400
500
Tokens (Billions)
Figure 15: Training loss curves of related works, smoothed using Exponential Moving Average (EMA) with a decay rate 0.99.
31

---

Published as a conference paper at ICLR 2025
Training Loss
2.60
OLMo-1B
OLMo-1B-DHCXI
OLMo-18-DHCX2
2.55
OLMo-1B-DHCx4
OLMo-1B-DHCx8
2.50
Loss
2.45
2.40
2.35
200
400
600
800
1000
Tokens (Billions)
Figure 16: Training loss curves of DHC with tanh over 500 billion tokens, smoothed using Exponential Moving Average (EMA) with decay rate 0.99.
Training Loss
2.60
OLMo-1B
OLMo-18-DHCx1 W/O tanh
OLMo-18-DHCx2 W/O tanh
2.55
OLMo-18-DHCx4 W/O tanh
OLMo-18-DHCx8 W/O tanh
2.50
Loss
2.45
2.40
2.35
200
400
600
800
1000
Tokens (Billions)
Figure 17: Training loss curves of DHC without tanh over 500 billion tokens, smoothed using Exponential Moving Average (EMA) with decay rate of 0.99.
32

---

Published as a conference paper at ICLR 2025
Training Loss
2.9
OLMo-1B
OLMo-1B-PTD PTB
2.8
OLMo-1B DHCx4 W/O tanh
OLMo-18 1B-DHCX4
2.7
Loss
2.6
2.5
2.4
100
200
300
400
500
Tokens (Billions)
Figure 18: Training loss curves comparied with parallel transformer blocks (PTB), smoothed using Exponential Moving Average (EMA) with decay rate of 0.99.
Table 15: Results on downstream benchmarks for IB models.
Method
are_easy
copa
hellaswag
openbook_gr
piqa
seig
winogrande
avg.
OLMo-IB
56.8
76.0
56.1
33.8
74.4
85.1
55.6
62.5
Scaling in DHC W/O tanh
W/O tanh
56.8
75.0
55.3
33.4
72.9
85.4
57.1
62.3
OLMo-IB-DHHCE W/O tanh
63.0
74.0
57.1
34.6
73.5
86.0
58.2
63.8
LLMo-IB-DHCC. W/O tanh
61.2
80.0
57.5
33.6
75.5
85.8
OLMo-IB-DHCXS W/O tanh
61.1
75.0
57.6
35.4
73.8
85.2
58.5
63.8
Scaling n in DHC
OLMo- OLMo-IB-DHCxI B-DHCxI
59.7
74.0
55.5
33.6
73.5
85.4
54.5
62.3
OL 10-DH22
59.7
73.0
56.7
34.0
74.7
85.2
57.9
63.0
OLMo-IB-DD IB-DHCX4
59,8
79.0
58.1
32.4
74.3
86.1
57.1
63.8
OLMo-IB-DHCK. IB-HHCX8
56.8
75.0
58.0
34.4
73.8
84.2
57.3
62.8
Scaling in SHC
DLMo-IB-SHCE OLMo-IB-SHCx4
59.1
77.0
56.6 56.7
35.4
74.2
85.3
56.4
63.4
59.3
77.0
34.0
74.3
86.6
57.1
63.6
Non-trainable WC
OLMo-IB-DIICA OLMo-IB-DIICS W/O tanh
60.5
78.0
56.2
34,0
73.5
86.0
55.8
63.4
59.1
72.0
56.8
35.0
73.3
86.0
55.5
62.5
Non-trainable
OLMo-IB-DHCC OLMo-IB-DHCx4 W/O tanh
59.5
77.0
$7.9
33.8
73.3
85.6
56.6
63.4
60.4
74.0
57.6
34.0
74.9
86.7
57.5
63.6
33

---

Published as a conference paper at ICLR 2025
OLMo-1 1D- DD-DICCt W/O umh
MG-
(LMM-1 SHCkkl
Mi-M- BB SHCC2
CH. Mo-1 18 C
OL Mu-1 18: HHH3
CIL Mo-Mo-1 Mo--------RC B-DRCC2
OLM ID- DICC umh
(LM6-) DIDICC
OLMo-1 ID DICC WHO
Mo-Mo Mlo-IB-DHC IB-HICH W:O unh
OLMMo-18
Method
H8
2.295
3,266
2.512
2,300
2. 3077
2,205
1.20
3,909
2.292
2.31
320
2,519
Achan
04. 100) Anmolis
2.593
2,008
2,610
2,501
2,591
2,088
2.589
2.591
2,600
2.636
2
2.739
2.743
2.752
2.757
2.750
2 SM
2.754
2.734
2.760
2.73
2.702
04. 01
Table 16: Losses of V2 validation sets for IB Model.
3,347
3,357
3,357
3,360
3,355
3,350
33367
3,350
3,344
3300
3.37
3,364
3,203
2.684
2
2.705
2,700
2.725
2.719
Scalin 21 DHC W/ cath
3.063
3,004
3.066
3.001
3,069
3. 102
3,085
2.567
2.
3,503
WC
585
SIC
2,507
2,564
2:587
2,506
2.500
2.56
5.553
2,609
2.504
m222 wiki
manospiere
3
5025
3014
3018
3,023
SNOW
5.005
3022
3,006
3,055
3,036
2
2.508
2.50
2.511
2.003
2
2,509
2
2.50
2.51
2,522
1004 .m
2
2.33
2.208
2.219
2.31
2
2.551
2.364
2,250
pile
2.888
2,500
3,880
2 3,503
2.87
2.890
2.99H
2.87%
3,588
5.998
2.54
2.953
IwitterAAF
3.543
3.631
3,641
3,635
3.203
3.672
wikkent 103
2,006
2.643
2,40
2611
2636
2.635
2.673
2.65
2.783
2,802
2,200
2.77%
2,78
2,902
2.773
2.779
2.292
2.822
2
34

---

Published as a conference paper at ICLR 2025
B-DDEC MLM OLMo--IB-Di DE-DCC WAD unh
OlL IB- SHOC2
OR Mi-M 1B DHDCCA
OLMo BED WHO unh
OLM- WHO
OEMo- OLMo-IB-DHCC 2 2 WAO kmh
OLMo- I8 / WDD knc
ULMo-188
Method
SHCC
anth
9.932
10:045
9923
9XTT
10961
9.897
9.200
10.174
10.66
4chan
c4 100 Aunoiiss
13 386
13
12
13
13 601
12.340
13
12 568
12
13:313
13 360
13
10.0
15510
154475
15.721
15755
15.66
15
15710
15.367
SS41
15000
155829
c4. .m
Table 17: Perplexities of V2 validation sets for 1B models.
28
25.117
28.009
28.001
782
28,500
28
20
28
25.340
29.338
28,901
[4661
14.222
147766
14931
14.641
14633
14925
14.58
47676
259
15.166
Scalnn ais DHC WD cath
Ia202 sdowe
21,130
22,
21 30
Scalia
21 21-110
21 340
21,357
21 162663
22.231
31.860
13.05
15.263
WC
1 200
SIC
15006
13
12,960
12.965
13 33.87
183373
m222 wiki
manoplere
20-253
502
20
20
20
20.18
20-823
20,651
12.122
12.35
12-276
12.310
12:30
12.100
120000
12.044
12.09
12.562
12:45
1n144 cm
9.200
9,300
9.15
9.374
9.199
9332
9.55
9.219
9.600
9.48
pile
18:
19/116
18.0
18.70
13.740
18.102
18
19.326
17.782
18 18
19007
19:161
37,610
38,5 500
38.212
57,341
136
20.592
37,650
37.708
40 580
30 33.3336
Wikileed 103
13.766
3.553
14.0001
$4.0600
13.570
13,006
13.957
594
1-162
14,251
17.50
17.912
17.778
17.445
13 17:509
12 $255H
17:425
17-451
18 270
18.0225
avg
35

---

Published as a conference paper at ICLR 2025
OL Mo-1B JB- D--DICC WWO Anh
Mo-
OLL
OL MB- SHOC
CH BL.M--RR-D MEAM--RR-DHCA DWCW
OL 18-M-118 1--I----HHH DHOS
OH M8-188-111 18 DHCO
ML ID DIIC WWO unh
OLM6-1B W/O
OLMc-IB MA------DHCCS-- DHC WAO kanh
OLMo-IB MM--B--DCCC DICH WHD unb
0LMb-1B
Method
SHCK
3,079
2.005
2.689
698
2,677
2.075
2.6044
2.67
3.676
2.712
2,202
.01
dolma books
2,0086
2 900
3,000
2,8766
2.993
2,87%
2 880
2,925
2,995
dolina common-csww
2.702
2.14
2,716
2.711
2,718
2.701
2
2212
2,695
3,706
3,608
2,232
2,722
Table 18: Losses of V3 validation sets for 1B model.
delma pen
2.900
2.
2.724
2.315
3.255
3.301
2. 3.101
2.921
2.903
2.317
3,3066
2,3/9
2,353
Scaling ain DHC W/ cath
Scating
dolno, peddit
2.0
2,953
2,900
2064
2.956
2,960
2.991 3.961
2,000
Betn
WC
STC
dolnoo stackt
1.001
1,0E5
1,028
1.003
1,022
1.001
1.002
1.045
1.024
1,015
1.041
doliaa miki
2.102
2.458
2,477
2.472
2.479
2,456
2,455
2.48
2.400
2.454
2.456
2.009
2.487
2,000
2.684
2,688
2.200
2,600
2.679
2.099
2,600
2. 2.653
2.721
2.715
10222 .2n
3,201
3.198
8.8
3.1
5.000
3.174
3.219
3.399
2.204
2.204
2.221
2.221
2.201
2,200
2,25
2.200
2.204
2.216
2,232
pile
wikilext 103
2.008
2,649
2,650
2.0
2.017
2,42
2
2.616
2.617
2.677
2.665
3,520
2.537
2,537
2.515
2.554
2.514
2,5
2.556
2,544
avg
36

---

Published as a conference paper at ICLR 2025
OLM OLLMo-L--DIR IB-DICx1 WAD anh
OLM
OLM SECCK
OL IB SHOC2
OL. OLM--IE-DCS BB-DAC
OLM6-1 FEMM-1B-DDES B--DHCK
OLMo-1 BLM--1B-DRE B-DACK2
OLMo ID-DIC unh
OLM--1 IB-DHCK
OLM- DD-DH WHO
OL EMM--B--DOC IB-DHCa WAO ont
01.M-118
Method
I--BCCdl
14,593
14.574
14:10
14,717
14:54
14,566
14-514
14.994
14.531
1.006
1449000
.0
dolina books
L7,920
18,224
L8,026
200
77.007
17:245
18.190
17,749
178177
180690
122280
dolna commoni-cran
Table 19: Perplexities of V3 validation sets for 1B models.
11.001
14
15.
15.019
15 150
14.500
14,320
15:051
153361
14.113
14.975
14
15 15.35
15 15.216
dolun pos2o
10092
10.255
10.21
10,230
10011
9.99
10.19)
10,000
10.1
100088
10-133
10,305
Scaling ain DHC WV cat
Scaling
delma reddit
19:405
19,6500
19.550
19,689
19.966
19:343
196612
19.306
19:479
199009 19,333
19,000
330
SIC
WC
dolmn stack
2.785
2.796
2,8077
2.779
2,776
2,80G
3,783
2,813
2,832
dedina wiki
11.724
11,816
19,655
18915
11.68 1,600
12.006
504
160
167
11,505
14,647
14:054
11,590
14.876
14.570
14.575
14820
14.597
14 14.608
15
15,098
11242 220r
24,108
24,233
24,552
24,107
24.478
23.004
23.945
24.509
23.94K
23,905
25.013
24.503
9.220
9.214
9.100
9A12K
9L181
9.021
9.061
9-651
9.319
pile
wikitest 103
13.009
14.135
14.150
13,089
13,084
13
14,540
14.354
13.900
13,874
14,125
14.152
19,025
1
14.14
13,819
13844
14-12
14,229
37


Reviews:
[{"review_id": "ve8bWNoGsx", "reviewer": "Reviewer_HHYn", "summary": "The paper introduces Hyper-Connections, a novel extension to residual connections that dynamically adjusts the strength of connections between layers in deep neural networks. This method addresses the limitations of traditional residual connections, such as gradient vanishing and representation collapse, by introducing Depth-Connections and Width-Connections thus enabling both cross-layer and intra-layer interactions. A dynamic variant of residual connection named Dynamic Hyper-Connections (DHC), further adapts connection strengths based on input. The approach is evaluated extensively across pretraining of large language models (LLMs), Mixture-of-Experts (MoE) models on vision tasks. Experimental results demonstrate significant improvements in training stability, convergence speed, and model performance on various benchmarks, highlighting Hyper-Connections as a general-purpose enhancement for neural architectures.", "questions": "- The author mentioned other baselines such as Altup and ResiDual had gains in the early stages of training, Can the author show the full loss curve of OLMo-1B-ResiDual and OLMo-1B-Altup×2 in Table4?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper provides a clear and systematic extension to residual connections named Dynamic Hyper-Connections (DHC), where residual could be consider a static hyperconnection, addressing the trade-off between gradient vanishing and representation collapse.\n- Experimenttal results demonstrated effectiveness of DHC across diverse domains, including LLM pretraining and vision tasks.", "weaknesses": "- There seem a lack of comparsion to a fullt enabled depth-connections and width-connections (DenseNet style) where all of the connection in Figure 2 are enabled and learnable.\n- The main result focus on LLM and downstream peformance on langugage tasks, the result on Vision task in the appendix seem to demonstrate less gain compare to langugage tasks, can the author elebrate more on this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vw3SMfdF6q", "reviewer": "Reviewer_DUBh", "summary": "This paper presents hyper-connexions, a new neural network architectural improvement which consists in dynamically adjusting the residual connections between layers, effectively managing the trade-off between vanishing gradients and feature collapse. Many experiments with LLMs and vision models demonstrate the effectiveness of hyper-connections to improve stability of training and downstream performance. Various hyper-connections patterns are also studied in depth, with thorough ablations and visualizations.", "questions": "- Why is expansion rate = 1 worse than no hyper-connection, do you have an intuition ?\n\n- Do these findings generalize to other types of architectures such as ResNets ?\n\n- Line 345 typo: “Static”", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- There is a clear signal that incorporating hyper-connections in LLMs architectures, without any other modification, improves the training loss for a given number of tokens, and boosts performance on downstream metrics. This result is validated for both dense and MOE architectures.\n\n- Hyper-connexions help reduce training instabilities. This is clear by looking at Figure 5 and 6, the training curves of the models with hyper-connections are smoother and do not have spikes, which is a major advantage for training large models.\n\n- The author did a thorough analysis of the learned connections patterns, with nice visualizations.\n\n- The results generalizes to the vision modalities with experiments on image generation and classification. Hyper-connections seem to be a general improvement for the transformer architecture.", "weaknesses": "- The main concern I have with this paper is the computational impact of replicating the activations of the network $n$ times for hyper-connections. There is no study on the computational impact both in terms of running time and memory usage. The authors mention Line 394 that “Both methods expand the hidden size by n times with negligible computational overhead” but it is not shown with a proper experiment on the throughput, overall running time, and peak memory usage. Also, it seems that n=1 performs worse than no hyper-connection, so if n>=2 is necessary, and the memory usage is high, it is necessary to study the trade-off between downstream performance, stability and computational cost.\n\n- Although the signal is promising, a full experiment with scaling the number of tokens beyond 500B will be necessary to fully validate the approach. Not asking for this experiment, but current best LLMs are trained on many more tokens and exhibit much better performance than the number reported. I would be curious to see if hyper-connections are useful for training state-of-the-art LLMs.\n\n- In section 3.2 several non-learnable patterns are presented but are not tried in practice. It is not clear whether learning the hyper-connection patterns is really better that having simple fixed patterns, and an analysis on that would be interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nw03LrO3V1", "reviewer": "Reviewer_VJDJ", "summary": "The paper introduced hyperconnection - width and depth connections with learnable strengths. Hyperconnections are proposed to find a balance between the seesaw effect noticed between vanishing gradients and representation collapse. The input is split into n copies for different connections, and these different copies are added together before layer computation.  The paper also introduced dynamic hyperconnections that dynamically update the strength of connections based on inputs. The paper performs comprehensive analysis of effect of hyperconnections on OLMO and OLMOE and image generation and classifcation problems and investigates its effect with prominent residual connections.", "questions": "Some questions that may add more value to the paper:\n1. With Dynamic Hyperconnections, there is a possibility for redundant connections (when strength becomes 0) how does these cases affect the seesaw effect of gradient vanishing and representation collapse.\n2. Do DHC connections behave similar for images belonging to the same class in case of image classification?\n\nThings I expect from the rebuttal are actions on points 2 and 3.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The results on LLM benchmarks and losses suggest a better balance between vanishing gradients and representation collapse.\n\n2. Section 4.5 discusses the effect of hyperconnections, which displays that hyperconnections eliminate input embeddings from the output, form parallel blocks which have less reliance on each other increasing chances for unique representations. \n\n3.Parallel block formation is particularly important as similar layers in a transformer block tends to learn similar representations. The ability to parallely form blocks dynamically based on input reduces the chances for similar representations enabling better models.\n\n4. Assessing the hyperconnections can reveal some internal logic of the neural network. For instance, we may find that a set of classes follow similar paths compared to other classes.\n\n5. The paper is well detailed and mathematically sound, proofs, hyperparameters and other implementation details are discussed appropriately.", "weaknesses": "1. The main drawback is when creating $n$ copies,  it leads to a considerable amount of increase in memory, though the burden can be reduced through engineering, the impact is yet to be known.\n\n2. If the goal of creating multiple copies is just to make sure multiple depth connections can be modelled parallelly, is creating such copies actually necessary? Can't a single copy be used with different residual strengths? The only difference would be in gradient computation , were additional terms for each depth connection would be added to the singular copy. Can an ablation be conducted between DHC ($n=4$ and $n=1$) and SHC ($n=4$ an $n=1$) to show the importance and need for additional copies. What I am particularly interested is the advantages with updating the gradients into different copies and then adding them before every layer computation verses  updating all the gradients with a single copy. If there is any other specific need for expansion please free to refute this point.\n\n3. Figure 2 caption improvement. The figure can discuss the diagram better, whats $\\beta$, and need to add $\\alpha_{1,0}$ and $\\alpha_{2,0}$ be explained in the diagram itself. This is important as Figure 2 is the central figure that tries to encompass hyperconnections therefore adding these information would make it more clearer for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cXcjRgFGcD", "reviewer": "Reviewer_fvae", "summary": "The paper introduces hyper-connections, a approach that aims to address the limitations of residual connections of transformers architectures.\nThe hyper-connections approach introduce depth-connections and width-connections, allowing a more customizable interaction between layers. Depth-connections are weighted connections across different layers, while width-connections facilitate two layers at the same depths. More importantly the hyper-connections are learned, so it improves model adaptability.\nModels with hyper-connections have significant performance improvements over the original architectures.", "questions": "1. In the introduction, you stated that \"Post-Norm applies normalization operations after the output of each residual block, weakening the\n\"strength\" of residuals.\" I don't think it is correct, since post-norm applies normalization after the summation of the skip and residual branch, so it shouldn't weaken the strength of the residuals.\n\n2. Can you also should the flops count for your model and the original model?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The experimental results seems good enough to claim the hyper-connections is better than the original architectures.", "weaknesses": "1. The presentation is horrible, it is really hard to understand what is model is doing. Probably a better figure or even pseudocode should be provided to explain the methods.\n2. Although the experimental results are good, there is no information on how much extra computation is need to achieve such good results. I suspected that the extra performance is highly due to the extra parameters or extra computation it needed. If you make your model have the same floats, probably you could see that the performance is really similar to the original transformers.\n3. The ordering of the paper is horrible, there is almost no explanation why you do each thing in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)