Hyper-Connections
Authors: name: Defa Zhu, name: Hongzhi Huang, name: Zihao Huang, name: Yutao Zeng, name: Yunyao Mao, name: Banggu Wu, name: Qiyang Min, name: Xun Zhou

We present hyper-connections, a simple yet effective method that can serve as
an alternative to residual connections. This approach specifically addresses
common drawbacks observed in residual connection variants, such as the seesaw
effect between gradient vanishing and representation collapse. Theoretically,
hyper-connections allow the network to adjust the strength of connections
between features at different depths and dynamically rearrange layers. We
conduct experiments focusing on the pre-training of large language models,
including dense and sparse models, where hyper-connections show significant
performance improvements over residual connections. Additional experiments
conducted on vision tasks also demonstrate similar improvements. We anticipate
that this method will be broadly applicable and beneficial across a wide range
of AI problems.