Frac-Connections: Fractional Extension of Hyper-Connections
Authors: name: Defa Zhu; name: Hongzhi Huang; name: Jundong Zhou; name: Zihao Huang; name: Yutao Zeng; name: Banggu Wu; name: Qiyang Min; name: Xun Zhou
Published: 2025-03-18 10:37:50

Abstract: Residual connections are central to modern deep learning architectures,
enabling the training of very deep networks by mitigating gradient vanishing.
Hyper-Connections recently generalized residual connections by introducing
multiple connection strengths at different depths, thereby addressing the
seesaw effect between gradient vanishing and representation collapse. However,
Hyper-Connections increase memory access costs by expanding the width of hidden
states. In this paper, we propose Frac-Connections, a novel approach that
divides hidden states into multiple parts rather than expanding their width.
Frac-Connections retain partial benefits of Hyper-Connections while reducing
memory consumption. To validate their effectiveness, we conduct large-scale
experiments on language tasks, with the largest being a 7B MoE model trained on
up to 3T tokens, demonstrating that Frac-Connections significantly outperform
residual connections.