Frac-Connections: Fractional Extension of Hyper-Connections
Authors: name: Defa Zhu, name: Hongzhi Huang, name: Jundong Zhou, name: Zihao Huang, name: Yutao Zeng, name: Banggu Wu, name: Qiyang Min, name: Xun Zhou

Residual connections are central to modern deep learning architectures,
enabling the training of very deep networks by mitigating gradient vanishing.
Hyper-Connections recently generalized residual connections by introducing
multiple connection strengths at different depths, thereby addressing the
seesaw effect between gradient vanishing and representation collapse. However,
Hyper-Connections increase memory access costs by expanding the width of hidden
states. In this paper, we propose Frac-Connections, a novel approach that
divides hidden states into multiple parts rather than expanding their width.
Frac-Connections retain partial benefits of Hyper-Connections while reducing
memory consumption. To validate their effectiveness, we conduct large-scale
experiments on language tasks, with the largest being a 7B MoE model trained on
up to 3T tokens, demonstrating that Frac-Connections significantly outperform
residual connections.