Ultra-Sparse Memory Network
Authors: name: Zihao Huang, name: Qiyang Min, name: Hongzhi Huang, name: Defa Zhu, name: Yutao Zeng, name: Ran Guo, name: Xun Zhou

It is widely acknowledged that the performance of Transformer models is
exponentially related to their number of parameters and computational
complexity. While approaches like Mixture of Experts (MoE) decouple parameter
count from computational complexity, they still face challenges in inference
due to high memory access costs. This work introduces UltraMem, incorporating
large-scale, ultra-sparse memory layer to address these limitations. Our
approach significantly reduces inference latency while maintaining model
performance. We also investigate the scaling laws of this new architecture,
demonstrating that it not only exhibits favorable scaling properties but
outperforms traditional models. In our experiments, we train networks with up
to 20 million memory slots. The results show that our method achieves
state-of-the-art inference speed and model performance within a given
computational budget.