Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend
  NPUs
Authors: name: Yichun Yin, name: Wenyong Huang, name: Kaikai Song, name: Yehui Tang, name: Xueyu Wu, name: Wei Guo, name: Peng Guo, name: Yaoyuan Wang, name: Xiaojun Meng, name: Yasheng Wang, name: Dong Li, name: Can Chen, name: Dandan Tu, name: Yin Li, name: Fisher Yu, name: Ruiming Tang, name: Yunhe Wang, name: Baojun Wang, name: Bin Wang, name: Bo Wang, name: Boxiao Liu, name: Changzheng Zhang, name: Duyu Tang, name: Fei Mi, name: Hui Jin, name: Jiansheng Wei, name: Jiarui Qin, name: Jinpeng Li, name: Jun Zhao, name: Liqun Deng, name: Lin Li, name: Minghui Xu, name: Naifu Zhang, name: Nianzu Zheng, name: Qiang Li, name: Rongju Ruan, name: Shengjun Cheng, name: Tianyu Guo, name: Wei He, name: Wei Li, name: Weiwen Liu, name: Wulong Liu, name: Xinyi Dai, name: Yonghan Dong, name: Yu Pan, name: Yue Li, name: Yufei Wang, name: Yujun Li, name: Yunsheng Ni, name: Zhe Liu, name: Zhenhe Zhang, name: Zhicheng Liu

We present Pangu Ultra, a Large Language Model (LLM) with 135 billion
parameters and dense Transformer modules trained on Ascend Neural Processing
Units (NPUs). Although the field of LLM has been witnessing unprecedented
advances in pushing the scale and capability of LLM in recent years, training
such a large-scale model still involves significant optimization and system
challenges. To stabilize the training process, we propose depth-scaled sandwich
normalization, which effectively eliminates loss spikes during the training
process of deep models. We pre-train our model on 13.2 trillion diverse and
high-quality tokens and further enhance its reasoning capabilities during
post-training. To perform such large-scale training efficiently, we utilize
8,192 Ascend NPUs with a series of system optimizations. Evaluations on
multiple diverse benchmarks indicate that Pangu Ultra significantly advances
the state-of-the-art capabilities of dense LLMs such as Llama 405B and Mistral
Large 2, and even achieves competitive results with DeepSeek-R1, whose sparse
model structure contains much more parameters. Our exploration demonstrates
that Ascend NPUs are capable of efficiently and effectively training dense
models with more than 100 billion parameters. Our model and system will be
available for our commercial customers.