LLaDA2.0: Scaling Up Diffusion Language Models to 100B
Authors: name: Tiwei Bie; name: Maosong Cao; name: Kun Chen; name: Lun Du; name: Mingliang Gong; name: Zhuochen Gong; name: Yanmei Gu; name: Jiaqi Hu; name: Zenan Huang; name: Zhenzhong Lan; name: Chengxi Li; name: Chongxuan Li; name: Jianguo Li; name: Zehuan Li; name: Huabin Liu; name: Ling Liu; name: Guoshan Lu; name: Xiaocheng Lu; name: Yuxin Ma; name: Jianfeng Tan; name: Lanning Wei; name: Ji-Rong Wen; name: Yipeng Xing; name: Xiaolu Zhang; name: Junbo Zhao; name: Da Zheng; name: Jun Zhou; name: Junlin Zhou; name: Zhanchao Zhou; name: Liwang Zhu; name: Yihong Zhuang
Published: 2025-12-10 09:26:18

Abstract: This paper presents LLaDA2.0 -- a tuple of discrete diffusion large language models (dLLM) scaling up to 100B total parameters through systematic conversion from auto-regressive (AR) models -- establishing a new paradigm for frontier-scale deployment. Instead of costly training from scratch, LLaDA2.0 upholds knowledge inheritance, progressive adaption and efficiency-aware design principle, and seamless converts a pre-trained AR model into dLLM with a novel 3-phase block-level WSD based training scheme: progressive increasing block-size in block diffusion (warm-up), large-scale full-sequence diffusion (stable) and reverting back to compact-size block diffusion (decay). Along with post-training alignment with SFT and DPO, we obtain LLaDA2.0-mini (16B) and LLaDA2.0-flash (100B), two instruction-tuned Mixture-of-Experts (MoE) variants optimized for practical deployment. By preserving the advantages of parallel decoding, these models deliver superior performance and efficiency at the frontier scale. Both models were open-sourced.