VeOmni: Scaling Any Modality Model Training with Model-Centric
  Distributed Recipe Zoo
Authors: name: Qianli Ma; name: Yaowei Zheng; name: Zhelun Shi; name: Zhongkai Zhao; name: Bin Jia; name: Ziyue Huang; name: Zhiqi Lin; name: Youjie Li; name: Jiacheng Yang; name: Yanghua Peng; name: Zhi Zhang; name: Xin Liu
Published: 2025-08-04 11:33:04

Abstract: Recent advances in large language models (LLMs) have driven impressive
progress in omni-modal understanding and generation. However, training
omni-modal LLMs remains a significant challenge due to the heterogeneous model
architectures required to process diverse modalities, necessitating
sophisticated system design for efficient large-scale training. Existing
frameworks typically entangle model definition with parallel logic, incurring
limited scalability and substantial engineering overhead for end-to-end
omni-modal training. % We present \veomni, a modular and efficient training
framework to accelerate the development of omni-modal LLMs. \veomni introduces
model-centric distributed recipes that decouples communication from
computation, enabling efficient 3D parallelism on omni-modal LLMs. \veomni also
features a flexible configuration interface supporting seamless integration of
new modalities with minimal code change. % Using \veomni, a omni-modal
mixture-of-experts (MoE) model with 30B parameters can be trained with over
2,800 tokens/sec/GPU throughput and scale to 160K context lengths via 3D
parallelism on 128 GPUs, showcasing its superior efficiency and scalability for
training large omni-modal LLMs.