VeOmni: Scaling Any Modality Model Training with Model-Centric
  Distributed Recipe Zoo
Authors: name: Qianli Ma, name: Yaowei Zheng, name: Zhelun Shi, name: Zhongkai Zhao, name: Bin Jia, name: Ziyue Huang, name: Zhiqi Lin, name: Youjie Li, name: Jiacheng Yang, name: Yanghua Peng, name: Zhi Zhang, name: Xin Liu

Recent advances in large language models (LLMs) have driven impressive
progress in omni-modal understanding and generation. However, training
omni-modal LLMs remains a significant challenge due to the heterogeneous model
architectures required to process diverse modalities, necessitating
sophisticated system design for efficient large-scale training. Existing
frameworks typically entangle model definition with parallel logic, incurring
limited scalability and substantial engineering overhead for end-to-end
omni-modal training. % We present \veomni, a modular and efficient training
framework to accelerate the development of omni-modal LLMs. \veomni introduces
model-centric distributed recipes that decouples communication from
computation, enabling efficient 3D parallelism on omni-modal LLMs. \veomni also
features a flexible configuration interface supporting seamless integration of
new modalities with minimal code change. % Using \veomni, a omni-modal
mixture-of-experts (MoE) model with 30B parameters can be trained with over
2,800 tokens/sec/GPU throughput and scale to 160K context lengths via 3D
parallelism on 128 GPUs, showcasing its superior efficiency and scalability for
training large omni-modal LLMs.