Baichuan-Omni-1.5 Technical Report
Authors: name: Yadong Li; name: Jun Liu; name: Tao Zhang; name: Tao Zhang; name: Song Chen; name: Tianpeng Li; name: Zehuan Li; name: Lijun Liu; name: Lingfeng Ming; name: Guosheng Dong; name: Da Pan; name: Chong Li; name: Yuanbo Fang; name: Dongdong Kuang; name: Mingrui Wang; name: Chenglin Zhu; name: Youwei Zhang; name: Hongyu Guo; name: Fengyu Zhang; name: Yuran Wang; name: Bowen Ding; name: Wei Song; name: Xu Li; name: Yuqi Huo; name: Zheng Liang; name: Shusen Zhang; name: Xin Wu; name: Shuai Zhao; name: Linchu Xiong; name: Yozhen Wu; name: Jiahui Ye; name: Wenhao Lu; name: Bowen Li; name: Yan Zhang; name: Yaqi Zhou; name: Xin Chen; name: Lei Su; name: Hongda Zhang; name: Fuzhong Chen; name: Xuezhen Dong; name: Na Nie; name: Zhiying Wu; name: Bin Xiao; name: Ting Li; name: Shunya Dang; name: Ping Zhang; name: Yijia Sun; name: Jincheng Wu; name: Jinjie Yang; name: Xionghai Lin; name: Zhi Ma; name: Kegeng Wu; name: Jia li; name: Aiyuan Yang; name: Hui Liu; name: Jianqiang Zhang; name: Xiaoxi Chen; name: Guangwei Ai; name: Wentao Zhang; name: Yicong Chen; name: Xiaoqin Huang; name: Kun Li; name: Wenjing Luo; name: Yifei Duan; name: Lingling Zhu; name: Ran Xiao; name: Zhe Su; name: Jiani Pu; name: Dian Wang; name: Xu Jia; name: Tianyu Zhang; name: Mengyu Ai; name: Mang Wang; name: Yujing Qiao; name: Lei Zhang; name: Yanjun Shen; name: Fan Yang; name: Miao Zhen; name: Yijie Zhou; name: Mingyang Chen; name: Fei Li; name: Chenzheng Zhu; name: Keer Lu; name: Yaqi Zhao; name: Hao Liang; name: Youquan Li; name: Yanzhao Qin; name: Linzhuang Sun; name: Jianhua Xu; name: Haoze Sun; name: Mingan Lin; name: Zenan Zhou; name: Weipeng Chen
Published: 2025-01-26 02:19:03

Abstract: We introduce Baichuan-Omni-1.5, an omni-modal model that not only has
omni-modal understanding capabilities but also provides end-to-end audio
generation capabilities. To achieve fluent and high-quality interaction across
modalities without compromising the capabilities of any modality, we
prioritized optimizing three key aspects. First, we establish a comprehensive
data cleaning and synthesis pipeline for multimodal data, obtaining about 500B
high-quality data (text, audio, and vision). Second, an audio-tokenizer
(Baichuan-Audio-Tokenizer) has been designed to capture both semantic and
acoustic information from audio, enabling seamless integration and enhanced
compatibility with MLLM. Lastly, we designed a multi-stage training strategy
that progressively integrates multimodal alignment and multitask fine-tuning,
ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads
contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of
comprehensive omni-modal capabilities. Notably, it achieves results comparable
to leading models such as Qwen2-VL-72B across various multimodal medical
benchmarks.