Baichuan-Omni-1.5 Technical Report
Authors: name: Yadong Li, name: Jun Liu, name: Tao Zhang, name: Tao Zhang, name: Song Chen, name: Tianpeng Li, name: Zehuan Li, name: Lijun Liu, name: Lingfeng Ming, name: Guosheng Dong, name: Da Pan, name: Chong Li, name: Yuanbo Fang, name: Dongdong Kuang, name: Mingrui Wang, name: Chenglin Zhu, name: Youwei Zhang, name: Hongyu Guo, name: Fengyu Zhang, name: Yuran Wang, name: Bowen Ding, name: Wei Song, name: Xu Li, name: Yuqi Huo, name: Zheng Liang, name: Shusen Zhang, name: Xin Wu, name: Shuai Zhao, name: Linchu Xiong, name: Yozhen Wu, name: Jiahui Ye, name: Wenhao Lu, name: Bowen Li, name: Yan Zhang, name: Yaqi Zhou, name: Xin Chen, name: Lei Su, name: Hongda Zhang, name: Fuzhong Chen, name: Xuezhen Dong, name: Na Nie, name: Zhiying Wu, name: Bin Xiao, name: Ting Li, name: Shunya Dang, name: Ping Zhang, name: Yijia Sun, name: Jincheng Wu, name: Jinjie Yang, name: Xionghai Lin, name: Zhi Ma, name: Kegeng Wu, name: Jia li, name: Aiyuan Yang, name: Hui Liu, name: Jianqiang Zhang, name: Xiaoxi Chen, name: Guangwei Ai, name: Wentao Zhang, name: Yicong Chen, name: Xiaoqin Huang, name: Kun Li, name: Wenjing Luo, name: Yifei Duan, name: Lingling Zhu, name: Ran Xiao, name: Zhe Su, name: Jiani Pu, name: Dian Wang, name: Xu Jia, name: Tianyu Zhang, name: Mengyu Ai, name: Mang Wang, name: Yujing Qiao, name: Lei Zhang, name: Yanjun Shen, name: Fan Yang, name: Miao Zhen, name: Yijie Zhou, name: Mingyang Chen, name: Fei Li, name: Chenzheng Zhu, name: Keer Lu, name: Yaqi Zhao, name: Hao Liang, name: Youquan Li, name: Yanzhao Qin, name: Linzhuang Sun, name: Jianhua Xu, name: Haoze Sun, name: Mingan Lin, name: Zenan Zhou, name: Weipeng Chen

We introduce Baichuan-Omni-1.5, an omni-modal model that not only has
omni-modal understanding capabilities but also provides end-to-end audio
generation capabilities. To achieve fluent and high-quality interaction across
modalities without compromising the capabilities of any modality, we
prioritized optimizing three key aspects. First, we establish a comprehensive
data cleaning and synthesis pipeline for multimodal data, obtaining about 500B
high-quality data (text, audio, and vision). Second, an audio-tokenizer
(Baichuan-Audio-Tokenizer) has been designed to capture both semantic and
acoustic information from audio, enabling seamless integration and enhanced
compatibility with MLLM. Lastly, we designed a multi-stage training strategy
that progressively integrates multimodal alignment and multitask fine-tuning,
ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads
contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of
comprehensive omni-modal capabilities. Notably, it achieves results comparable
to leading models such as Qwen2-VL-72B across various multimodal medical
benchmarks.