On Domain-Specific Post-Training for Multimodal Large Language Models
Authors: name: Daixuan Cheng; name: Shaohan Huang; name: Ziyu Zhu; name: Xintong Zhang; name: Wayne Xin Zhao; name: Zhongzhi Luan; name: Bo Dai; name: Zhenliang Zhang
Published: 2024-11-29 18:42:28

Abstract: Recent years have witnessed the rapid development of general multimodal large
language models (MLLMs). However, adapting general MLLMs to specific domains,
such as scientific fields and industrial applications, remains less explored.
This paper systematically investigates domain adaptation of MLLMs through
post-training, focusing on data synthesis, training pipelines, and task
evaluation. (1) Data Synthesis: Using open-source models, we develop a visual
instruction synthesizer that effectively generates diverse visual instruction
tasks from domain-specific image-caption pairs. Our synthetic tasks surpass
those generated by manual rules, GPT-4, and GPT-4V in enhancing the
domain-specific performance of MLLMs. (2) Training Pipeline: While the
two-stage training--initially on image-caption pairs followed by visual
instruction tasks--is commonly adopted for developing general MLLMs, we apply a
single-stage training pipeline to enhance task diversity for domain-specific
post-training. (3) Task Evaluation: We conduct experiments in two domains,
biomedicine and food, by post-training MLLMs of different sources and scales
(e.g., Qwen2-VL-2B, LLaVA-v1.6-8B, Llama-3.2-11B), and then evaluating MLLM
performance on various domain-specific tasks. To support further research in
MLLM domain adaptation, we will open-source our implementations.