Is Extending Modality The Right Path Towards Omni-Modality?
Authors: name: Tinghui Zhu, name: Kai Zhang, name: Muhao Chen, name: Yu Su

Omni-modal language models (OLMs) aim to integrate and reason over diverse
input modalities--such as text, images, video, and audio--while maintaining
strong language capabilities. Despite recent advancements, existing models,
especially open-source ones, remain far from true omni-modality, struggling to
generalize beyond the specific modality pairs they are trained on or to achieve
strong performance when processing multi-modal inputs. We study the effect of
extending modality, the dominant technique for training multimodal models,
where an off-the-shelf language model is fine-tuned on target-domain and
language data. Specifically, we investigate three key questions: (1) Does
modality extension compromise core language abilities? (2) Can model merging
effectively integrate independently fine-tuned modality-specific models to
achieve omni-modality? (3) Does omni-modality extension lead to better
knowledge sharing and generalization compared to sequential extension? Through
extensive experiments, we analyze these trade-offs and provide insights into
the feasibility of achieving true omni-modality using current approaches.