Stream-Omni: Simultaneous Multimodal Interactions with Large
  Language-Vision-Speech Model
Authors: name: Shaolei Zhang, name: Shoutao Guo, name: Qingkai Fang, name: Yan Zhou, name: Yang Feng

The emergence of GPT-4o-like large multimodal models (LMMs) has raised the
exploration of integrating text, vision, and speech modalities to support more
flexible multimodal interaction. Existing LMMs typically concatenate
representation of modalities along the sequence dimension and feed them into a
large language model (LLM) backbone. While sequence-dimension concatenation is
straightforward for modality integration, it often relies heavily on
large-scale data to learn modality alignments. In this paper, we aim to model
the relationships between modalities more purposefully, thereby achieving more
efficient and flexible modality alignments. To this end, we propose
Stream-Omni, a large language-vision-speech model with efficient modality
alignments, which can simultaneously support interactions under various
modality combinations. Stream-Omni employs LLM as the backbone and aligns the
vision and speech to the text based on their relationships. For vision that is
semantically complementary to text, Stream-Omni uses sequence-dimension
concatenation to achieve vision-text alignment. For speech that is semantically
consistent with text, Stream-Omni introduces a CTC-based layer-dimension
mapping to achieve speech-text alignment. In this way, Stream-Omni can achieve
modality alignments with less data (especially speech), enabling the transfer
of text capabilities to other modalities. Experiments on various benchmarks
demonstrate that Stream-Omni achieves strong performance on visual
understanding, speech interaction, and vision-grounded speech interaction
tasks. Owing to the layer-dimensional mapping, Stream-Omni can simultaneously
provide intermediate text outputs (such as ASR transcriptions and model
responses) during speech interaction, offering users a comprehensive multimodal
experience.