InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for
  Long-term Streaming Video and Audio Interactions
Authors: name: Pan Zhang; name: Xiaoyi Dong; name: Yuhang Cao; name: Yuhang Zang; name: Rui Qian; name: Xilin Wei; name: Lin Chen; name: Yifei Li; name: Junbo Niu; name: Shuangrui Ding; name: Qipeng Guo; name: Haodong Duan; name: Xin Chen; name: Han Lv; name: Zheng Nie; name: Min Zhang; name: Bin Wang; name: Wenwei Zhang; name: Xinyue Zhang; name: Jiaye Ge; name: Wei Li; name: Jingwen Li; name: Zhongying Tu; name: Conghui He; name: Xingcheng Zhang; name: Kai Chen; name: Yu Qiao; name: Dahua Lin; name: Jiaqi Wang
Published: 2024-12-12 18:58:30

Abstract: Creating AI systems that can interact with environments over long periods,
similar to human cognition, has been a longstanding research goal. Recent
advancements in multimodal large language models (MLLMs) have made significant
strides in open-world understanding. However, the challenge of continuous and
simultaneous streaming perception, memory, and reasoning remains largely
unexplored. Current MLLMs are constrained by their sequence-to-sequence
architecture, which limits their ability to process inputs and generate
responses simultaneously, akin to being unable to think while perceiving.
Furthermore, relying on long contexts to store historical data is impractical
for long-term interactions, as retaining all information becomes costly and
inefficient. Therefore, rather than relying on a single foundation model to
perform all functions, this project draws inspiration from the concept of the
Specialized Generalist AI and introduces disentangled streaming perception,
reasoning, and memory mechanisms, enabling real-time interaction with streaming
video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive
(IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module:
Processes multimodal information in real-time, storing key details in memory
and triggering reasoning in response to user queries. (2) Multi-modal Long
Memory Module: Integrates short-term and long-term memory, compressing
short-term memories into long-term ones for efficient retrieval and improved
accuracy. (3) Reasoning Module: Responds to queries and executes reasoning
tasks, coordinating with the perception and memory modules. This project
simulates human-like cognition, enabling multimodal large language models to
provide continuous and adaptive service over time.