ViSpeak: Visual Instruction Feedback in Streaming Videos
Authors: name: Shenghao Fu, name: Qize Yang, name: Yuan-Ming Li, name: Yi-Xing Peng, name: Kun-Yu Lin, name: Xihan Wei, name: Jian-Fang Hu, name: Xiaohua Xie, name: Wei-Shi Zheng

Recent advances in Large Multi-modal Models (LMMs) are primarily focused on
offline video understanding. Instead, streaming video understanding poses great
challenges to recent models due to its time-sensitive, omni-modal and
interactive characteristics. In this work, we aim to extend the streaming video
understanding from a new perspective and propose a novel task named Visual
Instruction Feedback in which models should be aware of visual contents and
learn to extract instructions from them. For example, when users wave their
hands to agents, agents should recognize the gesture and start conversations
with welcome information. Thus, following instructions in visual modality
greatly enhances user-agent interactions. To facilitate research, we define
seven key subtasks highly relevant to visual modality and collect the
ViSpeak-Instruct dataset for training and the ViSpeak-Bench for evaluation.
Further, we propose the ViSpeak model, which is a SOTA streaming video
understanding LMM with GPT-4o-level performance on various streaming video
understanding benchmarks. After finetuning on our ViSpeak-Instruct dataset,
ViSpeak is equipped with basic visual instruction feedback ability, serving as
a solid baseline for future research.