Chat with AI: The Surprising Turn of Real-time Video Communication from
  Human to AI
Authors: name: Jiangkai Wu, name: Zhiyuan Ren, name: Liming Liu, name: Xinggong Zhang

AI Video Chat emerges as a new paradigm for Real-time Communication (RTC),
where one peer is not a human, but a Multimodal Large Language Model (MLLM).
This makes interaction between humans and AI more intuitive, as if chatting
face-to-face with a real person. However, this poses significant challenges to
latency, because the MLLM inference takes up most of the response time, leaving
very little time for video streaming. Due to network uncertainty and
instability, transmission latency becomes a critical bottleneck preventing AI
from being like a real person. To address this, we propose Artic, an
AI-oriented Real-time Communication framework, exploring the network
requirement shift from "humans watching video" to "AI understanding video". To
reduce bitrate dramatically while maintaining MLLM accuracy, we propose
Context-Aware Video Streaming that recognizes the importance of each video
region for chat and allocates bitrate almost exclusively to chat-important
regions. To avoid packet retransmission, we propose Loss-Resilient Adaptive
Frame Rate that leverages previous frames to substitute for lost/delayed frames
while avoiding bitrate waste. To evaluate the impact of video streaming quality
on MLLM accuracy, we build the first benchmark, named Degraded Video
Understanding Benchmark (DeViBench). Finally, we discuss some open questions
and ongoing solutions for AI Video Chat.