Proactive Assistant Dialogue Generation from Streaming Egocentric Videos
Authors: name: Yichi Zhang, name: Xin Luna Dong, name: Zhaojiang Lin, name: Andrea Madotto, name: Anuj Kumar, name: Babak Damavandi, name: Joyce Chai, name: Seungwhan Moon

Recent advances in conversational AI have been substantial, but developing
real-time systems for perceptual task guidance remains challenging. These
systems must provide interactive, proactive assistance based on streaming
visual inputs, yet their development is constrained by the costly and
labor-intensive process of data collection and system evaluation. To address
these limitations, we present a comprehensive framework with three key
contributions. First, we introduce a novel data curation pipeline that
synthesizes dialogues from annotated egocentric videos, resulting in \dataset,
a large-scale synthetic dialogue dataset spanning multiple domains. Second, we
develop a suite of automatic evaluation metrics, validated through extensive
human studies. Third, we propose an end-to-end model that processes streaming
video inputs to generate contextually appropriate responses, incorporating
novel techniques for handling data imbalance and long-duration videos. This
work lays the foundation for developing real-time, proactive AI assistants
capable of guiding users through diverse tasks. Project page:
https://pro-assist.github.io/