Proactive Assistant Dialogue Generation from Streaming Egocentric Videos
Authors: name: Yichi Zhang; name: Xin Luna Dong; name: Zhaojiang Lin; name: Andrea Madotto; name: Anuj Kumar; name: Babak Damavandi; name: Joyce Chai; name: Seungwhan Moon
Published: 2025-06-06 09:23:29

Abstract: Recent advances in conversational AI have been substantial, but developing
real-time systems for perceptual task guidance remains challenging. These
systems must provide interactive, proactive assistance based on streaming
visual inputs, yet their development is constrained by the costly and
labor-intensive process of data collection and system evaluation. To address
these limitations, we present a comprehensive framework with three key
contributions. First, we introduce a novel data curation pipeline that
synthesizes dialogues from annotated egocentric videos, resulting in \dataset,
a large-scale synthetic dialogue dataset spanning multiple domains. Second, we
develop a suite of automatic evaluation metrics, validated through extensive
human studies. Third, we propose an end-to-end model that processes streaming
video inputs to generate contextually appropriate responses, incorporating
novel techniques for handling data imbalance and long-duration videos. This
work lays the foundation for developing real-time, proactive AI assistants
capable of guiding users through diverse tasks. Project page:
https://pro-assist.github.io/