Can Vision-Language Models Answer Face to Face Questions in the
  Real-World?
Authors: name: Reza Pourreza, name: Rishit Dagli, name: Apratim Bhattacharyya, name: Sunny Panchal, name: Guillaume Berger, name: Roland Memisevic

AI models have made significant strides in recent years in their ability to
describe and answer questions about real-world images. They have also made
progress in the ability to converse with users in real-time using audio input.
This raises the question: have we reached the point where AI models, connected
to a camera and microphone, can converse with users in real-time about scenes
and events that are unfolding live in front of the camera? This has been a
long-standing goal in AI and is a prerequisite for real-world AI assistants and
humanoid robots to interact with humans in everyday situations. In this work,
we introduce a new dataset and benchmark, the Qualcomm Interactive Video
Dataset (IVD), which allows us to assess the extent to which existing models
can support these abilities, and to what degree these capabilities can be
instilled through fine-tuning. The dataset is based on a simple
question-answering setup, where users ask questions that the system has to
answer, in real-time, based on the camera and audio input. We show that
existing models fall far behind human performance on this task, and we identify
the main sources for the performance gap. However, we also show that for many
of the required perceptual skills, fine-tuning on this form of data can
significantly reduce this gap.