RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
Authors: name: Pierre Sermanet, name: Tianli Ding, name: Jeffrey Zhao, name: Fei Xia, name: Debidatta Dwibedi, name: Keerthana Gopalakrishnan, name: Christine Chan, name: Gabriel Dulac-Arnold, name: Sharath Maddineni, name: Nikhil J Joshi, name: Pete Florence, name: Wei Han, name: Robert Baruch, name: Yao Lu, name: Suvir Mirchandani, name: Peng Xu, name: Pannag Sanketi, name: Karol Hausman, name: Izhak Shafran, name: Brian Ichter, name: Yuan Cao

We present a scalable, bottom-up and intrinsically diverse data collection
scheme that can be used for high-level reasoning with long and medium horizons
and that has 2.2x higher throughput compared to traditional narrow top-down
step-by-step collection. We collect realistic data by performing any user
requests within the entirety of 3 office buildings and using multiple robot and
human embodiments. With this data, we show that models trained on all
embodiments perform better than ones trained on the robot data only, even when
evaluated solely on robot episodes. We find that for a fixed collection budget
it is beneficial to take advantage of cheaper human collection along with robot
collection. We release a large and highly diverse (29,520 unique instructions)
dataset dubbed RoboVQA containing 829,502 (video, text) pairs for
robotics-focused visual question answering. We also demonstrate how evaluating
real robot experiments with an intervention mechanism enables performing tasks
to completion, making it deployable with human oversight even if imperfect
while also providing a single performance metric. We demonstrate a single
video-conditioned model named RoboVQA-VideoCoCa trained on our dataset that is
capable of performing a variety of grounded high-level reasoning tasks in broad
realistic settings with a cognitive intervention rate 46% lower than the
zero-shot state of the art visual language model (VLM) baseline and is able to
guide real robots through long-horizon tasks. The performance gap with
zero-shot state-of-the-art models indicates that a lot of grounded data remains
to be collected for real-world deployment, emphasizing the critical need for
scalable data collection approaches. Finally, we show that video VLMs
significantly outperform single-image VLMs with an average error rate reduction
of 19% across all VQA tasks. Data and videos available at
https://robovqa.github.io