NeurIPS 2025

Reading Recognition in the Wild
Charig Yang1,2, Samiul Alam3, Shakhrul Iman Siam3, Michael J. Proulx1, Lambert Mathias1,
Kiran Somasundaram1, Luis Pesqueira1, James Fort1, Sheroze Sheriffdeen1, Omkar Parkhi1,
Carl Ren1, Mi Zhang3, Yuning Chai1, Richard Newcombe1, Hyo Jin Kim1
1Meta Reality Labs Research
2VGG, University of Oxford
3The Ohio State University
charig@robots.ox.ac.uk, kimhyojin@meta.com
https://www.projectaria.com/datasets/reading-in-the-wild/
Abstract
To enable egocentric contextual AI in always-on smart glasses, it is crucial to be able
to keep a record of the user’s interactions with the world, including during reading.
In this paper, we introduce a new task of reading recognition to determine when
the user is reading. We first introduce the first-of-its-kind large-scale multimodal
Reading in the Wild dataset, containing 100 hours of reading and non-reading videos
in diverse and realistic scenarios. We then identify three modalities (egocentric
RGB, eye gaze, head pose) that can be used to solve the task, and present a flexible
transformer model that performs the task using these modalities, either individually
or combined. We show that these modalities are relevant and complementary to
the task, and investigate how to efficiently and effectively encode each modality.
Additionally, we show the usefulness of this dataset towards classifying types of
reading, extending current reading understanding studies conducted in constrained
settings to larger scale, diversity and realism. Code, model, and data will be public.
Figure 1: Am I reading? The left figure shows a timeline as the user navigates the world. We aim to solve the
task of reading recognition to enable AI assistants in always-on wearables. We identify three modalities: eye
gaze (in colored dot patterns), RGB crop around gaze (in red box), and inertial sensors performs the task to high
accuracy (with Prediction and GT shown). Images from our Reading in the Wild dataset, which features 100
hours ...