Title: Reading Recognition in the Wild
Venue: NeurIPS 2025
Accepted: yes
Keywords: eye gaze

egocentric perception

multimodal learning

reading

eye tracking
Abstract: To enable egocentric contextual AI in always-on smart glasses, it is crucial to be able to keep a record of the user's interactions with the world, including during reading. In this paper, we introduce a new task of reading recognition to determine when the user is reading. We first introduce the first-of-its-kind large-scale multimodal Reading in the Wild dataset, containing 100 hours of reading and non-reading videos in diverse and realistic scenarios. We then identify three modalities (egocentric RGB, eye gaze, head pose) that can be used to solve the task, and present a flexible transformer model that performs the task using these modalities, either individually or combined. We show that these modalities are relevant and complementary to the task, and investigate how to efficiently and effectively encode each modality. Additionally, we show the usefulness of this dataset towards classifying types of reading, extending current reading understanding studies conducted in constrained settings to larger scale, diversity and realism. Code, model, and data will be public.


Peer reviews:
comments: SUMMARY:
This paper proposes a method to determine whether a user's behavior is reading or not with the user's sensor information in an AR scenario. To implement the method, the paper presents a dataset containing three modalities. The paper reports that the use of the three modalities can complement each other and achieve efficient reading recognition.

STRENGTHS AND WEAKNESSES:
**Strengths.**

1.  The proposed task is interesting and the video demos are generally stable.
2.  The paper presents a large-scale dataset and provides a detailed introduction, which seems to have taken a lot of effort.
3.  The paper provides detailed experimental analysis.

**Weaknesses.**

1.  Recognition result still exists lags and flickering.
2.  The methodology is a bit too simple, and the architecture is not very original or insightful.
3.  Although the paper explores different inputs and model sizes, as well as high-level comparisons with other methods, there are no experimental comparisons with other related methods (warped to the task), and it's not clear that the method has an architectural lead.

QUESTIONS:
1.  Although the authors claim to have provided a large dataset, the total number of ids is still limited. Is it possible that some of the VLA methods are better in terms of generalization, accuracy, and stability if ft is performed on this downstream task?
2.  Notice that the encoders used in the model are small, especially the RGB encoder, is there any better performance if switching to the current popular visual encoder, such as DINO, etc.?

LIMITATIONS:
AR video data is highly correlated with human privacy, and it is hoped to discuss content related to user privacy.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-30

comments: SUMMARY:
This paper presents a novel method for detecting reading behavior in smartglass wearers using multimodal data. The authors introduce a dataset comprising 100 hours of egocentric video, eye gaze, and IMU data in diverse scenarios. They propose a transformer-based model capable of leveraging each modality individually or in combination. The work is timely, well-executed, and offers strong empirical results.

STRENGTHS AND WEAKNESSES:
Strengths:

+ Timely and relevant task in the context of XR and agentic AI, where understanding user behavior such as reading can enable more intelligent and context-aware systems.

+ The proposed dataset is a significant contribution, offering a rich and realistic benchmark for future work in egocentric activity recognition.

+ The transformer-based approach is well-motivated and clearly described. The ability to handle multiple modalities flexibly is a strong point.

+ Strong performance, achieving over 90% precision and 85% recall, indicating practical viability.

+ Extensive experiments, ablation studies, and thoughtful discussion on generalization to related tasks beyond reading detection.

+ Promised release of code, data, and models.


Minor Weaknesses:

+ The paper lacks discussion on computational efficiency and real-time deployment feasibility on the glasses. For instance, can the model run on the Aria Glasses? What are the implications for battery life and latency?

QUESTIONS:
Can the model be deployed on the glasses? What are the implications for battery life and latency?

LIMITATIONS:
Discussion on limitations is sufficient
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-04

comments: SUMMARY:
The paper addresses a practical problem for always-on smart glasses: efficiently detecting when users are reading to enable contextual AI. The authors introduce a 100-hour multimodal dataset collected using Project Aria glasses and propose a transformer-based model that combines eye gaze, RGB crops, and IMU data to achieve 86.9% accuracy on reading detection.  The proposed dataset can be used to study the reading behavior of people in realistic settings in greater detail which links to cognitive understanding.

STRENGTHS AND WEAKNESSES:
[Strength]

- This paper introduced a novel problem and datasets to the communities, which has good positioning within the broader context of contextual AI

- The introduced dataset is comprehensive, with 100 hours of diverse data across 111 participants, and diverse reading modes and environments. 

- The authors has thorough experimental design and comprehensive analysis of model components, input representations, and hyperparameters. 

[Weakness]

- There might be language bias: Training primarily on English, though cross-language experiments partially address this

- There might be reading material bias: May not cover all types of real-world reading scenarios

QUESTIONS:
Please check the weakness part.

LIMITATIONS:
None
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce reading recognition—a task to detect when a user is reading—for contextual AI in smart glasses. They present the Reading in the Wild dataset (100 hours of egocentric video, gaze, and IMU data across diverse real-world scenarios) and a lightweight transformer model leveraging RGB gaze crops, eye gaze, and head pose (IMU). The modalities are shown to be complementary, with combined use achieving 86.9% accuracy. The dataset and model enable applications like assistive tools for low-vision users and extend to classifying reading modes (e.g., skimming) and mediums (e.g., print).\n\nSTRENGTHS AND WEAKNESSES:\nThe authors introduce reading recognition—a task to detect when a user is reading—for contextual AI in smart glasses. They present the Reading in the Wild dataset (100 hours of egocentric video, gaze, and IMU data across diverse real-world scenarios) and a lightweight transformer model leveraging RGB gaze crops, eye gaze, and head pose (IMU). The modalities are shown to be complementary, with combined use achieving 86.9% accuracy. The dataset and model enable applications like assistive tools for low-vision users and extend to classifying reading modes (e.g., skimming) and mediums (e.g., print).\n\nModel struggles with vertical (Chinese: 35.5% accuracy) and right-to-left (Arabic: 21.0%) texts without augmentations (Table 3b). Reading mode (e.g., skimming vs. scanning) and medium (object vs. digital) tasks underperform (Tables 5–6), suggesting the 2s window is insufficient.\n\nQUESTIONS:\nCould personalizing models to individual reading styles (e.g., speed, gaze patterns) improve edge-case performance?  And for right-to-left languages, would training with minimal flipped-gaze samples (vs. inference-time flipping) enhance generalization?\n\nLIMITATIONS:\nTraining data limited to English; generalization to non-LTR scripts requires manual augmentation. And fixed 2s input may not capture long reading sessions or rapid transitions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}