Decoding Reading Goals from Eye Movements

Authors: Omer Shubi, Cfir Avraham Hadar, Yevgeni Berzak
Primary area: applications to neuroscience & cognitive science
Keywords: application of language models to cognitive science and psycholinguistic, eye movements in reading, reading goals, cognitive state decoding, multimodal models (eye movements and text)

Abstract:
Readers can have different goals with respect to the text they are reading. Can these goals be decoded from the pattern of their eye movements over the text? In this work, we examine for the first time whether it is possible to decode two types of reading goals that are common in daily life: information seeking and ordinary reading. Using large scale eye-tracking data, we apply to this task a wide range of state-of-the-art models for eye movements and text that cover different architectural and data representation strategies, and further introduce a new model ensemble. We systematically evaluate these models at three levels of generalization: new textual item, new participant, and the combination of both. We find that eye movements contain highly valuable signals for this task. We further perform an error analysis which builds on prior empirical findings on differences between ordinary reading and information seeking and leverages rich textual annotations. This analysis reveals key properties of textual items and participant eye movements that contribute to the difficulty of the task.

Decision: reject
Reviewer scores: 8, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730201154837

thread_type:
official_review

<thread_object>
creator: reviewer_y9qk

<content>
summary:
The authors investigate automatic classification of reading behaviour into information seeking and "ordinary reading". They recorded a new eye tracking dataset for this task and evaluated several relevant approaches.

strengths:
The article is very well written and easy to follow.
The experiments are extensive, sound, and deliver some meaningful insights.
The dataset can be a valuable resource for the community.

weaknesses:
The novelty is limited: the author conduct a number of evaluations of existing methods an a new dataset.
The novelty of the task is also very marginal and incorrectly portrayed as the authors miss highly relevant previous work. Xiuge et al. (2023) distinguished deep from skim reading using eye tracking, which is very similar to what the authors claim as novelty.

Further relevant previous work that should be discussed is by Kunze et al. (2013) who distinguished different document types.
It would also be meaningful to compare the reading goal recognition task to similar tasks that were investigated with eye tracking data. One example is informational versus navigational intent classification (Sharma et al., 2023).

In my opinion, if the authors re-formulate their contribution, taking all the relevant related work into account, the paper could still be interesting. I would not see it at ICLR due to the limited contribution w.r.t. to deep learning, but rather at a more HCI-focused venue such as ETRA, ICMI, IUI,...

References:
-----------

Chen, Xiuge, et al. "Characteristics of Deep and Skim Reading on Smartphones vs. Desktop: A Comparative Study." Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023.

Kunze, Kai, et al. "I know what you are reading: recognition of document types using mobile eye tracking." Proceedings of the 2013 international symposium on wearable computers. 2013.

Sharma, Mansi, et al. "Implicit Search Intent Recognition using EEG and Eye Tracking: Novel Dataset and Cross-User Prediction." Proceedings of the 25th International Conference on Multimodal Interaction. 2023.

questions:
no questions
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706814273

thread_type:
official_review

<thread_object>
creator: reviewer_v1ny

<content>
summary:
This paper adopts an existing dataset and existing methods to test reading goal prediction, which consists in predicting if a human is performing ordinary reading or information seeking (binary classification task). In addition to testing existing methods, the authors introduce a model ensemble which shows improved performance, highlighting that different models capture different information. Finally, the authors study how different trial features affect the model's prediction. Reading time before/after critical span, critical span length and paragraph position seem to be the most crucial features.

strengths:
Originality. The paper proposes an interesting research question, i.e. if one can distinguish between two reading tasks based on eye movements (+ text). This can be seen as an extension of previous work, in particular [1]. The authors suggest that their paper has broader scope and ecological validity than [1]. While this represents an important and valid extension, it is not particularly stark in originality. The methodologies are all adapted from previous works, as well as the data. While taking a different perspective, the conceptual framework is almost identical to previous work [1,2].

Quality. The methods used are appropriate and the analysis performed is appropriate, well presented, and interpreted. The authors released their code, which together with the implementation details provided in the paper should make the results reproducible.

Clarity. The paper is well written and exhaustively reports implementation details. To further improve clarity, I would add a more clear explanation of the difference between the proposed task (reading goal classification) and reading comprehension. 

Significance. The paper represents an interesting contribution to the field of psycholinguistics, exploring a new task. The analysis in Section 6 highlights important features to consider when studying eye movements on text. 

[1] Hollenstein, Nora, et al. "ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation." Proceedings of the Twelfth Language Resources and Evaluation Conference. 2020.
[2] Shubi, Omer, et al. "Fine-Grained Prediction of Reading Comprehension from Eye Movements." arXiv preprint arXiv:2410.04484 (2024).

weaknesses:
- As discussed above, the paper is an interesting extension of previous work but does not particularly shine in terms of originality.

- The authors do not discuss limitations of current methods or dataset. The authors briefly mention the need for different tasks and datasets in Section 8, but I believe a more structured and systematic discussion of limitations is necessary. For example:
  - The paragraphs considered in the study are all short. Do the findings generalise to longer text? 
  - Being taken from newspaper articles, all the paragraphs are expository - they provide information. Do the authors expect to find similar results for, e.g., narrative texts written to entertain? 
  - There are differences between different reading comprehension tasks [1]. For example, students that read a text when doing an exam have likely a different behaviour than students that read just to understand the text. Are these differences relevant to this paper?  

- The scientific implications of the authors’ findings can be better discussed. Based on their findings, how can we improve current methods? How can we fully exploit the potential of eye movements for such a task? 

- While the authors state that reading is an ubiquitous and essential skill, the paper lacks an explanation of why reading goal prediction is an important task to study and which are the applications or areas that will benefit the most from that. 

- Overall, this paper presents interesting research in psycholinguistics. However, I believe its contributions are not strong and original enough to be published at ICLR. In addition, the topic would probably fit better a psycholinguistics or computational linguistics venue.

[1] Mar, Raymond A., et al. "Memory and comprehension of narrative versus expository texts: A meta-analysis." Psychonomic Bulletin & Review 28 (2021): 732-749.

questions:
see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730849711637

thread_type:
official_review

<thread_object>
creator: reviewer_rdfv

<content>
summary:
The paper explores a novel task of predicting reading goals - specifically distinguishing between information-seeking and ordinary reading - based on eye movement data. Utilizing an extensive dataset and advanced machine learning models, the authors demonstrate that eye movement patterns contain valuable signals for goal decoding. The study systematically evaluates several state-of-the-art models, including both eye-movement-only and multimodal (eye movements + text) approaches, and introduces an ensemble model that further improves prediction accuracy. The paper provides an in-depth error analysis, identifying key challenges in goal decoding and highlighting textual and participant-specific factors affecting classification difficulty.

strengths:
Originality: The study introduces a novel problem - decoding reading goals from eye movements - that has not been widely explored. This new application area could encourage further research in cognitive science and multimodal data analysis.
Quality: The use of diverse models and a comprehensive evaluation protocol ensures the quality and reliability of the findings. The error analysis is particularly valuable, as it provides insights into the factors that influence classification success, such as the presence of critical spans and reading time.
Clarity: The paper explains the decoding task well and provides a logical flow of ideas. The inclusion of various model types and a mixed-effects model analysis demonstrates a comprehensive and thought-out approach.
Significance: This study lays a foundation for understanding goal-oriented reading behaviors. While applications are currently exploratory, the findings could aid cognitive science researchers and developers of educational tools who aim to personalize learning experiences based on user behavior.

weaknesses:
Complexity in Model Descriptions: Some model descriptions lack clarity, especially in multimodal integration approaches. Providing visual diagrams or more intuitive breakdowns could make these sections more understandable.
Limited Scope of Goal Types: The study only explores two reading goals (information seeking and ordinary reading). Extending this research to other goals (e.g., skimming, proofreading) could make the findings more broadly applicable.

questions:
Could the authors clarify how reading goal classification would perform in real-world applications where eye-tracking calibration is variable, such as web-based eye tracking?
Have the authors considered testing whether other task-specific goals (e.g., skimming) could also be accurately predicted using these models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730904065551

thread_type:
official_review

<thread_object>
creator: reviewer_yvgr

<content>
summary:
In their paper, the authors explore the option of predicting the goal of a person while reading a text. They distuingish between two different natural reading options. The first setting is a ordinary, general reading for pleasure, context, or prose. The second setting is a sharp turn, where the reader has a specfic goal in mind, answering in question, mimicing application scenarios for every person in daily life.

These are the two labels, for several binary classification models that try to distuingish, given the eye movements, and optionally the text, the goal of the reader. They also introduce a new way of interpreting the model's results using linear mixed models.

strengths:
The paper was a pleasure to read. It introduces all relevant background regarding eye movements and reading, has a clear outline and follows a nice story.

Related work is exhaustive and paints a good picture of both machine learning models used in any kind of reading setting as well as eye-tracking-while-reading in general. They then introduce several machine learning models which consume either eye movements, the text that was read during the recording, or both.

weaknesses:
The only true weakness is that the authors do not introduce a new model which exploits the eye movement while reading setting the authors investigate. 

Standard error not report, additionally, no statistical significance tests were done between trainable models, e.g. best model vs rest. The critical span (8./9. in the linear mixed effect model) for interpretability only works for already known texts. For binary classification AUROC would also be a great metric to report. 

Unfortunately, the anonymous repository is not accessible, as well as the data.

questions:
- Why didn't you report AUROC?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731992651567

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reply to reviewers - summary

comment:
We thank all the reviewers for their very thoughtful reviews and constructive feedback. All four reviewers have a positive opinion of the paper, and found the technical work to be comprehensive and insightful (e.g. Y9QK “experiments are extensive, sound, and deliver some meaningful insights”, v1ny “The methods used are appropriate and the analysis performed is appropriate, well presented, and interpreted”), the research questions and task to be interesting (e.g. v1ny ”the paper proposes an interesting research question”, RdFV “The study introduces a novel problem”) and the paper to be well written (e.g. yvGR “The paper was a pleasure to read”). 

We have taken all the comments into careful account, and have made multiple revisions which, as acknowledged by the reviewers, further improved the manuscript. We provide a summary of key revisions here:

- **Additional prior work**: We added all suggested references (as well as several additional references) to the manuscript and revised the related work section. It now better positions our contribution with respect to prior work on related tasks that involve eye movements, and further underlines the novelty of our open-ended question answering setup which enables studying for the first time arbitrary reader goals with respect the text. (lines 492-493, 512-515).
- **Additional statistical testing**: We added statistical tests for best model vs rest, as well as standard errors for all the models in Tables 1 and 3.
- **Improved clarity and detail in model and data exposition**: We made the model descriptions more detailed (Section 3.2) and added diagrams for all the considered models in Figures 5 and 6 in Appendix A. We also added more information on the data: variability of paragraph lengths (lines 216, 230, 233-234) and examples of eyetracking recordings in both conditions in Figure 7 in Appendix B.
- **Expanded discussion**: we expanded the discussion regarding the scientific value and the limitations of the current work (lines 526-533).

Regarding the scope of the paper’s contributions and the fit to ICLR, we would like to stress that first, we define a **new prediction task** of theoretical and practical importance, and benchmark it against state-of-the-art multimodal models for representing eye movements and for integrating them with text. Second, we propose a **new technique for model and task interpretability** by which we show how models can be leveraged for characterizing critical features for the task at hand via mixed effects modeling, a technique that to our knowledge has not been previously used for this purpose and can be applied to analysis of other classification tasks. Finally, we analyze the models’ performance and provide new insights for the new task, which lead to a **new and effective model ensemble** which outperforms all prior methods. This work is therefore highly appropriate for the *applications to cognitive science track* at ICLR, with additional contributions relevant to representation learning for eye movements, *interpretation of learned representations*, and *benchmarks*. More broadly, we believe that the challenge of multimodal modeling and analysis of eye movements and text, which (differently from other types of multimodal data) has not received much attention in ML thus far, will be of high interest to the ICLR community.

Please find our responses to each review below.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper initially had mixed reviews. The major issues raised were:

1. no new model is introduced that exploits the eye movements while reading  [yvGR]
2. standard errors not reported [yvGR]
3. extension of previous work, limited novelty [v1ny, Y9QK]
4. lack of discussion of limitations and scientific implications [v1ny]
5. contributions are not strong and original enough to be published at ICLR. In addition, the topic would probably fit better a psycholinguistics or computational linguistics venue. [v1ny]
6. I would not see it at ICLR due to the limited contribution w.r.t. to deep learning, but rather at a more HCI-focused venue such as ETRA, ICMI, IUI,.. [Y9QK]

The authors wrote a response. The response and discussion did not assuage the major concerns of the reviewers. In particular, the major concern is about the novelty (it follows previous studies), and the lack of technical contribution in ML. For the latter, authors claim to have several ML contributions: new technique for model and task interpretability (but this based on existing mixed effects models, L287); new type of model ensemble (but their proposed method is just "stacked generalization" in ML). Thus, the paper lacks ML novelty, and it is not suitable for an "applications track" paper in an ML conference -- The AC agrees with v1ny and Y9QK that the paper is more suitable to an HCI / psycholinguistics / computational linguistics venue.

Authors are encouraged to submit their paper to more suitable venues.