Comparative Evaluation of Computational Models Predicting Eye Fixation Patterns During Reading: Insights from Transformers and Simpler Architectures
Authors: Alessandro Lento, Andrea Nadalini, Nadia Khlif, Vito Pirrelli, Claudia Marzi, Marcello Ferro (CNR-ILC, Università Campus Bio-Medico, University Mohammed First, 2024)
Topic: Language Models

Abstract
Eye tracking data during reading provides significant insights into the cognitive processes underlying language comprehension. It allows for the estimation of lexical, contextual, and higher-level structural effects on word identification through metrics such as fixation duration. Despite advancements in psycholinguistic experiments that have elucidated these effects, the extent to which computational models can predict gaze patterns remains unclear. Recent developments in computational modeling, particularly the use of pre-trained transformer language models, have shown promising results in mirroring human reading behaviors. However, previous studies have not adequately compared these models to alternative architectures or considered various input features comprehensively. This paper addresses these gaps by replicating prior findings on English data, critically evaluating performance metrics, and proposing a stricter accuracy measurement method. Furthermore, it compares different computational models, demonstrating that simpler architectures can achieve results comparable to or better than transformers. The study also emphasizes the significance of individual differences in reading behavior, presenting challenges for simulating natural reading tasks.

Introduction
Eye-tracking data recorded while reading connected text offers valuable insights into the cognitive processes involved in language comprehension. For example, by looking at fixation duration it is possible to estimate the effects that lexical properties (e.g. length, frequencies, orthographic similarity AUTHOR AUTHOR), contextual constraints (e.g. predictability AUTHOR) and higher-level structures (e.g. syntactic structure or prosodic contour AUTHOR) can exert on word identification. While psycholinguistic experiments have reliably assessed how such effects modulate - either linearly or non-linearly - reading times, it is not clear to which extent computational models can directly predict behavioral metrics such as gaze patterns. Over the past 30 years, research in this field has advanced significantly, leading to the development of sophisticated architectures that can account for many aspects of eye movement behavior during reading (e.g. EZ-Reader AUTHOR, Swift AUTHOR). A great boost in such improvements came from the collection and release of eye-tracking corpora (e.g. Geco AUTHOR, Zuco AUTHOR, Meco AUTHOR) which allowed for deep learning techniques to be applied in prediction tasks of eye tracking metrics. Recent work AUTHOR used fine-tuned pre-trained transformer language models achieving remarkably accurate predictions of a wide range of eye-tracking features related to both early and late stages of lexical access. According to such results, these models would accurately mirror the average human reading behavior, suggesting that transformers inherently encode the relative importance of language elements similarly to human cognitive processing mechanisms. However, the authors did not make a direct comparison with alternative computational architectures (e.g. recurrent neural network) and alternative input features (e.g. word length, frequency, predictability, part-of-speech among others), but rather referred to previous literature to support their claim. In the current paper, we directly addressed this issue by replicating the results from AUTHOR on the English data. We critically evaluated the metrics used to assess models' performance and provided an alternative, stricter way to measure accuracy. Next, we tested a comprehensive set of different models on the same task, showing how simpler architectures can achieve comparable or even better results than transformers. Finally, we discussed the overall results in terms of group-level vs individual-level behavior, highlighting the importance of individual differences and how they represent a tough challenge when running simulations of natural reading tasks.

Conclusion
Transformer-based neural networks appear to reasonably predict fixation probability and first-pass duration of words in human reading of English connected texts. Our present investigation basically supports this conclusion, while providing new evidence on two related questions. Two questions naturally arise in this context. How accurate are transformer-based predictions compared with the best predictions of other neural network classifiers trained on the same task? How cognitively plausible are the mechanisms underpinning this performance? Here, we addressed both questions by testing various models on the task of predicting human reading measurements from the GECO corpus, using different evaluation metrics and regressing network predictions on a few linguistic factors that are known to account for human reading behaviour. Our first observation is that assessing a network's performance by looking at its MAE loss function provides a rather gross evaluation of the effective power of a neural network simulating human reading behaviour. A baseline model assigning each token a constant gaze duration that equals the average of all FPD values attested in GECO achieves a 95.7\% loss-based accuracy on both test and training data.\ That a transformer-based classification scores 97.2\% on the same metric and the same test data cannot be held, as such, as a sign of outstanding performance. In fact, it turns out that the MAE loss function is blind to both the magnitude of a network error, and possible biases in the prediction of very low\slash high target values. Thus, it provides an inflated estimate of a model's accuracy. %, to the extent that even the baseline \textit{const} model reached an accuracy of 95\%. We suggest that binary evaluation metrics, based on a fixed threshold partially overcome these limitations. Yet, as single word fixation times typically range between tens to hundreds of milliseconds, application of a fixed threshold will differently affect tokens with different fixation times. We conclude that a relative threshold based on each word's fixation time is a fairer way to measure prediction accuracy. Clearly, this comes at a cost. When assessed with a relative threshold, the accuracy of a transformer-based architecture on test data drops from 70\% down to 57.8\%. It turned out that all other network models tested for the present purposes showed accuracy levels that are comparable to the accuracy of a transformer-based architecture. Since the former are trained on a more restricted set of lexical and contextual input features than the latter, this seems to suggest that word embeddings are of limited use in the task at hand. Although fine-tuned word embeddings actually appear to score much higher on training data (even using accT and accS), we observe that this is due to data overfitting, as clearly shown by the considerably poorer performance of the fine-tuned model on test data. An analysis of the psychometric plausibility of the gaze patterns simulated with our neural models reveals that a relatively small set of linguistic factors that are known to account for a sizeable amount of variance in human fixation times can also account for the bulk of variance in models' behaviour. This is relatively unsurprising, as most of these models were trained on input features that encode at least some of these factors. Nonetheless, we believe that the result is interesting for at least two reasons. First, it shows a promising convergence between computational metrics of model accuracy and quantitative models of psychometric assessment.\ Secondly, it suggests that one can gain non trivial insights in a model's behaviour by analysing to what extent the behaviour is sensitive to the same linguistic factors human readers are known to be sensitive to. On the one hand, this is a step towards understanding what information a neural model is actually learning and putting to use for the task. On the other hand, this is instrumental in developing better models, as it shows what type of input information is more needed to successfully carry out a task, at least if one is trying to simulate the way the same task is carried out by speakers. In the end, it may well be the case that a 70\% fixed-threshold accuracy in simulating average gaze patterns in human reading is not as disappointing as it might seem. Given the wide variability in human reading behaviour (and even in a single reader when confronted with different texts), a considerable amount of variance in our data may simply be accounted for by by-subject (or by-token) random effects.