SlowGuess/NAACL_2021_OCR huggingface dataset:
Performance of Pretrained Models To test the language models' abilities on predicting human reading behavior only from pretraining on textual input, we take the provided model checkpoints and use them to predict the eye tracking features without any fine-tuning. The detailed results are presented in Appendix D.1. The achieved accuracy aggregated over all eye tracking features lies between  $75 - 78\%$  for English. For Dutch, the models achieve
<sup>6</sup>Plots for the remaining datasets are in Appendix D.2
Figure 3: Data ablation study for Dutch and English. The results are aggregated over all eye tracking features. In addition to the mean across five runs, the shaded areas represent the standard deviation. The dashed line is the result of the pretrained BERT-MULTI model without fine-tuning. Results are aggregated over all eye tracking features.
Figure 4: Results of individual eye tracking features for all models on the Dundee and RSC corpora.
Figure 5: Improvement of prediction accuracy for the XLM-100 model relative to the mean baseline for each eye tracking feature.
84% accuracy but for Russian merely 65%. Across the same languages the results between the different language models are only minimal. However, on the individual eye tracking features, the pretrained models do not achieve any improvements over the mean baseline (see Appendix D.1).
7 Data Sensitivity
For the main experiment, we always tested the models on held-out data from the same dataset. In this
section, we examine the influence of dataset properties (text domain and language) on the prediction accuracy. In a second step, we analyze the influence of more universal input characteristics (word length, text readability).
7.1 Cross-Domain Evaluation
Figure 6 shows the results when evaluating the eye tracking predictions on out-of-domain text for the English datasets. For instance, we fine-tune the model on the newspaper articles of the Dundee corpus and test on the literary novel of the GECO corpus. We can see that the overall prediction accuracy across all eye tracking features is constantly above.  $90\%$  in all combinations. This shows that our eye tracking prediction model is able to generalize across domains. We find that the cross-domain capabilities of BERT are slightly better than for XLM. BERT-EN performs best in the cross-domain evaluation, possibly because its training data is more domain-general since it includes text from Wikipedia and books.
7.2 Cross-Language Evaluation
Figure 7 shows the results for cross-language evaluation to probe the language transfer capabilities of the multilingual models. We test models finetuned on language A on the test set of language B. It can be seen that BERT-MULTI generalizes better across languages than the XLM models. This might be due to the fact that the multilingual BERT model is trained on one large vocabulary of many languages but the XLM models are trained with a cross-lingual objective and language information. Hence, during fine-tuning on eye tracking
Figure 6: Cross-domain evaluation on pretrained English models. The results are expressed as the difference in the prediction error compared to the in-domain prediction. A smaller error (i.e., a color more similar to the color of the diagonal) represents better domain adaptation.
Figure 7: Cross-language evaluation on multilingual models across English, Dutch, German and Russian data. The results are expressed as the difference in the prediction error compared to the prediction on the same language. A smaller error (i.e., a color more similar to the color of the diagonal) represents better language transfer.
data from one language the XLM models lose some of their cross-lingual abilities. Our results are in line with Pires et al. (2019) and Karthikeyan et al. (2020), who showed that BERT learns multilingual representations in more than just a shared vocabulary space but also across scripts. When fine-tuning BERT-MULTI on English or Dutch data and testing on Russian, we see surprisingly high accuracy across scripts, even outperforming the in-language results. The XLM models, however, show the expected behavior where transferring within the same script (Dutch, English, German) works much better than transferring between the Latin and Cyrillic script (Russian).
7.3 Input Characteristics
Gaze patterns are strongly correlated with word length. Figure 8 shows that the models accurately learn to predict higher fixation proportions for longer words. We observe that the predictions of
the XLM-100 model follow the trend in the original data most accurately. Similar patterns emerge for the other languages (see Appendix D.3). Notably, the pretrained models before fine-tuning do not reflect the word length effect.
On the sentence level, we hypothesize that eye tracking features are easier to predict for sentences with a higher readability. Figure 9 shows the accuracy for predicting the number of fixations (NFIX) in a sentence relative to the Flesch reading ease score. Interestingly, the pretrained models without fine-tuning conform to the expected behavior and show a consistent increase in accuracy for sentences with a higher reading ease score. After fine-tuning on eye tracking data, this behavior is not as visible anymore since the language models achieve constantly high accuracy independent of the readability of the sentences.
These results might be explained by the nature of the Flesch readability score, which is based only
Figure 8: Prediction accuracy of FPROP with respect to word length. The gray dashed line is the result of the pretrained BERT-MULTI model without fine-tuning.
on the structural complexity of the text (see Appendix B for a description of the Flesch Reading Ease score). Our results indicate that language models trained purely on textual input are more calibrated towards such structural characteristics, i.e., the number of syllables in a word and the number of words in a sentences. Hence, the Flesch reading ease score might not be a good approximation for text readability. In future work, comparing eye movement patterns and text difficulty should rely on readability measures that take into account lexical, semantic, syntactic, and discourse features. This might reveal deviating patterns between pretrained and fine-tuned models.
Our analyses indicate that the models learn to take properties of the input into account when predicting eye tracking patterns. These processing strategies are similar to those observed in humans. Nevertheless, the connection between readability and relative importance in text needs to be analysed in more detail to establish how well these properties are learned by the language models.
While the superior performance of pretrained transformer language models has been established, we have yet to understand to which extent these models are comparable to human language processing behavior. We take a step in this direction by finetuning language models on eye tracking data to predict human reading behavior.
We find that both monolingual and multilingual models achieve surprisingly high accuracy in predicting a range of eye tracking features across four languages. Compared to the XLM models, BERT-
Figure 9: Prediction accuracy for NFIX relative to the Flesch reading ease score of the sentence. A higher Flesch score indicates that a sentence is easier to read. The dashed lines show the results of the pretrained language models without fine-tuning on eye tracking data.
MULTI is more robust in its ability to generalize across languages, without being explicitly trained for it. In contrast, the XLM models perform better when fine-tuned on less eye tracking data. Generally, fixation duration features are predicted more accurately than fixation proportion, possibly because the latter show higher variance across subjects. We observe that the models learn to reflect characteristics of human reading such as the word length effect and higher accuracy in more easily readable sentences.
The ability of transformer models to achieve such high results in modelling reading behavior indicates that we can learn more about the commonalities between language models and human sentence processing. By predicting behavioral metrics such as eye tracking features we can investigate the cognitive plausibility within these models to adjust or intensify the human inductive biases.
Lena Jäger was partially funded by the German Federal Ministry of Education and Research under grant 01|S20043.
Samira Abnar, Lisa Beinborn, Rochelle Choenni, and Willem Zuidema. 2019. Blackbox meets blackbox: Representational similarity & stability analysis of neural language models and brains. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 191-203.
Phillip M Alday. 2019. M/EEG analysis of naturalistic stories: A review from speech to language processing. Language, Cognition and Neuroscience, 34(4):457-473.
Toni Amstad. 1978. Wie verstandlich sind unsere Zeitungen? Unpublished doctoral dissertation, University of Zürich, Switzerland.
Xuejun Bai, Guoli Yan, Simon P Liversedge, Chuanli Zang, and Keith Rayner. 2008. Reading spaced and unspaced Chinese text: Evidence from eye movements. Journal of Experimental Psychology: Human Perception and Performance, 34(5):1277.
Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. 2018. Sequence classification with human attention. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 302-312.
Maria Barrett, Joachim Bingel, Frank Keller, and Anders Søgaard. 2016. Weakly supervised part-of-speech tagging using eye-tracking data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, volume 2, pages 579-584.
Jasmijn Bastings and Katja Filippova. 2020. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In Proceedings of the Third Blackbox NLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 149-155.
Jose Canete, Gabriel Chaperon, Rodrigo Fuentes, and Jorge Pérez. 2020. Spanish pre-trained BERT model and evaluation data. PML4DC at ICLR.
Branden Chan, Timo Möller, Malte Pietsch, and Tanay Soni. 2019. German BERT.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451.
Uschi Cop, Nicolas Dirix, Denis Drieghe, and Wouter Duyck. 2017. Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading. Behavior Research Methods, 49(2):602-615.
Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. 2020. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3243-3255.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
WH Douma. 1960. De Leesbaarheid Van Landbouwbladen. Een Onderzoek Maar en Een Toepassing Van Leesbaarheidsformules.
Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48.
Rudolph Flesch. 1948. A new readability yardstick. Journal of applied psychology, 32(3):221.
Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pages 80-89. IEEE.
Anna Gladkova and Aleksandr Drozd. 2016. Intrinsic evaluations of word embeddings: What can we do better? In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 36-42.
Ana Valeria Gonzalez-Garduno and Anders Søgaard. 2017. Using gaze to predict text readability. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 438-443.
Michael Hahn and Frank Keller. 2016. Modeling human reading with neural attention. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
John Hale. 2017. Models of human sentence comprehension in computational psycholinguistics. In Oxford Research Encyclopedia of Linguistics.
Yiding Hao, Simon Mendelsohn, Rachel Sterneck, Randi Martinez, and Robert Frank. 2020. Probabilistic predictions of people perusing: Evaluating metrics of language model performance for psycholinguistic modeling. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 75-86.
Robert Hawkins, Takateru Yamakoshi, Thomas L Griffiths, and Adele Goldberg. 2020. Investigating representations of verb bias in neural language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4653-4663.
Nora Hollenstein, Maria Barrett, Marius Troendle, Francesco Bigiolli, Nicolas Langer, and Ce Zhang. 2019. Advancing NLP with cognitive language processing signals. arXiv preprint arXiv:1904.02682.
Nora Hollenstein, Jonathan Rotsztejn, Marius Troendle, Andreas Pedroni, Ce Zhang, and Nicolas Langer. 2018. ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading. Scientific Data.
Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer. 2020. ZuCo 2.0: A dataset of physiological recordings during natural reading and annotation. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 138-146.
Nora Hollenstein and Ce Zhang. 2019. Entity recognition at first sight: Improving NER with eye movement information. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
Albrecht Werner Inhoff and Keith Rayner. 1986. Parafoveal word processing during eye fixations in reading: Effects of word frequency. Perception & psychophysics, 40(6):431-439.
Lena Jäger, Thomas Kern, and Patrick Haller. 2021. Potsdam Textbook Corpus (PoTeC): Eye tracking data from experts and non-experts reading scientific texts. available on OSF, DOI 10.17605/OSF.IO/DN5HP.
Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543-3556.
Yu-Cin Jian, Ming-Lei Chen, and Hwa-wei Ko. 2013. Context effects in processing of Chinese academic words: An eye-tracking investigation. Reading Research Quarterly, 48(4):403-413.
Kaliyaperumal Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: An empirical study. In International Conference on Learning Representations.
Frank Keller. 2010. Cognitively plausible models of human language processing. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics: Short Papers, pages 60-67.
Alan Kennedy, Robin Hill, and Joel Pynte. 2003. The Dundee corpus. In Proceedings of the 12th European Conference on Eye Movement.
Sigrid Klerke, Yoav Goldberg, and Anders Søgaard. 2016. Improving sentence compression by learning to predict gaze. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1528-1533.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057-7075.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. In Advances in Neural Information Processing Systems.
AK Laurinavichyute, Irina A Sekerina, SV Alexeeva, and KA Bagdasaryan. 2019. Russian Sentence Corpus: Benchmark measures of eye movements in reading in Cyrillic. Behavior research methods, 51(3):1161-1178.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020. XGLUE: A new benchmark dataset for cross-lingual pretraining, understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008-6018.
Simon P Liversedge, Denis Drieghe, Xin Li, Guoli Yan, Xuejun Bai, and Jukka Hyönenä. 2016. Universality in eye movements and reading: A trilingual investigation. Cognition, 147:1-20.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations.
Jonathan Malmaud, Roger Levy, and Yevgeni Berzak. 2020. Bridging information-seeking human gaze and machine reading comprehension. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 142-152.
Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences.
Franz Matthies and Anders Søgaard. 2013. With blinkers on: Robust prediction of eye movements across readers. Proceedings of the 2013 Conference on empirical methods in natural language processing (EMNLP), pages 803-807.
Danny Merkx and Stefan L Frank. 2020. Comparing transformers and RNNs on predicting human sentence processing data. arXiv preprint arXiv:2005.09471.
James Michaelov and Benjamin Bergen. 2020. How well does surprisal explain N400 amplitude under different experimental conditions? In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 652-663.
Robin K Morris. 1994. Lexical and message-level sentence context effects on fixation times in reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20(1):92.
I Oborneva. 2006. Automatic assessment of the complexity of educational texts on the basis of statistical parameters.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001.
Keith Rayner. 1977. Visual attention in reading: Eye movements reflect cognitive processes. Memory & Cognition, 5(4):443-448.
Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. *Psychological bulletin*, 124(3):372.
Erik D Reichle, Alexander Pollatsek, Donald L Fisher, and Keith Rayner. 1998. Toward a model of eye movement control in reading. *Psychological review*, 105(1):125.
Martin Schrimpf, Idan A Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy G Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2020. Artificial neural networks accurately predict language processing in the brain. *BioRxiv*.
Ekta Sood, Simon Tannert, Philipp Mueller, and Andreas Bulling. 2020. Improving natural language processing tasks with human gaze-guided neural attention. Advances in Neural Information Processing Systems, 33.
Ece Takmaz, Sandro Pezzelle, Lisa Beinborn, and Raquel Fernandez. 2020. Generating image descriptions via sequential cross-modal alignment guided by human gaze. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4664-4677.
Mariya Toneva and Leila Wehbe. 2019. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). In Advances in Neural Information Processing Systems, pages 14928-14938.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampa Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arXiv preprint arXiv:1912.07076.
Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A Dutch BERT Model. arXiv preprint arXiv:1912.09582.
Ivan Vulic, Edoardo Maria Ponti, Ira Leviant, Olga Majewska, Matt Malone, Roi Reichart, Simon Baker, Ulla Petti, Kelly Wing, Eden Bar, et al. 2020. Multi-SimLex: A large-scale evaluation of multilingual and cross-lingual lexical semantic similarity. Computational Linguistics, pages 1-73.
Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. 2014. Aligning context-based statistical models of language with brain activity during reading. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 233-243.
Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. arXiv preprint arXiv:2006.01912.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Kuratov Yu and M Arkhipov. 2019. Adaptation of deep bidirectional multilingual transformers for Russian language. Computational Linguistics and Intellectual Technologies, (18):333-339.
A Eye Tracking Data
Table 6 presents information about the range of the eye tracking features.
Figure 10 shows the word length effect found in eye tracking data recorded during reading. i.e., the fact that longer words are more likely to be fixated. This effect is observable across all languages.
Figure 10: Word length effect on all datasets in all four languages.
Figure 11 shows the mean fixation duration (MFD) for adjectives, nouns, verbs, and adverbs for all six datasets. We use spacy $^7$  to perform part-of-speech tagging for our analyses. For Russian we load an externally trained model $^8$ , for Dutch, English and German we use the provided pretrained models.
Figure 12 shows an additional analysis where we explore which parts-of-speech can be predicted more accurately by the language models.
B Readability Scores
We use the Flesch Reading Easy score (Flesch, 1948) to define the readability of the English text in the eye tracking corpora. This score indicates how difficult a text passage is to understand. Since this score relies on language-specific weighting factors, we apply the Flesch Douma adaptation for Dutch (Douma, 1960), the adaptation by Amstad (1978) for German, and the adaptation by Oborneva (2006) for Russian.
C.1 Tokenization
When using BERT or XLM for token classification or regression, a pressing implementation issue is
represented by the subword tokenizers employed by the models. This tokenizer, in fact, handles unknown tokens by recursively splitting every word until all subtokens belong to its vocabulary. For example, the name of the Greek mythological hero "Philammon" is tokenized into the three subtokens "[phil', "#am', "#mon'']. In this case, our models for token regression would produce an eight-dimensional output for all three subtokens, and we had the choice as to what to do in order to compute the loss, having only one target for the full word "Philammon". We chose to compute the loss only with respect to the first subtoken.
C.2 Training Setup
As described in the main paper, all experiments are run over 5 random seeds, which are  $\{12,79,237,549,886\}$ .