SlowGuess/NAACL_2021_OCR huggingface dataset:
Nora Hollenstein $^{1}$ , Federico Pirovano $^{1}$ , Ce Zhang $^{1}$ , Lena Jäger $^{2,3}$ , Lisa Beinborn $^{4}$
2 University of Zurich
<sup>3</sup> University of Potsdam
4 Vrije Universiteit Amsterdam
{noraho,fpirovan,ce.zhang}@inf.ethz.ch,
jaeger@cl.uzh.ch, l.beinborn@vu.nl
We analyze if large language models are able to predict patterns of human reading behavior. We compare the performance of language-specific and multilingual pretrained transformer models to predict reading time measures reflecting natural human sentence processing on Dutch, English, German, and Russian texts. This results in accurate models of human reading behavior, which indicates that transformer models implicitly encode relative importance in language in a way that is comparable to human processing mechanisms. We find that BERT and XLM models successfully predict a range of eye tracking features. In a series of experiments, we analyze the cross-domain and cross-language abilities of these models and show how they reflect human sentence processing.
When processing language, humans selectively attend longer to the most relevant elements of a sentence (Rayner, 1998). This ability to seamlessly evaluate relative importance is a key factor in human language understanding. It remains an open question how relative importance is encoded in computational language models. Recent analyses conclude that the cognitively motivated "attention" mechanism in neural models is not a good indicator for relative importance (Jain and Wallace, 2019). Alternative methods based on salience (Bastings and Filippova, 2020), vector normalization (Kobayashi et al., 2020), or subset erasure (De Cao et al., 2020) are being developed to increase the post-hoc interpretability of model predictions but the cognitive plausibility of the underlying representations remains unclear.
In human language processing, phenomena of relative importance can be approximated indirectly by tracking eye movements and measuring fixation
Figure 1: From the fixation times in milliseconds of a single subject in the ZuCo 1.0 dataset, the feature vector described in Section 3.2 for the wors "Mary" would be [2, 233, 233, 431, 215.5, 1, 1, 1].
duration (Rayner, 1977). It has been shown that fixation duration and relative importance of text segments are strongly correlated in natural reading, so that direct links can be established on the token level (Malmaud et al., 2020). In the example in Figure 1, the newly introduced entity Mary French is fixated twice and for a longer duration because it is relatively more important for the reader than the entity Laurence, which had been introduced in the previous sentence. Being able to reliably predict eye movement patterns from the language input would bring us one step closer to understand the cognitive plausibility of these models.
Contextualized neural language models are less interpretable than conceptually motivated psycholinguistic models but they achieve high performance in many language understanding tasks and can be fitted successfully to cognitive features such as self-paced reading times and N400 strength (Merkx and Frank, 2020). Moreover, approaches to directly predict cognitive signals (e.g., brain activity) indicate that neural representations implicitly encode similar information as humans (Wehbe et al., 2014; Abnar et al., 2019; Sood et al., 2020; Schrimpf et al., 2020). However, it has not been analyzed to which extent transformer language models are able to directly predict human behavioral metrics such as gaze patterns.
The performance of computational models can
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 106-123 June 6-11, 2021. ©2021 Association for Computational Linguistics
be improved even further if their inductive bias is adjusted using human cognitive signals such as eye tracking, fMRI, or EEG data (Hollenstein et al., 2019; Toneva and Wehbe, 2019; Takmaz et al., 2020). While psycholinguistic work mainly focuses on very specific phenomena of human language processing that are typically tested in experimental settings with constructed stimuli (Hale, 2017), we focus on directly generating token-level predictions from natural reading.
We fine-tune transformer models on human eye movement data and analyze their ability to predict human reading behavior focusing on a range of reading features, datasets, and languages. We compare the performance of monolingual and multilingual transformer models. Multilingual models represent multiple languages in a joint space and aim at a more universal language understanding. As eye tracking patterns are consistent across languages for certain phenomena, we hypothesize that multilingual models might provide cognitively more plausible representations and outperform language-specific models in predicting reading measures. We test this hypothesis on 6 datasets of 4 Indo-European languages, namely English, German, Dutch and Russian.
We find that pretrained transformer models are surprisingly accurate at predicting reading time measures in four Indo-European languages. Multilingual models show an advantage over language-specific models, especially when fine-tuned on smaller amounts of data. Compared to previous psycholinguistic reading models, the accuracy achieved by the transformer models is remarkable. Our results indicate that transformer models implicitly encode relative importance in language in a way that is comparable to human processing mechanisms. As a consequence, it should be possible to adjust the inductive bias of neural models towards more cognitively plausible outputs without having to resort to large-scale cognitive datasets.
Using eye movement data to modify the inductive bias of language processing models has resulted in improvements for several NLP tasks (e.g., Barrett et al. 2016; Hollenstein and Zhang 2019). It has also been used as a supervisory signal in multi-task learning scenarios (Klerke et al., 2016; Gonzalez-
Garduno and Søgaard, 2017) and as a method to fine-tune the attention mechanism (Barrett et al., 2018). We use eye tracking data to evaluate how well transformer language models predict human sentence processing. Therefore, in this section, we discuss previous work on probing transformers models as well as on modelling human sentence processing.
2.1 Probing Transformer Language Models
Contextualized neural language models have become increasingly popular, but our understanding of these black box algorithms is still rather limited (Gilpin et al., 2018). Current intrinsic evaluation methods do not capture the cognitive plausibility of language models (Manning et al., 2020; Gladkova and Drozd, 2016). In previous work of interpreting and probing language models, human behavioral data as well as neuroimaging recordings have been leveraged to understand the inner workings of the neural models. For instance, Ettinger (2020) explores the linguistic capacities of BERT with a set of psycholinguistic diagnostics. Toneva and Wehbe (2019) propose an interpretation approach by learning alignments between the models and brain activity recordings (MEG and fMRI). Hao et al. (2020) propose to evaluate language model quality based on the degree to which they exhibit human-like behavior such as predictability measures collected from human subjects. However, their metric does not reveal any details about the commonalities between the model and human sentence processing.
The benefits of multilingual models are controversial. Transformer models trained exclusively on a specific language often outperform multilingual models trained on various languages simultaneously, even after fine-tuning. This curse of multilinguality (Conneau et al., 2020; Vulić et al., 2020) has been shown for Spanish (Canete et al., 2020), Finnish (Virtanen et al., 2019) and Dutch (Vries et al., 2019). In this paper we investigate whether a similar effect can be observed when leveraging these models to predict human behavioral measures, or whether in that case the multilingual models provide more plausible representations of human reading due to the common eye tracking effects across languages.
2.2 Modelling Human Sentence Processing
Previous work of neural modelling of human sentence processing has focused on recurrent neural networks, since their architecture and learn
$^{1}$ Code available on GitHub: https://github.com/DS3Lab/multilingual-gaze
Language	Corpus	Subjs.	Sents.	Sent. length	Tokens	Types	Word length	Flesch
English	Dundee	10	2,379	21.7 (1–87)	51,497	9,488	4.9 (1–20)	53.3
GECO	14	5,373	10.5 (1–69)	56,410	5,916	4.6 (1–33)	77.4
ZuCo	30	1,053	19.5 (1–68)	20,545	5,560	5.0 (1–29)	50.6
Dutch	GECO	19	5,190	11.64 (1–60)	59,716	5,575	4.5 (1–22)	57.5
German	PoTeC	30	97	19.5 (5–51)	1,895	847	6.5 (2–33)	36.4
Russian	RSC	103	144	9.4 (5–13)	1,357	993	5.7 (1–18)	64.7
Table 1: Descriptive statistics of all eye tracking datasets. Sentence length and word length are expressed as the mean with the min-max range in parentheses. The last column shows the Flesch Reading Ease score (Flesch, 1948) which ranges from 0 to 100 (higher score indicates easier to read). Adaptations of the Flesch score were used for Dutch (nl), German (de) and Russian (ru) (see Appendix B).
ing mechanism appears to be cognitively plausible (Keller, 2010; Michaelov and Bergen, 2020). However, recent work suggests that transformers perform better at modelling certain aspects of the human language understanding process (Hawkins et al., 2020). While Merkx and Frank (2020) and Wilcox et al. (2020) show that the psychometric predictive power of transformers outperforms RNNs on eye tracking, self-paced reading times and N400 strength, they do not directly predict cognitive features. Schrimpf et al. (2020) show that contextualized monolingual English models accurately predict language processing in the brain.
Context effects are known to influence fixations times during reading (Morris, 1994). The notion of using contextual information to process language during reading has been well-established in psycholinguistics (e.g., Inhoff and Rayner 1986 and Jian et al. 2013). However, to the best of our knowledge, we are the first to study to which extent the representations learned by transformer language models entail these human reading patterns.
Compared to neural models of human sentence processing, we predict not only individual metrics but a range of eye tracking features covering the full reading process from early lexical access to late syntactic processing. By contrast, most models of reading focus on predicting skipping probability (Reichle et al., 1998; Matthies and Søgaard, 2013; Hahn and Keller, 2016). Sood et al. (2020) propose a text saliency model which predicts fixation durations that are then used to compute the attention scores in a transformer network.
We predict eye tracking data only from naturalistic reading studies in which the participants read full
sentences or longer spans of naturally occurring text in their own speed. The data from these studies exhibit higher ecological validity than studies which rely on artificially constructed sentences and paced presentation (Alday, 2019).
3.1 Corpora
To conduct a cross-lingual comparison, we use eye tracking data collected from native speakers of four languages (see Table 1 for details).
English The largest number of eye tracking data sources are available for English. We use eye tracking features from three English corpora: (1) The Dundee corpus (Kennedy et al., 2003) contains 20 newspaper articles from *The Independent*, which were presented to English native readers on a screen five lines at a time. (2) The GECO corpus (Cop et al., 2017) contains eye tracking data from English monolinguals reading the entire novel *The Mysterious Affair* at *Styles* by Agatha Christie. The text was presented on the screen in paragraphs. (3) The ZuCo corpus (Hollenstein et al., 2018, 2020) includes eye tracking data of full sentences from movie reviews and Wikipedia articles.
Dutch The GECO corpus (Cop et al., 2017) additionally contains eye tracking data from Dutch readers, which were presented with the same novel in their native language.
German The Potsdam Textbook Corpus (PoTeC, Jäger et al. 2021) contains 12 short passages of 158 words on average from college-level biology and physics textbooks, which are read by expert and laymen German native speakers. The full passages were presented on multiple lines on the screen.
Note that the exact numbers might differ slightly from the original publications due to different preprocessing methods.
3We use Tasks 1 and 2 from ZuCo 1.0 and Task 1 from ZuCo 2.0.
Short Name	Language	Model Checkpoint	Reference
BERT-NL	Dutch	WIETSEDV/BERT-BASE-DUTCH-CASED	Vries et al. (2019)
BERT-EN	English	BERT-BASE-UNCASED	Wolf et al. (2019)
BERT-DE	German	BERT-BASE-German-CASED	Chan et al. (2019)
BERT-RU	Russian	DEEPPAVLOV/RUBERT-BASE-CASED	Yu and Arkhipov (2019)
BERT-MULTI	104 languages	BERT-BASE-MULTILINGUAL-CASED	Wolf et al. (2019)
XLM-EN	English	XLM-MLM-EN-2048	Lample and Conneau (2019)
XLM-ENDE	English + German	XLM-MLM-ENDE-1024	Lample and Conneau (2019)
XLM-17	17 languages	XLM-MLM-17-1280	Lample and Conneau (2019)
XLM-100	100 languages	XLM-MLM-100-1280	Lample and Conneau (2019)
Table 2: Pretrained transformer language models analyzed in this work.
Russian The Russian Sentence Corpus (RSC, Laurinavichyute et al. 2019) contains 144 naturally occurring sentences extracted from the Russian National Corpus. $^{4}$  Full sentences were presented on the screen to monolingual Russian-speaking adults one at a time.
3.2 Eye Tracking Features
A fixation is defined as the period of time where the gaze of a reader is maintained on a single location. Fixations are mapped to words by delimiting the boundaries around the region on the screen belonging to each word  $w$ . A word can be fixated more than once. For each token  $w$  in the input text, we predict the following eight eye tracking features that encode the full reading process from early lexical access up to subsequent syntactic integration.
Word-level characteristics We extract basic features that encode word-level characteristics: (1) number of fixations (NFIX), the number of times a subject fixates  $w$ , averaged over all subjects; (2) mean fixation duration (MFD), the average fixation duration of all fixations made on  $w$ , averaged over all subjects; (3) fixation proportion (FPROP), the number of subjects that fixated  $w$ , divided by the total number of subjects.
Early processing We also include features to capture the early lexical and syntactic processing, based on the first time a word is fixated: (4) first fixation duration (FFD), the duration, in milliseconds, of the first fixation on  $w$ , averaged over all subjects; (5) first pass duration (FPD), the sum of all fixations on  $w$  from the first time a subject fixes  $w$  to the first time the subject fixes another token, averaged over all subjects.
Late processing Finally, we also use measures reflecting the late syntactic processing and general
disambiguation, based on words which were fixated more than once: (6) total reading time (TRT), the sum of the duration of all fixations made on  $w$ , averaged over all subjects; (7) number of re-fixations (NREFIX), the number of times  $w$  is fixated after the first fixation, i.e., the maximum between 0 and the NFIX-1, averaged over all subjects; (8) re-read proportion (REPROP), the number of subjects that fixated  $w$  more than once, divided by the total number of subjects.
The values of these eye tracking features vary over different ranges (see Appendix A). FFD, for example, is measured in milliseconds, and average values are around  $200\mathrm{ms}$ , whereas REPROP is a proportional measure, and therefore assumes floating-point values between 0 and 1. We standardize all eye tracking features independently (range: 0-100), so that the loss can be calculated uniformly over all feature dimensions.
Eye movements depend on the stimulus and are therefore language-specific but there exist universal tendencies which remain stable across languages (Liversedge et al., 2016). For example, the average fixation duration in reading ranges from 220 to  $250\mathrm{ms}$  independent of the language. Furthermore, word characteristics such as word length, frequency and predictability affect fixation duration similarly across languages but the effect size depends on the language and the script (Laurinavichyute et al., 2019; Bai et al., 2008). The word length effect, i.e., the fact that longer words are more likely to be fixated, can be observed across all four languages included in this work (see Appendix A).
4 Language Models
We compare the ability to predict eye tracking features in two models: BERT and XLM. Both models are trained on the transformer architecture (Vaswani et al., 2017) and yield state-of-the
<sup>4</sup>https://ruscorpora.ru
Figure 2: True and predicted feature values for two example sentences. On the left the fixation proportion (FPROP) values for an English sentence from the ZuCo dataset, and on the right the number of fixations (NFIX) values for a Russian sentence from the RSC dataset.
art results for a wide range of NLP tasks (Liang et al., 2020). The multilingual BERT model simply concatenates the Wikipedia input from 104 languages and is optimized by performing masked token and next sentence prediction as in the monolingual model (Devlin et al., 2019) without any cross-lingual constraints. In contrast, XLM adds a translation language modeling objective, by explicitly using parallel sentences in multiple languages as input to facilitate cross-lingual transfer (Lample and Conneau, 2019). Both BERT and XLM use subword tokenization methods to build shared vocabulary spaces across languages.
We use the pretrained checkpoints from the HuggingFace repository for monolingual and multilingual models (details in Table 2).
We fine-tune the models described above on the features extracted from the eye tracking datasets. The eye tracking prediction uses a model for token regression, i.e., the pretrained language models with a linear dense layer on top of it. The final dense layer is the same for all tokens, and performs a projection from the dimension of the hidden size of the model (e.g., 768 for BERT-EN or 1,280 for XLM-100) to the dimension of the eye tracking feature space (8, in our case). The model is trained for the regression task using the mean squared error (MSE) loss.
Training Details We split the data into  $90\%$  training data,  $5\%$  validation and  $5\%$  test data. We initially tuned the hyper-parameters manually and set the following values for all models: We use an AdamW optimizer (Loshchilov and Hutter, 2018) with a learning rate of 0.00005 and a weight decay of 0.01. The batch size varies depending on the
model dimensions (see Appendix C.2). We employ a linear learning rate decay schedule over the total number of training steps. We clip all gradients exceeding the maximal value of 1. We train the models for 100 epochs, with early stopping after 7 epochs without an improvement on the validation accuracy.
Evaluation Procedure As the features have been standardized to the range 0-100, the mean absolute error (MAE) can be interpreted as a percentage error. For readability, we report the prediction accuracy as 100-MAE in all experiments. The results are averaged over batches and over 5 runs with varying random seeds. For a single batch of sentences, the overall MAE is calculated by concatenating the words in each sentence and the feature dimensions for each word, and padding to the maximum sentence length. The per-feature MAE is calculated by concatenating the words in each sentence. For example, for a batch of  $B$  sentences, each composed of  $L$  words, and  $G$  eye tracking features per word, the overall MAE is calculated over a vector of  $B^{*}L^{*}G$  dimensions. In contrast, the MAE for each individual feature is calculated over a vector of  $B^{*}L$  dimensions.
Tables 3 and 4 show that all models predict the eye tracking features with more than  $90\%$  accuracy for English and Dutch. For English, the BERT models yield high performance on all three datasets with standard deviations below 0.15. The results for the XLM models are slightly better on average but exhibit much higher standard deviations. Similar to the results presented by Lample and Conneau (2019), we find that more training data from multiple languages improves prediction performance. For instance, the XLM-100 model achieves higher accuracy than the XLM-17 model in all cases. For
$^{5}$ https://huggingface.co/transformers/pretrained_models.html
Model	Dundee (en)	GECO (en)	ZuCo (en)	ALL (en)
BERT-EN	92.63 (0.05)	93.68 (0.14)	93.42 (0.02)	93.71 (0.06)
BERT-MULTI	92.73 (0.06)	93.73 (0.12)	93.74 (0.05)	93.74 (0.07)
XLM-EN	90.41 (2.16)	91.15 (1.42)	92.03 (2.11)	90.88 (1.50)
XLM-ENDE	92.79 (0.15)	93.89 (0.12)	93.76 (0.15)	93.96 (0.08)
XLM-17	92.11 (1.68)	91.79 (1.75)	92.05 (2.25)	93.80 (0.38)
XLM-100	92.99 (0.05)	93.04 (1.40)	93.97 (0.09)	93.96 (0.06)
Table 3: Prediction accuracy over all eye tracking features for the English corpora, including the concatenated dataset. Standard deviation is reported in parentheses.
Model	GECO (nl)	PoTeC (de)	RSC (ru)	ALL-LANGS
BERT-NL	91.81 (0.23)	-	-	-
BERT-DE	-	78.38 (1.69)	-	-
BERT-RU	-	-	78.73 (1.38)	-
BERT-MULTI	91.90 (0.16)	76.86 (2.42)	76.54 (3.59)	94.72 (0.07)
XLM-ENDE	-	80.94 (0.88)	-	-
XLM-17	91.04 (0.70)	86.26 (1.31)	90.96 (3.96)	94.46 (0.83)
XLM-100	92.31 (0.22)	86.57 (0.54)	94.70 (0.60)	94.94 (0.11)
Table 4: Prediction accuracy over all eye tracking features for the Dutch, German and Russian corpora, and for all four languages combined in a single dataset. Standard deviation is reported in parentheses.
the smaller non-English datasets, PoTeC (de) and RSC (ru), the multilingual XLM models clearly outperform the monolingual models. For the English datasets, the differences are minor.
Size Effects More training data results in higher prediction accuracy even when the eye tracking data comes from various languages and was recorded in different reading studies by different devices (ALL-LANGS, fine-tuning on the data of all four languages together). However, merely adding more data from the same language (ALL (en), fine-tuning on the English data from Dundee, GECO and ZuCo together) does not result in higher performance.
To analyze this further, we perform an ablation study on varying amounts of training data. The results are shown in Figure 3 for Dutch and English. The performance of the XLM models remains stable even with a very small percentage of eye tracking data. The performance of the BERT models, however, drops drastically when fine-tuning on less than  $20\%$  of the data. Similar to Merkx and Frank (2020) and Hao et al. (2020) we find that the model architecture, along with the composition and size of the training corpus have a significant impact on the psycholinguistic modeling performance.
Eye Tracking Features The accuracy results are averaged over all eye tracking features. For a better understanding of the prediction output, we plot the true and the predicted values of two selected fea
tures (FPROP and NFIX) for two example sentence in Figure 2. In both examples, the model predictions strongly correlate with the true values. The difference to the mean baseline is more pronounced for the FIXPROPfeature.
Figure 4 presents the quantitative differences across models in predicting the individual eye tracking features. Across all datasets, first pass duration (FPD) and number of re-fixations (NREFIX) are the most accurately predicted features. Proportions (FPROP and REPROP) are harder to predict because these features are even more dependent on subject-specific characteristics. Nevertheless, when comparing the prediction accuracy of each eye tracking feature to a baseline which always predicts the mean values, the predicted features FPROP and REPROP achieve the largest improvements relative to the mean baseline. See Figure 5 for a comparison between all features for the best performing model XLM-100 on all six datasets.