Venue: NAACL 2021

# Multilingual Language Models Predict Human Reading Behavior

Nora Hollenstein1, Federico Pirovano1, Ce Zhang1, Lena Jager2,3, Lisa Beinborn4

1 ETH Zurich

2 University of Zurich

3 University of Potsdam 4 Vrije Universiteit Amsterdam

{noraho,fpirovan,ce.zhang}@inf.ethz.ch, jaeger@cl.uzh.ch,l.beinborn@vu.nl

###### Abstract

We analyze if large language models are able to predict patterns of human reading behavior. We compare the performance of language-specific and multilingual pretrained transformer models to predict reading time measures reflecting natural human sentence processing on Dutch, English, German, and Russian texts. This results in accurate models of human reading behavior, which indicates that transformer models implicitly encode relative importance in language in a way that is comparable to human processing mechanisms. We find that BERT and XLM models successfully predict a range of eye tracking features. In a series of experiments, we analyze the cross-domain and cross-language abilities of these models and show how they reflect human sentence processing.

## 1 Introduction

When processing language, humans selectively attend longer to the most relevant elements of a sentence Rayner (1998). This ability to seamlessly evaluate relative importance is a key factor in human language understanding. It remains an open question how relative importance is encoded in computational language models. Recent analyses conclude that the cognitively motivated attention mechanism in neural models is not a good indicator for relative importance Jain and Wallace (2019). Alternative methods based on salience Bastings and Filippova (2020), vector normalization Kobayashi et al. (2020), or subset erasure De Cao et al. (2020) are being developed to increase the post-hoc interpretability of model predictions but the cognitive plausibility of the underlying representations remains unclear.

In human language processing, phenomena of relative importance can be approximated indirectly by tracking eye movements and measuring fixation duration Rayner (1977). It has been shown that fixation duration and relative importance of text segments are strongly correlated in natural reading, so that direct links can be established on the token level Malmaud et al. (2020). In the example in Figure 1, the newly introduced entity _Mary French_ is fixated twice and for a longer duration because it is relatively more important for the reader than the entity _Laurence_, which had been introduced in the previous sentence. Being able to reliably predict eye movement patterns from the language input would bring us one step closer to understand the cognitive plausibility of these models.

Contextualized neural language models are less interpretable than conceptually motivated psycholinguistic models but they achieve high performance in many language understanding tasks and can be fitted successfully to cognitive features such as self-paced reading times and N400 strength Merkx and Frank (2020). Moreover, approaches to directly predict cognitive signals (e.g., brain activity) indicate that neural representations implicitly encode similar information as humans Webhe et al. (2014); Abnar et al. (2019); Sood et al. (2020); Schrimpf et al. (2020). However, it has not been analyzed to which extent transformer language models are able to directly predict human behavioral metrics such as gaze patterns.

The performance of computational models can

Figure 1: From the fixation times in milliseconds of a single subject in the ZuCo 1.0 dataset, the feature vector described in Section 3.2 for the wors “Mary” would be \([2,233,233,431,215.5,1,1,1]\).



be improved even further if their inductive bias is adjusted using human cognitive signals such as eye tracking, fMRI, or EEG data (Hollenstein et al., 2019; Toneva and Wehbe, 2019; Takmaz et al., 2020). While psycholinguistic work mainly focuses on very specific phenomena of human language processing that are typically tested in experimental settings with constructed stimuli Hale (2017), we focus on directly generating token-level predictions from natural reading.

We fine-tune transformer models on human eye movement data and analyze their ability to predict human reading behavior focusing on a range of reading features, datasets, and languages. We compare the performance of monolingual and multilingual transformer models. Multilingual models represent multiple languages in a joint space and aim at a more universal language understanding. As eye tracking patterns are consistent across languages for certain phenomena, we hypothesize that multilingual models might provide cognitively more plausible representations and outperform language-specific models in predicting reading measures. We test this hypothesis on 6 datasets of 4 Indo-European languages, namely English, German, Dutch and Russian.1

Footnote 1: Code available on GitHub: [https://github.com/DS3Lab/multilingual-gaze](https://github.com/DS3Lab/multilingual-gaze)

We find that pretrained transformer models are surprisingly accurate at predicting reading time measures in four Indo-European languages. Multilingual models show an advantage over language-specific models, especially when fine-tuned on smaller amounts of data. Compared to previous psycholinguistic reading models, the accuracy achieved by the transformer models is remarkable. Our results indicate that transformer models implicitly encode relative importance in language in a way that is comparable to human processing mechanisms. As a consequence, it should be possible to adjust the inductive bias of neural models towards more cognitively plausible outputs without having to resort to large-scale cognitive datasets.

## 2 Related Work

Using eye movement data to modify the inductive bias of language processing models has resulted in improvements for several NLP tasks (e.g., Barrett et al.2016; Hollenstein and Zhang 2019). It has also been used as a supervisory signal in multi-task learning scenarios (Klerke et al., 2016; Gonzalez-Garduno and Sogaard, 2017) and as a method to fine-tune the attention mechanism (Barrett et al., 2018). We use eye tracking data to evaluate how well transformer language models predict human sentence processing. Therefore, in this section, we discuss previous work on probing transformers models as well as on modelling human sentence processing.

### Probing Transformer Language Models

Contextualized neural language models have become increasingly popular, but our understanding of these black box algorithms is still rather limited (Gilpin et al., 2018). Current intrinsic evaluation methods do not capture the cognitive plausibility of language models (Manning et al., 2020; Gladkova and Drozd, 2016). In previous work of interpreting and probing language models, human behavioral data as well as neuroimaging recordings have been leveraged to understand the inner workings of the neural models. For instance, Ettinger (2020) explores the linguistic capacities of BERT with a set of psycholinguistic diagnostics. Toneva and Welhoe (2019) propose an interpretation approach by learning alignments between the models and brain activity recordings (MEG and fMRI). Hao et al. (2020) propose to evaluate language model quality based on the degree to which they exhibit humanlike behavior such as predictability measures collected from human subjects. However, their metric does not reveal any details about the commonalities between the model and human sentence processing.

The benefits of multilingual models are controversial. Transformer models trained exclusively on a specific language often outperform multilingual models trained on various languages simultaneously, even after fine-tuning. This _curse of multilinguality_(Conneau et al., 2020; Vulic et al., 2020) has been shown for Spanish (Canete et al., 2020), Finnish (Virtanen et al., 2019) and Dutch (Vries et al., 2019). In this paper we investigate whether a similar effect can be observed when leveraging these models to predict human behavioral measures, or whether in that case the multilingual models provide more plausible representations of human reading due to the common eye tracking effects across languages.

### Modelling Human Sentence Processing

Previous work of neural modelling of human sentence processing has focused on recurrent neural networks, since their architecture and learn 

ing mechanism appears to be cognitively plausible (Keller, 2010; Michaelov and Bergen, 2020). However, recent work suggests that transformers perform better at modelling certain aspects of the human language understanding process (Hawkins et al., 2020). While Merkx and Frank (2020) and Wilcox et al. (2020) show that the psychometric predictive power of transformers outperforms RNNs on eye tracking, self-paced reading times and N400 strength, they do not directly predict cognitive features. Schrimpf et al. (2020) show that contextualized monolingual English models accurately predict language processing in the brain.

Context effects are known to influence fixations times during reading (Morris, 1994). The notion of using contextual information to process language during reading has been well-established in psycholinguistics (e.g., Inhoff and Rayner 1986 and Jian et al. 2013). However, to the best of our knowledge, we are the first to study to which extent the representations learned by transformer language models entail these human reading patterns.

Compared to neural models of human sentence processing, we predict not only individual metrics but a range of eye tracking features covering the full reading process from early lexical access to late syntactic processing. By contrast, most models of reading focus on predicting skipping probability (Reichle et al., 1998; Matthies and Sogaard, 2013; Hahn and Keller, 2016). Sood et al. (2020) propose a text saliency model which predicts fixation durations that are then used to compute the attention scores in a transformer network.

## 3 Data

We predict eye tracking data only from naturalistic reading studies in which the participants read full sentences or longer spans of naturally occurring text in their own speed. The data from these studies exhibit higher ecological validity than studies which rely on artificially constructed sentences and paced presentation (Alday, 2019).

### Corpora

To conduct a cross-lingual comparison, we use eye tracking data collected from native speakers of four languages (see Table 1 for details).

EnglishThe largest number of eye tracking data sources are available for English. We use eye tracking features from three English corpora: (1) The Dundee corpus (Kennedy et al., 2003) contains 20 newspaper articles from _The Independent_, which were presented to English native readers on a screen five lines at a time. (2) The GECO corpus (Cop et al., 2017) contains eye tracking data from English monolinguals reading the entire novel _The Mysterious Affair at Styles_ by Agatha Christie. The text was presented on the screen in paragraphs. (3) The ZuCo corpus (Hollenstein et al., 2018, 2020) includes eye tracking data of full sentences from movie reviews and Wikipedia articles.3

Footnote 3: We use Tasks 1 and 2 from ZuCo 1.0 and Task 1 from ZuCo 2.0.

DutchThe GECO corpus (Cop et al., 2017) additionally contains eye tracking data from Dutch readers, which were presented with the same novel in their native language.

GermanThe Potsdam Textbook Corpus (PoTeC, Jager et al. 2021) contains 12 short passages of 158 words on average from college-level biology and physics textbooks, which are read by expert and laymen German native speakers. The full passages were presented on multiple lines on the screen.

\begin{table}
\begin{tabular}{l l r r r r r r} \hline \hline
**Language** & **Corpus** & **Subjs.** & **Sents.** & **Sent. length** & **Tokens** & **Types** & **Word length** & **Flesch** \\ \hline  & Dundee & 10 & 2,379 & 21.7 (1–87) & 51,497 & 9,488 & 4.9 (1–20) & 53.3 \\ English & GECO & 14 & 5,373 & 10.5 (1–69) & 56,410 & 5,916 & 4.6 (1–33) & 77.4 \\  & ZurCo & 30 & 1,053 & 19.5 (1–68) & 20,545 & 5,560 & 5.0 (1–29) & 50.6 \\ \hline Dutch & GECO & 19 & 5,190 & 11.64 (1–60) & 59,716 & 5,575 & 4.5 (1–22) & 57.5 \\ German & PoTeC & 30 & 97 & 19.5 (5–51) & 1,895 & 847 & 6.5 (2–33) & 36.4 \\ Russian & RSC & 103 & 144 & 9.4 (5–13) & 1,357 & 993 & 5.7 (1–18) & 64.7 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Descriptive statistics of all eye tracking datasets.2 Sentence length and word length are expressed as the mean with the min-max range in parentheses. The last column shows the Flesch Reading Ease score (Flesch, 1948) which ranges from 0 to 100 (higher score indicates easier to read). Adaptations of the Flesch score were used for Dutch (nl), German (de) and Russian (ru) (see Appendix B).

 

RussianThe Russian Sentence Corpus (RSC, Laurinavichyute et al.2019) contains 144 naturally occurring sentences extracted from the Russian National Corpus.4 Full sentences were presented on the screen to monolingual Russian-speaking adults one at a time.

Footnote 4: [https://ruscorpora.ru](https://ruscorpora.ru)

### Eye Tracking Features

A fixation is defined as the period of time where the gaze of a reader is maintained on a single location. Fixations are mapped to words by delimiting the boundaries around the region on the screen belonging to each word \(w\). A word can be fixated more than once. For each token \(w\) in the input text, we predict the following eight eye tracking features that encode the full reading process from early lexical access up to subsequent syntactic integration.

Word-level characteristicsWe extract basic features that encode _word-level_ characteristics: (1) number of fixations (nFix), the number of times a subject fixates \(w\), averaged over all subjects; (2) mean fixation duration (MFD), the average fixation duration of all fixations made on \(w\), averaged over all subjects; (3) fixation proportion (fProp), the number of subjects that fixated \(w\), divided by the total number of subjects.

Early processingWe also include features to capture the _early_ lexical and syntactic processing, based on the first time a word is fixated: (4) first fixation duration (FFD), the duration, in milliseconds, of the first fixation on \(w\), averaged over all subjects; (5) first pass duration (FPD), the sum of all fixations on \(w\) from the first time a subject fixates \(w\) to the first time the subject fixates another token, averaged over all subjects.

Late processingFinally, we also use measures reflecting the _late_ syntactic processing and general disambiguation, based on words which were fixated more than once: (6) total reading time (TRT), the sum of the duration of all fixations made on \(w\), averaged over all subjects; (7) number of re-fixations (nRefix), the number of times \(w\) is fixated after the first fixation, i.e., the maximum between 0 and the nFix-1, averaged over all subjects; (8) re-read proportion (ReProp), the number of subjects that fixated \(w\) more than once, divided by the total number of subjects.

The values of these eye tracking features vary over different ranges (see Appendix A). FFD, for example, is measured in milliseconds, and average values are around 200 ms, whereas ReProp is a proportional measure, and therefore assumes floating-point values between 0 and 1. We standardize all eye tracking features independently (range: 0-100), so that the loss can be calculated uniformly over all feature dimensions.

Eye movements depend on the stimulus and are therefore language-specific but there exist universal tendencies which remain stable across languages (Liversedge et al., 2016). For example, the average fixation duration in reading ranges from 220 to 250 ms independent of the language. Furthermore, word characteristics such as word length, frequency and predictability affect fixation duration similarly across languages but the effect size depends on the language and the script (Laurinavichyute et al., 2019; Bai et al., 2008). The word length effect, i.e., the fact that longer words are more likely to be fixated, can be observed across all four languages included in this work (see Appendix A).

## 4 Language Models

We compare the ability to predict eye tracking features in two models: BERT and XLM. Both models are trained on the transformer architecture (Vaswani et al., 2017) and yield state-of-the

\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Short Name** & **Language** & **Model Checkpoint** & **Reference** \\ \hline bert-nl & Dutch & wietsedv/bert-base-dutch-cased & Vries et al. (2019) \\ bert-en & English & bert-base-uncased & Wolf et al. (2019) \\ bert-de & German & bert-base-german-cased & Chan et al. (2019) \\ bert-ru & Russian & DeepPavlov/rubet-base-cased & Yu and Arkhipov (2019) \\ bert-multi & 104 languages & bert-base-multilliJouual-cased & Wolf et al. (2019) \\ \hline xlm-en & English & xlm-mlm-sen-2048 & Lample and Conneau (2019) \\ xlm-and-end & English + German & xlm-MLM-an-edhep-1024 & Lample and Conneau (2019) \\ xlm-17 & 17 languages & xlm-MLM-17-1280 & Lample and Conneau (2019) \\ xlm-100 & 100 languages & xlm-MLM-100-1280 & Lample and Conneau (2019) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Pretrained transformer language models analyzed in this work.

 

art results for a wide range of NLP tasks (Liang et al., 2020). The multilingual BERT model simply concatenates the Wikipedia input from 104 languages and is optimized by performing masked token and next sentence prediction as in the monolingual model (Devlin et al., 2019) without any cross-lingual constraints. In contrast, XLM adds a translation language modeling objective, by explicitly using parallel sentences in multiple languages as input to facilitate cross-lingual transfer (Lample and Conneau, 2019). Both BERT and XLM use subword tokenization methods to build shared vocabulary spaces across languages.

We use the pretrained checkpoints from the HuggingFace repository for monolingual and multilingual models (details in Table 2).5

Footnote 5: [https://huggingface.co/transformers/pretrained_models.html](https://huggingface.co/transformers/pretrained_models.html)

## 5 Method

We fine-tune the models described above on the features extracted from the eye tracking datasets. The eye tracking prediction uses a model for token regression, i.e., the pretrained language models with a linear dense layer on top of it. The final dense layer is the same for all tokens, and performs a projection from the dimension of the hidden size of the model (e.g., 768 for bert-en or 1,280 for xlm-100) to the dimension of the eye tracking feature space (8, in our case). The model is trained for the regression task using the _mean squared error_ (MSE) loss.

Training DetailsWe split the data into 90% training data, 5% validation and 5% test data. We initially tuned the hyper-parameters manually and set the following values for all models: We use an AdamW optimizer (Loshchilov and Hutter, 2018) with a learning rate of \(0.00005\) and a weight decay of \(0.01\). The batch size varies depending on the model dimensions (see Appendix C.2). We employ a linear learning rate decay schedule over the total number of training steps. We clip all gradients exceeding the maximal value of 1. We train the models for 100 epochs, with early stopping after 7 epochs without an improvement on the validation accuracy.

Evaluation ProcedureAs the features have been standardized to the range 0-100, the _mean absolute error_ (MAE) can be interpreted as a percentage error. For readability, we report the _prediction accuracy_ as \(100-\)MAE in all experiments. The results are averaged over batches and over 5 runs with varying random seeds. For a single batch of sentences, the overall MAE is calculated by concatenating the words in each sentence and the feature dimensions for each word, and padding to the maximum sentence length. The per-feature MAE is calculated by concatenating the words in each sentence. For example, for a batch of \(B\) sentences, each composed of \(L\) words, and \(G\) eye tracking features per word, the overall MAE is calculated over a vector of \(B^{*}L^{*}G\) dimensions. In contrast, the MAE for each individual feature is calculated over a vector of \(B^{*}L\) dimensions.

## 6 Results & Discussion

Tables 3 and 4 show that all models predict the eye tracking features with more than 90% accuracy for English and Dutch. For English, the BERT models yield high performance on all three datasets with standard deviations below 0.15. The results for the XLM models are slightly better on average but exhibit much higher standard deviations. Similar to the results presented by Lample and Conneau (2019), we find that more training data from _multiple_ languages improves prediction performance. For instance, the xlm-100 model achieves higher accuracy than the xlm-17 model in all cases. For

Figure 2: True and predicted feature values for two example sentences. On the left the fixation proportion (fProp) values for an English sentence from the ZuCo dataset, and on the right the number of fixations (nFix) values for a Russian sentence from the RSC dataset.

 

the smaller non-English datasets, PoTeC (de) and RSC (ru), the multilingual XLM models clearly outperform the monolingual models. For the English datasets, the differences are minor.

Size EffectsMore training data results in higher prediction accuracy even when the eye tracking data comes from various languages and was recorded in different reading studies by different devices (ALL-LANGS, fine-tuning on the data of all four languages together). However, merely adding more data from the same language (ALL (en), fine-tuning on the English data from Dundee, GECO and ZuCo together) does not result in higher performance.

To analyze this further, we perform an ablation study on varying amounts of training data. The results are shown in Figure 3 for Dutch and English. The performance of the XLM models remains stable even with a very small percentage of eye tracking data. The performance of the BERT models, however, drops drastically when fine-tuning on less than 20% of the data. Similar to Merkx and Frank (2020) and Hao et al. (2020) we find that the model architecture, along with the composition and size of the training corpus have a significant impact on the psycholinguistic modeling performance.

Eye Tracking FeaturesThe accuracy results are averaged over all eye tracking features. For a better understanding of the prediction output, we plot the true and the predicted values of two selected features (fProp and nFix) for two example sentence in Figure 2. In both examples, the model predictions strongly correlate with the true values. The difference to the mean baseline is more pronounced for the fixProfeature.

Figure 4 presents the quantitative differences across models in predicting the individual eye tracking features.6 Across all datasets, first pass duration (FPD) and number of re-fixations (nRefix) are the most accurately predicted features. Proportions (fProp and reProp) are harder to predict because these features are even more dependent on subject-specific characteristics. Nevertheless, when comparing the prediction accuracy of each eye tracking feature to a baseline which always predicts the mean values, the predicted features fProp and reProp achieve the largest improvements relative to the mean baseline. See Figure 5 for a comparison between all features for the best performing model xlm-100 on all six datasets.

Footnote 6: Plots for the remaining datasets are in Appendix D.2

Performance of Pretrained ModelsTo test the language models' abilities on predicting human reading behavior only from pretraining on textual input, we take the provided model checkpoints and use them to predict the eye tracking features without any fine-tuning. The detailed results are presented in Appendix D.1. The achieved accuracy aggregated over all eye tracking features lies between 75-78% for English. For Dutch, the models achieve

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **GECO (nl)** & **PoTeC (de)** & **RSC (ru)** & **ALL-LANGS** \\ \hline bert-nl & 91.81 (0.23) & – & – & – \\ bert-de & – & 78.38 (1.69) & – & – \\ bert-ru & – & – & 78.73 (1.38) & – \\ bert-multi & 91.90 (0.16) & 76.86 (2.42) & 76.54 (3.59) & 94.72 (0.07) \\ \hline xlm-ende & – & 80.94 (0.88) & – & – \\ xlm-17 & 91.04 (0.70) & 86.26 (1.31) & 90.96 (3.96) & 94.46 (0.83) \\ xlm-100 & **92.31** (0.22) & **86.57** (0.54) & **94.70** (0.60) & **94.94** (0.11) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Prediction accuracy over all eye tracking features for the Dutch, German and Russian corpora, and for all four languages combined in a single dataset. Standard deviation is reported in parentheses.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Model** & **Dundee (en)** & **GECO (en)** & **ZuCo (en)** & **ALL (en)** \\ \hline bert-en & 92.63 (0.05) & 93.68 (0.14) & 93.42 (0.02) & 93.71 (0.06) \\ bert-multi & 92.73 (0.06) & 93.73 (0.12) & 93.74 (0.05) & 93.74 (0.07) \\ \hline xlm-en & 90.41 (2.16) & 91.15 (1.42) & 92.03 (2.11) & 90.88 (1.50) \\ xlm-ende & 92.79 (0.15) & **93.89** (0.12) & 93.76 (0.15) & **93.96** (0.08) \\ xlm-17 & 92.11 (1.68) & 91.79 (1.75) & 92.05 (2.25) & 93.80 (0.38) \\ xlm-100 & **92.99** (0.05) & 93.04 (1.40) & **93.97** (0.09) & **93.96** (0.06) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Prediction accuracy over all eye tracking features for the English corpora, including the concatenated dataset. Standard deviation is reported in parentheses.

 

84% accuracy but for Russian merely 65%. Across the same languages the results between the different language models are only minimal. However, on the individual eye tracking features, the pre-trained models do not achieve any improvements over the mean baseline (see Appendix D.1).

## 7 Data Sensitivity

For the main experiment, we always tested the models on held-out data from the same dataset. In this section, we examine the influence of dataset properties (text domain and language) on the prediction accuracy. In a second step, we analyze the influence of more universal input characteristics (word length, text readability).

### Cross-Domain Evaluation

Figure 6 shows the results when evaluating the eye tracking predictions on out-of-domain text for the English datasets. For instance, we fine-tune the model on the newspaper articles of the Dundee corpus and test on the literary novel of the GECO corpus. We can see that the overall prediction accuracy across all eye tracking features is constantly above. 90% in all combinations. This shows that our eye tracking prediction model is able to generalize across domains. We find that the cross-domain capabilities of BERT are slightly better than for XLM. bert-en performs best in the cross-domain evaluation, possibly because its training data is more domain-general since it includes text from Wikipedia and books.

### Cross-Language Evaluation

Figure 7 shows the results for cross-language evaluation to probe the language transfer capabilities of the multilingual models. We test models finetuned on language A on the test set of language B. It can be seen that bert-multi generalizes better across languages than the XLM models. This might be due to the fact that the multilingual BERT model is trained on one large vocabulary of many languages but the XLM models are trained with a cross-lingual objective and language information. Hence, during fine-tuning on eye tracking

Figure 4: Results of individual eye tracking features for all models on the Dundee and RSC corpora.

Figure 5: Improvement of prediction accuracy for the xlm-100 model relative to the mean baseline for each eye tracking feature.

Figure 3: Data ablation study for Dutch and English. The results are aggregated over all eye tracking features. In addition to the mean across five runs, the shaded areas represent the standard deviation. The dashed line is the result of the pretrained bert-multi model without fine-tuning. Results are aggregated over all eye tracking features.

 

data from one language the XLM models lose some of their cross-lingual abilities. Our results are in line with Pires et al. (2019) and Karthikeyan et al. (2020), who showed that BERT learns multilingual representations in more than just a shared vocabulary space but also across scripts. When fine-tuning bert-multi on English or Dutch data and testing on Russian, we see surprisingly high accuracy across scripts, even outperforming the in-language results. The XLM models, however, show the expected behavior where transferring within the same script (Dutch, English, German) works much better than transferring between the Latin and Cyrillic script (Russian).

### Input Characteristics

Gaze patterns are strongly correlated with word length. Figure 8 shows that the models accurately learn to predict higher fixation proportions for longer words. We observe that the predictions of the xlm-100 model follow the trend in the original data most accurately. Similar patterns emerge for the other languages (see Appendix D.3). Notably, the pretrained models before fine-tuning do not reflect the word length effect.

On the sentence level, we hypothesize that eye tracking features are easier to predict for sentences with a higher readability. Figure 9 shows the accuracy for predicting the number of fixations (nFix) in a sentence relative to the Flesch reading ease score. Interestingly, the pretrained models without fine-tuning conform to the expected behavior and show a consistent increase in accuracy for sentences with a higher reading ease score. After finetuning on eye tracking data, this behavior is not as visible anymore since the language models achieve constantly high accuracy independent of the readability of the sentences.

These results might be explained by the nature of the Flesch readability score, which is based only

Figure 6: Cross-domain evaluation on pretrained English models. The results are expressed as the difference in the prediction error compared to the in-domain prediction. A smaller error (i.e., a color more similar to the color of the diagonal) represents better domain adaptation.

Figure 7: Cross-language evaluation on multilingual models across English, Dutch, German and Russian data. The results are expressed as the difference in the prediction error compared to the prediction on the same language. A smaller error (i.e., a color more similar to the color of the diagonal) represents better language transfer.

 

on the structural complexity of the text (see Appendix B for a description of the Flesch Reading Ease score). Our results indicate that language models trained purely on textual input are more calibrated towards such structural characteristics, i.e., the number of syllables in a word and the number of words in a sentences. Hence, the Flesch reading ease score might not be a good approximation for text readability. In future work, comparing eye movement patterns and text difficulty should rely on readability measures that take into account lexical, semantic, syntactic, and discourse features. This might reveal deviating patterns between pretrained and fine-tuned models.

Our analyses indicate that the models learn to take properties of the input into account when predicting eye tracking patterns. These processing strategies are similar to those observed in humans. Nevertheless, the connection between readability and relative importance in text needs to be analysed in more detail to establish how well these properties are learned by the language models.

## 8 Conclusion

While the superior performance of pretrained transformer language models has been established, we have yet to understand to which extent these models are comparable to human language processing behavior. We take a step in this direction by finetuning language models on eye tracking data to predict human reading behavior.

We find that both monolingual and multilingual models achieve surprisingly high accuracy in predicting a range of eye tracking features across four languages. Compared to the XLM models, bertmulti is more robust in its ability to generalize across languages, without being explicitly trained for it. In contrast, the XLM models perform better when fine-tuned on less eye tracking data. Generally, fixation duration features are predicted more accurately than fixation proportion, possibly because the latter show higher variance across subjects. We observe that the models learn to reflect characteristics of human reading such as the word length effect and higher accuracy in more easily readable sentences.

The ability of transformer models to achieve such high results in modelling reading behavior indicates that we can learn more about the commonalities between language models and human sentence processing. By predicting behavioral metrics such as eye tracking features we can investigate the cognitive plausibility within these models to adjust or intensify the human inductive biases.

Figure 9: Prediction accuracy for nFix relative to the Flesch reading ease score of the sentence. A higher Flesch score indicates that a sentence is easier to read. The dashed lines show the results of the pretrained language models without fine-tuning on eye tracking data.

Figure 8: Prediction accuracy of fProp with respect to word length. The gray dashed line is the result of the pretrained bert-multi model without fine-tuning.

 



## Acknowledgements

Lena Jager was partially funded by the German Federal Ministry of Education and Research under grant 01IS20043.

## References

* Abar et al. (2019) Samira Abnar, Lisa Beinborn, Rochelle Choenni, and Willem Zuidema. 2019. Blackbox meets blackbox: Representational similarity & stability analysis of neural language models and brains. In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 191-203.
* Alday (2019) Phillip M Alday. 2019. M/EEG analysis of naturalistic stories: A review from speech to language processing, _Language, Cognition and Neuroscience_, 34(4):457-473.
* Amstad (1978) Toni Amstad. 1978. Wie verstandlich sind unsere Zeitungen? _Unpublished doctoral dissertation, University of Zurich, Switzerland_.
* Bai et al. (2008) Xuejun Bai, Guoli Yan, Simon P Liversedge, Chuanli Zang, and Keith Rayner. 2008. Reading spaced and unspaced Chinese text: Evidence from eye movements. _Journal of Experimental Psychology: Human Perception and Performance_, 34(5):1277.
* Barrett et al. (2018) Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Sogaard. 2018. Sequence classification with human attention. In _Proceedings of the 22nd Conference on Computational Natural Language Learning_, pages 302-312.
* Barrett et al. (2016) Maria Barrett, Joachim Bingel, Frank Keller, and Anders Sogaard. 2016. Weakly supervised part-of-speech tagging using eye-tracking data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, volume 2, pages 579-584.
* Bastings and Filippova (2020) Jasminj Bastings and Katja Filippova. 2020. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_, pages 149-155.
* Canete et al. (2020) Jose Canete, Gabriel Chaperon, Rodrigo Fuentes, and Jorge Perez. 2020. Spanish pre-trained BERT model and evaluation data. _PML4DC at ICLR_.
* Chan et al. (2019) Branden Chan, Timo Moller, Malte Pietsch, and Tanay Soni. 2019. German BERT.
* Conneau et al. (2020) Alexis Conneau, Kartikav Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451.
* Cop et al. (2017) Uschi Cop, Nicolas Dirix, Denis Drieghe, and Wouter Duyck. 2017. Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading. _Behavior Research Methods_, 49(2):602-615.
* De Cao et al. (2020) Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. 2020. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3243-3255.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186.
* Douma (1960) WH Douma. 1960. _De Leesbaarheid Van Landbounvbladen. Een Onderzoek Naar en Een Toepassing Van Leesbaarheidsformules_.
* Ettinger (2020) Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. _Transactions of the Association for Computational Linguistics_, 8:34-48.
* Flesch (1948) Rudolph Flesch. 1948. A new readability yardstick. _Journal of applied psychology_, 32(3):221.
* Gilpin et al. (2018)Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining explanations: An overview of interpretability of machine learning. In _2018 IEEE 5th International Conference on data science and advanced analytics (DSAA)_, pages 80-89. IEEE.
* Gladkova and Drozd (2016) Anna Gladkova and Aleksandr Drozd. 2016. Intrinsic evaluations of word embeddings: What can we do better? In _Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP_, pages 36-42.
* Gonzalez-Garduno and Sogaard (2017) Ana Valeria Gonzalez-Garduno and Anders Sogaard. 2017. Using gaze to predict text readability. In _Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications_, pages 438-443.
* Hahn and Keller (2016) Michael Hahn and Frank Keller. 2016. Modeling human reading with neural attention. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* Hale (2017) John Hale. 2017. Models of human sentence comprehension in computational psycholinguistics. In _Oxford Research Encyclopedia of Linguistics_.



Yiding Hao, Simon Mendelsohn, Rachel Sterneck, Randi Martinez, and Robert Frank. 2020. Probabilistic predictions of people perusing: Evaluating metrics of language model performance for psycholinguistic modeling. In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_, pages 75-86.
* Hawkins et al. (2020) Robert Hawkins, Takateru Yamakoshi, Thomas L Griffiths, and Adele Goldberg. 2020. Investigating representations of verb bias in neural language models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4653-4663.
* Hollenstein et al. (2019) Nora Hollenstein, Maria Barrett, Marius Troendle, Francesco Bigiolli, Nicolas Langer, and Ce Zhang. 2019. Advancing NLP with cognitive language processing signals. _arXiv preprint arXiv:1904.02682_.
* Hollenstein et al. (2018) Nora Hollenstein, Jonathan Rotztejn, Marius Troendle, Andreas Pedroni, Ce Zhang, and Nicolas Langer. 2018. ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading. _Scientific Data_.
* Hollenstein et al. (2020) Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer. 2020. ZuCo 2.0: A dataset of physiological recordings during natural reading and annotation. In _Proceedings of The 12th Language Resources and Evaluation Conference_, pages 138-146.
* Hollenstein and Zhang (2019) Nora Hollenstein and Ce Zhang. 2019. Entity recognition at first sight: Improving NER with eye movement information. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_.
* Ihhoff and Rayner (1986) Albrecht Werner Inhoff and Keith Rayner. 1986. Parafoveal word processing during eye fixations in reading: Effects of word frequency. _Perception & psychophysics_, 40(6):431-439.
* Jager et al. (2021) Lena Jager, Thomas Kern, and Patrick Haller. 2021. Potsdam Textbook Corpus (PoTeC): Eye tracking data from experts and non-experts reading scientific texts. available on OSF, DOI 10.17605/OSF.IO/DN5HP.
* Jain and Wallace (2019) Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3543-3556.
* Jian et al. (2013) Yu-Chin Jian, Ming-Lei Chen, and Hwa-wei Ko. 2013. Context effects in processing of Chinese academic words: An eye-tracking investigation. _Reading Research Quarterly_, 48(4):403-413.
* Karthikeyan et al. (2020) Kaliyaperumal Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: An empirical study. In _International Conference on Learning Representations_.
* Keller (2010) Frank Keller. 2010. cognitively plausible models of human language processing. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics: Short Papers_, pages 60-67.
* Kennedy et al. (2003) Alan Kennedy, Robin Hill, and Joel Pyrne. 2003. The Dundee corpus. In _Proceedings of the 12th European Conference on Eye Movement_.
* Klerke et al. (2016) Sigrid Klerke, Yoav Goldberg, and Anders Sogaard. 2016. Improving sentence compression by learning to predict gaze. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1528-1533.
* Kobayashi et al. (2020) Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7057-7075.
* Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. In _Advances in Neural Information Processing Systems_.
* Laurinavichyute et al. (2019) AK Laurinavichyute, Irina A Sekerina, SV Alexeeva, and KA Bagdasaryan. 2019. Russian Sentence Corpus: Benchmark measures of eye movements in reading in Cyrillic. _Behavior research methods_, 51(3):1161-1178.
* Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fengfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Davain Jiang, Guihong Cao, et al. 2020. XGLUE: A new benchmark dataset for cross-lingual pretraining, understanding and generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6008-6018.
* Liversedge et al. (2016) Simon P Liversedge, Denis Drieghe, Xin Li, Guoli Yan, Xuejun Bai, and Judka Hyona. 2016. Universality in eye movements and reading: A trilingual investigation. _Cognition_, 147:1-20.
* Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In _Proceedings of the International Conference on Learning Representations_.
* Malmaud et al. (2020) Jonathan Malmaud, Roger Levy, and Yevgeni Berzak. 2020. Bridging information-seeking human gaze and machine reading comprehension. In _Proceedings of the 24th Conference on Computational Natural Language Learning_, pages 142-152.
* Manning et al. (2020) Christopher D Manning, Kevin Clark, John Hewitt, Uvashid Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. _Proceedings of the National Academy of Sciences_ 

Franz Matthies and Anders Sogaard. 2013. With blinkers on: Robust prediction of eye movements across readers. _Proceedings of the 2013 Conference on empirical methods in natural language processing (EMNLP)_, pages 803-807.
* Merkx and Frank [2020] Danny Merkx and Stefan L Frank. 2020. Comparing transformers and RNNs on predicting human sentence processing data. _arXiv preprint arXiv:2005.09471_.
* Michaelov and Bergen [2020] James Michaelov and Benjamin Bergen. 2020. How well does surprisal explain N400 amplitude under different experimental conditions? In _Proceedings of the 24th Conference on Computational Natural Language Learning_, pages 652-663.
* Morris [1994] Robin K Morris. 1994. Lexical and message-level sentence context effects on fixation times in reading. _Journal of Experimental Psychology: Learning, Memory, and Cognition_, 20(1):92.
* Oborneva [2006] I Oborneva. 2006. Automatic assessment of the complexity of educational texts on the basis of statistical parameters.
* Pires et al. [2019] Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4996-5001.
* Rayner [1977] Keith Rayner. 1977. Visual attention in reading: Eye movements reflect cognitive processes. _Memory & Cognition_, 5(4):443-448.
* Rayner [1998] Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. _Psychological bulletin_, 124(3):372.
* Reichle et al. [1998] Erik D Reichle, Alexander Pollatsek, Donald L Fisher, and Keith Rayner. 1998. Toward a model of eye movement control in reading. _Psychological review_, 105(1):125.
* Schrimpf et al. [2020] Martin Schrimpf, Idan A Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy G Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2020. Artificial neural networks accurately predict language processing in the brain. _BioRxiv_.
* Sood et al. [2020] Ekta Sood, Simon Tangert, Philipp Mueller, and Andreas Bulling. 2020. Improving natural language processing tasks with human gaze-guided neural attention. _Advances in Neural Information Processing Systems_, 33.
* Takmaz et al. [2020] Ece Takmaz, Sandro Pezzelle, Lisa Beinborn, and Raquel Fernandez. 2020. Generating image descriptions via sequential cross-modal alignment guided by human gaze. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4664-4677.
* Toneva and Weheb [2019] Mariya Toneva and Leila Weheb. 2019. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). In _Advances in Neural Information Processing Systems_, pages 14928-14938.
* Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, pages 5998-6008.
* Virtanen et al. [2019] Antti Virtanen, Jenna Kanerva, Rami Ito, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. _arXiv preprint arXiv:1912.07076_.
* de Vries et al. [2019] Wietse de Vries, Andreas van Cranenburg, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTJE: A Dutch BERT Model. _arXiv preprint arXiv:1912.09582_.
* Vulic et al. [2020] Ivan Vulic, Edoardo Maria Ponti, Ira Leviant, Olga Majewska, Matt Malone, Roi Reichart, Simon Baker, Ulla Petti, Kelly Wing, Eden Bar, et al. 2020. MultimodalLex: A large-scale evaluation of multilingual and cross-lingual lexical semantic similarity. _Computational Linguistics_, pages 1-73.
* Wehbe et al. [2014] Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. 2014. Aligning context-based statistical models of language with brain activity during reading. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 233-243.
* Wilcox et al. [2020] Ethan Gottlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. _arXiv preprint arXiv:2006.01912_.
* Wolf et al. [2019] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shlieifer, Patrick von Platen, Clara Ma, Yacine Jenrite, Julien Phu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. _ArXiv_, abs/1910.03771.
* Yu and Arkhipov [2019] Kuratov Yu and M Arkhipov. 2019. Adaptation of deep bidirectional multilingual transformers for Russian language. _Computational Linguistics and Intellectual Technologies_, (18):333-339 

Eye Tracking Data

Table 6 presents information about the range of the eye tracking features.

Figure 10 shows the word length effect found in eye tracking data recorded during reading. i.e., the fact that longer words are more likely to be fixated. This effect is observable across all languages.

Figure 11 shows the mean fixation duration (MFD) for adjectives, nouns, verbs, and adverbs for all six datasets. We use spacy7 to perform part-of-speech tagging for our analyses. For Russian we load an externally trained model8, for Dutch, English and German we use the provided pretrained models.

Footnote 7: spaCy.io

Footnote 8: [https://github.com/buriy/spacy-ru](https://github.com/buriy/spacy-ru)

Figure 12 shows an additional analysis where we explore which parts-of-speech can be predicted more accurately by the language models.

## Appendix B Readability Scores

We use the Flesch Reading Easy score [4] to define the readability of the English text in the eye tracking corpora. This score indicates how difficult a text passage is to understand. Since this score relies on language-specific weighting factors, we apply the Flesch Douma adaptation for Dutch [5], the adaptation by Amstad [39] for German, and the adaptation by Oborneva [5] for Russian.

## Appendix C Implementation Details

### Tokenization

When using BERT or XLM for token classification or regression, a pressing implementation issue is represented by the subword tokenizers employed by the models. This tokenizer, in fact, handles unknown tokens by recursively splitting every word until all subtokens belong to its vocabulary. For example, the name of the Greek mythological hero "Philammon" is tokenized into the three subtokens "['phiI', '#am', '#mnon']". In this case, our models for token regression would produce an eight-dimensional output for all three subtokens, and we had the choice as to what to do in order to compute the loss, having only one target for the full word "Philammon". We chose to compute the loss only with respect to the first subtoken.

### Training Setup

As described in the main paper, all experiments are run over 5 random seeds, which are \(\{12,79,237,549,886\}\).

All models were fine-tuned on a single GPU Titan X with 12 GB memory. Due to memory restrictions of the GPUs and the dimensions of the language models, the batch size was adapted as needed. Table 5 shows the batch sizes for each model.

On average the validation accuracy of BERT models stops improving after \(\sim 50\) epochs, while the XLM models only take \(\sim 10\) epochs. There is no noteworthy difference in training speed between monolingual and multilingual models.

## Appendix D Detailed Results

In this section we present addition plots that strengthen the results shown in the main paper.

### Pretrained Baseline

Tables 7 and 8 show the prediction accuracy of the pretrained models.

Moreover, Figure 13 shows the results of individual gaze features for all pretrained models (without

Figure 10: Word length effect on all datasets in all four languages.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Model** & **Batch size** \\ \hline Bert-en, bert-nl,bert-multi & 16 \\  bert-multi & \\ \hline bert-de, bert-ru, & 8 \\ xlm-ende, xlm-17, & \\ xlm-100 & \\ \hline xlm-en & 2 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Batch sizes used for each of the language models.

 

fine-tuning) on the Dundee (en) and RSC (ru) corpora.

Figure 14 presents the differences in prediction accuracy for the pretrained xml-100 model predictions relative to the mean baseline for each eye tracking feature. The pretrained models clearly cannot outperform the mean baseline for any language or dataset.

### Individual Feature Results

Figure 15 shows the prediction accuracy of the fine-tuned language models for the individual eye tracking features for all datasets.

### Word Length Effect

Figure 16 presents the comparison between models predictions and original word length effects for further languages.

 



\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline Corpus & nFix & MFD & fpProp & FFD & FPD & TRT & nRefix & reProp \\ \hline Dundee (en) & 0.8 (0.5) & 119.5 (62.1) & 0.6 (0.3) & 120.7 (63.4) & 140.6 (88.5) & 156.1 (105.5) & 0.2 (0.3) & 0.2 (0.2) \\ GECO (en) & 0.8 (0.5) & 128.4 (59.0) & 0.6 (0.2) & 129.3 (60.1) & 143.3 (77.5) & 168.2 (102.4) & 0.2 (0.3) & 0.2 (0.2) \\ ZuCo (en) & 1.1 (0.7) & 78.4 (34.8) & 0.7 (0.3) & 77.3 (34.4) & 92.3 (52.2) & 129.8 (89.7) & 0.4 (0.5) & 0.3 (0.2) \\ GECO (nl) & 0.8 (0.6) & 121.3 (80.1) & 0.6 (0.4) & 121.8 (81.1) & 134.1 (98.0) & 158.1 (131.2) & 0.2 (0.4) & 0.1 (0.2) \\ PoTeC (de) & 2.7 (2.9) & 217.5 (117.3) & 0.8 (0.4) & 167.9 (157.4) & 224.7 (264.2) & 675.6 (727.0) & 1.7 (2.2) & 0.6 (0.5) \\ RSC (ru) & 0.8 (0.4) & 203.4 (115.1) & 0.6 (0.3) & 233.6 (49.5) & 285.1 (101.9) & 314.2 (179.8) & 0.1 (0.1) & 0.1 (0.1) \\ \hline \end{tabular}
\end{table}
Table 6: Mean and standard deviation for all eye tracking features of the corpora used in this work.

Figure 11: Mean fixation duration (MFD) for the most common parts of speech across all six datasets.

Figure 12: Accuracy of the language models predicting the mean fixation duration (MFD) across various parts of speech for Dutch (left) and English (right).





## 6 Conclusion

In this paper, we have proposed a novel approach to the performance of the proposed method for generating and analyzing the data from different perspectives. We propose an efficient framework for generating and visualizing the features in terms of the parameters of the model. The proposed methods are based on the proposed method and can be used as a benchmark for generating and visualizing the features in terms of the parameters of the model. In particular, we also provide a new approach for generating and visualizing the features in terms of the parameters of the model. Our results show that our method is able to generate and visualize the features in terms of the parameters of the model.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline

- \\ bert-DE &
- & 73.55 (3.07) &
- \\ bert-RU &
- & 64.83 (2.09) &
- \\ bert-multi & 84.28 (0.10) & 73.47 (3.01) & 64.82 (2.11) & 86.22 (0.29) \\ \hline xlm-ende &
- & 73.49 (2.99) &
- \\ xlm-17 & 83.93 (0.16) & 73.17 (2.86) & 65.02 (2.11) & 85.84 (0.27) \\ xlm-100 & 83.94 (0.27) & 73.28 (2.91) & 64.67 (2.10) & 85.94 (0.38) \\ \hline \hline \end{tabular}
\end{table}
Table 8: Prediction accuracy of the pretrained language models aggregated over all eye tracking features for the Dutch, German and Russian corpora, and for all four languages combined in a single dataset. Standard deviation is reported in parentheses.

Figure 14: Differences in prediction accuracy for the pretrained XLM-100 model predictions (without fine-tuning on eye tracking data) relative to the mean baseline for each eye tracking feature.

Figure 13: Results of individual gaze features for all pretrained models (without fine-tuning) on the Dundee (en) and RSC (ru) corpora.



Figure 15: Results of individual eye tracking features for all fine-tuned models on all datasets not presented in the main paper.

 



## Appendix A

Figure 16: Word length versus predicted fixation probability for Russian, German and English. The gray dashed line is the result of the pretrained bert-multi model without fine-tuning.