SlowGuess/NAACL_2021_OCR huggingface dataset:
All models were fine-tuned on a single GPU Titan X with 12 GB memory. Due to memory restrictions of the GPUs and the dimensions of the language models, the batch size was adapted as needed. Table 5 shows the batch sizes for each model.
Model	Batch size
BERT-EN, BERT-NL, BERT-MULTI	16
BERT-DE, BERT-RU, XLM-ENDE, XLM-17, XLM-100	8
XLM-EN	2
Table 5: Batch sizes used for each of the language models.
On average the validation accuracy of BERT models stops improving after  $\sim 50$  epochs, while the XLM models only take  $\sim 10$  epochs. There is no noteworthy difference in training speed between monolingual and multilingual models.
In this section we present addition plots that strengthen the results shown in the main paper.
D.1 Pretrained Baseline
Tables 7 and 8 show the prediction accuracy of the pretrained models.
Moreover, Figure 13 shows the results of individual gaze features for all pretrained models (without
7spaCy.io
<sup>8</sup>https://github.com/buriy/spacy-ru
fine-tuning) on the Dundee (en) and RSC (ru) corpora.
Figure 14 presents the differences in prediction accuracy for the pretrained XML-100 model predictions relative to the mean baseline for each eye tracking feature. The pretrained models clearly cannot outperform the mean baseline for any language or dataset.
D.2 Individual Feature Results
Figure 15 shows the prediction accuracy of the fine-tuned language models for the individual eye tracking features for all datasets.
D.3 Word Length Effect
Figure 16 presents the comparison between models predictions and original word length effects for further languages.
Corpus	NFIX	MFD	FPROP	FFD	FPD	TRT	NREFIX	REPROP
Dundee (en)	0.8 (0.5)	119.5 (62.1)	0.6 (0.3)	120.7 (63.4)	140.6 (88.5)	156.1 (105.5)	0.2 (0.3)	0.2 (0.2)
GECO (en)	0.8 (0.5)	128.4 (59.0)	0.6 (0.2)	129.3 (60.1)	143.3 (77.5)	168.2 (102.4)	0.2 (0.3)	0.2 (0.2)
ZuCo (en)	1.1 (0.7)	78.4 (34.8)	0.7 (0.3)	77.3 (34.4)	92.3 (52.2)	129.8 (89.7)	0.4 (0.5)	0.3 (0.2)
GECO (nl)	0.8 (0.6)	121.3 (80.1)	0.6 (0.4)	121.8 (81.1)	134.1 (98.0)	158.1 (131.2)	0.2 (0.4)	0.1 (0.2)
PoTeC (de)	2.7 (2.9)	217.5 (117.3)	0.8 (0.4)	167.9 (157.4)	224.7 (264.2)	675.6 (727.0)	1.7 (2.2)	0.6 (0.5)
RSC (ru)	0.8 (0.4)	203.4 (115.1)	0.6 (0.3)	233.6 (49.5)	285.1 (101.9)	314.2 (179.8)	0.1 (0.1)	0.1 (0.1)
Figure 11: Mean fixation duration (MFD) for the most common parts of speech across all six datasets.
Figure 12: Accuracy of the language models predicting the mean fixation duration (MFD) across various parts of speech for Dutch (left) and English (right).
Table 6: Mean and standard deviation for all eye tracking features of the corpora used in this work.
Model	Dundee	GECO (en)	ZuCo (en)	ALL (en)
BERT-EN	77.42 (0.21)	77.67 (0.13)	76.06 (0.38)	78.69 (0.09)
BERT-MULTI	77.41 (0.21)	77.68 (0.13)	76.07 (0.37)	78.66 (0.07)
XLM-EN	77.21 (0.29)	77.65 (0.24)	75.97 (0.60)	78.47 (0.11)
XLM-ENDE	77.40 (0.29)	77.67 (0.10)	76.10 (0.41)	78.66 (0.12)
XLM-17	77.31 (0.23)	77.66 (0.19)	75.99 (0.39)	78.39 (0.15)
XLM-100	77.35 (0.29)	77.63 (0.34)	75.93 (0.43)	78.49 (0.11)
Table 7: Prediction accuracy of the pretrained language models aggregated over all eye tracking features for the English corpora, including the concatenated dataset. Standard deviation is reported in parentheses.
Model	GECO (nl)	PoTeC (de)	RSC (ru)	ALL-LANGS
BERT-NL	84.20 (0.10)	-	-	-
BERT-DE	-	73.55 (3.07)	-	-
BERT-RU	-	-	64.83 (2.09)	-
BERT-MULTI	84.28 (0.10)	73.47 (3.01)	64.82 (2.11)	86.22 (0.29)
XLM-ENDE	-	73.49 (2.99)	-	-
XLM-17	83.93 (0.16)	73.17 (2.86)	65.02 (2.11)	85.84 (0.27)
XLM-100	83.94 (0.27)	73.28 (2.91)	64.67 (2.10)	85.94 (0.38)
Table 8: Prediction accuracy of the pretrained language models aggregated over all eye tracking features for the Dutch, German and Russian corpora, and for all four languages combined in a single dataset. Standard deviation is reported in parentheses.
Figure 13: Results of individual gaze features for all pretrained models (without fine-tuning) on the Dundee (en) and RSC (ru) corpora.
Figure 14: Differences in prediction accuracy for the pretrained XLM-100 model predictions (without fine-tuning on eye tracking data) relative to the mean baseline for each eye tracking feature.
Figure 15: Results of individual eye tracking features for all fine-tuned models on all datasets not presented in the main paper.
Figure 16: Word length versus predicted fixation probability for Russian, German and English. The gray dashed line is the result of the pretrained BERT-MULTI model without fine-tuning.