ScanDL: A Diffusion Model for Generating Synthetic Scanpaths on Texts
Authors: Lena Sophia Bolliger
David Robert Reich
Patrick Haller
Deborah Noemie Jakobi
Paul Prasse
Lena Ann Jäger
Venue: EMNLP 2023 Main (2023)
Keywords: scanpath generation
eye movements
diffusion models
computational psycholinguistics
deep neural networks
transformer
eye tracking

Abstract: Eye movements in reading play a crucial role in psycholinguistic research studying the cognitive mechanisms underlying human language processing. More recently, the tight coupling between eye movements and cognition has also been leveraged for language-related machine learning tasks such as the interpretability, enhancement, and pre-training of language models, as well as the inference of reader- and text-specific properties. However,  scarcity of eye movement data and its unavailability at application time poses a major challenge for this line of research. Initially, this problem was tackled by resorting to cognitive models for synthesizing eye movement data. However, for the sole purpose of generating human-like scanpaths, purely data-driven machine-learning-based methods have proven to be more suitable. Following recent advances in adapting diffusion processes to discrete data, we propose ScanDL, a novel discrete sequence-to-sequence diffusion model that generates synthetic scanpaths on texts. By leveraging pre-trained word representations and jointly embedding both the stimulus text and the fixation sequence, our model captures multi-modal interactions between the two inputs. We evaluate ScanDL within- and across-dataset and demonstrate that it significantly outperforms state-of-the-art scanpath generation methods. Finally, we provide an extensive psycholinguistic analysis that underlines the model's ability to exhibit human-like reading behavior. Our implementation is made available at https://github.com/DiLi-Lab/ScanDL.

SCANDL: A Diffusion Model for Generating Synthetic Scanpaths on Texts
Lena S. Bolliger', David R. Reich", Patrick Haller', Deborah N. Jakobi' Paul Prasse", Lena A. Jäger'.
! Department of Computational Linguistics, University of Zurich, Switzerland DDpaarmeett of Computer Science, University of Potsdam, Germany {bolliger haller, haller,jakobi, jaeger uzh. ch {david. reich, paul (aprasseJuunn -potsdi.ptmdamm.ccm de
Abstract
Progresivv stccade
Regnessive succade
Fixation
Eye movements in reading play crucial role in psycholinguistic research studying the cog- nitive mechanisms underlying human language processing. More recently, the tight coupling between eye movements and cognition has also been leveraged for language-related machine learning tasks such as the interpretabilityy en- hancement, and pre-training of language mod- cls, as well the inference of reader- and text-specific properties. However, scarcity of eye movement data and its unavailability at application time poses a major challenge for this line of research. Initially, this problem tackled by resorting to cognitive models for synthesizing eye movement data. How- ever, for the sole purpose of generating human- like scanpaths, purely data-driven machine- Saaming-hased methods have proven to he more suitable. Following recent advances in adapt- ing diffusion processes to discrete data, pro- pose SCANDL, a novel discrete sequence-to- sequence diffusion model that generates syn- thetic scanpaths on texts. By leveraging pre- trained word representations and jointly em- bedding both the stimulus text and the fixation sequence, our model captures mudti-modal in- teractions between the two inputs. We evalu- ate SCANDL within- and across-dataset and demonstrate that it significantly outperforms state-of-tte aht tta scanpath generation methods. Finally, we provide an extensive psycholin- guistic analysis that underlines the model's ability exhibit human-like reading behay- ior. Our implementation is made available at https:///g /gi thub. www/Diii-Lab/ ScanDL.
Human Scanpa Lh:
[CL5]
you
imag he? [
Fixation Lndax:
SCANDL:
[CL8] Can you
imag he?
[SEP]
Lodexs
Figure I: Human scanpath vs. SCANDL.
psycholinguistics, and computer science. As a re- sult, computational models of eye movements in reading experienced upsurge over the past two decades. The earlier models are explicit compu- tational cognitive models designed for fundamen- tal research with the aim shedding light on i) the mechanisms underlying human language com- prehension at different linguistic levels and ii) the broader question of how the human language pro- cessing system interacts with domain-geneee cog- nitive mechanisms and capacities, such as working memory or visual attention (Reichle et al., 2003; Lingbert al., 2005; Engellaan et al., 2013). More recently, traditional and neural machine learning (ML) approaches have been adopted for the predic- tion of human eye movements (Nilsson and Nivre, 2009, 2011; Hahn and Keller, 2016; Wang et al.. 2019; Deng et al., 2023b) which, in contrast to cog- nitive models, do not implement any psychological or linguistic theory of eye movement control in reading. ML models, in turn, exhibit the flexibil- ity to learn from and adapt any kind of reading pattern on any kind of text. Within the field of ML. researchers have not only begun to create synthetic scanpaths but also, especially within NLP, to lever- age them for different use cases: the interpretability of language models (LMs) (Sood et al., 2020a; Hol- lenstein al., 2021, 2022; Merkx and Frank, 2021), enhancing the performance of LMs on downstream tasks (Barrett al., 2018: Hollenstein and Zhang, 2019; Sood al., 2020b; Deng et al., 2023a), and ppe-training models for all kinds of inference tasks concerning reader- and text-specific proper-
1 Introduction
As human eye movements during reading pro- vide both insight into the cognitive mechanisms involved in language processing (Rayner, 1998) and information about the key properties of the text (Rayner, 2009), they have been attracting in- creasing attention from across fields, including cog- nitive psychology, experimental and computational

---

ties (e.g., assessing reading comprehension skills or L2 proficiency, delecting dyslexia, or judging text readability) (Berzak et al., 2018; Raatikainen et al., 2021; Reich et al., 2022; Haller et al., 2022b; Hahn and Keller, 2023). For these NLP use cases, the opportunity to generate large amounts of synthetic scanpaths is crucial for two reasons: first, real hu- man eye movement data is scarce, and its collection is resource intensive. Second, relying on real hu- man scanpaths entails the problem of not being able to generalize beyond the respective dataset, as gaze recordings are typically not available at infer- ence time. Synthetic scanpaths resolve both issues. However, generating synthetic scanpaths is not a trivial task, as it is soquence-to-sequenee problem that requires the alignment of two different input se- quences: the word sequence (order of words in the sentence), and the scanpath (chronological order of fixations on the sentence).
and duration on a textual stimulus guided by lin- guistic variables such as lexical frequeney and pre- dictability. While these explicit cognitive models implement theories of reading and are designed to explain empirically observed psycholinguistic phe- nomena, second line of rescarch adopts a purely data-driven approach aiming solely at the accurate prediction of eye movement patterns. Nilsson and Nivre (2009) trained a logistic regression manu- ally engineered features extracted from a reader's eye movements and, in an extension, also the stim- ulus text to predict the next fixation (Nilsson and Nivre, 2011). More recent research draws inspi- ration from NLP sequence labeling tasks. For in- stance, Hahn and Keller (2016, 2023) proposed an unsupervised sequence-fo-sequennc architecture, adopting labeling network to determine whether the next word is fixated. Wang et al. (2019) pro- posed a combination of CNNs, LSTMs, and CRF to predict the fixation probability of each word in a sentence. A crucial limitation of these mod- els is their simplifying the dual-input sequence into a single-sequence problem, not accounting for the chronological order in which the words are fixated and thus unable to predict important aspects of eye movement behavior, such as regres- sions and re-fixations. To overcome this limitation. Deng et al. (2023b) proposed Eyettention, a dual- sequence encoder-encoder architecture, consisting of two LSTM encoders that combine the word se- quence and the fixation sequence by means of a cross-attention mechanism; their model predicts next fixations in an auto-regressive (AR) manner.
In this paper, we present a novel discrete diffusion model for the gen- eration of synthetic human scanpaths on a given stimulus text: SCANDL, Scanpath Diffusion condi- tioned on Language input (see Figure 1). SCANDL leverages pre-trained word representations for the text to guide the model's predictions of the location and the order of the fixations. Moreover, it aligns the different sequences and modalities (text and eye gaze) by jointly embedding them in the same continuous space, thereby capturing dependencies and interactions between the two input sequences.
The contributions of this work are manifold: We (i) develop SCANDL, the first diffusion model for simulating scanpaths in reading, which outper- forms all previous state-of-the-an methods; we (ii) demonstrate SCANDL's 's ability to exhibit human- like reading behavior, by means of a Bayesian psy- cholinguistic analysis; and we (iii) conduct an ex- tensive ablation study to investigate the model's different components, evaluate its predictive ca- pabilities with respect to scanpath characteristics, and provide a qualitative analysis of the model's decoding process.
Diffusion models for discrete input. Ap- proaches to apply diffusion processes to discrete input (mainly text) comprise discrete and contin- uous diffusion. Reid et al. (2022) proposed an based diffusion model for machine translation and summarization whose corruption process hap- pens in discrete space. Li et al. (2022) and Gong et al. (2023) both proposed continuous diffusion models for conditional text generation; whereas the former adopted classifiers to impose constraints on the generated sentences, the latter conditioned on the entire source sentence. All of these approaches consist of uni-modal input being mapped again to the same modality.
2
Related Work
Models of eye movements in reading. Two com- putational cognitive models of eye movement con- trol in reading have been dominant in the field dur- ing the past two decades: the F--Z reader model (Re- ichle et al., 2003) and the SWIFT model (Engbert et al., 2005). Both models predict fixation location
3 Problem Setting
Consider a scanpath f (fi...., which represents a sequence of fixations generated by

---

Told
Iilt
P
#irudenca
prudence
pludence
Mixtr
Mi
Blacc
Jess
ft
ft
Faxu
2
4
101
6404
1175
12087
185
19424
119
[PAD]
iPAd
(PAd]
(PAD)
[PAD]
Figure 2: Discrete input representation of the concatenation x of sentence x" and and scanpath xf. Each element of the sequence x represented by triple of word index Xidr- BERT input ID Xtent and position index Xpos-
reader while reading sentence Here, ʃj de- notes the location of the jth fixation represented by the linear position of the fixated word within the sentence w (word index). The goal to find a model that predicts a scanpath f given sentence w. We evaluate the model by computing the mean Normalized Levenshtein Distance (NLD) (Leven- shtein, 1965) between the predicted and the ground truth human scanpaths. Note that several readers In the follow-
derived from the BERT tokenizer (Devlin et al., 2019), refer to the tokenized subwords of the stim- ulus sentence for x while consisting merely of PAD tokens for the scanpath xf, as no mapping be- tween fixations and subwords is available (orange row in Figure 2). Finally, position indices Xpos cup- ture the order of words within the sentence and the order of fixations within the scanpath, respectively (red row in Figure 2).
read the same sentence ing, will denote the scanpath by f instead of f if the reader or sentence is unambiguous or the (predicted) scampath is not dependent on the reader.
4.2 Diffusion Model
A diffusion model (Sohl-Dickstein et al., 2015) latent variable model consisting forward and a reverse process. In the forward process, we sample from a real-world data distribution and gradually corrupt the data sample into standard Gaussian noise Zi-- (0, where T is the maximal number of diffusion steps. The latents Zi..... Zj are modeled as first-order Markov chain, where the Gaussian corruption of each termittent noising step & [1,..., is given by 2 ~ ((2 2 1) = -N(TT Biz B.T. where B, E (0, 1) is hyperparameter dependent on t. The reverse distribution p(z.-1/zı) gradually re- moves noise to reconstruct the original data sample 2o and is approximated by po-
4 SCANDL
Inspired by continuous diffusion models for text generation (Gong et al., 2023; et al., 2022), we propose SCANDL, a diffusion model that synthe- sizes scanpaths conditioned on stimulus sentence.
4.1 Discrete Input Representation
SCANDL uses discrete input representation for both the stimulus sentence and the scanpath. First,
we Sabbord-tokenize the stimulus sentence w (w.,..., ww) using the pre- pre-trained BERT Word- Piece tokenizer (Devlin et al., 2019; Song et al., 2021). We prepend special CLS and append special SEP tokens to both the sentence and the scanpath in order to obtain the stimulus sequence x (WCLS, Wl...., WM, WSEP) and corresponding ... fi..... We introduce these special tokens in order to sepa- rate the two sequences and to align their beginning and ending. The two sequences are concatenated along the sequence dimension into x = x + xf. An example of the discrete input x is depicted in Figure 2 (blue row). We utilize three features to provide discrete representation for every element in the sequence x. The word indices Xyd align fix- ations in xf with words in x w and align subwords in x" originating from the same word (yellow row in Figure 2). Second, the BERT input IDs Xbert-
4.3 Forward and Backward Processes
In the following, we describe SCANDL'S 's projec- tion of the discrete input into continuous space, its forward noising process and reverse denoising process pe (all depicted in Figure 3), well architectural details and diffusion-related methods.
fixation sequence x (fols, fi....
fN. fser).
4.3.1 Forward Process: Embedding of Discrete Input in Continuous Space Following Gong et al. (2023), our forward process deploys an embedding function EMB (-): NM+N+4
R(M+N+4)xd that maps from the discrete input representation into continu- ous space, where N and M denote the number of fixations and words, respectively, and d is the size of the hidden dimension. This embedding learns

---

Noising step T
lo (50 x)
Noising process
Gaussian
EMB
a 10/ 1)
Dert
0
Z
TT
EMB
x W
timestep EMB
pa
(2
poler 1
0
2,
ZT
pg( [ (0))
Denoising process
Discrete space
Latent continuous space
Figure 3: The embedding layer, the forward process (noising) and the reverse process (denoising) of SCANDL.
a joint representation the subword-tokenized sentence x" and the fixation sequence xf. More precisely, the embedding function EMB (x) =
and z is noised, i... 2f ~ q(2) 2f_1)
((v1- B B,11), with 1 < < T, where T is the amount of noise, and 1 S T S T, where i is the maximal number of diffusion steps. This partial noising (Gong et al., 2023) is erucial, as the sentence on which a scanpath is conditioned must remain uncorrupeed.
EMBia((Xux)) + EXBeee(Xteer) + po Xpos) is the sum of three independent embedding layers (see Figure 7 in Appendix C). While the word index embedding EMB and the position embedding EMBpos are learned during training, the pre-trained BERT model embedding EMB is frozen. It maps the input IDs to pre-trained BERT embeddings (Devlin et al., 2019) and adds semantic meaning to the sentence. Only the word index embedding EMB3, is corrupted by noise during the forward process; this embedding is what the model has learn. The other two embeddings remain unnoised.
4.3.3 Backward Process: Denoising
During the denoising process, a parametrized model fe learns to step-wise reconstruct by de- noising 27. Due to the first-order Markov property, the joint probability of all latents can be factor- ized po (2o.T) p((r) I Po (o This denoising process modeled po 1/2+) A/ (po(z,, t), (z, where pa(-) and (-(-) are the model's Le predicted mean and variance of the true posterior distribution g(z)-1/2)) l2:). The optimization criterion of the diffusion model is to maximize the marginal log lig-likelhhod of the data log Since directly computing and maxi- mizing log p(zo) would require access to the truc posterior distribution g (z.). maximize the variational lower bound (VLB) of log p(zo) as a proxy objective, defined Equation 1. However, as SCANDL involves mapping the discrete input into continuous space and back, the training ob- jective becomes the minimization of LSCANDL. a joint loss comprising three components, inspired by Gong et al. (2023). LSCANDL is defined in Equa- tion where fo is the parametrized neural network trained to reconstruct Zg from 2T (see Section 4.5).
Embedding the discrete input x into continu- ous space using our embedding function EMB(-) allows for new transition ž qo (z)lx)
N (EMB(x), BoI) to extend the original forward Markov chain, where An is the first latent variable at noising step / = 0 and d6 is the parametrized embedding step of the forward process. Note that this initial latent variable zg is not only projection of the discrete data x into continuous space, i.c. EMB(x), but is actually sampled from a normal distribution that is centered around Ex)(
4.3.2 Forward Process: Partial Noising
Let zo 2 @ of be initial latent variable. where 2 refers to the sentence subsequence and zf the fixation subsequence. Each subsequent latent 2; E .2, is given by 2; := zτ"" @ zf. where z," remains unchanged, i.e., zi"" = zi
Note that 7 can he 0 (see Section 4.5); if = 0, the model leams to reconstruct x from 2o ~ (X)EMMBx)

---

(zr) (zo 21)
T
(z)-1 2s)
VLB = Be 20)
log
log
(1)
9 (2r (0)
2 (2 ne 2a))
argmin LSCANDL argmin
Σ Ilfo (zs, 20112 [|FMM()M)))) ) fo 1)11
log po (x(2z) (zu)
(2)
LvLB
The first component LVLB is derived from the VLB (sce Appendix G for detailed derivation), and aims minimizing the difference between ground- truth Zo and the model prediction fe (zi, 1). The second component LEMB measures the difference between the model prediction Se (zt, 1) and the embedded input. The last component Ground cor- responds to the reverse embedding, or rounding operation, which pipes the continuous model pre- diction through reverse embedding layer to obtain the discrete representation.
12 blocks, with 8 attention heads and hidden dimen- sion d 256. An extra linear layer projects the pre-trained BERT embedding EMB to the hid- den dimension d. The maximum sequence length is 128, and the number of diffusion steps 7 2000. We use a sqrt noise schedule (Li at., 2022) to sam- ple 3r
V T+s' where 0.0001 is small constant corresponding to the initial noise level. To sample the noising step 7 0, . we employ importance sampling as defined by Nichol and Dhariwal (2021).
4.4 Inference
At inference time, the model fe needs to con- struct scanpath on a specific sentence Specif- ically, to synthesize the scanpath and condition it on the embedded sentence EMB(x"), we re- place the word index embedding of the scanpath EMB (xf idr.) with Gaussian noise, initializing as xide ~ (0, We then concatenate the new embedding EMB(x* PMB (xf par) with EMB to obtain the model input zy. The model fe then iteratively denoises Zij into Zo- At each denoising step t, an anchoring function is applied to ze that serves two different purposes. First, it performs rounding on (Li et al., 2022), which entails mapping it into discrete space and then projecting back into continuous enforce intermediate steps to com- mit to a specific discrete representation. Second, it replaces the part in Z that corresponds to the condition x" w with the original EMB(x") (Gong et al., 2023) to prevent the condition from being corrupted by being recovered by the model fø. Af- ter denoising Z into 2g. An is piped through the inverse embedding layer to obtain the predicted scanpath ff.
5 Experiments
To evaluate the performance of SCANDL against both cognitive and neural scanpath generation methods, we perform within- and an across- dataset evaluation. All models were implemented in PyTorch (Paszke et al., 2019), and trained for 80,000 steps on four NVIDIA GeForce RTX 3090 GPUs. For more details on the training, see Ap- pendix A. Our code is publicly available at https: Bgggthub.ccm/Dii-LL//SSaSaa
x + EMBber berr) +
5.1 Datasets
We use two train and/or evaluate model. The Corpus of Eye Movements in LI and L2 English Reading (CELER, Berzak et al., 2022) is an English sen- tence corpus including data from native (LI) and non-native (1.2) English speakers, of which we only use the L1 data (CELER L1). The Zurich Cognitive Language Processing Corpus (ZuCo, Hollenstein et al., 2018) is an English sentence corpus compris- ing both "lask-specific" and "natural" reading, of which only include the natural reading (ZuCo NR). Descriptive statistics for the two corpora in- cluding the distribution of reading measures and participant demographics can found Section B of the Appendix.
corpora to
space
4.5
Model and Diffusion Parameters
Our parametrized model fo consists of an encoder- only Transformer (Vaswani et al., 2017) comprising
RRecall that EMB(x) # Zo. as Zo ~ (EMMBB(x),

---

5.2 Reference Methods
New Reader setting. We test the model's ability to generalize to novel readers; the test set consists of scanpaths from readers that have not been seen during training, although the sentences appear both in training and test set.
We compare SCANDL with other state-of-the-art approaches to generate synthetic scanpaths includ- ing two well-established cognitive models, the H-Z reader model (Reichle et al., 2003) and the SWIFT model (Engbert et al., 2005), and one machine- learning-based model, Eyettention (Deng et al., 2023b). Morcover, we include a human bascline, henceforth referred to as Human, that measures the inter-reader scanpath similarity for the same sentence. Finally, we compare the model with two trivial baselines. One is the Uniform model, which simply predicts fixations iid over the sentence, and the other one, referred samples the saccade range from the training label distribution (Deng et al., 2023b).
Results. Again, our model both improves over the previous oftteoof-the-art, the cognitive models, as well over the Human baseline. Even more, the model achieves greater similarity novel readers as compared novel sentences in the previous setting.
New Reader/New Sentence setting. The test set comprises only sentences and readers that the model has not seen during training to assess the model's ability to simultaneously generalize to novel sentences and novel readers. Of the within- dataset settings, this setting exhibits the most out- of df-diitrribtionn qualities.
Train-label-dist model,
5.3 Evaluation Metric
To assess the model performance, we compute the Normalized Levenshtein Distance (NLD) between the predicted and the human scanpaths. The Lev- enshtein Distance (LD, Levenshtein, 1965) is similarity-base -based metric quantifying the minimal number of additions, deletions and substitutions required to transform a word-index sequence of a true scanpath into a word-index sequence of the model-predictee scanpath. Formally, the NLD is defined as NLD(S, $) = LD/ max([S),
Results. Again, SCANDL significantly outper- forms the previous state-of-the-art. Again, in con- trast previous approaches and in line with the other settings, SCANDL attains higher similarity as measured in NLD than the Human baseline.
5.6 Across-Datase Evaluation
To evaluate the generalization capabilities of our model, we train on CELER L.1 (Berzak et al., 2022), while testing it across-datasct on ZuCo NR (Hollenstein et al., 2018). Although the model has to generalize to unseen readers and sen- tences in the New Reader/New Sentence setting, the hardware setup and the presentation style includ- ing stimulus layout of the test data the same, and the readers stem from the same population. the
5.4 Hyperparameter Tuning
To find the best model-specific and training-specific hyperparameters of SCANDL, we perform triple cross-validation on the New Reader/New Sentence setting (for the search space, see Appendix A).
evaluation, therefore, we not only examine the model's ability to generalize to novel readers and sentences, but also to unfamiliar hardware and presentation style.
5.5 Within-Dataset Evaluation
For the within-dataset evaluation, we perform 5- fold cross-validation CELER L1 (Berzak et al., 2022) and evaluate the model in 3 different settings. The results for all settings are provided in Table
Results. The results for this setting can also be found in Table 1. SCANDL outperforms all reference models, and achieves similarity with true scanpath on par with the Human baseline.
New Sentence setting. We investigate the model's ability to generalize to novel sentences read by known readers; i.e., the sentences in the test set have not been seen during training, but the readers appear both in the training and test set. Results. SCANDL not only outperforms the previous state-of-the-an Eyettention (Deng al.. 2023b) as well as other reference methods by a significant margin, but even exceeds the similarity that is reached by the Human baseline.
5.7 Ablation Study
In this section, we investigate the effect of omit- ting central parts of the model: SCANDL without the position embedding EMB and the pre-trained BERT embedding EMB and SCANDL with- out the sentence condition (unconditional scanpath generation). Additionally, we also consider the two previously prevalent noise schedules, the linear

---

Model
New Sentence
New Reader
New Reader/Neww Sentence
Acruss-Datase
Uniform
0.779 0.002
0.781 = 0.003i
0.782 0.005t
0.802
Train-lahel-dist
0.672 0.003f
0.672 = 0.004f
0.674 0.005f
0.723
E-Z Reader
0.619 0.005t
0.620 = 0.006i
0.622 + 0.006f
0.667
SWIFT
0.607 0.004
0.608 = 0.006i
0.607 1 0.006
0.703
Eyettention
0.580 0.002
0,580 = 0.004
0.578 + 0.006
0.697
SCANDL
0,516 0.006
0,509 0.014
0,515 1 0.014
0.647
Human
0,538 + 0.006
0.536 0.004
0.538 + 0.006
0.646 = 0.002
Table 1: We report NLD = standard error for all settings. The dagger indicates models significantly worse than the best model. In the New Sentence, New Reader, and New Reader/New Senience settings, models are evaluated using five-fold cross-valiin valss-vali,tinn. In the
setting, the model is trained on CELER L1 and tested on ZuCo NR.
Ablation case
NLD |
hand. Effect sizes are estimated using Bayesian generalized linear-mixed models with reading mea- sures as target variables and psycholinguistic fea- tures predictors; logistic models for FPR and SR, Poisson models for FFC and TFC. For the hu- man data, fit random intercepts." 3 We compute posterior distributions over all effect sizes using brms (Bürkner, 2017), running 4 chains with 4000 iterations including 1000 warm-up iterations. Results. We present the posterior distributions of the effect estimates obtained for the four read- ing measures in the New Reader/New Sentence set- ting in Figure 4 and in Table 7 of the Appendix. We observe that across all reading measures and psycholinguistic predictor variables, SCANDL ex- hibits effects that are most consistent with the hu- man data. On the one hand, the qualitative pat- tern, i.e., the sign of the estimates, for SCANDL- scanpaths is identical to the pattern observed in the human data, outperforming not only previous state- of-the-art (SOTA) models such Eyettention, but even the two arguably most important cognitive models of eye movements in reading, E-Z reader and SWIFT, On the other hand, also the quantita- tive pattern, i.c., the estimated size of each of the effects, exhibited by ScanDL. are most similar to the ones observed in humans.
SCANDL (original)
0,515 + 0.014
Cosine
0.514 - 0.018
Linear
0.519 1 0.020
W/o condition
0,667 + 0.015
W/o EMB and EMB
0.968 + 0.002
Table 2: Ablation study. We report NL. standard emror for 5-fold cross-validation in the New Reader/New Sentence setting.
(Ho et al., 2020) and the cosine (Nichol and Dhari- wal, 2021) noise schedules (for their definition, Appendix C.2). All ablation cases are conducted in the New Reader/New Sentence setting.
Results. As shown in Table 2, omitting all em- beddings except for the word index embedding results in a significant performance drop, as well as training SCANDL on unconditional scanpath generation. Moreover, changing the sqrt to lin- ear noise schedule does not enhance performance, while there is slight increase in performance for the cosine noise schedule. However, this perfor- mance difference between the sqrt and the cosine schedule statistically not significant.
6 Investigation of Model Behavior
6.1 Psycholinguistic Analysis
We further assess SCANDL's ability exhibit human-like gaze behavior by investigating psy- cholinguistic phenomena observed humans. We compare the effect estimates of three well- established psycholinguistic predictors - word length, surprisal and lexical frequency effects - on range of commonly analyzed reading measures first-pass regression rate (FPR), skipping rate (SR), first-pass fixation counts (FFC) and total fixa- tion counts (TFC)- - between human scanpaths on the one hand and synthetic scanpaths generated by SCANDL and our reference methods on the other 3p = 0.68 in a paired r-test.
6.2 Emulation of Reading Pattern Variability
To assess whether SCANDL is able emulate the different reading patterns and their variability typ- ical for human data, we compare the true and the predicted scanpaths in the New Reader setting of both datasets with respect to the mean and standard deviation of range of commonly analyzed read- ing measures: regression rate, normalized fixation
"Full results plots, including the (psycholinguissiiclly ir- relevant) Uniform and Train-label-diss baselines, can be found Fig. 8 the Appendix.
*For more details on the reading measures, computation of the predictors and model specification, see Appendix D.

---

Human
ScanDL
Eyattontion
with this reader's mean NLD. The analysis sug- gests that SCANDL more easily predicts patterns of readers with shorter seanpaths and low propor- tion of regressions. Exact numbers are displayed in Table 10 in the Appendix.
Model
EZ-reader
SWIFT
skipping
frrst
0.25
0.50
0.00
0.25
-0.25
Posterior effect estimate
0.25
6.3 Qualitative Differences Between Models
0.75
+0.50
To investigate whether SCANDL and the baseline models exhibit the same qualitative pattern in their predictive performance, that is, whether they gen- erate good (or poor) predictions the same sen- tences, we compute the correlations between the mean sentence NLDs of SCANDL. and each of the reference models. The correlations are all signifi- cant, ranging from 0.36 (SWIFT) to 0.41 (Eyetten- tion), though neither very strong nor wide-spread (see Table 11 in the Appendix). For a more detailed inspection of each model's ability to predict certain reading patterns (scanpath properties), we compute the correlations between the ahove-introduced read- ing measures (see Section 6.2) of true scanpath and the NLD of its predicted counterpart for all models, in the New Reader/New Sentence setting. Results. All models exhibit a significant positive correlation between the NLD and both the regres- sion rate and the normalized fixation count (see Ta- ble 12 in the Appendix): long scanpaths with many regressions and high relative number of fixations result in predictions with higher NLD (i.e., are more difficult). Further, models of the same type, that is the two ML--based models SCANDL and Eyettention on the one hand, and the two cognitive models E-Z reader and SWIFT on the other hand, exhibit very similar patterns. Overall, the cogni- tive models have much stronger correlations than the ML-based models. In particular, E-Z reader and SWIFT exhibit large positive correlation for first-pass counts, meaning that they struggle with scanpaths with a high number of first-pass fixations on the words, while both SCANDL and Eyettention have correlations close to zero indicating that they cope equally well for scanpaths with high or low first-pass counts. In sum, while the cognitive mod- els appear to be overfitting to specific properties of a scanpath, SCANDL and Eyettention seem to generalize better across patterns.
1.00
-0755
total fnation oount
trsh-pass fxatton oount
0.2
0.1
0.1
0.0
-0.1
-0.1
0.2
lastcal froquency
suppical
word length
taxical Ireqquency
susprisal
woid langth
Psycholinguistic phenomenon
Figure 4: Comparison of posterior effect estimates for psycholinguistic phenomena on reading measures be- tween original and predicted scanpaths. Lines represent a 95% credible interval, means are denoted by dots.
count, progressive and regressive saccade length, skipping rate, and first-pass count (see Table 8 in the Appendix for a detailed definition).
Results. As depicted in Figure 5 and Table 9 in the Appendix, the scanpaths generated by SCANDL are similar in diversity to the true scanpaths for both datasets. Not only is SCANDL's mean value of each reading measure close to the true data, but, crucially, the model also reproduces the variability in the scanpaths: for instance, in both datasets, the variability big in both the true and the predicted scanpaths for regressive saccades, and is small in both true and predicted scanpaths with regards to progressive saccade lengths and first-pass counts.
Datiunt
Gale
18-
Iodses Ae mdd
mgo
Figure 5: Reading measures of true and predicted scan- paths by SCANDL of the CELER and ZuCo datasets.
6.4 Investigation of the Decoding Progress
We also inspect the model's ability to approxi- mate reader-specific patterns. To this end, we aver- age the above analyzed reading measures over all scanpaths of a reader, and compute the correlation
We further examine the denoising process of SCANDL to determine at which step the Gaussian noise is shaped into the embeddings representing the word IDs of the scanpath. Figure 6 depicts three

---

[-SNE plots (Hinton and Roweis, 2002) different steps of the decoding process. Only during the last 200 denoising steps do we see an alignment be- tween the words in the sentence and the predicted fixations on words in the scanpath, and a clear sep- aration between these predicted fixations and the predicted PAD tokens. At the very last denoising step, all PAD tokens mapped to small number of spatial representations.
main challenge (Reich al., 2022). Further- more, SCANDL's SOTA performance in the Across- Datasel setting, attaining parity with the human baseline, corroborates the model's generalizabil- ity. This generalizability is further underlined by the model emulating the variability in reading pat- terns observed in the human data, even when being evaluated across-datasct.
The omission of the positional embedding and the pre-trained BERT embedding in the ablation study highlights their importance the fact that discarding them yiclds a worse performance than omitting the sentence condition, in which case the model still receives the positional embedding, stresses the importance of sequential information, which is lost to transformer model if not explic- itly provided. Moreover, removing the sentence- condition also emphasizes the importance of the linguistic information contained in the sentence. Overall, the ablation study emphasizes the impor- tance of modeling scanpath prediction as a dual- nature problem.
Denoising timestep
100
Denoising timestep 1990
L-SNE component
Denolsing timestep
100
50
In contrast to cogmitive models, SCANDL was not designed to exhibit the same psycholinguistic phenomena human readers. However, psy- cholinguistic analysis demonstrates that SCANDL nevertheless captures the key phenomena observed in human readers which psycholinguistic theories build on. Even more, in contrast to the cognitive models, it appears to overfit less to certain read- ing patterns. These findings not only emphasize the high quality of the generated data, but also open the possibility to use SCANDL when design- ing psycholinguistic experiments: the experimental stimuli can be piloted by means of simulations with SCANDL to potentially detect unexpected patterns or confounds and address them before conducting the actual experiment with human readers. Fur- ther, we can use SCANDL. for human-centric NLG evaluations using synthetic scanpaths.
Scanooth
-100
-50
50
100
t. ENE component
Figure 6: t-SNE plots of the continuous model output ū; different steps of the 2000-step denoising process. Step 1999 refers to the last iteration (all noise removed).
7 Discussion
The experimental results show that our model estab- lishes a new state-of-the-art generating human- like synthetic eye movement patterns across all investigated evaluation scenarios, including within- and uctoss-dataset settings. Indeed, the similarity of the model's synthetic scanpaths to human scan- path even exceeds the similarity between two scan- paths generated by two different human readers on the same stimulus. This indicates that the model learns to mimic an average human reader, abstract- ing away from reader- speciffc idiosyncrasies." 6 In- terestingly, the model's performance is better in the New Reader than in both the New Sentence and New Reader/New Sentence setting, which stands in stark contrast to previous research which iden- tified the generalization to novel readers the
8 Conclusion
We have introduced a new state-of-the-art model for scanpath generation called SCANDL. It not only resolves the two major bottlenecks for cognitively enhanced and interpretable NLP, data scarcity and unavailability at inference time, but also promises to be valuable for psycholinguistics by producing high quality human- scanpaths. Further, we have extended the application of diffusion models to discrete sequence-to-ser to-sequence problems.
"This is further supported by the NLD not changing when computed as average NL.D hetween a SCANDI. scanpath and all human scanpaths on the same sentence in the test set.

---

Limitations
Finally, we would like emphasize that, al- though our model is able capture psycholinguis- tic key phenomena of human sentence processing, is nor a cognitive model and hence does not claim in any way that its generative process simulates or resembles the mechanisms underlying eye move- ment control in humans.
Eye movement patterns in reading exhibit a high de- gree of individual differences between readers (Ku- perman and Van Dyke, 2011; Jäger et al., 2020; Haller et al., 2022a, 2023). For generative model of scanpaths in reading, this brings about trade- off between group-level predictions and predictions accounting for between-reader variability. The fact that SCANDL. outperforms the human baseline in terms of NLD indicates that it learns to emulate an average reader. Whereas this might be the desired behavior for range of use case scenarios, it also means that the model is not able concomitantly predict the idiosyncrasies of specific readers. We plan to address this limitation future work by adding readcr-specific information to the model.
Ethics Statement
Working with human data requires careful ethical consideration. The eye-tracking corpora used for training and testing follow ethical standards and have been approved by the responsible ethics com- mittee. However, in recent years it has been shown that eye movements are a behavioral biometric char- acteristic that can be used for user identification, po- tentially violating the right of privacy (Jäger et al., 2020; Lohr and Komogortsev, 2022). The pre- sented approach using synthetic data at deploy- ment time considerably reduces the risk of poten- tial privacy violation, as synthetic eye movements do not allow to draw any conclusions about the reader's identity. Moreover, the model's capabil- ity to generate high-quality human-like scanpaths reduces the need to carry out eye-tracking experi- ments with humans across research fields and ap- plications beyond the use case of gaze-augmented NLP. Another possible advantage of our approach is that by leveraging synthetic data, we can over- come limitations associated with the availability and representativeness of real-world data, enabling the development of more equitable and unbiased models.
Relatedly, since our model has been trained and evaluated on natural reading task, it remains un- clear as to what extent it generalizes to task-specific datasets, which arguably might provide more in- formative scanpaths for the corresponding NLP downstream task. As for the reader-specific exten- sion of the model, this issue might be addressed by adding the task as an additional input condition.
On a more technical note, a major limitation of the presented model is its relatively high computa- tional complexity in terms of run time and memory at inference time (see Section A.2 in the Appendix). Moreover, the metric used for model evaluation, the Normalized Levensthein Distance, might not be the ideal metric for evaluating scanpaths. Other metries that have been used to measure scanpath similarity - MultiMatch (Jarodzka et al., 2010) and ScanMatch (Cristino et al., 2010) - have been questioned in terms of their validity in a recent study (Kümmerer and Bethge, 2021); both metrics have systematically scored incorrect models higher than ground-truth models. A better candidate to use in place of the Normalized Levenshtein Dis- tance might be the similarity score introduced by von der Malsburg et al. (2015), which has been shown to be sensitive to subtle differences between scanpaths on sentences that are generally deemed simple. However, the main advantage of this met- ric is that it takes into account fixation durations, which SCANDL, in its current version, is unable to predict.
In order to train downstream tasks using gaze data, substantial amount of training data is typically required. How- ever, the availability of such data has been limited thus far. The utilization of synthetic data offers promising solution by enabling the training of more robust models for various downstream tasks using gaze data. Nonetheless, the adoption of this approach raises important ethical considerations. as introduces the potential for training models that can be employed across wide range of tasks, including those that may be exploited for nefar- ious purposes. Consequently, there exists a risk that our model could be utilized for tasks that are intentionally performed in bad faith.
models for
This inability concomitantly predict fixation durations together with the fixation positions is another shortcoming of our model. However, we aim to tackle this problem in future work.

---

Acknowledgements
Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186. Association for Computational Linguis- tics.
This work was partially funded by the German
Federal Ministry of Education and Research un- der grant 01| S20043 and the Swiss National Sci- ence Foundation under grant 212276, and is sup- ported by COST Action MultiplEYF, CA21131, supported by COST (European Cooperation in Sci- ence and Technology).
Ralf Engbert, Antje Nuthmann, Eike M. Richter, and Reinhold Kliegl. 2005. SWIFT: A dynamical model of saccade generation during reading. Psychological Review, 112(4):777.
Felix Engelmann, Shravan Vasishth, Ralf Engbert, and Reinhold Kliegl. 2013. A framework for modeling the interaction of syntactic processing and eye move- ment control. Topics in Cognitive Science, 5(3):452- 474.
References
Shansan Gong. Mukai Li. Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2023. Diffuseq: Sequence to se- quence text generation with diffusion models. arXiv preprint arXiv:2210.088 10.0993.
Maria Barrett, Ana Valeria -ardulo Lea Fr- ermann, and Anders Søgaard. 2018. Unsupervised induction of linguistic categories with records of read- ing. speaking, and writing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:: Hu- man Language Technologies, Volume 1 (Long Pa- pers). pages tional Linguistics.
Michael Hahn and Frank Keller. 2016. Modeling human reading with neural attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 85-95. Association for Computational Linguistics.
Association for Computa-
Michael Hahn and Frank Keller. 2023. Modeling task effects in human reading with neural network-based attention. Cogmition, 230: 105289.
Yevgeni Berzak. Boris Katz, and Roger Levy. 2018. Assessing language proficiency from eye movements in reading. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 1986- 1996. Association for Computational Linguistics. Yevgeni Berzak, Chie Nakamura, Amelia Smith, Emily Weng, Boris Katz, Suzanne Flynn, and Roger Levy. 2022. CELER: A 365-Participant corpus of eye movements in L1 and L2 English reading. Open
Patrick Haller, Iva Koncic, David R. Reich, and Lena A. Jäger. 2023. Measurement reliability of individ- ual differences in sentence processing: A cross- methodological reading corpus and Bayesian analy- sis. preprint, PsyArXiv.
Patrick Haller, Iva Koncic, David R. Reich, Chiara Tschirner, Silvia Makowski, and Lena A. Jäger. 2022a. Measurement reliability of individual dif- ferences in sentence processing. In Archiectturs and Mechanisms of Language Processing 2022 (AMLaP 2022).
Mind, pages 1-10.
Paul-Christian Bürkner. 2017. brms: An R package for Bayesian multilevel models using Stan. Journal of Statistical Software, 80(1):1-28.
Patrick Haller, Andreas Säuberli, Sarah Kiener, Jinger Pan, Ming Yan, and Lena Jäger. 2022b. Eye-tracking based classification of Mandarin Chinese readers with and without dyslexia using neural sequence mod- els. In Proceedings of the Workshop on Text Simplifi- cation, Accessibility, and Readability (TSAR- 2022). pages 111-118. Association for Computational Lin- guistics.
Filipe Cristino. Sebastiaan Mathôt, Jan Theeuwes, and
lain D Gilchrist. 2010. Scanmateh: A novel method for comparing fixation sequences. Behavior Re- search Methods, 42(3):692-700
Imre Csiszar. 1975 I I-Divergence geometry of proba-
bility distributions and minimization problems. The Annals of Probability, 3(1): 158.
Geoffrey E. Hinton and Sam Roweis. 2002. Stochastic neighbor embedding. In Advances in Neural Infor- mation Processing Syxtems, volume 15. MIT Press. Jonathan Ho. Ajay Jain, and Pieter Abbeel. 2020. De- noising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840- 6851.
Shuwen Deng, Paul Prasse, David R. Reich, Tobias Scheffer, and Lena A. Jäger. 2023a. Pre-trained lan- guage models augmented with synthetic scanpaths for natural language understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Com- putational Linguistics.
Nora Hollenstein, Itziar Gonzalez- Dios, Lisa Beinborn, and Lena Jäger. 2022. Patterns text readability in human and predicted eye movements. In Proceed- ings of the Workshop Cognitive Aspects of the Lexicon, pages 1-15. Association for Computational Linguistics.
Shuwen Deng, David R. Reich, Paul Prasse, Patrick Haller, Tobias Schelfer, and Lena A. Jäger. 2023b. sequene model for predicting human scanpaths during read- ing. Proceedings of the ACM on Human- Compute Interaction, 7.
Eyettention: An aitention-based
Nora Hollenstein, Federico Pirovano, Ce Zhang, Lena Jäger, and Lisa Beinborn. 2021. Multilingual lan- guage models predict human reading behavior. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies,
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for

---

pages 106-123, Association for Computational Lin- guistics.
where to look: Modeling eye movements in read- ing. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, pages 93-101.
Nora Hollenstein, Jonathan Rotsztejn, Marius Troen- dle, Andreas Pedroni, Ce Zhang, and Nicolas Langer.
2018. ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading. Scientific Datu, 5:180211.
Mattias Nilsson and Joakim Nivre. 2011. Entropy- driven evaluation of models of eye movement control in reading. In Proceedings of the 8th International NLPCS Workshop, pages 201-212.
Nora Hollenstein and Ce Zhang. 2019. Entity recog- nition at first sight: Improving NER with eye move- ment information. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistios: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers). pages 1-10. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vitito Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Process- ing Systems, volume 32, pages 8024 8835. Curran Associates, Inc.
Lena A. Jäger, Silvia Makowski, Paul Prasse, Sascha Liehr, Maximilian Seidler, and Tobias Scheffer. 2020. Deep Eyedentification: Biometric identification us- ing micro- morrommtveeets of the eye. In Machine Learn- ing and Knowledge Discovery in Databases. ECML PKDD 2019, volume 11907 of Lecture Notes in Com- puter Science, pages 299-314 14. Springer International Publishing.
Peter Raatikainen, Jarkko Hautala, Otto Loberg, Tommi Kärkkäinen, Paavo Leppänen, and Paavo Nieminen. 2021. Detection of developmental dyslexia with machine learning using eye movement data. Array,
Halszka Jarodzka, Kenneth Holmqvist, and Marcus Nys- tröm. 2010. A vector-based, multidimensional scan- path similarity measure. In Proceedings of the 2010 Symposium on Eye-Tracking Research and Applica- tions, pages 211 1-2218.
12:100087.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Matthias Kümmerer and Matthias Bethge. 2021. State- oh-the-art in human scanpath prediction. arXiv preprint arXiv:2102. 12239.
Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psy- chological Bulletin, 124(3):37--422.
Victor Kuperman and Julie A. Van Dyke. 2011. Er- fects of individual differences in verbal skills on eye- movement patterns during sentence reading. Journal of Memory and Language, 65(1):22-73.
Keith Rayner. 2009. The 35th Sir Frederick Bartlett Lec- ture: Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62(8):1457-1500
Vladimir Levenshtein. 1965. Leveinshtein distance. Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S. Liang, and Tatsunori B Hashimoto. 2022. Diffusion- LM improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328- 4343.
David R. Reich, Paul Prasse, Chiara Tschirner, Patrick Haller, Frank Goldhammer, and Lena A. Jäger. 2022. Inferring native and non-native human reading com- prehension and subjective text difficulty from scan- paths in reading. In Proceedings of the 2022 ACM Symposium on Eye-Trucking Research and Applica- tions. Association for Computing Machinery. Erik D. Reichle, Keith Rayner, and Alexander Pollatsek. 2003. The B-Z reader model of eye-movement con- trol in reading: Comparisons to other models. The Behavioral and Brain Sciences, 26:445-526.
Dillon Lohr and Oleg V. Komogortsev. 2022. Eye Know You Too: Toward viable end-to-end eye movement biometrics for user authentication. IEEE Transac- tions on Information Forensics and Security, 17:3151- 3164.
Tlya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Machel Reid, Vincent J. Hellendoorn, and Graham Neu- big. 2022. Diffuser: Discrete diffusion via edit-based reconstruction. Xrviv preprint Xrx:v:22 16886. Jascha Sohh-Diikstein, Eric Weiss, Niru Mah- eswaranathan, and Surya Ganguli. 2015. Deep un- supervised learning using nonequilibrium thermody- namics. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 2256-2265. Proceedings of Machine Learning Re- search.
Calvin Luo. 2022. Understanding diffusion mod- cls: A unified perspective. arXiv preprint arXiv:2208.1 11970.
Danny Merkx and Stefan I.. Frank. 2021. Human sen- tence processing: Recurrence or attention? In Pro- ceedings of the Workshop on Cognitive Modeling and Computationnll Linguistics, pages 12-22. Association for Computational Linguistics.
Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast WordPiece tokenization. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2089-2103 Association for Computational Linguis- tics.
Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-811 8171. Proceedings of Machine Learning Research.
Mattias Nilsson and Joakim Nivre 2009. Learning
Ekta Sood, Simon Tannert, Diego Frassinelli, Andreas

---

Bulling, and Ngoe Thang Vu. 2020a. Interpreting attention models with human visual attention in ma- chine reading comprehension. In Proceedings of the 24th Conference on Computational Natural Lan- guage Learning, pages 12-25. Association for Com- putational Linguisties.
Ekta Sood, Simon Tannert, Philipp Mueller, and An- dreas Bulling. 2020b. Improving natural language processing tasks with human gaze-guided neural at- tention. In Advances Neural Information Process- ing Systems, volume 33, pages 6327-6341. Curran Associates, Inc.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Process- ing Systems, 30.
Titus von der Malsburg, Reinhold Kliegl, and Shravan Vasishth. 2015. Determinants scampath regularity reading. Cognítive Science, 3977):1675-1703,
Xiaoming Wang, Xinbo Zhao, Jinchang Ren, and Jun- gong Han. 2019. A type of eye movement model based on recurrent neural networks for simulating the gaze behavior of human reading. Complex., 2019.

---

Appendix for SCANDL: A Diffusion Model for Generating Synthetic Scanpaths on Texts
A Parameters and Hyperparameters
A.1 Diffusion-specilic Hyperparameters
Hyperparameters concerning SCANDL or the noising process include the number of encoder blocks, the number of attention heads in each attention layer, the hidden dimension of the transformer, the noise schedule, the schedule sampling, the number of diffusion steps, and the input sequence length. We set the number of diffusion steps to be T 2, 000, the input sequence length to 128, and opt for importance sampling (Nichol and Dhariwal, 2021) concerns the schedule sampling. The other hyperparameters are determined during hyperparameter tuning; the search space as well as the best values can be found in Table 3.
Concerinng the noise schedules specifically, we list and define the ones we investigate below: 1-1
linear noise schedule: 33 = 10 * (1 - i) + 0.02 i, with = 7-1'
sqri noise schedule: Bt =
where s is a constant corresponding to the starting noise level,
which wc set to s = 0.0001,
cosine noise schedule: B
f(t) /ay
with f(t) cos +0.00
1.008
truncated cosine noise schedule: 32 f(0)* with ((t) cos(
f(t)
t/ t/T+0.008
2
1.088
truncated linear noise schedule: Br (10 + 0.01) (1 (1- + (0.02 + 0.01) with i -
We uniformly sample 60 model configurations that evaluated in triple cross-validation manner in the New Reader/New Sentence setting, and employ the normalized Levenshtein Distance (Levenshtein, 1965) as selection criterion.
Parameter
Search space
Best value
Numbber encoder blocks Number of attention heads
[2, 12, 16) (2, 12, 16)
12
Hidden dimension
[128, 256, 512, 768)
256
Noise schedule
[linear, aqrt, cosine, truncated cosine, truncated linear)
sqrt
Table 3: Parameters and search space used for hyperparameter tuning.
A.2 Training-specific Hyperparameters
We train SCANDL for 80,000 learning steps parallel manner over NVIDIA GeForceRTX 3090 GPUs using the A optimizer (Loshchilov and Hutter, 2019), a learning rate of le-4, and a batch size of 64.
A.3 Training Time, Inference Time, Parameters
SCANDL consists of about 130 million trainable parameters?, with an exception for the ablation case of SCcNNDL without EMBbeN and EMB and the ablation without the sentence condition, which both consist of about 22 million trainable parameters. Training the above-mentioned configuration of SCANDL for 80,000 learning steps takes about hours. The inference/decoding done on one NVIDIA GeForceRTX 3090 GPU and takes about 75 minutes for 2200 sentences, which is around seconds per sentence.
B Datasets
As regards reader-specific demographic information in CELER (Berzak et al., 2022), there are 69 L1 English speakers in CELER, of which 38 are female and 28 are male, one identifying as "other" and two
1130,855,296 parameters

---

not divulging their gender. The mean age is 26.3 + 6.7 years. Participants were recruited via a variety of sources, such as mailing lists, advertisements on social media, or message boards. With respect ZuCo (Hollenstein al., 2018), there are 12 participants, of which 5 are female and male, all native English speakers between the ages of 25 and 51, with mean age of 35 9.8 years.
Descriptive statistics for the two corpora used in our experiments can be found in Table 4. Descriptive
statistics on mean reading measures values in the LI part of CELER Berzak et al. (2022) can be found in Table 5
Unique sentences
Words per sentence
Dataset
Eye-tracker (sampling fire..)
Readers
CELER LI (Berzak al., 2022) ZaCo NR (Hollenstein et al., 2018)
EyeLink (1000 Hz)
5456
11.2 1
EyeLink 1000 Plus (500 Hz)
700
19.6 +: 9.8
Table 4:
Descriptive statistics of the two eye-tracking corpora used for model training and evaluation, CELER (Berzak et al., 2022) and ZuCo (Hollenstein al., 2018). The number of words per sentence is ported using the mean + standard deviation.
Measure Number fixations per word Saccade length chars
Value -
=
Skip rate
0.36 0.02
Regression rate
0.24 = 0.01
Table 5: Mean eye movement measures with 95 % confidence intervals of the I.I speakers in CELER Berzak et al. (2022).
Implementation Details
C.1 Embedding Figure
Figure shows the embedding function (MB(x) (+): NM+N+4 + R(M+N+4)xd that maps from the discrete imput representation into continuous space, where d the size of the hidden dimension. This embedding learns a joint representation of the subword-tokenized sentence xw and the fixation sequence xf. More precisely, the embedding function EMB(x) = EM EMBben (Xber) + EMB pos (Xpos) is the sum of three independent embedding layers.
WCLS
1
WSEP
fels
fi
fN
fser
EMB (Xide)
EM (ong)
RMD (Mspor
EMiB
EMil (f1)
EMB
+
+
EMB (XMer))
EMtl xrl
EMIl
+
EMB pas (xos)
BMM (wegss
EMBparlong
EMBByol//zzs) pMppp/
EMByosljje
EMB(x)
RMBincusl
EMB((n))
(MB())))
EMBCfous
BMB(1)
EMB((N)
EMBU/SeR
Figure 7: Embeddings of the discrete input representations.
C.2 Ablation: Noise Schedules
For the ablation study, compare the sqrt noise schedule (Li al., 2022) with the linear noise schedule (Ho et al., 2020) and the cosine noise schedule (Nichol and Dhariwal, 2021). The definitions of these noise schedules can be found in Table

---

Noise schedule Definition
Sqrt
B Tis with 0.0001
Linear
3. 10-4 1) 0.02 with
(-1
f
Cosine
B f(D)
f(() with ft) cos 4/7 0.008 1.008
T )
Table 6: Noise schedules used for SCANDL and the ablation study.
D Psycholinguistic Analysis
D.1 Reading Measures and Predictors
D.1.1 Reading Measures
The reading measures used for the psycholinguistic analysis are defined as follows:
first-pass regression rate (FPR; binary); if regression was initiated for a given word when visiting
the first time, else 0.
Skipping rate (SR; binary): 1 if the word was skipped when visited for the first time.
first-pass fixation counts (FFC): Number of consecutive fixations on word when visiting it the first time.
total fixation counts (TFC): Total number of fixations given word.
D.1.2 Psycholinguistic Predictors
The psycholinguistic predictors were computed as follows:
Lexical frequency: Frequencies were extracted using the python library wordfreq:
https://gi thub. com//ssperr/worfffe
Suprisal: For each token wiyk in a given sentence j. surprisal is defined as log p(uyk wj<k). We extracted surprisal values from the GPT-2 language model gpt-xl provided by Hugging Face (Rad- ford et al., 2019).
D.2 Model specification
We model the reading measures y using Bayesian generalized linear models logistic regression models for the binary reading measures (skipping rate and first-pass regression rate), Poisson regression models for the count-based reading measures (total fixation counts and first-pass fixation counts). The predictors are word lenght (wl), lexical frequeny (freq) and surprisal (surp). For the human data, we include random intercepts for subjects.
Viy - 9 (30 + Bos + 31 wly + 922 freq + f3 surp,)
(3)
where Vij refers the reading measure of subject i for the jth word in a given sentence. 8 represents the global intercept, and Bix the random intercept for subject i. g(-) denotes the linking function: g(=) = In -ي for the binary measures, with Ytj following a Bernoulli distribution; and g(=) log for the count-based measures, with Yiy following a Poisson distribution. For the predicted data, we the same model, but without by-subject random intercepts. As priors, we use the brms standard priors.
D.3 Full Results
In addition to the figures in the main part, Fig. 8 contains shows posterior effect estimates for all baselines. Summary statistics (mean and Bayesian 95%-credible intervals) of the effect estimate posterior distributions in the New Reader/New Sentence setting can be found in Table 7.

---

Measure
Model
Mean
95%
Human
0.07
00.05, 0.08]
ScanDL
0.21
0.18, 0.24
Eyettention
-..3
-0.06,- -0.01]
E-Z reader
0.08
0.11,-0.05
SWIFT
0.07
0.10, 00.04
Train-label-dist
0.04
0.07, 0.01
Human
0.11
0.13, -0.09
ScanDL
- 0.26
0.31.-0.19
Eyettention
- 0.05
-0.09,-0.01
FPR
E-Z reader
0.09
-0.02.0.19
lexical frequency
SWIFT
0.16
-0.29,
Train-label-ding
0.11
10.16-0.00
Uniform
0.03
0..11,0.06
Human
-0.1
14.3,-0.11
ScanDL
0.20
0.21, 0.118
Eyettention
0.02
-0.03. 0.01
TEC
E-Z reader
0.03
0.01,0.00
SWIFT
0.03
0.01
Train-label-dist
0.03
0.02, 0.05
Uniform
0.19
0.22]
Human
0.07
0.08, -0.06]
ScanDL
0.17
-.11,
Eyettention
0.01
0..03,0.00
FFC
B-Z reader
0.03
0.01, 0.05)
SWIFT
0.03
0.01,
Train-label-dist
0.04
[0.02, 0.06
Uniform
0.17
0..14,
Human
-023
-0.24-0.221
ScanDL
-0225
-0.99,-023
Eyettention
-0.11
-0.14.
SR
E-Z reader
0.07,-0.01
SWIFT
0.02
0.05,
Train-label-dist
0.18,
Uniform
0.26
-.30.-0.22
Human
00.4
0.06. -0002
ScanDL.
0.10
0.14,-0.05
Eyettention
-0.1
-0.15,
EZ reader
0.39
-..53,-000
SWIFT
0.55
-0.71,
surprisat
Train-label-dist
0.04
0.009
Uniform
0.08
0.00
Human
0.09, 0.10
ScanDL
0.08
0.06,, 0.09
Eycttention
0.03
(0.01, 0.044
TFC
E-Z reader
0.01
-0.03, 0.01]
SWIFT
- 0.03
-0.05, -0011]
Train-label
0.13
0. 0.14)
Uniform
0.36
[0.34, 0.39)
Human
0.08
[0.07, 0.08)
ScanDL
0.08
0.06, 0.09
Eyettention
0.04
0.02, 0.05
FFC
F-Z reader
0.01
-0.03, 0.01]
SWIFT
0.03
0.05, -0.01]
-rain-label-dist
0.11
00.09.0 0.
Uniform
0.33
0..30, 0.36

---

Measure
Model
Mean
95% CT
Human
00.5
0.97, - 00.93
SCANDL
0.64
0.68,
0.60
Eyettention
0.34
-0.37 0
SR
B-Z reader
0.23
0.26, 0.20
SWIFT
-0.99
-0.23, 00.16
Train-label-dist
0.08
0.05, 0.11
Uniform
0.13
[0.08, 0.18
Human
0.35
(0.34, 0.37
SCANDL
00.36, 0.45
Eyettention
0.23
00 0.26
FPR
E-Z reader
00.28, 0.50
wond length
SWIFT
0.00, 0.24
Train-label-dist
-0.02
-0.07, 0.03
Uniform
0.10
0.02, 0.18)
Human
0.24
0.24, 0.25
SCANDL.
0.18
0.177 0.20
Eyettention
0.10
0.008, 0.11
E-Z reader
0.13
0.15
SWIFT
0.14
0.12,
Train-label-dist
-0.88
-0.09, -0.06
Uniform
0.25
1.2.009.0000 0.22
Human
0.22)
SCANDL
0.16
0.14,0.17
Eyettention
0.09
0..07, 0.10
FFC
E-Z reader
0.13
0.111, 0.15
SWIFT
0.14
0.
Train-label-dist
0.07
-0.09, 0.05
Uniform
0.23
-0.27,-0.0 -0.19
Table 7: Posterior distributions of effect sizes for three psycholinguistic predictors lexical frequency, surprisal and word length-in the New Reader/New Senience setting. Estimates were computed via Bayesian generalized linear models. We report means and 95%-creiibl intervals [2.5%; 97.5% for three psycholinguistic effects on four reading measures extracted from the fixation sequences.

---

Human
ScanDL
Eyettention
EZ-reader
Model
SWIFT
Train-label-dist
Uniform
skipping rate
first-pass regression rate
0.25
0.50
0.00
0.25
-0.25
$1
0.00
-0.50
-0.25
Posterior effect estimate
-0.75
-0.50
-1.00
-0.75
fixation count
first- pass fixation count
0.4
0.2
0.2
H1
H13
0.0
0.0
-0.2
-0.2
lexical frequency
surprisal
word length
lexical frequency
surprisal
word length
Psycholinguistic phenomenon
Figure 8: Comparison of posterior effect estimates for psycholinguistic phenomena on reading measures between original and predicted scanpaths. Lines represent 95% credible interval, means are denoted dots.

---

E
Emulation of Reading Pattern Variability
Reading measure
Definition
Regression rate
The average probahility of a word the sentence the starting point of regression The average number of fixations in sentence
Normalized fixation count
Progressive saccade length
The average length progressive saccades
Regressive saccade length
The average length regressive saccades
Skipping rate
The average probahility of a word to skipped
First-pass count
The average number of fixations on a word during the first first-pass reading of the sentence
Table 8: Definition of reading measures used for the investigation of the reading pattern predictability across readers and the predictability patterns across models.
Reading measure
CELER
ZuCo
(true)
mean sd (SCANDL.)
(trie)
mean + sd (SCANDI.)
First pass count
0.91 0.21
0.19
0.84 0.21
0.73
Normalized fixation count
1.24
0.30
1.15
0.34
Progr. saccade length
2.00
1.75 0.63
2.72 + 1.06
2.00 0.60
Regr. saccade length
2.55 2.16
1.82 + 2.24
4.44 3.58
2.44 + 3.05
Skipping rate
0.29 0.13
0.27 + 0.12
0.42 0.16
0.41 + 0.18
Regression rate
0.14 + 0.10
0.08 +0.07
0.15 0.09
0.07 + 0.07
Table 9: Mean and standard deviation of the reading measures of true and predicted scanpaths of both the CELER (Berzak et al., 2022) and ZuCo (Hollenstein al., 2018) datasets.
Reading measure
CELER
ZuCo
First-pass pass fixation count
.08
-39
Normalized fixation count
.52*
.07
Progr: saccade length
.88*
.77*
Regr. saccade length
.32**
.04
Skipping rate
.2
.70*
Regression rate
62
.75"
Table 10: Pearson correlations between reader's mean NLD and mean reading measures. Statistically significant correlations are marked with an asterisk. "**)p & 0.001,**) p S 0.01,*) p & 0.05.

---

Baseline
Pearson con coef.
Eyettention
Uniform
_1 **
Traindist
.39
E-Z Reader
-3
SWIFT
.36
Table II: Pearson correlations between the mean sentence NLDs of SCANDL and every baseline model. Statistically significant correlations are marked with an asterisk. "*'p S 0.001," p S 0.01," p S 0.05.
F Model-Specific Predictability Patterns
Model
Reading measure
Pearson
Normalized lixation count Regression
0.38
0.31
Progressive saccade length
0.31
SCANDL
Skipping rate
0.21
Regressive saccade length
0.22
First-pass fixation count
0.03
Progressive saccade length
0.31
Normalized fixation count
0.25
Regression
0.19
Eyettention
Skipping rate
0.18
Regressive saccade length
0.18
First-pass fixation count
0.02
Normalized fixation count
0.69
First-pass fixation count Regression rate
0.46
0.45
E-Z reader
Regressive saccade length Progressive saccade length
0.16
0.01
Skipping rate
0.2
Normalized fixation count First-puss fixation count Regression rate
0.71
0
0.47**
SWIFT
Regressive saccade length
0.16
Progressive saccade length
0.02
Skipping rate
0.24
Normalized fixation count Regression rate
0.3
0.22
Progressive saccade length
0.2
Traindist
Regressive saccade length
0.2*
First-pass fixation count
0.16
Skipping rate
0.01
Normalized fixation count
0.55
First-pass fixation count Regression rate
0.41
0.37**
Uniform
Regressive saccade length Progressive saccade length Skipping rate
0.23
0.03
-0.23***
Table 12: Pearson correlations between the NLD of predicted scanpath and reading measures of the corresponding true scanpath, for SCANDL and the baselines. Statistically significant correlations are marked with an asterisk. ***) < 0.001,**) p Σ 0.01,*) *) p S 0.05.

---

G Derivation of Objective
In this section we derive the component LVLB (Equation 1) of the model's training objective LSCANDL from the variational lower bound (Equation 2). This derivation closely follows the one given by Luo (2022). The diffusion posterior is defined
(ZiT) = II ((z 2t 21-
(4)
Each forward transition in the noising process follows first-order Markov property by being dependent only on the immediately preceding latent, and written as
glzllz 1) = =N(Vanz 1, - II)
(5)
The joint distribution of all latents of a diffusion model is then given by
I
p(zoT) p(zT) II po 1/)),
(6)
where p(zr) = ().
diffusion model can be optimized by maximizing the log- ligeliiel of the data log log p(zo), which is equivalent to maximizing the variational lower bound (VLB):

---

logp(zo) = log
p (ZorT) dzur
(7)
p (20.r) ) (21.1
(00) dæ1rr
log
(8)
q (Zi:T zn)
(Ze:r)
log Bogaar
(9)
4 (Z1rr %a))
y (Zo:T)
2 (y(ssrlAa))
log
(Jensen's Incquality)
(10)
9 (M11r %oo)
(2r) IT (z
zz)
log
Equations 4, 6
(11)
IT: y (zc xc-1)
log p (2x) po (2o 21) II.
po (Zc-1 1 21)
(12)
9 (z)
%o)
IT: 9 (Z+ z=11)
log Pe (2r) pe (zo
12)) + log II
T
po (zi-1 2e)
= Eggsesr/ (mo)
(13)
(2) zn)
@ (z:
2e-1 2a)
log p (xT) pe (xo
x11
+ log II
T
pe (xL-1 Xi)
(14)
9 (xx
xo)
lis < (m0)
jo(
log
(2z) po (2o
22)
+ log 2 (2) I 20)
+ log II
po (2--1 (c))
(15)
9 (z) zo0)
(zT zon)
y (Zr-1 Ze, 20)
1=2
log (2r) pe (zo 211)
pa (za-- 21)
Eq( (mn)
log
(16)
9 (Zr 20)
9 (2t-1
Ze, 2o)
log (22r 20)
(ZT)
T Egtar mo)
(2--1 2,)
= (ag [log po (z0 21)] + Evtas (mu)
log ② (2
(17)
Ze Bu)
t-2
Egtte no) [logee pe (8o [=1]]= + Eayiery (ma)
log (zr zo))
p p(rr
Σ 2=
B 20
log
po (21-1 2z)
(18)
9 (z.-1 Zz, 2o)
Egirs mo) po (zo 2x))
DEaL (er
kn) lle (2Tr)
(19)
reconntraation lorm
poir matching
Bo mo)
DiLt. (g (zi-1
Zz; zo) [|pa (Ze-1 zz
(20)
denobinng matchiing toan
The denoising matching term contains KL. Divergence terms (Csiszar, 1975) between the true denoising transition (z.-1 Zı. Zo) and the approximated transition (z,-1 z.). For our learned distribution pe to match the ground-truth distribution be minimized. However, the summation term has a high optimization cost, which is why make optimization tractable by leveraging the Gaussian transition via Bayes rule:
closely as possible, the denoising matching term has to
Z1, zn) 9 (Zz 2;-1, 2o) (Z--1 Zo) q (Z
(21)
q zo)
where g (z, 2, 1, 1,0) q (z:
and, applying the reparametrization trick, 2, = Vaizi-1+ + v1 - aye with e ~ (0, and 2,-1 = -ai-121-2 + VI- ai-ie with e
2, 141)= = N (Varz -1 a4)
II). The form of g (zr) 20) is derived by repeatedly applying the reparametrization trick. Let there
be 27 noise variables (4: c) T NN(0,)) I). A sample 2, (z; zn) can be re-written as:
iid

---

2: vaiz-1 + vI-
(22)
-Vai (Vai + 11-11--66 vI arei-1
(23)
- + Val - a10--10--1--- + v1 aiii-1
(24)
Vara+ 12, 2 V Vat 0tat + v añet-2 2
(25)
1010-121-2 V0000-12 12--2 + Vau a1ar-1 011412
(26)
= Vaya: 12, 2 + V1 a4a4 16--
(27)
(28)
II
a;Z0 + 1-
0:60
(29)
1=1
1=1
Vaizo + v1 - 0400
(30)
~N (Vōrzo, (1-at) II)
(31)
9 (2--1 Z1, zn) can now be obtained by substituting into the Bayes' expansion from Equation 21:

---

9 (z:
2e 1, k0) y (2 20)
9 (3 20)
(32)
9 (+t 20)
N (yarze-,, (1
(33)
N (vaizo, (1 [.
Z 1012-112
(2c-1 12)- 2(1 - -(--1)
(2c vo(2o)
exp
(34)
2(1 aa)
2(1 - (tc)
(zl v /
(Zt-1 - vay 120)
(z+ va
exp
(35)
at
1 at-1
Ete
1-2 0izi1b + aizi-1)
(zi-1 - -2/0n-110 126--111120)
exp
+
C (Zi, Zo)
(36)
1- at
āc-1
2 2a002i7--1
a:bi-1
2
2/ 1/1--161-100
X exp
(37)
2
1 a4
1- a2
1- ā2-1
64-1
exp
ai
6i-1 2
2
V017
v Vi 180
(38)
2
1-
1 1 al-1
1
a--1
exp
1
a. (1 ā 1) +)++1
ai 2
Va1zz
v (a-1160
21
(39)
2 (1-a.) (1 Ō)-1)
Oz
G/-1
1
ā + 11-
v
Vāt-120
exp
(1 - 0.)
Ō=-1)
-
a2
04-1
11
(40)
1
1 ā
Vasat
10:-12)
exp
2
(1 0z)
(1 01-1)
1-a a
11-1
(41)
Vaske
Vou
170
exp
1 a
6
1-91
1-71
21-1
(42)
2 (1 0z) (1 0=-1)
(1-mz)
(1)
a 120
(1 at) (1 - k(-1)
- 6z
1-nz
1-90-1
exp
2/-1 -2
2(-1
2 (1 az) (1 am-1)
1
(43)
z--1 2 (1 ae-1)2 21 + vāc- (1 cxe 80
exp
2 (1
1
71
(44)
1-m1
N1 far (1 at 1) 22
Vā- (1 ax) zo
(I ar) (1 - -
Ci
1 - 0e
(45)
Hig (g)
(1(
(46)
Therefore, each step 7t-1 q (Zt-1 21, 20) is normally distributed, with mean My (z,, 20) and variance (l).). Following Equation 45, the variance can be written (1),, where
(1 (1 - ore) (1 or-1) 1 at
(47)
As all a terms are given priori by the noise schedule, the variance can be immediately computed. The mean, however, must be parametrized, as a function of 2,, hence μ40 (2,, Since we choose the variances of the two distributions to match exactly, we can optimize the KL Divergence by minimizing the difference between the means of the two distributions:

---

argmin DKL (q (2r-1 22, 2o) lipo (2--1 kx))
(48)
argmin DEL (W (2:-1: (1)) | (me, (1)(
(49)
[t)
argmin
log
tr [2 (1) (me - na) Ey(1)) (No mn)
(50)
(t)
log 1 d+ d - (14 Ha) (t) (uo - na)
argmin
(51)
argmin (HO m²)² (t) (uo mn)
(52)
argmin (no n²)² (d)(t) -! (No ma)
(53)
argmin
(0) [llio Palli
(54)
This means that we want optimize μe (Zz, 1) that it matches My (zi, Zo). which is defined
Mg (Ze, 2o) vor (1 5g 1) z Vag 1 (1 (4)) Z
(55)
1 Ct
and μi () is defined as
Vai (1 ai-1
a)-1) Zz + Vou 1 (1 a4) 20 (2x.))
pe (Ze, 8)
(56)
āt
where ž (zz, t) is parametrized by our transformer model fo that is trained to predict zo from Zg. We henceforth denote &o (z,, t) by fo t), where fo is our transformer model and fo (zz, t) is the model prediction 2n for ground-truth Zo. The objective is then simplified to:
argmin DKL (g (Re-1 zt, 20) Ii (2--1 2z))
(57)
0
argmin DEL (V (Mg. Et) I| (no. (t))
(58)
1
Vae (1 -) 21 + van-1 fo (2.,t)
you (1 -4-1) nh + vo:-1 (1 0z)
(1
argmin
(t)
1 6
(59)
1
var-1 (1 - az) So (x),t)
Vai-i (1 - or) 20
= argmin (t)
(60)
1
(a-- (1 as)
argmin (l)
(Jo (2t,1) 20)
(61)
1 0o-1 (1 - au)
= argmin (t)
[llfo (2r,1) - 2oll
(62)
- a1)
This means that maximizing the VLB can be achieved by learning a neural network that predicts the ground truth from the noised version of the ground truth. In our case, we neglect the constant term and denote the part of our optimization criterion that stems from the VLB as
LV.. = 11.5o (Zt, 1) 20l12


Reviews:
[{"review_id": "i2GubaHaGf", "reviewer": "Reviewer_cEYa", "summary": "", "questions": "See above for questions.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer cEYa for their careful reading of the paper and helpful feedback and address their concerns as follows:\n<br>\n\n**1. The model learns scanpath predictions for an average reader. It is not clear *how diverse are the readers in the train set?* For eg. *how aligned are the scanpaths for readers who are on sides of the spectrum of the readers in the dataset?* Although the model is able to generalize to new readers, it is not clear *how different are the reading patterns of the readers* (for same sentence experiments) *in the train and test sets and to what extent the model is able to capture the differences*.**\n\nTo answer the *question of the diversity of readers in the train set*, we would like to first provide some demographic information: there are 69 L1 English speakers in CELER (Berzak et al., 2022), of which 38 are female and 28 are male. The mean age is 26.3 $\\pm$ 6.7 years. Berzak et al. (2022) have recruited the participants via a variety of sources, such as mailing lists, advertisements on social media, or message boards. We have now included these demographic statistics in Section B of the Appendix.\n\nFurther descriptive statistics taken from the original CELER publication on mean reading measure values can be seen in the table below. They describe the mean scanpath properties and are aggregated over readers and over sentences. We have also added this table to Section B of the Appendix.\n\n| Measure                      | Value           |\n|------------------------------|-----------------|\n| Number of fixations per word | 1.3 $\\pm$ 0.1   |\n| Saccade length in chars      | 8.5 $\\pm$ 0.4   |\n| Skip rate                    | 0.36 $\\pm$ 0.02 |\n| Regression rate              | 0.24 $\\pm$ 0.01 |\n\n\nHowever, the values in this table are aggregated over all L1 participants. To address both *the question of reader diversity* in more detail as well as *the question on how well ScanDL manages to capture the reading patterns of different readers, especially of readers that are towards the edge of the spectrum*, we have conducted an analysis to quantify the general reading behavior of each reader: we first computed a set of commonly used reading measures averaged over all scanpaths of a reader. The reading measures are: the average probability of a word in the sentence to be the starting point of a regression (regression rate), the average number of fixations in a sentence (normalized fixation count), the average lengths of progressive and regressive saccades (prog. saccade length, regr. saccade length), the average probability of a word to be skipped (skips), and the average number of fixations on a word during the first-pass reading of the sentence (first-pass count). We then computed the mean NLD for each reader between the reader's true scanpaths and the scanpaths predicted by the model, and then correlated the mean reader NLD with each per-reader mean reading measure. The reading measures and the correlations can be seen in the following table. Correlations greater than 0.3 are in bold font, statistically significant correlations are marked with an asterisk:\n\n\n\n\n| Reading measure           | Pearson corr. w. NLD |\n|---------------------------|----------------|\n| Regression rate           | **0.56*** |\n| Normalized fixation count | **0.48*** |\n| Progr. saccade length     | **0.45*** |\n| Regr. saccade length      | **0.30**  |\n| Skips                     | 0.26           |\n| First pass count          | 0.01           |\n\n\nThe  correlation of 0.56 between the NLD and a reader's average regression rate indicates that the model has more difficulties predicting the reading behavior of readers who show a large proportion of regressions than the reading patterns of readers that only regress rarely. Along the same lines, we observe a positive correlation between a reader's average normalized fixation count on a sentence and NLD: long scanpaths with many fixations (normalized by sentence length) are more difficult for the model to predict than short scanpaths. Looking at progressive and regressive saccades, the positive correlation between a reader's average saccade length and the mean NLD for that reader suggests that it is more difficult for the model to predict the reading behavior of readers that tend to make long saccades (both progressive and regressive).\n\nWhile the analysis above examines how well the model captures different reading patterns, we would also like to address reviewer cEYa's question on *the differences in reading patterns* in more detail. While we did perform 5-fold cross-validation to account for the diversity of readers between the train and test set and also performed an across-dataset evaluation to assess the model's ability to generalize to unseen readers and sentences, we have not specifically described the diversity within each set. To that end, for the above-mentioned reading measures, we computed the overall mean and standard deviation for both the predicted scanpaths and the true scanpaths for the CELER dataset (Berzak et al., 2022) so as to quantify the overall reading pattern variability and whether or not the model manages to emulate this variability. We are planning to perform the same analysis for the ZuCo dataset (Hollenstein et al., 2018) as well to be able to also compare how well the model manages to capture the diversity in a completely different dataset. Results can be see in the table below.\n\n\n\n\n| Reading measure           | mean $\\pm$ sd (true scanpaths)  | mean $\\pm$ sd   (predicted scanpaths)   |\n|---------------------------|-----------------|---------------------|\n| Regression rate           | 0.15 $\\pm$ 0.10 | 0.08 $\\pm$ 0.08     |\n| Normalized fixation count | 1.22 $\\pm$ 0.48 | 1.01 $\\pm$ 0.28     |\n| Progr. saccade length     | 2.01 $\\pm$ 0.59 | 1.72 $\\pm$ 0.47     |\n| Regr. saccade length      | 2.54 $\\pm$ 2.07 | 1.73 $\\pm$ 2.23     |\n| Skips                     | 0.30 $\\pm$ 0.13 | 0.28 $\\pm$ 0.12     |\n| First pass count          | 0.90 $\\pm$ 0.19 | 0.89 $\\pm$ 0.19 \n\n\nWe observe that the scanpaths generated by ScanDL are similar in diversity to the true scanpaths. Not only is ScanDL's mean value of each reading measure close to the true data, but, crucially, ScanDL also reproduces the variability in the scanpaths, underlining the model's ability to emulate the reader diversity: for instance, the variability with respect to regressive saccades is big in both the true scanpaths as well as the predicted ones, and it is small for both true and predicted with regards to skips and first pass counts. The only minor difference in variability occurs with respect to the normalized fixation count, with ScanDL depicting less overall variability in the number of fixations it produces.\n\nWe have added a new subsection  *Investigation of Reading Pattern Predictability* to the manuscript where we present the analyses summarized above. Moreover, we now discuss the above described findings in the Discussion section.\n\n\n\n\n**2. What does multi-modality mean in this case? It seems like scanpaths are also considered as text while modeling**\n\nThe term *multi-modal* can be understood on two different levels, so we thank reviewer cEYa for making us aware of possible misunderstandings. We used the term *multi-modal* on a merely conceptual level, i.e., to describe the difference between the two kinds of data: the eye movements and the textual data, which are of different modalities.\nHowever, on a technical level, as pointed out by the reviewer, we have erased the multi-modality by modeling the scanpaths as textual data. In order to avoid confusion, we have removed the term *multi-modal* in the abstract and the manuscript in lines 21, 88, 94, and 611, as well as from the keywords.\n\n\n\n\n**3. Scanpaths are required during training for the x\\_idx embedding, it is not clear how that is initialized during inference. how is the length of the scanpath decided when it is sampled to concatenate with EMB(x)**\n\nWe are grateful to reviewer cEYa for making us aware that the model input at inference time needs to be more clearly stated in order to avoid confusion.\n\nAt inference time, the word index embedding $Emb\\_{idx}(\\mathbf{x}\\_{idx})$ is replaced by Gaussian noise, i.e., we initialize it by means of sampling noise from a Standard Normal Distribution. This noise is added up with the position embedding and the BERT word embedding.\nFor better clarity, we have thus changed the corresponding part (lines 333ff. in Section 4.4) in the manuscript as follows:\n\n* *original*:  \"We initialize the scanpath as noise sampled from a standard normal distribution $ \\mathbf{z}\\_0^{\\mathbf{f}} \\sim \\mathcal{N}(0, \\mathbb{I})$, and concatenate it with $ Emb(\\mathbf{x^w}) $ to obtain the model input $ \\mathbf{z}\\_{\\tilde{T}} $.\"\n* *new*: \"We replace the word index embedding of the scanpath $Emb_{idx}(\\mathbf{x^f}\\_{idx})$ \nwith Gaussian noise, initializing it as \n$\\tilde{\\mathbf{x}}^{\\mathbf{f}}\\_{idx} \\sim \\mathcal{N}(0, \\mathbb{I})$. \nWe then concatenate the new embedding \n$\\tilde{Emb}(\\mathbf{x^f}) = \\tilde{\\mathbf{x}}^{\\mathbf{f}}\\_{idx} + Emb\\_{bert}(\\mathbf{x^f}\\_{bert}) + Emb\\_{pos}(\\mathbf{x^f}\\_{pos})$\n with $Emb(\\mathbf{x^w})$ to obtain the model input $\\mathbf{z}_{\\tilde{T}}$\"\n\nCorrespondingly, we have also changed Figure 3 (model architecture) to visualize unambiguously that only the word index embedding is noised and only afterwards is it added up with the position embedding and the BERT word embedding.\n\nRelatedly, we realized that  we had  not provided details on how the length of a scanpath is determined. Hence, we have now added a clarifying sentence in Section 4.4 after line 350:\"As the model learns to predict both the scanpath as well as PAD tokens up to the maximum length, we obtain the scanpaths by removing the model's predicted PAD tokens from the prediction.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The addressed problem is interesting and the proposed model has merits as shown by various ablation and evaluations under different settings (new reader, new sentence, new reader/sentence).\n2. The presented psycholinguistics analysis is an interesting addition to the work providing insights into its alignment with the cognitive aspects of human reading behaviour.\n3. The paper is well-written and easy to understand.", "reasons_to_reject": "1. The model learns scanpath predictions for an average reader. It is not clear how diverse are the readers in the train set? For eg. how aligned are the scanpaths for readers who are on sides of the spectrum of the readers in the dataset? Although the model is able to generalize to new readers, it is not clear how different are the reading patterns of the readers (for same sentence experiments) in the train and test sets and to what extent the model is able to capture the differences.\n2. What does multi-modality mean in this case? It seems like scanpaths are also considered as text while modeling,\n3. Scanpaths are required during training for the x_idx embedding, it is not clear how that is initialized during inference. how is the length of the scanpath decided when it is sampled to concatenate with EMD(x)?", "questions_to_authors": "See above for questions.", "paper_topic_and_main_contributions": "The paper addresses the problem of generating human-like scanpaths for a given sentence. It proposes a multi-modal discrete sequence-oto-sequence diffusion model for generating the scanpaths and achieves state-of-the-art results on various within and across-dataset settings. Ablation results emphasize the importance of sentence conditioning, bert and positional embeddings. A psycholinguistic analysis of the generated scanpaths is also presented showing human-like reading ability of their model's outputs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SlKu6Yev7W", "reviewer": "Reviewer_JBSW", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer JBSW for their helpful input and would like to address the following remark:\n\n**It is not clear how useful the generated scanpaths are in downstream tasks, but as the authors mentioned in the limitation section, this probably can be left for future work.**\n\nReviewer JBSW is right in that not showing how augmenting language models with synthetic eye movements can improve their performance on NLP downstream tasks might be regarded as a limitation of the paper. Unfortunately, the space was limited and we had to decide which aspects we wanted to focus on. As previous research already demonstrates the usefulness of synthetic eye movements, such as Sood et al. (2020, \"Improving natural language processing tasks with human gaze-guided neural attention\"), we opted for focusing on the model itself. As ScanDL is a novel approach to synthesizing eye movements, we chose to perform a thorough evaluation of every aspect of the model, not only assessing its performance in a range of settings that differ in the extent to which the model is forced to generalize out-of-distribution, but also investigating its cognitive plausibility in a psycholinguistic analysis and comparing it to other models (both cognitive and ML-based).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is generally well written with clear motivation, strong performance backed with solid experiments. The analysis part is also interesting to read.", "reasons_to_reject": "It is not clear how useful the generated scanpaths are in downstream tasks, but as the authors mentioned in the limitation section, this probably can be left for future work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, a diffusion model is proposed to generate synthetic scanpaths given sentence texts. Build upon the DiffSeq model, the proposed model include scan path in the input representation learning process, where in the inference time the scan path is obtained through an iterative de-noising process. The proposed method shows sota performance when compared to baselines.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qrlJ297XUj", "reviewer": "Reviewer_iKVr", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer iKVr for their valuable feedback and want to address the following remark: \n\n**With respect to its performance, the only regret is a lack of qualitative comparison with previous methods. Using diffusion models, how the ScanDL performs different from previous models? Where is it strong, and how it performs differently from human?**\n\nIn order to examine whether ScanDL and the baselines show the same trend in predictive performance, i.e., whether ScanDL and the baselines have good predictions on the same sentences, or bad predictions respectively, we averaged, for each model, the NLD of the different scanpaths generated by different readers on the same sentence, resulting in one mean NLD for each sentence and model. We then computed the correlations between the mean sentence NLDs of ScanDL and each of the baselines. Significant correlations are marked with an asterisk: \n\n\n| Baseline    | Pearson corr. |\n|-------------|---------------|\n| Eyettention | 0.40*         |\n| Traindist   | 0.37*         |\n| E-Z Reader  | 0.37*         |\n| Swift       | 0.36*         |\n| Uniform     | 0.35*         |\n\n\nWe observe positive correlations between ScanDL and all baseline models, indicating that there is a tendency of all models to perform well on the same sentences. However, these correlations are not extremely strong (0.4 for Eyettention, and 0.36 and 0.37 for the two cognitive models), which, in turn, indicates that there are indeed differences between how well ScanDL and the baselines cope with certain reading patterns. \n\nTo investigate in more detail what reading patterns ScanDL and the baselines struggle with or perform well on, we performed the following analyses: For ScanDL and each of the baselines, we computed the correlations between the NLD of a predicted scanpath and six reading measures characterizing the corresponding true scanpath from the test set. The reading measures are: the average probability of a word in the sentence to be the starting point of a regression (regression rate), the average number of fixations in a sentence (normalized fixation count), the average lengths of progressive and regressive saccades (prog. saccade length, regr. saccade length), the average probability of a word to be skipped (skips), and the average number of fixations on a word during the first-pass reading of the sentence (first-pass count).\nWhile for reviewer cEYa, the correlations we computed are on values aggregated over the readers and thus provide information about the model's reader-specific predictive ability, these instance-based correlations serve as indication of which specific scanpath properties are more or less difficult for a model. The results are depicted in the table below. Correlations with an absolute value greater than 0.3 are shown in bold font, statistically significant correlations with an asterisk: \n\n\n\n| Model       | Reading measure           | Pearson corr.  |\n|-------------|---------------------------|----------------|\n| ScanDL      | Normalized fixation count | **0.37***   |\n| ScanDL      | Prog. saccade length      | **0.35*** |\n| ScanDL      | Regression rate           | **0.32*** |\n| ScanDL      | Skips                     | 0.25*          |\n| ScanDL      | Regr. saccade length      | 0.21*          |\n| ScanDL      | First pass count          | 0.03           |\n| Eyettention | Prog. saccade length      | **0.32*** |\n| Eyettention | Normalized fixation count | 0.24* |\n| Eyettention | Skips                     | 0.19*          |\n| Eyettention | Regression rate           | 0.18 *         |\n| Eyettention | Regr. saccade length      | 0.18*          |\n| Eyettention | First pass count          | 0.01           |\n| E-Z Reader  | Normalized fixation count | **0.70*** |\n| E-Z Reader  | Regression rate           | **0.48*** |\n| E-Z Reader  | First pass count          | **0.47*** |\n| E-Z Reader  | Regr. saccade length      | 0.19*          |\n| E-Z Reader  | Prog. saccade length      | 0.02           |\n| E-Z Reader  | Skips                     | -0.20*         |\n| Swift       | Normalized fixation count | **0.68*** |\n| Swift       | Regression rate           | **0.46*** |\n| Swift       | First pass count          | **0.46*** |\n| Swift       | Regr. saccade length      | 0.15*          |\n| Swift       | Prog. saccade length      | 0.02           |\n| Swift       | Skips                     | -0.19*         |\n| Traindist   | Normalized fixation count | **0.33*** |\n| Traindist   | Regression rate           | 0.21*          |\n| Traindist   | Prog. saccade length      | 0.20*          |\n| Traindist   | Regr. saccade length      | 0.19*          |\n| Traindist   | First pass count          | 0.16*          |\n| Traindist   | Skips                     | 0.01           |\n| Uniform     | Normalized fixation count | **0.56*** |\n| Uniform     | First pass count          | **0.41*** |\n| Uniform     | Regression rate           | **0.37*** |\n| Uniform     | Regr. saccade length      | 0.23*          |\n| Uniform     | Prog. saccade length      | 0.03           |\n| Uniform     | Skips                     | -0.23*         |\n\n\nFor all models, there is a significant positive correlation between the NLD and both the normalized fixation count and the regression rate. In other words, long scanpaths that exhibit a high number of fixations for a given sentence length, or scanpaths that exhibit a high number of regressions result in predictions with a higher NLD, meaning that they are more difficult to predict than shorter scanpaths (with a lower fixation count) and scanpaths with fewer regressions. \nFurther, the patterns within types of models are similar, i.e., the two ML-based methods ScanDL and Eyettention exhibit the same correlation pattern, as well as the cognitive models E-Z Reader and Swift. Interestingly, both ScanDL and Eyettention have correlations close to zero with respect to the first-pass count, while both E-Z Reader and Swift display a strong correlation. Even more, the cognitive models have overall much stronger correlations than the ML-based models, with their respective correlations for the normalized fixation count being almost twice as high. They appear to be overfitting on specific reading patterns, while ScanDL and Eyettention both seem to generalize better across reading patterns. \n\nWe have added a new subsection *Predictability Patterns Across Models* to the manuscript in which we present the above described analyses. We have also added a sentence to the Discussion discussing these results.\n\nFurthermore, we are currently conducting an analysis investigating the instance-based correlation between the NLD and linguistically-based sentence metrics, such as the number of named entities, information density, and syntactic complexity. This analysis will also be included in the above-mentioned subsection.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Establishes a new state-of-the-art on generating eye movements that has human-level performance. Methodological contribution is also strong.", "reasons_to_reject": "Not especially.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a model to generate synthetic eye-movements on text. Based on a \ndiscrete diffusion model with appropriate tweaks, the authors succeed in generating\neye-movements that is even better than human performance. While the main contribution is\nmethodological, experimental investigation is also sufficient to support the main claim.\n\nThis paper is interesting and will be useful for many disciplines. Since gaze information\nis difficult to obtain experimentally, this kind of synthetic eye-movements are \nbeneficial for many problems with cognitive effects, such as readability, grammaticality,\nsentiment analysis, and so on. Improvements from previous state-of-the-art seems to be\nsignificant, thus this method could be a new standard on this task.\n\nWith respect to its performance, the only regret is a lack of qualitative comparison \nwith previous methods. Using diffusion models, *how* the ScanDL performs different from \nprevious models? Where is it strong, and how it performs diffenrently from human?\nSuch a qualitative information will further strengthen the arguments to adopt a diffusion\nmodel on this kind of problem.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a model for generating synthetic eye movements on text using a discrete diffusion model with adjustments that surpass human performance. While the primary contribution is methodological, the experimental evidence supports the paper's main claim. This work is anticipated to have broad applications across various disciplines due to the challenge of obtaining real gaze data. Synthetic eye movements can benefit cognitive research in areas like readability, grammaticality, sentiment analysis, and more. The improvements over previous state-of-the-art methods are substantial, potentially establishing this approach as a new standard for the task. Both AC and 3 reviewers acknowledged the contribution of this paper. 

However, one notable concern is the absence of qualitative comparisons with previous methods. Partly due to the novelty of this research. While the paper demonstrates the superior performance of the ScanDL model, it lacks qualitative insights into how it differs from earlier techniques that also employ diffusion models. Understanding the strengths and differences in performance compared to human eye movements could enhance the paper's arguments for adopting a diffusion model for this task.

In summary, the paper's methodological contribution is significant, establishing a new state-of-the-art in generating human-like eye movements. The model's performance is well-supported by experiments, and the paper is well-written with clear motivation and interesting analysis. Additionally, the psycholinguistic analysis adds valuable insights into the alignment of the model's outputs with human reading behavior. In my mind, the paper could benefit from further exploration of the diversity of readers in the training set and a clearer explanation of how scanpaths are handled during inference. Despite some minor concerns exists, the detailed rebuttal addressed most of the problems raised by reviewers.

Consider the original paper, the reviewer’s comments, and the rebuttal, as the area chair, would like to to recommend acceptance of the paper and invite the author to take an oral presentation in EMNLP.
Decision: Accept-Main