Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understanding
Authors: Shuwen Deng
Paul Prasse
David Robert Reich
Tobias Scheffer
Lena Ann Jäger
Venue: EMNLP 2023 Main (2023)
Keywords: Human gaze data
synthetic scanpaths
gaze-augmented language model
natural language understanding
transformer
deep neural networks

Abstract: Human gaze data offer cognitive information that reflects natural language comprehension. Indeed, augmenting language models with human scanpaths has proven beneficial for a range of NLP tasks, including language understanding. However, the applicability of this approach is hampered because the abundance of text corpora is contrasted by a scarcity of gaze data. Although models for the generation of human-like scanpaths during reading have been developed, the potential of synthetic gaze data across NLP tasks remains largely unexplored. We develop a model that integrates synthetic scanpath generation with a scanpath-augmented language model, eliminating the need for human gaze data. Since the model's error gradient can be propagated throughout all parts of the model, the scanpath generator can be fine-tuned to downstream tasks. We find that the proposed model not only outperforms the underlying language model, but achieves a performance that is comparable to a language model augmented with real human gaze data. Our code is publicly available.

Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understanding
Shuwen Deng', Paul Prasse', David R. Reich', Tobias Scheffer', Lena A. Jäger!
Department of Computer Science, University of Potsdam, Germany
2 Department of Computational Linguistics, University of Zurich, Switzerland [deng, prasse, david. reich, scliaassccceeffeeeererr powprrrraeannnoooaaaaammm de jaegerêcl uzh. ch
Abstract
Prediction
Ceanpath-Augmentad Language Model Scampath Enaader (GRU)
Human gaze data offer cognitive information that reflects natural language comprehension. Indeed, augmenting language models with hu- man scanpaths has proven beneficial for range of NLP tasks, including language understand- ing. However, the applicability of this approach is hampered because the abundance of text cor- pora is contrasted by scarcity of gaze data. Although models for the generation of human- like scanpaths during reading have been devel- oped, the potential of synthetic gaze data across NLP tasks remains largely unexplored. We develop a model that integrates synthetic scan- path generation with a scanpath-augmented lan- guage model, eliminating the need for human gaze data. Since the model's error gradient be propagated throughout all parts of the model, the scanpath generator can be fine-tuned downstream tasks. We find that the proposed model not only outperforms the underlying lan- guage model, but achieves a performance that is comparable to language model augmented with real human gaze data. Our code is publicly available.'
Fiention
Tuaer
Eysturtion
Language Mocel
Scanpath Generation Model
Tok
Tok
Tok
Tok
Tok (SEP)
Synthetic scanpath-augmented language model: the Scanpath Generation Model predicts se- quence of fixations for an input sentence; token embed- dings are rearranged according to the order of fixations.
Figure 1:
the use of aggregated word-level gaze features to regularize neural attention mechanisms (Bar- rett et al., 2018; Sood et al., 2020). Moreover, non-aggregated scanpaths, which capture the com- plete sequential ordering of the reader's 's gaze be- havior, have also demonstrated promise in NLP tasks (Mishra et al., 2017, 201 8a; Yang and Hollen- stein, 2023).
However, collecting gaze data is a resource- intensive endeavor, even for very small text corpora. Hence, human gaze data is scarce, and NLP task- specific gaze recordings are even scarcer. More- over, applying language model that additionally consumes gaze data requires gaze data to be avail- able for the input text at deployment time-which unrealistic for most use cases. To overcome these limitations, researchers have proposed a multi-task learning approach for NLP tasks such as sentence compression (Klerke et al., 2016), sentiment analy- sis (Mishra et al., 2018b), and predicting text read- ability (Gonzålez- and Søgaard, 2017). In this approach, labeled data for the specific NLP task is used the primary task, while a separate eye-tracking corpus utilized as an auxiliary task. While this approach helps mitigate the need for task-specific gaze data during training and testing, the problem of general scarcity of gaze samples
1 Introduction and Related Work
When humans read, they naturally engage in the cognitive process of comprehending language, which, in turn, reflected in their gaze behav- ior (Just and Carpenter, 1980). In a nutshell, a scanpath (i.e., sequence of consecutive fixations) on stimulus text approximates the reader's at- tention, which can be exploited to inform Natural Language Processing (NLP) tasks.
Gaze data has been shown to be beneficial in vari- ous NLP tasks, such as part-of-speech--aagin (Bar- rett et al., 2016), named entity recognition (Hol- lenstein and Zhang, 2019), generating image cap- tions (Takmaz et al., 2020) and question answer- ing (Sood et al., 2021). Researchers have explored
hnttps://github Sttp://tttrrrcccccaaaaasa------e

---

remains and hinders effective supervision for data- intensive architectures.
probability distribution over saccade ranges at each timestep. The next fixated word index is deter- mined by sampling from this distribution.
In this paper, we propose an alternative approach by using synthetic gaze data, which can be gener- ated easily for any given text, to provide cogni- tive signals across NLP tasks. The seminal work of Sood et al. (2020), which integrates eye move- ment data generated by computational cognitive model of gnge-movementccontol--diie
Scanpath-AAugmenter Language Model We adopt the PLM-AS framework (Yang and Hollen- stein, 2023), which augments pre-trained language models with human scanpaths for sentiment classi- fication. This framework uses a language model to extract token embeddings for a sentence, associat- ing each embedding with its position index. By uti- lizing human scanpath (fixation index sequence) as input, the model rearranges the token embedding sequence based on the order in which the words are fixated by the reader. The transformed sequence is then fed into a scanpath encoder, implemented as a layer of gated recurrent units (GRU), where the output of the last step is used as the final feature for sentiment classification. This framework allows for the use of different language models and achieves high performance through fine-tuning. In this work, we employ BERTBASE (Devlin et al., 2019) as the language model, following Yang and Hollenstein (2023).
for tasks such as sentence compression and paraphrase generation, demonstrated the potential of synthetic eye-gaze data. Khurana et al. (2023) explored a proof-of-concept model that integrated synthetic gaze data across multiple NLP tasks, but their re- sults did not reach the performance of fine-tuned BERT model (Devlin et al., 2019) without eye gaze on the General Language Understanding Evalua- tion (GLUE) benchmark. In our work, we build on recent advances in the development of machine- learning models for generating human-like sean- paths during reading (Deng et al., 2023; Bolliger et al., 2023; Khurana et al., 2023; Nilsson and Nivre, 2011).
We develop model that combines synthetic scanpath generation with scanpath-augmented language model, eliminating the need for human gaze data. The model allows for fine-tuning the scanpath generator to downstream tasks by propa- gating the error gradient through the entire model. Our upproach not only outperforms the underlying language model in multiple tasks on the GLUE, es- pecially in low-resource settings, but even reaches a performance comparable to an eye-gaze augmented model that uses real, rather than synthetic, eye movement data in sentiment classification.
Joint Modeling for NLP Tasks To eliminate the need for human gaze data, we integrate the syn- thetic scanpath generated by the Eyettention model consisting of a fixation index sequence into the PLM-AS framework. Before integration, the word index sequence generated by Eyettention is con- verted into token index sequence. During train- ing, the error gradient of the scanpath-augmennt language model can be back-propagated through the Eyettention model, allowing its parameters be adapted for specific NLP task. To handle the non-differentiable sampling from a categorical dis- tribution involved in scanpath generation, we em- ploy the Gumbel-softmax distribution (Jang et al., 2017) fully differentiable approximation. The training process consists of two phases. First, we pre-train the Eyettention model on natural read- ing task. Second, we train the entire model, which includes fine-tuning the language model and the Eyettention model, as well as training the scanpath encoder from scratch. For the Eyettention model, we add residual connections in both encoders to enhance its performance.
2 Model
We develop a model that combines scanpath gen- eration model with a campatth-augmenter language model to perform NLP downstream tasks. Figure 1 depicts the proposed model architecture.
Scanpath Generation Model We adopt Eyetten- tion (Deng et al., 2023), an open-source state-of- the-art model for scanpath generation over text. Eyettention predicts consecutive fixation locations, represented as word indices, based on stimulus sentence and the preceding fixations. It consists of two encoders, one for embedding the stimulus sentence, and the other for embedding the scanpath history. A cross-attention layer aligns the outputs of the two encoders, and a decoder produces a
Note that BERT can he substituted with other advanced pre-trained language models, potentially leading to further enhuncements in task performance.

---

3
Experiments
Model
Scanpath (8)
F1
AUC
BERT+
82.932.66
92.421.6
ScanTexIGAN ScanTexIGAN
real
83.34
In this section, we describe the data and present the evaluation results of our model for wide range of NLP tasks. Further details about training and hyper- parameter tuning can be found in Appendix B.
synthetic
84.77
PLM--S*
real (7)
85.81
94.79
synthetic
85.35
94.90
Eycticntion (frozen)
syathetic (7)
84.521.79
94,50
Eyettention (scratch)x
synthetic
85.0316
94.771.0
Table I: Results for sentiment classification ETSA, with standard errors indicated as subscript. Results ob- tained from our experiments are marked with *; other results are from the respective papers for recapitulation.
3.1 Data Sets
CELER (Berzak et al., 2022): We pre-train the scanpath generation model Eyettention the L1 subset of CELER, which contains eye-tracking recordings collected from 69 native speakers of English during natural reading of 5,456 sentences.
86
ETSA (Mishra et al., 2016) contains task- specific gaze recordings for sentiment classifica- tion of subjects who each read 383 positive and 611 negative sentences, including sarcastic quotes, short movie reviews, and tweets.
84
82
BERT
80
PLM-AS w/ human scanpaths
synthttic scanpaths
13 15
Scanpaths
GLUE (Wang et al., 2018) includes sentiment analysis (SST-2), linguistic acceptability (CoLA), similarity and paraphrase tasks (MRPC, STS-B, QQP), and natural language inference tasks (MNLI, QNLI, RTE). No gaze data are available.
Figure 2: Sentiment classification performance on ETSA with varying numbers of scanpaths at training and application time. Error bars show the standard error.
not pre-trained, the performance decreases. This demonstrates the importance of both pre-training and task-specific fine-tuning of the scanpath gener- ator.
3.2 Sentiment Classification
Table presents the results of our model on the sentiment classification task ETSA (Mishra et al., 2016), in comparison to BERT and previous state- of-the-art eye-gaze augmented models. We follow 10-fold cross-validation regime. In each iteration, BERT fine-tuned on the training portion of the ETSA text corpus, and PLM-AS is fine-tuned on the training portion of the ETSA text corpus and gaze data. Our model is fine-tuned on the training portion of the ETSA text corpus and, instead of the ETSA gaze data, synthetic gaze data generated by Eyettention. Since each sentence is associated with multiple scanpaths, tion by averaging the pre-softmax logits obtained from the models across all scanpaths for the PLM- AS baseline. Our model averages equally many synthetic scanpaths. We make multiple notable ob- servations Table 1:
Varying the number of scanpaths We analyze the impact of the number of scanpaths sampled both at training and at application time on model performance. Figure shows the FI score as a function of the number of scanpaths used by BERT without eye gaze, PLM-AS with human scanpaths, and our model with synthetic scanpaths. We ob- serve that the performance of Seanpath-augmeee models improves the number of scanpaths in- creases, reaching its peak seven scanpaths. Im- portantly, our model outperforms BERT and, when being augmented with five or more synthetic scan- paths, approaches the performance of PLM-AS augmented with human scanpaths.
compute the final predic-
Performance We hypothesize that eye gaze might be most beneficial in low- resource settings. To this hypothesis, we sam- ple small subset of the training sentences = (200, 400, 600) from the total number of around 800 training instances, and evaluate the perfor- mance of our model augmented with seven syn-
(a) Our model outperforms both BERT and the state-of-the-ar ScanTextGAN (Khurana et al., 2023) augmented with gaze data.
(b) Our model, augmented with synthetic scanpaths, achieves comparable performance the PLM-AS model augmented with human scanpaths, eliminat- ing the need for human scanpaths.
'The optimal number of scanpaths to be used by the model is considered a hyperparameter for the subsequent experi- ments.
(c) Ablation experiments (bottom two rows) show that when the Eyettention model is frozen or

---

MNLI
QQP
QNLI
SST-2
CoLA
STS-B
MRPC
RTE
Avg.
K
Model
Gaze
392k
363k
108k
67k
8.5k
5.7k
3.5k
2.5k
200
BERT x
42.90
57,422.0
73.07
16.952.74
79.43u.69
81.18
54,30
60.50
Ours
48,97ass
61.63
70.466.62
80.76u74
24.08
74.94).20
81.85m.7
59,35
62.75
500
BERT
12.09
65.1
17.04e.
82.35o47
35.611.74
83.14
81.533029
60.72n.sl
67.23
Ours
56.48
67.8
77.60
84.63u
16.41
81.99
82.32a.2
61.88
68.64
1000
BERT
58.977
67.35pm.9
78.88
85 55
39.89
85,42
84,181.0
63.39%.99
70.49
Ours
61.28g25
70.65p.14
80.74..00
26.06
41.19b.50
85.130.a
84.61
64.55
71.78
BERT
82.9
69.7
90.1
93.1
53.9
84.8
87.7
66.1
78.54
all
Ours
83.6
69.6
90.1
93.8
50.2
85.8
87.7
67.3
78.51
Table 2: Results on the GLUE benchmark with K = [200, 500, 1000, all) training samples. Below each task, the total number of training samples for each dataset indicated. We use for QQP and MRPC, Spearman correlation for STS-B, Matthews correlation for CoLA, and accuracy for the remaining tasks. The standard error is indicated the subscript.
out of 8 tasks, and achieves comparable perfor- mance in 3 tasks. However, our model's perfor- mance is notably poor in the CoLA task, possibly due the model's emphasis on gaze sequence or- dering, potentially overshadowing the importance of the original word order, which is critical to de- termine linguistic acceptability of sentences.
85
80
75
65
60
55
Ours w/ 7 synthetic scanpaths
200
400
600
all (8D4)
Training sentence instances
Figure 3: Sentiment classification performance ETSA in the low-resource setting, Error bars repre- sent standard error.
Performance We present the sults on the GLUE benchmark with K {200, 500, 1000} training samples in Table 2. We take ad- ditional 1,000 samples from the original training set as the development set used for early stopping.
thetic scanpaths (the best-performing configuration from the previous experiments). The performance comparison between our model and the baseline model BERT is shown in Figure 3. Our model con- sistently outperforms BERT, with larger improve- ments observed when using less training data.
The original development set is utilized for testing, We perform 5 runs with different random seeds to shuffle the data and report the average results.
Overall, our model consistently outperforms BERT across tasks, except for the STS-B task. terms of average score, our model shows perfor- mance gains of 2-4% compared to BERT.
3.3 GLUE Benchmark
contrast to the small and single task-specific ETSA data set, we extended our evaluation to as- sess whether gaze data could enhance language models across different tasks, including scenarios with substantial text data. To achieve this, we eval- uate our model on the GLUE benchmark, com- prehensive collection of diverse NLP tasks with a large number of text samples. As no eye gaze data is available for GLUE, we focus on the comparison with the BERT baseline, and investigate both, high- and low-resource settings.
4 Discussion and Conclusion
We developed model that integrates synthetic scanpath generation into auggeette lan- guage model. We observe that the model achieves results that are comparable to language model augmented with human scanpaths, which elimi- nates the need for human scanpaths during both training and testing. Human gaze data are only available for a very limited number of NLP tasks and data sets. At application time, under any standard use case scenario of NLP tasks, no gaze recordings are available. Synthetic gaze data not only open the possibility train high-capacity gaze-augmented models across tasks, which would otherwise require the collection an impractical large volume of gaze data, but also allow for the
HighhRReouure Performance The results of our model on the GLUE test set using all training sam- ples (K all) are reported in the bottom two rows Tablee The results are obtained from the GLUE leaderboard. Our model outperforms BERT in

---

exploitation of eye gaze signals as model input at application time.
during training and inference time is not con- ducive to high model efficiency. Future work could explore alternative scanpath generation mod- els that do not rely on auto-regressive architectures to improve efficiency.
Using the GLUE benchmark, we observe that gaze signals show benefits not only for sentiment classification tasks (SST-2), as reported in previ- ous research, but also for entailment classification tasks (MNLI, RTE) and sentence similarity task (STS-B). This highlights the potential of integrat- ing cognitive signals from eye gaze into a wider range of NLP tasks in the future. Nevertheless, is evident that not tasks derive equal benefits from gaze data. It remains up to future research to explore which types of tasks benefit most from gaze signals.
Ethics Statement
It is crucial to acknowledge potential privacy risks in collecting, sharing, and processing human gaze data. Since eye movements are highly individual, it can be possible extract participant's identity from gaze data (Jäger et al., 2020; Makowski al., 2021). Other personal information such as gen- der (Sammaknejad et al., 2017) and ethnicity (Blig- naut and Wium, 2014) that can be delected to some degree today may turn out to be extractable accu- rately in the future, which incurs a risk of leakage of personal information from gaze data. Synthetic gaze data can reduce the need for large-scale ex- periments with human subjects, even though some amount of human gaze data still necessary train generative models.
Our results further show that the potential bene- fit of augmenting language models with gaze data is higher for low-resource settings. Hence, we be- lieve that the augmentation with gaze data might be particularly interesting for low-resource lan- guages. Two ongoing multi-lab efforts to collect large multilingual eye-tracking saracking-whilee--e while-reading cor- pora (MECO* and MultiplEYE include range of low-resource languages, which will allow for training scanpath generators and augmenting lan- guage models with synthetic eye gaze for these languages in the near future.
A Acknowledgements
This work was partially funded by the German Federal Ministry of Education and Research under grant 01| $20043.
Limitations
One limitation of our work that the scanpath generation model Eyettention was pre-trained on eye-tracking data recorded on isolated sentences (single sentence reading paradigm). Since the ma- jority of tasks in the GLUE benchmark involve two-sentence classification, future work could in- volve pre-training the model on an eye-tracking data set specifically designed for two-sentence read- ing tasks to enhance its performance. Additionally, scanpath augmentation turned out to be detrimental to the language model's performance for the task of identifying linguistically acceptable sentences (CoLA). This finding was to be expected as the actual word order is more relevant for linguistic acceptability of a sentence than the order in which the words are fixated. Pre-training the scanpath generator on an eye-tracking corpus that includes both acceptable and unacceptable sentences may be beneficial for improving the model's performance.
References
Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. 2018. Sequence classi- fication with human attention. In Proceedings of the 22nd Conference on Computational Natural Lan- guage Learning (CoNLL), pages 302-312, Brussels, Belgium.
Maria Barrett, Frank Keller, and Anders Spgaard. 2016. Cross-lingual transfer of correlations between parts of speech and gaze features. Proceedings of the 26th International Conference on Computational Lin- guistics (COLING): Technical Pupers, pages 1330- 1339, Osaka, Japan.
Yevgeni Berzak, Chie Nakamura, Amelia Smith, Emily Weng, Boris Katz, Suzanne Flynn, and Roger Levy. 2022. CELER: A 365-participant corpus of eye movements in LI and L2 English reading. Open Mind, pages 1-10.
Pieter Blignaut and Daniël Wium. 2014. Eye-tracking data quality as affected by ethnicity and experimental design. Behavior Research Methods, 46:67-80.
Furthermore, in our proposed framework, the sampling process involved in scanpath generation
"https: com "https://multipleyee.u
Lena Bolliger. David R. Reich, Patrick Haller, Debo- rah N. Jakobi, Paul Prasse, and Lena A. Jäger. 2023.

---

ScanDL: A Diffusion Model for Generating Syn- thetic Scanpaths on Texts. In Proceedings of Em- pirical Methods in Natural Language Processing (EMNLP), Singapore, Singapore.
Sigrid Klerke, Yoav Goldberg, and Anders Søgaard. 2016. Improving sentence compression by learning to predict gaze. In Proceedings of North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (NAACL- H.TT", pages 1528-1533, San Diego, California.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bah- danau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Ennoddr-decoder ap- proaches. In Proceedings of SSST-8, Eighth Work- shop on Syntax, Semantics and Structure in Statistical Translation, pages 103 111, Doha, Qatar.
Nya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of In- ternational Conference on Learning Representations (ICLR), New Orleans, Louisiana, United States.
Shuwen Deng, David R Reich, Paul Prasse, Patrick Haller, Tobias Scheffer, and Lena A Jäger. 2023. Eye- ttention: An attention-base based sual-sequence model for predicting human scanpaths during reading. Proceed- ings of the ACM on Staman-Compun Interaction, 7(ETRA)11244 24.
Silvia Makowski, Paul Prasse, David R Reich, Daniel Krakowezyk, Lena A Jäger, and Tobias Scheffer. 2021. Derpeycdentffiattioniiee Oculomotoric bio- metric identification and presentation-attan tion using deep neural networks. IEFF Transac- tions on Biometrics, Behavior, and Identity Science, 3(4):506 518.
detec-
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (NAACI-HILT), pages 4171 7716, Minneapolis, MN, USA.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Alma- hairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. 2022. UniPELT: A unified framework for parameter-efficient language model tuning. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume Long Papers), pages 223--2644, Dublin, Ireland.
Ana Valeria Gordulez-Garduño and Anders Søgaard. 2017. Using gaze to predict text readability. In Pro- ceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, EMNLP. pages 0088-433000 Copenhagen, Denmark.
Abhijit Mishra, Pushpak Bhattacharyya. Abhijit Mishra, and Pushpak Bhattacharyya. 2018a. Scanpath com- plexity: modeling reading/annotation effort using gaze information. Cognitively Inspired Natural Lan- guage Processing: An Investigation Based on Eye- tracking, pages 77-98.
Nora Hollenstein and Ce Zhang. 2019. Entity recog- nition at first sight: Improving NER with eye move- ment information. In Proceedings of North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (NAACL- HLT), pages 1-10, Minneapolis, Minnesota.
Abhijit Mishra, Kuntal Dey, and Pushpak Bhattacharyya. 2017. Learning cognitive features from gaze data for sentiment and sarcasm classification using con- volutional neural network. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (ACL.), pages 377-387, Vancouver, Canada.
Lena A. Jäger, Silvia Makowski, Paul Prasse, Lichr Sascha, Maximilian Seidler, and Tobias Scheffer. 2020. Deep Eyedentification: Biometric identifica- tion using micro-movements of the eye. In Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2019, volume 11907 of Lecture Notes in Computer Science. pages 299-314 114 Cham, Switzer- land. Springer International Publishing.
Abhijit Mishra, Diptesh Kanojia, and Pushpak Bhat- tacharyya. 2016. Predicting readers' sareasm under- standability by modeling gaze behavior. In Proceed- ings of the AAAI Conference on Artificial Intelligence. volume 30, Phoenix, Arizona, USA.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Cate- gorical reparameterizatioon with gumbel-softmax. In Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France.
Abbijit Mishra, Srikanth Tamilselvam, Riddhiman Dasgupta, Seema Nagar, and Kuntal Dey. 2018b. Cognition-coonizzant sentiment analysis with multi- task subjectivity summarization based on annotators' gaze behavior. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 32, New Orleans, Lousiana, USA.
Marcel A Just and Patricia A Carpenter. 1980. A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4)3399
Mattias Nilsson and Joakim Nivre. 2011. Entropy- driven evaluation of models of eye movement con- trol in reading. In Proceedings of the 8th Interna- rional NLPCS Workshop, pages 201-212, Copen- hagen, Denmark.
Varun Khurana, Yaman Kumar, Nora Hollenstein, Ra- jesh Kumar, and Balaji Krishnamurthy. 2023. Syn- thesizing human gaze feedback for improved NLP performance. In Proceedings of the 17th Confer- ence of the European Chapter of the Association for Computational Linguistics| (EACL), pages 1895- 1908, Dubrovnik, Croatia.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor

---

Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Proceed- ings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), pages 8024-8035, Vancouver, Canada.
Negar Sammaknejad, Hamidreza Pouretemad, Changiz Eslahchi, Alireza Salahirad, and Ashkan Alinejad. 2017. Gender classification based on eye movements: A processing effect during passive face viewing, Ad- vances Cognitive Psychology. 13(3):232.
Ekta Sood, Fabian Kögel, Philipp Müller, Dominike Thomas, Mihai Bace, and Andreas Bulling. 2021. Multimodal integration of human-like attention in vi- sual question answering. Computing Research Repos- itory.
Ekta Sood, Simon Tannert, Philipp Müller, and Andreas Bulling. 2020. Improving natural language process- ing tasks with human gaze- gazdeg neural attention. In Proceedings of the 34th Conference on Newral Information Processing Systems (NeurIPS), pages 0327-6341 Online.
Takmaz, Sandro Pezzelle, Lisa Beinborn, and Raquel Fernández. 2020. Generating image descrip- tions via sequential cross modal alignment guided by human gaze. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 4664-4677. Online.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE; A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of EMNLP Workshop BlackboxNLP: Analyzing and In- terpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: thate-of-the-art natural language processing. Pro- ceedings of Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, pages 38-45, Online.
Duo Yang and Nora Hollenstein. 2023. PLM-AS: Pre- trained language models augmented with scanpaths for sentiment classification. In Proceedings the Northern Lights Deep Learning Workshop. Tromso. Norway.

---

Table 3: Optimal number of scanpaths used for our model in GLUE Benchmark with K (200, 500, 1000, all) training sentences.
K
MNLI
QQP
QNLI
SST-2
CoLA
STS-B B
MRPC
RTE
200
5
3
3
7
3
7
500
7
5
3
3
3
7
7
1000
5
7
3
5
all
2
4
2
3
3
3
Appendix for Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understanding
GLUE Benchmark We evaluate each GLUE data set using the metric specified in the bench- mark. We use the code provided in the Hugging- Face repository to train the BERT model and compute the metrics.
In the high-resource setting, we fine-tune the BERT model using the hyperparameter tuning pro- cedure outlined in the original paper (Devlin et al., 2019). We select the best learning rate from (Se-5, 4e-5, 3e-5, 2e-5] ] for each task based on the perfor- mance on the development set. The same learning rate used for training our model.
Model Details
PLA-AS Framework For the PLM-AS frame-
work, we adhere to the design of the original pa- per (Yang and Hollenstein, 2023). The scanpath en- coder consists of a single-direction GRU layer (Cho et al., 2014) with a hidden size of 768 and a dropout rate of 0.1. We initialize the hidden state of the scanpath encoder using the [CLS] token outputs from the final layer of BERT.
Additionally, for our model, we perform hyper- parameter search on the development set to deter- mine the optimal number of scanpaths to be used by the model for each task. We explore different numbers of scanpaths from (2, 3, 4) and select the configuration that achieves the best performance the development set. The optimal configuration for each task can be found Table 3.
Training Details
We train all neural networks using the Py- Torch (Paszke et al., 2019) library on an NVIDIA A100-SXM4-40GB GPU using the NVIDIA CUDA platform. For training, we use the AdamW optimizer (Loshchilov and Hutter, 2019), and a batch size of 32. We train 20 epochs and select the model with the best validation performance for evaluation. The training is early stopped if the validation performance does not increase for 3 consecutive epochs. During the training of our model, we employ the Gumbel-softmax distribu- tion with a temperature hyperparameter set to 0.5. We use the pre-trained checkpoints from the Hug- gingFace repository (Wolf et al., 2020) for the lan- guage model BERT BASE
In the low-resource setting, we use the same learning rate that was found optimal in the high- resource setting for each task. Besides, we perform hyperparameter search on the development set, investigating different numbers of scanpaths from (3, 5, 7) to be used by model. The optimal con- figurations for each task can be found in Table 3. To reduce variance, we apply shuffling the training data using different random seeds. We use the first K samples as the new training set, and the subsequent 1,000 samples as the develop- ment set. The data seeds used for shuffling are (111,222,333, (4333,444555)) while the seed s=42 is con- sistently used for model training across all models. The procedure was adapted from Mao et al. (2022).
Sentiment Classification
During training, each scanpath associated with one sentence is treated separate instance. However, during evalua- tion, the pre-softmax logits obtained from multiple scanpaths associated with the same sentence are averaged to generate a single prediction for this sen- tence. We use a lcarning rate of le-5 for training all investigated models.
'https://github. http:s///.ttti..oooonggg/aa///rr treerhaann/seeppaeeepccccrreeerr


Reviews:
[{"review_id": "3mz7HMHOVt", "reviewer": "Reviewer_3vtZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing your insightful feedback. We would like to address your concerns as follows.\n&nbsp;      \n&nbsp;      \n\nQ1: In Table 2, by increasing the data size K, the advantage of using synthesised gaze data become less obvious. For example when using all of the training data, the F1 score is basically the same for both of the model(see the Avg column), could this mean that, as long as we have enough language data, it is unnecessary to have gaze data anymore? Gaze data is only valuable when the corresponding language resources are limited?\n\n**A1:** We agree with the reviewer's observation regarding Table 2, where the advantage of synthesized gaze data becomes less prominent as training data size increases. However, when using all of the training data and examining the model's performance on individual tasks, we still observe performance improvements for specific tasks, namely MNLI, SST-2, STS-B, and RTE, as described in Section 3.3, paragraph ``High-Resource Performance\" (line 244). We analyzed these results in Section 4 (line 281), highlighting the potential utility of gaze data for enhancing certain NLP tasks, even when there is an abundance of text data available.   \n\nWe appreciate the reviewer's feedback, which has prompted us to include a more explicit discussion. While gaze data demonstrates benefits for certain tasks in the high-resource setting, it's evident that not all tasks benefit equally from its incorporation within the current framework.  In response, we have revised our manuscript by inserting the following sentences on line 288:  \n*Nevertheless, it's evident that not all tasks derive equal benefits from gaze data. In future research, exploring which types of tasks may benefit most from gaze signals would be of significant interest.* \n&nbsp;     \n&nbsp;            \n\nQ2: When I check this point with figure 3, I feel the necessity of using larger K size to check if the performance of Bert will be no different from the model with eye gaze. In a word, further experiment(for the figure 3) should be performed and the conclusion should rather be, under \"low resource\" situation, synthesised gaze data could help improve the performance of the language model.\n\n**A2:** Thank you for bringing this point to our attention. The ETSA sentiment classification dataset comprises only about 1k sentences (line 171). When reserving 10\\% for testing, the training data is limited to a maximum of around 800 sentences. Consequently, the rightmost dots in Figure 3 correspond to the scenario where all available training data is used. To clarify this, we have made the following revision in line 228:   \n*To test this hypothesis, we sample a small subset of the training sentences, K = {200, 400, 600}, from the total number of approximately 800 training instances.*     \nAdditionally, in Figure 3, we have updated the label on the x-axis to replace the last number *'800'* with *'all (804)'*.      \n \nWe would like to underline that the collection of high-quality gaze data for an extensive text corpus is a notably resource intense endeavor (besides the reimbursement of the participants, eye-tracking lab facilities and trained research assistants to supervise participants on a\none-to-one basis are needed). Consequently, all existing eyetracking datasets for NLP tasks are very limited in size. Overcoming this major bottleneck for the field was a key factor motivating our decision to investigate the potential of synthetic gaze data.       \n     \nWhile expanding the size K for further experimentation within the ETSA dataset was not possible, we conducted supplementary experiments using the GLUE dataset in our paper. This provided us with an opportunity to evaluate whether synthetic scanpaths could benefit language models even in scenarios with a large amount of text data.  Notably, our results revealed that for the SST-2 dataset, similar to the ETSA dataset, which also involves sentiment classification, our model augmented with synthetic scanpaths demonstrated improved performance even when utilizing the entirety of the available training data. To clarify this, we have now inserted the following sentences in line 238:    \n*In contrast to the small and single-task-specific ETSA dataset, we extended our evaluation to assess whether gaze data could enhance language models across different tasks, including scenarios with substantial amounts of text data. To achieve this, we evaluate our model on the GLUE benchmark.*    \n&nbsp;           \n&nbsp;        \n\n\nQ3: In Page 3, the part about low resources, the K set should be {200,400, 600, 800} according to Figure 3.  \n \n**A3:** Please also refer to our answer above concerning the training data size K. We have revised the manuscript in line 228:\n*To test this hypothesis, we sample a small subset of the training sentences, K = \\{200, 400, 600\\}, from the total number of approximately 800 training instances.* Additionally, in Figure 3, we have updated the label on the x-axis to replace the last number *'800'* with *'all (804)'*. We thank the reviewer for helping us achieve greater clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of using synthesised eye gaze data to improve language model is novel and interesting.\n\n2. Based on the results, the synthesised gaze data could improve the performance of language model, which could be considered as a nice contribution in terms of short paper based case study.", "reasons_to_reject": "In Table 2, by increasing the data size K, the advantage of using synthesised gaze data become less obvious.\nFor example when using all of the training data, the F1 score is basically the same for both of the model(see the Avg column), could this mean that, as long as we have enough language data, it is unnecessary to have gaze data anymore? Gaze data is only valuable when the corresponding language resources are limited? When I check this point with figure 3, I feel the necessity of using larger K size to check if the performance of Bert will be no different from the model with eye gaze. \nIn a word, further experiment(for the figure 3) should be performed and the conclusion should rather be, under \"low resource\" situation, synthesised gaze data could help improve the performance of the language model.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a new method to improve the performance of natural language comprehension, namely using synthesised eye gaze data to augment a pre-trained language model.\nThe results shows that by adding the synthesised gaze data, the performance of language model is in general improved, which has been tested in sentiment classification task and GLUE benchmark.\nAccording to author(s), gaze data in usual case is hard to get due to its scarce nature and expensive to collect.\nThis paper thus shows us the potential usage of synthesised gaze data, which cut off the reliance on eye tracking data.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Page 3, the part about low resources, the K set should be {200,400, 600, 800} according to Figure 3."}, "extra_text": {}}, {"review_id": "jQf13VUvJP", "reviewer": "Reviewer_ER3y", "summary": "", "questions": "- Q1: In Yang and Hollenstein (2023), the LPM-AS performance on ETSA (table 1 in their paper) is 90.48 in F1, which differs from the reproduced 85.81. Are these two numbers comparable? If so, what's the reason for this discrepancy?\n- Q2: Any insights behind the observation that, in Figure2, the performance peak is when the number of scanpaths is 7 and the performance degrades when considering more scanpaths?\n- Q3: In the low-resource setup for GLUE, why not use the original test set for testing?\n- Q4: What’s the train/dev/test split in ETSA dataset?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing your insightful feedback. We would like to address your concerns and answer your questions as follows.     \n&nbsp;   \n&nbsp;     \nQ1: One concern is that when the number of scanpath is less than 5 (i.e., 1 and 3 in Figure 2), the proposed model performs worse than the underlying models (i.e., BERT and PLM-AS) and the strong baseline ScanTextGan. This indicates that somehow the model relies on the aggregation of predicted scanpaths. More detailed analysis on this would benefit the paper.     \n\n**A1:** We agree that more analysis could enhance the interpretation of these results. In response, we have revised the manuscript and inserted the following paragraph after line 236:    \n*However, when using only a small number of scanpaths (1 and 3), our model exhibits lower performance compared to BERT. This could be attributed to the present framework's token embedding rearrangement based on fixation order, resulting in the removal of certain tokens. This process could potentially lead to information loss, thereby impacting the model's performance. This observation underscores the significance of combining a certain quantity of scanpaths, enabling the model to gather information from various perspectives.*   \n&nbsp;       \n&nbsp;       \n\nQ2: In Yang and Hollenstein (2023), the LPM-AS performance on ETSA (table 1 in their paper) is 90.48 in F1, which differs from the reproduced 85.81. Are these two numbers comparable? If so, what's the reason for this discrepancy?       \n\n**A2:** These two numbers are not directly comparable due to differences in the evaluation protocols. In the paper by Yang and Hollenstein (2023), their evaluation protocol, as outlined in section 4.2, involves a single train-test split. Since there is no standard train-test split for this dataset, we assume that they have conducted random subset selections for testing, followed by 25 runs with varying random model initializations. In contrast, for our reproduction, we chose a more robust approach: 10-fold cross-validation, aimed at providing a more comprehensive and accurate evaluation of the model's performance. These differing evaluation methodologies explain the observed performance discrepancy.   \n&nbsp;           \n&nbsp;      \n\nQ3: Any insights behind the observation that, in Figure 2, the performance peak is when the number of scanpaths is 7 and the performance degrades when considering more scanpaths?     \n\n**A3:** \n1) The number of scanpaths used by the model is regarded as a hyperparameter in our experiments. To clarify this, we have inserted the following footnote in line 220 after the words 'seven scanpaths':   \n*The optimal number of scanpaths to be used by the model is considered a hyperparameter for the subsequent experiments.*    \nIn Appendix Table 3, when examining the optimal number of scanpaths used for our model across different tasks within the GLUE benchmark, We observe that the optimal number of scanpaths for peak performance is likely linked to the size of the training sentences. In cases where the training sentences are limited in size, a higher number of scanpaths is often more helpful in enhancing the model's generalization performance.     \n\n2) We hypothesis that the observed performance degradation when incorporating more scanpaths could be attributed to a potential bias introduced by excessive scanpath inclusion. This bias may lead the model to overly focus on scanpath patterns, shifting its attention away from refining meaningful token and sentence representations and instead adapting to the variability in scanpaths.   \n&nbsp;              \n&nbsp;       \n\nQ4: In the low-resource setup for GLUE, why not use the original test set for testing?    \n\n**A4:** In the low-resource setting, we aimed to reduce variance by conducting 5 runs with distinct random seeds and averaging their results. However, this led to the need to submit 15 full results (one full result including all nine tasks) for the three values of the training data size K which we investigated in the low-resource setting, while the leaderboard allowed only 2 submissions per day. Due to the tight deadline, we decided to prioritize submitting results to the leaderboard for the high-resource setting, ensuring we could meet the evaluation deadline promptly.    \n&nbsp;                   \n&nbsp;          \n\nQ5: What’s the train/dev/test split in ETSA dataset?     \n\n**A5:** Since the ETSA dataset lacks a standard/predefined train-test split and its size is relatively modest, we opted for a 10-fold cross-validation evaluation approach (line 186).  We split the text into 10 parts, and for each CV fold, we use 9 parts for training and keep one part for testing the model. Additionally, we selected 10\\% of the training data as dev data to perform early stopping during training. Our rationale for employing cross-validation stems from its ability to deliver a more comprehensive and accurate evaluation of the model's performance. Mathematically, this approach reduces both bias and variance in the performance estimation, in contrast to the single random train-test split utilized in prior research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Novelty of the proposed model - the integration of the scanpath prediction model and the eye-tracking featured model alleviates the data scarcity problem and is shown promising for NLP understanding tasks.\n* Paper is well-written, well-structured and motivated.\n", "reasons_to_reject": "- One concern is that when the number of scanpath is less than 5 (i.e., 1 and 3 in Figure 2), the proposed model performs worse than the underlying models (i.e., BERT and PLM-AS) and the strong baseline ScanTextGan. This indicates that somehow the model relies on the aggregation of predicted scanpaths. More detailed analysis on this would benefit the paper.\n- Other minor concerns are listed in the question list below.\n", "questions_to_authors": "- Q1: In Yang and Hollenstein (2023), the LPM-AS performance on ETSA (table 1 in their paper) is 90.48 in F1, which differs from the reproduced 85.81. Are these two numbers comparable? If so, what's the reason for this discrepancy?\n- Q2: Any insights behind the observation that, in Figure2, the performance peak is when the number of scanpaths is 7 and the performance degrades when considering more scanpaths?\n- Q3: In the low-resource setup for GLUE, why not use the original test set for testing?\n- Q4: What’s the train/dev/test split in ETSA dataset?\n", "paper_topic_and_main_contributions": "This paper focuses on utilizing gaze data to improve NLP understanding tasks. The authors propose a unified model that integrates a scanpath generation model (Deng et al., 2023) and the PLM-AS framework (Yang and Hollenstein, 2023), which is a language model augmented with human scanpaths. By synthesizing the gaze data, the proposed model can be applied to the tasks without the annotations of human eye movement, effectively alleviating the data scarcity issue. It is shown that the proposed model performs on par with PLM-AS that takes use of real gaze data on sentiment classification. In addition, the model performs better than the baseline BERT on GLUE benchmark in a low-resource setup.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "41fvN3w0Vz", "reviewer": "Reviewer_biQN", "summary": "", "questions": "Question A: In l. 264, you say that your model consistently outperforms BERT across tasks, except for STS-B. However, QNLI with K=200 BERT wins with a noticeable difference of almost 3 points. Do you have a hypothesis for the better performance of BERT for QNLI with K=200?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We would like to answer your questions as follows.\n&nbsp;   \n&nbsp; \n    \nQ1:  In l. 264, you say that your model consistently outperforms BERT across tasks, except for STS-B. However, QNLI with K=200 BERT wins with a noticeable difference of almost 3 points. Do you have a hypothesis for the better performance of BERT for QNLI with K=200?    \n\n**A1:** While we may not have a specific hypothesis for this particular task, we can provide some general insights. In our approach, the Eyettention model (used for generating the synthetic scanpaths) is first pre-trained on a natural reading task and subsequently fine-tuned to adapt it to specific NLP tasks. However, in certain cases, such as specific tasks or data domains, the fine-tuning process might be challenging due to limited training data. As a result, the model may not be able to fully adjust and generalize effectively in those cases.     \n\nFuture research could explore the pre-training (or fine-tuning) of the Eyettention model on task-specific eye movement data rather than natural (i.e., task-unspecific) reading, even with limited data. This approach may result in a better downstream performance for NLP tasks that are similar to the tasks the humans performed during the recording of their eye movements.    \n&nbsp;       \n&nbsp;       \n\n\nQ2: Typos Grammar Style And Presentation Improvements.   \n1. Typo: 295: eyet-racking --> eye-tracking\n2. I think the accessibility of the \"Joint Modeling for NLP tasks\" paragraph could be much improved by placing Fig. 1 on page 2, so it can be used as a visual orientation. It would also bring it closer to Sec. 3.2.\n3. Space allowing, a footnote about the possibility and the potential effects of using models other than BERT_{BASE} (l. 135) could be added.    \n\n**A2:** Thank you for pointing this out. We have carefully revised the manuscript according to the reviewer's comments as follows:\n1. Typo: line 295, *eyet-racking* has been replaced by *eye-tracking*.\n2. Figure 1 has been positioned on page 2.\n3. A footnote has been added in line 135 after BERT_BASE: *Note that BERT can be substituted by other state-of-the-art pre-trained large language models from which (subword-) token embeddings can be extracted, potentially leading to further enhancements in task performance.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Improvement of the performance of a pre-trained LM through synthetic gaze data is achieved:\n* Synthetic gaze data does comes at a cheaper price as gaze data obtained from humans\n* Performance improvement on GLUE w.r.t unaugmented BERT is shown to carry over between different NLP tasks, especially in a low-resource scenario with less training data", "reasons_to_reject": "No reasons to reject.", "questions_to_authors": "Question A: In l. 264, you say that your model consistently outperforms BERT across tasks, except for STS-B. However, QNLI with K=200 BERT wins with a noticeable difference of almost 3 points. Do you have a hypothesis for the better performance of BERT for QNLI with K=200?", "paper_topic_and_main_contributions": "This paper presents an architecture for integrating synthetic gaze data (generated scanpaths) into a pre-trained language model, combining previous work on generating scanpaths with previous work on augmenting a language model with scanpaths obtained from humans. The model is evaluated on various NLP tasks, and shown to improve on the BERT baseline, especially in low-resource scenarios. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Typo: 295: eyet-racking --> eye-tracking\n* I think the accessibility of the \"Joint Modeling for NLP tasks\" paragraph could be much improved by placing Fig. 1 on page 2, so it can be used as a visual orientation. It would also bring it closer to Sec. 3.2.\n* Space allowing, a footnote about the possibility and the potential effects of using models other than BERT_{BASE} (l. 135) could be added."}}]

Author rebuttal:

Meta-review: This paper explores the question of whether scanpaths (from eye-tracking data) can improve various NLU tasks when the scanpaths used are fully synthetic, based on well known models of human scanpath data. The results show that the answer is yes, though the improvements diminish - unsurprisingly - as the amount of training data increases.

Overall this is a very interesting result, and my opinion as AC is that the authors have addressed the majority of the critical concerns brought up by reviewers. My only additional concern is that this paper does not really fit in "human centered NLP" as the point of this paper is exactly to remove humans from the process, not figuring out how to center humans. So I would suggest moving tracks.
Decision: Accept-Main