Fine-Tuning Pre-Trained Language Models with Gaze Supervision (2024)

Abstract: Human gaze data provide cognitive information that reflect human language comprehension, and has been effectively integrated into a variety of natural language processing (NLP) tasks, demonstrating improved performance over corresponding plain text-based models. In this work, we propose to integrate a gaze module into pre-trained language models (LMs) at the fine-tuning stage to improve their capabilities to learn representations that are grounded in human language processing. This is done by extending the conventional purely text-based fine-tuning objective with an auxiliary loss to exploit cognitive signals. The gaze module is only included during training, retaining compatibility with existing pre-trained LMbased pipelines. We evaluate the proposed approach using two distinct pre-trained LMs on the GLUE benchmark and observe that the proposed model improves performance compared to both standard fine-tuning and traditional text augmentation baselines. Our code is publicly available.1

heading: 1 Introduction
text: As humans read, the unconscious cognitive processes that unfold in their minds while comprehending the stimulus text are reflected in their eye movement behavior (Just and Carpenter, 1980). These gaze signals hold the potential to enhance NLP tasks. Research has focused on using aggregated word-level gaze features to enrich text features (Barrett et al., 2016; Mishra et al., 2016; Hollenstein and Zhang, 2019) or to regularize neural attention mechanisms, making their inductive bias more human-like (Barrett et al., 2018; Sood et al., 2020, 2023).
Moreover, there has been growing interest in adopting non-aggregated scanpaths (i.e., sequences
1https://github.com/aeye-lab/ ACL-GazeSupervisedLM
of consecutive fixations) to augment LMs. These scanpaths capture the complete sequential ordering of a reader’s gaze behavior and approximate their attention. Mishra et al. (2017) and Khurana et al. (2023) employed neural networks to independently encode scanpaths and text, followed by the fusion of the features extracted from both modalities. Yang and Hollenstein (2023) proposed rearranging the contextualized token embeddings produced by pre-trained LMs based on the order in which the reader fixates on the words, followed by applying sequence modeling to the reordered sequence. To tackle the issue of gaze data scarcity, Deng et al. (2023a) explored the possibility of augmenting LMs using synthetic scanpaths, generated by a scanpath generation model. Remarkably, synthetic scanpaths demonstrated advantages across various NLP tasks, particularly in settings with limited labeled examples for the downstream task.
In contrast to previous studies that concentrated on learning joint cross-modal representations of text and scanpath, we start from a different perspective and explore utilizing gaze data to improve on
217
the learned text representations of pre-trained LMs during the fine-tuning stage, without incurring additional computational effort when using the model at application time. To this end, we extend the standard pre-trained LM fine-tuning objective with an auxiliary loss by integrating a scanpath module, which serves a dual purpose. First, the auxiliary loss can effectively incorporate human-like gaze signals generated using a scanpath generation model and thus provide informative gradients to guide the LM towards more representative local minima. Second, reordering the token-embedding sequence based on the fixation sequence can diversify textual information, potentially improving generalization performance (Xie et al., 2020). This stands in contrast to heuristic text augmentation strategies, like random word insertion, replacement, swapping, and deletion (Wei and Zou, 2019; Xie et al., 2020). Scanpaths inherently contain cognitive information that better aligns with and complements textual information.
Notably, our proposed gaze module is only active during training (fine-tuning), ensuring alignment with the standard usage of LMs after this stage. This offers two key benefits. First, it facilitates seamless integration with existing LMbased pipelines. Second, at deployment time, it eliminates the need to either collect real-time gaze recordings, which is costly and impractical for most use-cases, or generate synthetic gaze data, which is often computationally challenging for devices with limited computational resources.
On the General Language Understanding Evaluation (GLUE) benchmark, our proposed gazeaugmented fine-tuning outperforms both standard text-only fine-tuning and traditional text augmentation baselines, without incurring additional computational effort at application time.

heading: 2 Method
text: In this section, we start out with a brief description of the conventional fine-tuning procedure for Transformer-based encoders on downstream tasks. Subsequently, we introduce our method, and explain how it incorporates synthetic scanpaths into this fine-tuning procedure to enhance representation learning of Transformer-based encoders. The overall model architecture is illustrated in Figure 1.
Preliminaries Our learning objective is to solve standard multi-class classification or regression problems. We assume access to a Transformer-
based pre-trained LM like BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019). In the conventional fine-tuning approach for downstream tasks, the pre-trained LM is adapted to a specific task by fine-tuning all the parameters end-to-end using task-specific inputs and outputs. The final hidden state of the “[CLS]” token typically serves as the aggregated sentence representation, which is then fed into a newly initialized (series of) dense layer(s) with output neurons corresponding to the number of labels in the task. We minimize the standard cross-entropy loss for classification and mean-squared-error loss for regression, denoted as Lstandard in Figure 1. Scanpath Integration We extend the standard fine-tuning framework by integrating a scanpath module. The design of the scanpath module follows the prior work of Deng et al. (2023a) and Yang and Hollenstein (2023). Specifically, the Transformer encoder produces contextualized token embeddings for a given sentence, with each embedding associated with its position index in the sequence. Simultaneously, a synthetic scanpath (fixation-index sequence) is generated based on the same sentence using the scanpath-generation model Eyettention (Deng et al., 2023b), which has demonstrated effectiveness in simulating humanlike scanpaths during reading (see Appendix A for detailed information about the Eyettention model). The scanpath module then rearranges the tokenembedding sequence based on the simulated fixation sequence. Subsequently, we use a scanpath encoder, implemented as a layer of Gated Recurrent Units (GRU), to process the reordered sequence. The output from the last step of the scanpath encoder is then forwarded to the subsequent dense layer. For the branch that takes the scanpath into account, we introduce an additional loss term, referred to as Lscanpath in Figure 1, which represents the cross-entropy loss for classification and the mean-squared-error loss for regression.
Training Objective We combine the standard purely text-based loss and the scanpath-integrated loss with a trade-off factor λ. The final training objective is defined as:
L := Lstandard + λLscanpath. The joint optimization of the two branches facilitates the flow of cognitive information from the scanpath module to the Transformer through backpropagation, thereby improving its capability to
process and comprehend text. Consequently, during testing, we can remove the scanpath module and generate predictions solely from the Transformer and the final dense layer. This ensures alignment with standard LM usage after the fine-tuning stage, notably preserving its intrinsic efficiency and compatibility.

heading: 3 Experiments

heading: 3.1 Evaluation Setup
text: Data Sets We conduct experiments on the GLUE benchmark (Wang et al., 2018), including sentiment analysis (SST-2), linguistic acceptability (CoLA), similarity and paraphrase tasks (MRPC, STS-B, QQP), and natural language inference tasks (MNLI, QNLI, RTE).
Model and Data Setup We use BERTbase (Devlin et al., 2019) and RoBERTabase (Liu et al., 2019) as the base models in the experiments. We primarily focus on a low-resource setting where only limited labeled examples for the downstream task are available. In such cases, effective finetuning strategies are crucial to enable high-capacity LMs to learn more informative representations for enhanced performance in downstream tasks (Zhang et al., 2021). For each task, we sample a small subset of training instances with sizes K = {200, 500, 1000}. We take an additional 1,000 instances from the original training set as the development set and use the original development set for testing. Additionally, we consider a high-
resource setting where we use the entire training set and report the results on the GLUE development sets. Appendix B gives further details about training and hyper-parameter tuning.
Baselines We compare our proposed method with the standard text-only fine-tuning using only Lstandard as the training objective. Moreover, we compare to the Easy Data Augmentation (EDA) method (Wei and Zou, 2019), which randomly performs word insertion, replacement, swap, and deletion in the text to augment the training data.

heading: 3.2 Results
text: Low-Resource Performance Table 1 shows that, overall, our scanpath-augmented fine-tuning (+SP) consistently outperforms the standard fine-tuning and EDA baselines, regardless of the number of training instances. We observe performance gains of 2-3% for BERT and 1-2% for RoBERTa over standard fine-tuning. At the per-task level, our method outperforms standard fine-tuning across all tasks in all setups for BERT, and on five, five and four out of eight tasks when trained with 200, 500, and 1,000 instances, respectively, for RoBERTa. The improvements are larger with fewer training instances, indicating the efficacy of our method in low-resource scenarios. Notably, for tasks like CoLA and STS-B, where the EDA method yields largely inferior results compared to standard finetuning (Model=BERT), our method shows superior performance. This suggests that the scanpath,
which inherently contains cognitive information, aligns with and complements textual information effectively.
High-Resource Performance In Table 2, we present the results of different methods when using all training instances. Our scanpath-augmented fine-tuning (+SP) achieves the highest overall performance. While the gains are not as significant as in the low-resource setting for most tasks, notable improvements persist for tasks like CoLA and RTE. In contrast, the EDA method fails to enhance performance over standard fine-tuning overall, which is in line with findings from previous research (Longpre et al., 2020).

heading: 3.3 Ablation Studies
text: Location of the Scanpath Module We explore the impact of integrating the scanpath module at different feature-representation levels on the model’s performance. Specifically, we experiment with placing the scanpath module after the 11th, 8th, 5th, and embedding layer of the Transformer. In these cases, it is straightforward to use the subsequent Transformer layers to process the scanpath-guided reordered sequence; we therefore remove the scanpath encoder from the module. Moreover, we add extra positional embeddings to the token embeddings after the rearrangement, providing information about the positions of tokens in the sequence.
Table 3 shows that integrating the scanpath module into the model, regardless of its placement, yields improved performance compared to standard text-only fine-tuning. However, placing it at a lower position within the Transformer results in smaller gains. This may be attributed to the top Transformer layers capturing richer semantic information (Jawahar et al., 2019). Placing the scanpath module at the top facilitates better access to this information, potentially aiding in leveraging cognitive information. Furthermore, adding extra positional information to the reordered sequence marginally impacts performance.
Scanpath vs Random Order The core principle of the scanpath module is to utilize the order of fixations to integrate estimated cognitive information into the model. To study whether the observed gains truly arise from the order of fixations, we compare our method which rearranges the tokenembedding sequence based on the scanpath to two baselines: (1) shuffling the scanpath ordering, and (2) randomly shuffling the token-embedding sequence. Table 4 shows that shuffling the scanpath results in consistent performance drops across all tasks, indicating the importance of the order of fixations. Furthermore, excluding the scanpath and randomly shuffling BERT token embeddings leads to a large decrease in performance gain, underscoring the importance of both fixated words and their order in enhancing model performance.

heading: 4 Conclusion
text: Our work contributes to the broad effort of enriching NLP models by grounding them in various domains of experience. Specifically, we focus on the use of scanpath data, demonstrating its vital role in enhancing textual representation learning. By extending the standard pre-trained LM finetuning objective with a scanpath-integrated loss, we ground the LM in human language processing. Finally, our experiments show that the proposed method surpasses standard fine-tuning and EDA baselines on the GLUE benchmark, pointing to the potentially promising future direction of enriching textual representations with gaze data, especially for low-resource tasks and languages (Reich et al., 2024). However, it should be noted that the performance gains achieved by incorporating gaze supervision vary across different NLP tasks. Future work may include further analysis of the impact of incorporating cognitive information into language models on specific downstream tasks.
Limitations
One limitation of our work is that the scanpathgeneration model—Eyettention—was pre-trained on a single eye-tracking corpus with a relatively small sample (see Appendix A). Participants read sentences covering only a single domain and a narrow range of text difficulty levels. This limitation may restrict the knowledge acquired by Eyettention concerning human language processing, thus potentially leading to limited benefits when integrating simulated gaze data into LMs. In our experiments, we observe that our proposed fine-tuning scheme provides smaller benefits to RoBERTa than BERT, even in the low-resource setting. The key difference between these models is the scale of unsupervised pre-training. We hypothesize that RoBERTa which is pre-trained on a larger scale of data has learnt sufficiently robust language representations, and to further improve its representation learning capability, a more competitive scanpath-generation model, trained on a large eye-tracking dataset that covers diverse domains of texts, might be required.
Furthermore, it is worth exploring the performance of the proposed approach when using other state-of-the-art scanpath generators. Different architectures have been developed recently in the field (Bolliger et al., 2023; Khurana et al., 2023). Exploring the strengths and weaknesses of different scanpath generators when integrated into LMs
could provide valuable insight into the development of improved scanpath generators for benefiting NLP tasks.
Ethics Statement
It is essential to acknowledge potential privacy risks in the collection, sharing, and processing of human gaze data. Due to the highly individual nature of eye movements, there exists a possibility of extracting sensitive information such as a participant’s identity (Jäger et al., 2020; Makowski et al., 2021), gender (Sammaknejad et al., 2017) and ethnicity (Blignaut and Wium, 2014) from gaze data, posing a risk of privacy leakage. The use of synthetic gaze data can help alleviate the necessity for large-scale experiments involving human subjects, although some amount of human gaze data remains necessary to train generative models.

heading: Acknowledgements
text: This work was partially funded by the German Federal Ministry of Education and Research under grant 01| S20043.

heading: A Model Details
text: Scanpath Generation Model For the utilization of the scanpath generation model Eyettention, we follow the work of (Deng et al., 2023a). The training process for the Eyettention model is conducted in two phases. First, we pre-train the Eyettention model on the L1 subset of the CELER corpus (Berzak et al., 2022), which comprises eyetracking recordings collected from native speakers of English during natural reading sentences. Second, the Eyettention model is fine-tuned on downstream NLP tasks. More specifically, in our proposed scanpath-augmented fine-tuning scheme, we fine-tune the Transformer encoder and the Eyettention model, as well as train the scanpath encoder and the final dense layer from scratch. We tailor the parameters of Eyettention for specific downstream tasks, aiming to provide targeted inductive biases. For further details on the Eyettention model, please refer to (Deng et al., 2023b,a).
In our experiments, we evaluate our proposed approach using two distinct pre-trained LMs, BERT and RoBERTa, each equipped with its unique tokenizer. The Eyettention model includes a pretrained LM in the text encoder for embedding the stimulus sentence. The generated fixation sequence (token index sequence) is based on the specific tokenizer associated with the pre-trained LM used. To facilitate a direct application of the arrangement operation based on the token-embedding sequence and fixation sequence without additional complex conversion, we maintain consistency by using the same pre-trained LMs in the Eyettention text encoder when evaluating specific pre-trained LMs as our base models. By replacing BERT with RoBERTa in the Eyettention text encoder, we observe a similar validation loss in scanpath prediction on the CELER corpus.
Scanpath Encoder The scanpath encoder is composed of a unidirectional GRU layer (Cho et al., 2014) with a hidden size of 768 and a dropout rate of 0.1. We initialize the hidden state of the GRU layer using the [CLS] token outputs from the final layer of the pre-trained LMs.

heading: B Training Details
text: We train all models using the PyTorch (Paszke et al., 2019) library on an NVIDIA A100-SXM4-40GB GPU using the NVIDIA CUDA platform. We use the pre-trained checkpoints from the Hugging-
Face repository (Wolf et al., 2020) for the language model BERTbase and RoBERTabase. The models are optimized using the AdamW optimizer (Loshchilov and Hutter, 2019). We set the maximum sequence length to 128 and the training batch size to 32.
In the high-resource setting, we train the models for 20 epochs and update the best checkpoint by measuring validation accuracy every 500 steps. For datasets with fewer than 500 steps per epoch, we update and validate at the end of each epoch. We tune the learning rates for BERT from {5e-5, 4e-5, 3e-5, 2e-5} and for RoBERTa from {3e-5, 2e-5, 1e5} for each task, following the recommendations in the original paper (Devlin et al., 2019; Liu et al., 2019).
In the low-resource setting, we train the models for 10 epochs and save checkpoints every epoch. We use the same learning rate that was found optimal in the high-resource setting for each task. We perform 5 runs with different data seeds ({111,222,333,444,555}) for shuffling, while the seed s=42 is consistently utilized for model training across all models.
In both high-resource and low-resource settings, for our proposed scanpath-augmented fine-tuning method, we conduct a hyperparameter search on the development set to determine the optimal tradeoff factor λ for each task, exploring values from {1, 0.7, 0.5, 0.3, 0.1, 0.01, 0.001}. For the EDA baseline, we tune the number of generated augmented sentences added to the original training set, exploring values from {1, 2, 4, 8, 16} based on the recommendations in the original paper (Wei and Zou, 2019).

heading: Future Work
text: Finally, our experiments show that the proposed method surpasses standard fine-tuning and EDA baselines on the GLUE benchmark, pointing to the potentially promising future direction of enriching textual representations with gaze data, especially for low-resource tasks and languages (Reich et al. , 2024). However, it should be noted that the performance gains achieved by incorporating gaze supervision vary across different NLP tasks. Future work may include further analysis of the impact of incorporating cognitive information into language models on specific downstream tasks. Limitations
One limitation of our work is that the scanpathgeneration model—Eyettention—was pre-trained on a single eye-tracking corpus with a relatively small sample (see Appendix A). Participants read sentences covering only a single domain and a narrow range of text difficulty levels. This limitation may restrict the knowledge acquired by Eyettention concerning human language processing, thus potentially leading to limited benefits when integrating simulated gaze data into LMs. In our experiments, we observe that our proposed fine-tuning scheme provides smaller benefits to RoBERTa than BERT, even in the low-resource setting. The key difference between these models is the scale of unsupervised pre-training. We hypothesize that RoBERTa which is pre-trained on a larger scale of data has learnt sufficiently robust language representations, and to further improve its representation learning capability, a more competitive scanpath-generation model, trained on a large eye-tracking dataset that covers diverse domains of texts, might be required. Furthermore, it is worth exploring the performance of the proposed approach when using other state-of-the-art scanpath generators. Different architectures have been developed recently in the field (Bolliger et al. , 2023; Khurana et al. , 2023). Exploring the strengths and weaknesses of different scanpath generators when integrated into LMs
could provide valuable insight into the development of improved scanpath generators for benefiting NLP tasks