An Eye Opener Regarding Task-Based Text Gradient Saliency (2024)

Abstract: Eye movements in reading reveal humans’ cognitive processes involved in language understanding. The duration a reader’s eyes fixate on a word has been used as a measure of the visual attention given to that word or its significance to the reader. This study investigates the correlation between the importance attributed to input tokens by language models (LMs) on the one hand and humans, in the form of fixation durations, on the other hand. While previous research on the internal processes of LMs have employed the models’ attention weights, recent studies have argued in favor of gradient-based methods. Moreover, previous approaches to interpret LMs’ internals with human gaze have neglected the tasks readers performed during reading, even though psycholinguistic research underlines that reading patterns are task-dependent. We therefore employ a gradient-based saliency method to measure the importance of input tokens when LMs are targeted on specific tasks, and we find that task specificity plays a crucial role in the correlation between humanand model-assigned importance. Our implementation is available at https://github.com/gjwubyron/Scan.

heading: 1 Introduction
text: Human eye movements during reading reflect cognitive processes involved in language processing (Just and Carpenter, 1980; Rayner, 1998): the fixation duration on a word correlates with reading comprehension (Rayner, 1977; Malmaud et al., 2020a). As such, fixation duration has been employed as proxy of the relative importance of a word to a reader (Hollenstein and Beinborn, 2021).
The introduction of neural attention mechanisms (Bahdanau et al., 2014) and the Transformer architecture (Vaswani et al., 2017), which relies on self-attention to compute input and output representations, has given fresh impetus to research into how language models (LMs) process language.
Attention mechanisms assign dynamic weights to input tokens, offering a method to understand a model’s internal functioning and decision-making processes (Wang et al., 2016; Ghaeini et al., 2018).
Recent research has compared model and human language comprehension by aligning model attention weights with human reading metrics, such as fixation durations (Sood et al., 2020; Eberle et al., 2022; Bensemann et al., 2022), presuming model attention effectively signifies the relative importance of input tokens. However, the findings are mixed (cf Section 2). While some studies (Sood et al., 2020) observed significant differences between transformer LMs’ attention patterns and human fixation patterns, other studies (Eberle et al., 2022; Bensemann et al., 2022) found strong correlations. Besides, research on attention (Jain and Wallace, 2019; Serrano and Smith, 2019; Brunner et al., 2019) has questioned the reliability of attention weights in accurately reflecting token significance.
In contrast, Hollenstein and Beinborn (2021) utilized gradient-based saliency (Simonyan et al., 2014; Li et al., 2016) to approximate relative importance in LMs through iterative token masking and discovered strong correlation between LMs gradient-based saliency and human fixation durations. However, the output space of this approach comprises tens of thousands of tokens, which could make gradient-based saliency uninformative (Yin and Neubig, 2022). Moreover, their work focused on natural reading. Since psycholinguistic studies show that human reading strategies vary with the task and differ from normal reading (Malmaud et al., 2020b; Shubi and Berzak, 2023; Mézière et al., 2023), it is crucial to take task specificity into account.
In this work, we align the LMs with the same tasks performed by human participants during taskspecific reading and measure the importance of
255
input tokens using gradient-based saliency. Additionally, we expand our analysis to include decoderbased LMs, which, due to their auto-regressive nature, more closely mirror the incremental nature of human processing. We find strong correlations between LMs and humans in this task-specific setting, and further fine-tuning on the task can enhance these correlations.

heading: 2 Related Work
text: Model attention and human attention Research comparing model attention to human visual attention, using fixation locations and durations as proxies, has produced mixed findings. Sood et al. (2020) observed distinct differences between transformer LM attention patterns and human fixation patterns. Conversely, studies by Eberle et al. (2022) and Bensemann et al. (2022) found strong correlations between early transformer layer attention weights, like those in BERT (Devlin et al., 2019), and human visual attention, contrasting with earlier results. This discrepancy can be attributed to methodological differences in processing attention weights: Sood et al. (2020) analyzed maximum attention values from the last layer’s sub-word tokens, while Bensemann et al. (2022) averaged attention across sub-word tokens in the first layer.

heading: Limitations of attention-based interpretation
text: The inconsistent results outlined above challenge the usefulness of methods based on model attention to investigate the internals of LMs. Indeed, Brunner et al. (2019) emphasize the lack of token identifiability as one moves to higher layers of a model, and Abnar and Zuidema (2020) show that distinct attention patterns are only found in earlier layers, while in higher layers the attention weights approximate a uniform distribution. Moreover, Jain and Wallace (2019) question whether attention weights can reliably identify the relative importance of inputs to the entire model, showing that different attention distributions yield equivalent model predictions. Similarly, Serrano and Smith (2019) find attention weights to be very noisy indicators of importance. Finally, an analysis of BERT’s (Devlin et al., 2019) attention (Clark et al., 2019) reveals a significant focus on the [SEP] token, which does not affect model outputs when its attention is altered, suggesting a“no-op” operation. Similarly, research on attention heads (Voita et al., 2019; Michel et al., 2019) finds that many of them can be pruned with minimal impact, further indicating the potential
redundancy or non-operational nature of certain attention mechanisms.
Saliency-based methods for analyzing LMs with human gaze As saliency-based methods are arguably more suited than methods based on attention (Bastings and Filippova, 2020) for model analysis, Hollenstein and Beinborn (2021) extract token importance by iteratively masking each input token, computing the L2 norm of the gradient for the correct output with respect to each token, and then summing all saliency scores for each input token. However, while they do emulate the LM’s pre-training objective, this does does not necessarily align with human processing: whereas the model “sees” the input only partially, and as many times as there are tokens, the readers see the input fully and only once. Moreover, the gaze data used in their study was, in parts, recorded while participants were completing a task, such as sentiment analysis and relation extraction (i.e., task-specific reading). In our approach, we thus compute gradients by having the model perform the same kind of classification task that humans performed during reading. Thereby the token importance attributed by both humans and the model refers to the importance within the constraint of a specific task, and the model sees the input only once, and fully.

heading: 3 Method
text: Consider an input sentence, formalized as x = ⟨x1, . . . , xN ⟩ of N tokens, where xj is the jth token (word) in the sentence, and two corresponding token importance vectors of the same length: the human importance vector h = ⟨h1, . . . , hN ⟩ and the model importance vector m = ⟨m1, . . . ,mN ⟩, where hj and mj are the human and model importance attributed to token xj . We obtain the mean Spearman correlation between model and human importance by computing the by-token Spearman correlations between the vectors m and h for all sentences x, then dividing the sum of these correlations by the number of sentences x.
Extracting model importance: gradient-based saliency The model importance vector m consists of gradient saliency values mj for each input token xj of the sentence x. “Saliency” refers to neural network interpretation methods that assign an importance distribution over the input in order to analyze a network’s prediction (Ding and Koehn, 2021). In other words, saliency methods aim at ex-
BERT base BERT large RoBERTa DistilBERT GPT-2 base GPT-2 large OPT Sentiment Analysis (SA)
fine-tuned 0.610.010 0.570.011 0.470.012 0.530.011 0.490.011 0.550.010 0.430.012 pre-trained (0-shot) 0.550.011 0.590.010 0.450.012 0.520.012 0.40.014 0.480.012 0.420.013 random init. (0-shot) 0.240.013 0.220.013 0.040.014 0.210.013 0.20.014 0.190.014 0.150.015 Relation Extraction (RE) fine-tuned 0.530.010 0.520.009 0.420.010 0.450.010 0.460.010 0.520.009 0.50.011 pre-trained (0-shot) 0.510.010 0.470.011 0.370.011 0.490.010 0.370.011 0.450.011 0.420.011 random init. (0-shot) 0.080.011 0.070.011 0.040.012 0.090.011 0.160.013 0.160.013 0.140.014
Table 1: We report mean Spearman correlations and standard errors between model and human importance for all models in their fine-tuned, pre-trained (0-shot), and randomly initialized (0-shot) version, for both tasks SA and RE. The difference in correlations is significant in all cases except for the ones indicated in italic.
plaining how sensitive the decision of a model is to changes in the input. The most common method of assigning this importance distribution is by means of the gradient (Simonyan et al., 2014). Given a parametrized language model fθ, we compute the gradient g with respect to an input token xj ∈ x as
g(xj) := ∂f cθ ∂xj (x), (1)
where c indexes the true class y in the model’s output, and f cθ refers to the predicted output logit for the true class y. We then follow Li et al. (2016) by defining the gradient saliency mj of token xj as the L1 norm of its gradient mj := |g(xj)|. Since different LMs employ different tokenization methods which split tokens into sub-word tokens (Sennrich et al., 2016; Song et al., 2021), we pool gradients back to token level by summing up the sub-word token-level gradient norms. We then normalize the token-level saliencies by dividing them by the sum of all saliency values in the sentence.
Extracting human importance: relative fixation duration To obtain the human importance vector h, we first extract raw total fixation durations tj,r for each token xj ∈ x, which is the sum of the durations of all fixations on that token by a reader r. However, due to variations in reading speed across readers and sentences, these raw durations can vary significantly between instances. We thus normalize them by dividing them by the sum of durations across all tokens within a sentence, resulting in relative fixation durations dj,r = tj,r/ ∑ j tj,r for each token xj . These relative durations are then averaged across all readers to bypass individual differences and to obtain a more robust signal, resulting in aggregated relative fixation durations hj = ∑ r dj,r/ |readers| for each token xj .

heading: 4 Experiments
text: Datasets The eye-tracking part of the Zurich Cognitive Language Processing Corpus (ZuCo; Hollenstein et al., 2018) comprises two task-specific readings: in the sentiment analysis (SA) reading, participants were presented with a subset from the Stanford Sentiment Treebank (SST; Socher et al., 2013) that consists of movie reviews, based on which they had to rate the movies; in the relation extraction (RE) reading, they performed relation extraction on a subset of sentences from the Wikipedia relation extraction corpus (Culotta et al., 2006).
Models and fine-tuning We include both encoder models and decoder models, as well as models from the same family but different in size. Encoders include BERT (Devlin et al., 2019) base and large, RoBERTa (Liu et al., 2019), and DistilBERT (Sanh et al., 2019); decoders include GPT-2 (Radford et al., 2019) base and large, and OPT (Zhang et al., 2022). As the models perform classification — ternary for SA, and 9-class for RE —, we utilize the architecture variants implemented for sequence classification in Huggingface (Wolf et al., 2019). For SA, we fine-tune the models on the SST dataset and for RE on the Wikipedia dataset (Culotta et al., 2006) , excluding the sentences used for ZuCo SA and RE, respectively.1
Baselines. We include two sets of baseline models: the above-mentioned models randomly initialized (random (0-shot)), and the models pre-trained but not fine-tuned (pre-trained (0-shot)).
Results As depicted in Table 1, the more similar the model’s training is to the human task, the more aligned are the model and human importance vectors. There exist medium to strong correlations between the fine-tuned model importance and human
1For training and implementation details as well as classification test results, see Appendix A.
importance vectors, exemplified by correlations of 0.61 by BERT base or 0.55 by GPT-2 large for SA. Additionally, most fine-tuned models produce significantly higher correlations than the pre-trained baselines, and the pre-trained models all have significantly higher correlations than their randomly initialized counterparts. Encoder models, on average, achieve higher correlations than decoders, despite variability within both types. Additionally, SA task model importance correlates more strongly on average than for RE.

heading: 5 Participant-level analysis
text: To investigate whether the models correlate more with certain participants, we perform an additional participant-level analysis in which we compute correlations between the model-extracted saliency values and relative fixation durations for each participant individually. We also extract the participants’ response accuracies for both their SA and RE, averaged over sentences. The underlying intuition is that the models possibly correlate more with participants that have a higher task accuracy.
Results The juxtaposition of correlations on participant level and participants’ accuracies reveals no discernible pattern, as exemplified by BERT base in Figure 1. The correlation coefficients between participants exhibit great variability in both tasks. Participants’ task accuracies are distributed across a wide range for SA but exhibit a ceiling effect for RE. Moreover, averaging the participantlevel correlations yields lower correlation values than using the aggregate relative fixation durations, e.g., the group-level correlation with BERT base is 0.61 and the average on participant-level is 0.41.2
2An overview of all by-participant accuracies and correlations, for all models can be found in Table 3 in Appendix B.

heading: 6 Discussion and Conclusion
text: The experimental results find medium to strong correlations between model importance vectors, derived from gradient saliencies, and human importance vectors, indicated by relative fixation durations, particularly when language models (LMs) are fine-tuned for tasks mirroring those undertaken by readers: task-specific fine-tuned models demonstrate notably stronger correlations than pre-trained zero-shot baselines. The discrepancy between the pre-trained and randomly initialized models suggests an initial understanding for human importance attribution acquired during pre-training. These findings underline the importance of matching tasks between models and humans for accurate gaze analysis, with task-specificity influencing reading behavior but remaining largely ignored in NLP (Shubi and Berzak, 2023). We further find that SA tasks show consistently higher correlations than RE, possibly due to the complexity introduced by more output classes affecting model predictions. Moreover, initial observations suggest encoders outperform decoders in correlation, potentially due to decoders’ unsuitability for classification tasks. Yet, this distinction may be incidental, influenced by factors like pre-training data or model architecture. Surprisingly, BERT base yields the highest correlation, while BERT large and RoBERTa, who achieve higher test accuracies than BERT, produce lower correlations. This indicates that emulating human importance attribution is neither a function of model parameters nor does it necessarily imply better model performance. The participant-level analysis reveals no distinct pattern, indicating that the models do not mirror the token importance attribution of more proficient humans. Moreover, averaging correlations across individual participants re-
sults in a lower correlation value compared to when participant fixation durations are aggregated across sentences. This implies both that by-participant aggregation of relative fixation durations produces a more robust signal, and that models correlate more with average human language processing than with subject-level idiosyncracies.
In conclusion, we have developed a gradient saliency-based method to analyze LMs with human gaze that does not neglect task-specificity and found that mirroring tasks yields higher correlations.

heading: Limitations
text: First of all, the number of sentences in the eye gaze dataset is quite low, as is the number of readers (which are all L1 English readers based in Zurich, and are not experts in sentiment analysis or relation extraction), which does not make for a representative sample of the population at large.
Relatedly, for a more extensive evaluation of our task-specific approach, one would have to apply it to the same sentences that contain eye movements from natural reading instead of task-specific reading. We leave it to future work to extend the data from ZuCo with eye movements from natural reading.
Moreover, while the studies outlined in Section 2 underline the superiority of gradient-based over attention-based methods, they might still not be the state-of-the-art for explainability methods and one might employ methods such as Integrated Gradients or Layer-wise Relevance Propagation.

heading: Ethics Statement
text: Working with human data requires careful ethical considerations. The eye-tracking dataset utilized in this study follows ethical standards and has been approved by the responsible ethics committees. It is licensed under the Creative Commons Attribution 4.0 International Public License (CC BY 4.0).

heading: Acknowledgements
text: We thank the anonymous reviewers for their comments. This work was partially funded by the Swiss National Science Foundation (SNSF) under grant IZCOZ0_220330/1 (EyeNLG), and by the German Federal Ministry of Education and Research under grant 01| S20043 (AEye), and is supported by COST Action MultiplEYE, CA21131, supported
by COST (European Cooperation in Science and Technology).

heading: Appendices

heading: A Fine-Tuning Details
text: We fine-tune the models outlined in Section 3 on the SST (Socher et al., 2013) dataset for ternary sentiment classification, excluding the sentences used for ZuCo SA, and on the Wikipedia dataset (Culotta et al., 2006) for 9-class relation classification, excluding the sentences used for ZuCo RE. After excluding sentences from ZuCo SA and RE, we are left with 5211 sentences allocated for SA and 889 sentences allocated for RE. Subsequently, we implement an 80/20 split for training and validation. For testing, there are 400 sentences from ZuCo SA and 335 sentences from ZuCo RE 3. We train the models for 10 epochs, with an early stopping patience of 3 epochs, using the AdamW (Loshchilov and Hutter, 2019) optimizer, a learning rate of 2 ∗ 10−5, and a batch size of 16. All models are implemented in PyTorch (Paszke et al., 2019).
B Participant-Level Analysis
(a) BERT large
(b) DistilBERT
(c) RoBERTa
3Out of the original 407 sentences in ZuCo RE, we retain only 335 sentences that contain a specific relation.

heading: Future Work
text: We leave it to future work to extend the data from ZuCo with eye movements from natural reading. Moreover, while the studies outlined in Section 2 underline the superiority of gradient-based over attention-based methods, they might still not be the state-of-the-art for explainability methods and one might employ methods such as Integrated Gradients or Layer-wise Relevance Propagation