Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models
Authors: Angela Lopez-Cardona
Carlos Segura
Alexandros Karatzoglou
Sergi Abadal
Ioannis Arapakis
Venue: ICLR 2025 Poster (2025)
Keywords: reward model
RLHF
visual attention
LLMs
eye tracking
implicit feedback
Primary area: foundation or frontier models, including LLMs
TL;DR: GazeReward: A novel framework that integrates implicit feedback into Reward Model (RM)

Abstract: Advancements in Natural Language Processing (NLP), have led to the emergence of Large Language Models (LLMs) such as GPT, Llama, Claude, and Gemini, which excel across a range of tasks but require extensive fine-tuning to align their outputs with human expectations. A widely used method for achieving this alignment is Reinforcement Learning from Human Feedback (RLHF), which, despite its success, faces challenges in accurately modelling human preferences. In this paper, we introduce GazeReward, a novel framework that integrates implicit feedback -- and specifically eye-tracking (ET) data -- into the Reward Model (RM). In addition, we explore how ET-based features can provide insights into user preferences. Through ablation studies we test our framework with different integration methods, LLMs, and ET generator models, demonstrating that our approach significantly improves the accuracy of the RM on established human preference datasets. This work advances the ongoing discussion on optimizing AI alignment with human values, exploring the potential of cognitive data for shaping future NLP research.

Published as a conference paper at ICLR 2025
SEEING EYE TO AI: HUMAN ALIGNMENT VIA GAZE- BASED RESPONSE REWARDS FOR LARGE LANGUAGE MODELS
Angela Lopez-Cardona Carlos Segura', Alexandros Karatzoglou', Sergi Abadal? and Ioannis
Arapakis
'Telefónica Scientific Research, Barcelona, Spain Universitat Politècnica de Catalunya, Barcelona, Spain
Amazonn, Barcelona, Spain
carlos seguraerras,,,,,,,,es, icannis eeee alexandros .aa.ssaaaaaaaaaa kara com, abadal Saadal@ac.oo upc edu
com,
ABSTRACT
Advancements in Natural Language Processing (NLP), have led to the emergence of Large Language Models (LLMs) such GPT, Llama, Claude, and Gemini, which excel across a range of tasks but require extensive fine-tuning to align their outputs with human expectations. A widely used method for achieving this align- ment is Reinforcement Learning from Human Feedback (RLHF), which, despite its success, faces challenges in accurately modelling human preferences. In this paper, we introduce GazeReward, novel framework that integrates implicit feed- back and specifically eye-tracking (ET) data into the Reward Model (RM). addition, we explore how ET-based features can provide insights into user pref- erences. Through ablation studies we test our framework with different integra- tion methods, L.M,s, and ET generator models, demonstrating that our approach significantly improves the accuracy of the RM on established human preference datasets. This work advances the efforts to optimise AI alignment with human values, and explores the potential of cognitive data to shape future NLP research.
INTRODUCTION
Recent advancements in Natural Language Processing (NLP) have led to the emergence of Large Language Models (LLMs) like GPT (OpenAl 2023). Llama (Dubey et al. 2024). Claude (An-) thropic 2024). and Gemini (Team al.) 2024), which excel across range of tasks. These models, often consisting of billions of parameters, are trained on massive datasets and typically require ex- tensive fine-tuning align their outputs with human expectations Several works have focused on refining the way LLMs interpret and respond to user intent (Wang et al. 20235). which has led to the development of novel alignment techniques. common approach to achieving human alignment involves leveraging explicit human feedback as preference information. Currently, the most widely adopted method is Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2024). RLHF has been implemented in many sate-offthhearr I.MMs (Cui et al. 2024 OpenAI 2023] Bai ct al. 2022b), and has been shown to help align models to human instructions and mitigate the gen- eration of toxic or harmful content (Kiegeland et al. 2024). However, persistent challenge with this approach the difficulty of acquiring sufficient high-quality training data (Casper et al. 2023).
To be able capture the complexities of real-world user instructions, there is need for meticu- lously handcrafted data (Wang et al. 2023b), which are resource-expensive and difficult to scale Yang et al. 2023). Obtaining high-quality feedback from human annotators, usually provided after
ILLMs that are trained only extensive datasets for language modeling referred to as "pre-trained" LLMss. Subsequent approaches, such human alignment, categorized "post-training".

---

Published as a conference paper at ICLR 2025
examining model response, suffers from several caveats (Casper et al. 2023 For instance, low inter-annotator agreement can result in inconsistent evaluations of the same model output due to varying interpretations, domain expertise, biases. Moreover, "scalable oversight" the ability to supervise models effectively with limited resources (Amodei et al.] 2016) remains an open prob- lem. Inconsistent data quality is another issue, as cost-quality tradeoff's often arise when collecting human feedback.
To address these challenges, researchers have increasingly turned to LLMs as form of Al-driven feedback, referred to as Reinforcement Learning from Al Feedback (RLAIF) Bai et al. (022b).
This method offers improved scalability, easier data collection, and cost-efficiecyy compared to traditional numan--riven approaches (Bai et al. [202bb] Wang et al., 2023a Madaan et al. 2023).
However, remains unclear what type of feedback signals, or a combination of feedback mecha-
nisms, is optimal to align LLM with human goals (Casper et al., 2023). More research is needed to explore the underlying beliefs and expectations of human users (Casper al.] 2023). and how
these can be incorporated into human alignment techniques. Furthermore, the alignment success of
a language model is dependent on the quality of the underlying RM (Pace et al. 2024). Various
alignment methods, such RLHF, RLAIF, and Direct Preference Optimization (DPO) (Rafailov
jet al., (023). rely on RM to incorporate feedback. Reward modelling is also essential for generat- ing synthetic data for preference alignment and is often used in LLM inference evaluate model outputs in techniques such as best-of-N sampling (Cui et al. 2024).
In this work, we propose a novel approach that incorporates Eye-tracking (ET) additional signal to address the challenge of human alignment. ET measures oculomotor behavior i.c. the movements and fixations of the eyes, which offers insight into visual attention and information pro- cessing (Kleinke 1986 Land & Furneaux] 1997). This allows researchers to correlate observable eye movement patterns with underlying cogmitive and perceptual processes during reading and lan- guage comprehension tasks (Kleinke, 1986; Krasich et al., 2018). Moreover, ET unlike other (explicit) forms of feedback (e.g., questionnaire data, data annotation) does not suffer from human biases, and offers better temporal and spatial resolution (Zhang & Hollenstein 2024). Several studies have shown a strong correlation between human eye movements and attention patterns in hansformer-basee models (Wang et al.) 2024a; Bensemann et al. 2022; Sood et al. 2020a). In- corporating ET data into NLP tasks has also proven valuable, as demonstrated by numerous works
(Huang et al. 2023 Khurana al.) 2023 Hollenstein al., 2019 Yang & Hollenstein 2023 Kiege-
land et al.] [2244] Deng et al.] [023a] a Mathias et al.) [018]] McGuire & Tomuro (021). Recently,
Kiegeland et al. (2024) proposed the integration of ET in controlled sentiment generation create a dataset that can be used in human alignment methods.
Gazo Souturie generation
Impul
Embeddinge OCTISE aution with syl--tacking featurrs
prompt
response
OOOO
OOOO
extedding
Comblree
O
madh to predictor
O aye OOOO
exteMiig
OOOO
Roward model
NN madiule
O Pye-tmni rg wmbbessingss
revrre
Comaineeenbecon
O
Reward model
Win Loss
Higher rwad pioteenco napotie
GazeReward Framework for using eye-tracking data for reward modelling. We use gen- erator model to compute ET features on a preference dataset D and we train the human preference by combining both text and ET embeddings (See section for details.)
Figure 1:
Human alignment remains one of the biggest challenges in the development of LLM, with RM playing important role in addressing this issue. This paper investigates how behavioural signals, particularly ET, can be operationalised as implicit feedback to improve human alignment (proposed approach is shown in Figure T). Furthermore, explore the use of ET prediction models that can
2

---

Published as a conference paper at ICLR 2025
generate automatically and with little effort ET features in response text input, which makes our solution not only cost-effective but also highly scalable. Our contributions are the following:
We propose GazeReward, novel and scalable framework that integrates implicit feedback in the form ET data into the RM, key component in modeling human preferences.
We perform for the first time an ablation study that examines several ffte-of-the-aa LLMs, various ET prediction models, and methods for incorporating ET features into the RM.
We demonstrate experimentally substantial performance improvements with the GazeRe- ward framework, showing accuracy gains of over 10% in RM predictions across diverse human preference datasets.
2 PRELIMINARIES
2.1 LARGE LANGUAGE MODELS- ALIGNMENT
LLMs-Human Alignment typically involves training LLMs on datasets curated by humans (learn- ing from human feedback data) (Ouyang al. 2024). This can be achieved through Supervised (SFT), where the model is traindd on pairs of prompts (x) and corresponding human- generated responses (3) (Liu et al. 2024). Alternatively, alignment can be pursued via preference optimization, using human preference dataset that differentiates between a better response (yw)
and worse one (z)) for the same prompt (x)): = {(x(f) you Vi
(2) (6) )
To this day, RLHF (Ouyang et al. 2024) remains the most popular technique used in state-of-the-ar LLMs like GPT-4 (OpenAl 2023). Claude (Bai et al., (022b), Bard (Google) 2023), and Llama 2-Chat TTuvrron et al. 2023). Different implementations of RI.FF can vary in terms of data col- lection, training processes, and choice of RI. algorithms. Typically, RLHF (Ouyang et al.) 2024) involves three main steps: (1) collecting feedback. (2) training RM based on that feedback, and (3) optimising the LLMs using RL techniques, such as Proximal Policy Optimization (PPO) Schullman
ct al. (2017) Since RLHF was first introduced, several advancements have been made, including fine-grained reward systems (Bai et al. 2022b; Wu et al. 2023b Dong et al.) 2023b; Wang et al,
2023c 2024c). or replaced the original PPO algorithm with other RL techniques (Wu et al. 2023a
An alternative to RLHF is DPO (Rafailov et al. 2023). which employs an offline RL approach to optimize language models based on preference data, without the need for separate RM. While DPO can be used independently, it is often complementary to other training methods like SFT or statistical rejection sampling, to further improve human alignment based on a RM (Zhao et al. 2023 Liu al. 2024 Dubey et al. (0224). Statistical rejection sampling, also called best-of-N or top-k-over-N (Bai
ct al. 2022b Touvron et al. 2023 Dubey et al.) 2024) is another widely used technique. Moreover, certain methods perform human alignment without RL to avoid instabilities, and fine-tune tune the model
on filtered samples by RM, or other sources (Dong et al., 2023a Yuan et al. 2023).
A major challenge in human alignment techniques is data acquisition (Casper et al. 2023). This in- cludes issues such as evaluator misalignment, supervision difficulties, and feedback quality (Casper et al. (023). However, as AI systems continue to improve, LLMs are increasingly employed for tasks traditionally handled by humans, such as data annotation and generation. Unlike human feed- back, Al-generated feedback offers better scalability, enabling faster and more cost-effective data collection. For example, RLAIF, introduced by Bai et al. (2022b). is a promising approach that trains reward models based preferences generated by off the shelf LLMs. Variations of RLAIF have been explored in several studies (Lee et al.) 2023 Jiao 2023 Cui et al. 2024 Li et al. 2024 Yang et al. 2024). In the context of self-
instructions, approaches like Selt Seff-Inntructt Wang et al.] (023a), Self-Refine (Madaan et al. 2023), , and Self-Alignment (Li et al.] 2023) demonstrate how models can autonomously generate datasets based on their learned human preferences.
Different alignment methods like RLHF and RLAIF rely on the RM to incorporate the human feed- back. The RM learns to predict human preference based on labeled examples, serving as a proxy for human judgment later. Therefore, the success of language model aligmment relies heavily on the quality of the underlying reward model (Pace et al., 2024). which in turn dictates the behaviour of
Before the process of human alignment, these models are referred to as "pre-trained" L.IMs.
3

---

Published as a conference paper at ICLR 2025
Table 1: Eye-tracking (ET) features computed per word.
Feature
Definition
First Fixation Duration (FFD)
Time spent on the initial fixation
Go-Past Time (GPT)
Cumulative fixation time before moving the right
Total Reading Time (TRT)
Overall time spent fixating on a word Number of fixations on each word
Number of Fixations (nFix)
Proportion of participants (fixProp)
Proportion of participants that fixated on the word
the resultant chatbot (Shen et al. 2023). Even in LLM inference, methods like best-of-N sampling use the RM to evaluate model outputs (Cui et al., 2024). RM has also become crucial for generating synthetic data for preference alignment. In recent RLAIF methods, reward modeling has expanded beyond its traditional role and is now used to generate artificial feedback.
2.1.1 REWARD MODELING
In the original implementation (Ouyang et al. 2024), the goal of RM training to train a classifier that predicts the probability of human preference p" between two completions (Equation I). mod- elled by a Bradley- Terry model (Bradley & Terry 1952). The typical setup involves showing two completions, with preferences being measured using win-loss-tie outcomes a Likert scale to cap- ture the strength of preference at. 2022a). The data is processed into prompt-chosen--ejeer trios, where the chosen completion, Yes is preferred over the rejected one, yt, forming the basis for training (Ouyang et al. 2024).
exp(r" ex, ywo)
pd (y > 31 xx) exp(r ) + exp(r"(x,
2.2 EYE-TRACKING
Eye-tracking (ET) systems monitor oculomotor behavior, such eye movements and fixations, of- fering valuable insights into visual attention, information processing, and expands our understanding of reading and language comprehension. (Zhang & Hollenstein 2024). Specifically. ET data often include fixations pauses eye movement to focus on specific areas (Mathias et al. 2020): sac-
cades rapid movements between two points (MeGuire & Tomuro 202T g scanpaths sequences of
fixations that reveal saccades and regressions (Yang & Hollenstein, 2023); and other temporal and spatial gaze behavior features (Zhang & Hollenstein 2024). Incorporating ET data into NLP tasks often involves the use of several features listed in Table
While several publicly available datasets such as ZUCO (Hollenstein al. 2020b). ZUCO2 (Hol-
lenstein et al. 2018), PROVO (Luke & Christianson 2018). ETSA-I (Mishra et al. 201 6a), ETSA-II MMisrra al. 2018), GECO (Cop et al., 2017), GECO-MT (Colman ET research, obtaining real ET data for NLP tasks remains challenge. This is primarily due to the cost and precision requirements of ET equipment, the unavailability of gaze data during inference, as well as privacy concerns (Khurana et al., (023). To address these challenges, two main approaches have been proposed. The first involves integrating ET data into the model during training through methods like Multi-task tasti learning (MTL), which eliminates the need for ET data during inference (Mishra et al., 20188: Klerke et al.] 20166; Ren & Xiong |021] Yu et al., 2024; Deng et al. 2024). The second approach involves techniques that directly predict users' gaze behaviour (Deng al.)
2022) are widely used in
2024 2023a: Zhang & Hollenstein 2024 Waag et al. (024a). creating synthetic ET data for any
given text stimulus (Deng et al. 2023b Bolliger al. 2023 Khurana al. 2023
Li & Rudziez
2021
Hollenstein et al. 2021 2022
3 RELATED WORK
Reward Modelling. The most popular approach to reward modeling follows the framework intro- duced by Ouyang et al. (2024). Several studies have examined alternative versions for refining RMs. For instance, Bai et al. 2022b) proposed more fine-grained reward structures, evaluating helpfulness
4

---

Published as a conference paper at ICLR 2025
and harmlessness separately. Other approaches have explored different reward modelling strategies [Wu et al.) 2023b Dong et al. 2023b Wang et al. (023c). Another line of research has focused
on Process Based Reward Models (PRMs) (Lightman et al. 2024 Uesato al. 2022 which differ from conventional RMs by predicting the correctness of intermediate steps, rather than solely eval- uating final outputs. Other studies implement data augmentation techniques (Shen al. (022)) or cross-attention mechanisms between encoded input text and candidate pairs (Jiang et al. 2023b) Moreover, some works have leveraged synthetic preference data for reward modelling (Cui et al. 2024 Jiao 2023). Wu et al. 2024b) built upon the LLM-as-a-Judge framework (Zheng et al. (2023) by introducing LLM-as- tarassa-Mess -aee----Mtta-iie Meta-Judge, which evaluates the model's judgments capabilities. Finally, Pace et al. (2024) incorporated self-training approach to improve reward model training. However, to date, no research has explored the integration of ET or other implicit feedback signals into RM.
generate prefer-
ence pairs that enhance its decision-makinn
Eye-tracking in Natural Language Processing. Several studies have investigated the use of ET data for a variety of NLP tasks, such as named entity recognition (Hollenstein & Zhang 2019 Ren
& Xiong 2021
Yu et al,, 2024 Hollenstein et al. 2019), text comprehension (Ahn et al. 2020) Re-
ich et al. 2022 Sood et al. (020b), language modelling (Huang et al.) 2023 Huang & Hollensteing
2023 Deng et al., (003b),, and question answering (Zhang & Hollenstem [024]] Wang et al., (024a))
Other applications include code comprehension (Alakmeh et al. (024). code summarization (Zhang
iet al. 2024) and hallucination detection (Maharaj et 2003). Eye-tracking has also been applied
t0 sentiment analysis and sarcasm detection tasks (Mishra et al. 601 bj [2018] Barrett al. 2018
Huang et al. 2023] Khurana et al. 2023 Hollenstein et al. [019 Yang & Hollenstein [2023] Kiege-
land et al.j 2024] Deng et [023a] Mathias al. 2018 McGuire & Tomuro (021). The most
relevant work to our approach is by Kiegeland et al. (2024). which introduced dataset generation
method using ET signals for DPO, building on the controlled sentiment generation framework pro-
posed by Deng et al. (2023a); Yang & Hollenstein| (2023). While this study has contributed to the first steps towards integrating ET for human alignment LLMs, it task- and dataset-specifie, often relying on ranking criteria that underutilize the potential of ET feedback. In contrast, our approach presents a more general framework by directly incorporating implicit feedback into the RM, rather than limiting its application to dataset creation.
4 GAZEREWARD: REWARD MODELING WITH ET FEEDBACK
In this section, we discuss the proposed framework for augmenting the RM using implicit feedback derived from ET signals (Figure 2D. Initially, we generate the ET features (subsection 4.T) consid- ering two tate-of-theearr ET prediction models. Next, combine the ET features with the text dsubsection 4.2). producing different types of combined embeddings, and finally pass them as input into the RM to obtain the reward for the prompt and its corresponding response (subsection 4.3).
1. Crnate ambaddiigg
Step embeddings
1. Compute
dor far Jaa La di
fi
O
Co ( xtl))
O-00000 OOOD
Pawiig cancass uadd
Tekaovoite
irputtoihe decade
Pantents
Embedding layer
Besenbeciig DOOO
decicr
Joing erowdding
Pelesece
repparec
OCOO O Recersadding
Hannnraapptte arradding inpitudde
resporse
Contoeeatignn promphiet best constructthe inpustunt
resporac
Inout
stams trave??
resporace
Figure 2: Overview of the GazeReward framework, incorporating eye-tracking features into the reward model. The architecture is illustrated in the figure using the second ET prediction model, but it would be identical if the first one were used instead (see subsection 5.1
5

---

Published as a conference paper at ICLR 2025
4.1 EYE-TRACKING FEATURES GENERATION
As discussed in subsection 2.2 obtaining organic ET features for NLP applications presents several challenges. this work, we consider an approach inspired by RLAIF research, where feedback is artificially generated from pre-trained LLMs and, in particular, from ET prediction models. Specifi- cally, we incorporate the output of two different ET prediction models (Li Rudziez, |021] Huang & Hollenstein, 2023) and evaluate the impact of different set of features. As input to these mod- els, we pass the same text as we do in the RM: a combination of prompt æ and response y. The output is a set of ET features, denoted where w represents the number of tokens in the tokenizer used by the ET prediction model, and f is the number of features. Depending the specific model, between one and five synthetic features fet = ( Jw E Rwx/ are generated per token for the input text.
fet. for each token fet = f2.
fw) E Ruxs,
4.2 RM AUGMENTATION USING EYE-TRACKING FEATURES
We implement two different approaches for incorporating ET features into the RM, as shown in Figure 2 In the first approach, GazeConcat, we concatenate the ET embeddings with the text embeddings. In the second approach, GazeAdd, we add the ET embeddings to the text embeddings. Furthermore, we concatenate the prompt and the response to be evaluated and pass them through the pre-trained embedding layer of, to generate the embeddings H {/1, h2...... hn } Rxx where n is the number of tokens in the tokenizer used by the RM and is the model embedding size.
To project these features to the model embedding size (d), ET feature projector fp(. The fp() consists of two linear layers, two dropout layers, two Layer Normalization layers, and ReLU activation, designed for stable, non-linear ET feature representation and overfitting prevention. The model's input dimension dynamically adjusts to accommodate the number of features used during training. The ET features projector can be formulated as emb ETF fr(fet) E Rexd (Figure 2}. This formula describes the projection of ETF features (fer) through a function fp, resulting in an embedding matrix embETF with dimensions w d. where represents the number of tokens and d the embedding dimension.
Multilayer Perceptron (MLP)
GazeConcat:
The ET embedding, denoted embETF, concatenated with the text embedding H to form the input for the RM. To distinguish between the two modalities, we introduce two special tokens: (eye) and //eye), which flag the start and end of the ET embedding, respectively (Figure 2) These special tokens are randomly initialized as one-dimensional vectors and added to the embedding layer or the RM model for training. The final input is structured as: (emb(<<eye>> embETF <mb(</eyy>>)
H). The same process is applied to the attentions masks.
The input to the RM consists of the ET embedding embETT F and the text embedding H. which are added in an elementwise fashion: (embETF + H). The two ET prediction models use different tokenizers, which also differ from those used by the base models in the RM. As a result, the number of tokens in the input for the RM and the number of tokens generated by the ET prediction model may not match. To address this embedding alignment issue, and have the same dimension, we remap the ET features from the w-token space the n-token space used by each base model in the RM. Further implementation details can be found in Appendix A.1.3
GazeAdd:
Table 2: Overview of different corpora used in the study train the reward model.
Corpus
Train set
Val. set
Test set
Lang.
Reference
OASSTI HelpSteer2
6567 5938
1160 1049
416
EN
Köpf et al. Wang el al. (2024c)
2023
364
EN
4.3 REWARD MODEL
The RM's architecture and hyperparameters are identical to those of the pretrained LLM, except that the classification head used for next-token prediction is replaced with regression head that outputs scalar reward (Touvron et al. 2023). This scalar reward indicates the quality of the model generation, corresponding to the predicted score for the final reply in conversation. Differences in

---

Published as a conference paper at ICLR 2025
these rewards represent the log-odds that one response is preferred over another. The loss function is defined in Eqquaion 2, where Yuu refers to the preferred response in a pair of completions Yo and yt- The dataset D consists of human comparisons, where ro (I, Yie) ro (æ, y) represents the RM 0
scalar outputs for the preferred and less preferred completions, respectively Ouyang et al. (2024).
loss (f) = - -..... [log (a (ra (a, Vw)) ro (r. k)))
(2)
In the proposed method, we augment the traditional RM, which uses text input (a combination of the prompt x and response y), by incorporating (artificial) implicit feedback through ET features generated from the same text. These ET features provide valuable information for capturing human preferences, thereby improving the RM's performance.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets. For our experiments, we use the OpenAssistant Conversations dataset's (OASSTI) (Köpf
et al. 2023) and HelpSteer2 (Wang et al. (022c) (Talle 2). OASSTI is human-generated, human- annotated, assistant-style conversation, created through global crowdsoureing and widely used for human alignment tasks (Köpf et al. 2023 Dettmers et al. 2023 Wu et al., 2024b). We filtered all non-English text, as the ET prediction models were exclusively trained on English data. Among the different responses in the dataset, we selected the two most distinct responses to compare the chosen and the rejected responses (Wang et al.) 2024b). HelpSteer2 a more recent, English-only dataset that has been used in studies such as Wang et al. (2024b/e). The dataset provides annotations for five response attributes: helpfulness, correctness, coherence, complexity, and verbosity. To transform it into a preference dataset, designate the response with the higher helpfulness score the chosen response and the other as the rejected response, following a method similar to that used in DPO training (Wang et al. 2024c) (see Appendix A.1.I for more details about the datasets).
Dataset Preparation. To tune LLMS for human-AI interaction, we need to define a chat dialogue protocol that allows the model to understand human instructions and rate them. To this end, we adopt a chat protocol that utilizes special header and termination tokens, similar to the format used in Llama 3. For example, in the case of the Llama 8B model, the concatenation of prompt and its corresponding response would follow this structure: <im sturs>>usssser Example Prompt <im end>> <im_start>assistonn Example Response <im_end> (see Appendix A. for more details).
Models. As RM base models we use the pretrained checkpoint of Hugging Face (Appendix A.1.4) for Llama 3 8B, Llama 8B-instruct (Dubey et al. 2024) and Mistral 7B (Jiang et al. 2023a).
ET prediction models. In our analyses, we utilise two state-of-the-art ET prediction models, both pre-trained to predict ET features and kept frozen in our implementation. The input to these models is the same text used for the RM, with minimal modifications (Appendix 1.2) The first model T5 embedding layer (Raffel et al. 2020), a two-layer
(Huang & Hollenstein| (023). consists BiL.TMM (Hochreiter & Schmidhuber 1997). and one-hidden-layer MLP. This model was trained on the Dundee, GECO (Cop et al. 2017 7). ZuCol (Hollenstein et af.) (0018) and ZuCo2 (Hollenstein al. (2220a) datasets, and predicts total reading time (TRT) per token (Figure 3). The second model (Li & Rudzicz 2021). is based on RoBERTa (Liu et al. 2019) with a regression head on each token. This head is a linear layer that outputs five features: FFD, fixProp, GPT, TRT, and nFix (Tbble T). The model is initialized with traitrd weights and fine-tuned on the ZuCol (Hollenstein et al., 2018 ZuCo2 (Hollenstein et al. 2020a) and PROVO (Luke & Christianson 2018) datasets. Since RoBERTa's maximum sequence length is 512 tokens and our input sequences are longer, we employ a sliding window approach. The input is split into 512-token segments with a 50-token overlap, and the results are combined using a linear weighted approach. Further details these models and their integration into our framework are provided in Appendix A.112
Baseline models. To evaluate the improvement in accuracy for a RM that incorporates implicit feed- back, and specifically ET signals, we compare the same RM with and without the ET embeddings. For each dataset and model, train and evaluate all combinations of integrating and combining ET features, and then compare them against a RM trained on the same base model and dataset but without implicit feedhack.
7

---

Published as a conference paper at ICLR 2025
Evaluation metrics. Performance is determined by measuring the model's ability to predict the better reponse from pairs of replies with different ranks. Accuracy is calculated as the percentage of cases where the reward score for the preferred response is higher than that of the less preferred re- sponse, based on held-out dataset. This method follows similar approaches found in Touvron et al. (2023); Yuan et al. (2023); Köpf al. (2023); ) Cui et al. (2024). We use the test split proposed by the authors for each dataset (Table 2). We also conduct complementary evaluation on RewardBench (Lambert al., 2024), benchmark dataset created for evaluating performance and safety features
Yuan
of R.M's (see Appendix B for more details).
Training procedure. In our implementetion the modules remains frozen (Figure 2). For the RM, we fine-tune the open-source models previously introduced with QLoRA (Dettmers et al.) (023) Sarameter-Ffficientt Fine-Tuning (PEFT) method based on Low-Rank Adaptation (LoRA) (Hu al, 2021), with other memory optimization techniques, We follow the training process for the RM as outlined in TTourron et al, (2023); Ouyang et al. (2024). We independently train each model on its respective dataset for two epochs, as detailed in Wang et al. (2024c). For hyperparameter tuning, we reserve 15% of each dataset for validation (shown in Table 2p. The best-performing checkpoints selected based the lowest validation loss and used for performance evaluation. We perform a grid search to determine the optimal batch size and testing values {8, 16. 32}. The Adam W optimizer (Loshchilov & Hutter 2019) is used, with the Learning Rate (LR) is tuned over the range (1,5. 5, 10, 50-e-6, following the values reported in Touvron et al. (2023); Cui et al. (2024); Waan al. (2024c). Additionally,
evaluate differennt LR schedulers: constant, linear, and cosine with a minimum LR. Further hyperparameter values and implementation details for both the RMM and the ET projector can be found in Appendix
5.2 RESULTS
The results of our experiments on the OASST and HelpSteer datasets, covering all possible combi- nations of ET features, models, and inclusion methods, as shown in Table 3] Table reespectiey. For the Mistral model, results for the GazeAdd method are unavailable due to the inability to map features between the ET prediction model's tokenizer and the reward model's ss tokenizer (details in Appendix A.1.3).. In what follows, we present key findings based on three seeds, reporting the average, mean, and statistical significance.
Table 3: Reward modeling accuracy (%) for OASST1 dataset. The highest results in bold and the second highest are underlined.
Llama-3-8B. Instruct
Llama-3-8B
Mistral-7B
baseline
65.9 + 0.5
diff (%)
65.5 2.1
diff (%)
66.3 = 0.1
diff (%)
ccomb
69,0 1 0.4+
4.7
69.3 1 0.6
5.9
67.6 1: 1.7
2.1
GazeConcat
comb22
70.2 - 0,3
6.6
71.5
0.5
9.2
70.2 = 0.4+
5,9
fcomb2.2
70.0 1 0.4*a
6.3
71.2 + 0.8
8.8
71.0 = 1.0
7.1
fcomb
68.9 + 0.9
4.6
68.9 1.0
5.3
GazeAdd
comb2s
70.2 1 0.1
6.6
69.5
1 0.3
6.1
fcomb2.2
69.0 0.4*
4.7
68.3
0.7
4.4
Table Reward modeling accuracy (%) for Helpsteer2 dataset. The highest results are in bold and the second highest are underlined.
Maama-3-8B-Instrueee
Llama-3-8B
Mistral-7B
baseline
54.7 + 0.7
diff (%)
53.3 = 0.8
diff (%)
54.1 + 0.3
diff' (%)
fcomb
61.1 + 1.2+
11.8
59.1 0.2+
10.8
57.6 = 2.3
6.5
GazeConcat
fcomb2s
58.5 + 1.6
7.0
60.3 + 0.5**
13.2
58.7 = 2,4
8.5
fcomb22
60.6 1 3.3
10.9
57.9 2.0
8.6
56.0 1 2.4
3.4
fcomb;
62.3 + 0.6**
13.9
62,4 + 1.0**
17.0
GazeAdd
comb2s
59.6 1.1*
9.0
58.6
1.2+
10.0
comb22
60.3 0.5**
10.2
59.3
0.1+
11.3
8

---

Published as a conference paper at ICLR 2025
Effect of Model Initialization. We evaluate the impact of model initialization performance. Open-access LLMs typically come in two forms: a pre-trained version without human alignment and final version that has undergone alignment with human feedback. Since lack access to in- termediate checkpoints, we experiment with both pre-trained models (Mistral 7B and Llama 3) and models that are already human-aligned (Ilama 3 Instruct). Our goal is confirm that our method is effective for RM initialized with both pre-trained and human-aligned checkpoints. When comparing accuracy improvements relative the baseline (without ET features), all models show consider- able gains from incorporating implicit feedback. Notably, the Llama 3 8B and Mistral 7B models, which had no prior alignment, demonstrate performance improvement from the incorporation of ET features, indicating that unaligned models can benefit from implicit feedback.
Inclusion method. The results shown Table and Table 4 indicate that both GazeConcat methods and GazeAdd introduce substantial performance improvements to the RM. Across both datasets, concatenating embeddings (GazeConcat) delivers more consistent results. Incorporating ET in- formation through specialized separator embeddings allows the model to process both text and ET features more robustly. However, in the HelpSteer dataset (Tall 4), directly adding ET information to the text embeddings (GazeAdd) results in the greatest improvement over the baseline.
Eye-tracking (ET) feature importance. Different ET features capture distinct aspects reading behaviour and information processing, influencing model performance uniquely (Zhang & Hollen- stein 2024). Here, we examine how model performance varies when incorporating three different feature combinations generated by two different ET prediction models: fcomb1 TRT generated by the first ET prediction model; fcomby.s five features (nFix, FFD, GPT, TRT, fixProp) gener- ated by the second ET prediction model; and fcomb2. TRT and FFD generated by the second ET prediction model. TRT and FFD are widely used in ET research (Huang et al. 2023 Huang & Hollenstein 2023] Zhang & Hollenstein 2024 Maharaj et al.) [023 Wang et al. 2022); J and they have been shown to correlate with attention scores from pre-trained transformer models ( (Wang et al.)
2024a Bensemann et al., 2022 Sood al., 2020a) and with gradient-based saliency(Hollen (Hollenstein & Beinborn [021]] Wu et al. (222a)). When comparing results, we observe that the R.M benefits from the ET feature combination or ET prediction model used. Specif- ically, in most cases, fcomb, yields the best results, particularly with the GazeAdd method. For GazeConcat, fcomb2.2 and fcomby.s performs best in general. We attribute the superior perfor- mance of fcombı to how the ET prediction model generating the fixations was trained, including the data and preprocessing methods used (see (A..nd)x(A.II2)) Moreover, when comparing f fcobb2. and f /comb2,5 both generated by the same model only in one case does integrating nFix, GPT, and fixProp improves performance. In some instances, using f fccomb2.5 results in worse performance than the baseline, confirming findings provided by prior studies, which suggest that features related to reading time, such as FFD and TRT, contribute most to performance gains.
implicit feedback regardless
RewardBench. As side contribution, we evaluate our best performing models (trained on the OAASTI dataset) on RewardBench. This evaluation is not intended to directly compare our method with larger, more essource-intensin RM, but rather to show that through the integration of multi- modal signals like ET features we can significantly enhance the performance of RM models. The results shown TTable demonstrate consistent improvements as previously observed, with gains exceeding 40% for the Mistral model a notable gain considering that the base RM is the same. We note that the performance of the bascline models is impacted by RM trained on base models with less than 9B parameters and on relatively small datasets (see details in Appendix B
Table 5: Reward modeling accuracy (%) evaluating on RewardBench dataset. All models are trained on OASSTI dataset. The highest results are in bold and the second highest are underlined.
Jlama-3-8B-Instuuct Llama-3-88
Mistral-7B
baseline
46.9
diff(%)
50.9
diff(%)
41.2
diff(
feombi
57.8
23.1 13.1%
58.4
14.5%
59.9
45.4%
GazeConcat
fcomb2.s
58.4
24.4%
58.1
14.1%
60.3
46.4%
fcomb22
58.1
23.8%
58.5
14.8% %
60.5
46.9 %
fcomb,
56.5
20.3%
56.6
11.2%
GazeAdd
54.9
16.9%
53.8
5.6%
f fcomb22
55.4
17.9%
52.5
3.1%

---

Published as a conference paper at ICLR 2025
6 DISCUSSION
In this work, we introduced a novel framework for integrating implicit feedback into the Reward Model, key component for aligning LLMs and generating synthetic data for further alignment. We validated our approach using widely-adopted, open-source models such Llama 3 and Mistral, for initializing the RMM By employing two different models to generate ET features, our results show that incorporating implicit feedback consistently improves the RM's ability predict user preferences, regardless of the model used and without the need to reach large parameter counts or train on massive datasets. Additionally, making it fully scalable and applicable to various human alignment methods, including those that involve artificially generated datasets. This work advances the ongoing discussion on optimizing AI alignment with human values and shows the potential of multimodal signals for NLP research enhancing current methods.
method leverages ET features generated by models,
6.1 LIMITATIONS & FUTURE WORK
Data. A limitation of our study is that both ET prediction models were trained on a relatively small datasets (Appendix (A..2) that are not tailored to tasks. Future work could benefit from directly collecting ET data specifically for LLMM-generaed responses, to offer insights into human reading comprehension and information processing of prompts, which could further improve model performance. Additionally, since the ET prediction models used in our experiments were trained on English corpora, the method's generalizability other languages requires further investigation. Moreover, we explored two methodologies for integrating ET features into the RM, but other ap- proaches could prove more effective. For instance, ET features could be used to modify the RM's attention mask, as suggested by Zhang & Hollenstein (2024). Regarding dataset selection, both models used, Mistral 7B and Llama 3, were fine-tuned on publicly available data, though specific details on the datasets are limited. Therefore, we cannot discount the possibility that the datasets we used may have been part of the models' pretraining, particularly for Llama 3 7B Instruct, which compare against basclines using the same
has already undergone human alignment. However, model checkpoints, any potential effects would be consistent across both conditions. Finally, is important to note that although the ET prediction models remain fixed during training, our solution has a slightly higher number of parameters than the baseline.
Training.
The scaling trends for the RM TTuvron al., 2023) show that larger models or models trained on massive datasets perform better. A promising direction would be to validate our frame- work on larger models, without relying on PEFT methods, and on larger datasets. We are confident that, despite the considerable computational costs this may entail, our framework is capable of scal- ing effectively. Another direction is integrating the proposed RM into an alignment method like RLHF, or applying it in rejection sampling to generate synthetic preference datasets, ensuring that accuracy gains in the RM translate to improvements in the final LLMs.
REPRODUCIBILITY STATEMENT
All the code necessary to reproduce is in the GitHub repository Both datasets used are publicly available KKöpf et 2023 Wang et 2024c). Additionally, both ET prediction models have been
3
trained with public datasets (Cop et al. 2017 7 Hollenstein et al.
2018 2020a).
IMPACT STATEMENT
Since our research uses only synthetic ET data, there are no privacy concerns or need for large-scale experiments involving human subjects. We should also raise attention to the limitations of human feedback and ET prediction models bias, that inevitably reflect aspects of their training data.
ACKNOWLEDGGMENTS
This research is supported by Horizon Europe's European Innovation Council through the Pathfinder program (SYMBIOTIK project, grant 101071147) and by the Industrial Doctorate Plan of the De-
https://github.rrt
Sci e ff ReRearchhhh wawaaarrrhgaaaeeeeaeeee
10

---

Published as a conference paper at ICLR 2025
partment of Research and Universities of the Generalitat de Catalunya, under Grant AGAUR 2023 DI060. We also want to thank Sebastian Macaluso for his important feedback during the project.
REFERENCES
Seoyoung Ahn, Conor Kelton, Aruna Balasubramanian, and Greg Zelinsky. Towards Predicting Reading Comprehension From Gaze Behavior. In ACM Symposium on Eye Tracking Research and Applications, ETRA '20 Short Papers, pp. 1-5, New York, NY. USA, June 2020, Association for Computing Machinery. ISBN 978-1-4503-71344
doi: 10. 10.1145/33991556339333. URL
https::/ddl d1, acm. wtt. wttpa//......o...o..oooooooo 114./ 0T..4//337915666 3391335
Tarek Alakmeh, David Reich, Lena Jäger, and Thomas Fritz. Predicting code comprehension: a novel approach to align human gaze with code using deep neural networks. In Alakmeh, Tarek; Reich, David; Jäger, Lena; Fritz, Thomas (2024). Predicting code comprehension: a novel ap- proach to align human gaze with code using deep neural networks. In: ACM International Con- ference on the Foundations of Software Engineering, Porto de Galinhas, Brazil, 17 July 2024 19 July 2024., Porto de Galinhas, Brazil, July 2024. University of Zurich. doi: 10.1145/3660795. URL https: ptttps://wwrr zora. wwtora..z..cooo ch/ http://hw.../////crctt////222220
Issue: Number: 1.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano,JJhn Schulman, and Dan Mané. Con- crete Problems in AI Safety, July 2016. arXiv:1606.06565 [cs].
06565
Anthropic. Claude 2, November 2024.
URL http::/www..nttroppc..o
claude-2
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-oodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Ölsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, April 2022a. URL h
http//aaxvv..o http:::/azaa..or//bbb///// 2204. 05862 arXiv:2204.0588 [cs].
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie KeTI, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton Tom Con- erly, Tom Henighan, Tristan Humn. Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI Feedback, December 2022b. URL http://///// /ttp///axivvv///a///a//eee org/ wrgpabb///ab2l///// 08C73 urXiv:2212.0800 [cs].
Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. Sequence Clas- sification with Human Attention. In Anna Korhonen and Ivan Titov (eds.), Proceedings of the 22nd Conference on Computational Natural Language Learning, pp. 302-312, Brussels, Bel- gium, October 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/K18.. 1030. URL https:: antho wgy.orgg/188... 1030
Joshua Bensemann, Alex Peng, Diana Senavides-Prado, Yang Chen, Neset Tan, Paul Michael Corballis, Patricia Riddle, and Michael Witbrock. Eye Gaze and Self-attention: How Hu- mans and Transformers Attend Words in Sentences. In Emmanuele Chersoni, Nora Hollenstein, Cassandra Jacobs, Yohei Oseki, Laurent Prévot, and Enrico Santus (eds.), Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pp. 75-87, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v http://222222lmlll// 1.9. URL https: //ccl antho wowhhhooyy...roo wow..ogg........... 1tt..oorrrr222.mccl1_....
cmcl-1.9
11

---

Published as a conference paper at ICLR 2025
Lena Bolliger, David Reich, Patrick Haller, Deborah Jakobi, Paul Prasse, and Lena Jäger. ScanDL: A Diffusion Model for Generating Synthetic Scanpaths on Texts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Language Processing, pp. 15513-15538, Singapore, December 2023. Association for Computa- tional Linguistics. doi: 10.18653/v1/2023.mmmll/ http:///022..mmmmmam.....mm 960. URL
Empirical Methods in Natural
emnip main. 960
Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The
Method of Paired Comparisons. Biometrika, 39(334):324-3455, 1952. ISSN 0006-3444. doi: 10. 2307/2334029, URL https: //www//wwwww./o////////// University Press, Biometrika Trust].
https://www jstor orrgerroog//soo http://saaleee223/3 http://ww.....ccororoa/aaa////// Publisher: [Oxford
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaél Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitri Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem BIyIk, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Saddield-Menee Open Problems and Funda- mental Limitations of Reinforcement Learning from Human Feedback. Transactions on Machine Learning Research, 2023.
Toon Colman, Margot Fonteyne, Joke Daems, Nicolas Dirix, and Lieve Macken. GECO-MT:
The Ghent Eye-tracking Corpus of Machine Translation. In Nicoletta Calzolari. Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hi- toshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 29- 38, Marseille, France, June 2022. European Language Resources Association. URL https: 1rec 1.4
7/ aa.lanthology.orrg220000
Uschi Cop, Nicolas Dirix, Denis Drieghe, and Wouter Duyck. Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading. Behav Res, 4922):602-615, April 2017. ISSN 1554-3528. doi: 10. 10.3758/133288-11-07744 URL https://dcii 3ttp://......///0..3333.
113428-016-0733
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. ULTRAFEEDBACK: boosting language models with scaled AI feedback. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of ICML'24, pp. 9722-9744, Vienna, Austria, July 2024. JMLR.org.
Shuwen Deng, Paul Prasse, David Reich, Tobias Scheffer, and Lena Jäger. Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understanding. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing, pp. 5500-6507, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 10.186553 wttp://.emmmmmmmmmm....
URL
coanthooogy..o ogy. org. 2023. ww0.2..emnlpp..mmmmmmmmmmm..e... emnle- in. 400
Shuwen Deng, David R. Reich, Paul Prasse, Patrick Haller, Tobias Scheffer, and Lena A. Jäger. Eyettention: An Attention-basse based Dual-Sequence Model for Predicting Human Scanpaths during Reading. Proc. ACM Hum. Comput. Interact., 7ETRAA), May 2023b. doi: 10.1 10.1145335 131. Place: New York, NY, USA Publisher: As-
URL http:htts:///e //doi. 0tgg/0..115/3355 sociation for Computing Machinery.
Shuwen Deng, Paul Prasse, David Reich, Tobias Scheffer, and Lena Jäger. Fine-Tuning Pre-Trained Language Models with Gaze Supervision. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 217-224, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. URL htRp::/::: UtL/sttss://ccaattaaa /ww.aaaaaaatto....aacaaaoa org. 2024 acl short.21|
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Fine- tuning of Quantized LLMss. November 2023. URL htttt::/ wet.:/ewweeeeeeeeeeeeemeeeeeeeee / forrm??
12

---

Published as a conference paper at ICLR 2025
5D (S22proffleeBFP8830022222 2o Hollzmanl)
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: Reward rAnked FinenT for Generative Foundation Model Align- ment, May 2023a. URL. http:/ttppxxii..oo/ abs/ 2304 06767 arXiv:2304.06767 [es, stat].
Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. SteerI.M: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF, October 2023b. URL http://arxiv. orw.a/v.rrraa/saee. htcp//bs//2333 05344 arXiv:2310.05344 [es].
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and et The Llama Herd of Models, August 2024. URL
arXiv:2407.2178..
Google. Google Bard Herramienta de IA Generativa y Bot Conversacional, 2023. URL https:
7/bard. le. com
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Tern Memory. Neural Computation, 18):1735-1780, November 1997. ISSN 0899-7667. doi: 199.6///...999..9...... URL
19977...199..33 99999999..17355
Nora Hollenstein and Lisa Beinborn. Relative Importance in Sentence Processing. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pp. 141-150, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v120211 acl-shhort.19 URL https:
alaclantholoo ott. acl
Nora Hollenstein and Ce Zhang. Entity Recognition at First Sight: Improving NER with Eye Move- ment Information. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- fics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1-10, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10. 1111116////N11--.1 1001.
URL http:s::///claantoooooooooo
1001
Nora Hollenstein, Jonathan Rotsztejn, Marius Troendle, Andreas Pedroni, Ce Zhang, and Nicolas Langer. ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading. Sci Data, 5(1)1,, December 2018 ISSN 2052-4463 doi: 10.1008/sdata.201..
URL Publisher: Nature Publishing
https: www. www.untuuee... com./ureeeoooeeeoooooooooeeeeeee Group.
Nora Hollenstein, Maria Barrett, Marius Troendle, Francesco Bigiolli, Nicolas Langer, and Ce Zhang, Advancing NLP with Cognitive Language Processing Signals, April 2019. URL http:///rxiv..ort wwwpaarwwww.ooo....aoo//a///a 9004... 02682 arXiv:1904.02682 [cs].
Nora Hollenstein, Maria Barrett, and Lisa Beinborn. Towards Best Practices for Leveraging Hu- man Language Processing Signals for Natural Language Processing. In Emmanuele Chersoni, Barry Devereux, and Chu-Ren Huang (eds.), Proceedings of the Second Workshop on Linguistic and Neurocognitive Resources, pp. 15-27, Marseille, France, May 2020a. European Language org/
Resources Association. ISBN 979-10-95546-522 URL 2020. 1incr.13
Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer. ZuCo 2.0: A Dataset of Phys- iological Recordings During Natural Reading and Annotation. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi. Hi- toshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Tweifth Language Resources and Evaluation Confer- ence, pp. 138-146, Marseille, France, May 2020b. European Language Resources Association. ISBN 10-91100555--344-0 URL httttttt.... //acl ogg org. 2200 rec- 1.18
13

---

Published as a conference paper at ICLR 2025
Nora Hollenstein, Emmanuele Chersoni, Cassandra L. Jacobs, Yohei Oseki, Laurent Prévot, and Enrico Santus. CMCL 2021 Shared Task on Eye- Trackirc Prediction. In Emmanuele Cher- soni, Nora Hollenstein, Cassandra Jacobs, Yohei Oseki, Laurent Prévot, and Enrico Santus (eds.), Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pp. 72-78, Online, June 2021, Association for Computational I Linguisties. doi: 10.
URL https: htt////caaanttttoooeeee hwwpt/thowoooooooooroooooo 2021 cmcl-1.7
Nora Hollenstein, Emmanuele Chersoni, Cassandra Jacobs, Yohei Oseki, Laurent Prévot, and En- rico Santus. CMCL 2022 Shared Task on Multilingual and Crosslingual Prediction of Human Reading Behavior. In Emmanuele Chersoni, Nora Hollenstein, Cassandra Jacobs, Yohei Oseki, Laurent Prévot, and Enrico Santus ), Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pp. 121-129, Dublin, Ireland, May 2022. Association for Com- putational Linguistics. doi: 18.18/53/v 1/0022.ccc2l-11-m1 1.14. URL https: haappss://allathhooogy. org/2022. wwc.-mc.1111...
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October 2021. URL http://arxiv www.:/gggga/....
09685 arXiv:2106.0990 066606 [cs].
Xinting Huang and Nora Hollenstein. Long-Range Language Modeling with Selective Cache. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Com- putational Linguistics: EMNLP 2023, pp. 1838-4854 Singapore, December 2023. Associa-
tion for Computational Linguisties. doi: 10.18653/w1/2023. http://ww//....nnnngeeeeee////// findings-emmlp.3221. URL https: org/2023. wtwpatthhoooo...og/22233333
//aclanthology.
findings emn emn 1a..pp...... 321
Xinting Huang, Jiajing Wan, Ioannis Kritikos, and Nora Hollenstein. Longer Fixations, More Com- putation: Gaze-Guided Guided Recurrent Neural Networks, October 2023. URL http:///rxiv.
arg/abs/2n 2311/ 00159 arXiv:2311 1.00159 [es].
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, October 2023a. URL org/ abs 2310 06825 arXiv:2310.0682 [es].
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. L..M-Beeddr: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion, June 2023b. 23C6. 02561 arXiv:2306.02561 [cs].
Tianhao Wu * Hanlin Zhu and Jiantao Banghua Zhu Jiao, Evan Frick Starling-7B: Increasing LLM Helpfulness Harmlessness with RLAIF, November 2023. URL.https:///tarrll
cs. .ss.brkelee..ed
Alan Kennedy, Joël Pynte, Wayne Murray, and Shirley-Anne Paul. Frequency and predictability effects the Dundee Corpus: An eye movement analysis. Quarterly journal of experimental psychology (2006), 66, March 2012. doi: 000000//177702182222..66666
Varun Khurana, Yaman Kumar, Nora Hollenstein, Rajesh Kumar, and Balaji Krishnamurthy. Syn- thesizing Human Gaze Feedback for Improved NLP Performance. pp. 1895-1831 1908, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10. 10.18653/v12003.. eacl-main. 139. bttts:::: how.elllaaooooooooo..ooooooooo
eac --aa-mmin 1m....139
Samuel Kiegeland, David Robert Reich, Ryan Cotterell, Lena Ann Jäger, and Ethan Wilcox. The Pupil Becomes the Master: Eye-Tracking Feedback for Tuning LLMs. July 2024. URL https:
Chris L. Kleinke. Gaze and eye contact: A research review. Psychological Bulletin, 100(1):7--100 1986. ISSN 1939-1455. doi: 00337/0033-2099100..1 Place: US Publisher: American Psychological Association.
14

---

Published as a conference paper at ICLR 2025
Sigrid Klerke, Yoav Goldberg, and Anders Søgaard. Improving sentence compression by learn- ing to predict gaze. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceed- ings of the 2016 Conference of the North American Chapter of the Association for Compu- tational Linguisties: Human Language Technologies, pp. 1528-1533, San Diego California, June 2016. Association for Computational Linguistics. doi: 10.18653/1//N16-11773 URL
1179
Kristina Krasich, Robert McManus, Stephen Hutt, Myrthe Faber, Sidney K. D'Mello, and James R. Brockmole. Gaze-based signatures of mind wandering during real-world scene processing. Jour- nal of Experimental Psychology. General, 147(8): August 2018. ISSN 1939-2222.
doi: 10.10377xge0000041.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam. Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shabul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Oppnnssistaa Cennerrraiiossneemmrriiiiinn Democratizing Large Language Model Align- arXiv:2304003327 [cs].
ment, April 2023. URL http:ittpp//arxii..r////a/s// wwtp:///////////// 22322233440777777
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, J. Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Ha- jishirzi. RewardBench: Evaluating Reward Models for Language Modeling, March 2024. URL
http://arxiv. org/ abs 2403. arXiv:2403.1375 [es].
M. F. Land and S. Furneaux. The knowledge base of the oculomotor system. Philosophical Trans- actions of the Royal Society of London. Series B. Biological Sciences, 352(13588)1231-1233, August 1997. ISSN 0962-8436. doi: 10. 1t..099//stb1199..1005
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinay Rastogi. RLAIF: Scaling Reinforcement Learning from Human Feed- back with AI Feedback, September 2023. URL http:://axi:: www.oorg//////////////.../eeee// abs 2209. 00267 arXiv:2309.00266 [cs].
Ang Li, Qiugen Xiao, Peng Cao, Jian Tang, Yi Yuan, Zijie Zhao, Xiaoyuan Chen, Liang Zhang, Xiangyang Li, Kaitong Yang, Weidong Guo, Yukang Gan, Xu Yu, Daniell Wang, and Ying Shan. HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback, March 2024. URL HrXiv:2403.08300 [cs].
Bai Li and Frank Rudzicz. TorontoCL at CMCL 2021 Shared Task: RoBERTa with Multi-Stage Fine- TuninTuni for Eye-Tracking Prediction. Emmanuele Chersoni. Nora Hollenstein, Cassan- dra Jacobs, Yohei Oseki, Laurent Prévot, and Enrico Santus (eds.). Proceedings of the Work- shop on Cognitive Modeling and Computational Linguisticx, pp. 85-89, Online, June 2021. Association for Computational Linguistics. doi: 10. 31533//1/2021.mmcl-11 URL https:
ottpthoooooo..rg2222111111 cmc.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-Alignment with Instruction Backtranslation, August 2023. URL http:
/ // wwwp:///g/aaaaa/aeeee 06259 arXiv:2308.068 00653990062991.1 [cs].
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. LET'S VERIFY STEP BY STEP. 2024.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Salch, Peter J. Liu, and Jialu Liu. Statistical Rejection Sampling Improves Preference Optimization, January 2024. URL http:
7/arxiv. org www.aaaas330 06657 mrXiv:2309.06653 [cs].
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach, July 2019. URL http://aaw.....og/a////////000// arXiv:1907.1 1692 [cs].
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, January 2019. URL http::/arxii www.ooogg/////o/eee .01101 1711:1711.001000 [es, math].
15

---

Published as a conference paper at ICLR 2025
Steven G. Luke and Kiel Christianson. The Provo Corpus: A large eye-tracking corpus with pre- dictability norms. Behay Res, 50(2):826 833, April 2018. ISSN 1554-3528, doi: 10.3758/ s13428-017-0008
hRLLts:s::/ee http::::///oo.orrg//////oo 10. 3758 10113/111111---111100000000
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uni Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder;
Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- Refine: Iterative Refinement with Self-Feedhack, May 2023. URL http::/arxiv.orrg arXiv:2303. [cs].
abs/ 2255/2233..0 17651
Kishan Maharaj, Ashita Saxena, Raja Kumar, Abhijit Mishra, and Pushpak Bhattacharyya. Eyes Show the Way: Modelling Gaze Behaviour for Hallucination Detection. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1442-11438, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18663/v1/2023. http:5////2222..nnnnneeemmme andingg-ennlp.764. URL
Findings-emnip 764.
Sandeep Mathias, Diptesh Kanojia, Kevin Patel, Samarth Agrawal, Abhijit Mishra, and Pushpak Bhattacharyya. Eyes are the Windows to the Soul: Predicting the Rating of Text Quality Using Gaze Behaviour. Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2352- 2362, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 18653/v 1/P18 1219. http:t:::
Sandeep Mathias, Diptesh Kanojia, Abhijit Mishra, and Pushpak Bhattacharya. A Survey on Using Gaze Behaviour for Natural Language Processing. volume 5, pp. 4907-4913, July 2020. doi:
10.24963/jjcai.22000 2020/683. URL https::////. //www//wwww/...// http://wwwwiiiiioocooo wttp://ccoooooeog////////a// ISSN: 1045-0823.
Erik S. McGuire and Noriko Tomuro. Sentiment Analysis with Cognitive Attention Supervision. Proceedings of the Canadian Conference Artificial Intelligence, June 2021. doi: 10.21428/ 594757ab.90170e50.
pubpub httgg//b///xxxtt11..
Abhijit Mishra, Diptesh Kanojia, and Pushpak Bhattacharyya. Predicting Readers' Sarcasm Un- derstandability by Modeling Gaze Behavior. Proceedings of the AAAI Conference on Artifi-
cial Intelligence, 30(1), March 2016a. ISSN 2374-3468. doi: 10.1609/aaii.v wat.aa...00......... URL
/tttp:://ojs.att org index phtp/AATTartticle///e
9
Number: 1.
Abhijit Mishra, Diptesh Kanojia, Seema Nagar, Kuntal Dey, and Pushpak Bhattacharyya. Lever- aging Cognitive Features for Sentiment Analysis. In Stefan Riezler and Yoav Goldberg (eds.), Proceedings of the 20th SIGNLL Conference on Computationnll Natural Language Learning, pp. 156-166, Berlin, Germany, August 2016b. Association for Computational Linguistics. doi: 10.18653/v1/K16-1100 URL pttttp:://aclaathooooo wrt.ooooyoog//11111100
Abhijit Mishra, Srikanth Tamilselvam. Riddhiman Dasgupta, Seema Nagar, and Kuntal Dey. Cognition-Cognizant Sentiment Analysis With Multitask Subjectivity Summarization Based on Annotators' Gaze Behavior. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, April 2018 doi: 10. 10.11609aaai.v3313 v32i1. 12221.12068. 12068. URL https://o/ aaaí oorg/ index. AAAIAAAAAAAAA
12068
ISSN: 2374 2159-5399
Issue: Journal
Abbreviation: AAAI.
OpenAl. GPT-4 Technical Report, March 2023. URL http://aariv..oggabbs///// 08774 arXiv:2303.08774 [cs].
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Pro- ceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22. pp. 27744, Red Hook, NY, USA, April 2024. Curran Associates Inc. ISBN 978-1- 71387-108-8.
16

---

Published as a conference paper at ICLR 2025
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-N: Synthetic Preference Generation for Improved Reward Modeling, January 2024. URL http: // wwwxx..org//bs//24000. 12086 arXiv:2401.12086 [cs].
Rafael Rafailov, Archit Sharma, Bric Mitchell, Christopher D. Manning, Stefano Ermon, and
Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Re-
ward Model. Advances in Neural Information Processing Systems, 1153778-537441 De-
cember 2023. URL http:://pppppp....sccccesee
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with Uni- fied Text-to-Tex Transformer, July 2020. URL http: /ttp:://axiv... org httgg/bb//191.. 10683
1EXiv:1910.10683 [es, stat].
David Robert Reich, Paul Prasse, Chiara Tschimer, Patrick Haller, Frank Goldhammer, and Lena A. Jäger. Inferring Native and Non-Native Human Reading Comprehension and Subjective Text Difficulty from Scanpaths in Reading. In 2022 Symposium on Eye Tracking Research and Applications, ETRA '22, pp. 1-8, New York, NY, USA, June 2022. Association for Comput- ing Machinery. ISBN 978-1 1 17788---503-9252--- 7/doi. wwg/oogg//0.0. 10. 114 .11.1709113333933333 3529639
doi: 10.1 10.1145/3517031 33526696,
URL https:
Yuqi Ren and Deyi Xiong. CogAlign: Learning to Align Textual Neural Representations to Cogni- tive Language Processing Signals. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- fies and the 1thh International Joint Conference on Natural Language Processing (Volume I:
Long Papers), pp. 3758-3769, Online, August 2021. Association for Computational Linguis- URL http::://eeca/anbooooooo.o//////
tics. doi: 10. http://w11122..ccooooogg.. acl long. 291
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Pol- icy Optimization Algorithms, August 2017. URLhttt://arxxi wwtp.//.wr...////...o/o// arXiv:1707.06344 [es].
org http//as//111..//0
06347
Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng. Haitao Mi, Daniel Khashabi, and Dong Yu. The Trickle-down Impact of Reward Inconsistency on RLHF, October 2023. URL
hattps://openreviewt net/ forum? deMe
Ekta Sood, Simon Tannert, Diego Frassinelli, Andreas Bulling, and Ngoc Thang Vu. Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension. In Raquel Fernández and Tal Linzen (eds.), Proceedings of the 24th Conference on Computational Natural Language Learning, pp. 12-25, Online, November 2020a. Association for Computational Lin- URL http::///eelatthooooo.oo////////
guistics. doi:
co.11-1.2
Ekta Sood, Simon Tannert, Philipp Mueller, and Andreas Bulling. Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention. In Advances in Neural Information Processing Systems, volume 33, pp. 6327-6341 Curran Asso- ciates, Inc., 2020b.
heereeeerdnngs.eeeiss.see httpllpp...pppppppreeee pccppappr//22//aass... html
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lil- licrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard. Paul Barham, Tom Hennigan, and et al. Gemini: A Family of Highly Capable Multimodal Models, June 2024. URL. http:://rxiit http:///axxvv.gg/aaaa/e http//bb//23133 11805 arXiv:2312.1 1805 [cs].
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom,
17

---

Published as a conference paper at ICLR 2025
Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, llyaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023. URLhtts //axxiv./axvvv.m// org/ abs 2307 09288 arXiv:2307 .090800 [cs].
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome- based feedback, November 2022. URL arXiv:2211. [es].
wwtpl//.rrgaab////eeeeee abs/ 2ttp//221..140 14275
Bingbing Wang, Bin Liang, Jiachen Du, Min Yang, and Ruifeng Xu. SEMGraph: Incorporating Sentiment Knowledge and Eye Movement into Graph Model for Sentiment Analysis. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 7521-7531, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v12022. emnlp- main.510. URL https: htt.t::://aaaaathooooooooooo org.oorgg//0// 2022 wmw.pmmmmmp...... mol-mmainn1511.
Bingbing Wang, Bin Liang, Lanjun Zhou, and Ruifeng Xu. Gaze-infused BERT: Do human gaze signals help pre-trained language models? Neural Comput & Applic, April 2024a. ISSN 0941- 0643, 1433-305 doi: 10.10077/000221-224-00222
URL https: //ink. springer....
com 10.
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-Taught Eval- uators, August 2024b. URL [cs].
02666
arXiv:2408.0266
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith. Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning Language Models with Self-Geeerated Instructions, May 2023a. URL http:ht:
org/ abs 2212. 10560 arXiv:2212.10566 [cs].
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng. Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning Large Language Models with Human: A Survey, July 2023b.
URL http:htt/////teee org/ abs /207 12966 arXiv:2307. [es].
Zhilin Wang, Yi Dong. Jiaqi Zeng. Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. HelpSteer: Multi-atrribute Helpfulness Dataset for SteerL.M, November 2023c. URL http:
09528 MxXiv:2311.0955 [cs].
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang. Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. HelpSteer2: Open-source dataset for training top-performing reward models, June 2024c. URL WRLhttp::/ arw.a.iiiv./.../ wrtp://////////////bb/ 08673 arXiv:2406.08673 [cs].
Guojun Wu, Lena Bolliger. David Reich, and Lena Jäger. An Bye Opener Regarding Task-Based Text Gradient Saliency. In Tatsuki Kuribayashi, Giulia Rambelli, Ece Takmaz, Philipp Wicke, and Yohei Oscki (eds.), Proceedings of the Workshop Cognitive Modeling and Computational Linguistics, pp. 255-263, Bangkok, Thailand, August 2024a. Association for Computational Lin- guistics. URL httpt::t/////...... /at:://allaae moclanntologyy. org / 2 2024. cmcl-1.22
Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for L.M Alignment, Oc- tober 2023a. URL
org. abtp://////333 00212 arXiv:2310.0021 [es].
Tianhao WU, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason We- ston, and Sainbayar Sukhbaatar. Meta-Rewarding Language Models: Self-Impgoving Alignment
18

---

Published as a conference paper at ICLR 2025
with MaA----s----Met-Juee July 2024b. URL http: bttp://arxx.v.org//b////// arXiv:240719594 [cs].
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-Grained Human Feedback Gives Bet- ter Rewards for Language Model Training, October 2023b.
2306. 01693 arXiv:2306.01699 [cs].
Duo Yang and Nora Hollenstein, PLM-AS:
Pre-trained Language Models Augmented with Scan- paths for Sentiment Classification. Proceedings of the Northern Lights Deep Learning Workshop. 4, January 2023. doi: 10.7557/18.6797.
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. Harnessing the Power of LLMSs in Practice: A Survey on ChatGPT and Beyond, April 2023. URI. htt.::ttp:/aaart
org/ abs/ 2304. 13712 arXiv:2304. 13712 [cs].
Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. RLCD: Reinforce- ment Learning from Contrastive Distillation for Language Model Alignment, March 2024. URL htttp://arxa
http//bs/23000 12950 arXiv:2307. 12950 [cs].
Jie Yu, Wenya Kong, and Fangfang Liu. CeER: A Nested Name Entity Recognition Model Incor- porating Gaze Feature. In Wenjie Zhang, Anthony Tung, Zhonglong Zheng, Zhengyi Yang, Xi- aoyang Wang, and Hongjie Guo (eds.), Web and Big Data, pp. 32 45, Singapore, 2024. Springer Nature Singapore. ISBN 978-981 087-2772--32-2
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank Responses Align Language Models with Human Feedback without tears, May 2023. URL httt:://aaxxxi org/ ab/p///ss//32//// 05302 #tXiv:230400330 [cs].
Leran Zhang and Nora Hollenstein. Eye-Tracking Features Masking Transformer Attention in Question-Answeringg Tasks. In Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessan- dro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint Interna- fional Conference on Computational Linguistics, Language Resources and Evaluation (LREC- COIING 2024), pp. 7057-7070, Torino, Italy, May 2024. ELRA and ICCL. URL https: /aaclanthology. wwg.
main. 619
Yifan Zhang, Jiliang Li, Zachary Karas, Aakash Bansal, Toby Jia-Jun Li, Collin McMillan, Kevin Leach, and Yu Huang. EyeTrans: Merging Human and Machine Attention for Neural Code Summarization. Proc. ACM Softw. Eng.. 1(FSE), July 2024. doi: 10.1 10/1644/364372. URL Place: New York, NY, USA Publisher: Asso-
https://doi wttp//...///..//....... ciation for Computing Machinery.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. SLiC-HF: Sequence Likelihood Calibration with Human Feedback, May 2023. URL http:
7/ wwwxii..rr//ab//2//
10425 arXiv:2305.10225 [cs].
Liammin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging LLM-as---Judge with MT-Bench and Chatbot Arena, December 2023. URL arXiv:2306.00 235522330000685 [cs].
http:
//arxiv.org/aatt
05685
A APPENDIX
A.1 IMPLEMENTATION DETAILS
This section provides further details on the implementation of our method. Subsection 1. vides more details on the datasets used and their preprocessing steps. In subsection A.112 further information is given about the models used for generating ET features, along with the specific pre- processing required for each. Subsection A.1.3 explains the process of mapping the fixations from the tokenizer used by the generation model to the tokenizer used by the Reward Model. Subsection AA..4]give more detials on the checkpoints used fot the RM backbone models. Finally, additional implementation details discussed in subsection A.1.5
pro-
19

---

Published as a conference paper at ICLR 2025
A.1.1 DATASET PROCESSING
In this subsection, we provide more details about the datasets used and the preprocessing to train the RM. We use two different datasets: OpenAssistant Conversations dataset's (OASSTI) (Köpf et al. 2023) and HelpSteer2 Wang et al. 2024c).
OASST1. A human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, resulting in over 10,000 complete and fully annotated conversation trees. The basic data structure is Conversation Tree (CT), with nodes representing written messages in a conversation. A CT's root node represents an initial prompt, given by the prompter. The data was collected using a web-app interface product of worldwide crowd- sourcing effort involving over 13,500 volunteers, dividing the collection of each tree into five sepa- rate steps: prompting, labelling prompts, adding reply messages as prompter assistant, labelling replies, and ranking assistant replies.
HelpSteer2. A CC-BY-4.0-licemssse open-source helpfulness dataset, designed to train state-of-the- art RM consisting on 10,000 response pairs. collects prompts mainly from ShareGPT focusing on user inputs and filtering out non-English and programming-related prompts for quality. The prompts are clustered into topics and sampled based on complexity to ensure diversity. Multi-turn prompts are generated using an in-house model, with responses sourced from various internal mod- els and human annotators. For each response, they annotate five attributes (helpfulness, correctness, coherence, complexity, and verbosity) on Likert-5 scale involving multiple annotators for each response, ensuring high-quality ratings across five attributes.
Conversation format and dataset preparation.
To fine-tune LLMs for human-Al interaction, we need to define a chat protocol. We use multi- message chat setup with a special header and termination tokens, similar to the one in Llama Dubey et al. (024). The header tokens differentiate the turns between the user and the system. For this, we use the apply www.ttthhatalleeoo
feature from FastTokenizers in the transformers library.
The tokenizer used by the Meta-Llama-3-3 3-BB--3--BBBB--- -nstruutt model already incorporates this chat format since this model has already undergone human alignment. Therefore, we use this format in our ex- periments. For the other models, employ the default chat format provided by their respective tokenizers. We add new tokens in the embeddings layer for these chat formats and we train them as part of our process. Below, we provide example of the template for each model.
Mcta-LJamma3-SS--nntruuc <begin of text> header id>user<end header id> Example Prompt <eot_id> <start heeder assistattttddndddd heuder id> Example Re- sponse <eotidd>
Met---Llama--8B: <im-start>user Example Prompt <im_end>
<im_start>assistom
Example Response <im end>
Mistral-7B: <>>[INST) Example Prompt 
 Example Response 

A.1.2 EYE-TRACKER FEATURES GENERATION MODELS
Special tokens are removed from the text before it is tokenized with the corresponding tokenizer used for the ET generator model. This is done to ensure that special tokens related to the chat format are not included in the input and are not assigned ET features them, since these tokens are just for the RM to understand the chat format.
First model: Model presented in Huang & Hollenstein 2023) The code for the model along with the weights is publicly available, so we used the pre-trained checkpoint and we adapted their code for our implementation. This model was trained on several eye-tracking datasets, including
Dundee (Kennedy et al. 2012), GECO (Cop et al., 2017), ZuCol (Hollenstein al., 2018). ZuCo2 Hollensteei et al. 2020a). More detailed information about this datasets is presented in Table 7
The best model achieves an mean squared emror (MSE) of 4.02 on randomly held-out test set (25%
20

---

Published as a conference paper at ICLR 2025
of all data). This model has 17.5M pararemetes and remains frozen during training. In Figure 3 we show an example of the synthetie total reading time (TRT) generated for the chosen and rejected response to prompt.
For training this model, since the fixation duration is distributed differently across corpora. the authors normalize the fixation duration for each corpus, by dividing by the mean duration of the corpus. Moreover, they map the duration values to discrete space [1, 2, K1. Using K-quantiles, the fixation values were partitioned into K subsets of nearly equal sizes, and each value was assigned to the index of the corresponding subset. The model is then trained in a multi-task setting, computing the mean and variance of the fixation duration. This quartile-bart baasdd processing used in other works (Huang et al. (023) that use ET data to improve performance in NLP tasks, and we believe it is part of the reason why we obtained better results with this combination when training the RM. The authors proposed method specifically for converting word-level TRT to token-level fixation data during model training. Initially, the TRT of word is assigned to its characters, then a small number is assigned to the last character of the word (mainly to give small values to punctuation). After tokenizing the word the span of each subword is obtained, and the maximum value in each span is taken as the final toven-level fixation data.
To use this model in our setup, we need to reverse this conversion process and recompute the fea- tures from token-level back to word-level, allowing to remap the features to different tokenizer. This is done by summing the orignal features for all tokens corresponding to a word and then dis- tributing them across the tokens mapped to the this conversion are explained in Appendix 1.3 and an example of the process in Table 8.
word in the other tokenizer. More details of
Second model: Model presented in Li & Rudzicz (2021) The code and training data are also publicly available, so we trained it following their original methodology and we adapted their code to incorporate it into our implementation. The model was trained using the ZuCo 1 (Hollenstein al. 2018) and ZuCo 2 (Hollenstein et al.) 2020a) and PROVO (Luke & Christianson (0188) datasets. For the ZuCo datasets, 800 sentences (15.7 tokens) were provided as training data and 191 sentences (3.5k tokens) were held out for evaluation. Information about the datasets used to train these models in TTble 7. The model based on RoBERTa (Liu et al., 2019) with a regression head on each token. This head is linear layer that outputs five features: FFD, fixProp, GPT, TRT, and nFix Table Ij. mean absolute error (MAE) for each feature is presented in Table This model has 125M parameters and remains frozen during training.
In this generative model, the conversion of word-level features to token-level features during training is done by assigning the features of word to the first token and it is assumed that the rest of the tokens of this word do not have features. We reversed this process similarly during inference by forcing the predictions for tokens that not the first in word be zero. Since the maximum sequence length for RoBERTa 512 and we are dealing with longer sequences, we implemented a sliding window approach. We split the input into sequences of 512 tokens with a 50-token overlap. After processing, we combine the results using a linear combination.
Table MAE performance of the model reported Li Rudzicz 2021 fixProp
nFix
FFD
GPT
TRT
All (Dev)
3,984
0.713
2.424
1,556
10.781
3.892
Table 7: Overview of different corpora used to train the ET features generator models.
Corpus
Lang.
Sents.
Tokens
Subjects
Reference
Dundee
EN
2367
58598
20
Kennedy et al.
201
Provo
EN
189
2659
84
Luke Christansoon
2018
ZuCo 1
EN
300
6588
12
Hollenstein et al.
(2013
ZuCo 2
EN
349
6828
18
Hollenstein et 2020b
Geco
EN
2449
23
Cop et al. (201
21

---

Published as a conference paper at ICLR 2025
here's
limerick
about cucumbers: there
once
there
once
was
cucumber
50
green
the
biggest
green
cupumber,
that really
quite
you've
ever
seen.
INGS
fresh
and
cool
bloomer,
grew
on
the
vine,
WDS
pool.
pickie.
oh
fine.
until
I
bocarn
Somcone's
consumer.
not
50
serene.
TRT per word in chosen response.
TRT per word in rejected response.
Figure 3:
TRT generated by first model (Huang & Hollenstein 2023 of the chosen and rejected response to prompt 'Create a limerick about cucumbers'. Deeper colour represents longer fixation.
A.1.3 MAPPING ET FEATURES BETWEEN DIFFERENT TOKENIZERS
Both ET features generator models used are based on different tokenizers, which are also different from the tokenizers employed by the based models used as RMM. As result, the number of tokens n in the input for the reward model and the number of tokens w for the ET features may not be the same. For GazeAdd, to be able to combine clementwise the ET feature embedding and the text embedding, they must have the same temporal dimensions. Therefore, we need to map the ET features per token from the ET tokenizer the tokens of the RM tokenizer. Specifically, convert our fer E Rwx (w is the number of tokens, and f is the number of features) to fet fraapedd " where n is the number of tokens in the RM imput. For that. we perform mapping between the two tokenizers to obtain the mapped features fet pmopped
To map tokens generated by two different tokenizers, we use our EyeTruckyy python library that will be publicly released. First, we perform an initial mapping of tokens to the words they belong to each tokenizer with some properties of FastTokenizers from the transformers library 8 Then, we map words from tokenizer to the words in the other and finally, assume that the combination of the tokens that are mapped to a word in one tokenizer correspond to the tokens that are mapped to the word that is mapped to the initial word in the other tokenizer. Each row in Table 8 refers to step in this process.
For each predictor, reverse the method used to convert word-level features into token-level fea- tures (more details in Appendix (A.12)) but passing from tokens in the first one, to tokens in the second tokenizer. For example, if for the first ET features predictor models tokens 41 l2 mapped to tokens f1, f2, ts in another second tokenizer, the values sum for all the tokens in the first list and distribute them equally across all the tokens in the second list: being 1 (Is TRT) and to (2s TRT) each of t f2, t3 are assigned a TRT of (1 + 2)/3 1s. In Table i represented a example of this process where row TRT(1) are the final TRT mapped for the first ET predictor and TRT(2) for the second one. Finally, because special chat tokens were removed when generating the ET features, we assign value 0 for all features in this tokens. At the time of publishing this work, some of the tokenizer functionalities needed for alignment between tokenizers were not available in Mistral 7B.
Table 8: Example of mapping TRT between two different tokenizers. TRT (1) represents the process used for the first ET predictor, and TRT(2) for the second ET predictor.
Tokenizer 1
Tokenizer 2
Words
astrophotography
astrophotography
Tokens str
"Astro', "photo', "graphy']
[Č; 'Ast', 'roph', 'ot', 'ography"
Tokens idx
[22, 23, 24] [15001, 17720, 16369] (11.23, 11.49, 10.16] [24.53, 0, 0]
[23, 24, 25, 26, 271]
Tokens IDs TRT (1)
[198, 62152, 22761, 354, 5814] [6.58, 6.58, 6.58, 6.58, 6.58]
TRT (2)
[24.53, 0, 0, 0, 0]
22

---

Published as a conference paper at ICLR 2025
1.1.4 MODELS
9 Dubey et al. 2024), Llama 3 8B-instruct 10 Dubey et al. 2024) and Mistral 7B (Jiang et al. 2023a 3a)
As RM base models we use the pretrained checkpoint Hugging Face for Llama 3 8B
A.1.5 TRAINING DATAILS
During training, the ET features predictor model remains frozen. The RM is fine-tuned on top of the open-source models using QLoRA (Dettmers et al.) 2023 based on Low-Rank Adaptation (LoRA) (Hu et al. 202T), which fine-tunes select dense layers by optimizing low-rank decomposi- tion matrices representing weight changes, instead of directly updating pre-trained weights. QLoRA introduces memory optimization techniques such as the 4-bit NormalFloat (NF4), novel data type, to improve performance without increasing memory usage. Following Dettmers al. (2023) we use hyperparameters: r=8, alpha=32, and dropout=0.1.
We also fine-tune the RM embedding layer, since we are adding new tokens for the chat format and special separators tokens in our RewardConcat method (section 4). Also, the last layer added to the RM for the scalar reward is trained from scratch without adapters. Our implementation is based in pytorch and
transformers 12 from Hugging Face.
Hardware. We trained the models on servers equipped with 2x Intel Xcon Platinum 8470 CPUs, ITB of RAM, and either 2x NVIDIA H100 (80GB) 4x NVIDIA A100 (80GB) GPUs. We always train using only GPU at a time per each model and training times were between 20 and 50 hours depending mainly on the number of steps between model evaluations.
B REWARD BENCHMARK
As described in section 6, future direction would be train Reward Model larger model with more data. It has been proven the scaling trends for the reward model; More data and a larger- size model generally improve accuracy (Touvron et al., 2023). Nevertheless, as complement to our results, we evaluated our trained models on the dataset with the best results, OAASTI, this RewardBench, a benchmark for Reward Models. RewardBench, proposed in Lambert et (2024), is a benchmark designed to evaluate the performance and safety of reward models. It consists of set of datasets intended for measuring how reward models perform on challenging prompts across chat, reasoning, and safety domains, using a trio structure of prompt-choxen-reeette pairs. It comprises 2985 diverse tasks, each sample is formatted as a prompt with a manual or human-verified chosen and rejected completion. Due to its diversity of tasks (4 categories and 23 sub-categories) this benchmark minimizes the likelihood of overfitting. Task accuracy is calculated based on whether the chosen response receives higher reward than the rejected response. We directly evaluate their open dataset reward-bench
C HYPER PARAMETER TUNING
We performed hyperparameter tuning for the GazeConcat method and the baseline, and we repli- cated them in the GazeAdd method, as testing all combinations is computationally very expen- sive. For each dataset, 15% is reserved for validation to perform hyperparameter tuning. The best- performing checkpoints are selected based on the lowest validation loss and are subsequently used for performance evaluation in the test split. We trained for two epochs, as described in Wang et al.) (024c) and in line with trends observed in TTovron et al. (2023) where they found that training longer can lead to over-fitting We perform a grid search for the optimal batch size, testing {8. 16, 32) values. AdamW optimizer (Loshchilov & Hutter 2019) is used and the learning rate tuned within the range of (1, 5, 10, 50)c-6, inspired by the values reported in Touvron et al. (2023):
https:// [tteas///maaaaaseecc///ee/llllls
23

---

Published as a conference paper at ICLR 2025
Epoch
0.000001
1e-00
Valldation Loss
Epoch
Figure 4: Validation loss with different LR ConcatReward, batch size features: fl and Meta-
Lamaa-3-8B-Inae base model
Validation Loss Epoch
Validation Loss
Epoch
Figure 5: Validation loss with different batch size, learning rate: 5e-5, features: fl and Meta-Llama- Inst-unst base model
Cui et al. (2024): ) Wang et al. (024c). Additionally, explored different scheduler configuration, comparing constant, linear, and cosine with a minimum learning rate.
In general, the parameter that most affected the validation results the learning rate. For the others, we ended up choosing values that worked well across all combinations. We achieved better results in both the baseline and the models concatenating the ET features with learning rate of 0.0005 (Figure 4). In Figure 6a and Figure 6b) the validation loss and learning rate with different schedulers shown. For lower learning rates, such 0.00001, the scheduler had little effect. How- ever, with higher learning rates, using a scheduler helped to mitigate overfiting. In Figure 7a and Figure 7b represents validation loss and learning rates with a higher learning rate of 0.0005, We opted to use this 0,00005 learning rate for all experiments, employing cosine learning rate sched- uler with a minimum learning rate of 0.7, in line with other studies such as Wang et al. (2024c); Touvron et al. (2023). We did not find significant effect of training batch size on validation accu- racy, but we opted for value of 8, which often (especially with high learning rates and without scheduler) was the one that tended to overfit the least (Figure 5).
ET features projector The PyTorch architecture of our ET features projector is shown below. mum-ffeauuee varies between 1, 5, and (depending the configuration used fcombi, fcomb2.6. fcomb2.2 2 subsection 5.2). P1.P2 are dropout values. Finally, after testing different combinations, used 0.1 and 0.3. This model has 0.53M parameters.
Listing 1: PyTorch architecture of gaze features projector
24

---

Published as a conference paper at ICLR 2025
Loss Epoch
Learning Rate
Incer
de
cooite 2.5
0.0
Learning Rate
130
(a) Validation Loss Comparison for different LR Schedulers
(b) Learning rate comparison for different LR Schedulers
Figure 6: ConcatReward, LR: 0.00001, features: fl and Meta-LLama-3-8B-nnsee base model
Validation Loss vs Epoch
Learning Rate vs Epoch
0/9
Velldaton Lowe
Leamiing Rate
Epoch
300
0.0
655
015
Epoch
100
1.5
130
(a) Validation Loss Comparison for different LR
Learning comparison for different LR Schedulers
Schedulers
Figure 7: ConcatReward, LR: 0.00005, features: fl and Oeta-Llama-3-8B--mmee base model
self. bikxattons__mbedddggggoojjeeeeee nn Ln.ein (num_features, 128), (anyeNNomm(128) nn.ReLU nn.Dropout (p=p_1), In. (128, hidden_size), nn.Dropout (p=p_2),
nn im.eueenttall
25


Reviews:
[{"review_id": "zHx1Gezzt6", "reviewer": "Reviewer_sHLc", "summary": "The paper examines the possibility of using eye-tracking data to train language reward models. The authors integrate synthetic eye tracking data from existing models into the reward model using two strategies: concatenation or addition of eye tracking features. They test their methods across multiple eye tracking models and LLMs and show that eye tracking data can improve reward models at a small to medium scale.", "questions": "Q1. You don't scale this approach to larger models/datasets (understandably!). Do you foresee any potential challenges of scaling this approach other than compute cost? Do you believe this data would be best used to fully replace existing methods for reward modeling, or in conjunction?\n\nQ2. Somewhat related to the above-- do you have a sense of any error category differences between RMs trained with eye tracking data and without?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. The idea of using eye tracking data as reward model signal is exciting, interesting, and well-explored in this work. The paper analyzes several methods of integration and discusses the pros and cons of (synthetic or real) eye tracking data.\n\nS2. The paper is beautifully written and clearly lays out prior work in both reward modeling and eye tracking for NLP. It was very easy to follow.\n\nS3. The appendix reports details on hyperparameter tuning, implementations, and modeling assumptions. I believe it would be easy to replicate this work with the details provided.", "weaknesses": "W1. Very minor, but I think section 2+5 could be better structured, perhaps by moving them both into section 2. I was a bit surprised on my first read to see a related work section near the end, since section 2 discusses much of the reward modeling literature; I think it makes sense to put these in the same spot, and I liked the placement near the beginning.\n\nW2. Because the datasets are somewhat constrained, it's not clear that this would provide consistent gains over existing reward models in a large-scale setting (could this be a useful signal only in small data regimes?). I don't think the paper necessarily needs to provide evidence against this concern, as this might require excessive amounts of compute, but I think more discussion of this possibility would help temper the claims in the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2XrQt9QrMf", "reviewer": "Reviewer_TgMi", "summary": "The paper introduces GazeReward, a framework that incorporates eye-tracking data into the reward modeling phase of RLHF to better align LLMs with human preferences. This represents a novel solution to the reward modeling task with extra information from eye-tracking features. Experimental results also show the effectiveness of the proposed approach with an impormvent in reward modeling accuracy.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Eye-tracking is being introduced as a novel feature to assist reward modeling. \n+ Substantial performance improvements with the GazeReward framework", "weaknesses": "+ The experiments show that eye-tracking signals can improve the accuracy of reward modeling. As the reward model is used for RLHF processes like PPO and Best-of-n sampling, I would suggest the authors add relevant experiments to show the improvement in this down-stream application of the reward model.\n+ What is the size of eye-tracking feature extractor? If the size of the eye-tracking component is not negligible, it might not be a fair comparison between the original model the model with eye-tracking feature extractor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mZCcXy90OW", "reviewer": "Reviewer_WBeE", "summary": "This work incorporates features extracted from eye-tracking prediction models into reward models for RLHF. They find that adding eye-tracking information improves reward modeling accuracy on two reward modeling datasets. In addition, the authors also perform ablation studies to study what type of eye-tracking features lead to the highest reward modeling accuracy.", "questions": "- L.46-48 Could you elaborate how eye-tracking data mitigates the problems you cited in human feedback for RLHF? Would the eye-tracking features not also reflect biases from humans from which the ET models were trained on, or their domain expertise?\n- L.48-49 How is scalable oversight relevant to the problem studied in this paper?\n- L. 73 It's unclear what you mean by \"better temporal and spatial resolution\", could you please clarify?\n- L. 74 What is the connection to attention patterns and human preference for reward modeling?\n- S. 3.1 It would be helpful to describe more in this section what these ET models are trained on, the accuracy of these ET models.\n- I would be curious to see results, or if not feasible a discussion, on how your GazeReward framework compares to if we used eye-tracking data directly collected on the preference prompts instead of features generated by an ET prediction model\n- Please also discuss the ethical impact of this work, mainly to address concerns that incorporating an ET model may amplify biases present in the ET models\n- Suggestion: make the text in your figures bigger", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The idea behind incorporating eye-tracking features to reward models is quite original, and to my knowledge, has not been previously explored. The authors are also able to demonstrate accuracy gains in reward modeling after adding eye-tracking features, which brings a positive impact to RLHF that is widely used in current LLMs.", "weaknesses": "I find two main weaknesses that prevent me from recommending this paper for acceptance, but that could be feasibly addressed by the authors. First, it is difficult for me to assess whether the improvements on reward modeling reported in the paper are significant. According to Table 2, the test sets used for reward model evaluation are quite small (416 and 364 examples) and I did not find information on the statistical significance of the differences in results. I am also skeptical whether the improvements in scores would be robust to different random seeds and other parameters. It would be beneficial to add statistical significance tests and further discussion on how you ensured your findings are robust.\nThe second weakness is the lack of motivation in using eye-tracking data. There are a few points the authors allude to in the introduction (see some of my questions below), but it would be very helpful to explain the intuition on why eye-tracking data would have a connection or be correlated with human preferences. Intuitively, I would think that people would spend more time gazing at sentences that are difficult to parse for example, rather than having a preference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)