Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models

Authors: Angela Lopez-Cardona, Carlos Segura, Alexandros Karatzoglou, Sergi Abadal, Ioannis Arapakis
Primary area: foundation or frontier models, including LLMs
Keywords: reward model, RLHF, visual attention, LLMs, eye tracking, implicit feedback

Abstract:
Advancements in Natural Language Processing (NLP), have led to the emergence of Large Language Models (LLMs) such as GPT, Llama, Claude, and Gemini, which excel across a range of tasks but require extensive fine-tuning to align their outputs with human expectations. A widely used method for achieving this alignment is Reinforcement Learning from Human Feedback (RLHF), which, despite its success, faces challenges in accurately modelling human preferences. In this paper, we introduce GazeReward, a novel framework that integrates implicit feedback -- and specifically eye-tracking (ET) data -- into the Reward Model (RM). In addition, we explore how ET-based features can provide insights into user preferences. Through ablation studies we test our framework with different integration methods, LLMs, and ET generator models, demonstrating that our approach significantly improves the accuracy of the RM on established human preference datasets. This work advances the ongoing discussion on optimizing AI alignment with human values, exploring the potential of cognitive data for shaping future NLP research.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730696471353

thread_type:
official_review

<thread_object>
creator: reviewer_wbee

<content>
summary:
This work incorporates features extracted from eye-tracking prediction models into reward models for RLHF. They find that adding eye-tracking information improves reward modeling accuracy on two reward modeling datasets. In addition, the authors also perform ablation studies to study what type of eye-tracking features lead to the highest reward modeling accuracy.

strengths:
The idea behind incorporating eye-tracking features to reward models is quite original, and to my knowledge, has not been previously explored. The authors are also able to demonstrate accuracy gains in reward modeling after adding eye-tracking features, which brings a positive impact to RLHF that is widely used in current LLMs.

weaknesses:
I find two main weaknesses that prevent me from recommending this paper for acceptance, but that could be feasibly addressed by the authors. First, it is difficult for me to assess whether the improvements on reward modeling reported in the paper are significant. According to Table 2, the test sets used for reward model evaluation are quite small (416 and 364 examples) and I did not find information on the statistical significance of the differences in results. I am also skeptical whether the improvements in scores would be robust to different random seeds and other parameters. It would be beneficial to add statistical significance tests and further discussion on how you ensured your findings are robust.
The second weakness is the lack of motivation in using eye-tracking data. There are a few points the authors allude to in the introduction (see some of my questions below), but it would be very helpful to explain the intuition on why eye-tracking data would have a connection or be correlated with human preferences. Intuitively, I would think that people would spend more time gazing at sentences that are difficult to parse for example, rather than having a preference.

questions:
- L.46-48 Could you elaborate how eye-tracking data mitigates the problems you cited in human feedback for RLHF? Would the eye-tracking features not also reflect biases from humans from which the ET models were trained on, or their domain expertise?
- L.48-49 How is scalable oversight relevant to the problem studied in this paper?
- L. 73 It's unclear what you mean by "better temporal and spatial resolution", could you please clarify?
- L. 74 What is the connection to attention patterns and human preference for reward modeling?
- S. 3.1 It would be helpful to describe more in this section what these ET models are trained on, the accuracy of these ET models.
- I would be curious to see results, or if not feasible a discussion, on how your GazeReward framework compares to if we used eye-tracking data directly collected on the preference prompts instead of features generated by an ET prediction model
- Please also discuss the ethical impact of this work, mainly to address concerns that incorporating an ET model may amplify biases present in the ET models
- Suggestion: make the text in your figures bigger
</content>
</thread_object>

</thread>
<thread>
creation date:
1730786975032

thread_type:
official_review

<thread_object>
creator: reviewer_tgmi

<content>
summary:
The paper introduces GazeReward, a framework that incorporates eye-tracking data into the reward modeling phase of RLHF to better align LLMs with human preferences. This represents a novel solution to the reward modeling task with extra information from eye-tracking features. Experimental results also show the effectiveness of the proposed approach with an impormvent in reward modeling accuracy.

strengths:
+ Eye-tracking is being introduced as a novel feature to assist reward modeling. 
+ Substantial performance improvements with the GazeReward framework

weaknesses:
+ The experiments show that eye-tracking signals can improve the accuracy of reward modeling. As the reward model is used for RLHF processes like PPO and Best-of-n sampling, I would suggest the authors add relevant experiments to show the improvement in this down-stream application of the reward model.
+ What is the size of eye-tracking feature extractor? If the size of the eye-tracking component is not negligible, it might not be a fair comparison between the original model the model with eye-tracking feature extractor.

questions:
See weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730856848085

thread_type:
official_review

<thread_object>
creator: reviewer_shlc

<content>
summary:
The paper examines the possibility of using eye-tracking data to train language reward models. The authors integrate synthetic eye tracking data from existing models into the reward model using two strategies: concatenation or addition of eye tracking features. They test their methods across multiple eye tracking models and LLMs and show that eye tracking data can improve reward models at a small to medium scale.

strengths:
S1. The idea of using eye tracking data as reward model signal is exciting, interesting, and well-explored in this work. The paper analyzes several methods of integration and discusses the pros and cons of (synthetic or real) eye tracking data.

S2. The paper is beautifully written and clearly lays out prior work in both reward modeling and eye tracking for NLP. It was very easy to follow.

S3. The appendix reports details on hyperparameter tuning, implementations, and modeling assumptions. I believe it would be easy to replicate this work with the details provided.

weaknesses:
W1. Very minor, but I think section 2+5 could be better structured, perhaps by moving them both into section 2. I was a bit surprised on my first read to see a related work section near the end, since section 2 discusses much of the reward modeling literature; I think it makes sense to put these in the same spot, and I liked the placement near the beginning.

W2. Because the datasets are somewhat constrained, it's not clear that this would provide consistent gains over existing reward models in a large-scale setting (could this be a useful signal only in small data regimes?). I don't think the paper necessarily needs to provide evidence against this concern, as this might require excessive amounts of compute, but I think more discussion of this possibility would help temper the claims in the work.

questions:
Q1. You don't scale this approach to larger models/datasets (understandably!). Do you foresee any potential challenges of scaling this approach other than compute cost? Do you believe this data would be best used to fully replace existing methods for reward modeling, or in conjunction?

Q2. Somewhat related to the above-- do you have a sense of any error category differences between RMs trained with eye tracking data and without?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733170535580

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final comment by authors

comment:
As a final comment on our paper review, we’d like to summarize the clarifications provided during the rebuttal:
* Reviewer sHLc: Expanded the discussion on large-scale settings, models, and datasets.
* Reviewer TgMi: Added details on ET feature extractor models and explored the effects of model size.
* Reviewer WBeE: Included tables showing statistical significance, presented state-of-the-art examples with similar training methods, increased training seeds across all 36 runs, and committed to reporting these results upon acceptance. We also addressed the motivation of ET signals by citing relevant literature and updated the Impact Statement accordingly.

We have requested feedback from both TgMi and WBeE on our responses. As the rebuttal period concludes, we welcome any final clarifications and kindly ask that reviewers consider our updates when finalizing their scores. Thank you for your time and consideration.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel framework that integrates eye-tracking data as implicit feedback into the reward model for aligning LLMs with human expectations, called GazeReward. The reviewers appreciate the novelty of the approach and the performance gains it can achieve by incorporating gaze data. However, they also expressed concerns about the limited scale of the experiments, the lack of clear justification for using eye-tracking data, the absence of downstream evaluation, and the lack of statistical analysis.