RadEyeVideo: Enhancing general-domain Large Vision Language Model for chest X-ray analysis with video representations of eye gaze

Authors: Yunsoo Kim, Jinge Wu, Honghan Wu
Primary area: foundation or frontier models, including LLMs
Keywords: LVLM, Eye Gaze, Video, Medicine, Medical Image, Chest X-ray, Chest X-ray Report Generation

Abstract:
Large Vision-Language Models (LVLMs) have demonstrated promising performance in chest X-ray (CXR) analysis. To enhance human-computer interaction, several studies have incorporated radiologists' eye gaze, typically through heatmaps or textual prompts. However, these methods often overlook the sequential order of eye movements, which could provide valuable insights by highlighting both the areas of interest and the order in which they are examined. In this work, we propose a novel approach called RadEyeVideo that integrates radiologists’ eye-fixation data as a video sequence, capturing both the temporal and spatial dynamics of their gaze. The video, featuring a red gaze point overlaid on CXR images, emphasizes regions of focused attention during interpretation. We evaluate this method in CXR report generation and disease diagnosis using three general-domain, open-source LVLMs with a video input capabilities. When prompted with eye-gaze videos, model performance improves by up to 25.4% on Impression generation task and on average 7.9% for all tasks using scaled evaluation metrics. Our approach enhanced open-domain LVLM models, when combined with exemplar reports for in-context learning, outperform medical models as well as those specifically trained for CXR report generation on the benchmark dataset. This work highlights that domain expert's knowledge (eye-gaze information in this case), when effectively integrated with LVLMs, can significantly enhance general-domain models' capabilities in clinical tasks, pointing out a new effective approach of utilising LVLMs in healthcare and beyond.

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730298485378

thread_type:
official_review

<thread_object>
creator: reviewer_ge5k

<content>
summary:
This article introduces a novel prompting method called RadEyeVideo, which presents radiologists’ eye-tracking data as a video sequence to capture the temporal and spatial order of their gaze (i.e., scan paths). The authors evaluated the effectiveness of this approach in chest X-ray report generation and disease diagnosis tasks using large vision-language models (LVLMs) with video input capabilities. Results show that incorporating eye-gaze video prompts improved model performance by 25.4% on the Impression generation task, with an average performance increase of 7.9% across all tasks.

strengths:
1. The RadEyeVideo method converts radiologists’ eye-tracking data into video sequences for use in chest X-ray report generation and diagnostic tasks, significantly enhancing the performance of general large vision-language models (LVLMs). Experimental results indicate a 25.4% improvement in the Impression generation task and an average improvement of 7.9% across all tasks, outperforming models specifically designed for medical applications.
2. Comprehensive Evaluation of Eye-Tracking Prompting Methods: This study presents the first thorough evaluation of various eye-tracking integration techniques, including static heatmaps, eye-tracking text prompts, and dynamic video prompts. The results indicate that RadEyeVideo outperforms the other methods in terms of diagnostic accuracy and clinical relevance, establishing a new standard for eye-tracking data prompts.

weaknesses:
- Methodological Clarity: a) The paper lacks sufficient detail about the video prompt integration process, particularly how the video information is encoded and processed by the LVLMs. The technical implementation of combining video sequences with textual prompts could be more thoroughly explained. b) There's limited discussion of potential alternatives to full video sequence processing that might achieve similar results with lower computational costs.

- Data Collection and Generalizability Limitations: a) The methodology requires synchronized eye-tracking data collection during radiologist readings, which is resource-intensive and difficult to scale. b) The approach may not generalize well to scenarios where real-time eye-tracking data is unavailable or impractical to collect. c) The current evaluation is limited to a relatively small dataset (2,298 CXR images), raising questions about broader applicability.

- Computational Efficiency Concerns: The direct use of complete eye-tracking video sequences as prompts likely increases computational overhead significantly. While the authors mention sampling k frames (typically 16) from the total sequence, there's limited analysis of the computational trade-offs or optimal sampling strategies. The method may be computationally prohibitive for real-time clinical applications.

questions:
Please address the concern raised in weakness part.
Moreover, there are some questions may help you to address the weakness.
1) What is the computational overhead of processing video prompts compared to traditional image-only or text-only prompts?
2) Have you explored more efficient alternatives to using complete video sequences, such as key frame selection or compressed representations? 3) How do you envision this approach being implemented in real-world clinical settings where real-time eye-tracking data may not be available? 4) Have you considered alternative methods for generating synthetic eye-tracking data that could make the approach more broadly applicable? 5)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730547269928

thread_type:
official_review

<thread_object>
creator: reviewer_8pz9

<content>
summary:
This paper introduces eye-tracking data into medical LVLMs and uses video formats to retain the temporal and spatial characteristics of radiologist’s eye movements, providing more comprehensive prior information for the diagnostic model. Compared to heatmaps and textual prompts, this method fully considers temporal features. Evaluation on multiple downstream tasks demonstrate the effectiveness of the approach.

strengths:
1. This work introduces radiologists' eye-tracking data into LVLMs in video format, highlighting the temporal features of eye movement sequences.
2. The experiments in the paper are comprehensive, validating multiple diagnostic tasks across various datasets.

weaknesses:
1. The paper mentions using simple stacking of gaze points for heatmap generation, but it does not specify the radius size of the gaze points. Different gaze point sizes can affect the model's interpretation.
2. In-context learning often heavily relies on the provided examples, which can significantly influence the generated results. The paper does not discuss what constitutes suitable examples.

questions:
1. Are the eye-tracking data collected from a single radiologist? If there are multiple radiologists, how do you handle individual behavioral differences?
2. How is the duration mentioned in line 176 used to control the frame count? This aspect is not explained in the paper. Similarly, the sampling method mentioned in line 194 is not clearly described.
3. Does the sampling of eye-tracking data risk losing short-term diagnostic behaviors, potentially affecting the diagnosis of various types of diseases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730570459641

thread_type:
official_review

<thread_object>
creator: reviewer_r59d

<content>
summary:
The paper introduces an innovative approach that incorporates radiologists' eye-tracking data as video sequences. This method captures both spatial and temporal patterns in gaze, which provides a more accurate representation of radiologists' attention during chest X-ray interpretation. The approach was evaluated using three general-domain LVLMs, showing significant improvements.

strengths:
- RadEyeVideo's use of video-based eye-gaze data is a unique contribution that effectively captures the temporal and spatial dynamics of radiologists' focus. I like this idea.
- The study demonstrates substantial improvements, particularly in impression generation, highlighting RadEyeVideo's effectiveness in enhancing diagnostic tasks.
- The language is clearly presented. The authors use precise and concise language so that the reader can easily understand the methodology, and results of the study.

weaknesses:
- Although this idea is interesting, it still relies on temporal and spatial information in the inference phase, which is difficult to apply to real clinical scenarios. Do the authors consider involving multiple information inputs only in the training phase and simulating zero-shot scenarios as much as possible in the inference phase?
- The study’s findings are limited by the small size of the MIMIC-Eye dataset, which may not fully capture the variability in real-world clinical settings, raising questions about the generalizability of the results.
- The comparisons primarily focus on selected models with minimal tuning for this domain. Including a wider range of task-specific medical LVLMs could provide a more comprehensive evaluation.

questions:
- Why the paper lacks an section of related work? e.g., some recent evaluation work of Med-LVLMs [1,2,3]
- The formats of reference is weird. e.g., in Line 311-321. Please check it.

[1] Gu Z, Yin C, Liu F, et al. MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context[J]. arXiv preprint arXiv:2407.02730, 2024.

[2] Jiang Y, Chen J, Yang D, et al. MedThink: Inducing Medical Large-scale Visual Language Models to Hallucinate Less by Thinking More[J]. arXiv preprint arXiv:2406.11451, 2024.

[3] Xia P, Chen Z, Tian J, et al. CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models[J]. arXiv preprint arXiv:2406.06007, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695970552

thread_type:
official_review

<thread_object>
creator: reviewer_lhmg

<content>
summary:
To enhance the reliability of large vision-language models (LVLMs) in real clinical environments, this study proposes a novel video prompting method called RadEyeVideo, which integrates radiologists' eye-tracking data as video sequences, capturing the spatial and temporal dynamics of gaze. Red fixation points are superimposed on CXR images to highlight the areas that doctors pay attention to, so as to dynamically present the doctor's eye movement path. Improve the ability of radiologists or AI models to diagnose chest diseases by providing rich contextual information.

strengths:
1. The proposed method, RadEyeVideo, combines video, text, and eye movement data to realize the fusion of multi-modal information and improve the accuracy and efficiency of diagnosis.
2. The authors conducted a thorough assessment of various eye-tracking integration techniques, providing strong empirical support for their claims.

weaknesses:
1. In Figure 2, the authors illustrate different prompting methods; however, the figure does not clearly distinguish between the use of text descriptions and video inputs. This ambiguity makes it challenging to understand whether the authors used text descriptions to guide the prompt along with the video input or if they only provided video data. 
2. The author's experiments conducted on only one dataset are clearly insufficient in terms of persuasiveness. This limitation may affect the generalizability and applicability of the research findings. To enhance the credibility of the study, it is recommended that the authors validate their approach on different datasets.
3. When the authors use radiologist's eye-tracking sequences as video input, this approach does enhance the model's understanding of prior knowledge in the diagnostic reading process to a certain extent. However, converting eye-tracking data into video format substantially increases the number of tokens, leading to significant computational resource consumption and longer inference times for the model.

questions:
1. The RadEyeVideo method integrates the eye movement data of the radiologist as a video sequence to capture the spatiotemporal dynamics of his gaze. However, the article does not explain in detail how this video sequence is specifically generated, for example, how the eye movement data is sampled and converted into video frames, and how these frames contain temporal and spatial information.
2. In Section 2.3 INPUT REPRESENTATION, to fit the input requirements of LVLM, the authors uniformly sampled the video sequence and selected 16 frames as inputs. However, the article does not explain why 16 frames were chosen and whether this choice had a significant impact on model performance. In addition, whether the impact of using more or fewer frames on model performance has been explored is also a question worth exploring. This helps to further understand the effect of video data length and quality on model performance.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes RadEyeVideo, a novel prompting method that integrates radiologists' eye-tracking data as video sequences to capture the spatial and temporal dynamics of gaze. By leveraging these video prompts, the study enhances large vision-language models for chest X-ray diagnosis and report generation.

Reviewers found that the strengths of this paper lie in its RadEyeVideo method integrating radiologists’ eye-tracking data as video sequences to enhance LVLM performance, comprehensive evaluations validating its superior performance, and a clearly articulated methodology with broad applicability in medical imaging. However, reviewers also have major concerns in generalizability, methodological clarity, and computational efficiency. The study relies on a single, small dataset (MIMIC-Eye, 2,298 images), limiting its applicability to broader clinical settings (Reviewers Lhmg, r59d, ge5k). The methodology lacks detail on video prompt integration and alternatives to full video sequence processing, which could reduce computational overhead and improve scalability (Reviewers ge5k, Lhmg). Additionally, the reliance on synchronized eye-tracking data raises feasibility concerns for real-world applications, and comparisons with minimally tuned models leave the evaluation incomplete (Reviewers r59d, ge5k). These issues limit the study’s persuasiveness and practicality.

During the discussion, the authors acknowledged the limitation of using a single dataset (MIMIC-Eye) due to availability constraints but emphasized that their method is dataset-agnostic and can be applied to future, larger datasets. They answered that converting eye-tracking data to video format results in only a marginal increase (10% GPU memory) in computational resource consumption. The authors acknowledged the challenges of synchronized eye-tracking data collection but emphasized that their approach improves performance compared to task-specific LVLMs without retraining. They also provided detailed explanations of the video prompt integration process and added missing details. The reviewers provided mixed final responses. Reviewer Lhmg did not respond. Reviewer r59d remained unconvinced, attributing improvements partly to the lack of pretraining in baseline models and withheld acceptance. Reviewer 8pZ9 appreciated the rebuttal and raised their score. Reviewer ge5k acknowledged that most concerns were addressed but maintained generalizability concerns, particularly about using mouse cursor movements as a proxy for eye gaze. While some issues were resolved, concerns about generalizability and robustness persist.

Based on the paper, reviewers' comments, and discussions, this work, while presenting intriguing ideas, is not yet sufficiently developed for publication due to unresolved concerns regarding generalizability, methodological robustness, and practical applicability. While the authors provided thoughtful responses to many issues, key challenges remain. Aligning with the reviewers, this AC has two main concerns. First, the study relies on a single, small dataset, raising questions about its broader applicability, especially in scenarios without eye-tracking data, where suggested proxies like mouse cursor movements lack investigation. Second, the methodological contributions, though novel, leave important questions about scalability and real-world clinical implementation unanswered. While some reviewer acknowledged the authors’ efforts and raised the score, the persistent concerns regarding generalizability and practical impact ultimately outweigh the strengths of this work.