Title: From Human Attention to Diagnosis: Semantic Patch-Level Integration of Vision-Language Models in Medical Imaging
Venue: NeurIPS 2025
Accepted: yes
Keywords: Gaze prediction

medical image analysis

CXR

deep learning

fixation modeling

attention mechanisms

visual attention

neural networks
Abstract: Predicting human eye movements during goal-directed visual search is critical for enhancing interactive AI systems. In medical imaging, such prediction can support radiologists in interpreting complex data, such as chest X-rays. Many existing methods rely on generic vision--language models and saliency-based features, which can limit their ability to capture fine-grained clinical semantics and integrate domain knowledge effectively. We present \textbf{LogitGaze-Med}, a state-of-the-art multimodal transformer framework that unifies (1) domain-specific visual encoders (e.g., CheXNet), (2) textual embeddings of diagnostic labels, and (3) semantic priors extracted via the logit-lens from an instruction-tuned medical vision--language model (LLaVA-Med). By directly predicting continuous fixation coordinates and dwell durations, our model generates clinically meaningful scanpaths. Experiments on the GazeSearch dataset and synthetic scanpaths generated from MIMIC-CXR and validated by experts demonstrate that LogitGaze-Med improves scanpath similarity metrics by 20--30\% over competitive baselines and yields over 5\% gains in downstream pathology classification when incorporating predicted fixations as additional training data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LogitGaze-Med, a multimodal framework that integrates domain-specific visual and textual embeddings with semantic patch-level priors to predict task-driven scanpaths in medical images. The textual embeddings are derived from diagnostic labels, and semantic priors are extracted using a logit-lens from a medical vision–language model (LLaVA-Med). The goal is to enable clinically meaningful, patch-level attention modeling for gaze prediction. The authors formulate scanpath prediction as a continuous regression task over spatial coordinates and dwell durations. Evaluations show that the model produces clinically meaningful gaze patterns and improves downstream pathology classification when predicted fixations are used in training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ This paper presents a well-motivated approach to scanpath prediction in medical imaging by integrating domain-specific visual encoders, textual embeddings of diagnostic labels, and semantic patch-level priors via logit-lens decoding from a medical VLM. The proposed framework is methodologically sound and leverages clinically grounded semantic priors to model gaze behavior in a task-driven setting.\n+ The paper applies logit-lens decoding to LLaVA-Med for extracting patch-level semantic priors. \n+ The paper demonstrates experimental results on the GazeSearch dataset where the model outperforms existing baselines across multiple scanpath similarity metrics. In the downstream classification task with MIMIC-CXR, it improves the pathological classification. \n+ The paper has detailed descriptions of evaluation metrics. Figures for the qualitative results are used effectively to illustrate the performance.  \n+ The significance of the work lies in its ability to bridge gaze prediction with clinical reasoning by generating fixations that align with human attention and diagnostically meaningful regions.\n\nWeaknesses:\n- Although this method can do a good job of predicting the gaze scanpath, there are some theoretical and experimental gaps in understanding how the architecture works.\n- A substantial portion of the framework’s novelty hinges on logit-lens decoding of pretrained models (e.g., LLaVA-Med), which may raise concerns about the originality of the core algorithmic contributions. While the integration is creative and domain-specific, more justification in modeling fixations beyond architecture reuse would strengthen the paper.\n- In Line 200, the authors claim to test generalization by discarding human fixations and replacing them with synthetic ones. However, it is not explained how this strategy robustly tests generalization. Notably, although the MIMIC-CXR dataset has real fixation data [1], Tables 1 and 2 only report results on GazeSearch. The absence of performance metrics using real fixations from MIMIC-CXR raises concerns about the model’s claimed generalization ability.\n- The authors introduce joint losses for coordinate regression and dwell time regression, but it is not specified whether the associated weighting coefficients (λ₁, λ₂) are fixed or learnable. An ablation study analyzing the sensitivity to these hyperparameters would help clarify the relative importance of each component and improve the transparency of the training process.\n- In line 181, the authors claim that LogitGaze-Med produces more human-like scanpaths. However, in Figure 3, especially in the “Cardiomegaly” and “Pneumonia” tasks, the predicted paths of LogitGaze-Med and GazeSearch look quite similar. Maybe including other qualitative examples could strengthen the claim.\n- It remains unclear whether, during inference, the model requires the X-ray image, the semantic embeddings (e.g., “rib, lung”, “rib, gray”), and the disease labels. If such semantic phrases and/or labels are essential for prediction, it raises practical deployment concerns (whether radiologists must manually provide these prompts, or if they are generated automatically). A clarification on whether semantic priors are predicted, retrieved, or externally provided at test time would remove this confusion.\n- In Figure 4, the attention maps produced by the Eye-Gaze baseline for the “Normal” and “CHF” conditions appear nearly identical. This casts doubt on the meaningfulness of these comparisons. Either more or improved qualitative illustrations are needed to validate the claim that LogitGaze-Med yields more discriminative and clinically aligned attention maps.\n\nMinor Concern: \n- Although the model is said to produce interpretable fixations, the paper lacks feedback from radiologists to validate whether these gaze patterns truly reflect diagnostic reasoning.\n- In Line 166, please provide a citation to support the statement that 'AUROC serves as a proxy for clinical utility and interpretability,' as this is a strong claim that would benefit from justification. \n\n[1] Hsieh, C., Ouyang, C., Nascimento, J. C., Pereira, J., Jorge, J., & Moreira, C. (2023). Mimic-eye: Integrating mimic datasets with reflacx and eye gaze for multimodal deep learning applications. PhysioNet (version 1.0. 0).\n\nQUESTIONS:\n1.\tCan you use the real fixation data for MIMIC-CXR to report the performance in Tables 1 and 2? A clarification for why they are not used in the evaluation would be helpful for reassessing the paper.\n2.\tAre the weighting parameters (coordinate loss: λ₁, dwell time loss: λ₂) fixed or learnable during training? How sensitive is model performance to these hyperparameters? Can you include an ablation study varying λ₁ and λ₂, and report the effects on scanpath similarity. This would clarify their roles in model optimization.  \n3.\tIn Figure 3, the predicted scanpaths from LogitGaze-Med and GazeSearch appear visually similar for certain diagnoses (e.g., cardiomegaly, pneumonia), and in Figure 4, attention maps for different conditions seem indistinct. Can you provide different qualitative examples to reduce the confusion?\n4.\tCan you clarify what novel modeling contributions your method makes beyond reusing pretrained components, and how LogitGaze-Med fundamentally differs from prior works such as LogitGaze? Please elaborate on how your model introduces new mechanisms for fixation prediction beyond leveraging existing VLM infrastructure.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduced LogitGaze-Med, a framework to predict the spatio-temporal sequence of scanpaths that occur during visual search in radiology. Specifically, LogitGaze-Med leverages visual features from domain-specific encoders, text embeddings, and patch-level priors extracted from a medical vision language model.\n\nThe authors claim three contributions: \n\n(1) LogitGaze-Med, the first framework to apply logit-lens decoding to an instruction-tuned medical VLM (LLaVA-Med), enabling clinically grounded patch-level attention for gaze prediction.\n\n(2)  Show that domain-specific components—LLaVA-Med and a CheXNet-style encoder—consistently outperform generic baselines in scanpath prediction metrics.\n\n(3) Formulate scanpath prediction as continuous regression over spatial coordinates and dwell durations, and demonstrate that predicted fixations improve downstream pathology classification.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nPlug-and-play framework: The authors leverage an existing framework to integrate existing domain-specific models (instead of training a model from stracth). \n\nStrong empirical performance: LogitGaze-Med achieves state-of-the-art results across multiple scanpath similarity metrics and improves downstream pathology classification (AUROC).\n\nComprehensive evaluation across multiple datasets: The authors collect multiple datasets to evalaute their models. Additionally they created a synthetic dataset using MIMIC.\n\nWeakeness: \n\n* The main novelty of this work is to leverage an existing method (LOGITGAZE) in radiology. There is no new technical contribution \n\n* Many claims are not justified with empirical experiments. For example, \"Existing methods often rely on generic vision–language models and saliency-based features, which limits their ability to capture clinical semantics and integrate domain knowledge effectively.\" Where is the comparison (e.g. performance on classification) of LogitGaze-Med against Llava-med, Llava-rad, biomedclip, BMC-clip, etc?\n\n* The selection of feature extracores is not thoroughly justified. For example, the selection of Chexpert (a model published 6 years ago) or llava-med (instead of llava-rad, a specific llava model for radiology) seems a bit arbitrary. As such, an ablation for the selection of models is required. Regarding the optimality of this method, would it be possible to achieve similar performance with newer/ smaller models? \n\n* Not sufficient basic baselines. For instance, are all modalities required? What happens if you remove text? or the llava feautres? Furthermore, is this approach required? Why not just add a decoder on top of BiomedClip? \nMIMIC-CXR\n\n* There is no evaluation  (nor clear details of data construction) to demonstrate the quality of the synthetic data generated from MIMIC-CXR.\n\nQUESTIONS:\nPlease provide an experiment behind each claim. If the experiment is not provided, please remove the claim. \n\nOverall, the value of this work could increase if ablations are provided:\n1 Test adding a decoder on top of a CLIP model (e.g. biomedclip, bmcclip, etc) and compare against your method.\n\n2. Please add an extensive empirical experiment or add different encoders. What happens if you use lava vs lava-med or lava-rad? \n\n3. Add an experiment to quantify how much improvement you get from each step of your method.\n\n4. Please provide a human evaluation of the quality of your synthetic dataset/ remove it if it's not possible to assess its quality.\n\n5. Compare the performance of your method against other VLLMs (if you wish to keep the claim: \n\"Existing methods often rely on generic vision–language models and saliency-based features, which limits their ability to capture clinical semantics and integrate domain knowledge effectively.\"\n\n6. Please include newer encoders and VLMs in your evaluation and experimentation.\n\nLIMITATIONS:\nWhile the work does address an interesting and relevant problem, it does not offer enough technical novelty, in makes claims without adequate experiments, and it's vague about some of its components. \n\nIn the absence of adequate technical novelty, an empirical study would add sufficient value for this work to be accepted in this venue. I encourage authors to explore the questions provided above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new multimodal model called LogitGaze-Med, which leverages vision-language grounding and is loosely based on concepts from visual attention and how humans process images. This approach is applied to the interpretation of radiological chest X-rays in the medical imaging domain (i.e. MIMIC-CXR and GazeSearch datasets). The model accurately predicts both fixation locations and durations, which enables generation of realistic scanpaths based on expert human eye tracking data. Quantitative experiments show that the proposed model improves predicted scanpath similarity and can also be useful when applied to downstream classification tasks which leverage the predicted fixations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: Overall, this paper is well-written and tackles the meaningful clinical problem of chest X-ray interpretation and classification. The proposed LogitGaze-Med architecture and associated loss functions are relatively intuitive, enabling modular use of different modalities as separate components of the model. The use of the model for both scanpath prediction and classification makes for a harder set of tasks, increasing the novelty of the approach. In general, the experiments in the paper are well designed with appropriate metrics, with appropriate baseline comparisons to prior work and ablation studies.\n\n**Weaknesses**: There are a few minor weaknesses to the paper:\n\n- LogitGaze and Gazeformer are trained on natural images, not chest X-rays, yet they are included in the results as baselines, which may be somewhat unfair given the domain mismatch. Are there other foundational vision-language models (VLMs) in the radiology domain would be useful to compare against?\n\n- I'm not very familiar with the other proposed baselines for scanpath prediction, but the authors state that, \"The initial query Q_0 is modulated with the first human fixation via 2D positional encoding\" (line 127). This may be providing the proposed LogitGaze-Med approach with an unfair advantage, as it may not need to fully predict the first fixation. Can the authors clarify if all approaches are conditioned with the first fixation?\n\n- In terms of ablation studies, only semantic continuity and domain specificity are evaluated. I would like to see how the different model components impact the final performance (e.g. removing the vision encoder, logit lens, or text encoder, respectively). This is needed to prove that the proposed architecture and joint embeddings are indeed beneficial.\n\n- In addition to the GazeSearch dataset, the authors generated synthetic scanpaths based on the MIMIC-CXR dataset, but they do not clearly state how they validated this dataset as many details are missing (e.g. do statistics of dwell times match real data?) This limits reproducibility and may affect the conclusions that can be drawn from the synthetic data results.\n\nQUESTIONS:\nI have a few questions:\n\n1.  The attention maps are mostly used for downstream classification, but it may be interesting to evaluate similarity with the human-derived maps (e.g. Figure 4). Are attention maps from LogitGaze-Med more similar to human maps, relative to the other baseline approaches?\n\n2. Did the authors consider an additional ablation related to temporal shuffling of the predicted scanpath? This could be done by either shuffling the durations of predicted fixations, or the order in which fixations are visited in the scanpath.\n\n3. The authors only show results on chest X-rays, but presumably the framework could be applied to other medical image interpretation tasks such as digital pathology. Do the authors have thoughts on this?\n\nLIMITATIONS:\nThe paper addresses potential limitations in the Conclusion section, although additional limitations could be enumerated there. For example, use of synthetic scanpath data based on the MIMIC-CXR dataset may introduce biases in the evaluation based on how the data is generated.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}