From Human Attention to Diagnosis: Semantic Patch-Level Integration of Vision-Language Models in Medical Imaging
Authors: Dmitry Lvov
Ilya Pershin
Venue: NeurIPS 2025 poster ()
Keywords: Gaze prediction
medical image analysis
CXR
deep learning
fixation modeling
attention mechanisms
visual attention
neural networks
Primary area: machine_learning_for_sciences
TL;DR: This paper explores gaze prediction and fixation modeling in medical images using deep learning techniques.

Abstract: Predicting human eye movements during goal-directed visual search is critical for enhancing interactive AI systems. In medical imaging, such prediction can support radiologists in interpreting complex data, such as chest X-rays. Many existing methods rely on generic vision--language models and saliency-based features, which can limit their ability to capture fine-grained clinical semantics and integrate domain knowledge effectively. We present \textbf{LogitGaze-Med}, a state-of-the-art multimodal transformer framework that unifies (1) domain-specific visual encoders (e.g., CheXNet), (2) textual embeddings of diagnostic labels, and (3) semantic priors extracted via the logit-lens from an instruction-tuned medical vision--language model (LLaVA-Med). By directly predicting continuous fixation coordinates and dwell durations, our model generates clinically meaningful scanpaths. Experiments on the GazeSearch dataset and synthetic scanpaths generated from MIMIC-CXR and validated by experts demonstrate that LogitGaze-Med improves scanpath similarity metrics by 20--30\% over competitive baselines and yields over 5\% gains in downstream pathology classification when incorporating predicted fixations as additional training data.

From Human Attention to Diagnosis: Semantic Patch-Level Integration of Vision-Language Models in Medical Imaging
Dmitry Lvov Research Center of the Artificial Intelligence Institute Innopolis University Innopolis, Russia 420500 w..vvv@inoopolis....
Ilya Pershin
Research Center of the Artificial Intelligence Institute Innopolis University Innopolis, Russia 420500
i wwwppein@imooollsss.....
Abstract
Predicting human eye movements during goal-directed visual search is critical for enhancing interactive Al systems. medical imaging, such prediction can support radiologists in interpreting complex data, such as chest X-rays. Many exist- ing methods rely on generic vision-language models and saliency- features, which can limit their ability to capture fine-grained clinical semantics and integrate state-of-the-arr domain knowledge effectively. We present LogitGaze-Med, multimodal transformer framework that unifies (1) domain-specifee visual encoders (e.g., Xhe)) (2) textual embeddings diagnostic labels, and (3) semantic pri- extracted via the logit-lens from an instruction- seneiidon-teeddddd medical vision-langue language model (LLaVA-Med), By directly predicting continuous fixation coordinates and dwell durations, our model generates clinically meaningful scanpaths. Experiments on the GazeSearch dataset and synthetic scanpaths generated from MIMIC-CXR and validated by experts demonstrate that LogifGaze-Med improves scanpath sim- ilarity metrics by 20-30% over competitive baselines and yields over 5% gains in downstream pathology classification when incorporating predicted fixations additional training data.
Introduction
Understanding and predicting human eye movements during visual search is long- long-standing problem in both cognitive science and computer vision 14 In medical imaging, modeling expert gaze- gaze-such such how radiologists examine chest offers new avenues support diagnostic decision-making, enhance training, and develop interactive AI systems.
However, existing gaze prediction methods often fail in real-world clinical settings, where visual targets are subtle, abstract, and highly variable depending on the diagnostic task (5 6). While most prior work has focused on free-viewing saliency-based gaze prediction 7 g these approaches typically neglect the observer's 's goal. In contrast, we study goal-directed gaze modeling: predicting the spatio-temporal sequence of fixations (scanpaths) that occur during active visual search for specific diagnostic targets [9].
This task poses unique challenges in medical imaging. Diagnostic categories like "pleural effusion" may not correspond to well-defined objects or regions [10 II] Visual cues can be ambiguous or distributed across anatomical structures. Moreover, collecting large-scale gaze data across all possible pathologies is infcasible due to cost, data availability, and privacy concerns
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

bone, grey
struct, ankle
rib, lines
metal, white
cardio, heart
ves, "M spi
Scanpath visualization on a chest X-ray image labeled "normal", with fixations (yellow circles) indicating attention shifts. Image patches are semantically decoded via VLM into labels such as "bone, gray, metal", "cardio", and "heart", aligning gaze prediction with clinical visual understanding.
Figure 1:
To address these challenges, we introduce LogitGaze-Medd transformer-basee 13
framework that integrates three complementary modalities: (1) visual features from domain-specific encoder (e.g.. CheXNet [14); (2) text embeddings that represent diagnostic queries labels; and (3) semantic patch-level priors extracted from a medical Vision-language model (LLaVA-Med)) using the logit lens technique [85 As shown in Figure I these decoded keywords help align predicted scanpaths with clinically meaningful image regions.
We evaluate LogitGaze-Mor Med on both the GazeSearch dataset [1 and large-scale synthetic gaze dataset generated over MIMIC-CXR (20 11, with synthetic scanpaths validated through expert human evaluation. Our model shows strong performance across standard scanpath similarity met- rics and maintains robustness when tested with alternative medical VLMs (LLaVA-Ra) (21)) and modern encoders (CheSS, PEAC 22 (3) In addition, we find that our model's predicted fixations significantly improve downstream pathology classification, suggesting practical utility beyond gaze modeling. Our main contributions
We introduce LogitGaze-Med, the first framework to apply logit-lens decoding to medical VLMs for clinically grounded patch-level gaze prediction, with comprehensive ablation studies demonstrating the importance domain-specific components.
Our formulation of scanpath prediction continuous regression over spatial coordinates and dwell durations improves downstream pathology classification by over 5%.
Expert human evaluation validates synthetic scanpaths with high realism (4.3/5.0) and clinical relevance scores (4.2/5.0).
2
Related Work
We review prior work in three relevant areas: scanpath prediction, VLMs(vision-language models), and interpretabilityy
Scanpath Prediction Early models of scanpath generation relied on saliency maps or hand-crafted
heuristics (24), lacking semantic understanding and goal-awareness. Recent transformer-basee
methods have advanced goal-directed gaze modeling. GazeFormer [9] introduced a zero-shot
"ZeroGaze" task by encoding search goals via natural language, achieving strong spatial-temporal accuracy while being over five times faster than prior approaches. HAT(The Human Attention Transformer) (25] unified top-down and bottom-up attention single framework, leveraging simplified foveated retina to model human- like spatio-temporal attention. GazeXplain (26) extended this by generating natural language rationales alongside fixation sequences, bridging "where" and "why" people look. LookHear (ART) 27 tackled multimodal gaze by modeling real-time fixations during spoken object reference. LogitGaze 28 integrated semantic priors from VLMS using logit- lens decoding, improving prediction accuracy 15% and enhancing interpretability through explicit

---

concept-level alignment. Foundational insights such as inhibition of return (IOR) [29 continue to inform fixation dynamics by discouraging re-attending to previously viewed regions.
In the medical domain, GazeSearch [19] introduced the first task-aligned chest X-ray visual search dataset and dedicated baseline ("ChestSearch"), revealing that general-purpose gaze models under- perform in clinical settings. Despite these ad advances, few approaches jointly model clinical semantics, task conditioning, and probabilistic scanpaths in a unified, end-to-end architecture.
Vision-Languaal Models iirructiiontuned VLMs have shown strong generalization across do-
mains. LLaVA [5] bootstrapped multimodal instruction tuning using GPT-4 [30], demonstrating
robust performance open-ended vision tasks. LLaVA-Med [16] adapted this approach to biomedi- cal images by generating self-instruction data from PubMed figure captions [31] outperforming prior
Med-VQA models with minimal domain- specific fine-tuning. LLaVA-Rad 21 further specialized
this approach for radiology, training on 697k image-report pairs. Voila-A [32] leveraged AR/VR gaze data to align VLM attention with human fixations, using GPT-4 to annotate the VOILA-COCO dataset and integrating gaze into perceiver modules for interpretabillity R-LLaVA [3] injected region-level priors into CLIP inputs to enhance Med- VQA accuracy, emphasizing the utility of explicit visual context. These studies show the promise of task-aligned or gaze-aware VLMs, but most do not directly model spatio-temporal attention fixations.
Interpretability of VLMs Interpreting VLMs is essential for trustworthy AI, especially in medicine. Neo et al. [18) showed that LLaVA gradually refines object-level semanties across layers. The logit lens 17 projects intermediate activations into the output space, revealing how representations evolve from generic concepts in early layers to clinical terms like "consolidation." " Originally for language models, this method was adapted to vision-language settings in scan path prediction task [28]. offering interpretable semantic priors.
Methodology
Our scanpath prediction pipeline consists of four main stages: (1) logit-lens semantic extraction; (2) preprocessing of visual and textual features; (3) joint transformer encoding with multiterm loss formulation; and (4) fixation decoding and scanpath regression. All components are aligned with the schematic shown in Figure
3.1 Logit-Lens Semantic Extraction
Given an input chest X-ray R we extract patch-level semantics using vision-language
transformer. The image is divided into P x P grid (with /' = 24, yielding M = 576 patches), and a pretrained multimodal model-either LLaVA (15), LLaVA-Med (34), LLa [2] fine-tuned on radiological data- the final hidden state for each patch:
h; Rdd i= 1= M.
After layer normalization, logits over the vocabulary are computed:
LayerNorm((h)),
Wmm h4 bm. pt = softmax((e) e RV
From the top-k logits (j.,,...,, Jr, extract corresponding word vectors using the frozen language embedding matrix ET 18 Vxd,
S, (ja,),....... Em (r,k)) Eam
Rxd
These are pooled (e.g., via mean) into a single semantic vector .dd. This logit-lens mecha-
nism 18 exposes latent clinical concepts such "consolidation" or "effusion" "_a the patch level.
3.2 Visual and Textual Feature Preprocessing
Visual Feature Encoders. Each image is resized 768 768, normalized, and passed through one of several CNN backbones:
3

---

Task object:
edemna consolidation, pncummnis normal
ardddmeg
Encodar
Visian Model
Lxp
FCC
R
R
Stack
Transformer
Trnnformer
VXT
TxT
TxT
R
RR
R
Joint Embeddaings
TxT
R
Isl
+
R
Txll
31xT
Flvations
R
Encoding (pleers)
Encoding ipatch)
Decoder
Scan path pr prediction
Figure 2: LogitGaze-Mel architecture integrating vision-language processing with multiscale posi- tional encoding. The framework fuses visual features (patch/pixel levels) with semantic embeddings (e.g., bone, heart, opacity) through decoder. Joint optimization across classification tasks such as edema, pneumonia, and normal, along with spatial (C,, Cy) and temporal (C2) losses, enables human-like scanpath generation. Closed-loop decoding projects hidden states into interpretable clinical vocabulary space.
ResNetCOCO. ResNet50-FPN 355 from Mask R-CNN 36 pretrained on MS COCO 337 processes batches E R 788876
backbone(x) IR B
C 2048, H' W' 24 24.
Flattening and permuting yields:
Xvis E R BxMxiiss dvis = 2048.
CheXNet 14] DenseNet-121 38), pretrained on the ChestX-ray 14 dataset [39] outputs
Bx122 xW" A x convolution expands channels to 2048, yielding:
X B x2038
Alternative encoders. We also evaluate modern medical image encoders including CheSS [22] and PEAC 23 to assess framework robustness across different visual back- bones.
Textual Task Embeddings.
Each diagnostic label (c.g., "normal," "pncumonia") is encoded with SentencceTranffommer (e.g., stsb-roberta-aase- 40]
Stask d
W. Ctask 6, E Rd

---

3.3 Joint Transformer Encoding and Loss Formulation
Embedding Fusion. Visual features Uvis E R Rx and semantic embeddings Usem E IR RxMxd are linearly projected into a shared dimension d. These are concatenated:
Fjoint
Uvis
ssa
E BBBx2Mx
Learnable 2D positional encodings and broadcasted task embedding Tiask E BRdx are added:
= "raansormerrEncoddr((BBiio + PEpatch + Task @ ... + Tissk) IR Bx2Mxx
Loss Functions.
We jointly optimize three objectives with fixed weighting coefficients \ and A
C
Cdss
+
A1 Capatial
X2
Ltime dwell time
coordinate regression
classlication
The weighting parameters are fixed throughout training, with baseline values X 1.0 and X2 = 1.0 selected via preliminary validation.
Token Classificatton:: At each timestep E [1, LJ. the decoder predicts a binary stop token my E [0, 1]. A masked cross-entropy loss is used:
Lels
T. log ( (cc)
a (0.
Nvalid
Spatial Loss: We compute the LI distance between predicted and ground truth coordinates:
Sapatial
Σ Ie (lã: t=1
xzl l0ee vil)-
Nvalid
Temporal Loss: The predicted dwell time is supervised using mean squared error:
Ltime
Le (Ge T1)²
Nvalid
3.4 Fixation Decoding and Scanpath Regression
Decoder Architecture. We use transformer decoder with Nacr layers and L learnable query embeddings (Qo..... ... (Q.....QQ..1 C Rd. The initial query Qo is modulated with the first human fixation via 2D positional encoding, ensuring all compared methods are equally conditioned on initial gaze position. Self- and cross-attention operations over encoder outputs Z produce:
Fdec E RR Bx
Fixation Parameter Regression. For each step six MLP heads predict the spatial means Ma, Mye dwell time mean Mr., and varianciss Azu Ages Arc- Predictions are sampled via the reparameteri- zation trick:
2, = Mz, + Ex, exp \w.), Ez, (O,,
and similarly for in and 7t. This stochastic sampling enables the model represent variability in human gaze behavior.
Results
4.1 Database
We evaluate LogitGaze-Med on several datasets that capture radiologists' eye movements during chest X-ray interpretation. Preference was given datasets aligned with diagnostic search tasks.
Our primary dataset GazeSearch [19,, designed for scanpath prediction in medical imaging. Unlike carlier datasets that reflect free viewing, GazeSearch employs target-present search paradigm with
5

---

known findings (e.g., cardiomegaly). It filters fixations to emphasize task-relevant sequences, making scanpaths more representative of purposeful diagnostic behavior.
To assess generalization, we also use the Eye Gaze dataset [1, which contains expert fixations but limited in scale. To overcome this, we generate synthetic scanpaths for large MIMIC- subset [20), enabling downstream tasks like gaze-informed pathology classification and large-scale evaluation.
For cross-domain benchmarking, we include COCO-Search18 41 a visual search dataset from general vision domain. LogitGaze-Med performs worse here, validating its specialization for medical imaging and the need for task-specific inductive biases.
4.2 Evaluation Metries
To evaluate scanpath prediction, we follow the GazeSearch protocol that capture spatial, temporal, and dynamic gaze characteristics.
19 and report multiple metrics
ScanMatch 42
aligns scanpaths using a variant of the Needleman- Wunsch algorithm 43 Each
fixation is encoded by spatial bin and temporal order, with optional inclusion of duration(w: Dur.). For "w/o duration" Dur.) variants, the alignment omits fixation length information.
SED 44 45 (Scanpath Edit Distance) is Levenshtein-base metric that counts insertions, deletions,
and substitutions between discretized fixations. While duration and geometry are ignored, it remains an intuitive measure of sequence similarity.
STDE [46] embeds scanpaths into temporal-spatial space and is sensitive to rhythm and timing variations. It captures biologically plausible gaze behavior, common in diagnostic reading.
MultiMatch 47 compares scanpaths along five continuous dimensions: shape, direction, length, position, and duration.
For downstream classification tasks, also report AUROC (Area Under the Receiver Operating Characteristic Curve) to assess the effectiveness of gaze-informed models in distinguishing between clinical conditions. AUROC serves proxy for clinical utility and interpretabillity
4.3 Scanpath Prediction
We evaluate scanpath prediction on the GazeSearch dataset 19, focusing on the target present condi- tion. Our model, LogitGaze-MMd, is compared against GazeFormer 9 HAT 25) ChestSearch 19
and LogitGaze (28].
LogitGaze-Med leverages LLaVA-Med [34] and optionally integrates CheXNet [4] for pathology- aware encoding. All models are trained from scratch on the GazeSearch dataset and identically conditioned the first fixation to ensure fair comparison. All models are trained for 100 epochs (batch size 32) on 3090Ti GPU. We use 6-layer encoder-decoder transformer (hidden size 512), a staged learning rate schedule (1e-6/ 2 1c 1c-4), and dropout (0.4 the classifier).
LogitGaze-MM outperforms all baselines on both sequence-level and component-wel wise metrics. With CheXNet features, we observe relative ScanMatch gain of +26% (w/o duration) and +48% (w/ duration), and SED is reduced by 4%. As shown in Tables and 2 improvements in vector and duration similarity confirm enhanced alignment with human scan dynamics.
Table Performance scanpath similarity metrics (higher is better for ScanMatch/STDE, lower is better for SED).
Method
ScanMatch
SED
STDE
w/o Dur.
w/ Dur.
GazeFormer
0.293 0.021
0.201 = 0.015
5.11 + 0.08
0.799 + 0.004
HAT 25
0.309 + 0.020
5.07 + 0.07
0.800 - 0.004
GazeSearch 19
0.332 + 0.019
0.223 + 0.014
4.88 + 0.06
0.809 + 0.004
LogitGaze 28
0.328 + 0.018
0.225 = 0.015
5.07 - 0.07
0.810 - 0.004
LogitGaze-Med (Res)
0.416 + 0.017
0.325 + 0.012
4.68 + 0.05
0.852 + 0.003
LogitGaze-Meed (CheX)
0.419 0.016
0.330 + 0.010
4.68 + 0.05
0.855 + 0.003

---

Table MultiMatch similarity (higher better) across five components.
Method
Vector
Direction
Length
Position
Duration
GazeFormer 9
0.902 + 0.008
0.644 = 0.010
0.899 + 0.009
0,803 = 0.007
0.595 + 0.015
HAT 25
0.909 0.007
0.649 0.010
0.910 + 0.008
0.825
0.006
GazeSearch
19
0.917 0.006
0.679
0.010
0.917 + 0.007
0.829
0.006
0.695 + 0.014
LogitGaaze
28
0.882 0.009
0.643
0.008
0.923 + 0.005
0.809
0.006
0.625 + 0.013
LogilGaze-Mir (Res)
0.935 0.004
0.650
0.939
0.006
0.005
0.743 + 0.010
GazetGad (CheX)
0.938 0.004
0.651 = 0.009
0.948 + 0.005
0.823
0.005
0.740 + 0.010
(a) Normal
(b) Cardiomegaly
(c) Consolidation
(d) Pneumonia
Figure 3: Comparison of human scanpaths (top), LogitGaze-Med predictions (middle), and Gaze- Search predictions (bottom) across tasks: normal, cardiomegaly, consolidation, and pneumonia. The models evaluated under target-present conditions.
Figure 3 presents qualitative comparisons of scanpaths for several diagnostic tasks. Each triplet shows the human scanpath (top), predictions from LogitGaze-Mo (middle), and predictions from Gaze- Search (bottom). Visually, LogitGaze-Medt produces more human-like trajectories, with smoother saccades and task-relevant fixations that better reflect real radiological reading behavior. In contrast to baselines, it captures the diagnostic flow more accurately. These examples highlight the model's ability to generate plausible and clinically coherent visual search patterns.
4.4 X-Ray classification analysis
To assess the utility of gaze information in medical imaging, reproduced and extended the classification setups introduced by [IT] on the MIMIC-CXR dataset [20]. In addition to their original settings, we introduced new variant using model-generated fixations, allowing us to evaluate generalization beyond human gaze annotations. We addressed a 3-class classification task (normal, congestive heart failure (CHF), and pneumonia) and explored how different mize-informed architectures affect performance.
The baseline model uses convolutional encoder and linear classification head, taking only X-ray images as input. In the second setup, we augment image with temporal fixation heatmaps: the image is encoded into visual vector VCXR, while each of m gaze heatmaps passes through separate
7

---

CNN. These are aggregated via BiLSTM with self ceff-atteniion into gaze vector gaze, and the final prediction based on \CXR: Ugaree The third setup introduces multi-task U-Net [48] that jointly predicts pathology and reconstructs static gaze heatmap. An EfficientNer-Bi BO 44 encoder is used, with classification head the bottleneck and decoder predicting the gaze map. The total loss is weighted sum of binary cross-entropy for both tasks.
To generalization, we replaced human fixations with synthetic ones generated by various models (GazeFormer [9]. LogitGaze [28]. LogitGaze producing 8 scanpaths per image. Heatmaps derived from these were substituted into each pipeline without changing architecture or training protocol.
To validate the quality of our synthetic MIMIC-CXR scanpaths, conducted structured human evaluation with a board-certified radiologist (5+ years experience). The expert reviewed a hold-out of 100 chest X-rays (50 with real and 50 with synthetic seanpaths, randomized and blinded) and provided ratings on a 5-point Likert scale for visual realism and clinical relevance.
Table 3: AUROC scores across three classification setups on MIMIC-CX--JPG using human and synthetic gaze. For LogitGaze- per-class AUROC scores are follows: Baseline (Normal 0.87, CHF 0.85, Pneumonia 0.74). Temporal (Normal 0.93, CHF 0.92, Pneumonia 0.85), and U-Net (Normal 0.94, CHF 0.92, Pneumonia 0.87).
Method
Baseline
Temporal
U-Net
Eye-Gaze 11
0.77 = 0.02
0.82 + 0.03
0.87 + 0.02
GazeFormer 9
0.78 # 0.02
0.84 = 0.02
0.89 + 0.01
LogitGaze 28
0.80 + 0.01
0.87 = 0.02
0.90 + 0.01
LogitGaze-Med
0.82 + 0.01
0.90 0.02
0.91 0.01
(a) Original CXRs
(b) Human
(c) LazitGaze-Med
(d) Eye-Gaze baseline
Figure 4: Attention maps for three conditions: normal (top), CHF (middle), and pneumonia (bot- tom). (a) Original X-rays; (b) Human gaze reference; (c) LogitGaze-Med;; (d) Eye-Gaze baseline. Heatmaps generated by aggregating raw fixation coordinates into static heatmaps Gaussian kernel smoothing [50]. Yellow denotes high attention.
The synthetic scanpaths received mean scores of 4.3+0.5 visual realism and 4.2+0.6 for clinical relevance. In binary classification task (real vs. synthetic), the expert achieved only 58% accuracy,

---

indicating that synthetic scanpaths were often indistinguisablle from real ones. These results confirm the plausibility and clinical validity of our generated scanpaths.
Table 3 reports AUROC scores across all three setups. Temporal and multi-task models consis- tently outperform the baseline, showing that gaze information expeeiiall when structured tem- porally-enhances diagnostic accuracy. Among gaze sources, LogitGaze-Med achieves the best performance all cases, suggesting that its vision-language grounding improves semantic alignment with pathology.
Figure 4 contrasts attention maps from LogitGaze- Gede and Eye-Gaze LogitGaze-Mer limited to six predicted fixations as in GazeSearch [19], produces focused and pathology-allee aligned maps. In contrast, human and Eye- Eye-Gaze maps often broader and less selective, reflecting more chaotic viewing behavior and weaker diagnostic relevance.
4.5 Ablation Analysis
We conduct comprehensive ablation experiments to evaluate the roles of different components, semantic continuity, and domain specificity in gaze prediction.
Hyperparameter Sensitivity Analysis. We analyzed the sensitivity our model to the loss weighting parameters Å (coordinate loss) and A2 (dwell time loss). As shown in Table 4 the model maintains robust performance across a broad range of values (Aι E [0.5 - 5.0]. X2 E [O.T 5.0)), with optimal performance at X 2.0, A2 0.5. This configuration improves shape and position consistency while maintaining strong duration modeling.
Table 4: Effect of varying A (coordinate loss) and A2 (dwell time loss) on scanpath similarity (MultiMatch components). Best values in bold.
Loss Weights
Shape
Direction
Length
Position
Duration
A1 = 1.0, A2 1.0
0.921
0.633
0.932
0.812
0.712
A1 = 2.0, d2 0.5
0.938
0.651
0.948
0.823
0.740
A1 0.1, dσ 1.0
0.866
0.570
0.882
0.743
0.707
A 1.0, Aσ 0.1
0.917
0.624
0.926
0.805
0,610
A1 = 1.0, A2 5.0
0.910
0.615
0,920
0.792
0.689
A = 5.0, d 1.0
0.887
0.593
0.902
0.760
0.701
Component-wis. Ablation. Table shows the importance of each modality. The image-only baseline performs poorly, confirming that visual features alone are insufficient. Adding clinical text substantially improves performance, highlighting the value of semantic intent. The full model with logit-lens achieves the best results, demonstrating the benefit of combining clinical context with localized attention.
Table 5: Stepwise ablation analysis of LogitGaze-Med GazeSearch.
Method
ScanMatch w/o Dur.
ScanMatch w/ Dur. +
SED L
STDE
Image only
0.148 0.039
0.126 0.034
8.54 + 0.28
0.562 + 0.094
+ Text only
0,280 + 0.020
0.206 0.013
5.95 + 0.06
0.810 + 0.004
LogitGaze-Med (full)
0.419 + 0.016
0.330 + 0.010
4.68 0.05
0.855 + 0.003
Encoder and VLM Comparison. Table evvlluates different combinations of VLMs and visual encoders. LLaVA-Me and LLaVA-Ras yield comparable results, confirming robustness to VLM choice. While modern encoders like CheSS and PEAC offer slight improvements, CheXNet remains competitive, supporting its use as a strong baseline.
Semantic Continuity. To assess the importance of structured semantic alignment, we compare LogitGaze-Med with two baselines: (1) random predictor [27] and (2) a variant with shuffled reference alignments (LogilGaze: - Medahws), which disrupts the scanpath's semantic structure. Table 7 shows that LogilGaze Medstaf suffers a >40% drop in ScanMatch and a similar decline in Multiatch, confirming that preserving alignment essential for meaningful predictions.

---

Table Performance comparison across different VLM and encoder combinations on GazeSearch
dataset.
Method
ScanMatch w/o Dur.
ScanMatch w/ Dur. f
SED
STDE
LogitGaze-Mer (ResNet)
0.416 + 0.017
0.325 = 0.012
4.68 0.05
0.852 0.003
LogitGaze-Mar (CheXNet)
0.419 + 0.016
0.330 = 0.010
4.68 0.05
0.855 + 0.003
LogitGaze-Mer (CheSS)
0,425 + 0.017
0,321 = 0,012
4.66 460
0,857 = 0.003
LogitGaze-Med (PEAC)
0.428 + 0.018
0.319 = 0.012
4.64 464+0000
0.858 + 0.004
LLaVA-Rad CheXNet
0.417 + 0.016
0.329 0.010
4.67 + 0.05
0.854 + 0.004
LLaVA-Rad + CheSS
0.426 + 0.018
0.320 = 0.012
4.65 0.05
0.857 + 0.004
LLaVA-Raal + PEAC
0.429 + 0.019
0.318 = 0.012
4.63 0.05
0.859 + 0.004
Table 7: Effect of semantic continuity gaze prediction.
Method
ScanMatch w/o Dur
ScanMatch w/ Dur
MultiMatch Avg t
LogitGaze-Mor Med (CheX)
0.419 = 0.016
0.330 0.010
0.820 0.009
LogitGaze 28
0.328 = 0.017
0.225 = 0.020
0.776 0.011
Random
27
0.159 + 0.014
0.148 = 0.016 0.158 0.015
0.269 0.012
LogitGaze-Ao Maeeaaeeeeedaaa Med
0.178 + 0.016
0.295 0.013
Domain Specificity. We further evaluate the models the COCO18 dataset 41 analyze cross-
domain generalization. While LogitGaze and GazeFormer were trained on COCO18, LogitGaze-Med trained only on GazeSearch [19]. As shown in Table g LogitGaze-Mas performs worse COCO, reflecting its specialization for medical imagery. This gap highlights the importance of domain-aligned training for accurate scanpath prediction.
Table 8: Cross-domain evaluation on COC018 dataset.
Method
ScanMatch w/o Dur
ScanMatch w/ Dur
MultiMatch Avg t
LogitGaze
28
0.527 = 0.012
0.454 0.013
0.862 = 0.008
GazeFormer 9
0.492 0.014
0.441 0.015
0.816 = 0.010
LogitGaze-Med
0.353 + 0.018
0.289 + 0.020
0.749 # 0.014
5 Conclusion
We introduced LogitGaze-Med, a multimodal transformer that integrates domain-specito visual encoders, textual features, and semantic priors via logit-lens decoding from medical VLM. Evaluated on real (GazeSearch) and validated synthetic (MIMIC-CXR) scanpaths, model outperforms prior approaches, achieving 20-30% gains in scanpath similarity and over 5% improvement in downstream pathology classification.
Comprehensive evaluation demonstrates robustness across different medical VLMs and visual en- coders. Expert human evaluation confirms the clinical plausibility of synthetic scanpaths with high realism (4.3/5.0) and clinical relevance scores (4.2/5.0).
Inference-time analysis shows a modest overhead (85 ms vs. 70 ms per sample) due to logit-lens extraction, maintaining suitability for clinical workflows with superior interpretability.
Key limitations include sensitivity to noisy activations from non-content tokens, as recent work [5]] shows transformers encode important context in punctuation and function words, leading to potential fixation errors from unstable noun activations. Training on synthetic scanpaths also risks overfitting. Deployment in diverse clinical settings requires further validation across institutions and expertise levels.
We aim deploy our framework in radiology for lesion tracking and training, with extensions to other medical imaging domains. Code will be released to support vision-language grounding and human-AI collaboration.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the Ministry Economic Development of the Russian Federation (agreement No. 139-10-2025-034 dd. 19.06.2025, IGK 000000C11325PP40000000
References
[1] Akash Awasthi, Anh Mai Vu, Ngan Le, Zhigang Deng, Supratik Maulik, Rishi Agrawal, Carol Wu, and Hien Van Nguyen. Modeling radiologists' cognitive processes using a digital gaze twin to enhance radiology training. Scientific Reports, 15(1):13685, 2025.
[2] Carolina Camirez-Tamayo Syed Hasib Akhter Faruqui, Stanford Martinez, Angel Brisco, Nicholas Czarnek, Adel Alaeddini, Jeffrey R Mock, Edward J Golob, and Kal L Clark. Incorpo- ration of eye tracking and gaze feedback characterize and improve radiologist search patterss of chest x-rays: A randomized controlled clinical trial. Journal of the American College of Radiology, 96) 946, 2024.
[3] Tim Halverson and Anthony Hornof. A computational model of "active vision" for visual search in human- coman-computer interaction. Human- Computer Interaction, 26(4):285-314 2011.
[4] Gregory J Zelinsky, Hossein Adeli, Yifan Peng, and Dimitris Samaras. Modelling eye move- ments in categorical search task. Philosophical Transactions of the Royal Society B: Biological Sciences, (266)22221000555 2013.
[5] John M Henderson. Gaze control prediction. Trends in cognitive sciences, 21(1):15-23, 2017.
[6] Shreya Ghosh, Abhinav Dhall, Munawar Hayat, Jarrod Knibbe, and Qiang Automatic gaze analysis: A survey of deep learning based approaches. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(1):61-88) 2023.
[7] Fei Yan, Cheng Chen, Peng Xiao, Siyu Qi, Zhiliang Wang, and Ruoxiu Xiao. Review of visual saliency prediction: Development process from neurobiological basis to deep models. Applied Sciences, 12(1):309, 2021.
[8] Mohamed A Kerkouri, Marouane Tliba, Aladine Chetouani, and Rachid Harba. Salypath: A deep-based architecture for visual attention prediction. In 2021 IEEE International Conference on Image Processing (ICIP), pages 1464-1468. IEEE, 2021.
[9] Sounak Mondal, Zhibo Yang, Seoyoung Ahn, Dimitris Samaras, Gregory Zelinsky, and Minh Hoai. Gazeformer: Scalable, effective and fast prediction of goal-directed human attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1441-1450, 2023.
[10] José Neves, Chihcheng Hsich, Isabel Blanco Nobre, Sandra Costa Sousa, Chum Ouyang, Anderson Maciel, Andrew Duchowski, Joaquim Jorge, and Catarina Morcira. Shedding light on ai in radiology: A systematic review and taxonomy of eye gaze-driven interpretability in deep learning. European Journal of Radiology, 172:111341, 2024.
[11] Karargyris, S Kashyap, Lourentzou, JT Wu, Sharma, MH Tong, Abedin, DJ Beymer, VV Mukherjee, EA Krupinski, et al. Creation and validation of a chest x-ray dataset with eye-tracking and report dictation for ai tool development, 2020.
[12] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.077 2021.
[13] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.

---

[14] Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, al. Chexnet: Radiologist-evel pneumonia detection chest x-rays with deep learning. arXiv preprint arXiv: 2017.
[15] Haotian Liu,, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances neural information processing systems, (1134899-344166 2023.
[16] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and-visee assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541-2888 285644, 2023.
[17] Nostalgebraist. Interpreting gpt: the logit lens, 2020.
[18) Clement Neo, Luke Ong. Philip Torr, Mor Geva, David Krueger, and Fazl Barez. To- wards interpreting visual information processing in vision-language models. arXiv preprint arXiv:2410.07149, 2024.
[19] Trong Thang Pham, Tien-Phat Nguyen, Yuki lkebe, Akash Awasthi, Zhigang Deng, Carol C Wu, Hien Nguyen, and Ngan Le. Gazesearch: radiology findings search benchmark. In 2025 IEFF/CVF Winter Conference on Applications of Computer Vision (WACV), pages 96-106. IEEE, 2025.
[20] Alistair Johnson, Tom Pollard, Roger Mark, Seth Berkowitz, and Steven Horng. Mimic-exr database. PhysioNet10, 13026:C2/T1Q. 2024.
[21] Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademi, Ziyi Yang, al. A clinically accessible small multimodal radiology model and evaluation metric for chest x-ray findings. Nature Communications, 16(1):3108, 2025.
[22] Kyungjin Cho, Ki Duk Kim, Yujin Nam, Jiheon Jeong, Jeeyoung Kim, Changyong Choi, Soyoung Lee, Jun Soo Lee, Seoyeon Woo, Gil-Sun Hong, et al. Chess: Chest x-ray pre-trained model via self-supervised contrastive learning. Journal of Digital Imaging, 36(3):902-910, 2023.
[23] Ziyu Zhou, Haozhe Luo, Jiaxuan Pang, Xiaowei Ding, Michael Gotway, and Jianming Liang. Learning anatomically consistent embedding for chest radiography. In BMVC: proceedings of the British Machine Vision Conference. British Machine Vision Conference, volume 2023, page 617, 2023.
[24] Minh Tri Nguyen, Prarinya Siritanawan, and Kazunori Kotani. Saliency map extraction in human crowd rgb data. In 2019 58th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE), pages 941-9461 19-6. IEEE, 2019.
[25] Zhibo Yang, Sounak Mondal, Seoyoung Ahn, Ruoyu Xue, Gregory Zelinsky, Minh Hoai, and Dimitris Samaras. Unifying top-down and bottom-up scanpath prediction using transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1683-1693, 2024.
[26] Xianyu Chen, Ming Jiang, and Qi Zhao. Gazexplain: Learning to predict natural language explanations visual scanpaths. European Conference on Computer Vision, pages 314-333. Springer, 2024.
[27] Sounak Mondal, Seoyoung Ahn, Zhibo Yang, Niranjan Balasubramanian, Dimitris Samaras, Gregory Zelinsky, and Minh Hoai. Look hear: Gaze prediction for speech-directed human attention. In European Conference on Computer Vision pages 236-255. Springer, 2024.
[28] Dmitry Lvov and Ilya Pershin. Logitgaze: Predicting human attention using semantic infor- mation from vasion- models. In Workshop on Reasoning and Planning for Large Language Models, 2025.
12

---

[29] Kepei Zhang, Meiqi Lu, Zheng Lu, and Xuetao Zhang. Scanpath prediction via semantic representation of the scene. In 2022 IEEE International Conference on Image Processing (ICIP), pages 1976-1980. IEFE, 2022.
[30] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[31] Jacob White. Pubmed 2.0. Medical reference services quarterly, 39(4)338--877, 2020.
[32] Kun Yan, Zeyu Wang, Lei Ji, Yuntao Wang, Nan Duan, and Shuai Ma. Voila-a: Aligning vision-language models with user's gaze attention, Advances in Neural Information Processing Systems, 37:1890-199 1918, 2024.
[33] Xupeng Chen, Zhixin Lai, Kangrui Ruan, Shichu Chen, Jiaxiang Liu, and Zuozhu Liu. R- Ilava: Improving med-vqa understanding through visual region of interest. arXiv preprint arXiv:2410.20327, 2024. 2024.
[34] Yucheng Shi, Quanzheng Li, Jin Sun, Xiang Li, and Ninghao Liu. Enhancing cognition and explainability of multimodal foundation models with self-synthesized data. arXiv preprint arXiv:2502. 14044, 2025.
[35] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[36] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[37] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[38] Forrest Iandola, Matt Moskewicz, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint MrXiv:1404. 1869, 2014.
[39] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly- supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097-2106, 2017.
[40] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907.11699999 2019.
[41] Yupei Chen, Zhibo Yang, Scoyoung Ahn, Dimitris Samaras, Minh Hoai, and Gregory Zelinsky. Coco-search18 18 fixation dataset for predicting goal-directed attention control. Scientific reports, 11(1):8776, 2021.
[42] Filipe Cristino, Sebastiaan Mathôt, Jan Theeuwes, and lain Gilchrist. Scanmatch: A novel method for comparing fixation sequences. Behavior research methods, 42:692-700, 2010.
[43] Saul Needleman and Christian Wunsch. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of molecular biology, 48(3):443 453,, 1970.
[44] Stephan A Brandt and Lawrence Stark. Spontaneous eye movements during visual imagery reflect the content of the visual scene. Journal of cognitive neuroscience, 911:277-3, 1997.
[45] Li Yujian and Liu Bo. normalized levenshtein distance metric. IEFE transactions on pattern analysis and machine intelligence, 1996)100 2007. 2007.
13

---

[46] Wei Wang, Cheng Chen, Yizhou Wang. Tingting Jiang, Fang Fang, and Yuan Yao. Simulating human saccadic scanpaths on natural images. In CVPR 2011, pages 441 IEEE, 2011.
[47] Richard Dewhurst, Marcus Nyström, Halszka Jarodzka, Tom Foulsham, Roger Johansson, and Kenneth Holmqvist. It depends on how you look at it: Scanpath comparison in multiple dimensions with multimatch, a vector- based approach. Behavior research methods, 44:1079
1100, 2012.
[48] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-asiisted intervention-MICCAl 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[49] Brett Koonce. Efficientnet. In Convolutional neural networks with swift for Tensorflow: image recognition and dataset categorization, pages 103-123. Springer, 2021.
[50] Jeffrey Heer. Fast & accurate gaussian kernel density estimation. In 202/ IEEE Visualization Conference (VIS), pages 11-15. IEEE, 2021.
[51] Anton Razzhigaev. Matvey Mikhalchuk. Temurbek Rahmatullaev, Elizaveta Goncharova, Polina Druzhinina, Ivan Oseledets, and Andrey Kuznetsov. min-miccoscoop: Uncovering the hidden role of punctuation in context memory of transformers. arXiv preprint arXiv:2502. 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction accurately reflect the main contributions of the paper: (I) the introduction of LogitGaze-Med, a domain-specific gaze prediction model lever- aging semantic priors from medical vision-language model; (2) consistent improvements in scanpath similarity metrics and downstream classification tasks; and (3) interpretabil- ity and domain specialization key strengths. These claims are directly supported by experimental results and ablation analyses Section 4.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The limitations of the proposed method are discussed in the Conclusion section. In particular, we note that the logit-lens approach sensitive noisy non- content activations, and that reliance on unstable content- word embeddings (e.g.. from volatile tokens like nouns) can lead to fixation prediction errors in cluttered medical images. We also outline future directions to address these issues by incorporating syntactic signals such as punctuation and structure-related tokens.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
15

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
The paper does not contain formal theoretical results, theorems, or proofs. It focuses primarily on empirical evaluation and the application of a multimodal transformer- based framework for gaze prediction in medical diagnostics.
Justification:
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We describe the architecture of LogitGaze- -ed in detail (Section 3), including input representations, fusion strategies, and optimization setup. All datasets, evaluation protocols, and baseline implementations are fully specified (Section 4). We also provide instructions for processing both real and synthetic scanpaths.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
16

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We commit to releasing our training and evaluation code, including detailed instructions and environment setup. All real scanpath data (GazeSearch) is already publicly available. For MIMIC-CXR, we describe how to obtain access and generate synthetic scanpaths using pretrained model. All relevant scripts for preprocessing, training, and evaluation will be released alongside the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
btcips.cc/puble ctcc//ublic//ee wwwp://gwgeeeeeeeeeeeeeeeeeeeeee httpg//giddeeeeddubbmbmiiee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4 (Results) provides a detailed description of datasets (GazeSearch and MIMIC-CXR), evaluation metrics, and experimental design. Hyperparameters, optimizer settings, learning rates, and data are described in Results section.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
17

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report standard deviations for all evaluation metrics across experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative етrоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments conducted single NVIDIA RTX 3090 Ti GPU with 24 GB of VRAM. The computational requirements are moderate and reproducible using standard hardware.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:://wwnnrppssccccppllttt/tta
Answer: [Yes]
Justification: We have carefully reviewed the NeurIPS Code of Ethics. Our research complies with all guidelines, including transparency, reproducibility, and the responsible use of data and models. All datasets used are publicly available and anonymized, and we disclose limitations and potential biases as appropriate.
18

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work primarily focuses on improving the prediction of gaze patterns for medical image analysis, which positively impact fields like radiology and assistive technologies. By improving gaze prediction, we may help doctors to better understand attention allocation when diagnosing medical images, potentially leading to more accurate diagnoses.
However, there are potential risks associated with this technology, particularly in sensitive domains such as medical diagnostics. Misuse of gaze prediction models could lead to biased decision-makin prediction could exacerbate existing biases in medical diagnostics if not rigorously tested across diverse patient populations.
if applied without proper safeguards. For instance, inaccurate gaze
To mitigate these risks, we advocate for transparent and responsible deployment of the technology, including continuous monitoring and validation of model outputs in real-world clinical settings. Additionally, we encourage collaboration with domain experts to ensure fairness and avoid unintended harms.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification: The work described in the paper does not involve high-risk models or data that could pose significant misuse risks. Our gaze prediction model, designed for medical
19

---

image analysis, does not fall under categories such pretrained language models or image generators with high potential for harmful or dual-use applications. However, acknowledge the importance of responsible research practices.
While our model itself does not pose direct misuse risk, we have taken care to ensure that datasets used are either publicly available and ethically sourced or have been anonymized prevent the disclosure of sensitive information. In cases where the technology could potentially be misapplied, such as in other domains outside medical image analysis, we encourage the community implement safeguards for their respective models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets used in this work, including datasets and models, are properly credited with clear citations and licensing terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Any new assets introduced in this paper, including custom datasets, models, or code, are well documented, and the documentation provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
20

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does involve crowdsoureing or research with human subjects. No such data collection or experimentation was conducted for the work presented.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects, and thus no IRB approval or equivalent was required.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMS if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [No]
Justification: The core methodology and scientific rigor of the research do not involve LLMs as an important or original component. L.LMs were not used in the development of the core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
21

---

Please refer to our LLM policy http:s//heerriisccc///ofrrrrrrsr for what should or should be described.
22


Reviews:
[{"review_id": "3SNjnknwF4", "reviewer": "Reviewer_e5fz", "summary": "This paper proposes a new multimodal model called LogitGaze-Med, which leverages vision-language grounding and is loosely based on concepts from visual attention and how humans process images. This approach is applied to the interpretation of radiological chest X-rays in the medical imaging domain (i.e. MIMIC-CXR and GazeSearch datasets). The model accurately predicts both fixation locations and durations, which enables generation of realistic scanpaths based on expert human eye tracking data. Quantitative experiments show that the proposed model improves predicted scanpath similarity and can also be useful when applied to downstream classification tasks which leverage the predicted fixations.", "questions": "I have a few questions:\n\n1.  The attention maps are mostly used for downstream classification, but it may be interesting to evaluate similarity with the human-derived maps (e.g. Figure 4). Are attention maps from LogitGaze-Med more similar to human maps, relative to the other baseline approaches?\n\n2. Did the authors consider an additional ablation related to temporal shuffling of the predicted scanpath? This could be done by either shuffling the durations of predicted fixations, or the order in which fixations are visited in the scanpath.\n\n3. The authors only show results on chest X-rays, but presumably the framework could be applied to other medical image interpretation tasks such as digital pathology. Do the authors have thoughts on this?", "limitations": "The paper addresses potential limitations in the Conclusion section, although additional limitations could be enumerated there. For example, use of synthetic scanpath data based on the MIMIC-CXR dataset may introduce biases in the evaluation based on how the data is generated.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**: Overall, this paper is well-written and tackles the meaningful clinical problem of chest X-ray interpretation and classification. The proposed LogitGaze-Med architecture and associated loss functions are relatively intuitive, enabling modular use of different modalities as separate components of the model. The use of the model for both scanpath prediction and classification makes for a harder set of tasks, increasing the novelty of the approach. In general, the experiments in the paper are well designed with appropriate metrics, with appropriate baseline comparisons to prior work and ablation studies.\n\n**Weaknesses**: There are a few minor weaknesses to the paper:\n\n- LogitGaze and Gazeformer are trained on natural images, not chest X-rays, yet they are included in the results as baselines, which may be somewhat unfair given the domain mismatch. Are there other foundational vision-language models (VLMs) in the radiology domain would be useful to compare against?\n\n- I'm not very familiar with the other proposed baselines for scanpath prediction, but the authors state that, \"The initial query Q_0 is modulated with the first human fixation via 2D positional encoding\" (line 127). This may be providing the proposed LogitGaze-Med approach with an unfair advantage, as it may not need to fully predict the first fixation. Can the authors clarify if all approaches are conditioned with the first fixation?\n\n- In terms of ablation studies, only semantic continuity and domain specificity are evaluated. I would like to see how the different model components impact the final performance (e.g. removing the vision encoder, logit lens, or text encoder, respectively). This is needed to prove that the proposed architecture and joint embeddings are indeed beneficial.\n\n- In addition to the GazeSearch dataset, the authors generated synthetic scanpaths based on the MIMIC-CXR dataset, but they do not clearly state how they validated this dataset as many details are missing (e.g. do statistics of dwell times match real data?) This limits reproducibility and may affect the conclusions that can be drawn from the synthetic data results.", "was_revised": true, "final_justification": "I believe the authors' clarification of their experiments have addressed my concerns in that area; I would like to see additional information and results on validating the proposed MIMIC-CXR synthetic dataset in the final paper, as it currently used to show the benefit of their approach.", "rebuttal": "Thank you for the positive feedback, and interesting questions.\n\n**Q1: The attention maps are mostly used for downstream classification, but it may be interesting to evaluate similarity with the human-derived maps (e.g. Figure 4). Are attention maps from LogitGaze-Med more similar to human maps, relative to the other baseline approaches?**\n\nTo assess the alignment between our model’s attention maps and human gaze-derived maps, we conducted a blinded qualitative evaluation on a hold‑out set of 100 randomly selected chest X‑rays. A board‑certified radiologist with over five years of experience reviewed side‑by‑side attention overlays from LogitGaze‑Med and each baseline model without knowing which map belonged to which method. In every case, the expert found that LogitGaze‑Med’s maps were more faithful to true gaze behavior—focusing on clinically relevant regions and avoiding spurious activations seen in baseline outputs.\n\nWe note that, for fairness, all models in our comparison (including LogitGaze, Gazeformer, and our own) were trained from scratch on the GazeSearch dataset rather than using their original pre‑trained weights on natural images. Furthermore, to eliminate any bias from initialization, all approaches—including the baselines—were identically conditioned on and initialized with the first fixation. This ensures that performance differences are not due to domain‑specific pre‑training or unequal input conditioning, but rather to the architectural and loss innovations of LogitGaze‑Med.\n\nWhile this assessment is inherently qualitative, it provides strong evidence of our model’s superior perceptual alignment with expert reasoning. In the revised manuscript, we will include these findings—together with richer visual comparisons—and outline plans for a more comprehensive, quantitative evaluation—we plan to measure KL divergence on the same 100‑image hold‑out set, in future work.\n\n**Q2: Did the authors consider an additional ablation related to temporal shuffling of the predicted scanpath? This could be done by either shuffling the durations of predicted fixations, or the order in which fixations are visited in the scanpath.**\n\nWe agree that assessing the impact of temporal structure—such as the order or duration of fixations—on downstream tasks is a valuable direction. In this work, we focused our ablation studies on spatial shuffling and permutation of fixation coordinates to isolate the effect of spatial attention, which we found to be already quite informative.\n\nIn fact, we have conducted a comprehensive component‑wise ablation (detailed in our response to Reviewer 2 and summarized in Table 2). The results show a clear progression in performance across all scanpath similarity metrics. Notably, removing both the text prompt and the patch‑level attention mechanism results in a drastic drop in performance—indicating that clinical intent and region‑level semantic grounding are critical—while the text‑only variant performs on par with strong baselines like GazeFormer, and our full model achieves the best overall performance.\n\nWhile temporal‑shuffling ablations (e.g., permuting fixation order or durations) would yield further insights into the role of temporal dynamics, we have deferred that study to future work. Once the spatial and semantic contributions are fully characterized, we plan to extend our evaluation to include these temporal analyses.\n\n**Q3: The authors only show results on chest X-rays, but presumably the framework could be applied to other medical image interpretation tasks such as digital pathology. Do the authors have thoughts on this?**\n\nWhile our experiments focus on chest X-rays, the LogitGaze‑Med framework is inherently modality-agnostic and can be extended to other medical imaging domains, including digital pathology.\n\nThe key idea—predicting human-like scanpaths and using them to guide spatial attention within feature maps—is applicable wherever fine-grained interpretability and localization are critical. In the case of digital pathology, for example, our scanpath prediction module could be integrated with detection or segmentation pipelines to prioritize diagnostically relevant regions in large-scale whole‑slide images (e.g., CAMELYON16).\n\nWe view this as a natural next step, especially given the increasing importance of interpretable visual reasoning in complex domains. Exploring such extensions—potentially in combination with task-specific backbones (e.g., U-Net, YOLO, MedSAM)—is one of the parts of our planned future work."}, {"review_id": "9X6W1bA3Wp", "reviewer": "Reviewer_Fg6X", "summary": "The authors introduced LogitGaze-Med, a framework to predict the spatio-temporal sequence of scanpaths that occur during visual search in radiology. Specifically, LogitGaze-Med leverages visual features from domain-specific encoders, text embeddings, and patch-level priors extracted from a medical vision language model.\n\nThe authors claim three contributions: \n\n(1) LogitGaze-Med, the first framework to apply logit-lens decoding to an instruction-tuned medical VLM (LLaVA-Med), enabling clinically grounded patch-level attention for gaze prediction.\n\n(2)  Show that domain-specific components—LLaVA-Med and a CheXNet-style encoder—consistently outperform generic baselines in scanpath prediction metrics.\n\n(3) Formulate scanpath prediction as continuous regression over spatial coordinates and dwell durations, and demonstrate that predicted fixations improve downstream pathology classification.", "questions": "Please provide an experiment behind each claim. If the experiment is not provided, please remove the claim. \n\nOverall, the value of this work could increase if ablations are provided:\n1 Test adding a decoder on top of a CLIP model (e.g. biomedclip, bmcclip, etc) and compare against your method.\n\n2. Please add an extensive empirical experiment or add different encoders. What happens if you use lava vs lava-med or lava-rad? \n\n3. Add an experiment to quantify how much improvement you get from each step of your method.\n\n4. Please provide a human evaluation of the quality of your synthetic dataset/ remove it if it's not possible to assess its quality.\n\n5. Compare the performance of your method against other VLLMs (if you wish to keep the claim: \n\"Existing methods often rely on generic vision–language models and saliency-based features, which limits their ability to capture clinical semantics and integrate domain knowledge effectively.\"\n\n6. Please include newer encoders and VLMs in your evaluation and experimentation.", "limitations": "While the work does address an interesting and relevant problem, it does not offer enough technical novelty, in makes claims without adequate experiments, and it's vague about some of its components. \n\nIn the absence of adequate technical novelty, an empirical study would add sufficient value for this work to be accepted in this venue. I encourage authors to explore the questions provided above.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nPlug-and-play framework: The authors leverage an existing framework to integrate existing domain-specific models (instead of training a model from stracth). \n\nStrong empirical performance: LogitGaze-Med achieves state-of-the-art results across multiple scanpath similarity metrics and improves downstream pathology classification (AUROC).\n\nComprehensive evaluation across multiple datasets: The authors collect multiple datasets to evalaute their models. Additionally they created a synthetic dataset using MIMIC.\n\nWeakeness: \n\n* The main novelty of this work is to leverage an existing method (LOGITGAZE) in radiology. There is no new technical contribution \n\n* Many claims are not justified with empirical experiments. For example, \"Existing methods often rely on generic vision–language models and saliency-based features, which limits their ability to capture clinical semantics and integrate domain knowledge effectively.\" Where is the comparison (e.g. performance on classification) of LogitGaze-Med against Llava-med, Llava-rad, biomedclip, BMC-clip, etc?\n\n* The selection of feature extracores is not thoroughly justified. For example, the selection of Chexpert (a model published 6 years ago) or llava-med (instead of llava-rad, a specific llava model for radiology) seems a bit arbitrary. As such, an ablation for the selection of models is required. Regarding the optimality of this method, would it be possible to achieve similar performance with newer/ smaller models? \n\n* Not sufficient basic baselines. For instance, are all modalities required? What happens if you remove text? or the llava feautres? Furthermore, is this approach required? Why not just add a decoder on top of BiomedClip? \nMIMIC-CXR\n\n* There is no evaluation  (nor clear details of data construction) to demonstrate the quality of the synthetic data generated from MIMIC-CXR.", "was_revised": true, "final_justification": "My concerns were addressed", "rebuttal": "We are grateful for the reviewer’s comments, which highlighted important aspects of model design, empirical validation, and clarity. We have addressed each point in detail below.\n\n**Q1: Test adding a decoder on top of a CLIP model (e.g. biomedclip, bmcclip, etc) and compare against your method.**\n\nWe appreciate the suggestion and agree that CLIP-style architectures like BioMedCLIP represent an important baseline.\n\nPrior to selecting our current approach, we analyzed CLIP-based models for gaze prediction. While BioMedCLIP offers strong global image–text alignment, it lacks two key properties required for our setting: (1) Patch-level semantic grounding — CLIP is trained with global contrastive objectives and does not produce fine-grained, spatially interpretable logits, making regional fixation modeling difficult; (2) Instruction-style conditioning — CLIP models do not support prompt-based attention control, which is central to our approach via LLaVA-Med and logit-lens decoding.\n\nIn contrast, our method leverages patch-level attention guided by clinical intent, enabling interpretable and semantically aligned scanpath prediction. That said, we agree that comparing to a decoder built atop BioMedCLIP would be informative. We are currently implementing a lightweight version and plan to report results in future work to better highlight the impact of regional grounding.\n\n**Q2: Please add an extensive empirical experiment or add different encoders. What happens if you use lava vs lava-med or lava-rad?**\n\nWe conducted additional experiments to evaluate how different combinations of VLMs and visual encoders affect scanpath prediction within the LogitGaze framework.\n\nWe compared two instruction-tuned medical VLMs: LLaVA-Med (used in our original setup) and LLaVA-Rad, a radiology-specific model trained on 697k image–report pairs using modular adapters. For encoders, we tested CheXNet, CheSS [1] (a ViT trained on 4.8M chest X-rays via contrastive learning), and PEAC [2] (a patch-level SSL model enforcing anatomical consistency across views). All variants were evaluated on the GazeSearch dataset using standard scanpath metrics (Table 1).\n\nLLaVA-Med and LLaVA-Rad yielded comparable results with the same encoder, confirming robustness to VLM choice. While CheSS and PEAC slightly improve some metrics, CheXNet remains competitive, particularly in duration-aware ScanMatch — supporting its use as a strong baseline. These findings suggest that performance gains stem primarily from integrating clinically grounded attention (via logit-lens and text prompts), rather than from any single backbone.\n\nWe will include these results and implementation details in the revised manuscript to support our claim that spatial–semantic grounding is more impactful than raw model scale or recency.\n\n$$\n\\\\begin{array}{lcccc}\n\\\\hline\n\\\\textbf{Method} & \\\\textbf{ScanMatch w/o Dur.} \\\\uparrow & \\\\textbf{ScanMatch w/ Dur.} \\\\uparrow & \\\\textbf{SED} \\\\downarrow & \\\\textbf{STDE} \\\\uparrow \\\\\\\\\n\\\\hline\n\\\\text{LogitGaze-Med (ResNet)} & 0.416 \\\\pm 0.017 & 0.325 \\\\pm 0.012 & 4.68 \\\\pm 0.05 & 0.852 \\\\pm 0.003 \\\\\\\\\n\\\\text{LogitGaze-Med (CheXNet)} & 0.419 \\\\pm 0.016 & \\\\mathbf{0.330} \\\\pm \\\\mathbf{0.010} & 4.68 \\\\pm 0.05 & 0.855 \\\\pm 0.003 \\\\\\\\\n\\\\text{LogitGaze-Med (CheSS)} & 0.425 \\\\pm 0.017 & 0.321 \\\\pm 0.012 & 4.66 \\\\pm 0.05 & 0.857 \\\\pm 0.003 \\\\\\\\\n\\\\text{LogitGaze-Med (PEAC)} & 0.428 \\\\pm 0.018 & 0.319 \\\\pm 0.012 & 4.64 \\\\pm 0.05 & 0.858 \\\\pm 0.004 \\\\\\\\\n\\\\text{LLaVA-Rad + CheXNet} & 0.417 \\\\pm 0.016 & 0.329 \\\\pm 0.010 & 4.67 \\\\pm 0.05 & 0.854 \\\\pm 0.004 \\\\\\\\\n\\\\text{LLaVA-Rad + CheSS} & 0.426 \\\\pm 0.018 & 0.320 \\\\pm 0.012 & 4.65 \\\\pm 0.05 & 0.857 \\\\pm 0.004 \\\\\\\\\n\\\\textbf{LLaVA-Rad + PEAC} & \\\\mathbf{0.429} \\\\pm \\\\mathbf{0.019} & 0.318 \\\\pm 0.012 & \\\\mathbf{4.63} \\\\pm \\\\mathbf{0.05} & \\\\mathbf{0.859} \\\\pm \\\\mathbf{0.004} \\\\\\\\\n\\\\hline\n\\\\end{array}\n$$\n**Table 1:** Performance on scanpath similarity metrics (higher is better for ScanMatch/STDE, lower is\nbetter for SED).\n\n**Q3: Add an experiment to quantify how much improvement you get from each step of your method.**\n\nTo evaluate the contribution of each component, we performed a stepwise ablation that isolates the effects of the image encoder, text prompt, and logit-lens priors. We tested three variants: (1) image-only baseline (no text, no logit-lens); (2) +text, without logit-lens; and (3) full LogitGaze-Med with both modalities. Results on GazeSearch are shown in Table 2.\n\nThe image-only model performs poorly across all metrics, confirming that visual features alone are insufficient. Adding clinical text substantially improves performance, highlighting the value of semantic intent. The full model, which includes patch-level grounding via logit-lens, achieves the best results — especially on spatially and temporally sensitive metrics — demonstrating the benefit of combining clinical context with localized attention.\n\n$$\n\\\\begin{array}{lcccc}\n\\\\hline\n\\\\textbf{Method} & \\\\textbf{ScanMatch w/o Dur.} \\\\uparrow & \\\\textbf{ScanMatch w/ Dur.} \\\\uparrow & \\\\textbf{SED} \\\\downarrow & \\\\textbf{STDE} \\\\uparrow \\\\\\\\\n\\\\hline\n\\\\text{Image only (no text, no logit-lens)} & 0.148 \\\\pm 0.039 & 0.126 \\\\pm 0.034 & 8.54 \\\\pm 0.28 & 0.562 \\\\pm 0.094 \\\\\\\\\n\\\\text{w/ Text only (no logit-lens)} & 0.280 \\\\pm 0.020 & 0.206 \\\\pm 0.013 & 5.95 \\\\pm 0.06 & 0.810 \\\\pm 0.004 \\\\\\\\\n\\\\textbf{LogitGaze-Med} & \\\\mathbf{0.419} \\\\pm \\\\mathbf{0.016} & \\\\mathbf{0.330} \\\\pm \\\\mathbf{0.010} & \\\\mathbf{4.68} \\\\pm \\\\mathbf{0.05} & \\\\mathbf{0.855} \\\\pm \\\\mathbf{0.003} \\\\\\\\\n\\\\hline\n\\\\end{array}\n$$\n\n**Table 2:** Stepwise ablation analysis of LogitGaze-Med on GazeSearch. Adding textual prompts improves alignment with clinical intent; incorporating logit-lens guidance provides further gains via patch-level priors.\n\n**Q4: Please provide a human evaluation of the quality of your synthetic dataset/ remove it if it's not possible to assess its quality.**\n\nWhile the synthetic scanpaths are not intended as a standalone contribution or benchmark, they play an important role in demonstrating the potential of learned gaze priors for guiding model attention and enhancing interpretability.\n\nTo assess their plausibility, we conducted a blinded qualitative evaluation with a board-certified radiologist who reviewed a hold-out set of 100 chest X-rays. Each image was overlaid with either a real or a synthetic scanpath, and the expert was asked to evaluate the fixation patterns in terms of realism and alignment with clinically relevant regions, without knowledge of their origin.\n\nIn the majority of cases, the expert noted that the synthetic scanpaths closely resembled typical radiological search behavior. The fixations were generally coherent, avoided implausible image regions, and often corresponded to diagnostically meaningful structures. While the expert could distinguish synthetic from real in some instances—particularly where the timing of fixations appeared overly uniform—the overall impression was that the generated scanpaths were realistic and clinically interpretable.\n\nWe agree that more systematic evaluation, including inter-rater analysis and larger-scale studies, would be necessary to establish generalizability. We view this as an important direction for future work, and we are currently conducting a more extensive investigation into the use of synthetic gaze data in human-in-the-loop diagnostic settings.\n\n**Q5: Compare the performance of your method against other VLLMs (if you wish to keep the claim: \"Existing methods often rely on generic vision–language models and saliency-based features, which limits their ability to capture clinical semantics and integrate domain knowledge effectively.\"**\n\nWe agree that the original phrasing may be too broad, and we appreciate the opportunity to clarify our intent.\n\nIn our statement, we primarily aimed to emphasize that many prior approaches—particularly those built on general-purpose VLMs—do not explicitly adapt or fine-tune their architecture or supervision to reflect the clinical semantics inherent in medical imaging. For example, standard VLMs trained on COCO or other natural image-text pairs may overlook subtle pathologies or fail to prioritize medically relevant regions due to domain mismatch.\n\nIn contrast, our method incorporates domain-specific adaptations in multiple ways: we utilize a medical vision encoder pretrained on chest X-rays, introduce a semantic projection module that creates interpretable task-specific embeddings, and supervise gaze prediction through a clinically grounded multi-loss objective. This allows our model to integrate domain knowledge more explicitly, improving both fixation modeling and downstream diagnostic utility. To avoid overstatement, we will revise the original sentence in the manuscript.\n\n**Q6: Please include newer encoders and VLMs in your evaluation and experimentation.**\n\nWe agree, and as noted in our response to Q2, we have already incorporated several recent components into our evaluation. Specifically, we compared LLaVA-Med with LLaVA-Rad — a radiology-specific VLM — and tested modern medical image encoders such as CheSS and PEAC. While these newer models offer slight performance gains, our results demonstrate that the proposed framework is robust across a range of VLM and encoder configurations.\n\n[1] Cho, Kyungjin, et al. \"CHESS: Chest X-Ray Pre-trained model via self-supervised contrastive learning.\" Journal of Digital Imaging 36.3 (2023): 902-910.\n\n[2] Zhou, Ziyu, et al. \"Learning anatomically consistent embedding for chest radiography.\" BMVC: proceedings of the British Machine Vision Conference. British Machine Vision Conference. Vol. 2023. 2023."}, {"review_id": "9KU0I1A5ZZ", "reviewer": "Reviewer_mK2d", "summary": "This paper proposes LogitGaze-Med, a multimodal framework that integrates domain-specific visual and textual embeddings with semantic patch-level priors to predict task-driven scanpaths in medical images. The textual embeddings are derived from diagnostic labels, and semantic priors are extracted using a logit-lens from a medical vision–language model (LLaVA-Med). The goal is to enable clinically meaningful, patch-level attention modeling for gaze prediction. The authors formulate scanpath prediction as a continuous regression task over spatial coordinates and dwell durations. Evaluations show that the model produces clinically meaningful gaze patterns and improves downstream pathology classification when predicted fixations are used in training.", "questions": "1.\tCan you use the real fixation data for MIMIC-CXR to report the performance in Tables 1 and 2? A clarification for why they are not used in the evaluation would be helpful for reassessing the paper.\n2.\tAre the weighting parameters (coordinate loss: λ₁, dwell time loss: λ₂) fixed or learnable during training? How sensitive is model performance to these hyperparameters? Can you include an ablation study varying λ₁ and λ₂, and report the effects on scanpath similarity. This would clarify their roles in model optimization.  \n3.\tIn Figure 3, the predicted scanpaths from LogitGaze-Med and GazeSearch appear visually similar for certain diagnoses (e.g., cardiomegaly, pneumonia), and in Figure 4, attention maps for different conditions seem indistinct. Can you provide different qualitative examples to reduce the confusion?\n4.\tCan you clarify what novel modeling contributions your method makes beyond reusing pretrained components, and how LogitGaze-Med fundamentally differs from prior works such as LogitGaze? Please elaborate on how your model introduces new mechanisms for fixation prediction beyond leveraging existing VLM infrastructure.", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n+ This paper presents a well-motivated approach to scanpath prediction in medical imaging by integrating domain-specific visual encoders, textual embeddings of diagnostic labels, and semantic patch-level priors via logit-lens decoding from a medical VLM. The proposed framework is methodologically sound and leverages clinically grounded semantic priors to model gaze behavior in a task-driven setting.\n+ The paper applies logit-lens decoding to LLaVA-Med for extracting patch-level semantic priors. \n+ The paper demonstrates experimental results on the GazeSearch dataset where the model outperforms existing baselines across multiple scanpath similarity metrics. In the downstream classification task with MIMIC-CXR, it improves the pathological classification. \n+ The paper has detailed descriptions of evaluation metrics. Figures for the qualitative results are used effectively to illustrate the performance.  \n+ The significance of the work lies in its ability to bridge gaze prediction with clinical reasoning by generating fixations that align with human attention and diagnostically meaningful regions.\n\nWeaknesses:\n- Although this method can do a good job of predicting the gaze scanpath, there are some theoretical and experimental gaps in understanding how the architecture works.\n- A substantial portion of the framework’s novelty hinges on logit-lens decoding of pretrained models (e.g., LLaVA-Med), which may raise concerns about the originality of the core algorithmic contributions. While the integration is creative and domain-specific, more justification in modeling fixations beyond architecture reuse would strengthen the paper.\n- In Line 200, the authors claim to test generalization by discarding human fixations and replacing them with synthetic ones. However, it is not explained how this strategy robustly tests generalization. Notably, although the MIMIC-CXR dataset has real fixation data [1], Tables 1 and 2 only report results on GazeSearch. The absence of performance metrics using real fixations from MIMIC-CXR raises concerns about the model’s claimed generalization ability.\n- The authors introduce joint losses for coordinate regression and dwell time regression, but it is not specified whether the associated weighting coefficients (λ₁, λ₂) are fixed or learnable. An ablation study analyzing the sensitivity to these hyperparameters would help clarify the relative importance of each component and improve the transparency of the training process.\n- In line 181, the authors claim that LogitGaze-Med produces more human-like scanpaths. However, in Figure 3, especially in the “Cardiomegaly” and “Pneumonia” tasks, the predicted paths of LogitGaze-Med and GazeSearch look quite similar. Maybe including other qualitative examples could strengthen the claim.\n- It remains unclear whether, during inference, the model requires the X-ray image, the semantic embeddings (e.g., “rib, lung”, “rib, gray”), and the disease labels. If such semantic phrases and/or labels are essential for prediction, it raises practical deployment concerns (whether radiologists must manually provide these prompts, or if they are generated automatically). A clarification on whether semantic priors are predicted, retrieved, or externally provided at test time would remove this confusion.\n- In Figure 4, the attention maps produced by the Eye-Gaze baseline for the “Normal” and “CHF” conditions appear nearly identical. This casts doubt on the meaningfulness of these comparisons. Either more or improved qualitative illustrations are needed to validate the claim that LogitGaze-Med yields more discriminative and clinically aligned attention maps.\n\nMinor Concern: \n- Although the model is said to produce interpretable fixations, the paper lacks feedback from radiologists to validate whether these gaze patterns truly reflect diagnostic reasoning.\n- In Line 166, please provide a citation to support the statement that 'AUROC serves as a proxy for clinical utility and interpretability,' as this is a strong claim that would benefit from justification. \n\n[1] Hsieh, C., Ouyang, C., Nascimento, J. C., Pereira, J., Jorge, J., & Moreira, C. (2023). Mimic-eye: Integrating mimic datasets with reflacx and eye gaze for multimodal deep learning applications. PhysioNet (version 1.0. 0).", "was_revised": true, "final_justification": "The authors responded to a few questions, but the majority items in the weakness were not mentioned. The reviewer still has concerns on those items and remains the current reject decision.", "rebuttal": "We thank the reviewer for thorough and insightful comments, which help us better explain and contextualize the innovations and evaluation of LogitGaze-Med.\n\n**Q1: Can you use the real fixation data for MIMIC-CXR to report the performance in Tables 1 and 2? A clarification for why they are not used in the evaluation would be helpful for reassessing the paper.**\n\nYes, we do use real fixation data from MIMIC-CXR in our experiments. Specifically, the evaluation results in Tables 1 and 2 are based on human scanpaths from the GazeSearch framework, which were collected via eye-tracking on MIMIC images.\n\nImportantly, the GazeSearch authors identified that the raw fixation data can be ambiguous for modeling targeted visual attention, since radiologists were asked to assess multiple conditions simultaneously without explicit prompts. As a result, individual fixations may reflect multiple potential findings, and the aggregated gaze data often spans broad regions, making it poorly suited for localization tasks.\n\nTo address this, the GazeSearch team applied preprocessing steps to segment, clean, and temporally refine the raw fixations—producing more diagnostically meaningful scanpaths. Our study builds directly on this processed dataset, rather than the raw fixations, in order to better model expert visual search behavior and to provide reliable attention signals for downstream tasks.\n\nWe acknowledge the availability of MIMIC-Eye fixations, but our study relies on the GazeSearch framework, which includes preprocessing steps to isolate task-driven attention patterns. We found that using unprocessed fixations leads to diffuse, non-discriminative scanpaths that degrade downstream interpretability.\n\nUsing the raw fixations would likely result in overly diffuse attention maps and reduced interpretability, especially in tasks requiring precise localization. We believe the use of processed gaze data provides a more faithful and clinically relevant benchmark for evaluating our approach.\n\n**Q2: Are the weighting parameters (coordinate loss: λ₁, dwell time loss: λ₂) fixed or learnable during training? How sensitive is model performance to these hyperparameters? Can you include an ablation study varying λ₁ and λ₂, and report the effects on scanpath similarity. This would clarify their roles in model optimization.**\n\nIn our current setup, the weighting parameters for coordinate loss (λ₁) and dwell time loss (λ₂) are fixed throughout training. The baseline values (λ₁=1.0, λ₂=1.0) were selected via preliminary validation to balance spatial versus temporal scanpath fidelity.\nWe conducted a targeted ablation on the held-out GazeSearch test set using the MultiMatch metric (shape, direction, length, position, duration) on LogitGaze-Med(CheX) model to assess sensitivity to these hyperparameters.\n\nAs the results show in Table 1, increasing spatial emphasis (λ₁) improves shape and position consistency, while reducing temporal emphasis (λ₂) degrades duration modeling. Across a broad range (λ₁∈[0.5–5.0], λ₂∈[0.1–5.0]), the model maintains robust performance, with (2.0, 0.5) offering an optimal trade‑off.\nWe will include these findings in the revised manuscript. Exploring learnable weighting schemes or automated hyperparameter optimization remains a promising avenue for future work.\n\n$$\n\\\\begin{array}{lccccc}\n\\\\hline\n\\\\textbf{Loss Weights} & \\\\textbf{Shape} & \\\\textbf{Direction} & \\\\textbf{Length} & \\\\textbf{Position} & \\\\textbf{Duration} \\\\\\\\\n\\\\hline\n\\\\lambda_1=1.0,\\\\ \\\\lambda_2=1.0 & 0.921 & 0.633 & 0.932 & 0.812 & 0.712 \\\\\\\\\n\\\\lambda_1=2.0,\\\\ \\\\lambda_2=0.5 & \\\\mathbf{0.938} & \\\\mathbf{0.651} & \\\\mathbf{0.948} & \\\\mathbf{0.823} & \\\\mathbf{0.740} \\\\\\\\\n\\\\lambda_1=0.1,\\\\ \\\\lambda_2=1.0 & 0.866 & 0.570 & 0.882 & 0.743 & 0.707 \\\\\\\\\n\\\\lambda_1=1.0,\\\\ \\\\lambda_2=0.1 & 0.917 & 0.624 & 0.926 & 0.805 & 0.610 \\\\\\\\\n\\\\lambda_1=1.0,\\\\ \\\\lambda_2=5.0 & 0.910 & 0.615 & 0.920 & 0.792 & 0.689 \\\\\\\\\n\\\\lambda_1=5.0,\\\\ \\\\lambda_2=1.0 & 0.887 & 0.593 & 0.902 & 0.760 & 0.701 \\\\\\\\\n\\\\hline\n\\\\end{array}\n$$\n\n**Table 1:** Effect of varying \\\\(\\\\lambda_1\\\\) (coordinate loss) and \\\\(\\\\lambda_2\\\\) (dwell time loss) on scanpath similarity (MultiMatch components). Best values in bold.\n\n**Q3: In Figure 3, the predicted scanpaths from LogitGaze-Med and GazeSearch appear visually similar for certain diagnoses (e.g., cardiomegaly, pneumonia), and in Figure 4, attention maps for different conditions seem indistinct. Can you provide different qualitative examples to reduce the confusion?**\n\nWe agree that in the current Figures 3 and 4, some scanpaths and attention maps may appear visually similar across different diagnoses and methods, which can obscure the interpretability of the comparisons.\n\nTo address this, we conducted an additional blinded review with a board-certified radiologist (5+ years of experience), who helped us identify cases(N = 100) where the differences between LogitGaze‑Med and baseline methods are more visually and clinically distinguishable.\n\nIn the revised manuscript, we will replace the current figures with these curated examples, which better highlight class-specific scanpath dynamics and attention localization. We will also include the radiologist’s blinded preference scores, which consistently favored LogitGaze‑Med for its sharper focus on diagnostically meaningful regions and clearer distinction from other methods.\n\n**Q4: Can you clarify what novel modeling contributions your method makes beyond reusing pretrained components, and how LogitGaze-Med fundamentally differs from prior works such as LogitGaze? Please elaborate on how your model introduces new mechanisms for fixation prediction beyond leveraging existing VLM infrastructure.**\n\nWhile our method builds upon existing vision–language model (VLM) infrastructure, LogitGaze-Med is not a trivial reuse of pretrained blocks; it introduces novel architectural, representational, and training innovations explicitly designed for fixation prediction in medical imaging, which fundamentally distinguish it from prior works like LogitGaze.\n\n1. Unlike LogitGaze, which directly decodes fixations from generic VLM hidden states (typically trained on COCO-like data), LogitGaze‑Med explicitly disentangles the intermediate representations of a medical-adapted VLM. Through empirical analysis, we found that high-level features from LLaVA-style models lack spatial and semantic resolution. To address this, we introduce a dedicated semantic projection module that filters and restructures these representations into a compact embedding space aligned with clinical semantics. This \"semantic bottleneck\" enhances interpretability and supports multi-task transfer (e.g., gaze prediction, classification).\n\n2. Prior works rely on general-purpose backbones (e.g., ResNet trained on COCO), which are suboptimal for medical tasks. We redesign the visual front-end by incorporating a CheXNet-derived encoder pretrained on chest X-rays, enabling extraction of fine-grained diagnostic signals that are essential for clinically realistic scanpath modeling.\n\n3. LogitGaze‑Med introduces a novel training objective that decomposes the gaze prediction task into three components:\n(i) Token-level alignment loss, to ground fixations in semantic visual tokens;\n(ii) Spatial density loss, to approximate human fixation maps;\n(iii) Temporal consistency loss, to preserve the sequential structure of expert scanpaths.\n\nThis spatiotemporal loss formulation goes beyond the weakly supervised or single-objective setups in previous methods and is essential for learning structured, human-aligned attention.\n\nIn sum, LogitGaze-Med fundamentally departs from prior works by restructuring semantic representations and introducing targeted, multi-faceted training objectives tailored to the unique challenges of medical scanpath prediction."}]

Author rebuttal:

Meta-review: This paper proposes a new multimodal model called LogitGaze-Med, which leverages vision-language grounding and is based on concepts from visual attention and how humans process images. This model is applied to the interpretation of radiological chest X-rays in the medical imaging domain (i.e., MIMIC-CXR and GazeSearch datasets). The model accurately predicts both fixation locations and durations, which enables the generation of realistic scanpaths based on expert human eye tracking data. Experiments show that the model improves predicted scanpath similarity and can also be useful when applied to downstream classification tasks that leverage the predicted fixations.

Initially, a number of concerns were raised by the reviewers regarding the novelty and empirical validation; however, during an active rebuttal phase, the authors were able to push back with a convincing rebuttal on most fronts.

Specifically, the primary strengths of the paper are: 
- A well-written paper solving a clinically relevant problem, using chest X-ray interpretation and classification as a use case.
- Methodologically sound, intuitive, plug-and-play framework that integrates domain-specific models without training from scratch, supporting modular use of different modalities and joint tasks of scanpath prediction and classification. 
- Strong empirical performance, achieving state-of-the-art scanpath similarity metrics, and improving downstream pathology classification (AUROC).
- Comprehensive evaluation: Tested across multiple datasets, including a synthetic dataset based on MIMIC.
- Effectively uses logit-lens decoding: Applied to LLaVA-Med for patch-level semantic priors.
- Bridges' gaze prediction with clinical reasoning by generating fixations aligned with human attention and diagnostically relevant regions.

The weaknesses pointed out to the reviewers (some of which were already addressed by the authors) are: 
- Main novelty lies in applying existing LogitGaze to radiology, with limited technical innovation. Furthermore, several claims (e.g., superiority over generic VLMs) are not supported with direct comparisons.
- Baseline comparisons: More relevant radiology-specific VLM baselines (e.g., LLaVA-Med, LLaVA-Rad, BiomedCLIP, BMC-CLIP) should be compared.
- Concerns with ablation studies: Current ablations only cover semantic continuity and domain specificity.  Missing analysis of how individual components (vision encoder, logit lens, text encoder) affect performance. Model selection (CheXpert, LLaVA-Med) is not fully justified; ablations on alternative/newer models are needed. Unclear whether all modalities are necessary or whether simpler alternatives could achieve comparable performance.
- Synthetic scanpaths from MIMIC-CXR are insufficiently described and not validated (e.g., dwell time statistics). Lack of validation limits reproducibility and weakens conclusions.

As mentioned, most of these concerns were addressed by a thorough rebuttal from the authors. Keeping that in mind, given the uniqueness of the work (even though no new method is developed, the motivation and the ideas behind the entire system are new) and its thorough empirical validation and ablation studies make it worthwhile for this paper to be recommended for publication.
Decision: Accept (poster)