Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning
Authors: Chong Ma
Hanqi Jiang
Wenting Chen
Yiwei Li
Zihao Wu
Xiaowei Yu
Zhengliang Liu
Lei Guo
Dajiang Zhu
Tuo Zhang
Dinggang Shen
Tianming Liu
Xiang Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Medical Multi-modal Alignment
Eye-gaze
Radiology
Primary area: deep_learning_architectures
TL;DR: A Multi-modal Framework Guided by Human Eye-gaze.

Abstract: In the medical multi-modal frameworks, the alignment of cross-modality features presents a significant challenge. However, existing works have learned features that are implicitly aligned from the data, without considering the explicit relationships in the medical context. This data-reliance may lead to low generalization of the learned alignment relationships. In this work, we propose the Eye-gaze Guided Multi-modal Alignment (EGMA) framework to harness eye-gaze data for better alignment of medical visual and textual features. We explore the natural auxiliary role of radiologists' eye-gaze data in aligning medical images and text, and introduce a novel approach by using eye-gaze data, collected synchronously by radiologists during diagnostic evaluations. We conduct downstream tasks of image classification and image-text retrieval on four medical datasets, where EGMA achieved state-of-the-art performance and stronger generalization across different datasets. Additionally, we explore the impact of varying amounts of eye-gaze data on model performance, highlighting the feasibility and utility of integrating this auxiliary data into multi-modal alignment framework.

Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning
Chong Ma', Hanqi Jiang?, Wenting Chen², Yiwei Li,, Zihao Wu?
Xiaowei Yu', Zhengliang Liu?, Lei Guo', Dajiang Zhu', Tuo Zhang'
Dinggang Shen2, Tianming Liu?, Xiang Li
Northwest Polytechnical University University of Georgia CCity University of Hong Kong *University of Texas at Arlington "ShanghaiTech University & Shanghai United Imaging Intelligence Co. "Massachusetts General Hospital, Harvard University
Abstract
In the medical multi-modal frameworks, the alignment of cross-modality features presents a significant challenge. However, existing works have learned features that are implicitly aligned from the data, without considering the explicit relationships in the medical context. This data-reliance may lead to low generalization of the learned alignment relationships. In this work, we propose the Eye-gaze Guided Multi-modal Alignment (EGMA) framework to harness eye-gaze data for better alignment of medical visual and textual features. We explore the natural auxil- iary role of radiologists' eye-gaze data in aligning medical images and text, and introduce a novel approach by using eye-gaze data, collected synchronously by radiologists during diagnostic evaluations. We conduct downstream tasks of image classification and image-text retrieval on four medical datasets, where FGMM achieved state-of-the -hrtt--atttttttt performance and stronger generalization across different datasets. Additionally, we explore the impact of varying amounts of eye-gaze data on model performance, highlighting the feasibility and utility of integrating this auxiliary data into multi-modal alignment framework.
Introduction
With the development of multi-modal learning, pre-trained models can now utilize large amounts of paired multi-modal data, such as image-text pairs, audio-text pairs, etc., optimize the multi- modal feature extraction and alignment capabilities. With the emergence the CLIP 42 model, contrastive learning has become the prominent framework of multi-modal learning. The advantage of this framework lies in its simplicity of structure and it does not require sample-level annotations. However, the main drawback is its heavy reliance on the scale of training data. Subsequent works have optimized this framework by leveraging potential auxiliary information between image and text data. For instance, GLIP [32] and RegionCLIP [6]] utilized pre-predicted annotation information to perform fine- ine-ggrined region-level pre-training. They introduced detection networks firstly to predict image regions relevant to the text prompt, and then trained the model to align these image regions with their corresponding text descriptions. However, these models heavily rely on the performance of the ROI detector and have high computational complexity. FILIP (77] proposed a refined multi-modal alignment operation after the encoder, relying solely on image patches and text tokens. Although this further explores the local feature relationships between multi-modal data, still requires sufficient data support. When training small-scale datasets, especially in the medical field, accurately learning alignment features between modalities becomes more challenging 66 60)
"Corresponding author; xli60 ali60@mgh.hmll www.harvrdd.dddd.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To address the scarcity of medical data, studies 58 have introduced self-supervised training into the CLIP framework further enhance encoder performance. Additionally, weak labels between images and texts have been incorporated during pre-training to aid multi-modal alignment [55] Some studies [5] 53 utilized fine-grained alignment between chest image patches and text tokens for pre-training 157. However, unlike natural images and text, the relationship between medical images and diagnostic text is often more complex and challenging to learn. Moreover, with insufficient data, models are prone to learning shortcut features unrelated to disease diagnosis, resulting in poor generalization ability Do 36 35]. Therefore, it is crucial to learn useful alignment information from relatively limited medical multi- multi-model datasets.
'normal heart'
E
Aligned
Gaze Gaze-guided alignment
E
Radiologist
A Atention Region
Gaze Point [ Audio Transcripls
Patch Feature
Sentence Feature
Figure 1: The guiding role of radiologists' eye-gaze data. The text provided by radiologists during diagnosis aligns naturally with the attention regions.
In this study, we fully explore the auxiliary role of eye-gaze data from radiologists in multi-model alignment. Eye-gaze data can intuitively reflect the image regions radiologists focus on, providing insights into their cognitive behavior during diagnosis 77]. Therefore, compared to refined annotations like bounding boxes and masks, eye-gaze data can also provide useful auxiliary information for the model [24] (4445 35). Moreover, collecting eye-gazz data from radiologists during the diagnostic process is more time-efficient than annotating bounding boxes and masks 25 35). For the modal medical dataset, EYE GAZE [22) and REFLACX BI] collected eye-gaze data from radiologists while diagnosing chest X-rays. Additionally, these datasets recorded synchronized voice data, where radiologists verbalized their diagnoses while observing the images. As shown in Fig. [ we found that the radiologists' attention regions on the image naturally align with the diagnostic text over time. Therefore, we believe this type of eye-gaze data can provide expert prior knowledge for training the alignment between medical visual and textual features. Thus, considering the utilization of eye-gaze data to assist in multi-modal model training, we propose the Eye-gaze Guided Multi-modal Alignment framework (EGMA). Our model first segments the transcribed text into individual sentences and obtains radiologists' attention heatmaps. Subsequently, we obtain encoded features of image patches and sentences through image and text encoders, generating instance-level similarity matrix. Then, we compute the loss between this matrix and the attention heatmaps, integrating refined feature representations for subsequent contrastive loss. To further leverage the assisting role of eye-gaze data in aligning images and texts, we combine the eye-gaze heatmaps with the similarity matrix derived from model, serving as weights to calculate cross-modality mapping loss. Experimental results on zero-shot classification and retrieval tasks reveal that our framework surpasses other leading methods in performance across diverse datasets and under multiple dataset size scenarios. Specifically, the EGMA framework yielded a remarkable 3.9% improvement in image-to-text matching tasks and an impressive 19.75% increase in text-to-image matching tasks. These results underscore the cutting-edt edgee and efficacious nature of our approach, highlighting its substantial advancements over existing methodologies. We also explore the auxiliary effect of using eyc-gaze data of different scales on the model, finding that even a small portion of eye-gaze data can enhance the model's multi-modal processing capability. Moreover, the fine-tuned classification results of EGMA achieved the best performance across multiple datasets. The code of this work is available Githul
In summary, the main contributions of this work are as follows:
htatp:///github..co/MMbaaaaa

---

We propose EGMA, novel framework for medical multi-modee alignment, marking the first attempt integrate eye-gaze data into Dasounlanguage pre-training.
EGMA outperforms existing mate-offthee medical multi- modal pre- me-training methods, and realizes notable enhancements in image classification and image-text retrieval tasks.
EGMA demonstrates that even small amount of eye-gaze data can effectively assist in multi-modal pre-training and improve the feature representation ability of the model.
2
Related Works
Medical Vision-language Pre-training (Med-VLP): In pursuit of Artificial General Intelligence (AGI), Vision- Vision-language Pre-training (VLP) has become a pivotal area in Al research. The advent of the transformer architecture [51] has significantly accelerated progress in the multi-modal domain by integrating vision and language, with VLP frameworks focusing on fusion encoders that use cross-attention mechanisms amalgamate visual and textual features (34 47 The introduction of
CLIP [42] marked breakthrough, leading to numerous CLIP-bascd VLP frameworks incorporating contrastive loss as a core component [57 32]. In the medical field. COnVIRT [59] serves as an equiv- alent to CLIP, while MedCLIP [55] addresses the challenge of insufficient paired image-text data by integrating knowledge extraction techniques. BioViL (2 enhances performance through specialized biomedical text BERT encoders in contrastive learning tasks. GLoRIA [I5] proposes multi-modal global-local representation learning, and MGCA [53] introduces alignment pathological region,
instance, and disease levels. Furthermore, study 56 incorporates knowledge bases to infuse expert
medical knowledge into the system.
Eye-tracking Technology in Radiology: In medical imaging diagnostics, eye-tracking technology has proven valuable for decades, revealing how experienced radiologists can quickly identify hidden lesions through comprehensive observation [28] 29 7 [6).. Integrating radiologists eye-gaze data with deep learning models has significantly advanced the field. For instance, merging eye-gaze data with Convolutional Neural Networks (CNNs) has enhanced lesion detection accuracy [24]. and visual search patterns in mammography have linked human visual attention with CNN performance (37). Comprehensive datasets combining eye-gaze data and disease diagnoses have facilitated multi-task processing (21), while attention consistency modules have improved CNN accuracy in diagnosing osteoarthritis from knee X-rays [54]. Recently, integrating eye-gaze data with Vision Transformer (ViT) models has further pushed the boundaries of medical image processing 35 Additionally, multi- modal guidance systems replicating eye tracking and probe manipulation in ultrasound examinations have significantly enhanced scanning accuracy and efficiency [38]. Despite these advancements, fully integrating eye-gaze data with image-text alignment strategies in medical vision-language models remains an ongoing research challenge.
Method
As shown in F the framework of our proposed method consists of four main components. Firstly, we extract features from image and text in part A to obtain a refined instance-level similarity matrix. Secondly, in part B, we integrate textual transcripts derived from radiologists' audio, images, and eye-gaze data, to visualize and map radiologists' attention onto specific regions images during diagnosis. This process establishes alignment between texts and images, facilitating model training. The detailed gaze data processing methods are described in Sec. Sec.3.1l Given that eye-gaze data tightly links textual and localized visual information, after obtaining auxiliary information from part B, we introduce eye-gaze guided refined alignment training strategies, as depicted in Parts C' and D of Fig. 2 Specifically, we introduce the optimization algorithm for eye-gaze guided fine-grained text-image similarity matrix in Part C Sec. 3.2 Finally, in Sec. 3.3 we present the algorithm for eye-gaze guided cross- cross-modality mapping.
3.1 Multi-modal Data Processing
With the development of data collection technologies such as eye-tracking and speech recognition, has become possible to collect and process multi-modal interaction data of radiologists during the diagnostic process. In this work, we utilize MIMIC-EYE of 3689 images extracted from the MIMIC datasets
[4 datasets as our training set, consisting
19
20
18 17] Each sample is accompanied
3

---

Image and Text Feature Extraction
Eye-gaze Guided Fine-gr Fine-grained Aignment
Sent.
Sent.m
Text Encoder
Er
Row Max
wio gaze
Sentence Feature
geze
Potch Sent
Simillriy Batch
edllas ogti kuy bus
Diagnostic Text
Contrastive
Loss
ga2n
Image Encoder
Column Max
Katt-Patch Simuluitty Butch
oge goan
Image
Patch Feature
Sent- Patch Similanity
D: Eye-gaze Guided Cross moddlity Mapping
Eye-gaze Feature Extraction
MLCE Loss
Weighted
Repeat & Weighted Som
Scatence Embbddlings
EGM
Patched
Rinarize
Coos mapppag
Flatten
Normalizod
A Aignment Wenglts
Sentannce
Sm Kolkoer
bustemn
Patch Embecdings
Gaze Healmap of Sentence
Gaze Gaze-guide Similarity
Gaze-guided
Label
EGM
Figure 2: The framework of EGMA. After images and text are processed by the encoder in Part A. patch feature and sentence feature representations are obtained, resulting fine fine- seeegga similarity matrix for instances. Subsequently, the two types of eye-gaze see-gaze-based based auxiliary information obtained Part B are used for fine-grained and mrapsig alignment in Part and Part D, respectively.
by corresponding eye-trate tre-tracking data and transcripts text. These eye-tracking data provided by
the publicly available EYE GAZE [22] and REFLACX [31] datasets on PhysioNet [12]. Since each modality synchronized, the audio data is aligned with the eye- aye-gaze data in time. By segmenting the audio based on the time before and after the pronunciation of each word, we can align the transcripts with the audio, thereby aligning sentence-level text with eye-gaze data. Subsequently, attention heatmap based on eye-gaze data and images to represent the image regions the radiologist focuses Through the aforementioned data processing steps, achieve precise alignment between sentence-level text and image regions. Detailed processing method of eye-gaze and audio transcripts can be found at Appendix D
generate
3.2 Eye-gaze Guided Fine-grained Alignment
The core idea of contrastive learning is bring the features of related samples closer while pushing away the features unrelated samples. During the training progress of CLIP [4) model, assuming
batch size of b and input data { xx{ (k =1.... ,b) representing image-text pairs, global features
=f E, (x{{) E IR and 27 (x e R Ixd are obtained through image encoder E, and text encoder Er. Subsequently, the cosine similarity sk computed, with the following formula:
J2T' and s between the two modalities is
J2T %.l
= cossim(sk) 2f), THI cossim(zf 1515b
(I)
I2T is the image-to-text similarity, T2l is the text-to-image similarity, and I is the index number the another modality. Then, the image-to-text contrastive loss 1 for æ/ and text-to-image contrastive loss L for #x can be formulated as:
where cossim is cosine similarity. sg.l
1 I (art, (at) )
1 log expls 12T /T)
Exerp(s /7)) L 2 (o. (
1 log Eeerp(( /T)) (2)
exp( 8 P /T)
where 7 is a learned temperature. above, both the image and text utilize global-level features, while the auxiliary information generated from eye-gaze data emphasizes the local-level features between modalities. Therefore, based on [57].
worth noting that the calculation the loss mentioned
replace instance feature z/ and of with pm gnxd and 5t pmxd where P1 < n)

---

is the i-th patch feature of z and ( و m) is the j-th sentence feature of 2 and n, m are the image patch number and the sentence number of report. Then we calculate the similarities of sentence-to-patch ok S2P E R and patch-to-sentence of22 E Raxm in one instance:
cossim(s) Pi), "k
P28 cossim P S.
For each heatmap corresponding to a sentence, we initially divide it into patches similar to the image. Subsequently, concatenate the heatmaps of sentences obtain the Gaze-guided Similarity matrix GS2 for input { x (as illustrated in Fig. B). In this matrix, non-zero elements indicate the semantic correlation between the corresponding sentences and image patches. Thus, we binarize GSk, setting non-zero regions to 1, resulting in the Gaze-guided Label matrix GLk- After this step, we compute the multi-label cross-entree cross-entropy (MLCE) loss for z S22 and x P25 completing the optimization for fine- ame-grained alignment between positive sample pairs { Th f. as follows:
fL = mlce( S2P, GL,), fL = mlce( P25 )F
(4)
where mlce is the multi-label cross-entropy loss. Subsequently, we calculate the fine-grained features &! and # as follows:
mari Jol, 2 m
S2P mar [
$2 )s+]
(5)
12
Then, replace the z = with the updated # 2 in Eq. Finally, the fine-grained image-to-text
loss _ and text-to-image loss L are computed based on Eq The formula for our Eye-gaze
Guided Fine-grained (EGF) alignment loss is as follows:
1 b CLI
1 LEGF
CULE S2P + +LRR
+ 2(1E" 1
(6)
2b
3.3 Eye-gaze Guided Cross-modality Mapping
In the previous section, we replaced the global instance logits in the traditional batch clip loss with fine-grained instance logits that consider local features and optimized the alignment between these local features using gaze information. The text our work recorded by radiologists while observing images, implying a close semantic relationship between the focus region and the corresponding text. To further optimize the alignment between modalities, we continue to incorporate eye-gaze data assistance into the cross-modality mapping process. In this work, we first utilize matrices GSk, x r2s and æ S2 to generate the image-to-text and text-to-image alignment weight matrix W/2T E 1 and WTT E mm The calculation formula is as follows:
W 12T = norm(w(a: P28) + CS*), W = norm(u(u #2P + )
(7)
where norm is normalization and w consists of sparse and binarize operations. After obtaining the weight matrix, we perform the mapping from text features S to image features Cross_Pt e Rax and from image features P. to text features Cross_S E IRd according to the following formula:
Cross_PP P
! Cross_S{ St.
P
W T2
(8)
where i [1, n/| is the i-th patch feature of Ph and j E [1, m] is the j-th sentence feature of s Subsequently, we use the mapped features along with the target features as inputs to compute the alignment contrastive loss defined in Eq. 2 obtaining the image mapping loss mLJ. and the text mapping loss ml.-. The formula for our Eye-gaze Guided cross-model Mapping (EGM) loss is as follows:
1 LEGM = - Immt + mLI))
Finally, the total loss of our model within a batch is process, considering the proportion of eye-gaze data, batches may contain both types of data. When encountering samples without eye-gaze data, the EGF module does not compute the loss from Eq. and the weight matrix in the Eq. of EGM module also excludes the GS
LEGF + LEGM- In our training
5

---

4 Experiments
In this study, we first conduct supervised and zero-shot classification as well as zero-shot retrieval experiments in Sec. 4.1 to validate the model's generalization performance and its representation capability of multi-modal features. Then, in Sec. 4.2 we perform ablation studies on various modules of EGMA. Additionally, to further investigate the auxiliary effect of eye-gaze data, we compare the performance when guided by different amounts of eye- data. Finally, in Sec. [43] we visualize the model's feature representations and the learned image-text relationships, further demonstrating the model's performance and interpretability.
4.1 Comparison with State-of-the-Arts
Image Classification We conduct supervised classification experiments on the CheXpert [16]. RSNA 44 and SIIM-ACR (45] datasets. CheXpert [[6] is a large-scale public dataset for chest
radiograph interpretation, comprises 224,316 chest radiographic images. Following (53), we utilize the official training split as our training set, and the official validation set of 202 images with expert-label as our test set. RSNA [4 is a comprehensive dataset for Pneumonia diagnosing. contains 39,700 chest X-ray images categorized into normal and pneumonia positive category.
We follow 153 to divide the data into 70% for training, 15% for validation, and 15% for testing. 45 is a chest dataset used for pneumothorax diagnosing, It consists of 2379 images with pneumothorax and 8300 images without pneumothorax. In this work, we utilize a subset defined in 43 as our test set, with the remaining data used for training and validation. More details of dataset can be found in the Appendix C
SIIM-ACR
In the supervised classification experiments, we adopt the linear classification settings 15 where the pre-trained image encoder is frozen, and only a randomly initialized linear classification head is trained. We adopt area under ROC curve (AUROC) metric to evaluate all model's performance. And for better validate the model's efficiency, we test its performance using 1%, 10%, and 100% of the training set. As shown in Tab. [ our model achieved the best results compared to other models. Additionally, with only 1% of the training set, our model outperformed the second-best model by 1.11%. 0.8%, and 1.94% on the CheXpert, RSNA, and SIIM-ACR datasets, respectively. Moreover, as the amount of training data increased, the model's 's performance improved significantly. This demonstrates that, with the assistance of radiologists' eye-gaze data, model possesses strong multi-modal feature representation capabilities.
Table 1: Comparison results of supervised classification task with other SOTA models on CheXpert, RSNA, and SIIM-ACR datasets. Area under ROC curve (AUROC) is reported with different portions of training data: 1%, 10%, 100%. Red and blue denote the best and second-best results.
CheXpert [6]
RSNA 44
SIR
45
Method
1%
10%
100%
1%
10%
100%
1%
10%
100%
CIRT
59
85.90
86.80
87.30
77.40
80.10
88.60
BioViL. 2
81.95
85.37
88.62
81.76
85.68
88.64
80.26
82.79
90.51
MedKLIP 556
87.31
87.99
89.31
85.27
90.71
91.88
MGCA [53
85,80
87.66
89.30
85.22
87.54
89.24
86.12
89.66
92.16
GLoRIA [15
86.60
87.80
88.10
86.10
88.00
88.60
PRIOR [S]
86.16
87.08
89.08
86.72
88.07
89.19
88.35
89.72
92.49
McdCLIP [55]
85.74
87,49
88.02
87.61
88.19
89.10
88.84
91.13
92.18
EGMA(Ours)
87.71
88.92
89.50
88.41
89.40
90.10
90.78
92.17
93.29
We further conduct zero-shot classification tasks on the CheXpert5x200 [15]. RSNA [44], and SIIM-ACR (45) datasets. CheXpert5x200 includes five common chest diseases, Atelectasis, Cardiomegaly. Consolidation, Edemo, and Pleural fff each with 200 chest X-rays. It important to note that the CheXpert training set does not include any data from CheXpert5x200, so there is no data leakage issue. The test sets for RSNA and SIIM-ACR are the same as those used in the supervised classification task. All text prompts are provided by professional radiologist [15] During testing, we calculated the similarity between image features and text prompt features for all diseases, with the highest similarity indicating the predicted category. As shown in Tab. 2 CLIP 42
performs poorly on medical images due to its training data primarily consisting of natural images. The

---

Table Comparison results of zero-shot classification tasks with other SOTA models on CheXpert 5x200, RSNA, and SIIM-ACR datasets. The Accuracy (Acc.) and Fl-score (FI) metrics are reported. Red and blue denote the best and second-best results.
CheXpert 5x200
15
RSNA
44
SIIM-ACR
45
Method
Acc.1
FI
Acc.t
FIf
Acc.1
FIf
CLIP
20.10
9.12
25.03
22.07
49.39
47.98
GLoRIA
15
53.30
48.99
29.15
28.54
22.57
22.57
PRIOR S
34,90
30,56
76.77
51.80
50.00
33.33
MGCA
53
43,60
41.37
60.83
57.77
30.03
25,45
MedCLIP [55
57.50
55.97
43.09
31.01
58.40
57.85
EGMA(Ours)
61.30
60.38
76.97
43.49
63.62
61.46
models in rows two to five use encoders pre-trained on medical datasets, and thus, their performance better than that of CLIP. Interestingly, the GLoRIA [I5] and MGCA (53) perform worse than the CLIP model in diagnosing pneumonia on the SIIM-ACR dataset. This indicates that these models are significantly influenced by the data distribution, resulting in poor generalization performance. Conversely, our EGMA achieves the best results in all other metrics, except for the Fi-score on the RSNA dataset. This demonstrates that our model, enhanced by eye-gaze data, has learned more generalizable feature relationships between medical images and text, significantly improving its generalization performance.
Image-text Retrieval To further validate the alignment capability of our model between visual and tex- tual features, compare the zero-shot retrieval performance of EGMA with other models on CheX- pert 8x200 dataset [59]. Unlike CheXpert5x200 (15), CheXpert8x200 includes eight common chest diseases, No Finding, Cardiomegaly, Edema, Pneumonia, Atelectasis, Pneumothooaz, Pleural E fusion, and Fracture, each with 200 chest X-rays and five corresponding text prompts. is worth noting that the prompts for retrieval tasks are different from those for classification tasks in the previous section, but all are written by task, we first compute the similarity between the image and all candidate texts, and then rank the retrieved results. Similarly, in the text-to-imane task, we compute the similarity between the textual prompts and all images, and rank the retrieval results. We report Precision at Top-1. Top-5, and Top-10, 10, which reflect how many relevant examples are retrieved. As shown in Tab. 3 our model achieves the best results in both retrieve tasks. Our model outperforms the second-best model in the image-to-text and text-to-image retrieval tasks by 3.9%, 5.88%, and 4.33%, and 19.75%, 14.50% and 12% in terms of P@ P@5, and P@ metrics, respectively. This indicates that our model has fully learned the relationship between images and texts, achieving better alignment effects.
radiologists. In the image-to-tes to-text retrieval
Table 3: Comparison results of zero-shot retrieval task with other SOTA models on CheXpert 8x200 dataset. The Precision at 1op-1, Top-5, and Top-10 reported. Red and blue denote the best and second- results.
Image-to-text
Text-to-image
Method
P@ 1
P@5t
P@ 101
P@ 1
P@5t
P@ 101
CLIP
42
12.75
12.48
10.03
5.00
12.50
12.50
MedCLIP 5S
14.50
15.98
15.86
12.50
12.50
15.00
MGCA [33
35.00
27.80
23.33
45.00
47.50
44.00
GLoRIA [[
38.75
31.62
24.51
52.50
49.00
50.25
Con VIRT [59]
60.25
60.00
57.50
EGMA(Ours)
42.65
37.50
28.84
80.00
74.50
69.50
4.2 Ablation Study
To further validate the model's performance, we conducted ablation experiments on the proposed EGF and EGM modules, while also assessing the impact of the proportion of eye-gaze data on the model results. As shown the upper half of Tab. the first row represents our Baseline model, where we utilize the initialized weights pre- trainrd on CheXpert [16] and MIMIC-CXR 220] datasets [55). The
7

---

second row "MLCE" indicates that within our EGF module, the EGF loss is not further computed beyond the Eq. f instead, only the multi-label cross-entropy (MLCE) loss between the eye-gaze guided similarity matrix and the model's output similarity matrix is calculated. The third row "EGF" utilizes the Eye-gaze Guided Fine-graines loss described in Eq. 6 The fourth row "EGM" indicates that the model is trained solely through the Eye-gaze Guide cross-model Mapping method. Finally, the fifth row presents our proposed EGMA model, which integrates the aforementioned modules guided by eye-gaze data.
Table 4: Comparison results of zero-shot classification ablation experiments on CheXpert 5x200, RSNA, and SIIM-ACR datasets. The Accuracy (Acc.) and F1-score (F1) metrics are reported. Each value the lower part is the average of three runs. Red and blue denote the best and second-best results.
CheXpert5x200
16
RSNA
SIIM-ACR
Method
Acc."
F1
Acc.t
F17
Acc.t
FIf
Baseline
57.50
55.97
43.09
31.01
58.40
57.85
MLCE
60.90
59.59
47.06
33.04
27.43
22.81
EGF
60.30
58.44
53.81
35.52
63.54
65.70
EGM
59.30
57.74
54.68
35.80
52.61
47.85
Unirsed(Ours)
61.30
60.38
76.97
43.49
63.62
61,46
1% Gaze
58.93=0.006
56.62:10.05
40.38:0.01
29.75=10.01
57.90+:.22
57.37=00.24
5% Gaze
58.93+0006
56.69+-0.06
53.00+00.55
35.13++0001
59.20++0.11
58.51+0.10
10% Gaze
59.30=0001
57.82+00001
53.78+00.01
35.37+.001
58.27++0.11
57.71+0.122
50% Gaze
59.55=0.07
58.8410.01
58.54110.7
37.0110.20
61.4110.31
58.8810.22
In Tab. 4 it can be observed that the method using only gaze-guided MLCE loss significantly improves performance compared to the baseline on CheXpert 5x200 dataset, with a slight improvement on RSNA but severe decline SIIM-ACR dataset. However, models using EGF or EGM show significant improvements on SIM-ACR. This indicates that while MLCE improves performance on some datasets, it simultaneously reduces the model's generalization ability. Thus, relying solely on simple loss for similarity matrix is insufficient. In this work, by combining eye-gaze guided image- text relationships with fine-gra mme-graieed feature alignment (EGF), although the model's performance slightly decreases on CheXpert 5x200, its overall generalization improves. Similarly, to enhance the model's multi-modal alignment ability, introducing eye-gaze guided cross-modal mapping results in improved performance and generalization, with EGM achieving optimal performance on RSNA dataset. Finally, when optimizing both fine-grained alignment and cross-modal alignment using eye-gaze, the model achieves dominant performance on all three datasets, demonstrating further enhancement in generalization.
Numerous studies 24 54 35 36 have demonstrated that training models using eye-gaze data can achieve comparable performance to models trained with fine-grained manual annotations. Meanwhile, the cost of collecting fine-grained manual annotations is significantly higher than that of collecting eye-gaze data. Therefore, incorporating eye-gaze into pre-training tasks is a feasible approach to enhancing model performance. To further validate the efficiency of our model using eye-gaze data, conduct ablation experiments MIMIC-EYF, consists of a total of 3695 samples. We perform ablation experiments using 1%, 5%, 10%, and 50% of the eye-gaze data, resulting 37, 185, 370, and 1848 samples with prior information from radiologists, respectively. We repeat each experiment three times to eliminate the bias caused by random sampling, and report the average results. As shown the lower part of Tab, 4 the model's performance on the CheXpert 5x200 dataset improved when trained with 1% of eye-gaze data. However, due to the limited data volume, the model's 's performance on other datasets is inferior to the baseline. When increasing the eye-gaze data to 5%, the model shows significant improvements on all three datasets. With the continuous increase in eye-gaze data, the performance of the model also improves. Therefore, even with small amount of eye-gaze data (185 samples), our framework can effectively guide the model's multi-moddl processing capability, ensuring performance enhancement. This further illustrates the applicability of our model and its low training cost characteristics.
the proportion of it in the training set. Our training dataset,
8

---

CLIP
EGMA
(4)
(b)
Image
EGMA
MGCA
GLoRIA
Alelectasis
Cardoonegll
Conselidarion
Eiless
Pleural Fiflaiion
Figure 3: Results of cross-modality attention maps visualization. Related text content: (a) "heart size borderline enlarged"; (b) "increased bibasilar opacities are the combination of in- creased bilateral pleural effusions and bibasilar atelectasis".
Figure 4: t-SNE visualization on CheXpert 5x200 dataset by CLIP and our EGMA. The figures dis- play points of different colors representing vari- ous ground truth disease types and their cluster assignments. The color-coded points illustrate the clustering results of each algorithm.
4.3 Visualization
To better demonstrate the correspondence learned by the EGMA framework between text and radiographic images, we conducted cross-modality attention maps visualization Fig. 3 Guided by eye-gaze data, the EGMA framework clearly outperforms other state-of-the-ar methods in the field accurately localizing disease regions. Fig. 4 we visualize the feature representations of CLIP [2) and our EGMA model on images of CheXpert 5x200 dataset using the t-SNE (50]. It can be observed that our model exhibits better clustering representation. The CLIP model, which was not trained on medical data, is unable to effectively differentiate these diseases. More results of t-SNE visualization can be referred to the Appendix E clustering performance other SOTA also inferior to our EGMA.
methods 15 53
5 Discussion and Conclusion
In this work, we reveal the significant role of radiologists' eye-gaze data in multi-modal alignment and propose an Eye-gaze Guided Multi-modal Alignment framework called EGMA. In fact, the EGF and EGM modules in our EGMA framework are similar to the diagnostic process radiologists. Many studies [39 46 30 8 have pointed out that radiologists perform global search first and then conduct a detailed examination of the local areas when suspicious lesions are found. Firstly, the EGF module of EGMA corresponds to the global search by radiologists, aligning the local image patch features with the individual sentence features. Secondly, EGM uses the key values from the local similarity matrix computed in the first step as weights, focusing the alignment on certain important image patches and texts, akin to the detailed observation stage after identifying suspicious lesions. Thus, by mimicking the real diagnostic behavior of radiologists, our EGMA further enhances the model's multimodal processing capability and diagnostic accuracy. We evaluate EGMA's zero-shot capabilities and fine-tuned performances on multiple datasets and observe significant improvement in classification and retrieval tasks. Additionally, we investigate the impact of eye-gazz data scale on performance, finding that even small amounts of eye-gaze data can enhance the model's 's multi- modal alignment capabilities during pre-training. Overall, our EGMA framework explores the feasibility of incorporating eye-gaze data from radiologists to assist in multi-modal feature alignment during model training, laying the foundation for the application of eye-gaze data in the medical multi-modal domain.
Limitations and Discussion Our work only compared state-of-the-ar methods in classification and retrieval tasks, without conducting downstream tasks such lesion localization segmentation. Additionally, our model heavily relies on multi-modal datasets like MIMIC-EYE [14), which can simultaneously collect eye-gaze data, medical images, and diagnostic text. The scenarios for collecting these data are also significant consideration. For instance, in clinical ultrasound diagnosis [38]. radiologists often use both hands to operate the equipment and verbally communicate their diagnostic information to assistant. In this context, is convenient to simultaneously record ultrasound images, eye-gaze data, and audio. contrast, during chest X-ray diagnosis MIMIC-EYF, radiologists typically record diagnostic information directly in text form rather than verbally. Fortunately, some

---

are focusing on how to naturally collect during diagnosing. They have designed more flexible collection systems that better accommodate the routine work of radiologists, which is crucial for the widespread adoption of collecting multi-modal diagnostic data such as eye-gaze information.
recent efforts 22 35
data of radiologists
Potential Impacts Although the eyc-gaze data we used is publicly available and we have permission use it, some studies [27 [33 have indicated that private information such gender, age, and mental state of observers can be extracted from eye-gaze data. Therefore, privacy concerns have always been a focal point in using eye-gaze data. To address this, we recommend using de-identification methods to filter eye-gaze data or releasing the data in the form of heatmaps rather than the raw data.
Future Work In the future, will continue to optimize these proposed collection systems 27 and explore the guidance role of eye-gaze data between images and handwritten diagnostic reports to accelerate their application in real medical diagnostic scenarios. This will provide research foundation to alleviate data annotation pressure and enhance model interpretability. Additionally, we will continue to analysis the eye-gaze features, such as temporal features, and further optimize their role in multi-modal feature alignment. We believe this work can serve as a valuable reference for the application of eye-gaze data in multi-modal frameworks and promote its development in the field of medical multi-modality. Moreover, thanks to the inherent flexibility of the EGMA model, it is well- sulted for multi- multi-modal alignment tasks involving natural images, such as in human robott interaction and control as well as for education/tainiig purposes. Therefore, we believe this is promising direction for future expansion.
References
[1] Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint 3XXiv:904.03322 2019.
[2] Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez- Valle, et al. Making the most of text semantics to improve biomedical vision-language processing. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 1-21. Springer, 2022.
[3] Adrian P Brady. Error and discrepancy in radiology: inevitable avoidable? Insights into imaging, 1:12-1 2017.
[4] Wenting Chen, Xiang Li, Linlin Shen, and Yixuan Yuan. Fine-grained image-text alignment in medical imaging enables cyclic image-report generation. arXiv preprint urXiv:2312.08077 2023.
[5] Pujin Cheng, Li Lin, Junyan Lyu. Yijin Huang, Wenhan Luo, and Xiaoying Tang. Prior: Prototype representation joint learning from medical images and reports. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[6] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language- image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818 2829, 2023.
[7] T. Drew, K. Evans, M. L.-H. Vö, F. L. Jacobson, and J. M. Wolfe. Informatics in radiology: What can you see in a single glance and how might this guide visual search in medical images? RadioGraphics,
Jan 2013.
[8] Trafton Drew, Karla Evans, Melissa L-H Vö, Francine L Jacobson, and Jeremy M Wolfe. Informatics in radiology: what can you see in a single glance and how might this guide visual search in medical images? Radiographics,
((1))263-2744 2013.
[9] Andrew T Duchowski and Andrew T Duchowski. Eye tracking methodology: Theory and practice. Springer, 2017.
10

---

[10] Robert Geirhos, Jörn Hennike Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 (11):665-673) 2020.
[11] Nicholas A Giovinco, Steven M Sutton, John D Miller, Timothy M Rankin, Grant W Gonzalez, Bijan Najafi, and David G Armstrong. A passing glance? differences in eye tracking and gaze patterns between trainees and experts reading plain film bunion radiographs. The Journal of Foot and Ankle Surgery, (3(3):3823391, 2015.
[12] Ary Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanow, Roger G Mark, Joseph E Mietus, George B Moody, Chung- Kang Peng, and H Eugene Stanley, Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23):e215-e220 2000.
[13] David M Hansell, Alexander A Bankier, Heber MacMahon, Theresa C McLudd, Nestor L Muller, and Jacques Remy. Fleischner society: glossary of terms for thoracic imaging. Radiology, 246(3):697-722, 2008
[14] Chihcheng Hsieh, Chun Ouyang, Jacinto C Nascimento, Joao Pereira, Joaquim Jorge, and Catarina Moreira. Mimic-eye: Integrating mimic datasets with reflacx and eye gaze for multimodal deep learning applications. PhysioNer (version 1.0. 0), 2023.
[15] Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942-3951, 2021.
[16] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ileus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 590-597, 2019.
[17] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Leo Anthony Celi, Roger Mark, and Steven Horng Mimic-iv-ed (version 1.0). PhysioNet, 2021.
[18] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv. PhysioNet. Available online at: https:/pphysione.. argleomtee/nimiii//.//acssss August 23, 2021), 2020.
[19] Alistair Johnson, Tom Pollard, Roger Mark, Seth Berkowitz, and Steven Horng. Mimic-exr database. PhysioNet10, 13026:C2JTIQ, 2019.
[20] Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew Lungren, Chih- ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. Mimic-cxr-jpg. large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042. 2019.
[21] A. Karargyris et al. Creation and validation of a chest x-ray dataset with eye-tracking and report dictation for ai development. Sei. Data, 8(1):1-18, Mar 2021.
[22] Alexandros Karargyris, Satyananda Kashyap, Ismini Lourentzou, Joy Wu, Matthew Tong, Arjun Sharma, Shafiq Abedin, David Beymer, Vandana Mukherjee, Elizabeth Krupinski, et al. Eye gaze data for chest x-rays. PhysioNet https://doi. org/10. 20g/100.1002/0QFDZZ2R87,, 2020.
[23] Christina Katsini, Yasmeen Abdrabou, George E Raptis, Mohamed Khamis, and Florian Alt. The role of eye gaze in security and privacy applications: Survey and future hei research directions. In Proceedings of the 2020 CHI Conference Human Factors in Computing Systems, pages 1-21, 2020.
[24] N. Khosravan et al. Gaze2segment: A pilot study for integrating eye-tracking technology into medical image segmentation. Medical Computer Vision and Bayesian and Graphical Models for Biomedical Imaging, pages 94-104, Cham, Switzerland, 2016. Springer.
11

---

[25] Naji Khosravan, Haydar Celik, Baris Turkbey, Elizabeth C Jones, Bradford Wood, and Ulas Bagei. A collaborative computer aided diagnosis (c-cad) system with eye-tracking, sparse attentional model, and deep learning. Medical image analysis, 51:101-115, 2019.
[26] E. M. Kok and H. Jarodzka. Before your very eyes: The value and limitations of eye tracking in medical education. Med. Educ., ((():114-1222 Jan 2017.
[27] Jacob Leon Kröger, Otto Hans-Martin Lutz, and Florian Müller. What does your gaze reveal about you? on the privacy implications of eye tracking. In IFIP International Summer School on Privacy and Identity Management, pages 226-2226222222 241.241..25 Springer, 2020.
[28] E. A. Krupinski. Current perspectives in medical image perception. Attention, Perception, & Psychophysics, 72(5):1205 1217, Jul 2010.
[29] H. L. Kundel, C. F. Nodine, E. F. Conant, and S. P. Weinstein. Holistic component of image perception in mammogram interpretation: Gaze-tracking study, Radiology, 242(2):396 402, Feb 2007.
[30] Harold L Kundel, Calvin F Nodine, Emily F Conant, and Susan P Weinstein. Holistic component of image perception in mammogram interpretation: gaze-tracking study. Radiology, 242(2):396- 402, 2007.
[31] Ricardo Bigolin Lanfredi, Mingyuan Zhang, William Auffermann, Jessica Chan, Phuong-Anh Duong, Vivek Srikumar, Trafton Drew, Joyce Schroeder, and Tolga Tasdizen. Reflacx: Reports and eye-tracking data for localization of abnormalities in chest x-rays, 2021.
[32] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975, 2022.
[33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[34] Jiasen La, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for aision-and-languuge tasks. In Advances in Neural Information Processing Systems, pages 2019.
[35] C. Ma et al. Eye-gaze-guided vision transformer for rectifying shortcut learning. IEEE Transactions on Medical Imaging, 42 (2(11):3384-339 3394,334, Nov 2023.
[36] Chong Ma, Lin Zhao, Yuzhong Chen, Lei Guo, Tuo Zhang, Xintao Hu, Dinggang Shen, Xi Jiang, and Tianming Liu. Rectify vit shortcut learning by visual saliency. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[37] S. Mall, E. A. Krupinski, and C. R. Tello-Thoms. Missed cancer and visual search of mammo- grams: What feature based machine-learning can tell us that deep-convolution learning cannot. In Proc, SPIE, pages 281 2877 Mar 2019.
[38] Q Men, Teng, L Drukker, et al. Gaze-probe joint guidance with multi-task learning in obstetric ultrasound scanning. Medical Image Analysis, 10:102 2023.
[39] Calvin F Nodine and Harold L Kundel. Using eye movements to study visual search and to 1987.
improve tumor detection. Rudiographics,
[40] Calvin F Nodine, Harold L. Kundel, Sherri C Lauver, and Lawrence C Toto. Nature of expertise in searching mammograms for breast masses. Academic radiology, 3(12); 10000 1996.
[41] Marcus Nyström and Kenneth Holmqvist. adaptive algorithm for fixation, saccade, and glissade detection in eyetracking data. Behavior research methods, 42(1):188-22) 204, 2010.
12

---

[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 1748-8763. PMLR, 2021.
[43] Khaled Saab, Sarah M. Hooper, Nimit Sharad Sohoni, Jupinder Parmar, Brian P Pogatchnik, Sen Wu, Jared Dunnmon, Hongyang Zhang, Daniel I.. Rubin, and Christopher Ré. Observational supervision for medical image classification using gaze data. In Medical Image Computing and Computer-Asissted Intervention, 2021.
[44] George Shih, Carol C Wu, Safwan S Halabi, Marc D Kohli, Luciano M Prevedello, Tessa Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya Galperin-Aizenber et al. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiology: Artificial Intelligence, 1(1), 2019.
[45] SIIM-ACR pneumothorax segmentation, 2020. [online] Available: https://wwwi kaggle com/ hemm/c/siimmeen acr ppeuunothhora--ssggaattaaio
[46] Richard G Swensson. A two-stage detection model applied to skilled visual search by radiolo- gists. Perception & Psychophysics, 27(()) 1980.
[47] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) Association for Computational Linguistics, 2019.
[48] Georgia Tourassi, Sophie Voisin, Vincent Paquit, and Elizabeth Krupinski. Investigating the link between radiologists' gaze, diagnostic decision, and image content. Journal of the American Medical Informatics Association, 20(6): 1067-1075. 2013.
[49] A Van der Gijp, CJ Ravesloot, H Jarodzka, MF Van der Schaaf, IC Van der Schaaf, Jan PJ van Schaik, and Th Ten Cate. How visual search relates to visual diagnostic performance: a narrative systematic review of eye- eye-tracking research in radiology. Advances in Health Sciences Education, 22:765-787 2017.
[50] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] Sophie Voisin, Frank Pinto, Songhua Xu, Garnetta Moron-Ducote, Kathy Hudson, and Georgia D Tourassi. Investigating the association of eye gaze pattern and diagnostic error in mammog- raphy. In Medical Imaging 2013: Image Perception, Observer Performance, and Technology Assessment, volume 8673, page 867302. SPIE, 2013.
[53] Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti, and Lequan Yu. Multi- granularity cross-modal alignment for generalized medical visual representation learning. arXiv preprint arXiv:22 2022.
[54] S. Wang, X. Ouyang, T. Liu, Q. Wang, and D. Shen. Follow my eye: Using gaze to supervise computer-aided diagnosis. IEEE Trans. Med. Imag. 1187)11688-169, Jul 2022.
[55] Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163. 2022.
[56] Chaoyi Wu, Xiaoman Zhang. Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-image pre-training. medRxiv, pages 2023-01, 2023.
[57] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training, In International Conference on Learning Representattinss 2021.
13

---

[58] Ke Zhang, Yan Yang, Jun Yu, Hanliang Jiang. Jianping Fan, Qingming Huang, and Weidong Han. Multi-task paired masking with alignment modeling for medical vision-lange -aiiugl pre-training. IEFF Transactions on Multieeia,, 2023.
[59] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pages 25. PMLR, 2022.
[60] Zihao Zhao, Yuxiao Liu, Han Wu, Yonghao Li, Sheng Wang, Lin Teng, Disheng Liu, Xiang Li. Zhiming Cui, Qian Wang, et al. Clip in medical imaging: A comprehensive survey, arXiv preprint (xX1v:2312.07353, 2023.
[61] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-basee language- image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-16803, 2022.
14

---

A Appendix
This appendix organized as follows. In Sec. B we provide more experimental settings, including training parameters, detailed parameters of image and text encoders. In Sec. C we introduce the detailed information of the datasets used in this work. In Sec. D we provide more details of multi-modal data processing of MIMIC-EYE [14] dataset. In Sec. E we provide additional visualization results of feature representation. In Sec. F we provide additional experiments of zero-shot classification task after continue pre- vre-training using the backbones of other SOTA models in our EGMA framework.
B Experimental Details
B.1 Image/Text Encoder
In this study, we use SwinTransformer 33 as the image encoder, BioClinicalBERT encoder. Specifically, we use a 4 stages Swin Twannfraasormer, including 2, 2, 6, and 2 wwwiirraasformee blocks. Other parameters are: patch size 4; window size And we use layers BioClinicalBERT with 12 attention heads. In our EGMA framework, we add linear projection layer after both the image encoder and text encoder to map the embeddings dimension to 512, and we use a learnable temperature in contrastive loss calculation initialized on 0.07.
the text
B.2 Training Settings
Pre-training Settings In the pre-training process, we utilize the following image augmentations to the chest X-ray images: scale images to 224 224; color jittering with brightness and contrast ratios from [0.8, 1.2]: randomly change the contrastprrobubilly, = 0.5). And we train our model with 50 epochs with an initial learning rate 10 and weight decay 1 10 and 10 epochs of warm-up.
Fine-tuning Settings In the supervised classification experiments, adopt the linear classification settings [55, where the pre-trained image encoder is frozen, and only randomly initialized linear classification head is trained. We choose the same image augmentations the above pre-training settings. And we fine-tune our model with 30 epochs with an initial learning rate 5 x 10-7 and weight decay 1 x 10- and 6 epochs of warm-up. And all our training tasks are completed four RTX 3090 GPUs.
C Dataset Descriptions
C.1 MIMIC-EYE
The MIMIC-EYE 14 dataset includes comprehensive range of patient information, including medical images and reports, clinical data, patient's hospital journey, and eye-tracking data and audio of radiologists during diagnosis. The dataset comprises total of 3689 images from the MIMIC- IV v1.0 dataset [18]. each accompanied by transcripts text from audio and eye-tracking data of radiologists. this work, we use this dataset as our training set.
C.2 CheXpert
CheXpert [16] is large-scale public dataset for chest radiograph interpretation, developed by team from Stanford University. The dataset comprises 224,316 chest radiographic images involving 65,240 patients, annotated for the presence of 14 common chest radiographic findings [13] These annotations are categorized into three types: positive, negative, uncertain. In our study, we follow [5] and [59], using two subsets of this dataset, namely CheXpert 5x200 and CheXpert 8x200. for our zero-shot classification and zero-shot retrieval testing tasks. The CheXpert 5x200 dataset [[5] comprises five common chest diseases, Aleleclasis, Cardiornegaly, Consolidation, Edema, and Pleural E.J. f /usion, each with 200 chest X-rays. In [15), radiologist provided possible sub-types, severities, and locations for these five diseases. As depicted in Tab. []] all combinations of these three types of information form the text queries for CheXpert 5x200 dataset. In the zero-shot classification
15

---

Table 5: Examples of possible sub-types, severities, and locations provided by the radiologist in CheXpert 5x200 dataset.
Atelectasis
Consolidation
Pleural Effusion
mild
increased improved apperance of
small stable
severity
minimal
subsegmental atelectasis linear atelectasis trace atelectasis
bilateral consolidation reticular consolidation patchy consolidation airspace consolidation partial consolidation
bilateral pleural effusion subpulmonic pleural effusion bilateral pleural effusion
subtype
bibasilar atelectasis retrocardiac atelectasis bandlike atelectasis
at the mid lung zone at the upper lung zone at the right lung zone at the left lung zone at the lung bases
at the lower lung zone at the upper lung zone at the left lower lobe at the right lower lobe at the left upper lobe
left
right
location
tiny
task, image embeddings are compared with the embeddings of these text queries, and the class with the highest similarity is assigned as the predicted classification for the image. The CheXpert 8x200 dataset (59) comprises eight categories, Nofiiding,, Cardiomegaly, Edema, Pneumonia, Alelectasis, Pneumothorar, Pleural Ej fusion, and Fracture, each with 200 images. In [59], a radiologist expert was also invited to compose five expert queries for each category, used for image-text retrieval tasks. Specific queries are detailed in Tab.
C.3
RSNA
The RSNA Pneumonia Detection Dataset [44], encompasses a comprehensive set of medical imaging data types, including X-rays, CT (Computed Tomography), and MRI (Magnetic Resonance Imaging) images. In this work, we utilized the stage 2 version of this dataset, comprising 29,700 700 chest X-ray images categorized into normal and pncumonia positive category. Following (15), 15% this dataset for our zero-shot classification testing set. And we utilize the text queries from the "no finding" and "Pneumonia" categories in the CheXpert 8x200 dataset as the text queries for zero-shot classification in this data.
allocated
C.4 SIMMACR
The SIIM-ACR 45 dataset is a chest dataset used for pnewmothorar classification and segmenta- tion. It consists of 2379 images with pneumothorax and 8300 images without pneumothorax. this study, we utilized subset of the dataset filtered by Saab et al. [43] the test data to evaluate the zero-shot classification performance of the model for pneumothorax disease. And we utilize the text queries from the "no finding" and "Pneumothorax" categories in the CheXpert 8x200 dataset the text queries for zero-shot classification in this data.
D Details of Multi-modal Data Processing
As illustrated in Fig. 5 the presentation of multi-modal data in the MIMIC-EYE 14 dataset includes radiologists' audio, text transcript, eye-gaze data, and image. Since each modality is synchronized, the audio data aligned with the eye- eye-gaze data in time. By segmenting the audio based on the time before and after the pronunciation of each word, we can align the transcripts with the audio, thereby aligning word-level text with eye-gaze data. Subsequently, we generate attention heatmap based on eye-gaze data and images to represent the image regions the radiologist focuses on. Through the aforementioned data processing steps, we achieve precise alignment between word-level text and image regions. It is noteworthy that due to the rapid speech rate of radiologists, there may be
16

---

Table Examples of text queries for different categories in the CheXpert 8x200 dataset.
Categories
Text Query
The lungs are clear.
No abnormalities are present.
No Finding
The chest is normal.
No clinically significant radiographic abormalities.
No radiographically visible abnormalities in the chest.
The heart is mildly enlarged.
Cardiomegaly present.
Cardiomegaly
The heart shadow is enlarged.
The cardiac silhouette is enlarged.
Cardiac enlargement is seen.
Mild interstitial pulmonary edema is present.
The presence of hazy opacity suggests interstitial pulmonary edema.
Edema
Moderate alveolar edema is present.
Mild diffuse opacity likely represents pulmonary edema.
Cardiogenic edema likely is present.
A consolidation at the base likely represents pneumonia. Pneumonia is present.
Pneumonia
The presence of air bronchograms suggest pneumonia.
A fluffy opacity suggests pneumonia.
A pulmonary opacity with defined borders likely represents pneumonia.
Platelike opacity likely represents atelectasis.
Geometric opacity likely represents atelectasis.
Atelectasis
Atelectasis present.
Basilar opacity and volume loss is likely due to atelectasis.
Patchy atelectasis is seen.
An apical pneumothorax is present.
basilar pneumothorax seen.
Pneumothorax
A medial pneumothorax is present adjacent to the heart. lateral pleural line suggests pneumothorax.
Pleural air is present.
A pleural effusion is present.
Blunting of the costophrenic angles represents pleural effusions.
Pleural Effusion
Trace plcural fluid is present.
The pleural space is partially filled with fluid.
Layering pleural effusions are present.
An angulated fracture present.
An oblique radiolucent line suggests a fracture.
Fracture
cortical step off indicates the presence of fracture.
A communuted displaced fracture is present.
fracture present.
17

---

Timeline
Audio
Transcript
hyper-inflated
lungs
with
prominent
interstitial
markings.
Gaze Point
Heatmap
Word-level
Sentence- level
Figure 5: The generation methods for heatmap both word-level and sentence-level.
available eye-gaze data within the time interval corresponding to single word. Fig. 5 the word "with" in the transcript has no corresponding gaze data. Another common and unavoidable issue is the loss of eye-gaze data caused by blinking and intense head movement of radiologist, last two words of the transcript. Due to these technical challenges, achieving perfect pairing between words and image regions is difficult. However, as shown the right side of Fig. [ adjusting the text to the sentence level largely mitigates the issue of missing word-level heatmap (Heatmap with red edge), and the semantic information of the entire sentence also encompasses the information of each word. Therefore, in this work, we process text features at the sentence level.
seen in the
During the pre-training of EGMA, the size of the input heatmap is determined by the number of patches in the image. For example, after the image is processed by the image encoder, the size of image embedding 196 768, where 196 represents the number of image patches. Therefore, resize the heatmap directly to 14 x 14 to match the image embedding and further process it into the Gaze-guided Similarity and Gaze-guided Label mentioned in the main manuscript.
D.1 Eye-gaze Data Denoising
In fact, noise and errors in eye-gaze data common as noise in images, and these can all affect the model's final performance. this work, the errors in eye-gaze data primarily stem from two factors: involuntary saccades and microsaccades of the radiologists" eyes 19, and subjective fixation errors 3
Human eye movements can be categorized into two main types: saccades and fixations (9] Saccades the rapid movements between different areas of the visual field and are generally considered noise data that do not involve cognitive processes. Fixations, on the other hand, are brief pauses of the eyes on small area and are regarded as the primary cognitive behavior. Additionally, due to the structure of the eye, fixations are accompanied by microsaccades, causing the fixation point to drift within the fixation area. Thus, microsaccades are also major source noise in eyc-gaze data. Fortunately, eye-tracking technology has evolved significantly over the past century, and noise reduction techniques for eye-gaze data have become highly advanced. Currently, all commercial eye-tracking devices come with preprocessing software that uses adaptive methods (c.g.. 41 filter out noise and output fixations, greatly facilitating the use of eye-tracking technology. Fig. 6 shows example of MIMIC-EYE data used in our work. All gaze data is denoised by the filter operation, ensuring that noise from blinks, saccades, and microsaccades the radiologists has been eliminated.
Additionally, due to variations in radiologists' expertise and cognitive levels, fixation data may contain errors, such the reviewer mentioned, "occasionally shift to locations not exactly on the objects."
18

---

A daptive Filter
Raw Gaze
(d) Processed Gaze
(b) Raw Gaze Heatmap
(c) Speed of Raw Gaze Points
t
(e) Processed Gaze
Healmap
Figure 6: Attention heatmap (b) generated from raw gaze data (a) is susceptible to noise. The adaptive filter employed the preprocessing step of this work removes noisy data (saccades and microsaccades) based on characteristics such as the speed of gaze points (c), resulting in more accurate fixation data (d) and heatmap (e).
For the minor shift error of fixations, where the duration of incorrect fixations is small compared to the overall duration, their values in the attention heatmap are low, thus having minimal impact on the mlce loss utilized in work (Eq. in main paper). In our EGMA, EGF module (Sec. 3.2 in main paper) selects the highest similarity value local features before averaging, ensuring that these minor errors do not affect the model. Additionally, in the EGM module (Sec. 33 in main paper), the sparse operation for alignment weight also filters out minor errors in the heatmap. For the significant fixation errors, MIMIC-EYE data has been cleaned to address this issue the first time. Specifically, during data collection, radiologists were required to provide a diagnostic label. These data were then reviewed by professionals to exclude inconsistent diagnoses and fixation errors. Moreover, the MIMIC-EYE E dataset includes eye-gaze data from six radiologists. Even if one or two radiologists' fixations for specific diagnostic text incorrect, the correct fixation data from the other radiologists under the same semantic context can mitigate the adverse effects. Fig. shows an example of this compensation.
(a) Inacuuate gaze
(b) The accurate gaze of rachloggss
Figure Inaccurate eye-gaze data of one radiologist (a) in the heart region and several correct eye-gaze data (b) of other radiologists in the same region that compensate for this error, which are included the dataset used in this work.
In summary, EGMA was designed with considerations for the noise and errors inherent in eye-gaze data from the beginning. And effectively identifying and mitigating errors radiologists' diagnostic behavior during model training will be focus of our future work.
D.2 Eye-gaze Data Properties
Numerous studies have investigated and proven the close relationship between radiologists' gaze behavior, image content, and diagnostic results. For example, studies [40] 52
49 48 have found
19

---

that radiologists fixate more on diseased areas than on healthy ones. Moreover, they discovered that novice radiologists, due to their lack of experience. repeatedly look at diseased areas, resulting in even more fixations in these areas than those of expert radiologists. Fig. shows some comparison cases. Additionally, the abnormal regions with more fixations have higher weights in the attention heatmap. Our EGMA model can leverage these weights, along with image content and text, to learn better disease diagnosis capabilities and feature representations.
8
(8) Cartiomegaly
(b) Normal Heart:
(c) Pneumonia
(d) Clear Lungs
Figure 8: Comparison of eye-gaze data in normal and abnormal cases. For the heart region, there more fixations on disease area (a) compared normal heart (b). For the lung region, fixations on disease area (c) are more concentrated, whereas fixations normal lungs more dispersed.
E
Additional Visualization Results
CLIP
GLoRIA
MGCA
EGMA
telectass
Cardiomegaly
Consolidation
Edema
Pleural Effusion
Figure 9: Visualization of feature representation of CheXpert 5x200 dataset by CLIP, GLORIA,
MGCA, and our EGMA.
In Fig. we visualize the feature representations CLIP 42 GLoRIA 13
53 and our EGMA model images of CheXpert 5x200 dataset using the t-SNE [50 It can be observed that model exhibits better clustering representation. The CLIP model. which was not trained on medical data, is unable effectively differentiate these diseases. Additionally, while the representation capability of GLoRIA and MGCA has improved noticeably, their clustering performance still inferior our EGMA.
MGCA
Additional Analysis Results
To further validate the generality of our framework, we utilize the encoders and pre-training weights provided by CLIP 42), GLoRIA [15), and MGCA [53] in our EGMA framework. Subsequently. we continue training the MIMIC-EYE [1] dataset and present the zero-shot classification results the CheXpert 5x200 [66,, RSNA A4 and SIIM-ACR 45) datasets Tab.
We present accuracy Accuracy and F1 score metries on three datasets. The values in parentheses indicate the improvement over the baseline metrics (as shown in Tab. ( It can be observed that, all models show improvement after training with our EGMA framework, except for the decrease metrics for trained MGCA model on the RSNA dataset. For MGCA, when tested with its provided pre-trained weights, it performs the best F1-score on the RSNA dataset (as shown in Tab. 2 after training with the EGMA framework, performance improves on CheXpert 5x200 and SIIM but decreases RSNA dataset. This may reflect that the features extracted by MGCA
20

---

Table 7: Comparison results of zero-shot classification after continue pre- pre-training using the backbones of other SOTA models in our EGMA framework. Red and blue denote the best and second-bes results. The values (parentheses) represents the improvement over the baseline metrics in Table of main manuscript.
CheXpert 5x200 [6
RSNA 44
SIIM-ACR
Method
Acc.t
FIf
Acc.1
FIf
Acc.t
FI1
CLIP [42
20.30(0.2)
10.73(1.61)
34,04(9.01)
33.68(11.61)
50.19(0.8)
49.03(1.05)
GLoRIA 1.5
14,40(1.1) 49.31(0.32)
49.11(19.96)
38.82(7.81)
31.07(8.5)
31.10(8.53)
MGCA 53
50.20X6.6) (8.29(6.92)
57.08(-3.75)
40.40(-17.37)
32.65(2.62)
27.78(2.33)
EGMA(Ours)
61.30
60.38
76.97
43.49
63.62
61.46
RSNA dataset are not truly disease-related features but rather shortcut features, indicating that the high baseline metries were based easily distinguishable shortcut features. Furthermore, after training with EGMA, the performance of other models significantly improves on all three datasets.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
The paper provides the full set of assumptions and a complete experimental
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and data used in this paper have been available on Gil ub. once the paper is accepted. we will provide detailed links.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
ooicc) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwppppguuuiiieeeee /otpc//uuuieee/oeeuubiiiiii
(CodeSubiisss wwtaaabmmsssooni ssiiiiolcccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The full details are provided with the main context and supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information on the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /thiss/hhhrr////orrrp....//////p
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential positive societal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The paper describes safeguards that have been put in place for responsible release of data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Meet the requirements.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Meet the requirements.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not Applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not Applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "JZSRYmfh1d", "reviewer": "Reviewer_hPhG", "summary": "This paper proposes utilizing eye-gaze information to aid in learning representations from paired images and texts.  The method was validated on four medical datasets, demonstrating the feasibility of integrating gaze information for multi-modal alignment.", "questions": "If the gaze information is not very accurate, for example, if the gazes occasionally shift to locations not exactly on the objects (e.g., cardiac), would using the gaze information harm the learning performance? If not, why, and if yes, how would you handle such an issue?", "limitations": "Please see weakness.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The idea of using additional information beyond the image-text pairs is interesting.", "weaknesses": "First, if eye-gaze coordinates are noisy and contain errors, introducing eye-gazing into the learning process could negatively impact representation learning with image-text pairs. The image-text pairs are well-defined and reliable data resources. In contrast, the quality of eye-gaze information can be highly dependent on the operators, working conditions, and devices. Introducing potentially noisy and/or unreliable information into the learning process is risky and may lead to degraded performance.This paper does not mention and address this issue and the proposed method lacks for dealing with potentially noisy gaze information (could happen.)\n\nEye-gaze locations may fall on the cardiac areas in both normal and abnormal cases, but the critical consideration in learning is to develop a representation that can distinguish between these situations. Eye-gazes might provide little to no useful information for this learning process, as there is can be no significantly clear distinction between the eye-gaze coordinates in abnormal versus normal cases. Introducing eye-gaze data is not useful for learning to distinguish between these conditions. The only benefit might be identifying the cardiac area, which can be easily captured by conventional contrastive learning without the need for explicitly provided location information.\n\nThe writing is poor. The notations and formulas used in the method section are unprofessional. Notations such as COS, Mean, and Max are too casual and should be changed to be more rigorous and formal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1\\&Q1:** We thank the reviewer for these great questions and apologize for any confusion caused by the lack of discussion in our paper. Indeed, noise and errors in eye-gaze data are as common as noise in images, and these can all affect the model's final performance. In this work, the errors in eye-gaze data primarily stem from two factors: involuntary saccades and microsaccades of the radiologists' eyes [1], and subjective fixation errors [2]. \n\nHuman eye movements can be categorized into two main types: saccades and fixations [1]. Saccades are the rapid movements between different areas of the visual field and are generally considered noise data that do not involve cognitive processes. Fixations, on the other hand, are brief pauses of the eyes on a small area and are regarded as the primary cognitive behavior. Additionally, due to the structure of the eye, fixations are accompanied by microsaccades, causing the fixation point to drift within the fixation area. Thus, microsaccades are also a major source of noise in eye-gaze data. Fortunately, eye-tracking technology has evolved significantly over the past century, and noise reduction techniques for eye-gaze data have become highly advanced. Currently, all commercial eye-tracking devices come with preprocessing software that uses adaptive methods (e.g., [3]) to filter out noise, greatly facilitating the use of eye-tracking technology. In the attached PDF, Figure R1 shows an example of denoising process in MIMIC-EYE dataset.\n\nAdditionally, due to variations in radiologists' expertise and cognitive levels, fixation data may contain errors, such as the reviewer mentioned, \"occasionally shift to locations not exactly on the objects.\"\n\nFor the minor shift error of fixations, where the duration of incorrect fixations is small compared to the overall duration, their values in the attention heatmap are low, thus having minimal impact on the mlce loss utilized in our work (Eq. 4 in main paper). In our EGMA, EGF and EGM modules (Sec. 3.2 and Sec. 3.3 in main paper) can also filter out this minor errors in the heatmap. For the significant fixation errors, MIMIC-EYE data has been cleaned to address this issue at the first time. Specifically, these data were first reviewed by professionals to exclude inconsistent diagnoses and fixation errors. Moreover, the MIMIC-EYE dataset includes eye-gaze data from six radiologists. Even if one or two radiologists' fixations are incorrect, the correct fixations from the other radiologists under the same semantic context can mitigate the adverse effects. In the attached PDF, Figure R2 shows an example of this compensation.\n\nAs the reviewer pointed out, effectively identifying and mitigating errors in radiologists' diagnostic behavior during model training is crucial, and this will be a focus of our future work. This discussion and results will be incorporated into the revised version of the paper.\n\n**W2:** We thank the reviewer for this insightful question and apologize for any confusion caused by the lack of relevant discussion in our paper. In fact, numerous studies have investigated and proven the close relationship between radiologists' gaze behavior, image content, and diagnostic results. For example, studies [4-9] have found that radiologists fixate more on diseased areas than on healthy ones. Moreover, they discovered that novice radiologists, due to their lack of experience, repeatedly look at diseased areas, resulting in even more fixations in these areas than those of expert radiologists. Figure R3 in the attached PDF shows some comparison cases. \n\nAdditionally, the abnormal regions with more fixations have higher weights in the attention heatmap. Our EGMA model can leverage these weights, along with image content and text, to learn better disease diagnosis capabilities and feature representations. This discussion and results will be incorporated into the revised version of the paper.\n\n**W3:** We are grateful for the reviewer’s attentive analysis and helpful feedback. We have revised the formulas as follows: (1) Change \"COS\" (Eq. 1 and Eq. 3 in main paper) to lowercase, as shown in Equations 1 and 2 below:\n$$\n    s_{k,l}^{I2T} = cos(z_k^I, z_l^T), \\ s_{k,l}^{T2I} = cos(z_k^T, z_l^I) \\quad  1 \\leqslant l \\leqslant b  ~~ (1) \\\\\n$$\n\n$$\n    x_k^{S2P} = cos(S_k^j, P_k^i), \\ x_k^{P2S} = cos(P_k^i, S_k^j)   ~~ (2)\n$$\n(2) Change \"Mean/Max\" (Sec. 3.2 in main paper) to the following Equations 3 and 4:\n$$\n\\hat{z}^I_k= \\frac{1}{m} \\sum^{m}_{j=1}[\\underset{j}{max}  {(x^{P2S}_k)} ], ~~ 1 \\leqslant j \\leqslant m  ~~ (3)\n$$\n\n$$\n\\hat{z}^T_k= \\frac{1}{n} \\sum^{n}_{i=1}[\\underset{i}{max} {(x^{S2P}_k)} ], ~~ 1 \\leqslant i \\leqslant n  ~~ (4)\n$$\nThese revised formulas will be incorporated into the revised version of the paper.\n\nReferences:\n\n[1] Chamberlain, L.. (2007). Eye tracking methodology: theory and practice.\n\n[2] Brady, A.P.. (2017). Error and discrepancy in radiology: inevitable or avoidable?\n\n[3] Nyström, M. and Holmqvist, K.. (2010). An adaptive algorithm for fixation, saccade, and glissade detection in eyetracking data.\n\n[4] Nodine, C.F., et al. (1996). Nature of expertise in searching mammograms for breast masses.\n\n[5] Mallett, S., et al. (2010). Tracking eye gaze during interpretation of endoluminal three-dimensional CT colonography: visual perception of experienced and inexperienced readers. \n\n[6] Voisin, S., et al. (2013). Investigating the association of eye gaze pattern and diagnostic error in mammography.\n\n[7] Giovinco, N. A., et al. (2015). A passing glance? Differences in eye tracking and gaze patterns between trainees and experts reading plain film bunion radiographs.\n\n[8] Van der Gijp, A., et al. (2017). How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology.\n\n[9] Tourassi, G., et al. (2013). Investigating the link between radiologists' gaze, diagnostic decision, and image content.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sT5VFY5SOC", "reviewer": "Reviewer_YZxW", "summary": "This paper proposes a cross-modal alignment method that can optionally learn from eye tracking data that is collected together with the speech of radiologists. The proposed Eye-gaze Guided Multi-modal Alignment (EGMA) system consists of losses that optionally incorporate alignment objectives between sentences and gaze regions (expressed as heatmaps) which allows the contrastive learning scheme to out-perform existing vision-language models for radiology, such as MedCLIP. This is in terms of linear evaluation (supervised classification), zero-shot classification, image-to-text retrieval and text-to-image retrieval. The EGMA method also shows highly interpretable cross-modal attention maps and well clustered embeddings (with respect to known ground-truth classes).", "questions": "Please clarify how the Mean(Max()) operation described in Sec. 3.2 enables fine-grained alignment.\n\nI would also like to understand the motivation for the EGM module, in relation to existing prior art.\n\nThank you.", "limitations": "The authors are very forward and clear about the limitation of using or collecting data where gaze and speech are aligned. However, they also mention that medical practises are evolving to allow for this better. In this reviewer’s opinion, the framework proposed in this paper could be easily applied to non-medical areas as well as its design is domain-agnostic.\n\nPrivacy or societal concerns are also adequately mentioned.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "A key challenge with papers that propose to incorporate eye gaze data into vision-language model training, is the lack of paired training samples. This paper proposes a very clever solution that is clever in two ways: (a) they propose fine-grained alignment between sentence tokens (originating from speech) and scanpath heatmaps corresponding to the sentences, and (b) design their contrastive objectives to allow for gaze supervision to be optional. This results in a rare demonstration of the benefit of gaze data for improving not only performance (as measured by classification or retrieval) but also interpretability.\n\nThe paper is also written very well, with sufficient figures and descriptive text - which is a plus.\n\nThe presented quantitative results are comprehensive, on several datasets, and consistently out-performing multiple existing state-of-the-arts. The ablation study, in particular, with regards to varying the amount of available gaze data is very interesting. It shows a gradual and (mostly) consistent increase in zero-shot classification performance with an increasing number of gaze-labeled data. The value of EGMA is further supported by the fact fine-tuning SotA models using EGMA typically results in performance improvements (Appendix F).", "weaknesses": "Despite the authors’ best efforts, Fig. 2C and Fig. 2D (and the corresponding Sec. 3.2 and 3.3) are hard to understand. In particular, it is unclear to me why the Mean(Max()) reduces features are considered to be “fine-grained” (it rather seems to be an instance-level feature that allows for gaze-free supervision), and why the cross-modality mapping is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1\\&W1: \"Please clarify how the Mean(Max())...\" \"Despite the authors’ best efforts, Fig. 2C and Fig. 2D...allows for gaze-free supervision)\"**\n\nWe thank the reviewer for this great question and apologize for any confusion caused by casual description in our paper. First, to clarify the role of fine-grained alignment, we have rewritten the Mean(Max()) operation in Sec. 3.2 of main paper as follows:\n\n$$\n\\hat{z}^I_k= \\frac{1}{m} \\sum^{m}_{j=1}[\\underset{j}{max}  {(x^{P2S}_k)} ], ~~ 1 \\leqslant j \\leqslant m   ~~  (1)\n$$\n\n$$\n\\hat{z}^T_k= \\frac{1}{n} \\sum^{n}_{i=1}[\\underset{i}{max} {(x^{S2P}_k)} ], ~~ 1 \\leqslant i \\leqslant n  ~~  (2)\n$$\n\nIn the above equations, $x_k^{P2S} \\in \\mathbb{R}^{n \\times m}$ and $x_k^{S2P} \\in \\mathbb{R}^{m \\times n}$ are similarity matrices between local features within a single instance, and these matrices have been optimized with eye-tracking data as shown in Eq. 4 in main paper. We know that CLIP's contrastive loss can align features well within a batch by computing the similarity between global features of the samples. To continue using this contrastive loss on top of instance-level local features, we need the global similarity between the corresponding text and image for each instance. In this work, instead of directly using the global features output by the encoder as CLIP does, we first calculate local feature similarities and then derive global similarity from the local similarity matrix. Specifically, to compute the global similarity between an image and a diagnostic text within a batch, we first calculate the similarity matrix $x_k^{P2S}$ between the local features of image patches and sentence tokens using Eq. 3 in main paper. Then, we take the maximum value of each column of this matrix, meaning each sentence token gets a similarity score with its most related image patch. Finally, we average these values to obtain the global similarity refined by local similarity. The calculation of the fine-grained global similarity from diagnostic text to image within a batch is similar. This discussion and results will be incorporated into the revised version of the paper.\n\n\n**Q2\\&W2: \"I would also like to understand the motivation for the EGM module...\" \"why the cross-modality mapping is necessary.\"**\n\nWe thank the reviewer for this great question. In fact, the EGF and EGM modules proposed in our work are similar to the diagnostic process of radiologists. Many studies [1-5] have pointed out that radiologists perform a global search first and then conduct a detailed examination of the local areas when suspicious lesions are found. Firstly, the EGF module of EGMA corresponds to the global search by radiologists, aligning the local image patch features with the individual sentence features. Secondly, EGM uses the key values from the local similarity matrix computed in the first step as weights, focusing the alignment on certain important image patches and texts, akin to the detailed observation stage after identifying suspicious lesions. Thus, by mimicking the real diagnostic behavior of radiologists, our EGMA further enhances the model's multimodal processing capability and diagnostic accuracy. This discussion will be incorporated into the revised version of the paper.\n\n\n**L: \"In this reviewer’s opinion, the framework proposed in this paper could be easily applied to non-medical areas as well as its design is domain-agnostic.\"**\n\nWe greatly appreciate the reviewer's suggestion. In fact, thanks to the inherent flexibility of the EGMA model, it is well-suited for multimodal alignment tasks involving natural images, such as in human-robot interaction and control as well as for education/training purposes. Therefore, we believe this is a promising direction for future expansion. Once again, thank the reviewer for this valuable suggestion. This discussion will be incorporated into the revised version of the paper.\n\nReferences:\n\n[1] Nodine, C.F., et al. (1987). Using eye movements to study visual search and to improve tumor detection.\n\n[2] Swensson, R.G.. (1980). A two-stage detection model applied to skilled visual search by radiologists. \n\n[3] Kahneman, D.. (2003). A perspective on judgment and choice: Mapping bounded rationality.\n\n[4] Kundel, H.L., et al. (2007). Holistic component of image perception in mammogram interpretation: gaze-tracking study.\n\n[5] Drew, T., et al. (2013). Informatics in radiology: what can you see in a single glance and how might this guide visual search in medical images?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fiIS8lq8Vy", "reviewer": "Reviewer_V2ie", "summary": "The paper proposes EGMA, a novel framework for medical multi-modal alignment integrating eye-gaze data into vision-language pre-training. EGMA outperforms existing methods in image classification and image-text retrieval tasks, demonstrating significant advancements and improved feature representation with even minimal eye-gaze data.", "questions": "The paper mentions that even a small amount of eye-gaze data can enhance model performance. Can you provide more quantitative results or analysis on how performance scales with different amounts of eye-gaze data?\n\nHave you considered applying the EGMA framework to other types of medical imaging modalities (e.g., MRI, CT scans) or diagnostic tasks? If so, what challenges or modifications would be necessary?", "limitations": "The authors discuss privacy concerns associated with using eye-gaze data, suggesting the use of de-identification methods or releasing data in the form of heatmaps to mitigate these issues.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Innovative approach that diverges from traditional reliance on annotated datasets, providing a fresh perspective on multi-modal learning in medical contexts.\n\nRobust experimental design that includes comparisons with state-of-the-art methods, highlighting the efficacy of the proposed framework.\n\nComprehensive visualizations and tables that effectively communicate the results and support the paper's claims.\n\nProvides a scalable solution that shows strong generalization across different datasets, indicating its broader applicability in various medical settings.", "weaknesses": "The paper primarily focuses on classification and retrieval tasks. Including additional evaluations, such as lesion localization or segmentation tasks, could provide a more comprehensive assessment of the framework’s effectiveness.\n\nThe use of eye-gaze data raises potential privacy issues, as such data can inadvertently reveal sensitive information about the observers. Addressing these concerns through robust de-identification methods would strengthen the ethical considerations of the work.\n\nAlthough the paper includes ablation studies, more detailed analysis on the impact of different components of the model, such as varying the amount of eye-gaze data or different types of medical images, could provide deeper insights into the robustness and adaptability of the proposed framework.\n\nThe model relies on multi-modal datasets like MIMIC-EYE, which simultaneously collect eye-gaze data, medical images, and diagnostic text. This dependency might limit the generalizability of the approach to other datasets that lack such rich multi-modal annotations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** We greatly appreciate the reviewer's suggestion. Supervision information used for localization and segmentation tasks, such as bounding boxes and masks, is stronger than the labels used for classification tasks. Additionally, the amount of refined manual annotation required for localization and segmentation is relatively limited. Therefore, we believe that if EGMA performs well on classification tasks with relatively weak supervision and across multiple classification datasets, it is likely to also excel in localization and segmentation tasks. Furthermore, the visualization results in our work already provide preliminary evidence of EGMA's localization capability. Finally, thank the reviewer again for this suggestion. Conducting additional validation experiments is indeed a promising direction for our future work.\n\n**W2:** We thank the reviewer for this great comment. Privacy concerns in eye-gaze data are often overlooked by researchers. Studies [1,2] have shown that eye-gaze data can reveal private information such as the collector's personality traits, age, gender, and mental state, making privacy protection crucial. One approach [3] involves adding noise to the original eye-gaze data to obscure some private information, but this can compromise the quality of the data. Another commonly used method [4], which is employed in EGMA, is to convert the original eye-gaze data into a two-dimensional attention heatmap. This heatmap retains rich visual information while mitigating the risk of privacy leaks compared to raw eye-gaze data. Therefore, the EGMA framework protects the privacy of the participants during the data preprocessing phase.\n\n**W4:** We thank the reviewer for this great comment. In fact, when designing experiments for zero-shot classification and zero-shot image-text retrieval, EGMA took this issue into account. Experimental results show that after multimodal training on datasets with eye-gaze data, the model performs well on other datasets without eye-gaze data. Furthermore, with the advancement of eye-gaze data collection systems [5-8], collecting multimodal data with eye-gaze has become increasingly feasible. Therefore, we propose a strategy where pre-training is conducted in scenarios where eye-gaze data is more easily collected, followed by fine-tuning on more challenging scenarios. This approach is similar to mainstream multimodal pre-training models, where a strong pre-trained model is first trained on a widely available dataset and then fine-tuned on other datasets for downstream tasks. The performance of EGMA in this work supports the feasibility of this approach.\n\n**W3\\&Q1:** We thank the reviewer for this great question. In fact, we have discussed this issue in the ablation study of our paper (Tab. 4 in main paper). Given the current scarcity of eye-gaze data, EGMA was designed to support training with both eye-gaze and non-eye-gaze data within the same batch. In Tab. 4, we conducted comparative experiments with four different proportions of eye-gaze data: 1\\%, 5\\%, 10\\%, and 50\\%, corresponding to approximately 37, 185, 370, and 1848 instances with eye-gaze data in the training set, respectively. It can be observed that with 1\\% of eye-gaze data, the model shows only slight improvement on the CheXpert5x200 dataset due to limited auxiliary information. As the proportion of eye-gaze data increases, the performance of EGMA gradually improves across the three datasets. This aligns with the expectation that providing more auxiliary information allows the model to learn better feature representations and thus enhance its performance on the datasets.\n\n**Q2:** We thank the reviewer for this great question. Our EGMA model is designed based on the CLIP architecture and supports the co-training of both eye-gaze and non-eye-gaze data, which provides it with significant flexibility. Recently, a study has seamlessly integrated eye-tracking technology into ultrasound imaging [8], where ultrasound images, eye-gaze data, and diagnostic texts were collected. EGMA can be easily adapted to this dataset with minimal modifications, primarily involving the replacement of the encoder with one pre-trained on ultrasound images.\n\nReferences:\n\n[1] Kröger, J.L., et al. (2020). What does your gaze reveal about you? On the privacy implications of eye tracking.\n\n[2] Katsini, C., et al. (2020). The role of eye gaze in security and privacy applications: Survey and future HCI research directions.\n\n[3] Steil, J., et al. (2019). Privacy-aware eye tracking using differential privacy.\n\n[4] Liu, A., et al. (2019). Differential privacy for eye-tracking data. \n\n[5] Stember, J.N., et al. (2020). Integrating eye tracking and speech recognition accurately annotates MR brain images for deep learning: proof of principle. \n\n[6] Khosravan, N., et al. (2019). A collaborative computer aided diagnosis (C-CAD) system with eye-tracking, sparse attentional model, and deep learning. \n\n[7] Ma, C., et al. (2023). Eye-gaze-guided vision transformer for rectifying shortcut learning. \n\n[8] Men, Q., et al. (2023). Gaze-probe joint guidance with multi-task learning in obstetric ultrasound scanning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dl3sfWpsq2", "reviewer": "Reviewer_n21y", "summary": "This paper introduces the Eye-gaze Guided Multi-modal Alignment (EGMA) framework, which leverages radiologists' eye-gaze data to enhance the alignment of medical visual and textual features. By using synchronously collected eye-gaze data during diagnostic evaluations, EGMA improves generalization and achieves state-of-the-art performance in image classification and image-text retrieval tasks across four medical datasets. The study also investigates the impact of varying amounts of eye-gaze data on model performance, demonstrating the feasibility and utility of integrating this auxiliary data into multi-modal alignment frameworks.", "questions": "Impact of Eye-Gaze Data Quality and Quantity:\nQuestion: How does the quality and quantity of eye-gaze data affect the performance of the EGMA framework? Are there specific thresholds or optimal amounts of eye-gaze data required to achieve the best results?\nContext: Detailed insights into the relationship between eye-gaze data quality/quantity and model performance would help in optimizing the framework and understanding its practical requirements.", "limitations": "No", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Innovative Use of Eye-Gaze Data:\nThe introduction of the Eye-gaze Guided Multi-modal Alignment (EGMA) framework is a novel approach that leverages eye-gaze data from radiologists to improve the alignment of visual and textual features in medical data. This innovative use of auxiliary data opens new avenues for enhancing model performance and provides a unique perspective on incorporating human cognitive processes into machine learning models.\nEmpirical Validation and Generalization:\nThe EGMA framework is rigorously evaluated across four different medical datasets, demonstrating its ability to achieve state-of-the-art performance in image classification and image-text retrieval tasks. This comprehensive empirical validation underscores the robustness and effectiveness of the proposed method, highlighting its potential for broader application in various medical contexts.\nExploration of Eye-Gaze Data Utility:\nThe paper not only introduces a novel framework but also explores the impact of varying amounts of eye-gaze data on model performance. This detailed analysis provides valuable insights into the feasibility and utility of integrating eye-gaze data into multi-modal alignment frameworks, offering practical guidelines for future research and application in the field.", "weaknesses": "Dependency on Eye-Gaze Data Collection:\nIssue: The reliance on eye-gaze data from radiologists introduces a significant dependency that may not be feasible for all institutions due to the need for specialized equipment and the additional effort required to collect this data. Especially when the eye-gaze prior is not accurate.\nImpact: This dependency could limit the scalability and widespread adoption of the EGMA framework, particularly in resource-constrained settings or where the collection of eye-gaze data is not practical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W: \"The reliance on eye-gaze data from...where the collection of eye-gaze data is not practical.\"**\n\nWe thank the reviewer for this great comment. In fact, when designing experiments for zero-shot classification and zero-shot image-text retrieval, EGMA took this issue into account. Experimental validation has shown that after multimodal training on datasets with eye-gaze data, the model performs well on other datasets without eye-gaze data (Tab. 1\\&2\\&3 in main paper). Furthermore, with advancements in eye-gaze data collection systems [1-4], acquiring multimodal data with eye-gaze has become increasingly feasible. Compared to having radiologists spend additional time and effort annotating refined labels, eye-tracking allows for data collection during their routine diagnostic work, minimizing disruption and reducing annotation burdens. Additionally, some studies have demonstrated [3,5,6] that eye-gaze data can achieve performance comparable to refined annotations. Therefore, we propose initially pre-training on scenarios where eye-gaze data is more easily collected and then fine-tuning on more challenging datasets. This approach is similar to current mainstream multimodal pre-training models, where a robust model is trained on a broadly available dataset and then fine-tuned on other datasets to perform downstream tasks. The performance of EGMA in this work supports the feasibility of this approach.\n\n\n**Q1: \"How does the quality and quantity of eye-gaze data affect the performance of the EGMA framework?\"**\n\nWe thank the reviewer for this great question. The quality of eye-gaze data is crucial for model performance. In this work, we first filter out interfering data such as blinks and saccades by using only the fixation data from the eye-tracking recordings. We then generate a 2D attention heatmap from these fixation points to further mitigate the influence of interfering data. Figure R1 in the attached PDF shows an example of denoising in MIMIC-EYE dataset. Additionally, in the design of the EGMA framework, we use the EGM module to further filter out potentially problematic regions. For catastrophic errors could happened, the MIMIC-EYE dataset provides a robust solution by involving multiple expert radiologists in data collection. This approach ensures overall data quality and allows for the compensation of errors made by one or two radiologists with correct data from other radiologists. Figure R2 in the attached PDF shows an example of this compensation.\n\nRegarding the issue of the quantity of eye-gaze data, we conducted extensive comparisons in the ablation study of this work (Tab. 4 in main paper). Specifically, we evaluated the model performance with 1\\%, 5\\%, 10\\%, and 50\\% of the eye-gaze data. We found that with a small amount of data, such as 1\\% (approximately 37 samples), the model's performance improvement was limited, which is attributed to insufficient auxiliary information. As the proportion of eye-gaze data increased, the performance of EGMA improved progressively across the three datasets. This aligns with the expectation that providing more supervised information allows the model to learn better feature representations and consequently enhance performance on the datasets.\n\n**Q2: \"Are there specific thresholds or optimal amounts of eye-gaze data required to achieve the best results?\"**\n\nWe thank the reviewer for this great question. For our EGMA framework, having more eye-gaze data is certainly beneficial, similar to other types of annotations. Additionally, EGMA is designed to handle situations where eye-gaze data may be limited, as it supports training with both eye-gaze and non-eye-gaze data. In our ablation experiments (Tab. 4 in main paper), we also investigated the impact of the proportion of eye-gaze data on model performance. Our goal was to demonstrate that even with limited eye-gaze data (e.g., using only 185 samples), EGMA's performance can still improve steadily, highlighting the framework's flexibility and effectiveness.\n\nReferences:\n\n[1] Stember, J.N., et al. (2020). Integrating eye tracking and speech recognition accurately annotates MR brain images for deep learning: proof of principle.\n\n[2] Khosravan, N., et al. (2019). A collaborative computer aided diagnosis (C-CAD) system with eye-tracking, sparse attentional model, and deep learning.\n\n[3] Ma, C., et al. (2023). Eye-gaze-guided vision transformer for rectifying shortcut learning.\n\n[4] Men, Q., et al. (2023). Gaze-probe joint guidance with multi-task learning in obstetric ultrasound scanning.\n\n[5] Wang, S., et al. (2022). Follow my eye: Using gaze to supervise computer-aided diagnosis.\n\n[6] Ma, C., et al. (2023). Rectify vit shortcut learning by visual saliency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their careful reading, valuable comments, and recognition of the contributions of our work. We have provided itemized responses to the questions and suggestions from the reviewers. We are also pleased to receive the positive feedbacks from reviewers, particularly:

1. Novel and interesting method [Reviewers hPhG, YZxW, V2ie, and n21y].

2. Thorough evaluations and ablation studies [Reviewers YZxW, V2ie, and n21y].

3. Potential for broader application [Reviewers V2ie and n21y].

4. Excellent and clear written [Reviewer YZxW].

We found that reviewers are particularly concerned about the general availability and quality of the eye-gaze data. Here are brief responses to these two concerns:

**Data availability issue [Reviewers V2ie and n21y].**

Multiple studies [1-5] have investigated the cognitive behavior in radiologists' eye-gaze data, demonstrating its close relationship with images and diagnostic results. This highlights the availability and value of eye-gaze data in the medical field. In addition, with the growing research and solution development in the practical application of eye-tracking systems in the medical field [6-13], we envision that both the quality and the availability of eye-gaze data will be greatly improved. The recent success of eye-tracking technology in commercial applications (e.g., Apple Vision Pro, eye-tracking in iOS 18) further highlights its core role as a next-generation human-computer interaction technology. In our work, the model's zero-shot capability on other datasets significantly improved after eye-gaze guidance. This demonstrates that the model has learned more generalizable feature representation through the use of eye-gaze data. Furthermoer, we conducted ablation experiments (Tab. 4 in the main paper) to verify that our model can benefit from very limited eye-gaze data. Therefore, the ability to efficiently utilize eye-gaze data can provide valuable assistance to the model both now and future. Detailed explanations are provided in response to the respective questions from the reviewers (**Reviewer V2ie**:W4, **Reviewer n21y**:W).

**Data quality issue [Reviewers hPhG and n21y].**

Eye-tracking technology has evolved significantly over the past century, and noise reduction techniques for eye-gaze data have become highly advanced [14-16]. We also included examples in the attached PDF (Figure R1 and Figure R2) to illustrate how our model can mitigate the impact of noise and errors in the eye-gaze data. Detailed explanations are provided in response to the respective questions from the reviewers (**Reviewer hPhG**:W1\&Q1, **Reviewer n21y**:Q1).


References:

[1] Nodine, C.F., et al. (1987). Using eye movements to study visual search and to improve tumor detection.

[2] Swensson, R.G.. (1980). A two-stage detection model applied to skilled visual search by radiologists. 

[3] Kahneman, D.. (2003). A perspective on judgment and choice: Mapping bounded rationality.

[4] Kundel, H.L., et al. (2007). Holistic component of image perception in mammogram interpretation: gaze-tracking study.

[5] Drew, T., et al. (2013). Informatics in radiology: what can you see in a single glance and how might this guide visual search in medical images?

[6] Khosravan, N., et al. (2019). A collaborative computer aided diagnosis (C-CAD) system with eye-tracking, sparse attentional model, and deep learning. 

[7] Stember, J.N., et al. (2020). Integrating eye tracking and speech recognition accurately annotates MR brain images for deep learning: proof of principle. 

[8] Wang, S., et al. (2022). Follow my eye: Using gaze to supervise computer-aided diagnosis.

[9] Ma, C., et al. (2023). Eye-gaze-guided vision transformer for rectifying shortcut learning. 

[10] Men, Q., et al. (2023). Gaze-probe joint guidance with multi-task learning in obstetric ultrasound scanning. 

[11] Ma, C., et al. (2023). Rectify vit shortcut learning by visual saliency.

[12] Ji, C., et al. (2023). Mammo-net: Integrating gaze supervision and interactive information in multi-view mammogram classification.

[13] Zhao, Z., et al. (2024). Mining gaze for contrastive learning toward computer-assisted diagnosis.

[14] Salvucci, D.D. and Goldberg, J.H.. (2000). Identifying fixations and saccades in eye-tracking protocols.

[15] Smeets, J.B. and Hooge, I.T.. (2003). Nature of variability in saccades.

[16] Nyström, M. and Holmqvist, K.. (2010). An adaptive algorithm for fixation, saccade, and glissade detection in eyetracking data.

Meta-review: This paper introduces an eye gaze guided vision language learning to improve visual and text representation learning for medical applications .  It receives one strong accept , one accept, and two borderline accepts . The reviewers in general like this work, finding its ideas using gaze to guide learning novel, papers well written, with extensive experiments that demonstrate the superior performance of their method.  They have some concerns about data quality, data accessibility, and data privacy.  The authors rebuttal is extensive and well addresses the reviewers concerns .  I hence recommend its acceptance.
Decision: Accept (poster)