Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning

Authors: chong  ma
hanqi  jiang
wenting  chen
yiwei  li
zihao  wu
xiaowei  yu
zhengliang  liu
lei  guo
dajiang  zhu
tuo  zhang
dinggang  shen
tianming  liu
xiang  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 11:16:17

Abstract: In the medical multi-modal frameworks, the alignment of cross-modality features presents a significant challenge. However, existing works have learned features that are implicitly aligned from the data, without considering the explicit relationships in the medical context. This data-reliance may lead to low generalization of the learned alignment relationships. In this work, we propose the Eye-gaze Guided Multi-modal Alignment (EGMA) framework to harness eye-gaze data for better alignment of medical visual and textual features. We explore the natural auxiliary role of radiologists' eye-gaze data in aligning medical images and text, and introduce a novel approach by using eye-gaze data, collected synchronously by radiologists during diagnostic evaluations. We conduct downstream tasks of image classification and image-text retrieval on four medical datasets, where EGMA achieved state-of-the-art performance and stronger generalization across different datasets. Additionally, we explore the impact of varying amounts of eye-gaze data on model performance, highlighting the feasibility and utility of integrating this auxiliary data into multi-modal alignment framework.

Main paper: 
 introduction: 
 With the development of multi-modal learning, pre-trained models can now utilize large amounts of paired multi-modal data, such as image-text pairs, audio-text pairs, etc., to optimize the multimodal feature extraction and alignment capabilities. With the emergence of the CLIP [42] model, contrastive learning has become the prominent framework of multi-modal learning. The advantage of this framework lies in its simplicity of structure and it does not require sample-level annotations. However, the main drawback is its heavy reliance on the scale of training data. Subsequent works have optimized this framework by leveraging potential auxiliary information between image and text data. For instance, GLIP [32] and RegionCLIP [61] utilized pre-predicted annotation information to perform fine-grained region-level pre-training. They introduced detection networks firstly to predict image regions relevant to the text prompt, and then trained the model to align these image regions with their corresponding text descriptions. However, these models heavily rely on the performance of the ROI detector and have high computational complexity. FILIP [57] proposed a refined multi-modal alignment operation after the encoder, relying solely on image patches and text tokens. Although this further explores the local feature relationships between multi-modal data, it still requires sufficient data support. When training on small-scale datasets, especially in the medical field, accurately learning alignment features between modalities becomes more challenging [6, 60] .To address the scarcity of medical data, studies [4, 58] have introduced self-supervised training into the CLIP framework to further enhance encoder performance. Additionally, weak labels between images and texts have been incorporated during pre-training to aid multi-modal alignment [55] . Some studies [15, 53] utilized fine-grained alignment between chest image patches and text tokens for pre-training [57] . However, unlike natural images and text, the relationship between medical images and diagnostic text is often more complex and challenging to learn. Moreover, with insufficient data, models are prone to learning shortcut features unrelated to disease diagnosis, resulting in poor generalization ability [10, 36, 35] . Therefore, it is crucial to learn useful alignment information from relatively limited medical multi-model datasets. In this study, we fully explore the auxiliary role of eye-gaze data from radiologists in multi-model alignment. Eye-gaze data can intuitively reflect the image regions radiologists focus on, providing insights into their cognitive behavior during diagnosis [7] . Therefore, compared to refined annotations like bounding boxes and masks, eye-gaze data can also provide useful auxiliary information for the model [24, 54, 35] . Moreover, collecting eye-gaze data from radiologists during the diagnostic process is more time-efficient than annotating bounding boxes and masks [25, 35] . For the multi-modal medical dataset, EYE GAZE [22] and REFLACX [31] collected eye-gaze data from radiologists while diagnosing chest X-rays. Additionally, these datasets recorded synchronized voice data, where radiologists verbalized their diagnoses while observing the images. As shown in Fig. 1 , we found that the radiologists' attention regions on the image naturally align with the diagnostic text over time. Therefore, we believe this type of eye-gaze data can provide expert prior knowledge for training the alignment between medical visual and textual features. Thus, considering the utilization of eye-gaze data to assist in multi-modal model training, we propose the Eye-gaze Guided Multi-modal Alignment framework (EGMA). Our model first segments the transcribed text into individual sentences and obtains radiologists' attention heatmaps. Subsequently, we obtain encoded features of image patches and sentences through image and text encoders, generating instance-level similarity matrix. Then, we compute the loss between this matrix and the attention heatmaps, integrating refined feature representations for subsequent contrastive loss. To further leverage the assisting role of eye-gaze data in aligning images and texts, we combine the eye-gaze heatmaps with the similarity matrix derived from model, serving as weights to calculate cross-modality mapping loss. Experimental results on zero-shot classification and retrieval tasks reveal that our framework surpasses other leading methods in performance across diverse datasets and under multiple dataset size scenarios. Specifically, the EGMA framework yielded a remarkable 3.9% improvement in image-to-text matching tasks and an impressive 19.75% increase in text-to-image matching tasks. These results underscore the cutting-edge and efficacious nature of our approach, highlighting its substantial advancements over existing methodologies. We also explore the auxiliary effect of using eye-gaze data of different scales on the model, finding that even a small portion of eye-gaze data can enhance the model's multi-modal processing capability. Moreover, the fine-tuned classification results of EGMA achieved the best performance across multiple datasets. The code of this work is available on Github 2 .In summary, the main contributions of this work are as follows:• We propose EGMA, a novel framework for medical multi-modal alignment, marking the first attempt to integrate eye-gaze data into vision-language pre-training. • EGMA outperforms existing state-of-the-art medical multi-modal pre-training methods, and realizes notable enhancements in image classification and image-text retrieval tasks. • EGMA demonstrates that even a small amount of eye-gaze data can effectively assist in multi-modal pre-training and improve the feature representation ability of the model. 
related works: 
 Medical Vision-language Pre-training (Med-VLP): In pursuit of Artificial General Intelligence (AGI), Vision-language Pre-training (VLP) has become a pivotal area in AI research. The advent of the transformer architecture [51] has significantly accelerated progress in the multi-modal domain by integrating vision and language, with VLP frameworks focusing on fusion encoders that use cross-attention mechanisms to amalgamate visual and textual features [34, 47] . The introduction of CLIP [42] marked a breakthrough, leading to numerous CLIP-based VLP frameworks incorporating contrastive loss as a core component [57, 32] . In the medical field, ConVIRT [59] serves as an equivalent to CLIP, while MedCLIP [55] addresses the challenge of insufficient paired image-text data by integrating knowledge extraction techniques. BioViL [2] enhances performance through specialized biomedical text BERT encoders in contrastive learning tasks. GLoRIA [15] proposes multi-modal global-local representation learning, and MGCA [53] introduces alignment at pathological region, instance, and disease levels. Furthermore, study [56] incorporates knowledge bases to infuse expert medical knowledge into the system.In medical imaging diagnostics, eye-tracking technology has proven valuable for decades, revealing how experienced radiologists can quickly identify hidden lesions through comprehensive observation [28, 29, 7, 26] . Integrating radiologists' eye-gaze data with deep learning models has significantly advanced the field. For instance, merging eye-gaze data with Convolutional Neural Networks (CNNs) has enhanced lesion detection accuracy [24] , and visual search patterns in mammography have linked human visual attention with CNN performance [37] .Comprehensive datasets combining eye-gaze data and disease diagnoses have facilitated multi-task processing [21] , while attention consistency modules have improved CNN accuracy in diagnosing osteoarthritis from knee X-rays [54] . Recently, integrating eye-gaze data with Vision Transformer (ViT) models has further pushed the boundaries of medical image processing [35] . Additionally, multimodal guidance systems replicating eye tracking and probe manipulation in ultrasound examinations have significantly enhanced scanning accuracy and efficiency [38] . Despite these advancements, fully integrating eye-gaze data with image-text alignment strategies in medical vision-language models remains an ongoing research challenge. 
method: 
 As shown in Fig. 2 , the framework of our proposed method consists of four main components. Firstly, we extract features from image and text in part A to obtain a refined instance-level similarity matrix. Secondly, in part B, we integrate textual transcripts derived from radiologists' audio, images, and eye-gaze data, to visualize and map radiologists' attention onto specific regions of images during diagnosis. This process establishes alignment between texts and images, facilitating model training.The detailed gaze data processing methods are described in Sec. 3.1. Given that eye-gaze data tightly links textual and localized visual information, after obtaining auxiliary information from part B, we introduce eye-gaze guided refined alignment training strategies, as depicted in Parts C and D of Fig. 2 . Specifically, we introduce the optimization algorithm for eye-gaze guided fine-grained text-image similarity matrix in Part C in Sec. 3.2. Finally, in Sec. 3.3, we present the algorithm for eye-gaze guided cross-modality mapping.With the development of data collection technologies such as eye-tracking and speech recognition, it has become possible to collect and process multi-modal interaction data of radiologists during the diagnostic process. In this work, we utilize MIMIC-EYE [14] datasets as our training set, consisting of 3689 images extracted from the MIMIC datasets [19, 20, 18, 17] . Each sample is accompanied by corresponding eye-tracking data and transcripts text. These eye-tracking data are provided by the publicly available EYE GAZE [22] and REFLACX [31] datasets on PhysioNet [12] . Since each modality is synchronized, the audio data is aligned with the eye-gaze data in time. By segmenting the audio based on the time before and after the pronunciation of each word, we can align the transcripts with the audio, thereby aligning sentence-level text with eye-gaze data. Subsequently, we generate attention heatmap based on eye-gaze data and images to represent the image regions the radiologist focuses on. Through the aforementioned data processing steps, we achieve precise alignment between sentence-level text and image regions. Detailed processing method of eye-gaze and audio transcripts can be found at Appendix D.The core idea of contrastive learning is to bring the features of related samples closer while pushing away the features of unrelated samples. During the training progress of CLIP [42] model, assuming a batch size of b and input datax I k , x T k (k = 1, • • • , b) representing image-text pairs, global features z I k = E I (x I k ) ∈ R 1×d and z T k = E T (x T k ) ∈ R 1×dare obtained through image encoder E I and text encoder E T . Subsequently, the cosine similarity s I2T k,l and s T 2I k,l between the two modalities is computed, with the following formula:EQUATION)where cossim is cosine similarity, s I2T k,l is the image-to-text similarity, s T 2I k,l is the text-to-image similarity, and l is the index number of the another modality. Then, the image-to-text contrastive loss L I2T k for x I k and text-to-image contrastive loss L T 2I k for x T k can be formulated as:EQUATIONwhere τ is a learned temperature. It is worth noting that in the calculation of the loss mentioned above, both the image and text utilize global-level features, while the auxiliary information generated from eye-gaze data emphasizes the local-level features between modalities. Therefore, based on [57] , we replace instance feature z I k and z T k with P n k ∈ R n×d and S m k ∈ R m×d , whereP i k (1 ≤ i ≤ n)is the i-th patch feature of x I k and S j k (1 ≤ j ≤ m) is the j-th sentence feature of x T k , and n, m are the image patch number and the sentence number of report. Then we calculate the similarities of sentence-to-patch x S2P k ∈ R m×n and patch-to-sentence x P 2S k ∈ R n×m in one instance:EQUATIONFor each heatmap corresponding to a sentence, we initially divide it into n patches similar to the image. Subsequently, we concatenate the heatmaps of m sentences to obtain the Gaze-guided Similarity matrix GS k for input x I k , x T k (as illustrated in Fig. 2 .B). In this matrix, non-zero elements indicate the semantic correlation between the corresponding sentences and image patches. Thus, we binarize GS k , setting non-zero regions to 1, resulting in the Gaze-guided Label matrix GL k . After this step, we compute the multi-label cross-entropy (MLCE) loss for x S2P k and x P 2S k , completing the optimization for fine-grained alignment between positive sample pairs x I k , x T k , as follows:EQUATION)where mlce is the multi-label cross-entropy loss. Subsequently, we calculate the fine-grained features ẑI k and ẑT k as follows:EQUATIONThen, we replace the z I k , z T k with the updated ẑI k , ẑT k in Eq. 1. Finally, the fine-grained image-to-text loss LI2Tk and text-to-image loss LT 2I k are computed based on Eq. 2. The formula for our Eye-gaze Guided Fine-grained (EGF) alignment loss is as follows:EQUATIONIn the previous section, we replaced the global instance logits in the traditional batch clip loss with fine-grained instance logits that consider local features and optimized the alignment between these local features using gaze information. The text in our work is recorded by radiologists while observing images, implying a close semantic relationship between the focus region and the corresponding text.To further optimize the alignment between modalities, we continue to incorporate eye-gaze data assistance into the cross-modality mapping process. In this work, we first utilize matrices GS k , x P 2S k and x S2P k to generate the image-to-text and text-to-image alignment weight matrix W I2T ∈ R n×m and W T 2I ∈ R m×n . The calculation formula is as follows:EQUATIONwhere norm is normalization and ω consists of sparse and binarize operations. After obtaining the weight matrix, we perform the mapping from text features S m k to image features Cross_P n k ∈ R n×d and from image features P n k to text features Cross_S m k ∈ R m×d according to the following formula:EQUATION)where i ∈ [1, n] is the i-th patch feature of P n k and j ∈ [1, m] is the j-th sentence feature of S m k . Subsequently, we use the mapped features along with the target features as inputs to compute the alignment contrastive loss defined in Eq. 2, obtaining the image mapping loss mL I k and the text mapping loss mL T k . The formula for our Eye-gaze Guided cross-model Mapping (EGM) loss is as follows:EQUATIONFinally, the total loss of our model within a batch is L = L EGF + L EGM . In our training process, considering the proportion of eye-gaze data, batches may contain both types of data. When encountering samples without eye-gaze data, the EGF module does not compute the loss from Eq. 4, and the weight matrix in the Eq. 7 of EGM module also excludes the GS k .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code and data used in this paper have been available on GitHub. If once the paper is accepted, we will provide detailed links.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this study, we first conduct supervised and zero-shot classification as well as zero-shot retrieval experiments in Sec. 4.1 to validate the model's generalization performance and its representation capability of multi-modal features. Then, in Sec. 4.2, we perform ablation studies on various modules of EGMA. Additionally, to further investigate the auxiliary effect of eye-gaze data, we compare the performance when guided by different amounts of eye-gaze data. Finally, in Sec. 4.3, we visualize the model's feature representations and the learned image-text relationships, further demonstrating the model's performance and interpretability.Image Classification We conduct supervised classification experiments on the CheXpert [16] , RSNA [44] , and SIIM-ACR [45] datasets. CheXpert [16] is a large-scale public dataset for chest radiograph interpretation, it comprises 224,316 chest radiographic images. Following [53] , we utilize the official training split as our training set, and the official validation set of 202 images with expert-label as our test set. RSNA [44] is a comprehensive dataset for Pneumonia diagnosing. It contains 29,700 chest X-ray images categorized into normal and pneumonia positive category.We follow [53] to divide the data into 70% for training, 15% for validation, and 15% for testing. SIIM-ACR [45] is a chest dataset used for pneumothorax diagnosing. It consists of 2379 images with pneumothorax and 8300 images without pneumothorax. In this work, we utilize a subset defined in [43] as our test set, with the remaining data used for training and validation. More details of dataset can be found in the Appendix C.In the supervised classification experiments, we adopt the linear classification settings [15] , where the pre-trained image encoder is frozen, and only a randomly initialized linear classification head is trained. We adopt area under ROC curve (AUROC) metric to evaluate all model's performance.And for better validate the model's efficiency, we test its performance using 1%, 10%, and 100% of the training set. As shown in Tab. 1, our model achieved the best results compared to other models. Additionally, with only 1% of the training set, our model outperformed the second-best model by 1.11%, 0.8%, and 1.94% on the CheXpert, RSNA, and SIIM-ACR datasets, respectively. Moreover, as the amount of training data increased, the model's performance improved significantly. This demonstrates that, with the assistance of radiologists' eye-gaze data, our model possesses strong multi-modal feature representation capabilities. We further conduct zero-shot classification tasks on the CheXpert5x200 [15] , RSNA [44] , and SIIM-ACR [45] datasets. CheXpert5x200 includes five common chest diseases, Atelectasis, Cardiomegaly, Consolidation, Edema, and P leural Ef f usion, each with 200 chest X-rays. It is important to note that the CheXpert training set does not include any data from CheXpert5x200, so there is no data leakage issue. The test sets for RSNA and SIIM-ACR are the same as those used in the supervised classification task. All text prompts are provided by a professional radiologist [15] . During testing, we calculated the similarity between image features and text prompt features for all diseases, with the highest similarity indicating the predicted category. As shown in Tab. 2, CLIP [42] performs poorly on medical images due to its training data primarily consisting of natural images. The models in rows two to five use encoders pre-trained on medical datasets, and thus, their performance is better than that of CLIP. Interestingly, the GLoRIA [15] and MGCA [53] perform worse than the CLIP model in diagnosing pneumonia on the SIIM-ACR dataset. This indicates that these models are significantly influenced by the data distribution, resulting in poor generalization performance. Conversely, our EGMA achieves the best results in all other metrics, except for the F1-score on the RSNA dataset. This demonstrates that our model, enhanced by eye-gaze data, has learned more generalizable feature relationships between medical images and text, significantly improving its generalization performance.Image-text Retrieval To further validate the alignment capability of our model between visual and textual features, we compare the zero-shot retrieval performance of EGMA with other models on CheXpert 8x200 dataset [59] . Unlike CheXpert5x200 [15] , CheXpert8x200 includes eight common chest diseases, N o F inding, Cardiomegaly, Edema, P neumonia, Atelectasis, P neumothorax, P leural Ef f usion, and F racture, each with 200 chest X-rays and five corresponding text prompts.It is worth noting that the prompts for retrieval tasks are different from those for classification tasks in the previous section, but all are written by board-certified radiologists. In the image-to-text retrieval task, we first compute the similarity between the image and all candidate texts, and then rank the retrieved results. Similarly, in the text-to-image task, we compute the similarity between the textual prompts and all images, and rank the retrieval results. We report Precision at Top-1, Top-5, and Top-10, which reflect how many relevant examples are retrieved. As shown in Tab. 3, our model achieves the best results in both retrieve tasks. Our model outperforms the second-best model in the image-to-text and text-to-image retrieval tasks by 3.9%, 5.88%, and 4.33%, and 19.75%, 14.50%, and 12% in terms of P@1, P@5, and P@10 metrics, respectively. This indicates that our model has fully learned the relationship between images and texts, achieving better alignment effects. To further validate the model's performance, we conducted ablation experiments on the proposed EGF and EGM modules, while also assessing the impact of the proportion of eye-gaze data on the model results. As shown in the upper half of Tab. 4, the first row represents our Baseline model, where we utilize the initialized weights pre-trained on CheXpert [16] and MIMIC-CXR [20] datasets [55] . The second row "MLCE" indicates that within our EGF module, the EGF loss is not further computed beyond the Eq. 4, instead, only the multi-label cross-entropy (MLCE) loss between the eye-gaze guided similarity matrix and the model's output similarity matrix is calculated. The third row "EGF" utilizes the Eye-gaze Guided Fine-grained loss described in Eq. 6. The fourth row "EGM" indicates that the model is trained solely through the Eye-gaze Guide cross-model Mapping method. Finally, the fifth row presents our proposed EGMA model, which integrates the aforementioned modules guided by eye-gaze data. In Tab. 4, it can be observed that the method using only gaze-guided MLCE loss significantly improves performance compared to the baseline on CheXpert 5x200 dataset, with a slight improvement on RSNA but a severe decline on SIIM-ACR dataset. However, models using EGF or EGM show significant improvements on SIIM-ACR. This indicates that while MLCE improves performance on some datasets, it simultaneously reduces the model's generalization ability. Thus, relying solely on simple loss for similarity matrix is insufficient. In this work, by combining eye-gaze guided imagetext relationships with fine-grained feature alignment (EGF), although the model's performance slightly decreases on CheXpert 5x200, its overall generalization improves. Similarly, to enhance the model's multi-modal alignment ability, introducing eye-gaze guided cross-modal mapping results in improved performance and generalization, with EGM achieving optimal performance on RSNA dataset. Finally, when optimizing both fine-grained alignment and cross-modal alignment using eye-gaze, the model achieves dominant performance on all three datasets, demonstrating further enhancement in generalization.Numerous studies [24, 54, 35, 36] have demonstrated that training models using eye-gaze data can achieve comparable performance to models trained with fine-grained manual annotations. Meanwhile, the cost of collecting fine-grained manual annotations is significantly higher than that of collecting eye-gaze data. Therefore, incorporating eye-gaze into pre-training tasks is a feasible approach to enhancing model performance. To further validate the efficiency of our model using eye-gaze data, we conduct ablation experiments on the proportion of it in the training set. Our training dataset, MIMIC-EYE, consists of a total of 3695 samples. We perform ablation experiments using 1%, 5%, 10%, and 50% of the eye-gaze data, resulting in 37, 185, 370, and 1848 samples with prior information from radiologists, respectively. We repeat each experiment three times to eliminate the bias caused by random sampling, and report the average results. As shown in the lower part of Tab. 4, the model's performance on the CheXpert 5x200 dataset improved when trained with 1% of eye-gaze data. However, due to the limited data volume, the model's performance on other datasets is inferior to the baseline. When increasing the eye-gaze data to 5%, the model shows significant improvements on all three datasets. With the continuous increase in eye-gaze data, the performance of the model also improves. Therefore, even with a small amount of eye-gaze data (185 samples), our framework can effectively guide the model's multi-modal processing capability, ensuring performance enhancement. This further illustrates the applicability of our model and its low training cost characteristics. To better demonstrate the correspondence learned by the EGMA framework between text and radiographic images, we conducted a cross-modality attention maps visualization in Fig. 3 . Guided by eye-gaze data, the EGMA framework clearly outperforms other state-of-the-art methods in the field in accurately localizing disease regions. In Fig. 4 , we visualize the feature representations of CLIP [42] and our EGMA model on images of CheXpert 5x200 dataset using the t-SNE [50] . It can be observed that our model exhibits better clustering representation. The CLIP model, which was not trained on medical data, is unable to effectively differentiate these diseases. More results of t-SNE visualization can be referred to the Appendix E, clustering performance of other SOTA methods [15, 53] also inferior to our EGMA. 
discussion and conclusion: 
 In this work, we reveal the significant role of radiologists' eye-gaze data in multi-modal alignment and propose an Eye-gaze Guided Multi-modal Alignment framework called EGMA. In fact, the EGF and EGM modules in our EGMA framework are similar to the diagnostic process of radiologists.Many studies [39, 46, 30, 8] have pointed out that radiologists perform a global search first and then conduct a detailed examination of the local areas when suspicious lesions are found. Firstly, the EGF module of EGMA corresponds to the global search by radiologists, aligning the local image patch features with the individual sentence features. Secondly, EGM uses the key values from the local similarity matrix computed in the first step as weights, focusing the alignment on certain important image patches and texts, akin to the detailed observation stage after identifying suspicious lesions. Thus, by mimicking the real diagnostic behavior of radiologists, our EGMA further enhances the model's multimodal processing capability and diagnostic accuracy. We evaluate EGMA's zero-shot capabilities and fine-tuned performances on multiple datasets and observe significant improvement in classification and retrieval tasks. Additionally, we investigate the impact of eye-gaze data scale on performance, finding that even small amounts of eye-gaze data can enhance the model's multi-modal alignment capabilities during pre-training. Overall, our EGMA framework explores the feasibility of incorporating eye-gaze data from radiologists to assist in multi-modal feature alignment during model training, laying the foundation for the application of eye-gaze data in the medical multi-modal domain.Limitations and Discussion Our work only compared state-of-the-art methods in classification and retrieval tasks, without conducting downstream tasks such as lesion localization or segmentation. Additionally, our model heavily relies on multi-modal datasets like MIMIC-EYE [14] , which can simultaneously collect eye-gaze data, medical images, and diagnostic text. The scenarios for collecting these data are also a significant consideration. For instance, in clinical ultrasound diagnosis [38] , radiologists often use both hands to operate the equipment and verbally communicate their diagnostic information to an assistant. In this context, it is convenient to simultaneously record ultrasound images, eye-gaze data, and audio. In contrast, during chest X-ray diagnosis in MIMIC-EYE, radiologists typically record diagnostic information directly in text form rather than verbally. Fortunately, some recent efforts [25, 35] are focusing on how to naturally collect multi-modal data of radiologists during diagnosing. They have designed more flexible collection systems that better accommodate the routine work of radiologists, which is crucial for the widespread adoption of collecting multi-modal diagnostic data such as eye-gaze information.Potential Impacts Although the eye-gaze data we used is publicly available and we have permission to use it, some studies [27, 23] have indicated that private information such as gender, age, and mental state of observers can be extracted from eye-gaze data. Therefore, privacy concerns have always been a focal point in using eye-gaze data. To address this, we recommend using de-identification methods to filter eye-gaze data or releasing the data in the form of heatmaps rather than the raw data.Future Work In the future, we will continue to optimize these proposed collection systems [27, 23] and explore the guidance role of eye-gaze data between images and handwritten diagnostic reports to accelerate their application in real medical diagnostic scenarios. This will provide a research foundation to alleviate data annotation pressure and enhance model interpretability. Additionally, we will continue to analysis the eye-gaze features, such as temporal features, and further optimize their role in multi-modal feature alignment. We believe this work can serve as a valuable reference for the application of eye-gaze data in multi-modal frameworks and promote its development in the field of medical multi-modality. Moreover, thanks to the inherent flexibility of the EGMA model, it is well-suited for multi-modal alignment tasks involving natural images, such as in human-robot interaction and control as well as for education/training purposes. Therefore, we believe this is a promising direction for future expansion. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The full details are provided with the main context and supplemental materials. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The paper reports the statistical significance of the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: The paper describes safeguards that have been put in place for responsible release of data or models that have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Meet the requirements.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Meet the requirements.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Not Applicable.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: Not Applicable.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This appendix is organized as follows. In Sec. B, we provide more experimental settings, including training parameters, detailed parameters of image and text encoders. In Sec. C, we introduce the detailed information of the datasets used in this work. In Sec. D, we provide more details of multi-modal data processing of MIMIC-EYE [14] dataset. In Sec. E, we provide additional visualization results of feature representation. In Sec. F, we provide additional experiments of zero-shot classification task after continue pre-training using the backbones of other SOTA models in our EGMA framework.In this study, we use SwinTransformer [33] as the image encoder, BioClinicalBERT [1] as the text encoder. Specifically, we use a 4 stages SwinTransformer, including 2, 2, 6, and 2 SwinTransformer blocks. Other parameters are: patch size 4; window size 7. And we use a 6 layers BioClinicalBERT with 12 attention heads. In our EGMA framework, we add a linear projection layer after both the image encoder and text encoder to map the embeddings' dimension to 512, and we use a learnable temperature τ in contrastive loss calculation initialized on 0.07.Pre-training Settings In the pre-training process, we utilize the following image augmentations to the chest X-ray images: scale to images to 224 × 224; color jittering with brightness and contrast ratios from [0.8, 1.2]; randomly change the contrast(probability = 0.5). And we train our model with 50 epochs with an initial learning rate 1 × 10 -6 and weight decay 1 × 10 -4 and 10 epochs of warm-up.Fine-tuning Settings In the supervised classification experiments, we adopt the linear classification settings [15] , where the pre-trained image encoder is frozen, and only a randomly initialized linear classification head is trained. We choose the same image augmentations to the above pre-training settings. And we fine-tune our model with 30 epochs with an initial learning rate 5 × 10 -7 and weight decay 1 × 10 -4 and 6 epochs of warm-up. And all our training tasks are completed on four RTX 3090 GPUs.The MIMIC-EYE [14] dataset includes a comprehensive range of patient information, including medical images and reports, clinical data, patient's hospital journey, and eye-tracking data and audio of radiologists during diagnosis. The dataset comprises a total of 3689 images from the MIMIC-IV v1.0 dataset [18] , each accompanied by transcripts text from audio and eye-tracking data of radiologists. In this work, we use this dataset as our training set.CheXpert [16] is a large-scale public dataset for chest radiograph interpretation, developed by a team from Stanford University. The dataset comprises 224,316 chest radiographic images involving 65,240 patients, annotated for the presence of 14 common chest radiographic findings [13] . These annotations are categorized into three types: positive, negative, or uncertain. In our study, we follow [15] and [59] , using two subsets of this dataset, namely CheXpert 5x200 and CheXpert 8x200, for our zero-shot classification and zero-shot retrieval testing tasks. The CheXpert 5x200 dataset [15] comprises five common chest diseases, Atelectasis, Cardiomegaly, Consolidation, Edema, and P leural Ef f usion, each with 200 chest X-rays. In [15] , a radiologist provided possible sub-types, severities, and locations for these five diseases. As depicted in Tab. 5, all combinations of these three types of information form the text queries for CheXpert 5x200 dataset. In the zero-shot classification task, image embeddings are compared with the embeddings of these text queries, and the class with the highest similarity is assigned as the predicted classification for the image. The CheXpert 8x200 dataset [59] comprises eight categories, N oF inding, Cardiomegaly, Edema, P neumonia, Atelectasis, P neumothorax, P leural Ef f usion, and F racture, each with 200 images. In [59] , a radiologist expert was also invited to compose five expert queries for each category, used for image-text retrieval tasks. Specific queries are detailed in Tab. 6.The RSNA Pneumonia Detection Dataset [44] , encompasses a comprehensive set of medical imaging data types, including X-rays, CT (Computed Tomography), and MRI (Magnetic Resonance Imaging) images. In this work, we utilized the stage 2 version of this dataset, comprising 29,700 chest X-ray images categorized into normal and pneumonia positive category. Following [15] , we allocated 15% of this dataset for our zero-shot classification testing set. And we utilize the text queries from the "no finding" and "Pneumonia" categories in the CheXpert 8x200 dataset as the text queries for zero-shot classification in this data.The SIIM-ACR [45] dataset is a chest dataset used for pneumothorax classification and segmentation. It consists of 2379 images with pneumothorax and 8300 images without pneumothorax. In this study, we utilized a subset of the dataset filtered by Saab et al. [43] as the test data to evaluate the zero-shot classification performance of the model for pneumothorax disease. And we utilize the text queries from the "no finding" and "Pneumothorax" categories in the CheXpert 8x200 dataset as the text queries for zero-shot classification in this data.As illustrated in Fig. 5 , the presentation of multi-modal data in the MIMIC-EYE [14] dataset includes radiologists' audio, text transcript, eye-gaze data, and image. Since each modality is synchronized, the audio data is aligned with the eye-gaze data in time. By segmenting the audio based on the time before and after the pronunciation of each word, we can align the transcripts with the audio, thereby aligning word-level text with eye-gaze data. Subsequently, we generate attention heatmap based on eye-gaze data and images to represent the image regions the radiologist focuses on. Through the aforementioned data processing steps, we achieve precise alignment between word-level text and image regions. It is noteworthy that due to the rapid speech rate of radiologists, there may be no available eye-gaze data within the time interval corresponding to a single word. In Fig. 5 , the word "with" in the transcript has no corresponding gaze data. Another common and unavoidable issue is the loss of eye-gaze data caused by blinking and intense head movement of radiologist, as seen in the last two words of the transcript. Due to these technical challenges, achieving perfect pairing between words and image regions is difficult. However, as shown in the right side of Fig. 5 , adjusting the text to the sentence level largely mitigates the issue of missing word-level heatmap (Heatmap with red edge), and the semantic information of the entire sentence also encompasses the information of each word. Therefore, in this work, we process text features at the sentence level.During the pre-training of EGMA, the size of the input heatmap is determined by the number of patches in the image. For example, after the image is processed by the image encoder, the size of image embedding is 196 × 768, where 196 represents the number of image patches. Therefore, we resize the heatmap directly to 14 × 14 to match the image embedding and further process it into the Gaze-guided Similarity and Gaze-guided Label mentioned in the main manuscript.In fact, noise and errors in eye-gaze data are as common as noise in images, and these can all affect the model's final performance. In this work, the errors in eye-gaze data primarily stem from two factors: involuntary saccades and microsaccades of the radiologists' eyes [9] , and subjective fixation errors [3] .Human eye movements can be categorized into two main types: saccades and fixations [9] . Saccades are the rapid movements between different areas of the visual field and are generally considered noise data that do not involve cognitive processes. Fixations, on the other hand, are brief pauses of the eyes on a small area and are regarded as the primary cognitive behavior. Additionally, due to the structure of the eye, fixations are accompanied by microsaccades, causing the fixation point to drift within the fixation area. Thus, microsaccades are also a major source of noise in eye-gaze data. Fortunately, eye-tracking technology has evolved significantly over the past century, and noise reduction techniques for eye-gaze data have become highly advanced. Currently, all commercial eye-tracking devices come with preprocessing software that uses adaptive methods (e.g., [41] ) to filter out noise and output fixations, greatly facilitating the use of eye-tracking technology. Fig. 6 shows an example of MIMIC-EYE data used in our work. All gaze data is denoised by the filter operation, ensuring that noise from blinks, saccades, and microsaccades of the radiologists has been eliminated.Additionally, due to variations in radiologists' expertise and cognitive levels, fixation data may contain errors, such as the reviewer mentioned, "occasionally shift to locations not exactly on the objects." For the minor shift error of fixations, where the duration of incorrect fixations is small compared to the overall duration, their values in the attention heatmap are low, thus having minimal impact on the mlce loss utilized in our work (Eq. 4 in main paper). In our EGMA, EGF module (Sec. 3.2 in main paper) selects the highest similarity value of local features before averaging, ensuring that these minor errors do not affect the model. Additionally, in the EGM module (Sec. 3.3 in main paper), the sparse operation for alignment weight also filters out minor errors in the heatmap. For the significant fixation errors, MIMIC-EYE data has been cleaned to address this issue at the first time. Specifically, during data collection, radiologists were required to provide a diagnostic label. These data were then reviewed by professionals to exclude inconsistent diagnoses and fixation errors. Moreover, the MIMIC-EYE dataset includes eye-gaze data from six radiologists. Even if one or two radiologists' fixations for a specific diagnostic text are incorrect, the correct fixation data from the other radiologists under the same semantic context can mitigate the adverse effects. Fig. 7 shows an example of this compensation. In summary, EGMA was designed with considerations for the noise and errors inherent in eye-gaze data from the beginning. And effectively identifying and mitigating errors in radiologists' diagnostic behavior during model training will be a focus of our future work.Numerous studies have investigated and proven the close relationship between radiologists' gaze behavior, image content, and diagnostic results. For example, studies [40, 52, 11, 49, 48] have found that radiologists fixate more on diseased areas than on healthy ones. Moreover, they discovered that novice radiologists, due to their lack of experience, repeatedly look at diseased areas, resulting in even more fixations in these areas than those of expert radiologists. Fig. 8 shows some comparison cases. Additionally, the abnormal regions with more fixations have higher weights in the attention heatmap. Our EGMA model can leverage these weights, along with image content and text, to learn better disease diagnosis capabilities and feature representations. In Fig. 9 , we visualize the feature representations of CLIP [42] , GLoRIA [15] , MGCA [53] , and our EGMA model on images of CheXpert 5x200 dataset using the t-SNE [50] . It can be observed that our model exhibits better clustering representation. The CLIP model, which was not trained on medical data, is unable to effectively differentiate these diseases. Additionally, while the representation capability of GLoRIA and MGCA has improved noticeably, their clustering performance still inferior to our EGMA.To further validate the generality of our framework, we utilize the encoders and pre-training weights provided by CLIP [42] , GLoRIA [15] , and MGCA [53] in our EGMA framework. Subsequently, we continue training on the MIMIC-EYE [14] dataset and present the zero-shot classification results on the CheXpert 5x200 [16] , RSNA [44] , and SIIM-ACR [45] datasets in Tab. 7.We present accuracy Accuracy and F1 score metrics on three datasets. The values in parentheses indicate the improvement over the baseline metrics (as shown in Tab. 2). It can be observed that, all models show improvement after training with our EGMA framework, except for the decrease in metrics for the trained MGCA model on the RSNA dataset. For MGCA, when tested with its provided pre-trained weights, it performs the best F1-score on the RSNA dataset (as shown in Tab. 2), but after training with the EGMA framework, its performance improves on CheXpert 5x200 and SIIM but decreases on RSNA dataset. This may reflect that the features extracted by MGCA on Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.https://github.com/MoMarky/EGMA 


Decision: This paper introduces an eye gaze guided vision language learning to improve visual and text representation learning for medical applications .  It receives one strong accept , one accept, and two borderline accepts . The reviewers in general like this work, finding its ideas using gaze to guide learning novel, papers well written, with extensive experiments that demonstrate the superior performance of their method.  They have some concerns about data quality, data accessibility, and data privacy.  The authors rebuttal is extensive and well addresses the reviewers concerns .  I hence recommend its acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes utilizing eye-gaze information to aid in learning representations from paired images and texts.  The method was validated on four medical datasets, demonstrating the feasibility of integrating gaze information for multi-modal alignment.
strength_weakness: strength_weakness: First, if eye-gaze coordinates are noisy and contain errors, introducing eye-gazing into the learning process could negatively impact representation learning with image-text pairs. The image-text pairs are well-defined and reliable data resources. In contrast, the quality of eye-gaze information can be highly dependent on the operators, working conditions, and devices. Introducing potentially noisy and/or unreliable information into the learning process is risky and may lead to degraded performance.This paper does not mention and address this issue and the proposed method lacks for dealing with potentially noisy gaze information (could happen.)

Eye-gaze locations may fall on the cardiac areas in both normal and abnormal cases, but the critical consideration in learning is to develop a representation that can distinguish between these situations. Eye-gazes might provide little to no useful information for this learning process, as there is can be no significantly clear distinction between the eye-gaze coordinates in abnormal versus normal cases. Introducing eye-gaze data is not useful for learning to distinguish between these conditions. The only benefit might be identifying the cardiac area, which can be easily captured by conventional contrastive learning without the need for explicitly provided location information.

The writing is poor. The notations and formulas used in the method section are unprofessional. Notations such as COS, Mean, and Max are too casual and should be changed to be more rigorous and formal.
questions: questions: If the gaze information is not very accurate, for example, if the gazes occasionally shift to locations not exactly on the objects (e.g., cardiac), would using the gaze information harm the learning performance? If not, why, and if yes, how would you handle such an issue?
limitations: limitations: Please see weakness.
paper_summary: paper_summary: This paper proposes a cross-modal alignment method that can optionally learn from eye tracking data that is collected together with the speech of radiologists. The proposed Eye-gaze Guided Multi-modal Alignment (EGMA) system consists of losses that optionally incorporate alignment objectives between sentences and gaze regions (expressed as heatmaps) which allows the contrastive learning scheme to out-perform existing vision-language models for radiology, such as MedCLIP. This is in terms of linear evaluation (supervised classification), zero-shot classification, image-to-text retrieval and text-to-image retrieval. The EGMA method also shows highly interpretable cross-modal attention maps and well clustered embeddings (with respect to known ground-truth classes).
strength_weakness: strength_weakness: Despite the authors’ best efforts, Fig. 2C and Fig. 2D (and the corresponding Sec. 3.2 and 3.3) are hard to understand. In particular, it is unclear to me why the Mean(Max()) reduces features are considered to be “fine-grained” (it rather seems to be an instance-level feature that allows for gaze-free supervision), and why the cross-modality mapping is necessary.
questions: questions: Please clarify how the Mean(Max()) operation described in Sec. 3.2 enables fine-grained alignment.

I would also like to understand the motivation for the EGM module, in relation to existing prior art.

Thank you.
limitations: limitations: The authors are very forward and clear about the limitation of using or collecting data where gaze and speech are aligned. However, they also mention that medical practises are evolving to allow for this better. In this reviewer’s opinion, the framework proposed in this paper could be easily applied to non-medical areas as well as its design is domain-agnostic.

Privacy or societal concerns are also adequately mentioned.
paper_summary: paper_summary: The paper proposes EGMA, a novel framework for medical multi-modal alignment integrating eye-gaze data into vision-language pre-training. EGMA outperforms existing methods in image classification and image-text retrieval tasks, demonstrating significant advancements and improved feature representation with even minimal eye-gaze data.
strength_weakness: strength_weakness: The paper primarily focuses on classification and retrieval tasks. Including additional evaluations, such as lesion localization or segmentation tasks, could provide a more comprehensive assessment of the framework’s effectiveness.

The use of eye-gaze data raises potential privacy issues, as such data can inadvertently reveal sensitive information about the observers. Addressing these concerns through robust de-identification methods would strengthen the ethical considerations of the work.

Although the paper includes ablation studies, more detailed analysis on the impact of different components of the model, such as varying the amount of eye-gaze data or different types of medical images, could provide deeper insights into the robustness and adaptability of the proposed framework.

The model relies on multi-modal datasets like MIMIC-EYE, which simultaneously collect eye-gaze data, medical images, and diagnostic text. This dependency might limit the generalizability of the approach to other datasets that lack such rich multi-modal annotations.
questions: questions: The paper mentions that even a small amount of eye-gaze data can enhance model performance. Can you provide more quantitative results or analysis on how performance scales with different amounts of eye-gaze data?

Have you considered applying the EGMA framework to other types of medical imaging modalities (e.g., MRI, CT scans) or diagnostic tasks? If so, what challenges or modifications would be necessary?
limitations: limitations: The authors discuss privacy concerns associated with using eye-gaze data, suggesting the use of de-identification methods or releasing data in the form of heatmaps to mitigate these issues.
paper_summary: paper_summary: This paper introduces the Eye-gaze Guided Multi-modal Alignment (EGMA) framework, which leverages radiologists' eye-gaze data to enhance the alignment of medical visual and textual features. By using synchronously collected eye-gaze data during diagnostic evaluations, EGMA improves generalization and achieves state-of-the-art performance in image classification and image-text retrieval tasks across four medical datasets. The study also investigates the impact of varying amounts of eye-gaze data on model performance, demonstrating the feasibility and utility of integrating this auxiliary data into multi-modal alignment frameworks.
strength_weakness: strength_weakness: Dependency on Eye-Gaze Data Collection:
Issue: The reliance on eye-gaze data from radiologists introduces a significant dependency that may not be feasible for all institutions due to the need for specialized equipment and the additional effort required to collect this data. Especially when the eye-gaze prior is not accurate.
Impact: This dependency could limit the scalability and widespread adoption of the EGMA framework, particularly in resource-constrained settings or where the collection of eye-gaze data is not practical.
questions: questions: Impact of Eye-Gaze Data Quality and Quantity:
Question: How does the quality and quantity of eye-gaze data affect the performance of the EGMA framework? Are there specific thresholds or optimal amounts of eye-gaze data required to achieve the best results?
Context: Detailed insights into the relationship between eye-gaze data quality/quantity and model performance would help in optimizing the framework and understanding its practical requirements.
limitations: limitations: No

Comments: comment: Thank you very much for the detailed and kind answers.

I understand the EGF and EGM losses slightly better after reading your explanation. One follow-up question would be whether the EGF loss, as it expects fine-grained feature consistency (specifically, that the max value corresponds to the correct location of interest), would require that the image encoder is pre-trained prior to pre-training. I see in Sec. B.1 that you mention the use of SwinTransformer and BioClinicalBERT as the image and text encoders, respectively, but it is not clear to me whether you begin from the publicly available pre-trained weights.

I would think that the EGF loss may not work very well when training from scratch (as taking the max can lead to improper features being selected).

In any case, I will retain my rating as I believe the authors have addressed both mine and most other reviewers' concerns. The authors need not respond to me.
We appreciate the timely response. We are pleased to know that we have successfully addressed your concerns.

Thanks for engaging with our work and adjusting your evaluation.
I have read the authors' rebuttal which has addressed my concerns to some degree. I have raised my rating accordingly. Thanks.