GazeDiff: A radiologist visual attention guided diffusion model for zero-shot disease classification

Authors: moinak  bhattacharya
Venue: MIDL_io_2024_Conference
Published: 2024-01-27 19:40:44

Abstract: We present GazeDiff, a novel architecture that leverages radiologists' eye gaze patterns as controls to text-to-image diffusion models for zero-shot classification. Eye-gaze patterns provide important cues during the visual exploration process; existing diffusion-based models do not harness the valuable insights derived from these patterns during image interpretation. GazeDiff utilizes a novel expert visual attention-conditioned diffusion model to generate robust medical images. This model offers more than just image generation capabilities; the density estimates derived from the gaze-guided diffusion model can effectively improve zero-shot classification performance. We show the zero-shot classification efficacy of GazeDiff on four publicly available datasets for two common pulmonary disease types, namely pneumonia, and tuberculosis.

Main paper: 
 introduction: 
 Understanding radiologists' eye gaze patterns is crucial to deciphering the intricacies of spatial presentation of disease patterns in radiological scans. This auxiliary signal, in the form of eye gaze maps, has been recently harnessed by deep learning systems for medical image diagnosis (Bhattacharya et al., 2022b,a) . Medical experts dedicate years to honing their skills in diagnosing diseases from radiology images, meticulously mastering the identification of intricate disease patterns. This experience enables their visual-cognitive working mechanism to modify/improve with time and, in turn, finesses their way of looking at scans (Bertram et al., 2016; Kelahan et al., 2019; Tourassi et al., 2013) . Hence, the visual patterns of an expert can provide critical sub-visual information for a deep learning model to improve its meta-understanding of a radiology image (Stember et al., 2020) .Several works have been done in the last decade on generative modeling with a special focus on content generation. Recently, a significant improvement has been made on this front with diffusion models. Diffusion models are likelihood-based generative models that model the data distribution via an iterative noising and denoising procedure (Ho et al., 2020) and achieve state-of-the-art performance in text-based image generation. Improvements have been made in diffusion models by introducing additional controls (Zhang et al., 2023) . Furthermore, conditional generative models can be converted to a classifier (Ng and Jordan, 2001) and, similarly, text-to-image diffusion models can be used as zero-shot classifiers without any additional training (Li et al., 2023a) . This can be achieved by repeatedly adding noise to the input image and computing a Monte Carlo estimate of the expected noise reconstruction losses for every class in the dataset. Recent advances in controllable diffusion models enable text-to-image diffusion models with additional controls for guided image generation. Providing additional controls can be done in two ways: a) training the diffusion models from scratch (Huang et al., 2023) , and b) introducing light-weight adapters into pretrained diffusion models (Zhang et al., 2023; Li et al., 2023b; Mou et al., 2023) . More recently, multiple controls are also used to generate more diverse images (Zhang et al., 2023; Zhao et al., 2023a; Qin et al., 2023) . Deep learning models can recognize a shape better if they can learn to generate better (Hinton, 2007) . This fact goes back deep into the psychological paradigm of mechanisms to improve deep models, where generative modeling can act as a crucial player in discriminative tasks like classification. In medical image generation tasks, clinically explainable conditions and text conditions are important in generating realistic radiology images; this is still relatively unexplored. Radiologists' eye gaze patterns are strong clinical meta-features that are highly relevant in understanding disease patterns and associated diagnoses. Can these eye gaze patterns serve as suitable controls for diffusion models? In this work, we propose a novel approach to integrate this expert visual attention as an additional control to the text-to-image diffusion models. Here, the text condition is the radiologist's transcript while viewing an image and contains disease-specific and context-rich information. Our proposed architecture, GazeDiff, utilizes these text conditions and visual attentions as additional controls for medical image generation (Figure 1 ).Even though machine learning models can benefit from experts' eye gaze patterns, it is time-consuming, expensive, and often impractical to obtain eye gaze in real-time decisionmaking scenarios. We address this problem by adapting GazeDiff as a zero-shot classifier. Similar to (Li et al., 2023a) , the gaze-conditioned stable diffusion model is used as a zeroshot classifier without any additional training. In our work, we show that the proposed method outperforms the baselines in classifying both known and unknown classes. In summary, we propose a novel gaze-guided zero-shot diffusion classifier, GazeDiff, for pulmonary disease classification. Motivation and Overview. The motivation for our work stems from generating clinically accurate medical images. We hypothesize that the context-rich visuo-cognitive information of radiologists' eye gaze patterns can be used as a clinically-relevant condition for image generation. To do this, first, we add the eye gaze patterns of experts as additional controls and radiologists' transcripts as text conditions to the text-to-image diffusion models. Then, we show that this helps in generating clinically accurate images and finally we use this finetuned diffusion model as a zero-shot classifier for downstream pulmonary diseases like pneumonia and tuberculosis classification tasks. The key contributions of this paper are as follows: a) we propose to add eye gaze patterns of experts as additional control and radiologists' text as prompt to the text-to-image diffusion models. b) we use this finetuned diffusion model as a zero-shot classifier for downstream pulmonary diseases like pneumonia and tuberculosis classification tasks. Eye gaze patterns HVA mapEnc. + Noise schedule ) is locked and a trainable copy is created connected with zero convolution blocks to train with radiologists' eye gaze patterns as conditions. Then, a chest xray image χ and classifier prompt are fed to the finetuned gaze-conditioned model (CN-Gaze). This is used as a zero-shot classifier using the classifier objective.𝜖 ! ℱ % 𝔾 f , ℱ % 𝔾 g 𝑦 𝔾 argmin ! !⊕# $ 𝐸 ",$ 𝜖 -𝜖 % I " , 𝑐 &⊕( $ ) (I,𝔾ʹ) SD-CXR CN-Gaze 
methodology: 
 Figure 2 shows the pipeline on the GazeDiff architecture. GazeDiff consists of three major components: the Stable Diffusion (SD-CXR) model (Rombach et al., 2022) (shown in yellow and red), the ControlNet-Gaze (CN-Gaze model) that enhances the SD-CXR with more radiologists' eye gaze patterns as additional controls, (shown in green), and finally the class conditioned density estimates are calculated from the CN-Gaze model, (shown in blue) First, we provide a preliminary overview of diffusion models. In Section 2.1, we propose a method to use radiologists' eye gaze patterns as additional controls for text-to-image diffusion models, and in Section 2.2, we discuss a technique to use diffusion models as zero-shot classifiers.Preliminary. Diffusion probabilistic models (Ho et al., 2020) or diffusion models are generative models with a parameterized Markov chain trained using variational inference.Let us consider an input image x, the diffusion or forward process (shown in yellow in Figure 2 ) is a fixed Markov process that adds Gaussian noise ϵ ∼ N (µ, σ 2 ) to x, according to a variance schedule β = {β 1 , ..., β T }, shown as q(x 1:T |x 0 ) := q(x t |x t-1 ), where q(x t |x t-1 := N (x t ; √ 1 -β t x t-1 ; β t I)). And, the reverse process(shown in red in Figure 2 ) is a learned Gaussian transition to denoise x, which can be conditioned on a variable c, shown as p θ (x 0:T ) := p(x T ) p θ (x t-1 |x t ). In our case, x is a CXR image, c is the radiologist's findings (radiologist's transcripts and disease labels), and T is the number of timesteps. So, diffusion models define x 0 conditioned on c asp θ (x 0 |c) = x 1:T p(x T ) p θ (x t-1 |x t , c)dx 1:T with p θ (x t-1 |x t ) := N (x t-1 ; µ θ (x t , t), σ θ (x t , t)). Now, the diffusion model is trained to minimize the variational lower bound (ELBO) of the log-likelihood, defined as,log p θ (x 0 |c) ≥ E q log p θ (x 0:T , c) q(x 1:T |x 0 ) (1)First, we train the SD with CXR images and we term this model SD-CXR. Then, radiologists view a CXR image I and generate eye gaze patterns G, discussed in detail in 2.1. Similar to (Bhattacharya et al., 2022b,a) , Human Visual Attention(HVA) maps are computed from G. In this work, we compute separate Focal HVA and Global HVA maps. The focal HVA captures fine-grained disease-relevant features while the global HVA captures coarse disease-relevant features; a weighted combination of these maps captures the entire feature space of disease-relevant regions, discussed in detail in Appendix B.Here, we discuss CN-Gaze, in which the radiologists' eye gaze patterns are injected as additional conditions into the SD-CXR model. Let us assume that the SD-CXR model is F(.). We represent radiologists' eye gaze patterns as G r ∈ R (F,T ) , where r ∈ {1, 2, ..., R}.Here, F are the eye gaze fixations over time T and R is the number of radiologists whose eye gaze are collected for CXR image I ∈ R H×W ×C with {H, W, C} are height, width, and number of channels of the image. Now, HVA edge maps are computed for Global and Focal HVA from G, represented as G ′ (discussed in detail in the Appendix B). Similar to (Zhang et al., 2023) , for training with additional control G ′ , we freeze F(.) with initial parameters represented as Θ and clone the frozen model parameters into a trainable model to train with the gaze condition, shown as Θ ′ G . The input I is fed to both F Θ (.) and F Θ ′ G (.) in a manner where F Θ ′ G blocks are connected to F Θ blocks through zero convolution(represented as Z(.), which is a Conv 1×1 layer with W = 0 and b = 0) layers, as shown in Figure 2 . The output y ′ G is shown asy G = F Θ (I) + Z 2 (F Θ G (I + Z 1 (G))). In our case, the radiologist's eye gaze patterns are represented as two separate entities, namely, Focal HVA and Global HVA. Hence, two separate ControlNets are trained, F Θ f , and F Θg . The resulting outputs from these ControlNets are added with no extra weighting or linear interpolation to make it a Multi-ControlNet F Θ f . Hence, y ′ G can be represented as a weighted combination of focal-conditioned, y f and global-conditioned, y g , shown as,y G = λ 1 y f + λ 2 y g , λ 1 , λ 2 ∈ R + , y f = F 1 Θ (I) + Z 1 2 (F 1 Θ G f (I + Z 1 1 (G f ))) y g = F 2 Θ (I) + Z 2 2 (F 2 Θ Gg (I + Z 2 1 (G g )))    HVA (2)In common medical scenarios, during real-time inference, radiologists' eye gaze patterns and transcripts are not available. Here we discuss how CN-Gaze is used for for zero-shot classification. Given each noised samplex t = √ α t x + √ 1 -α t ϵ, diffusion model learns ϵ θ (x t , c). Using this parameterization, Equation (1) can be rewritten as, -E[ T t=2 w t ∥ϵ -ϵ θ (x t , c)∥ 2log p θ (x 0 |x 1 , c)] + C. From (Li et al., 2023a) , assuming w t = 1 and log p θ (x 0 |x 1 , c) ≈ 0 as T = 1000 is large, the simplified ELBO term is represented as -E t,ϵ ∥ϵ -ϵ θ (I t , c)∥ 2 + C. Now, classification tasks using generative models can be defined using Bayes Theorem asp θ (c i |I) = p(c i )p θ (I|c i ) j p(c j )p θ (I|c j ), where c i is the label, and I is the input image. Using the simplified ELBO term, it can be re-written asp θ (c i |I) = exp -Et,ϵ[∥ϵ-ϵ θ (It,c i )∥ 2 ] j exp -Et,ϵ[∥ϵ-ϵ θ (It,c j )∥ 2 ]. In our case, from Equation ( 2), p θ (c i |I) can be rewritten as p Θ (c (f ⊕g) i |I), shown asEQUATIONThen, an unbiased Monte Carlo estimate is calculated for each expectation by samplingN (t i , ϵ i ), shown as 1 N N i=1 ϵ i -ϵ Θ ( √ α t I + √ 1 -α t ϵ i , c j )2 . Now, plugging this formulation into Equation (3) makes the zero-shot GazeDiff classifier. 
datasets and experiments: 
 We use the radiologist's eye gaze and corresponding transcriptions from publicly available Eye Gaze Data for Chest X-rays (Karargyris et al., 2021 (Karargyris et al., , 2020) ) (Canny, 1986) of the global and focal HVA maps separately, as shown in Figure 3 . This training was performed for 10 epochs with a batch size of 2 and a learning rate of 1e -5. During inference, we merge the global and focal ControlNets with UniPCMultistepScheduler (Zhao et al., 2023b) for sampling. The number of time steps is set to 50 and the condition scale is set to 2.5. Also, we use ℓ 2 norm for ϵ-prediction error.We show results on the known class (samples from this class present in training SD-CXR and CN-Gaze), i.e. pneumonia classification, and the unknown class (samples from this class not present in training SD-CXR and CN-Gaze), i.e. tuberculosis classification. Evaluation of Zero-shot classification Performance. GazeDiff is compared against standard diffusion models like SD (Ho et al., 2020) , ControlNet (Zhang et al., 2023) trained on natural images, and the same models finetuned on CXRs (i.e. SD-CXR and CN-CXR). We also evaluate the performance of GazeDiff against RoentGen (Chambon et al., 2022) , a baseline SD model finetuned with CXR images. To measure the classification performance, we report Accuracy(↑) and F1-score(↑) in Table 1 , and we report additional metrics in Appendix Table 4 . GazeDiff outperforms the baselines on all 4 benchmark datasets for Pneumonia and Tuberculosis classification. We observe that GazeDiff outperforms the finetuned SD model by 0.62±0.48% [Cell: 1.35%, RSNA: 0.60%, CHN: 0.51%, MCU: no improvement], the finetuned ControlNet model by 1.15±0.65% [Cell: 2.16%, RSNA: 0.66%, CHN: 0.51%, MCU: 1.27%] and RoentGen by 15.60±4.55% [Cell: 8.90%, RSNA: 17.13%, CHN: 14.87%, MCU: 21.51%]. We also show comparisons with CLIP (Radford et al., 2021) and PubMedCLIP (Eslami et al., 2021) . Evaluation of image quality. In Table 2 , we report FID(↓), and CLIP-score(↑) to evaluate the performance of GazeDiff for generated images quality and compare it with ControlNet. We show that GazeDiff outperforms ControlNet on 4 pulmonary disease types. Additional results are reported in Appendix Table 5 and Table 6. In Table 3 , we show the performance of GazeDiff, when trained with different human visual attentions. From a radiologist's eye gaze patterns, we calculated two different visual attention patterns, namely global attention and focal attention, described in detail in Appendix B. Here, we observe that the ControlNet finetuned with the combined global and focal attention mechanisms generate better noise representation for zero-shot classifications and outperform the Global model by 1.35% and the Focal model by 1.62%. In Figure 3 , we show the generated CXRs for the different human visual attention canny edge maps. Here, we observe that the generated images show distinct irregularities in locations where there are canny edges of the human visual attentions. This demonstrates the robust interpretation of the experts' eye gaze content semantics for medical image generation.In Figure 5 , we compare the generated CXRs from GazeDiff with different baselines like Stable Diffusion, ControlNet, and RoentGen for pneumonia and tuberculosis disease names as text conditions. Here, we observe that the GazeDiff generates more realistic disease patterns when compared to the baselines. We also show the location of the generated "left ij catheter is in the svc. right picc line is at the cavoatrial junction. there is a left ventricular assist device. cardiac pacer is noted with leads in the right ventricle and coronary sinus. there is moderate to severe cardiomegaly. mild diffuse hazy opacity likely represents pulmonary edema. there is left basilar atelectasis or consolidation.no pneumothorax. no acute osseous abnormality."Original Image Radiologist's transcript Generated Image disease patterns annotated by a radiologist (7 years experience). In Figure 4 , we show the generated CXRs of the proposed method for different transcriptions as text conditions. We show the location of the generated disease patterns annotated in red. GazeDiff not only generates disease patterns/irregularities and devices (text highlighted in red) as mentioned in the transcript but also generates them in the mentioned location (text highlighted in blue).Prompt: A photo of chest x-ray with pneumonia Prompt: A photo of chest x-ray with tuberculosis SD-CXR CN-CXR RoentGen GazeDiff  
conclusion: 
 In this work, we show that eye gaze patterns are indeed critical in generating high-fidelity and robust medical images. With eye gaze patterns as conditional inputs to diffusion models, the generated medical images contain disease-specific features or irregularities in the exact location or periphery of radiologists' eye gaze fixations. This makes the medical image generation more robust and disease-specific, possibly minimizing hallucinations. In another important contribution of our work, we show that the diffusion model trained with eye gaze patterns can also be used as a zero-shot classifier for two common disease types without any additional training. This work is a significant step forward in gaze-conditioned medical discriminative tasks using a generative modeling approach. However, there is a scope for improvement in zero-shot classification performance by selecting prompts that are more gaze-relevant and hence induce more diversity in the image generation step.dimensional Gaussian filter with standard deviation, σ = 64, is used to generate these attention heatmaps, shown in Figure 7 . 
 
 Appendix: Focal HVA Global HVA Focal HVA edge Global HVA edge  


Decision: This paper proposes a novel text-to-image diffusion models for zero-shot classification of CXRs that also uses radiologists' gaze for conditioning. In general the ideas was thought to be interesting and the use of more medical information in image generation was thought to be a strength. The reviewers had some questions about contributions of the work, minor improvements against methods compared against and lack of evaluation by radiologists, these appear to have been resolved in the rebuttal. Even though the improvements are minor, it was pointed out that they are statistically significant. Some reassurance from radiologist feedback was also provided in the rebuttal. There was some remaining questions by one reviewer about limited novelty, all reviewers leaned towards acceptance of the paper in their final ratings.

Reviews: review: paper_summary: paper_summary: This paper develops the GazeDiff model that utilizes the radiologist's eye gaze patterns as additional control of the text-to-image diffusion model. And show that this model can not only improve image generation capability, but also improve the zero-shot classification performance.
main_review: main_review: The research topic this paper explores is interesting and potentially inspires other research in utilizing human expertise in guiding or controlling the ML model. However, the zero-short classification task the model evaluated on is not very convincing and not very suitable for the application in medical domain.
strength_weakness: strength_weakness: * The authors explored a way to integrate human attention into the model. However, the supurior of the proposed model compare with foundation model (e.g.CheXagent) which is trained on large number of chest x-ray images and reports is not very clear. Especially considering that the inference time of diffusion model may hinder its application as a classifier.
* Since the generated image based on input and text are quite arbitrary and very different from the input (see  Figure.4), it makes the motivation of using the guided diffusion model as a zero-shot classifier questionable (diffusion model may perform classification by adding arbitrary information that may not exist in the input), especially in the high-risk medical domain. 
* It is not clear why the authors use HVA edge maps instead of directly using HVA maps as control.
questions: questions: Please address the Weaknesses points
paper_summary: paper_summary: Overall, this is an interesting paper that presents a new approach to integrate expert visual attentions as an additional control to the text-to-image diffusion models. Four public domain datasets were used to evaluate the performance of the proposed GazeDiff. Given the common medical scenarios radiologists’ eye gaze patterns and transcripts are not available; the authors used zero-shot classification.
main_review: main_review: The authors have addressed all concerns during the rebuttal process. The dataset description as well as the effectiveness and significance of the proposed approach are provided. I'm happy with the quality of the rebuttals.
strength_weakness: strength_weakness: •	The key contributions of the proposed approach are not clearly outlined.

•	The list of datasets is provided, but there is no description accompanying each dataset.

•	In section 3.1, the justifications of selecting parameters to tune the model are not provided.

•	The results show marginal improvements which make me question the effectiveness and significance of the proposed approach.
questions: questions: •	Provide the key contributions in the introduction section.

•	A detailed description of datasets with samples. 

•	The results show marginal improvements which make me question the effectiveness and significance of the proposed approach. In Table 1, how the performance of GazeDiff compares to other methods in terms of statistical significance.
paper_summary: paper_summary: This work presents a novel architecture that leverages radiologists’ eye gaze patterns as controls for text-to-image diffusion models for zero-shot classification. This model offers more than just image generation capabilities; the density estimates derived from the gaze-guided diffusion model can effectively improve zero-shot classification performance. With eye gaze patterns as conditional inputs to diffusion models, the generated medical images contain disease-specific features or irregularities in the exact location or periphery of radiologists’ eye gaze fixations.
main_review: main_review: This paper uses the gaze information to control the chest X-ray image generation. This is a good idea to control the disease location in the generated image. The generated image is expected to have higher quality compared with previous methods. I think this paper deserves accepted.
strength_weakness: strength_weakness: 1. The experimental results lack evaluation from radiologists. The involvement of doctors in the evaluation of generated images can make the experiment more convincing.
2. The zero-shot classification lacks comparison with other methods.
questions: questions: 1. Introducing the doctor's evaluation work in the generated pictures. Due to the particularity of CXR pictures, the doctor's evaluation of the pictures is more in line with medical standards.
2. Compare the disease conditions in the pictures generated by different methods. Can these generated images pass disease diagnostic models and doctors’ inspections?
3. It would be better to show the results of zero-shot classification compared with other methods.
paper_summary: paper_summary: In this work, the authors proposed a method named GazeDiff which aims to utilize radiologists' eye gaze patterns to improve the classification results on chest X-ray diseases. This work uses the framework of stable diffusion as the backbone of the diffusion model to perform disease classification in a zero-shot manner. In details, the authors finetuned the stable diffusion network based on chest X-ray from a public dataset with available eye gaze data and conditioned the diffusion model with processed eye gaze data, actually HVA images, to improve the performance. This conditioned diffusion model follows previously proposed method where the diffusion model is used as a classifier given different class labels. The authors chose to use ControlNet as the condition method where the processed eye gaze patterns are used as input with zero-init conv. The work achieves relatively good performance compared with the baselines (stable diffusion and stable diffusion + controlNet).
main_review: main_review: Even though I think the novelty of the paper is still limited. But I'm satisfied with the rebuttals of the other concerns. After the rebuttal, the paper now is rich and well-written. I think it's improved in other aspects. I'm changing my rating from weak reject to borderline accept.
strength_weakness: strength_weakness: 1. The novelty is limited. This work basically uses diffusion model as a classifier and chooses ControlNet as the condition method which is quite common.

2. Lack of comparisons with other X-ray disease classification model. There are many works related to this domain. It would be better if the authors could add more baselines from other X-ray disease classification model to demonstrate the superiority. 

3. The performance doesn't clearly outperform other baselines such as CN-CXR. The metrics are very close.

4. The overall writing is good but the details or descriptions for baselines are not clearly defined. I understand it might be due to the 8 page limits.
questions: questions: 1. Finetuning of Stable Diffusion: How is stable diffusion fine-tuned, e.g., how do the authors handle one-channel X-ray input; Is the text module also finetuned?

2. What's the input of the copied and trained ControlNet compared with your proposed GazeDiff since ControlNet also requires an image or 2D input.

3. Experiments using only canny edge of the X-ray images instead of HVA. How would the performance of this be?

4. Experiments of other condition methods if time allows.

Comments: title: No feedback yet
Request for feedback
Please read and respond to author comments
comment: Thanks for your clarification. I have no further questions.
Thank you for your efforts during the rebuttal process and for your revisions to the paper I'm pleased with the quality of the rebuttals.
Thank you for the efforts during the rebuttals as well as your revisions to the paper! I'm happy with the rebuttals.
Dear Area Chair, 

We have not received any response to the rebuttal yet despite your earlier reminder. We are concerned that this delay may not give us adequate time for further experimentation should any comments be forthcoming today. We have posted another reminder.

Regards,
Authors of Paper #73
We express our gratitude to the reviewers again for their comprehensive assessment during the initial round and eagerly await any additional feedback to address potential clarifications in our responses.
We thank the reviewers for their insightful comments and suggestions on our work which really helped improve our paper. The reviewers point out that integrating the eye gaze patterns of radiologists to a diffusion model is interesting and helps in improving the zero-shot classification performance. The reviewers also appreciate the structure and clarity of the paper. However, the reviewers have a few concerns about comparison with other methods, improvements in the performance, comparison with other conditions, etc. We have addressed all these concerns in this rebuttal in detail. Also, the reviewers had a few comments requiring additional information, we have addressed them and modified the paper likewise.
We thank the reviewer for the excellent comments on our work. We answer the questions/comments below:

> In Fig.4, "We show the generated pathology/object mentioned in the radiologists’ transcript in the red box.". Are these bounding boxes marked by doctors? Similarly, In Fig.5, are these bounding boxes also marked by doctors? Why are these bounding boxes not marked by other methods?

Yes, in Figure 4 and 5, the bounding boxes and arrows are marked by a radiologist (mentioned in Subsection 3.4 in the paper.)

> Can FID and CLIP scores better evaluate medical images? Are doctors involved in the evaluation of the generated images? If so, how are they evaluated?

FID and CLIP scores are standard metrics for evaluating the quality of generated images. Also, in [1], FID and CLIP have been used for quantitative assessment of image fidelity and diversity.

> Introducing the doctor's evaluation work in the generated pictures. Due to the particularity of CXR pictures, the doctor's evaluation of the pictures is more in line with medical standards.

The reviewer points out a very interesting evaluation mechanism. The quality and clinical accuracy of the generated CXRs should be evaluated by a radiologist. In this work, in Figure 4 and Figure 5, we show the radiologist's annotation using red arrows or bounding boxes. The feedback of the radiologist indicated that the CXRs generated by the ControlNet model are of sub-optimal quality whereas the CXRs generated by GazeDiff are of high-quality and clinically accurate.

> Compare the disease conditions in the pictures generated by different methods. Can these generated images pass disease diagnostic models and doctors’ inspections?

Initial examination by an experienced radiologist suggests that the CXRs generated from GazeDiff are of good quality and diagnostic accuracy. We also showed CXRs generated by the ControlNet to the radiologist, the feedback was that the generated images using the ControlNet model were of poor quality (Figure 5).

> It would be better to show the results of zero-shot classification compared with other methods.

In Table 1 of  [2], CLIP has been used as another baseline method.  We have added CLIP and PubMedCLIP as  additional methods for comparison. See Table 1.

**Reference**

[1] Chambon, Pierre, et al. "Roentgen: Vision-language foundation model for chest x-ray generation." arXiv preprint arXiv:2211.12737 (2022).

[2] Li, Alexander C., et al. "Your diffusion model is secretly a zero-shot classifier." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
We thank the reviewer for the excellent comments on our work. We answer the questions/comments below:

 > The authors explored a way to integrate human attention into the model. However, the supurior of the proposed model compare with foundation model (e.g.CheXagent) which is trained on large number of chest x-ray images and reports is not very clear. Especially considering that the inference time of diffusion model may hinder its application as a classifier.

The reviewer rightly points out that CheXagent is trained on a large number of CXR images and reports from MIMIC-CXR, PadChest, and BIMCV-COVID-19 and demonstrates superior performance for multiple tasks. The main difference between foundation models like CheXagent and diffusion models is that diffusion models demonstrate excellent zero-shot performance on unseen tasks. Finetuning CheXagent, a model with 8 billion parameters with eye-gaze of radiologists, is challenging. Moreover, the probabilistic nature of diffusion models enables them to capture intricate data distributions, in our case visual attention patterns, more effectively than foundation models. 
The reviewer points to an interesting concern regarding inference time of diffusion models. Diffusion models are indeed comparatively slower during inference but the advantages outweighs this limitation. This can be addressed in the mentioned points: 
Model Complexity: In clinical scenarios, generalization to unseen tasks and zero-shot inference is of prime importance. Existing zero-shot classification methods typically involve more data and complex inference processes whereas the architecture and the probabilistic inference mechanisms of the diffusion model helps in better generalization to unseen classes with very less data. 
Real-time clinical scenarios:  In clinical scenarios, where the training data is often sparse and, in cases unavailable, diffusion models demonstrate superior ability to generalize to unseen classes without retraining the model. Diffusion models can generate high quality medical images that also help in diagnosis.

> Since the generated image based on input and text are quite arbitrary and very different from the input (see Figure.4), it makes the motivation of using the guided diffusion model as a zero-shot classifier questionable (diffusion model may perform classification by adding arbitrary information that may not exist in the input), especially in the high-risk medical domain. 

GazeDiff generates CXRs with the disease pathology in Figure 4 verified by an experienced radiologist. The reviewer raises a concern on the ambiguity between the text and the generated CXR. This is a critical problem in text-to-image diffusion models. To address this, we propose a HVA guided CXR generation using ControlNet (CN-Gaze). This helps in generating high-quality and clinically accurate CXR images by integrating the visual attention information from radiologists. We are using this CN-Gaze architecture as a zero-shot classifier (GazeDiff). Then, we show that GazeDiff performs better in zero-shot classification compared to CN-CXR and SD-CXR. This implies that GazeDiff generates better disease patterns than the baselines. Hence, the applicability of these types of conditional diffusion models in data augmentation for classification can be useful. 
Concern in high-risk medical domains: The arbitrary nature of the images generated from a diffusion model can be risky if it is used in real-time clinical practice. In recent times, diffusion models are used in improving performance for diagnostic tasks like disease classification, segmentation and zero-shot inference. Here, the main application is to generate images that have the mentioned disease pathology and are of diagnostic quality. 

> It is not clear why the authors use HVA edge maps instead of directly using HVA maps as control.

In this work, we use HVA edge maps rather than using HVA maps as the former is a strong control that helps in reducing ambiguity between the control and the generated CXRs. HVA maps are heatmaps that are visual representations of eye gaze fixation distribution across a spatial domain and hence sparsifies the control resulting in ambiguity whereas HVA edge maps are edge maps obtained from applying Canny edge filtering to the HVA map that contain information regarding both the physical anatomy and disease presence information - this minimizes the ambiguity and acts as a strong control.

> Interesting work that explores how to use human expert knowledge to improve ML learning models. However, the application scenario and the tasks are not very convincing.

The method outlined offers a broad range of applications, spanning from unseen task generalization to missing modality generalization and time point image generation, among others. Also, finetuning diffusion models with radiologist’s eye gaze attention maps have immense potential towards generating clinically accurate medical images.
Dear reviewers, The authors have posted responses to your reviews. Please take the time to read and respond before March 27.
We thank the reviewer for the excellent comments on our work. We answer the questions/comments below:

> In section 3.1, the justifications of selecting parameters to tune the model are not provided.

We apologize we could not provide a detailed description on the parameter selection due to space constraints. Here, we elaborate on the parameters used for finetuning our models. For finetuning the Stable Diffusion v1.5 model, we use the same image size of 512x512 and learning rate of 1e-5 that was used to train the model on natural images in HuggingFace. We use a batch size of 4,the training was done on a Quadro RTX 8000 (48 GB). For finetuning the ControlNet model, we use the same parameters as in HuggingFace, the only difference is that we train for 10 epochs as the eye gaze dataset is comparatively small and requires more iterations to converge well. 

> Provide the key contributions in the introduction section.

We have added key contributions in the introduction section. 
‘The key contributions of this paper are as follows: a) we propose to add eye gaze patterns of experts as additional control and radiologists' text as prompt to the text-to-image diffusion models. b) we use this finetuned diffusion model as a zero-shot classifier for downstream pulmonary diseases like pneumonia and tuberculosis classification tasks.’

> A detailed description of datasets with samples.

We have added detailed description of the datasets with image samples used in Appendix A.

> The results show marginal improvements which make me question the effectiveness and significance of the proposed approach. In Table 1, how the performance of GazeDiff compares to other methods in terms of statistical significance.

Though the difference in accuracy and F1 is marginal. We show the statistical significance by calculating the paired t-test between the predictor errors ($\epsilon_\theta$) for 2 datasets. For the Cell Pneumonia dataset, the p value is 0.0067 and for the RSNA Pneumonia dataset, the p value is 0.0317.
We thank the reviewer for the excellent comments on our work. We answer the questions/comments below:

> The novelty is limited. This work basically uses diffusion model as a classifier and chooses ControlNet as the condition method which is quite common.

The reviewer expresses concern that using ControlNet is common and hence the novelty is limited. We would like to point out that we are not just using ControlNet as the conditioning method for downstream zero-shot classification. We train the ControlNet using HVA maps that contain the radiologist’s eye gaze attention patterns for downstream zero-shot classification. This approach is still unexplored and opens up avenues for future method developments.

> Lack of comparisons with other X-ray disease classification model. There are many works related to this domain. It would be better if the authors could add more baselines from other X-ray disease classification model to demonstrate the superiority.

There are very few existing methods that demonstrate zero-shot classification with CXRs. In Table 1 of [2], SD and CLIP are used for comparison. Hence, we also add CLIP[3] and PubMedCLIP[4] as a baseline method in this rebuttal (see Table 1).

> The performance doesn't clearly outperform other baselines such as CN-CXR. The metrics are very close.

Although the metrics are close, we show statistical significance of the method by calculating the paired t-test between the predictor errors ($\epsilon_\theta$ in Figure 1) for 2 datasets. For the Cell Pneumonia dataset, the p value is 0.0067 and for the RSNA Pneumonia dataset, the p value is 0.0317.

> The abbreviation of the name for ControlNet (CN) is not clearly stated. It would be better to clearly define somewhere in your paper. Same for RoentGen. A reference in the paper or appendix is sufficient.

Thank you very much for pointing out this inconsistency. We modify the first line of Section 2 where we define that SD is Stable Diffusion and CN is ControlNet. The name of the RoentGen[1] paper is RoentGen and we are using the same name everywhere in the paper. For better understanding, we add the references for the baselines in Subsection 3.2.

> SD and CN is not really necessary for the results since they are all trained with large dataset including natural imaging not specifically with X-ray image.

The author rightly points out that SD and CN are trained with large natural imaging datasets. We wanted to compare that the performance of these standard SD and CN features are not better than SD and CN finetuned on CXRs. We modify Table 1 and add another column ‘CXR’ that shows models trained with CXRs or natural images.

> Finetuning of Stable Diffusion: How is stable diffusion fine-tuned, e.g., how do the authors handle one-channel X-ray input; Is the text module also finetuned?

The CXRs are converted to 3-channels and then the SD v1.5 model is finetuned with CXRs and texts from the MIMIC-CXR dataset. The SD v1.5 model weights are obtained from Hugging Face and the finetuning is done using accelerate. The text module is also finetuned.

> What's the input of the copied and trained ControlNet compared with your proposed GazeDiff since ControlNet also requires an image or 2D input.

The reviewer is right in pointing out that the ControlNet requires an image as input. We show in Figure 2 that (I, G’) is fed to the trainable copy. We show the HVA edge maps that are fed in Figure 3 and Figure 7 in the Appendix. In summary, the input to GazeDiff is the HVA edge maps and the text conditions.

> Experiments using only canny edge of the X-ray images instead of HVA. How would the performance of this be?

Using only canny edge, for Cell dataset, the accuracy is 58.01 (GazeDiff:59.29, 2.16% improvement) and the F1 score is 71.21 (GazeDiff:72.08, 1.21% improvement) and for the CHN dataset the accuracy is 58.61 (GazeDiff:58.91, 0.51% improvement) and the F1 score is 70.41 (GazeDiff:70.69, 0.40% improvement).

> Experiments of other condition methods if time allows.

We show results of ControlNet with various radiomics filter maps as controls on 2 datasets. We are adding this as Table 7 in the Appendix. We observe that GazeDiff outperforms all the ControlNets trained with these different conditions.

**Reference**:

[1] Chambon, Pierre, et al. "Roentgen: Vision-language foundation model for chest x-ray generation." arXiv preprint arXiv:2211.12737 (2022).

[2] Li, Alexander C., et al. "Your diffusion model is secretly a zero-shot classifier." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

[4] Eslami, Sedigheh, Gerard de Melo, and Christoph Meinel. "Does clip benefit visual question answering in the medical domain as much as it does in the general domain?." arXiv preprint arXiv:2112.13906 (2021).