Parameter-Efficient Generation of Natural Language Explanations for Chest X-ray Classification

Authors: isabel  rio-torto
jaime s cardoso
luís f teixeira
Venue: MIDL_io_2024_Conference
Published: 2024-02-01 09:21:30

Abstract: The increased interest and importance of explaining neural networks' predictions, especially in the medical community, associated with the known unreliability of saliency maps, the most common explainability method, has sparked research into other types of explanations. Natural Language Explanations (NLEs) emerge as an alternative, with the advantage of being inherently understandable by humans and the standard way that radiologists explain their diagnoses. We extend upon previous work on NLE generation for multi-label chest X-ray diagnosis by replacing the traditional decoder-only NLE generator with an encoder-decoder architecture. This constitutes a first step towards Reinforcement Learning-free adversarial generation of NLEs when no (or few) ground-truth NLEs are available for training, since the generation is done in the continuous encoder latent space, instead of in the discrete decoder output space.
However, in the current scenario, large amounts of annotated examples are still required, which are especially costly to obtain in the medical domain, given that they need to be provided by clinicians. Thus, we explore how the recent developments in Parameter-Efficient Fine-Tuning (PEFT) can be leveraged for this use-case. We compare different PEFT methods and find that integrating the visual information into the NLE generator layers instead of only at the input achieves the best results, even outperforming the fully fine-tuned encoder-decoder-based model, while only training 12\% of the model parameters. Additionally, we empirically demonstrate the viability of supervising the NLE generation process on the encoder latent space, thus laying the foundation for RL-free adversarial training in low ground-truth NLE availability regimes. The code is publicly available at https://github.com/to_be_added.

Main paper: 
 introduction: 
 The need for providing explanations is essential in domains like medicine, given the life-ordeath nature of the decisions involved (Vayena et al., 2018) . Given their training-free nature, saliency maps are, by far, the most commonly employed explanation method, despite their known problems (Adebayo et al., 2018; Camburu et al., 2019; Rudin, 2019) .Among other explanation modalities are Natural Language Explanations (NLEs). These are textual descriptions of the decision-making process of a neural network for a given input image, thus combining image descriptions (e.g. obtained via image captioning models) -image-relevance -and model predictions -class-relevance (Hendricks et al., 2016; Rio-Torto et al., 2022) . NLEs are intrinsically understandable by humans and can provide complementary insights to those of other types of explanations. For example, while visual explanations are more spatially precise, NLEs can be more semantically meaningful (Rio-Torto et al., 2022) . In the medical domain in general, and in chest X-ray diagnosis in particular, NLEs can be even more useful, given that they correspond to how radiologists explain their diagnoses (Gale et al., 2018; Miller, 2019) .There are several works on NLEs for natural images (Hendricks et al., 2016; Park et al., 2018; Wu and Mooney, 2019; Do et al., 2020; Marasović et al., 2020; Kayser et al., 2021; Majumder et al., 2022; Plüster et al., 2022; Sammani et al., 2022) and for text-only models (Camburu et al., 2018; Rajani et al., 2019; Kumar and Talukdar, 2020; Narang et al., 2020) ; they usually apply the traditional cross entropy loss on the generated NLEs and condition the NLE generator on information from the predictions of the model that is being explained. We highlight the work by Hendricks et al. (Hendricks et al., 2016) since it explicitly imposes class-relevance on the generated NLEs through the proposal of a Reinforcement Learningbased (RL) loss. Inspired by this and by the work on adversarial text generation (Donahue and Rumshisky, 2018) , we propose replacing the traditional decoder-only NLE generator with an encoder-decoder architecture (see Figure 1 ). This paves the way for the RL-free adversarial generation of NLEs, when no (or few) ground-truth data is available to train the model.In the medical domain, text generation mostly consists of report generation approaches (Hou et al., 2021; Monshi et al., 2020; Ouis and A. Akhloufi, 2024) . To the best of our knowledge, Kayser et al. (Kayser et al., 2022) is the first and only work on NLE generation for the medical domain. Therefore, we build upon this work; namely, we use the dataset the authors introduce, MIMIC-NLE, and their proposed evaluation framework.However, contrary to the models proposed by Kayser et al. (Kayser et al., 2022) , we adopt Parameter-Efficient Fine-Tuning (PEFT) when training our NLE generator to receive visual features and classification predictions. Instead of fine-tuning all model parameters, PEFT methods fine-tune only a small percentage, while still performing competitively (Zhang et al., 2023) . A plethora of PEFT methods has been proposed, not only for single modality use-cases (Rebuffi et al., 2017; Houlsby et al., 2019; Lester et al., 2021; Li and Liang, 2021; Liu et al., 2021; Chen et al., 2023; Yu et al., 2023; Zhang et al., 2023) , but also for multi-modal applications (Luo et al., 2023; Zhang et al., 2023; Gao et al., 2024) , and in the medical domain as well (Dutt et al., 2023) . Applying PEFT to NLE generation is still an emergent topic, with work only on uni-modal applications (Solano et al., 2023) .Therefore, our contributions are the following:• Replacement of the decoder-only NLE generator of Kayser et al. (Kayser et al., 2022) by an encoder-decoder architecture that lays the foundation for the RL-free adversarial generation of NLEs when no (or few) ground-truth data is available during training.• Introduction of the first work exploring different PEFT methods for NLE generation for chest X-ray classification. The encoder receives the image features and the prediction information, while the decoder generates the NLE from the latent space of the encoder. 
mimic-nle dataset: 
 NLEs are fundamentally different from image descriptions, since the former need to be not only image-relevant but also decision-relevant, i.e. take into account the decision of the underlying model they are trying to explain (Hendricks et al., 2016; Rio-Torto et al., 2022) .Although there are some datasets for chest X-ray diagnosis that include medical reports in text form (i.e. image descriptions) (Demner-Fushman et al., 2015; Johnson et al., 2019; Bustos et al., 2020) , the work by Kayser et al. (Kayser et al., 2022) was the first and, to the best of our knowledge, the only one to propose a dataset with NLEs.The MIMIC-NLE dataset (Kayser et al., 2022) is obtained automatically from the MIMIC-CXR dataset (Johnson et al., 2019) Besides introducing MIMIC-NLE, Kayser et al. (Kayser et al., 2022 ) also tested different architectures to generate NLEs for this multi-label scenario. These architectures, which we describe below, are used here as baselines to compare our proposals with. All architectures work in a similar fashion: a vision model (DenseNet-121 (Huang et al., 2017) ) extracts features and classifies the input image; the features and a vector representation of the prediction, together with the pathology for which the NLE is being generated, are fed into an autoregressive language model, which generates the NLE. During training, instead of using the prediction vector, the ground-truth label is used, and the vision model is trainable. Thus, this can be considered an in-model explanation architecture that works within a predict-explain paradigm (Do et al., 2020) .RAdiological Text Captioning for Human Examined Thoraxes (RATCHET) was first proposed by Hou et al. (Hou et al., 2021) as a chest X-ray captioning method. Kayser et al. (Kayser et al., 2022) leverage this architecture to generate NLEs instead. It consists of a GPT-2 (Radford et al., 2018) model with cross-attention layers, that receive the vision classifier's features concatenated with a projection of the prediction vector, as depicted in Figure 2 (top left). As input, the model receives the NLE-related pathology.DPT (DenseNet-121 + GPT-2) is proposed by Kayser et al. (Kayser et al., 2022) and it consists of a GPT-2 model that receives as input the vision features, the prediction vector, and the NLE-related pathology. An illustration of this architecture can be found in Figure 2 (top right).Architecture Our approach builds upon the work of Kayser et al. (Kayser et al., 2022) , but replaces the NLE decoder with a denoising auto-encoder (see Figure 1 ). Not only does this increase the representation capabilities of the NLE generator, but, most importantly, it lays the foundation for the RL-free adversarial generation of NLEs when no (or few) ground-truth NLEs are available for training. In this scenario, teacher forcing is unfeasible. While an adversarial approach can solve the problem, if the generator is a decoder-only architecture (e.g. DPT), RL is needed, since its output (and the input to the discriminator) is discrete and, as such, no gradient would flow from the discriminator to the generator (Donahue and Rumshisky, 2018) . By using an encoder-decoder architecture, where the adversarial learning happens on the continuous latent space of the encoder at the sentence level, gradient propagation is ensured without needing RL, thus making the training process faster (obtaining an NLE now only involves a single encoder forward pass instead of N decoder passes, where N is the number of tokens in the generated sentence) and more stable. Finally, decoding this latent space into text to obtain the NLE is still possible thanks to the pre-trained decoder.We use the auto-encoder proposed by (Montero et al., 2021) , which consists of a frozen BERT-based encoder, a multi-head attention mechanism as a bottleneck layer, and a single transformer decoder layer with cross attention. Since this cross attention layer receives the same input at every timestep, the authors introduce a gating mechanism that controls the amount of information to keep at each timestep. Trained on MIMIC-NLE (Kayser et al., 2022) , it achieves a reconstruction BLEU-4 score of 93.9.PEFT Our auto-encoder is trained on text inputs, but it needs to be adapted to incorporate the products of the vision model we want to explain. Recently, the transfer learning paradigm has shifted from fine-tuning entire models to only certain layers. This is especially important in NLE generation, and even more so in post-model methods, where given a pretrained model, we want to obtain explanations for its decision process with the shortest training time possible. Therefore, we experiment with applying several PEFT techniques to the encoder of our NLE Generator, while keeping its decoder module frozen. These methods are depicted in Figure 2 (last three rows).In total, we try five PEFT methods, four of which are built on top of an input configuration similar to that of DPT, i.e. where both image features, prediction vector and associated pathology are given as inputs to the encoder.LoRA (Yu et al., 2023) represents, through low-rank decomposition, the model's weight updates into two smaller matrices, which are trained, while the original model weights are kept frozen. After training, both weight matrices are combined. Prompt and Prefix Tuning (Lester et al., 2021; Li and Liang, 2021) insert a given number of learnable soft prompts into the frozen model. While Prompt Tuning (Lester et al., 2021) only appends these prompts to the input embeddings, Prefix Tuning (Li and Liang, 2021) applies these prompts to several layers of the frozen model (in our case, to all layers).LlaMA-Adapter (Zhang et al., 2023) was proposed to efficiently fine-tune a large language model, LlaMA (Touvron et al., 2023) , and it differs from Prefix Tuning in the way the learnable adapters are merged into the frozen model. The authors propose to separate the adapter prompts from the word tokens and introduce a zero-initialized gating mechanism that adaptively controls the influence of the adapter prompts during training. This way, in the early training stages, the influence of the adapter prompt is very small, and it is progressively increased, which guarantees training stability while still ensuring the model is gradually being adapted and simultaneously retaining the knowledge of the original model.However, none of the aforementioned methods take into account the multi-modality of our scenario, as they were developed for uni-modal applications. To tackle this, the authors of LlaMA-Adapter also propose a multi-modal version. Instead of providing the imagerelated information through the input of the model, it is directly added to the adaptation tokens. To ensure the image features have the same dimensions as the adaptation prompts, a Transformer encoder is used, as shown in Figure 2 (last row).As previously mentioned, using an encoder-decoder architecture opens the door to the RL-free adversarial generation of NLEs in low groundtruth availability regimes. To test the viability of such an approach despite having, as we do in this work, ground-truth NLEs for the whole dataset, we experiment with supervising the NLE generation directly in the latent space of the encoder, as this is where the adversarial learning would happen. Thus, in this experiment, we replace the cross entropy loss computed between the discrete output of the decoder and the ground-truth NLE, by the Mean Squared Error (MSE) computed between the encoder latent representation of the ground-truth NLE and the continuous output of the encoder. 
experimental setup: 
 Evaluation Metrics We follow the evaluation protocol proposed in the MIMIC-NLE paper (Kayser et al., 2022) ; several aspects are evaluated, namely the chest X-ray multilabel classification performance (through balanced AUC -Area Under the Curve), and the NLEs' quality (only for correctly predicted instances). Given that there is evidence that automated natural language generation (NLG) metrics correlate badly with the way humans evaluate how similar two NLEs are (Kayser et al., 2021) , because two NLEs can be simultaneously correct while having very different syntactic forms, the authors propose the CLEV (CLinical EVidence) score. This score uses the CheXbert labeler (Smit et al., 2020) to verify if an NLE refers to the same clinical evidence as the ground-truth NLE. Thus, it corresponds to the accuracy regarding clinical evidence. Several NLG metrics are considered, with a bigger importance given to BERTScore (Zhang et al., 2020) and METEOR (Lavie and Agarwal, 2007) since it has been shown that these exhibit the highest correlation with human evaluation (Kayser et al., 2021) .Implementation Details Regarding the pre-training of the auto-encoder, the decoder is a single transformer layer, as proposed in the original AUTOBOT paper (Montero et al., 2021) , and the encoder is the CXR-BERT model (Boecking et al., 2022) . CXR-BERT is a BERT masked language model trained on PubMed abstracts (U.S. National Library of Medicine), clinical notes from MIMIC-III (Johnson et al., 2016) and MIMIC-CXR (Johnson et al., 2019) . The model is trained with the AdamW optimizer for 1 million steps with a batch size of 64 and a linearly decayed learning rate of 0.001 with 1000 warmup steps. The best model is chosen based on the reconstruction BLEU-4.For the explanation generation model, we follow the training details described in the MIMIC-NLE paper (Kayser et al., 2022) : for RATCHET, the learning rate is 5 × 10 -5 , and 5 × 10 -4 for all remaining experiments. All models are trained with the AdamW optimizer for 50 epochs with a batch size of 16 and a linearly decayed learning rate with 1000 warmup steps. We use Early Stopping with a patience of 25.LoRA parameters follow the original LoRA paper (Yu et al., 2023) , i.e. an alpha of 32 and a dropout of 0.01. For the remaining PEFT methods, the learnable prompts have 10 tokens (as in the original LlaMA-Adapter (Zhang et al., 2023) ), and for both LlaMA-Adapter models, the prompts are injected at every layer of the NLE generator encoder. The transformer used in the multi-modal LlaMA-Adapter has the following configuration: 2 transformer layers with 4 attention heads, an embedding dimension of 768, and the linear layers have 3072 neurons.The best model and corresponding hyperparameters are chosen with the same criteria as in the MIMIC-NLE paper (Kayser et al., 2022) : the product of the classification score (balanced AUC), CLEV score, and the average of BERTScore (Zhang et al., 2020) and METEOR (Lavie and Agarwal, 2007) , computed on the validation set.We learn the weighting scheme of the classification and text losses, following the methodology proposed in (Cipolla et al., 2018) . Thus, our loss function is given by L= L clf + 1 σ 2 L text + log(σ), where σ is learned during training and has a starting value of 1.0. 
results and discussion: 
 Table 1 contains the results of the different NLE generation frameworks evaluated in terms of balanced AUC, CLEV and NLG metrics. We also present the number of correctly classified instances (for which the CLEV and NLG metrics are computed), the number of additional trainable parameters introduced, and the overall performance.Besides the base DenseNet-121 model, we present the results of two of the models of (Kayser et al., 2022) , RATCHET and DPT, trained with our loss weighting scheme. The obtained results follow the trends verified in the original paper, i.e. RATCHET improves AUC over the DenseNet-121, while DPT does not, and the former is better overall than the latter.Regarding our proposals, replacing the decoder-only NLE generator (i.e. DPT) with the encoder-decoder (Base AE) improves the overall performance: while the AUC slightly decreases, the CLEV and BERT scores increase. However, this comes at a cost: 145M trainable parameters are added, 20M more than DPT introduces.Uni-modal PEFT techniques slightly decrease or perform on par with the Base AE model in terms of overall performance. However, these methods introduce much less trainable parameters (11M in the worst case). Given the small loss in performance but the high gain in training efficiency, PEFT methods are, in fact, suitable for NLE generation for chest X-ray diagnosis, as had already been shown in other domains. Our results also validate the findings of the original LlaMA-Adapter (Zhang et al., 2023) , i.e. that the zero-initialized gating mechanism surpasses traditional Prefix Tuning.The multi-modal version of LLaMA-Adapter achieves the best overall results among our proposals, even surpassing the fully fine-tuned model. This confirms our hypothesis that incorporating the visual information throughout the NLE generator layers instead of just at the input is beneficial. Indeed, this is probably the same reason why RATCHET outperforms DPT. This result demonstrates that explainability and task performance are not always at odds with each other and, in fact, explainability might improve classification accuracy.Finally, we train the best PEFT configuration (multi-modal LLaMA-Adapter) directly with MSE on the latent space of the encoder. While the decrease in NLG metrics is expected since supervision now occurs at the sentence level instead of at the token level, the CLEV score significantly increases, which shows that, although the generated NLEs might by syntactically different from the ground-truth NLEs, they contain the correct clinical evidence. Thus, this confirms that our proposed approach is viable, laying the foundation for the RL-free adversarial generation of NLEs.We present qualitative examples in Figure 3 of the Appendix. They corroborate the quantitative results, mainly that the PEFT approaches produce NLEs consistent with fully fine-tuned models, even while training only a fraction of the parameters, and that reasonable NLG metric values do not always correlate with correct clinical evidence. 
conclusion: 
 We build upon the seminal work of Kayser et al. (Kayser et al., 2022) , which tackled the problem of generating NLEs for chest X-ray diagnosis, by replacing the decoder-only NLE generator by an encoder-decoder, coupled with PEFT techniques to drastically reduce the number of trainable parameters while maintaining competitive performance. We show that using the multi-modal-aware LLaMA-Adapter (Zhang et al., 2023) achieves the best overall results, even outperforming the fully fine-tuned encoder-decoder model. We also empirically demonstrate that directly supervising the NLE generation process on the encoder latent space at the sentence level is viable, thus consolidating this approach as a first step towards the RL-free adversarial generation of NLEs when no (or few) ground-truth NLEs are available. 
 
 Appendix:  


Decision: The paper addresses natural language explanations for chest x-ray classification by exploring how to tune such models more efficiently. The reviewers were fairly positive but raised some questions, in particular about replications of results from earlier work. This appears to be fixed in the revision, along with some other questions. Although the scores are not overwhelmingly positive I think I agree that this would make a good conference contribution.

Reviews: review: paper_summary: paper_summary: The authors develop a new model for natural langage explanations of chest x-ray classification using a a langage model conditioned on features from a vision model and the classification results. They train using parameter efficient methods and ablate the different methods and losses.
main_review: main_review: - Good ablation of Parameter-efficient Fine-Tuning (PEFT): the number of PEFT methods is increasing steadily but no grounding and ablations have been made in the medical text-generation field. This is a valuable insight for future work as model size increases and compute osts become an issue.
- Good experiments on which visual conditioning technique is the best: multi-modal langage generation is becoming more and more prominent and finding the most efficient way to condition text-generation to visual feature is still an open question.

My main concerns have been addressed and I think that the paper can add value to the NLE community.
strength_weakness: strength_weakness: 1. The paper is a little bit difficult to follow:
- The MSE loss is introduced but no explanation about what it is applied on.
- The author talk about their method allowing to train the NLE model without reinforcement learning but no baseline use RL.
- The author introduce their method as using an auto-encoder both in the main text and in the Table 1 but not all methods are auto-encoder based according the Figure 2. The encoder is a BERT model and the decoder a transformer layer, but are the text tokens inputed in the BERT model or the decoder? 
2. The reproduction of the baselines is significantly lower than in the original paper: it seems that both this paper and the original Kayser et al. paper (https://arxiv.org/pdf/2207.04343.pdf) use the same splits so the reproductions should match. This hinder the possibility to get conclusive results as DPT is said to be better than RATCHET according to this paper’s results but the reverse is true in the Kayser et al. paper.
3. The paper lacks better discussion on the results: Why « LLaMA-Adapter produces the best overall results » but is falling behind LoRA AE for all classical text metrics? It seems to be a trade-off between CLEV/BERTS and other metrics. Results are interesting as-is but the conclusion should be more nuanced and clearly explain why a metric is more important than the other (maybe with a cherry-picked example).
questions: questions: 1. Reformulate some of the passage about MSE, auto-encoder and reinforcement learning
2. Extend the discussion about the results

Optional:
3. I get that doing the experiments again using a better DenseNet is time consuming but baseline results should match or you should explain why they don't (DenseNet is trained with less epochs/less tricks etc.). If the paper is accepted and you manage to make results match, updating the table in the camera-ready version would be great.
paper_summary: paper_summary: The authors propose a mostly exploratory style work, investigating various parameter efficient fine tuning methods for natural language explanations of chest X-ray classifications. From a technical novelty perspective, the authors build on one of the only NLE approaches so far tested for medical imaging and replace the decoder only style approach to an encoder-decoder style. The authors show this change allows them to only update a small fraction of parameters without needing RL to achieve strong results.
main_review: main_review: Thanks to the authors for the comments. I stand by my original rating. I think this would make a great poster at MIDL, and maybe even meet the bar for an oral presentation due to the wide interest I expect the see from the community for this explorative work. I still disagree with the authors on how much technical novelty there is here, but that's of little consequence, the paper is a solid accept either way.
strength_weakness: strength_weakness: Given that metrics for NLE's are always very weak at best. It would be great to find a way to fit in some representative examples of NLE's generated by the various methods. I understand this is incredibly tough with the space limitations, but if accepted, I would strongly recommend including these in a supplementary information section (if allowed) or at least on an arXiv version of this paper. 

Given a weak accept instead of strong because there isn't really any technical novelty in this work. That's fine, there are other types of papers than just novel techniques (for example exploration), but without any technical novelty, it's hard to support a very strong accept.
questions: questions: See weaknesses. If there is more technical novelty, laying this out in the contributions would be helpful. Just switching from a decoder only to an encoder-decoder (which already exists) isn't a technical novelty.
paper_summary: paper_summary: This work extends upon previous work on NLE generation for multi-label chest X-ray diagnosis by replacing the traditional decoder-only NLE generator with an encoder-decoder architecture. The results show that using the multi-modal-aware LLaMA-Adapter achieves the best overall results
main_review: main_review: This paper extends previous work on NLE generation for multi-label chest X-ray diagnosis by replacing the traditional decoder-only NLE generator with an encoder-decoder architecture. This would be a contribution to the AI model explanation method. The experiments in this work are sufficient and the results presented are relatively good. Taken together, this work deserves to be accepted.
strength_weakness: strength_weakness: 1. The model does not seem to perform better than previous methods on AUC and other metrics.
2. Whether this improvement has a medical motivation or is it the assembly of a model, a detailed explanation of the motivation is lacking.
3. Given some specific examples to explain your results would be better.
questions: questions: 1. Page 7 says: "We use a weight of 5 for the classification loss for all models, except for the model that uses MSE for the NLE loss, for which the weight is 2." How is this weight parameter set?
2. The model does not seem to perform better than previous methods on AUC and some other metrics. Please provide a detailed explanation.
3. Given some specific examples to explain your results would be better. Especially the comparison between your method and the previous method.

Comments: title: Invitation to reply to authors
Rebuttal - General Response
comment: Thank you for your satisfactory answers, I will improve my rating accordingly.
Thank you for your response and clarifications, I am satisfied with the replies and have no further comments.
We thank the reviewer for the valuable comments and suggestions indicating that our work fills a gap in the medical imaging field by providing ablations on which multi-modal adaptation framework works best. With this rebuttal we aim to clarify our work, especially regarding the MSE loss, the discussion of the obtained results, and the reproduction of the chosen baselines.
&nbsp;

&nbsp;

### MSE Loss
Throughout the paper we reformulated the passages regarding the motivation for the MSE loss experiment and how it is conducted (i.e., where the loss is applied on). Please refer to the general comment where we clarify the connection between the MSE Loss experiment, the choice regarding the replacement of the decoder-only model by the auto-encoder architecture, and the statement that no RL would be needed.
&nbsp;

&nbsp;

### Figure 2
We clarified Figure 2, making it more explicit that our proposal uses an encoder-decoder architecture, setting it apart from RATCHET and DPT.
&nbsp;

&nbsp;

### Reproduction of results
Please refer to the general comment for further discussion of this topic.
&nbsp;

&nbsp;

### Discussion of results and qualitative examples
We included some examples of the NLEs generated by the different approaches in the supplementary material, together with an extended discussion (both in the main paper and in the appendix) regarding the relative importance of the NLG metrics and the CLEV score. Namely, even though some NLG metrics might be lower for some models, indicating that the syntax of their generated NLEs might be different from the one used in ground-truth NLEs, the clinical evidence remains correct.
Dear reviewers,

The authors have prepared responses to your comments, which you should now be able to see in OpenReview. We encourage you to reply to their comments, and where necessary, adjust your rating. Please do so before the 27th of March.
We thank the reviewers for their valuable comments. As summarized by all reviewers, we propose replacing the traditional decoder-only architecture for generating Natural Language Explanations (NLEs) with an encoder-decoder trained with Parameter-Efficient Fine-Tuning (PEFT). The reviewers highlighted that our experimental methodology is well conducted, with sound ablation studies, the exploration of a wide range of PEFT methods, and that the topic is relevant for the medical imaging community.

This rebuttal addresses all reviewers’ concerns and the manuscript has been updated to include qualitative examples, further clarifications regarding the MSE loss, how the encoder-decoder architecture might avoid the need for Reinforcement Learning (RL), and a more extensive discussion of the results.
&nbsp;
&nbsp;

### Encoder-decoder and MSE Loss
Reviewer 8Pm1 asked for further clarification regarding the MSE loss, reviewer  QYz9 mentioned a lack of technical novelty (although still finding “plenty of value to justify publication”), and reviewer R54G states that the manuscript lacks a detailed explanation of the motivation. We believe these three concerns are connected and, thus, we answer them here. The MSE loss experiment is a preliminary test which paves the way and hints at the viability of a future architecture where no (or few) ground-truth NLEs are available during training, but no RL is needed. In this case (when teacher forcing is unfeasible), an adversarial approach can be used. However, if the generator is a decoder-only architecture (e.g. DPT), RL is needed to propagate approximate gradients from the discriminator to the generator, because the output of the decoder (which is also the input to the discriminator) is discrete. By using an encoder-decoder architecture, the adversarial learning happens on the continuous latent space of the encoder, so gradient propagation is ensured without needing RL, thus making the training process faster and more stable. Finally, decoding this latent space into text to obtain the NLE during testing is still possible thanks to the pre-trained decoder. Despite having ground-truth data available in this work, we wanted to test the viability of supervising the NLE generation directly in the output of the encoder, as this is where the adversarial learning would happen in the aforementioned RL-free adversarial NLE generation in low ground-truth availability regimes. We included this clarification throughout the updated manuscript, mainly in Section 2.3.
&nbsp;
&nbsp;

### Reproduction of results and AUC
We thoroughly revised our implementation and found that it was inconsistent with how Kayser et al. described the AUC computation. Additionally, through further experiments, we observed a big impact on the results of the weighting scheme between the classification and text losses. Given computational resource and time constraints, running an extensive hyperparameter search on the combination of these two losses was infeasible, so we opted for automatically learning the loss weights. We have updated the manuscript to include this weighting scheme and the updated results, which are now closer to the original work of Kayser et al.. Small differences might be due to the different loss weighting schemes and a different number of epochs. Moreover, NLG metrics are only computed on correctly classified instances, so small deviations on the classification side might have a big impact on the NLG metrics. Additionally, this makes it harder to compare with the work of Kayser et al.: even though two models might correctly classify the same number of instances, it might not be the same set of instances for the two models. The updated results show similar trends as before, with the multi-modal LlaMA-Adapter obtaining the best results among our models. The results of the MSE loss experiment improved, given the new automatic loss weighting scheme.
We thank the reviewer for their insightful comments stressing that the proposed model architecture is clearly introduced and that an ablation study of different models is provided.
&nbsp;

&nbsp;
### Model performance
Please refer to the general comment for a more in depth discussion concerning model performance, especially the AUC.
&nbsp;

&nbsp;
### Lack of motivation
We clarified the motivation for replacing the decoder-only NLE generator by an encoder-decoder architecture and refer the reviewer to the general comment where this is further discussed.
&nbsp;

&nbsp;
### Classification loss weight
All hyperparameters were chosen using the validation set of MIMIC NLE. As in Kayser et al., the best model corresponds to the one with the best overall results, i.e. the product between AUC, CLEV, and the average of BERTScore and METEOR. We clarified this in the updated manuscript.
&nbsp;

&nbsp;
### Qualitative examples
We included some examples and further discussion of the NLEs generated by the different approaches in the supplementary material.
We thank the reviewer for their insightful comments highlighting that our work is relevant to a “very interesting emerging area”, is “of incredible relevance and clinical importance” and “very welcomed”, and it covers a wide range PEFT techniques.  
&nbsp;

### Qualitative examples
We included some examples and further discussion of the NLEs generated by the different approaches in the supplementary material.
&nbsp;

&nbsp;


### Novelty
Although we recognise our work does not constitute a technical novelty in the sense that it does not propose a completely new architecture, we believe the novelty of this work lies in the fact that it is, to the best of our knowledge, the first to investigate PEFT for NLE generation in the medical domain. Furthermore, we also believe that this constitutes the first attempt at replacing the decoder-only architecture by an encoder-decoder framework for NLE generation and it paves the way for interesting future possibilities, mainly if one does not have ground-truth NLEs available during training. Please refer to the general comment where we dive deeper into these aspects. Throughout the updated manuscript (and in the contributions at the end of Section 1) we reformulated the parts regarding the motivation for the encoder-decoder architecture and the MSE loss experiment so that they are clearer.
Thank you for the clarification. We will add the necessary figures in the supplementary material of the updated version of the paper.
Dear authors,

You should be able to upload an updated version of your paper. Perhaps you can add an Appendix with the images? 

If you mean you want to show some images but not include them in the supplementary material, I suppose since the process is single-blind, you could put them on a public repository on Github, or some other platform which does not track the viewer (so do not use Google Docs for example).
Dear Area Chairs,

In our rebuttal it would be very helpful if we could submit a pdf with some of the images requested by the reviewers.
Is it possible to do so? And if yes, how do you suggest we do that since we cannot directly upload the file via open review.

Thank you in advance.