BiCAL: Bi-directional Contrastive Active Learning for Clinical Report Generation (2024)

Abstract: State-of-the-art performance by large pretrained models in computer vision (CV) and natural language processing (NLP) suggests their potential for domain-specific tasks. However, training these models requires vast amounts of labelled data, a challenge in many domains due to the cost and expertise required for data labelling. Active Learning (AL) can mitigate this by selecting minimal yet informative data for model training. While AL has been mainly applied to single-modal tasks in the fields of NLP and CV, its application in multi-modal tasks remains underexplored. In this work, we proposed a novel AL strategy, Bidirectional Contrastive Active Learning strategy (BiCAL), that used both image and text latent spaces to identify contrastive samples to select batches to query for labels. BiCAL was robust to class imbalance data problems by its design, which is a problem that is commonly seen in training domain-specific models. We assessed BiCAL’s performance in domainspecific learning on the clinical report generation tasks from chest X-ray images. Our experiments showed that BiCAL outperforms Stateof-the-art methods in clinical efficacy metrics, improving recall by 2.4% and F1 score by 9.5%, showcasing its effectiveness in actively training domain-specific multi-modal models.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 328–341 August 16, 2024. ©2024 Association for Computational Linguistics
328

heading: 1 Introduction
text: Active Learning (AL) is a branch of machine learning that aims to select a small set of the most informative data to annotate for model training (Settles, 2009). This technique allows the model to achieve optimal performance while lowering the cost of annotation. Moreover, by actively selecting data to train on, a model trained under active learning can sometimes surpass the performance that is trained on the full dataset. AL has shown its great potential in the field of natural language processing (NLP) (Shelmanov et al., 2021; Dor et al., 2020; Shen
et al., 2017; Margatina et al., 2021a) and computer vision (CV) (Slade and Branson, 2022; Takezoe et al., 2023). However, relatively few have explored the application of active learning in a multi-modal setting.
In addition, as the capabilities of general largepretrained models arise (Bai et al., 2023; OpenAI, 2023), a rising interest has been seen in fine-tuning them to become domain-specific models. However, when training models in specific domains, obtaining quality labelled data is challenging due to the domain expertise required for accurate annotation, which is costly in both time and money. This motivates us to explore active learning’s application in the domain-specific setting. We identify that in domain-specific active learning, there exists one key challenge – class imbalance in datasets is often seen in domain-specific settings, existing AL methods struggle to actively select samples that have less population but may be more important – in medicine, common (healthy) samples often out populate rare (unhealthy) samples. Models trained
under such active learning strategies converge on the commonly seen samples and perform poorly in identifying rare sickness cases.
In this study, we introduce a novel AL strategy Bidirectional Contrastive Active Learning strategy (BiCAL) that is tailored to address the challenge in domain-specific active learning. We assess BiCAL and other established AL methods on clinical report generation from chest X-ray images. Our key contributions are:
1. We propose a novel AL strategy BiCAL that is able to select rare but important cases inherently to be robust against the class imbalance limitations, which is a common problem in clinical setting.
2. We present an in-depth analysis of existing AL strategies for multi-modal task – clinical report generation.

heading: 2 Related Work
text: This section provides the background of our proposed AL strategy BiCAL. We first formalize the active learning problem under the image-to-text generation task and set up the notation for the rest of the paper. Given a modelM, unlabelled image data pool Xpool. We denote an unlabelled input image as x ∈ Xpool, and the labelled text report as y ∈ Y , where y = (y1, ..., yn), n is the number of tokens in the generated report. We define the labelled data pool Xlabel to contain image-report pairs, where Xlabel ∩ Xpool = ∅ . The whole data pool is Xall := Xlabel ∪ Xpool. The model is parameterized by vector w, as follows:
M = pw(y | x) = pw(y1, ..., yn | x) (1)
An acquisition function representing the query heuristic in the AL setting is denoted as a(x,M). At each active learning iteration, we acquire the label of a batch Q of b number of unlabelled instances from Xpool and add to the labelled data pool Xlabel using a(x,M). The updated labelled data pool Xlabel is used to train the underlying model every iteration. This process iterates until a predefined budget B is depleted. Sampling from the pool is determined by the acquisition function as follows :
x∗ = argmaxx∈Xpool a(x,M) (2)

heading: 2.1 Uncertainty-based and Diversity-based Active Learning
text: Uncertainty-based AL strategies often use a heuristic that can measure the model’s uncertainty toward unlabelled data and choose the unlabelled data with the highest uncertainty (Lewis, 1995; Wang et al., 2019; Shannon, 2001). Gal et al. (2017) demonstrated the idea of measuring model uncertainty by combining Bayesian Active Learning by Disagreement (BALD) (Houlsby et al., 2011) with Bayesian formulation of Neural Networks such as Bayesian by Backprop (Blundell et al., 2015) and MC dropout (Gal and Ghahramani, 2016). However, uncertainty-based active learning typically depends on the underlying training model’s predictions for uncertainty measurements. This dependence results in the “cold-start” problem (Yuan et al., 2020; Ash and Adams, 2020), where these methods are ineffective early in training due to the initial model’s naivety.
On the other side, diversity-based Active Learning aims to select a subset of the data that can best represent the whole dataset, such that the model achieves similar performance to full-tuning when trained on the selected subset. There has been much previous work in this stream of designing AL strategies (Kim et al., 2006; Citovsky et al., 2021; Sener and Savarese, 2018).

heading: 2.2 Hybrid Active Learning
text: There have also been some hybrid AL methods that combine diversity and uncertainty in their design (Ash et al., 2019; Yuan et al., 2020). Approaches that infuse reinforcement learning into AL strategies which learn the selection heuristic from scratch were also seen (Fang et al., 2017; Liu et al., 2018; Vu et al., 2019). There has been close work on active learning to ours in natural language generation and abstractive text summarization, however, they focused on the single modal generation task (Tsvigun et al., 2023; Gidiotis and Tsoumakas, 2021a; Perlitz et al., 2023; Gidiotis and Tsoumakas, 2021b).
The closest work to ours is Contrastive Active Learning (CAL) proposed by (Margatina et al., 2021b). They hypothesized that if two data points are close in the underlying model feature space but result in very different underlying model predictive likelihood, then they may be lying on the model’s decision boundary and therefore are a good can-
didate to query. CAL uses K-Nearest Neighbors (KNN) (Cover and Hart, 1967) to find and record the top k neighbouring points by their model representation encodings from the input. Then it computes the KL divergence (Kullback and Leibler, 1951) between the model’s output probability of each unlabelled instance with their recorded k neighbours. The contrastive score of each unlabelled instance is then calculated by the average of all KL-divergence values of the neighbours. Ultimately, the data point with the highest contrastive score is selected to be queried.

heading: 3 Bidirectional Contrastive Active Learning
text: We identify the following limitations that existing AL methods have when training models in a clinical setting. In clinical settings, data for healthy or common sicknesses is often seen, while unhealthy or rare sicknesses are rare in the population, leading to an imbalanced dataset. This leads to models trained on such datasets that can converge easily on the common cases, and have poor performance on rare but important cases. Previous AL methods have not yet addressed this problem, as they are not able to explicitly identify important cases within the dataset automatically. The original CAL would identify two data points are neighbours if two data points have the same sickness, and if the model predicts differently for the two data points, they are considered as ’contrastive’ and queried. Such a heuristic cannot locate the positive (unhealthy) cases efficiently, because negative (healthy) neighbour pairs would outweigh the positive (unhealthy) neighbour pair in the population, leading to the sampling process suffering from class imbalance and queries too many negative instances. Therefore, models trained using CAL achieve a bad performance in clinical efficacy and recalling positive cases, as revealed by our experiments in Table 1.

heading: 3.1 BiCAL Algorithm
text: BiCAL is robust to class imbalance datasets by its design and can automatically select rare but valuable cases within a dataset for the model to learn. This is done by bi-directionally augmenting the contrastive definition and measuring the contrastiveness in pre-trained embedding space, empowering the algorithm to select rare samples in domain datasets inherently.
We redefine two types of contrastive samples. For
BiCAL, contrastive examples have to satisfy one of the following definitions:
1. Two data points with similar pre-trained embeddings but different pre-trained embeddings of their model generation outputs.
2. Two data points with different pre-trained embeddings but similar pre-trained embeddings of their model generation outputs.
The intuition behind the second augmented definition is that common cases and rare cases will most likely have the most different representations of each other within the dataset. Therefore, if a model generates similar outputs for two data points that have different representations, this means it is highly possible that at least one rare sample is within the two data points, and the current model hasn’t trained enough on at least one of the two data points. Hence by augmenting the contrastive definition in BiCAL, we have increased the chance of querying a rare case, compared to CAL. Moreover, by leveraging pre-trained encoders, we isolate the underlying model in generating the uncertainty measure, alleviating the Cold-Start Learning problem (Yuan et al., 2020; Ash and Adams, 2020) – in the initial stage of training, the underlying model is naive due to the absence of domain knowledge, if we use the underlying model’s encoder to generate uncertainty measure it would result in a decrease in the ineffectiveness of such uncertainty-based AL strategies.
Formally, each data point xi should obtain k number of similar neighbours Xclose and k number of dissimilar neighbours X f ar.
Xclose := f ( Φ(xi),Φ(x j)) < ϵ
X f ar := f ( Φ(xi),Φ(x j)) > γ
(3)
For the first contrastive sample, the data point should satisfy the following condition:
f ( Ω(M(xi)),Ω(M(xmclose)) ) > γ (4)
For the second contrastive sample, the data point should satisfy the following conditions:
f ( Ω(M(xi)),Ω(M(xmf ar)) ) < ϵ (5)
Where Φ(.) ∈ Rd′ is a selected pre-trained image encoder that maps input xi and x j to its feature space. Ω(.) ∈ Rd′′ is the selected pre-trained text encoder that maps the predicted output of underlying
Algorithm 1 Single iteration of BiCAL Input: all data Xall, unlabeled data Xpool, acquisition size b, modelM, number of neighbours k, distance metric function f (.), pre-trained image
(encoding) function Φ(.), pre-trained text (encoding) function Ω(.), contrastive ratio c ∈ [0, 1], Total number of unlabelled data N, . 1 S close := ∅ ; S f ar := ∅ 2 for i in 1, . . . ,N do 3 d j ← f ( Φ(xi),Φ(x j) ) ▷ x j ∈ Xall, j = 1, ...,N 4 Xclose ← Select k number of x∈ Xall with lowest d j ▷ Xclose = {x1close, ..., x k close}; j , i 5 X f ar ← Select k number of x∈ Xall with highest d j ▷ X f ar = {x1f ar , ..., x k f ar} 6 Ŷclose ←M(Xclose) 7 Ŷ f ar ←M(X f ar) 8 ŷi ←M(xi)
9 siclose ← 1 k k∑ m=1 f ( Ω(ŷi),Ω(ŷ mclose) ) 10 sif ar ← 1 k k∑ m=1 f ( Ω(ŷi),Ω(ŷ mf ar)
) 11 S close := S close ∪ {siclose} ; S f ar := S f ar ∪ {s i f ar} 12 end 13 Q1 ← Select b × c number of x ∈ Xpool with the highest sclose ▷ sclose ∈ S close 14 Q2 ← Select b × (1 − c) number of x ∈ Xpool with the lowest s f ar ▷ s f ar ∈ S f ar
Output: Q1 ∪ Q2
model ŷi to its feature space. f(.) is a distance metric, such as Euclidean distance or cosine similarity. ϵ and γ represent the threshold for a very small and a very large distance value respectively, although in practice we adopt ranking instead of using a threshold. M(.) is the underlying training model of the active learning loop, such that ŷi ← M(xi). We detail the single iteration of BiCAL’s algorithm as follows:
Compute Neighbours We use the encoding function from the pre-trained model Φ(.) to map all the data points to its pre-trained embedding space. For each unlabelled instance xi, we use cosine similarity f (.) to measure the distances between the embeddings of xi and all the other data points in the Xall (line 3). We record xi’s nearest (top k) and furthest (bottom k) neighbours in the embedding space by the distance calculated (lines 4-5).
Compute Contrastive Scores The unlabelled instance xi and all its neighbours Xclose and X f ar will be passed to the underlying modelM to generate their text outputs ŷ (lines 6-8). The generated text from the model is then encoded by the selected pretrained language model Ω(.) to obtain text embedding of the generated text. Using these embeddings, we can calculate two different contrastive scores for the unlabelled instance xi (lines 9-10). The first contrastive score siclose is calculated by the average distance between the embedding of generated output of the unlabelled instances with their nearest neighbours, and the second one sif ar is calculated with its furthest neighbours.
Query Two Contrastive Batches For each unlabelled instance xi, we obtain two lists of contrastive scores S close and S f ar. We select the unlabelled instances using the two contrastive scores separately. For S close, we select the top b × c number of instances, where b is the total intended batch size for query, and c is a hyperparameter “contrastive ratio” that controls the ratios of samples sampled from the two contrastive definitions. This gives us a batch of instances Q1 of the first contrastive definition (line 13). For S f ar, we select the bottom b × (1 − c) number of instances. This gives us a batch of instances Q2 of the second contrastive definition (line 12). Ultimately, two batches Q1 and Q2 combines to give the output of BiCAL.

heading: 4 Experiment Settings
text: We assess BiCAL and other established AL methods’ performance in training general multi-modal models to specify on the task of clinical report generation from chest X-ray images. In every active learning loop, the underlying model denoted as M, was fine-tuned twice on the labelled pool Xlabel. Subsequently, we evaluated the model on the test dataset using various NLG metrics. Each experiment was run in 3 folds with different random seeds, each fold containing 10 active learning iterations, where 100 data points were queried per iteration, i.e. 1000 data points were queried in total. This choice of 1000 data points reflects realworld scenarios where active learning is applied when labelled data is not available. Our goal was to examine the efficiency and performance of active learning methods under constrained labelling budgets in a medical setting. In real-world AL sce-
narios, labelling a large size of unlabelled data is often impractical due to the significant expertise labelling effort required. Therefore, 1000 data points were deemed sufficient to assess the performance of the AL methods in our focus while mimicking a real-world AL situation. Future work could explore varying the number of training examples (e.g., 1500, 2000) to understand further the impact of labelled data quantity on active learning strategies in training medical models.

heading: 4.1 Baselines
text: We evaluate our proposed BiCAL against various literature Active Learning strategies:
1. Random Sampling (RS): Unlabelled instances are drawn at random.
2. Normalized Sequence Probability (NSP): Uses the probability of the generated sequence by the model as a measure of uncertainty.
NSP = 1−exp 1n n∑ i=1 logP(yi | y1 . . . , yn, x) 
(Tsvigun et al., 2023; Wang et al., 2019).
3. Expected Normalised Sentence Probability (ENSP): Bayesian AL method where it has the same intuition as NSP.
ENSP = 1 − Ew∼qθ̂ p̄w(y|x)
(Tsvigun et al., 2023; Ueffing and Ney, 2007; Wang et al., 2019).
4. Expected Normalised Sentence Variance (ENSV): Similar to ENSP but uses variance instead of expectation between the sequence probability.
ENSV = Varw∼qθ p̄w(y|x)
(Tsvigun et al., 2023; Ueffing and Ney, 2007; Wang et al., 2019).
5. Contrastive Active Learning (CAL): SOTA AL method described in section 3 (Margatina et al., 2021b).
In addition, for BiCAL, we implemented two variants by varying the choice of pre-trained image encoder Φ(.) in the BiCAL algorithm. We have experimented with two types of pre-trained models, Dinov2 and CheSS, to examine the effect of different types of pre-trained image encoders in our algorithm. Dinov2 is an image model that is pre-trained
on a general image dataset (Oquab et al., 2023), whereas CheSS is pre-trained on a CXR dataset (Cho et al., 2023). For the pre-trained text encoder Ω(.), we have fixed the selection to GatorTron (Yang et al., 2022) based on its SOTA performance in clinical NLP tasks (that outperforms BioBERT (Lee et al., 2019), ClinicalBERT (Huang et al., 2020), BioMegatron (Shin et al., 2020)).

heading: 4.2 Datasets
text: We used the labelled datasets MIMIC-CXR (Johnson et al., 2019a) and IU X-Ray (Demner-Fushman et al., 2015) for our simulation of active learning conditions. The IU X-Ray dataset contains 3,955 radiology reports with 7,470 associated chest Xray images, while MIMIC-CXR includes 227,835 radiology reports with 377,110 associated chest X-ray images. Following the methodology from Chen et al. (2022), we excluded samples without accompanying reports. We partitioned the IU XRay dataset into training and testing sets using an 85%:15% ratio and used the official train-test split for MIMIC-CXR.
In our simulated active learning experiments, we queried only 1,000 data points. As it was impractical in terms of running time to run the experiment on the entire MIMIC-CXR dataset of 377,110 images, we leveraged the structured labels from MIMIC-CXR-JPG (Johnson et al., 2019b) and conducted stratified sampling to obtain a 10% subset of the training split (34,463 data points). This ensured that the subset closely mirrored the label distribution of the full MIMIC-CXR dataset. We used this stratified subset for training and the official test set for evaluation. We release the processed reports with their image IDs for both datasets in CSV files in the repository and provide the data distribution of MIMIC-CXR before and after subset sampling in Table 5 and 6 in the Appendix.

heading: 4.3 Setup
text: Experiments were conducted on a single NVIDIA RTX6000 GPU. We adopted the AdamW optimizer (Loshchilov and Hutter, 2019) with a learning rate of 3e-5 and a weight decay of 3e-7. A warm-up scheduler was applied to the learning rate for the initial 200 steps. Due to computational constraints, we used a training batch size of 8 and limited the maximum number of tokens for generation to 100.
In our experiment, we fine-tuned a vision encoderdecoder model initialized with pre-trained Vision
Transformers (ViT) (Dosovitskiy et al., 2020) and GPT-2 (Radford et al., 2019). These models were chosen for their popularity and strong performance in computer vision and natural language processing, respectively. Our primary focus was to investigate active learning strategies in a multi-modal task, so we did not explore other model choices. We utilized HuggingFace (Wolf et al., 2020) and Deepspeed (Rasley et al., 2020) to facilitate our experiment setup.

heading: 5 Results and Analysis
text: We used two types of evaluation metrics: natural language generation (NLG) metrics and domainspecific (clinical efficacy) metrics. This provided a comprehensive evaluation of the generated reports in terms of general and domain-specific performance. For NLG metrics, we reported BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) scores at each active learning iteration. For clinical efficacy metrics, we used the CheXpert (Irvin et al., 2019) model to label the generated and reference reports. We reported precision, recall, and F1 scores for the labeled categories of the generated and reference reports. This evaluation approach is widely used in chest X-ray clinical report gen-
eration tasks (Chen et al., 2022; Liu et al., 2019, 2021).

heading: 5.1 Clinical Efficacy Metrics
text: We first assessed the baseline methods and our strategy after 1000 queries on MIMIC-CXR using domain-specific metrics to examine the performance of active learning (AL) strategies, which is crucial for training clinical models. Table 1 displays the clinical efficacy metrics of various AL strategies based on 1000 data queries from a MIMIC-CXR dataset subset. The table’s last three rows show the performance of our underlying model after fine-tuning for 10 epochs on the full MIMIC-CXR dataset subset, R2Gen (Chen et al., 2022), and the model (CCR) from Liu et al. (2019). These latter two are fully supervised models trained on the full MIMIC-CXR dataset, designed to excel in chest radiology report generation tasks, with their performance referenced directly from their published papers.
A notable observation is that BiCAL CheSS surpassed baseline methods in recall and F1 scores while maintaining a competitive average precision score. This suggests that the BiCAL CheSS approach effectively recognizes more rare cases (un-
healthy scenarios) than other AL strategies, though it may occasionally increase false positives, as indicated by the precision score. In medical diagnostics, catching every potential disease case (reducing false negatives) is crucial. Therefore, high recall is preferable to high precision, making BiCAL’s performance desirable in our context and demonstrating BiCAL CheSS’s superiority in generating clinically accurate reports.
Remarkably, the BiCAL CheSS method achieved a recall score that surpasses models fine-tuned on the entire MIMIC-CXR subset (Full Tune). Additionally, it achieved competitive performance with fully supervised models R2Gen and CCR, with a better recall score and an F1 score not much lower. This is noteworthy, considering this performance was achieved with only 1000 data points (less than 0.3% of the whole MIMIC-CXR).
An interesting observation is that although CAL performed well in the NLG metrics on the MIMICCXR dataset (Figure 3), its clinical precision and recall scores were the least impressive among all methods. This suggests that while CAL trains models to produce seemingly accurate reports, these might not be clinically sound. By augmenting the contrastive bidirectionally and utilizing pre-trained encoders, the domain-specific performance of this contrastive active learning approach is largely enhanced, demonstrating the success of our approach. We include a case study of generation performance on rare cases using various AL methods in Table 7
in the Appendix.
Furthermore, evidence of the task’s complexity is seen in the last three rows of Table 1. These rows include results from R2Gen and CCR, models specifically tailored for chest X-ray report generation and comprehensively trained on the full MIMIC-CXR dataset. Despite their specialized design, their clinical performance remains relatively low. This underscores the inherent challenge of our downstream task—clinical report generation. The intricacies in medical images may be difficult for the underlying model’s capability to learn, suggesting that to truly elevate clinical accuracy, superior clinical models adept at the task may need to be designed.

heading: 5.2 Natural Language Generation Metrics
text: We found that for the IU X-ray dataset, no single strategy consistently outperformed the others. Notably, RS and NSP showed marginally better performance during the initial four iterations in both BLEU and ROUGE metrics. For the MIMIC-CXR dataset, CAL performed slightly better in ROUGE scores, while BiCAL was competitive with CAL in BLEU scores, as shown in Figure 3.
The varying performance of CAL across the MIMIC-CXR and IU X-ray datasets suggests that CAL’s superiority did not extend to the IU X-ray dataset. This may be due to the different data volumes. Smaller datasets result in a limited unlabeled data pool, potentially narrowing batch sample variance and minimizing observable performance vari-
ance. Consequently, the queried batches of different AL strategies on the IU X-ray dataset have more overlap than on MIMIC-CXR, leading to similar performance across strategies.
For the BLEU score, BiCAL Dinov2 performed better than all strategies before 500 queries but was surpassed by CAL afterwards (≥ 500), though it remained competitive. For ROUGE scores, CAL consistently retained slightly better performance starting from 300 queried data. This comparison demonstrates BiCAL’s competitiveness in NLG metrics. As shown in Table 3, after 1000 queries, BiCAL Dinov2 achieved the best performance in all BLEU scores and the second-best performance in all ROUGE scores.
Although BiCAL only surpassed other literature AL methods in some NLG metrics, it remained competitive with the best-performing baseline methods. However, language models have been criticized for producing hallucinated text (Ouyang et al., 2022; Stiennon et al., 2020; Ziegler et al., 2019). In a medical setting, our priority is creating accurate clinical reports, not just authoritativesounding ones. We believe the relatively worse performance of BiCAL CheSS is due to the hallucination problem of LLMs.
CAL and other methods suffer from class imbalance data and may select more healthy cases for training, leading to hallucinated models, that are good at generating good negative (healthy) reports containing many common phrases. In contrast, BiCAL may have a higher proportion of positive cases, training a model with higher clinical efficacy. However, this model’s ability to write comprehensive healthy reports that match the reference deteriorates. This results in worse performance on NLG metrics due to the class imbalance problem (more negative cases than positive in the test set, causing the model to generate negative reports more often). This hypothesis is supported by our analysis of the generation results of the models under different active learning methods, including a case study in Table 2. It can be seen that although all reports are saying the candidate contains no significant diseases, but other methods learn to give a more comprehensive healthy report, which results in a higher BLEU score. Thus due to the imbalanced dataset problem, the average NLG score of the other methods may exceed BiCAL despite being less clinically accurate in positive cases (shown
in clinical efficacy metrics in Table 1). We also include a positive case study from our analysis to show BiCAL’s ability to train clinically accurate models in Table 7.

heading: 5.3 Ablation Study
text: In Sections 5.1 and 5.2, we discussed the impact of different image encoders on the BiCAL algorithm, comparing those pre-trained on a general image dataset (Dinov2) and a Chest X-ray dataset (CheSS). Additionally, a crucial component of the BiCAL algorithm is the contrastive ratio, denoted as c, which determines the sampling ratio between two contrastive definitions in a batch. Our previous experiments used a default c value of 0.5, meaning an equal split between the two contrastive definitions. As shown in Table 4, for clinical efficacy metrics, BiCAL performs best when c is 0.5 in terms of clinical recall and F1 scores. For clinical precision, a c value of 0.75 seems optimal. The poorest performance in terms of clinical recall is observed at c = 0.25. This suggests that while a c value of 0.5 may not be the best for NLG metrics, it ensures the generation of higher clinical quality reports by achieving the best recall of diseases in the generated reports.

heading: 6 Conclusion
text: In this work, we present a study on the effectiveness of current active learning methods for domainspecific multi-modal learning, specifically on the task of clinical report generation from chest Xray images. We identified the challenge of class imbalance in domain-specific active learning and addressed it by introducing BiCAL, a new active learning technique. BiCAL excelled in both NLG and domain-specific (clinical efficacy) metrics, notably outperforming baselines in clinical recall and F1-score.
We found that existing AL strategies demonstrate similar performance in NLG metrics for the task
of clinical report generation from chest X-ray images. This may be due to the complexity of our task, which requires training the model to acquire clinical expertise to generate accurate and clinically sound reports. Interestingly, our tests revealed that an AL strategy’s high performance in NLG metrics does not ensure equal success in domain-specific (clinical) performance, possibly due to the hallucination properties of language models. We hope this work provides valuable insights and can act as a starting point for researchers in the future on the task of active learning in multi-modal clinical tasks.
Ethical Consideration and Limitations
We note that despite the success of BiCAL in our study of clinical report generation, in practice, its performance is yet to be confirmed. We have simulated our experiments based on a labelled dataset where the radiology report was collected under a monitored condition such that their format may achieve a certain level of consistency (Johnson et al., 2019a; Demner-Fushman et al., 2015). However, in practice, the queried data’s label report may vary based on different radiologist labellers, which may cause noise in the training dataset, which may affect the effectiveness of BiCAL.
We identify that for this work have used sensitive personal data that is related to the health sector. We used MIMIC-CXR (Johnson et al., 2019a) and IU X-Ray (Demner-Fushman et al., 2015) datasets in this project. We note that both datasets have been de-identified, where they have removed all personal health information (PHI). This has ensured the privacy and confidentiality of the individuals. During this project, we handled the data responsibility and used it only for the purpose of research. No attempt at re-identification of the datasets is made. We have also signed the data use agreement for MIMIC-CXR before we use the data. We note that MIMIC-CXR and IU X-rays, just like all datasets, may contain inherent biases based on patient information such as where the data is collected. Moreover, active learning is a technique that samples data based on a certain heuristic, which therefore may introduce additional bias in the sampling and training of the model. This work researches the effectiveness of active learning in clinical report generation, we recognize this potential bias that may be introduced by our research, and this also comes along with our work’s contribution to the
improvement of the field of active learning in the clinical sector.
Due to the difficulties in acquiring publicly available domain-specific image-report pair data, we chose to work with the task of clinical report generation from chest X-rays. As we designed the BiCAL algorithm, it was not tailored to the clinical report generation task that we conducted our experiments. Moreover, we believe that with the intricacies and high level of expertise required in the medical domain, we believe experiments conducted in this domain can provide valuable insight and act as a good reference for AL’s performance on domain-specific learning in general. However, further work should be done in the future to test the performance of BiCAL in other domains, given that the data is available.

heading: 7 Appendix

heading: Future Work
text: We hope this work provides valuable insights and can act as a starting point for researchers in the future on the task of active learning in multi-modal clinical tasks. Ethical Consideration and Limitations
We note that despite the success of BiCAL in our study of clinical report generation, in practice, its performance is yet to be confirmed. We have simulated our experiments based on a labelled dataset where the radiology report was collected under a monitored condition such that their format may achieve a certain level of consistency (Johnson et al. , 2019a; Demner-Fushman et al. , 2015). However, in practice, the queried data’s label report may vary based on different radiologist labellers, which may cause noise in the training dataset, which may affect the effectiveness of BiCAL. We identify that for this work have used sensitive personal data that is related to the health sector. We used MIMIC-CXR (Johnson et al. , 2019a) and IU X-Ray (Demner-Fushman et al. , 2015) datasets in this project. We note that both datasets have been de-identified, where they have removed all personal health information (PHI). This has ensured the privacy and confidentiality of the individuals. During this project, we handled the data responsibility and used it only for the purpose of research. No attempt at re-identification of the datasets is made. We have also signed the data use agreement for MIMIC-CXR before we use the data. We note that MIMIC-CXR and IU X-rays, just like all datasets, may contain inherent biases based on patient information such as where the data is collected. Moreover, active learning is a technique that samples data based on a certain heuristic, which therefore may introduce additional bias in the sampling and training of the model. This work researches the effectiveness of active learning in clinical report generation, we recognize this potential bias that may be introduced by our research, and this also comes along with our work’s contribution to the
improvement of the field of active learning in the clinical sector. Due to the difficulties in acquiring publicly available domain-specific image-report pair data, we chose to work with the task of clinical report generation from chest X-rays. As we designed the BiCAL algorithm, it was not tailored to the clinical report generation task that we conducted our experiments. Moreover, we believe that with the intricacies and high level of expertise required in the medical domain, we believe experiments conducted in this domain can provide valuable insight and act as a good reference for AL’s performance on domain-specific learning in general. However, further work should be done in the future to test the performance of BiCAL in other domains, given that the data is available