A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning (2024)

Abstract: Diagnostic Captioning (DC) automatically generates a diagnostic text from one or more medical images (e.g., X-rays, MRIs) of a patient. Treated as a draft, the generated text may assist clinicians, by providing an initial estimation of the patient’s condition, speeding up and helping safeguard the diagnostic process. The accuracy of a diagnostic text, however, strongly depends on how well the key medical conditions depicted in the images are expressed. We propose a new data-driven guided decoding method that incorporates medical information, in the form of existing tags capturing key conditions of the image(s), into the beam search of the diagnostic text generation process. We evaluate the proposed method on two medical datasets using four DC systems that range from generic image-to-text systems with CNN encoders and RNN decoders to pretrained Large Language Models. The latter can also be used in fewand zero-shot learning scenarios. In most cases, the proposed mechanism improves performance with respect to all evaluation measures. We provide an opensource implementation of the proposed method at https://github.com/nlpaueb/dmmcs.

heading: 1 Introduction
text: Diagnostic Captioning (DC) systems receive one or more medical images of a patient, such as X-Rays or Magnetic Resonance Images (MRIs), which they analyse to draft a diagnostic report (Ting et al., 2023). Such systems can function as supportive tools for doctors and clinical staff, assisting them in their daily workload. Possible benefits, as summarized by Pavlopoulos et al. (2021), include (i) increased overall throughput of medical departments, since improving a partially correct draft report may be faster than writing it from scratch, (ii) reduced diagnostic errors, by providing suggestions for the clinical findings of the input images, which might
*Corresponding author.
otherwise be missed, and (iii) decreased cost of medical imaging examinations. Despite the rapid advancements in deep learning methods, draft diagnostic reports generated by DC systems still exhibit shortcomings, such as hallucinations or lack of accurate descriptions of medical findings (Xu et al., 2023). The medical accuracy of a generated diagnostic text strongly depends on whether key medical conditions depicted in the images are considered during text generation (Huang et al., 2019; Wang et al., 2020). Such key conditions can be captured by tags, reflecting medical concepts to be mentioned in the generated text. Tags of this kind can be obtained by medical image taggers (Rajpurkar et al., 2017; Lu et al., 2020) and are also present (as gold tags) in diagnostic captioning datasets. Assigning tags to an image is to some extent similar to content selection, i.e., deciding which concepts to express, which was the first stage in symbolic-based text generation systems (Reiter and Dale, 2000). More background information on DC is provided in Appendix A, and in the DC survey by Pavlopoulos et al. (2021).
In this work, we propose Distance from Median Maximum Concept Similarity (DMMCS), a novel data-driven guided decoding method that aims to integrate information from medical image tags into the diagnostic text generation process. This is achieved by imposing a new penalty at each decoding step. The penalty is designed to prioritize the generation of words that are semantically similar to the medical tags of the input images, also taking into account how often each tag is explicitly or implicitly expressed in gold captions. DMMCS involves calculating a series of statistical distributions that model the relationship between each tag and the tokens of the diagnostic captions it is associated with in the training data. It is the first guided decoding method specifically developed for DC, as well as the first data-driven method that uses image tags to guide the generation of image captions.
7450
DMMCS is applicable to any encoder-decoder DC system, as it can be integrated in the decoding process. For experimental purposes, we train four DC systems, ranging from a generic CNNRNN image-to-text method (Vinyals et al., 2015), to Transformer-based architectures (Vaswani et al., 2017), and state-of-the-art prompt-based systems (Dai et al., 2023; Alayrac et al., 2022). Furthermore, we investigate the impact of DMMCS on few-shot captioning scenarios. We evaluate the performance of all models on two medical datasets, ImageCLEFmedical 2023 (Rückert et al., 2023) and MIMIC-CXR (Johnson et al., 2019). We use two evaluation measures, BLEU and BLEURT, comparing the results with and without DMMCS, demonstrating the effectiveness of the proposed algorithm. As one would expect, the performance boost provided by DMMCS is larger when using gold tags, but we show that DMMCS is also beneficial with noisy tags predicted by medical image classifiers.
Our main contributions are: (i) We introduce DMMCS, a data-driven guided decoding method for DC, which leverages medical tags of the input images to improve the generated captions. (ii) We incorporate DMMCS in four DC systems covering a range of learning scenarios, including finetuning and few-shot learning. (iii) We demonstrate that DMMCS significantly enhances performance across all four models in most cases, even when using noisy predicted tags.

heading: 2 Related Work
text: Substantial research has been dedicated to Controllable Text Generation and guided decoding strategies (Prabhumoye et al., 2020). Standard decoding methods, such as greedy or standard beam search, provide minimal control over the model’s output (Zhou et al., 2023). Therefore, decoding techniques that partially guide the model’s choices to adhere to task-specific requirements have been proposed. Most guided decoding methods can be categorized based on three control conditions: semantic, structural, and lexical (Zhang et al., 2023).
Semantic constraints guide the model to generate text conforming to specific attributes such as tense or sentiment (Yang et al., 2018; Gu et al., 2022). For instance, Ghazvininejad et al. (2017) introduced Hafez. It was initially designed for generating poetry-styled texts, but is adaptable to meet any specified content-based decoding constraint. This is achieved by integrating a constraint-specific
score in the next word selection process at each decoding step. The score is calculated by a feature function. Such functions can, for example, encourage or discourage specific word choices, prevent repetitions, or guide the model to prefer longer words. The constraints may also be based on supervised learning (Holtzman et al., 2018) or reinforcement learning (Li et al., 2017), rather than just heuristics (Baheti et al., 2018).
Structural constraints direct the model’s output to adhere to a specific syntax structure (Yang et al., 2022; Chen et al., 2019; Kumar et al., 2020). Finally, lexical-based constraints guide the model’s choices to incorporate a set of specified keywords. He (2021) introduced CBART (constrained BART), which shifts a part of the generation process from the decoder to the encoder. The encoder guides the generation towards some specified must-include tokens. This is achieved by adding a dense layer over BART’s encoder (Lewis et al., 2020) that generates a sequence of labels. The latter guides the decoder on which actions should be taken. The model undergoes a refinement process, regenerating multiple outputs until all constraints are fulfilled.
A recent related method for semantic guided decoding, which comprised the starting point of our research, is Contrastive Search (Su et al., 2022). It attempts to tackle the problem of text degeneration, where models produce unnatural and repetitive text. It imposes a degeneration penalty at each decoding step to guide the model’s output towards more natural and less repetitive text sequences. The penalty for a candidate token u at decoding step t is:
Du = max 1≤j≤t−1 sim (h(u), h(xj)), (1)
where xj are the preceding tokens of the incomplete generated sequence x, h (·) calculates word embeddings, and sim(·, ·) denotes cosine similarity. The degeneration penalty Du is the maximum cosine similarity between the word embeddings of the candidate token u and the preceding sequence x<t. It is subtracted from the score that the decoder would otherwise assign to u, thus guiding the decoder to generate less repetitive text.

heading: 3 The Proposed DMMCS Method
text: Our proposed method introduces a tag-guided decoding strategy for DC systems. It aims to guide the model to select words that appropriately express the tags (medical concepts) associated with
the input image.1 For instance, if a radiology image is associated with the tag “Atelectasis”, but the generated caption makes no implicit or explicit reference to the aforementioned medical condition, then it is probably inaccurate.
As a first exploration, we calculated the FastText word embeddings (Bojanowski et al., 2017) of all the tags and all the tokens of the gold captions in the training set of the ImageCLEF 2023 dataset (Rückert et al., 2023). Tags consisting of multiple tokens were represented by the centroid of the tokens’ embeddings. We then investigated the relationship between each tag and the gold captions it was associated with (the gold captions of training images tagged with the particular tag). Figure 1 presents a heatmap that visualizes the relationship between the tokens of a caption s (x-axis) and its corresponding set of tags T (y-axis). Each heatmap cell represents the cosine similarity between the (centroid) embedding of the corresponding tag t and the respective caption token sj . Darker colors correspond to larger similarity values. Hence,
for instance, the lower right cells show that tokens s11 and s12 (“pancreas head”) of the caption have a very high cosine similarity with tag t3 (“Head of pancreas”). Indeed, t3 is almost explicitly expressed in s (almost the same words), while the other two tags are expressed more implicitly (with different words, e.g., “Neoplasms” vs. “tumour”, “X-Ray CT” vs. “computed tomography”). We define the similarity between a tag t and a caption s as the maximum cosine similarity (MCS) between the (centroid) word embedding of t and the word embedding of each token in s, i.e.,
MCS(t, s) = max 1≤j≤|s| sim(h(t), h(sj)). (2)
1In our experiments, the input is always a single image (and its tags), but DMMCS also applies to multi-image inputs.
Tag: Angiogram - Boxplot
A high MCS(t, s) between a tag t and a caption s indicates a significant presence of the tag’s meaning in the caption. Next, we investigated the relationship between each tag t and the set S of all the gold captions tag t is associated with in the training data (gold captions of images tagged with t). For each tag t and its associated captions S, we compute the distribution R(t, S) of its corresponding MCS scores, as the set:
R(t, S) = {MCS(t, s)|s ∈ S}. (3)
Figure 2 illustrates the distribution R(t, S) of the tag “Angiogram”, using the gold captions of the images associated with this particular tag in the ground truth of the ImageCLEF 2023 training set. The blue box represents the interquartile range (IQR) of the distribution, while the coral line denotes the median value, denoted MMCS(t):
MMCS(t) = median(R(t, S)). (4)
We repeated the calculation of the distribution R(t, S) for every tag t and the set S of gold captions associated with t in the training set of ImageCLEF 2023. Figure 3 shows the interquartile range (IQR, black vertical lines) and MMCS(t, S) (median, coral line) of the distribution R(t, S) for each tag t. In other words, Figure 3 contains many box-plots, like the one of Figure 2, side by side. Intuitively, Figure 3 shows how strongly each tag t is expressed in the gold captions associated with it. The side-by-side box-plots of Figure 3 are sorted by ascending MMCS(t, S) (coral), in order to highlight the observation that the tags of the ImageCLEFmedical 2023 dataset are not expressed equally strongly in the ground truth captions. For instance, we observe a variation in MMCS(t, S) values (coral line), ranging from as low as 0.3 for
tags on the left end, to values close to 1 for tags on the right end. The former are overall expressed more implicitly in the diagnostic captions they are associated with, while the latter are explicitly mentioned. Some tags conveying information that may be trivial to a clinician (e.g., that the image is an X-Ray) may actually not be expressed at all (not even implicitly).
Considering these findings, we propose a new decoding penalty that aims to improve the generated diagnostic captions by integrating information provided by the image’s medical tags. The tags are in practice predicted by a medical image tagger, but we experiment with both predicted and gold (oracle) tags. The penalty encourages the decoder to select words that express more or less explicitly (or not at all) the tags of the image. The target level of explicitness of each tag is determined by its MMCS(t, S) score, which is computed on the training captions. Tags with larger MMCS(t, S) should be explicitly mentioned, while tags with lower MMCS(t, S) should be expressed less explicitly (or not at all).
During inference, if an image is associated with a single tag t, we calculate the MCS(t, s) (Eq. 2) between the tag t and the tokens of each candidate (possibly still incomplete) caption s being considered by the beam search decoder. The penalty is the squared difference between the computed MCS(t, s), which shows how strongly the tag is expressed in the candidate caption, and the tag’s MMCS(t, S), which shows how strongly the tag is expressed (median value) in the ground-truth training captions it is associated with. If an image is associated with multiple tags, then a separate
penalty is calculated for each tag, as above, and the total penalty is the sum of the penalties divided by the number of associated tags. Formally, given a (possibly incomplete) caption s and a set of image tags T to be expressed, the penalty is calculated as:
DMMCSp(T, s) =
1 |T | · ∑
t∈T (MCS(t, s)−MMCS(t))2. (5)
Intuitively, the goal is to generate a caption that expresses each tag of the image as strongly as the training captions associated with the tag.
At each decoding step, each candidate (possibly incomplete) caption s considered by the beam search decoder is scored as follows:
DMMCS(s) = α · DMMCSp(T, s) + (1− α) · (1− Dscore), (6)
where T is the set of tags the input image is associated with. Dscore and α are explained below. Dscore: This is the score the decoder assigns to each candidate caption, i.e., the sum of the (log) probabilities of the decoder (Eq. 7). The decoder can be any type of generative model conditioned on the input image, e.g., a Recurrent Neural Network (RNN) or a Transformer-based architecture (Vaswani et al., 2017). We conducted experiments with both types of architectures in order to test the effectiveness of the DMMCS-based decoding. Dscore is also normalized in [0, 1], using min-max scaling, in order to align with the score range of DMMCSp. As the goal is to minimize the overall score, Eq. 6 uses 1− Dscore.
Dscore = − t∑
t=1
logP (st|s<t) (7)
α: This hyper-parameter controls the effect of the two terms in the overall DMMCS(s) score. The larger the value of α, the more significant the influence of the penalty DMMCSp on the overall score, and vice-versa. When α = 0, standard beam search is applied. The value of α is tuned by experimenting with several values on the validation set.
In summary, at each decoding step the DMMCS(s) score (Eq. 6) is calculated for each candidate sequence s of the beam search. The score combines DMMCSp (Eq. 5) and Dscore (Eq. 7). The former measures how well the input image’s tags are expressed in the candidate (possibly incomplete) caption, while the latter denotes the score
assigned by the decoder to the candidate caption. The n sequences with the best DMMCS(s) scores are selected and expanded at the next decoding step. Algorithm 1 presents our proposed guided decoding method in pseudo-code form.
Algorithm 1 DMMCS algorithm - Pseudocode T, S, I ▷ the dataset tags, captions, images i ∈ I ▷ an image in question tagspi , tags g i ⊆ T ▷ predicted, gold tags of i
Phase 1 - Training Statistics
for t ∈ T do S′← {si ∈ S : t ∈ tagsgi } ▷ captions per t MMCS(t)← median{MCS(t, s)|s ∈ S′} ▷ Eqs. 2–4 end for
Phase 2 - Inference
for each decoding step of the beam search do for each beam search sequence s do
get Dscore ▷ decoder’s score of s
▷ Calculate DMMCS penalty (Eq. 5) T ′ ← tagspi ▷ predicted tags of i DMMCSp ← 1
|T ′| ∑
t∈T ′ (MCS(t, s)−
MMCS(t))2
▷ Calculate DMMCS score (Eq. 6) DMMCSs ←α · DMMCSp + (1− α) · (1−Dscore) ▷ Set updated beam scores beamScores[s]← DMMCSs
end for end for

heading: 4 Experiments
text: For experimental purposes, we employed DMMCS decoding on four DC models, spanning diverse architectures and learning techniques, including longestablished generic image captioning models and more advanced state-of-the-art ones. Moreover, we evaluated DMMCS using two datasets: the dataset of ImageCLEFmedical 2023 (Rückert et al., 2023), and a subset of the MIMIC-CXR dataset (Johnson et al., 2019). The performance of each model is evaluated using two measures, briefly discussed in
§4.3. We demonstrate that DMMCS leads to substantial performance improvements in most cases.

heading: 4.1 Datasets
text: ImageCLEFmedical 2023 Dataset: This dataset (Rückert et al., 2023) is a subset of the Radiology Objects in Context (ROCO) dataset (Pelka et al., 2018). It consists of 71,355 images, accompanied by their gold tags and gold diagnostic captions. The maximum number of words in a single caption is 315, the minimum is 1, while the average is 16.04. The dataset contains 2,125 distinct tags, and the images are from multiple medical modalities, such as X-Ray, CT, MRI. No ground truth captions for the test set are available. We split the provided data into three subsets, following a 75%-10%-15% split, holding out a test subset for evaluation purposes. Thus, we employed 53, 516 images as our training data, 7, 135 images as our validation set, while the remaining 10, 704 images comprised our held-out test set.
MIMIC-CXR Dataset: MIMIC-CXR (Johnson et al., 2019) originally consists of 377,110 deidentified chest X-ray examinations. Each of them is accompanied by a set of gold medical tags and a gold diagnostic caption. The maximum number of words in a single caption is 1, 014, the minimum is 2, while the average is 137.8. There are 14 distinct biomedical tags in total, generated by the CheXpert labeler system (Irvin et al., 2019b). We only considered examinations containing a single radiology image per patient, because the DC systems we used could only receive a single image as input. Thus, we used 32,637 images, along with their gold tags and captions. We randomly split them into training, validation and test sets, containing 25,301, 2,752 and 4,584 samples, respectively.

heading: 4.2 DC Models
text: Show & Tell: This is a basic generic image captioning model (Vinyals et al., 2015), consisting of a CNN image encoder followed by an RNN text decoder (no attention used).2 The CNN extracts image features that are then fed to the RNN decoder, which generates the diagnostic caption word by word, based on the image features.
ViT-GPT2: This is also based on the encoderdecoder approach, but now both components are based on Transformers (Vaswani et al., 2017). Vi-
2Show, Attend & Tell (Xu et al., 2015) with attention performed worse in DC in preliminary experiments.
sion Transformer (ViT) (Dosovitskiy et al., 2021) is employed as the image encoder, while GPT2 (Radford et al., 2019) is responsible for caption generation. Both models were loaded from a HuggingFace checkpoint3 for a joint ViT-GPT2 encoderdecoder pipeline, and were further pre-trained on generic image-caption pairs. We then fine-tuned the model on the two employed DC datasets.
InstructBLIP: This is vision-language instructiontuned model (Dai et al., 2023). Such models are designed to swiftly adapt to new tasks based on specific instructions. Hence, their performance strongly depends on the provided instructions (presented in Appendix F).
Flamingo: This is a few-shot (in-context learning) generic image captioning system (Alayrac et al., 2022). Flamingo can generate a diagnostic caption based on a few demonstrative examples of image-caption pairs provided as a multi-modal prompt. Checkpoints of the original Flamingo architecture are not publicly available, hence we employed OpenFlamingo, an open-source implementation that obtains around 80-89% of the original Flamingo’s performance (Awadalla et al., 2023).

heading: 4.3 Evaluation Measures
text: BLEU, BLEURT: We use BLEU (Papineni et al., 2002) and the more recent (BERT-based) BLEURT (Sellam et al., 2020) as main evaluation measures. Both are frequently used in text generation, including generic image-to-text and DC.4
Clinical Accuracy: Previous work has shown measures like BLEU and BLEURT may not adequately capture clinical correctness in DC (Pavlopoulos et al., 2021). Hence, we also measure clinical accuracy (CA) by comparing medical concepts extracted from the generated diagnostic captions (as silver labels by a multi-label text classifier) to those extracted from the corresponding gold captions. To compute CA, we follow previous work (Liu et al., 2019) that employs CheXBERT (Irvin et al., 2019a) to determine whether or not each one of the 14 thoracic ailments (treated as classes) are mentioned in a given diagnostic caption. Given a caption (generated or gold), CheXBERT produces “present”, “negative”, “unsure” or “blank”, for each one of the 14 ailments. We treat “blank” as “negative”. When the “unsure” label is predicted, we change the label to either “present” or “negative” with equal prob-
3 https://huggingface.co/nlpconnect/vit-gpt2-image-captioning 4We also report ROUGE scores in Appendix B.
ability, as in the work of Liu et al. (2019). CA is defined as:
CA = 1
m
m∑
j=1
1
n
n∑
i=1
1{yR[i, j] = yP [i, j]},
where n is the number of classes, m is the number of caption pairs (gold-generated) being compared, and yR, yP denote predictions from reference (gold) and predicted (generated) captions, respectively.

heading: 4.4 Baselines
text: Beam search decoding: The first baseline we compare DMMCS against is standard beam search.
Constrained beam search: The second baseline is a constrained beam search decoding method (Anderson et al., 2017), where decoding is guided by a set of manually defined constraints, aiming to enforce specific lexical requirements in the generated captions. We experimented with two versions of constrained beam search, namely strict and disjunctive. The former (denoted ∀) ensures that all specified keywords are present in the generated captions. The latter (denoted ∃) only enforces the inclusion of at least one of the given tags in the generated captions. For both versions, we used the HuggingFace implementation of constrained beam search,5 which is based on a plethora of guided decoding methods (Anderson et al., 2017; Post and Vilar, 2018; Hu et al., 2019; Li et al., 2021).

heading: 4.5 Experimental Results
text: We repeat each experiment for three random nonintersecting subsets of the test set, with each subset containing 1, 000 images. We report the average (over the the three test subsets) score for each evaluation measure and the standard deviation, offering insights into the stability of each model’s performance across different test subsets. To obtain the medical tags of each image, we use a medical image tagger trained on the training subset of each dataset. Specifically, we employ the top-performing encoder of the ImageCLEFmedical 2023 campaign (Ionescu et al., 2023; Rückert et al., 2023), namely a DenseNet-121 instance (Huang et al., 2017), initially pre-trained on ImageNet (Deng et al., 2009). We fine-tuned it (separately per DC dataset) on the training images and corresponding gold tags of the two DC datasets we
5 https://huggingface.co/blog/constrained-beam-search
use (Section 4.1). We also report scores using the gold (oracle) tags of the test images (Appendix B). Tuning α: For each model, dataset, and evaluation measure, we tuned the α factor of DMMCS (Eq. 6) by trying values from 0.05 to 0.95 and keeping the value with the best development score (measured with the particular evaluation measure). As an example, Fig. 4 shows the performance of InstructBLIP on ImageCLEFmedical 2023, in terms of BLEU (left) or BLEURT (right), when using DMMCS (continuous lines) or standard beam search decoding (dotted horizontal lines), as a function of α. For α values in [0.4, 0.8], DMMCS improves the model’s performance, while smaller and larger values deteriorate it, comparing to standard beam search. BLEU, BLEURT results: Table 1 reports the performance of each model on the two datasets, averaging over the three test subsets. For each evaluation measure (BLEU, BLEURT), four scores are provided: one for each baseline method (standard beam search, strict and disjunctive constrained beam search), as well as one for our proposed DMMCS method. The scores for DMMCS are obtained with the best α for each model, dataset, measure, using development data, as discussed above. Tags predicted by a medical image tagger are used. We observe that DMMCS always outperforms both standard (BS) and strict constrained (∀) beam search. Moreover, DMMCS is on par with the disjunctive constrained (∃) beam search method, outperforming it in most cases. We show the number of “wins” of each decoding method in the last row of Table 1. We also experimented using gold (oracle) tags instead of predicted tags; the results, discussed in Appendix B, show that DMMCS again improves performance, comparing to standard beam search. Clinical accuracy results: In Table 2, we present the best performing method across models and
datasets in terms of clinical accuracy (CA). When InstructBLIP is used as the backbone model, DMMCS is the best mechanism across datasets. For the rest of the models, however, there is no clear winner, although ConBS∀ is the best in Mimic (see Appendix I). We note that CA is based on silver labels (automatically generated), for a limited number of classes, and only on captions regarding chest (i.e., less than 20%). Consequently, the reliability of these results may be compromised to some extent. Future work should focus on the development of better medical image taggers that could improve automated clinical evaluation (e.g., making it applicable not only to chest images).
Fluency results: In the last two rows of Table 2 we present the results based on fluency (using Perplexity, see Appendix H) instead of CA. Our proposed method is the best in both InstructBLIP and ViT-GPT2, and the best across models for ImageCLEFmedical 2023.
Qualitative results: Additionally, we conducted a qualitative analysis of the diagnostic captions produced by the DC models with and without applying our proposed guided decoding method. We show a sample from this analysis in Figure 5. Without DMMCS, the beam search decoder generated a partially inaccurate caption, referring to a “parasternal long axis view” instead of an “echocardiogram”. While related, these terms are not precisely the same. Given the importance of precise medical terminology in diagnostic captions, addressing such inaccuracies is crucial. The DMMCSenhanced caption rectified the examination type to an “echocardiography parasternal long axis view” aligning more closely with the reference caption. Moreover, it correctly identified the diagnosed medical condition as “pericardial effusion” without introducing extra inaccurate information, unlike beam search decoding which incorrectly stated that the examination type showed a “left ventricular outflow tract”. However, it is crucial to note that the proposed method may not address all hallucinations, or more generally, inaccuracies of the model. Nevertheless, the draft reports generated by our method are more accurate according to all evaluation metrics, and in practice they would be checked and improved by medical experts.
Varying Sentence Order Analysis: The sentence order in medical reports varies (e.g., the same diagnosis may appear in different reports, using the same sentences yet reordered). In light of this
observation, we conducted a sentence-level analysis to investigate the potential influence of sentence order on the quality of the generated captions. We considered pairs of gold and generated captions consisting of the same number of sentences. We then created sentence pairs using sentences at the same position in the gold and generated captions. We evaluated each pair in terms of BLEU and BLEURT and then averaged the sentence-pair scores across the test set. If the order of the generated sentences is often wrong compared to the corresponding gold ones, we should notice a substantial difference relative to the original scores, which do not penalize generating reasonably correct sentences but in the wrong order. Our analysis, however, revealed that the scores of sentence pairs remained consistent with the original ones (within the reported standard deviation), ultimately indicating that this issue did not affect our experiments. Cases where performance declines: As shown in Tables 1 and 2, the proposed algorithm generally improves performance on both Natural Language Generation (NLG) metrics and clinical accuracy. However, there are instances where the performance declines when using the proposed algorithm. We investigated these cases by manually examining them, but no specific pattern explaining the decrease in performance could be identified. We plan to investigate this issue further in the future, possibly requiring assistance from medical professionals. Computational Overhead: It is important to measure the computational overhead associated with our proposed method. Implementing our method results in an additional time overhead, requiring approximately 25 to 27% more time to generate diagnostic captions compared to baseline methods such as standard beam search. In the case of InstructBLIP, this translates to an increase of around 13 to
15 minutes in processing time per 1, 000 captions in our experiments. Furthermore, there is an additional computational overhead in terms of memory usage, which increases by approximately 5% compared to baseline approaches (Section 4.4). While these overheads are important considerations, the improved performance, in terms of NLG metrics and clinical accuracy, of the generated captions justify the investment in time and computational resources. Per-Modality Results The ImageCLEFmedical 2023 dataset consists of four primary medical modalities; namely X-Ray, CT, MRI and Ultrasonography. We explored the individual permodality results of both the standard beam search and the proposed decoding method in order to identify any modality-specific features that might require attention. We observed that the proposed algorithm consistently outperforms the standard beam search across both metrics and nearly all modalities and models. No modality-specific peculiarities were noted, so we did not deem it necessary to implement any case-specific data handling procedures. Detailed per-modality evaluations are provided in Table 4 (Appendix G).

heading: 5 Discussion
text: Performance with gold tags We also experimented with the ground-truth (instead of the predicted) tags per image (see also Table 3 in the Appendix). As one would expect, DMMCS-enhanced models perform better with gold, compared to predicted tags. However, the performance of DMMCS-enhanced models with predicted tags (Table 1) is relatively close to the corresponding performance with ground-truth tags (Table 3), showing the robustness of DMMCS to noisy tags.
Guidance balancing We also experimented by balancing the two basic components of DMMCS (Eq. 6), the decoder’s scores and the proposed data-driven penalty. By introducing a dynamically computed weight called Histogram Divergence (HD), we adjusted the contribution of the two components, so that when DMMCS could not be trusted, more weight would be assigned to the decoder’s score (more details in Appendix E). We observe that this balancing results in steadier performance across α values, though it comes at the cost of lower overall performance compared to when HD is not considered. In other words, if tuning is not an option, dynamically balancing the two components of DMMCS could lead to the selection of a better optimal α.

heading: 6 Conclusion
text: In this work we introduced the DMMCS datadriven guided decoding method that enhances the performance of automatically generated diagnostic captions by integrating medical tags associated with a radiology image in the generation process. We assessed the effectiveness of the proposed method by applying it to four DC systems with diverse architectures and on two datasets, namely ImageCLEFmedical 2023 and a subset of MIMIC-CXR. Subsequently, we compared the generated captions with those obtained using a typical beam search decoding approach, employing widely used evaluation metrics, namely BLEU and BLEURT. Our results demonstrate that using the proposed DMMCS mechanism during decoding consistently outperforms the typical beam search approach across almost all models and datasets for most of the metrics.
In future work, we plan to experiment with more domains and focus on a broader range of tasks to investigate the benefits of our method in a wider context. Furthermore, we will explore our method’s capabilities in generic image captioning (Lin et al., 2015), as well as other text generation tasks. A final direction for future work concerns the use of contextual representations, in order to enhance the quality of the embeddings used when computing the penalty of DMMCS (Eq. 5).

heading: 7 Limitations
text: Although we used all the publicly available medical datasets we could obtain, the experimental results are limited to the specific conditions, regions, and
language (English) these datasets concern. However, this limitation could be addressed by collaborating with medical institutions, under the license of respective review boards, which is a direction we plan to consider in the future.

heading: 8 Ethical Considerations
text: Assisting clinicians by providing them more accurate draft diagnostic reports promotes ensuring good health and well-being, as well as reduced inequalities. However, emphasis on biomedical data privacy has long been a sensitive issue because of crossing ethical, legal, and technical boundaries, thus apprehension of clinical information privacy needs to be taken into serious consideration when patients’ data are used for model training.

heading: Acknowledgements
text: This work has been partially supported by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program.

heading: A Background on DC
text: Various DC approaches have been explored in previous work, ranging from earlier ontology and rule-based systems (Varges et al., 2012) to modern encoder-decoder architectures. Notably, the bestperforming techniques in generic image captioning (Bai and An, 2018) are not always the most efficient for DC tasks. Unlike generic image captioning models, DC systems face challenges in accurately describing medical images based solely on visual content. For instance, conventional retrieval methods, despite their simplicity, have shown promising results in DC tasks by leveraging captions from similar archived exams (Pavlopoulos et al., 2021). Even simple models like 1-NN, which retrieve and use the caption from the visually nearest image, can outperform more complex systems, such as Transformer-based or encoder-decoder architectures (Liu et al., 2019). However, despite the noteworthy performance of retrieval methods, recent advancements in the field of deep learning have established the encoder-decoder framework as the predominant approach in the DC domain. CNNs or Vision Transformers (Dosovitskiy et al., 2021) are commonly employed for image encoding, while RNNs or Transformer-based LMs are selected for the report generation process. Recent
advancements in diagnostic captioning have also incorporated attention mechanisms, visual attention, and reinforcement learning techniques into the encoder-decoder framework (Li et al., 2018).

heading: B DMMCS based on ground-truth tags
text: In this section, we report the performance of the four models on both datasets using the ground-truth tags for each image (Table 3), instead of predicted tags (Table 1). As shown in the main section of the paper, employing the proposed algorithm enhances model performance compared to standard beam search. Furthermore, employing ground-truth tags, instead of tags predicted by a medical image classifier, results in superior model performance. The extent of this performance difference depends on the accuracy of the classifier. Less accurate classifiers generate noisier tags, which subsequently affect the generated captions. Conversely, more accurate classifiers provide guidance for the model to include relevant concepts, as well as words closely aligned with the medical examination’s findings. In our experiments, we observed that the performance of the models utilizing predicted tags (Table 1) was relatively close to the ones employing ground-truth tags (Table 3). This indicates that despite the use of predicted tags, the model’s performance remained competitive, indicating the robustness of the proposed decoding algorithm.

heading: C ROUGE scores
text: In the main part of this paper, we primarily focused on a precision-based measure (i.e., BLEU (Papineni et al., 2002)) and a learned evaluation metric (i.e., BLEURT (Sellam et al., 2020)) that relies on precision-driven metrics during training. BLEU evaluates the precision of n-grams in the generated text compared to reference texts, while BLEURT is a learned metric that leverages transformer-based embeddings and is trained on human judgment data. It incorporates BLEU in order to enhance its evaluation capabilities. Nevertheless, recall is also important, as it measures the model’s ability to capture all relevant details from the reference captions. Unlike precision, which emphasizes the correctness of the generated information, recall ensures that the model doesn’t miss any pertinent details. However, clinicians face time constraints and cannot feasibly process overly long or potentially inaccurate information. Therefore, while recall is crucial for ensuring thoroughness, it must be balanced with
the practical considerations of clinical workflow. Our main focus remains on precision, ensuring that the information provided is concise, relevant, and accurate. Nevertheless, the model’s performance in a recall-based metric, specifically ROUGE (Lin, 2004), is also included in this section (Table 3). This approach allows us to provide a more comprehensive evaluation while maintaining a focus on the precision needed for practical clinical use.

heading: D MIMIC-CXR MCS distributions
text: In this part, we also provide a figure with multiple interquartile plots for the employed subset of the
MIMIC-CXR (Johnson et al., 2019) dataset, in correspondance with Figure 3. Each plot represents the MCS distribution of each one of the 14 tags. The coral lines denote the median value of each interquartile plot (MMCS). We observe that, similarly to the ImageCLEFmedical 2023 dataset (see Figure 3), the MMCS value of the distinct tags vary significantly, ranging from 0.3 to 1.0.

heading: E Histogram Divergence (HD)
text: As part of further exploration, we experimented with adding an additional term in our method’s guided decoding scoring function. Histogram Divergence (HD) is a dynamically computed parameter that serves as a weighting factor of the two components, DMMCSp and 1 − Dscore. In detail, at each decoding step, we consider a set of generated candidate sequences G and a histogram of the MCS distribution R(t, G) (Eq. 3) for each tag t is calculated. It represents the maximum lexical representation of the tag t in each candidate sequence. Therefore, for a given list of ground-truth tags T , |T | histograms are generated. In addition, a similar histogram has been pre-calculated for each tag on its associated ground-truth captions with respect to the training data (see Section 3). The rationale behind HD is that in candidate sequences with similar MCS distributions (one for each tag) to those computed on the training data the DMMCSp factor should be assigned a larger weight compared to 1 − Dscore during that decoding step. In contrast, if the distributions do not match, the conventional Dscore should be trusted more.
Given a single tag c, the divergence of its two corresponding histograms (computed on the generated and ground-truth captions respectively) is
calculated by performing a Kolmogorov-Smirnov Goodness of Fit Test (KS-test) (Massey, 1951), which yields two values: the p-value and the ksstatistic. The p-value, as in any other goodness of fit test, can be examined to either accept or reject a null hypothesis, but it does not precisely quantify the discrepancy between the two provided distributions. Unlike p-value, the ks-statistic measures the distance between two given distributions, while it is an appropriate metric in the specified context since it takes values in [0, 1]. A value of 0 suggests that the two samples are drawn from the same distribution, while a value of 1 indicates the opposite. The ks-statistic is calculated as the maximum absolute vertical distance between the Empirical Cumulative Distribution Functions (ECDF) of the two distributions. Formally, the ks-statistic for a single tag t at a random decoding step can be calculated as:
ks(t) = max |F (R(t, S))− F (R(t, G))|, (8)
where S denotes the training captions associated with t and G represents the generated captions up to this decoding step. Moreover, F (·) calculates the ECDF of the given input, while R is the MCS distribution as defined in Eq. 3. Overall, the HD for a set of tags T is calculated as:
HD = 1 |T | · ∑
t∈T ks(t). (9)
HD is integrated in Eq. 6 as an additional weighting factor:
DMMCS(s) = α · (1− HD) · DMMCSp(T, s) + (1− α) · HD · (1− Dscore)
(10)
As can be seen in Fig. 7, both blue (with HD) and green (without HD) lines surpass the red dashed line of the beam search baseline for various α values when using the (oracle) gold tags. The green line achieves a higher peak compared to the blue, while the blue line has lower standard deviation. Therefore, HD improves robustness across α values at the cost of (maximum) performance.
F Instructions
InstructBLIP (Dai et al., 2023) operates as a VisionLanguage instruction-tuning model, which can
adapt rapidly to new tasks based on specific instructions provided in its query prompt during training. Consequently, its performance depends largely on the quality of the instruction it is prompted with. While identifying the optimal instruction remains elusive, it is commonly agreed that concise and coherent instructions tend to yield better performance. However, due to the substantial memory demands of an InstructBLIP instance, conducting numerous experiments to find the optimal or nearoptimal instruction can be very impractical. As a result, we conducted experiments using three different instruction prompts, which are presented in Table 5.

heading: G Per-modality scores
text: In this section, we conducted a modality-specific evaluation in order to assess and highlight any notable differences in performance between different modalities, which is presented in Table 4. After our exploratory analysis, we observed that the ImageCLEFmedical 2023 dataset contains examinations originating from four main medical modalities, which are described in Thus, we split it into four subsets and evaluated the performance of the standard beam search and our proposed decoding method individually. In the rare case that an image belongs to more than one modality, we randomly select one of them to assign it. We observed that DMMCS outperforms beam search across almost all methods and modalities, remaining consistent with our findings on the entire dataset.

heading: H Perplexity
text: We also provide perplexity scores (Table 6) obtained using ClinicalT5 to measure the fluency of each model.6 ClinicalT5 is a biomedical version of T5 (Raffel et al., 2019), pre-trained on the MIMICIII dataset (Johnson et al., 2016). We observe that our proposed method generates more fluent captions than the baseline decoding methods, as it achieves lower perplexity scores in most cases.
6We were granted access to ClinicalT5 through PhysioNet: https://www.physionet.org/content/clinical-t5/1. 0.0/, Last accessed: 2024-06-05.

heading: I Clinical Accuracy - Extensive Results
text: Table 7 presents an extensive evaluation of the Clinical Accuracy measure across all models and decoding methods. We share more details on how we
calculate the CA between the gold and generated captions in Section 4.3.

heading: Future Work
text: However, this limitation could be addressed by collaborating with medical institutions, under the license of respective review boards, which is a direction we plan to consider in the future