Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation (2024)

Abstract: Fine-grained vision-language models (VLM) have been widely used for inter-modality local alignment between the predefined fixed patches and textual words. However, in medical analysis, lesions exhibit varying sizes and positions, and using fixed patches may cause incomplete representations of lesions. Moreover, these methods provide explainability by using heatmaps to show the general image areas potentially associated with texts rather than specific regions, making their explanations not explicit and specific enough. To address these issues, we propose a novel Adaptive patchword Matching (AdaMatch) model to correlate chest X-ray (CXR) image regions with words in medical reports and apply it to CXRreport generation to provide explainability for the generation process. AdaMatch exploits the fine-grained relation between adaptive patches and words to provide explanations of specific image regions with corresponding words. To capture the abnormal regions of varying sizes and positions, we introduce an Adaptive Patch extraction (AdaPatch) module to acquire adaptive patches for these regions adaptively. Aiming to provide explicit explainability for the CXR-report generation task, we propose an AdaMatch-based bidirectional LLM for Cyclic CXR-report generation (AdaMatch-Cyclic). It employs AdaMatch to obtain the keywords for CXR images and ‘keypatches’ for medical reports as hints to guide CXR-report generation. Extensive experiments on two publicly available CXR datasets validate the effectiveness of our method and its superior performance over existing methods.

heading: 1 Introduction
text: Inter-modality alignment, such as vision and language, has been an important task with growing interests in the field of computer vision, especially with the recent advancement in representation
*Xiang Li and Yixuan Yuan are corresponding authors.
(c) AdaMatch: Adaptive Patch-Word Matching for Medical Vision-Language Model
Adaptive Patches
Sa m
pl in
g
Align
(b) Vision-Language Model with Local Alignment
Align
Visual Features
Medical ReportTextual Features
Patch Features
Adaptive Patch Features Word Features
Align
Word Features
!! !"
!! !"
Medical Report
!! !"
Medical Report
(a) Vision-Language Model with Global Alignment
Medical Image
Medical Image
Medical Image
Figure 1: Current vision-language models (VLM) achieve (a) global alignment and (b) local alignment by matching overall visual with textual features, and aligning patches with word features, respectively. (c) To exploit the relation between textual words and abnormal patches with varied sizes, our AdaMatch obtains adaptive patch features and aligns them with word features.
learning (Radford et al., 2021). Technologies like contrastive learning and self-supervised learning have dramatically improved state-of-the-art alignment performance. Recent vision-language models (VLMs) demonstrate two approaches: global contrastive alignment, which integrates images and texts at a global level (Radford et al., 2021; Jia et al., 2021; Jang et al., 2023; Wang et al., 2023; Yang et al., 2022), and local alignment, focusing on detailed connections between visual objects and textual words (Chen et al., 2020a; Li et al., 2020b,a; Zhan et al., 2021; Kim et al., 2021; Yao et al., 2021), as illustrated in Fig. 1.
Current VLMs with local alignment either adopt the pre-trained object detector to extract regionof-interest (ROI) features from images and match the corresponding object features with textual words (Chen et al., 2020a; Li et al., 2020b,a; Zhan et al., 2021), or align the visual token from each
9494
patch and the textual token into the same embedding space (Kim et al., 2021; Yao et al., 2021; Ji et al., 2021; Wang et al., 2022a). The former highly relies on the quality of the object detector and its predefined classes, which is less generalizable to new domains. The latter family of methods learns the alignment in a more automatic and datadriven manner. However, most of these methods depend on a pre-defined patch size and positions (e.g., grids) across images. In the most challenging cases, such as the analysis of medical image, lesions can exhibit a wide range of shapes, sizes, and positions. A fixed partition of image patches can lead to incomplete or ambiguous representations of the key imaging abnormalities. Therefore, it is highly desirable to adaptively exploit the finegrained relationship between image embeddings derived from a more flexible patching scheme and textual embeddings.
Another challenge in the current VLMs lies in their explainability: it is generally difficult to delineate the image-text relationship learned by the model, especially for the current medical VLMs. Current solutions to provide such explanations in medical VLMs leverage the attention maps from the intermediate layer to visualize the location of the abnormalities (Moon et al., 2022; Huang et al., 2021; Yan and Pei, 2022). Other methods (Wan et al., 2023; Chen et al., 2023; Liu et al., 2023) utilize network gradients such as Grad-CAM (Selvaraju et al., 2017) to generate the heatmaps according to lesion types based on ground-truth reports. However, both maps can only show the general areas potentially associated with the corresponding text data rather than pinpointing a specific region. In addition, gradient-based methods need groundtruth reports, prohibiting them from functioning correctly beyond training data. It is thus highly necessary to develop a mechanism that could provide explicit and specific explanations of input image or text during inference time.
To address these two challenges above, we propose a novel Adaptive patch-word Matching (AdaMatch) model to match fine-grained image regions of various sizes and positions with textual data. AdaMatch introduces an image encoder with multiple Adaptive Patch extraction (AdaPatch) modules to adaptively acquire the patches associated with certain text tokens. It then performs patch-word alignment based on contrastive learning. AdaMatch is specifically developed in the context of aligning radiology images (chest X-ray,
CXR) and their corresponding radiology reports with the capability of achieving cyclic (CXR-toreport and report-to-CXR) generation based on the learned alignment. Our premise is that such a cyclic generation task would serve as the best use case and evaluation criterion for the desired fine-grained alignment. Also, fine-grained cyclic generation between CXR and report will provide natural explainability for how the model aligns two modalities: for any given text token, we can visualize its matching imaging manifestation; and for any image region within a CXR image, we can tell the type of lesion or anatomical region it belongs to.
To implement the cyclic CXR-report generation, we propose an AdaMatch-based bidirectional model (AdaMatch-Cyclic). AdaMatch-Cyclic employs AdaMatch to identify the keywords for CXR images and the ‘keypatches’ for medical reports to guide the generation tasks. Since the potential keywords for CXR images cover a wide range and ground-truth reports cannot be used during inference, we predefine a textual codebook with the most common entities from medical reports as prior knowledge during fine-grained alignment. With the textual codebook, AdaMatch aligns it with the adaptive patches to obtain matched keywords to facilitate report generation. Next, a VQ-GAN model encodes the CXR image into image tokens, and a Large Language Model (LLM) takes image tokens, the matched keywords, and the instructions as input to generate medical reports. Similarly, we also build a visual codebook with the most commonly seen patches as ‘keypatches’, and use AdaMatch to obtain the matched keypatches from given text reports as hints for CXR generation. Utilizing medical reports, matched keypatches, and instructions, LLM generates image tokens, subsequently decoded by the VQ-GAN model to produce the resulting CXR image. Our contributions are summarized as follows:
• To exploit the fine-grained relation between CXR image patches and words of medical reports, we propose an Adaptive patch-word Matching (AdaMatch) model to obtain adaptive patches for abnormal regions and perform alignment between them and texts in medical reports.
• We devise an AdaMatch-based bidirectional LLM for Cyclic CXR-report generation (AdaMatch-Cyclic) to facilitate the bidirectional generation between CXR and re-
ports. Moreover, we build the textual and visual codebook to utilize AdaMatch to extract useful keywords and keypatches for the report and CXR generation, respectively.
• Experiments on two publicly available chest X-ray datasets demonstrate the effectiveness of our method and its superior performance over the state-of-art methods.

heading: 2 Related Works

heading: 2.1 Fine-Grained Vision-Language Models
text: Recently, several fine-grained vision-language models (VLM) achieve the local alignment by exploiting the fine-grained relation between visual objects and textual words. Some methods (Chen et al., 2020a; Li et al., 2020b,a; Zhan et al., 2021) employ the pre-trained object detector to obtain object features from images and align them with textual features, and others (Kim et al., 2021; Yao et al., 2021; Wang et al., 2022a; Ji et al., 2021; Xu et al., 2023; Jiang et al., 2023) aim to align the fixed patches with the textual words locally. The former relies on a precise object detector, and the latter focuses on the relation between fixed patches inside the predefined grid and words. However, varying lung lesion characteristics may cause these methods to divide them into separate patches, resulting in incomplete semantic information. Thus, we devise an Adaptive patch-word Matching (AdaMatch)
model to adaptively exploit the fine-grained relation between flexible patches and textual words.

heading: 2.2 CXR-Report Generation
text: Medical VLMs are widely used in downstream tasks for chest radiographs, including CXR-toreport generation (Chen et al., 2020b, 2021a; Yang et al., 2021; Wang et al., 2022b; Voutharoja et al., 2023; Yang et al., 2023; Shi et al., 2023; Huang et al., 2023) and report-to-CXR generation (Rombach et al., 2022; Chambon et al., 2022b,a; Lee et al., 2023a,b; Han et al., 2024; Shentu and Al Moubayed, 2024; Hou et al., 2023; Hashmi et al., 2024; Chen et al., 2024) tasks. For CXR-to-report generation, Chen et al., 2021a introduces a crossmodal memory network with shared memory to align images with texts to promote report generation performance. In the report-to-CXR generation task, prior techniques create annotated CXR images from medical reports to augment training data and address privacy concerns, which are categorized into diffusion-based and transformer-based methods. In this paper, we use the cyclic generation (i.e. CXR-to-report and report-to-CXR generation) as use case and evaluation criteria for our finegrained alignment method (AdaMatch). We also design an AdaMatch-Cyclic to employ AdaMatch to improve the explainability of cyclic generation.

heading: 3 Methods
text: In Fig. 2, we propose an adaptive patch-word matching (AdaMatch) model to associate CXR image regions with words in medical reports, and apply it to CXR-report generation to enhance explainability. Given an input CXR image xI , the image encoder with several Adaptive Patch extraction modules (AdaPatch) predicts the location and scale of multiple adaptive patches and processes the adaptive patch embeddings zI . With zI and text embeddings zT extracted by text encoder for the medical report xT , we compute the similarities sI , sT between adaptive patches and text tokens to calculate the contrastive loss LI2T , LT2I to optimize AdaMatch. For CXR-to-report generation, we utilize the frozen AdaMatch model to match a predefined textual codebook with the input CXR image xI to obtain matched keywords kT and feed a LLM with kT , the instruction, and image tokens encoded by VQ-GAN from xI to generate a medical report xT ′ . Then, the AdaMatch model is used to match a predefined visual codebook with the generated report xT ′ to acquire the matched keypatches kI . With instruction, xT ′
and kI , LLM outputs image tokens of the generated CXR image xI ′ , which is optimized through Ltext and Limg.

heading: 3.1 Adaptive Patch-Word Matching (AdaMatch)
text: Current VLMs (Chen et al., 2020a; Yao et al., 2021) achieve fine-grained alignment between visual objects and textual words, but they may split lung lesions into separate fixed patches due to various sizes of lung lesions. Thus, we propose an Adaptive patch-word Matching (AdaMatch) to locate the important regions, extract adaptive patches for these regions, and align them with corresponding textual words, as shown in Fig. 2 (a). Image Encoder. To obtain the adaptive patches, the image encoder comprises four stages with feature maps of decreasing scales. Specifically, in the first stage, we first adopt the patch embedding module to split the CXR image xI ∈ RH×W×C into N patches with fixed size s × s and project them to patch embeddings z(i)(1 ≤ i ≤ N) through a fully connected (FC) layer g, z(i) = g([a(i,1); . . . ; a(i,s×s)]), where a(i,j) indicates the image features for the pixel located at the q(i,j). q(i,j) indicates the coordinates for the image pixel. [·] represents the concatenation operation among the features. Afterward, we pass the patch em-
beddings z(i) with the position embeddings epos into a transformer encoder R to obtain the outputs, z(i) = R(z(i) + epos). To locate the potential lung lesions, we devise an Adaptive Patch Extraction module (AdaPatch) for the rest of stages. In AdaPatch, patch embeddings z(i) are fed into an FC layer, followed by four separate FC layers f1, f2, f3, f4 to predict offsets (δ (i) x , δ (i) y ) and patch sizes (s(i)w , s (i) h ) for adaptive patches, with offsets indicating shifts to the center (c(i)x , c (i) y ) of fixed patches,
δ(i)x = Tanh(f1(z (i))), δ(i)y = Tanh(f2(z (i))), (1)
s(i)w = ReLU(Tanh(f3(z (i))), s (i) h = ReLU(Tanh(f4(z (i))). (2) With the offset and patch size, we compute the position of left-top (a(i)x , a (i) y ) and right-bottom corners (b (i) x , b (i) y ) for each adaptive patch,
a(i)x = c (i) x + δ (i) x −
s (i) w
2 , a(i)y = c (i) y + δ (i) y −
s (i) h
2 ,
(3)
b(i)x = c (i) x + δ (i) x +
s (i) w
2 , b(i)y = c (i) y + δ (i) y +
s (i) h
2 ,
(4) and uniformly sample m × m feature points inside the patches. Since the coordinates may be fractional, bilinear interpolation is used to obtain the sampled feature points { p̂(j)
} 1≤j≤m×m. The
embeddings of all sampled points are flattened and fed into an FC layer f5 to obtain patch embeddings, z(i) = f5([p̂(i,1); . . . ; p̂(i,m×m)]). Finally, we pass z(i) with position embeddings epos into a transformer encoder R. The final adaptive patch embeddings zI ∈ RN×d are the ensemble of z(i)(1 ≤ i ≤ N), zI = EI(xI) = { zI1 , · · · , zIN } , where EI , xI , and d denote the image encoder, the CXR image, and the dimension of zI , respectively. Text Encoder. We adopt a pre-trained text encoder ET to encode the medical report xT into text embeddings zT . To make zT with the same dimension as zI , we feed an FC layer with zT to reduce its dimension, zT = ET (xT ) = { zT1 , · · · , zTK } (zT ∈ RK×d), where K is the number of text tokens. Patch-Word Alignment. To exploit the relation between the adaptive patches and textual tokens, we perform fine-grained contrastive representation learning to achieve patch-word alignment. Concretely, for the i-th CXR image xIi and j-th medical report xTj , we first compute the similarities between all the adaptive patch embeddings zIn(1 ≤ n ≤ N) and all the text embeddings
zTk (1 ≤ k ≤ K), and use the largest similarity max(1≤k≤K) (zIn) ⊤ zTk as the patch-word maximum similarity for n-th adaptive patch embedding. Then, the patch-word maximum similarities for all the adaptive patch embeddings are averaged as the similarity sIi,j of the i-th CXR image to the j-
th medical report, sIi,j = 1 N
N∑
n=1
(zIn) ⊤zTmIn , where
mIn = argmax(1≤k≤K) (z I n) ⊤ zTk . Similarly, the similarity of the j-th medical report to the i-th
CXR image is defined as, sTi,j = 1 K
K∑
k=1
(zI mTk )⊤zTk ,
where mTk = argmax(1≤n≤N) (z I n) ⊤ zTk . We exclude the padded textual tokens when computing the similarity.
With the cross-modal similarities sIi,j and s T i,j for the i-th CXR image and j-th medical report, we compute the adaptive patch-word contrastive loss LIi for i-th CXR image x I i ,
LI2Ti (x I i , { xTj }b j=1 ) = −1 b log exp(sIi,i/τ)∑ j exp(s I i,j/τ) ,
(5) where b, τ , and { xIi , x T i } represent the batch size, temperature hyperparameter and the positive CXRreport pair, respectively. Similarly, the adaptive word-patch contrastive loss LTi for i-th medical report xTi is formulated as,
LT2Ii (x T i , { xIj }b j=1 ) = −1 b log exp(sTi,i/τ)∑ j exp(s T j,i/τ) .
(6) The final contrastive loss for a mini-batch is calculated by, L = 12 ∑b i=1 (L I2T i + L T2I i ). With L, AdaMatch learns to locate important patches with varied sizes and exploits the fine-grained relation.

heading: 3.2 AdaMatch-based LLM for Cyclic CXR-Report Generation (AdaMatch-Cyclic)
text: To provide explicit explainability for CXR-report generation task, we propose an AdaMatch-based bidirectional LLM for the cyclic CXR-report generation (AdaMatch-Cyclic) by locating the potential lesions in CXR images and visualizing the appearance of description in reports to guide the generation process, as depicted in Fig. 2 (b).

heading: 3.2.1 CXR-to-Report Generation
text: In CXR-to-report generation, we use AdaMatch to match a predefined textual codebook with the CXR
image, providing keywords to guide LLM. Building textual codebook. Specifically, we first use a pre-trained BioEN (Raza et al., 2022) to extract the related entities from medical reports in the training set, where the related entities are divided into four entity groups, i.e., biological structure, detailed description, disease disorder, and sign symptom. Next, we compute the frequency of each entity and pick top κ0 entities for each entity group as keywords in the textual codebook. Keywords Extraction. With the textual codebook, we employ the frozen AdaMatch model to match keywords from the textual codebook with the adaptive patches of CXR images, and obtain a patchword similarity matrix sP ∈ RM×N between the keyword tokens and adaptive patch embeddings, where M and N denote the number of keyword tokens and adaptive patches. To extract the most matched keywords for each adaptive patch, we rank the patch-word similarity along the dimension of keyword tokens, obtain the top κ1 patch-word similarities for each adaptive patch, and extract corresponding keywords kT . The matched keywords can explain potential lesions in each adaptive patch, to assist LLM in generating medical reports. Instruction tuning LLM. After keywords extraction, we use a frozen VQ-GAN (Esser et al., 2021) encoder E to encode the input CXR image xI to the quantized image latent vectors as image tokens E(xI). For CXR-to-report generation, we adopt the dolly-v2-3b (Conover et al., 2023) model as the pre-trained LLM, and convert CXR-to-report generation dataset in instruction-following format. An example of instruction data for CXR-to-report generation is depicted in Fig. 3, with the instruction, input (the image tokens of the input CXR image), keywords (extracted by AdaMatch), and response parts (the ground-truth medical report). During training, the LLM learns to generate the hidden response part in an autoregressive manner.
By adopting AdaMatch-Cyclic, we can locate potential lesions and interpret them with keywords to enhance explainability.

heading: 3.2.2 Report-to-CXR Generation
text: In report-to-CXR generation, AdaMatch matches a predefined visual codebook with generated reports xT ′ , providing keypatches as guidance for the LLM to synthesize CXR images, where keypatches are important image patches related to the reports. Building visual codebook. Concretely, we first construct a visual codebook with the most common adaptive patches as keypatches. To collect the most common adaptive patches for CXR images in the training set, AdaMatch matches the adaptive patches of CXR images with textual tokens of medical reports to obtain top κ2 CXR-report pairs with the highest report-to-CXR similarities sT . For each CXR-report pair, we compute the word-patch maximum similarity max(1≤n≤N) (zIn) ⊤ zTk for each textual token, rank the word-patch maximum similarities, and extract the adaptive patches for top κ3 similarities as keypatches in the visual codebook. Each keypatch includes its adaptive patch and the corresponding features. Keypatches Extraction. With the visual codebook, the frozen AdaMatch matches the features of keypatches in the visual codebook with textual tokens of the generated report to acquire the wordpatch similarity matrix sW ∈ R(κ2×κ3)×K , where K is the number of textual tokens. To obtain keypatches related to the generated report, we rank the word-patch similarity along the dimension of keypatches, obtain the top κ4 word-patch similarity for each textual token, and extract the features of corresponding keypatches kI . Instruction tuning LLM. After keypatches extraction, we use a frozen VQ-GAN encoder to convert the matched keypatches kI into image tokens E(kI), and feed the LLM with the instruction, generated report, and image tokens of keypatches in the instruction-following format, as shown in Fig. 4. Then, LLM predicts image tokens, decoded by the VQ-GAN into the generated CXR image xI ′ .
AdaMatch-Cyclic allows us to interpret generated reports with matched keypatches, thereby providing explainability to the generation procedure.

heading: 3.2.3 Overall Objective
text: To optimize AdaMatch-Cyclic, we use the standard language modeling objective for both CXR-toreport and report-to-CXR generation. For CXR-
to-report generation, LLM receives the instruction It, image tokens of input CXR image E(xI), and matched keywords kT , aiming to generate the ground-truth medical report of response part autoregressively. We compute the conditional probability of the k-th token, P (uk) = LLM(It, E(xI), kT ), where k is the token index after the response key (### Response:). The report generation loss for the response area is calculated by, Ltext = ∑n i=k − logP (ui|u1, u2, · · · , ui−1), where [u1, u2, · · · , ui−1] denotes the tokenized texts before the response part, and n represents the maximum length of output tokens. Similarly, for report-to-CXR generation, we feed the LLM with the instruction Iv, generated report xT ′ , and image tokens of keypatches E(kI), and obtain the conditional probability of k-th token wk, P (wk) = LLM(Iv, xT ′ , E(kI)). The CXR generation loss is defined as: Limg =∑n i=k − logP (wi|w1, w2, · · · , wi−1). With Ltext and Limg, LLM can implement bidirectional CXR and report generation according to instructions.

heading: 4 Experiments

heading: 4.1 Experiment Setting
text: Datasets. We experiment on two main publicly available chest X-ray datasets, i.e. MIMICCXR (Johnson et al., 2019) and OpenI (DemnerFushman et al., 2016) datasets. MIMIC-CXR dataset comprises 473,057 images and 206,563 reports from 63,478 patients. We use the official splits, i.e. 368,960 for training, 2,991 for validation, and 5,159 for testing. OpenI contains 3,684 report-image pairs with 2,912 for training and 772 for testing. Unlike prior methods, we utilize both finding and impression sections as medical reports. Implementation Details. In the AdaMatch-Cyclic model, we first train the AdaMatch model and then use the frozen AdaMatch to train LLM. We adopt VQ-GAN (Esser et al., 2021) models pre-trained on
the MIMIC-CXR and OpenI datasets, respectively, and the dolly-v2-3b (Conover et al., 2023) model as pre-trained LLM. The source code will be released. Please see appendix C for more details.
Evaluation Metrics. We assess CXR-to-report generation using BLEU (B), METEOR (M), and ROUGE-L (R-L) (Chen et al., 2020b), and reportto-CXR generation using FID (Heusel et al., 2017) and NIQE (Mittal et al., 2012). The retrieval performance between CXR and report is assessed through the exact report in the top K retrieved reports for a given CXR image (R@K, K={1, 5, 10}).

heading: 4.2 Comparison with State-of-the-Arts.

heading: 4.2.1 CXR-to-Report Generation
text: We compare AdaMatch-Cyclic with current CXRto-report generation methods on the MIMIC-CXR and the OpenI datasets, including R2Gen (Chen et al., 2020b), R2GenCMN (Chen et al., 2021a), Joint-TriNet (Yang et al., 2021), XProNet (Wang et al., 2022b), ITHN (Voutharoja et al., 2023), and M2KT (Yang et al., 2023). Since these methods are mainly trained on the finding section, we reimplement them on both the finding and impression sections. In Table 1, AdaMatch-Cyclic achieves the best performance of 0.3793 in BLEU-1 on the MIMIC-CXR dataset, with superior generalization
to OpenI dataset using the same model. Compared to GPT-4V, AdaMatch-Cyclic significantly outperforms in BLEU-1 by 0.1479, as listed in Table 2.
In Fig. 5, we compare AdaMatch-Cyclic’s performance with M2KT and R2GenCMN. AdaMatchCyclic accurately captures relevant keywords like ‘scarring’, and describes ‘osseous abnormality’ and ‘clips in the right upper quadrant of the abdomen’ in the first case, showcasing superior report quality over previous methods.

heading: 4.2.2 Report-to-CXR Generation
text: We quantitatively compare our method with textto-image generation method like Stable Diffusion (Rombach et al., 2022) and report-to-CXR generation methods, such as Adapting-Med (Chambon et al., 2022b), RoentGen (Chambon et al., 2022a), UniXGen (Lee et al., 2023a), and LLMCXR (Lee et al., 2023b). In Table 3, AdaMatchCyclic achieves the highest FID scores on both datasets, showing its superior effectiveness in generating CXR. To compare the high-level feature distribution of CXR images generated by different methods, we randomly select 1,000 cases from test set, and apply t-SNE visualization to the real and synthetic CXR images on the MIMIC-CXR dataset. In Fig. 6, while existing methods’ synthetic CXR images differ from real ones, AdaMatch-Cyclic’s almost overlap with real ones, indicating its superiority in report-to-CXR generation.
In Fig. 7, we display generated CXR images from the MIMIC-CXR and OpenI datasets. In the
first example, AdaMatch-Cyclic accurately synthesizes ‘left and right pleural effusions’, while other methods fail to generate such features. This suggests the superior ability of our method to produce realistic CXR images based on input reports.

heading: 4.2.3 CXR-Report Retrieval
text: To demonstrate the superiority of AdaMatch, we evaluate the CXR-to-report and report-to-CXR retrieval performance on the MIMIC-CXR dataset compared to existing methods. We utilize the AdaMatch to compute the similarities between CXR images and reports, rank the similarities, and obtain the retrieval results. As shown in Table 4, AdaMatch significantly outperforms LIMITR with R@1 of 11.77% and 13.48% for CXR-to-report and report-to-CXR retrieval. This indicates the superior ability of our method to extract distinct semantic features and align them accurately between CXR images and medical reports.
In Table 5, to ablate AdaPatch, we remove the layers to predict offset and scale. We then aligned grid features from the last stage of the pyramid vision Transformer with textual tokens to compute the image-text similarity for retrieval tasks.

heading: 4.3 Ablation Study
text: Effectiveness of AdaPatch. We evaluate the effectiveness of AdaPatch by comparing the CXRreport retrieval performance of AdaMatch with and without AdaPatch. To ablate AdaPatch, we remove the layers to predict offset and scale, and compute the similarity between grid features of the last stage and textual tokens for retrieval tasks.
Table 6: Effectiveness of report-to-CXR and CXR-to-report generation tasks.
Report-to-CXR CXR-to-Report B-1↑ B-2↑ B-3↑ B-4↑ M↑ R-L↑ FID↓ NIQE↓ ✓ 0.3542 0.1973 0.1190 0.0758 0.1329 0.2392 - -
✓ - - - - - - 1.7128 4.0391 ✓ ✓ 0.3793 0.2346 0.1540 0.1060 0.1625 0.2813 1.0916 3.3931
In Table 5, AdaPatch significantly improves R@1 by approximately 3% for both retrieval directions, highlighting its effectiveness. To demonstrate the optimality of the AdaPatch design, we compare AdaPatch with other adaptive vision models, such as A-ViT (Yin et al., 2022) and AdaViT (Meng et al., 2022), in the CXR-report retrieval task. As shown in Table 7, AdaPatch exhibits superior retrieval performance compared with other methods. Effectiveness of Cyclic Generation. To validate the necessity of report-to-CXR and CXR-to-report generation tasks, we remove each task and evaluate their performance, as listed in Table 6. When ablating report-to-CXR generation task, the performance of CXR-to-report generation decreases substantially compared to our method, and vice versa. These suggest that the CXR-to-report and report-to-CXR generation can indeed benefit each other, and their interaction is crucial for the overall performance of the model. Image Encoder with Different Stages. In AdaMatch, the image encoder comprises several stages, with AdaPatch modules in the stages. To assess the effectiveness of the stage number, we compare the retrieval performance of AdaMatch with two, three, or four stages. Table 8 shows that the three-stage image encoder achieves the highest R@1 of 51.47%, indicating that the features from
the 3rd stage are most beneficial for CXR-report retrieval. Visualization of Adaptive Patches. We visualize adaptive patches in AdaPatch of some examples from the MIMIC-CXR dataset in Fig. 8. Each example includes text, its tokens, and the corresponding CXR image with marked adaptive patches. The patches are represented by colored bounding boxes with arrows indicating the center shift from the fixed patch. In the first example, adaptive patches cover the pacemaker and wire, and the second one can find pleural effusions, implying the capacity of AdaPatch to localize regions relevant to input text tokens accurately.

heading: 5 Conclusion
text: We propose AdaMatch for fine-grained image-text alignment, which presents the first work to adaptively associate image patches with words and improve the explainability of cyclic CXR-report generation. It includes AdaPatch to acquire adaptive patches for abnormal regions and performs patchword alignment between adaptive patches with textual tokens. We implement cyclic CXR-report generation by using AdaMatch to provide explanations for the generation process. In addition, the finegrained cyclic generation process provides a natural explainability for the alignment between CXR images and reports. Extensive experiments on two CXR datasets show the effectiveness of our method and its superiority over previous methods.
Limitation
While our research has made significant strides in utilizing chest X-ray datasets, it is important to acknowledge certain limitations. Our experiments predominantly focus on chest X-ray datasets due to their availability of large-scale images paired with high-quality medical reports. However, these datasets primarily consist of patients from the ICU, potentially skewing our model towards severe disease domains. In our future endeavors, we intend to expand the scope of our methodology by applying the proposed AdaMatch-Cyclic to multi-domain scenarios, thereby mitigating this limitation and enhancing the versatility of our approach.
Acknowledge
This work was supported by the Hong Kong Research Grants Council (RGC) General Research Fund under Grant 14220622, Innovation and Technology Commission Innovation and Technology Fund ITS/229/22, and the National Natural Science Foundation of China under Grant 82261138629.

heading: A Ablation Studies on AdaMatch-Cyclic
text: In the proposed AdaMatch-Cyclic, we leverage the AdaMatch to obtain keywords for CXR-to-report generation and keypatches for the report-to-CXR generation. To analyze the effectiveness of keywords and keypatches for the generation process, we ablate them in the AdaMatch-Cyclic on the MIMIC-CXR dataset to evaluate the CXR-report generation performance.
Effectiveness of keywords. As listed in Table 9, we ablate both keywords and keypatches of AdaMatch-Cyclic to create a baseline model. When we employ the keywords in the baseline model, the CXR-to-report generation performance improves significantly by about 0.03 in BLEU-4, indicating the effectiveness of the keywords obtained from AdaMatch. To further analyze the influence of the number of keywords Nw, we train the proposed AdaMatch-Cyclic with different Nw = {10, 15, 20}. In Table 10, AdaMatch-Cyclic achieves the best report generation performance with the ROUGE-L of 0.2859, when the Nw is set to 10.
Effectiveness of keypatches. In Table 9, when we further apply keypatches to the baseline model with keywords, the CXR generation performance boosts remarkably by about 0.5 in FID score, implying the effectiveness of keypatches provided by AdaMatch. To investigate the influence of the number of keypatches Np, we analyze the CXR
generation performance when the number of keypatches ranges from 5 to 15. As shown in Table 11, AdaMatch-Cyclic achieves the best CXR generation performance with the FID score of 1.0916, when the number of keypatches is 5.
B Visual Results
To prove the effectiveness of AdaMatch-Cyclic for CXR-to-report and report-to-CXR generation tasks, we demonstrate some visual results for both tasks. Moreover, to further visualize the explanation provided by AdaMatch, i.e. the adaptive patches and the corresponding texts. CXR-to-Report Generation. As shown in Fig. 9, we compare CXR-to-report generation performance with existing methods on the MIMIC-CXR and OpenI datasets. In the first row, the proposed AdaMatch-Cyclic can capture the ‘Feeding tube tip’ and ‘mild bibasilar atelectasis’, while M2KT (Yang et al., 2023) cannot observe such device and lung abnormality. The second case of the OpenI dataset shows that ‘cardiac silhouette is mildly enlarged’ and ‘degenerative changes of the thoracic spine’ can be discovered by our method. These imply that our AdaMatch-Cyclic can generate a more comprehensive and complete medical report with the guidance of generated keywords in comparison to current methods. Report-to-CXR Generation. Fig. 10 visualizes the real and synthetic CXR images of the MIMICCXR and OpenI datasets in comparison with existing methods. As depicted in the first example, the CXR image generated by AdaMatch-Cyclic shows ‘patchy opacity in the right lung’, while the CXR images generated by other methods do not include this pattern. In the second example, our AdaMatch-Cyclic can generate the CXR image
pulmonary edema 'pulmonary', 'ed', '##ema'
left sided picc: 'left', 'sided', 'p', '##ic', '##c'
enlarging right pleural effusion: 'en', '##lar', '##ging', 'right', 'p', '##le',
'##ural', 'e', '##ff', '##usion' heart size is mildly enlarged: 'heart', 'size', 'is', 'mildly', 'enlarged'
right pleural effusion: 'right', 'p', '##le', '##ural', 'e', '##ff', '##usion' bilateral basilar opacities: 'bilateral', 'b', '##asi', '##lar', 'op', '##ac', '##ities'
pacemaker: 'pace', '##maker'
left pleural effusion 'left', 'p', '##le', '##ural', 'e', '##ff', '##usion'
with ‘low lung volumes’. These indicate the superiority of our AdaMatch-Cyclic over existing methods in report-to-CXR generation. Adaptive Patches and Texts. In Fig. 11, we visualize adaptive patches in AdaPatch, the textual words, and the textual tokens for different cases from the MIMIC-CXR datasets. We highlight the adaptive patches with bounding boxes in different colors. Each bounding box has an arrow inside to show the shift of center from the fixed patch to the adaptive patch. In the first example, adaptive patches cover the pulmonary edema. Meanwhile, adaptive patches of the second example show the correct position of PICC device in the left lung of the CXR image. These suggest that AdaMatch-Cyclic can show the correspondence between the adaptive patches and textual words to provide the correct explanation for the CXR-report generation.
C Implementation Details
In the AdaMatch-Cyclic model, we first train the AdaMatch model and then use the frozen AdaMatch to train LLM. The AdaMatch model consists of an image encoder and a text encoder. We utilize the DPT-medium (Chen et al., 2021b) as the image encoder that includes AdaPatch module
in stage 2 and 3. The image encoder is pre-trained on the MIMIC-CXR dataset with the disease classification task. We adopt the pre-trained BioClinicalBERT (Alsentzer et al., 2019) as text encoder. The image and text encoders are followed by two convolutional layers with batch normalization and the ReLU activation function, respectively, to reduce the feature dimension to 256. The patch size s is set to 4 for stage 1, and 2 for stage 2, 3, and 4. The number of sampled feature points m is 3. We optimize the AdaMatch using LAMB optimizer (β1 = 0.9, β2 = 0.999, ϵ = 10−4). The cosine learning rate scheduler with the base learning rate (lr_b) of 6×10−3 is adopted to linearly warm up to the peak learning rate (i.e. lr_p = lr_b× √ bs_all 512 ) during the first quarter of the total training epochs, where bs_all denotes the effective total batch size. The total training epoch is 15 and the per GPU batch size is 112. In AdaMatch-Cyclic, we use the pre-trained VQ-GAN (Esser et al., 2021) model to encode CXR images into image tokens and decode image tokens into CXR images. We adopt the dolly-v2-3b (Conover et al., 2023) model as the pre-trained LLM. The LLM has 5,1845 token types with the first 5,0821 token types for text tokens and the rest 1,024 token types for image tokens. We add 5,0821 to each image token value encoded by VQ-GAN. To decode the image tokens into images,
we subtract 5,0821 from the image tokens generated by LLM and feed image tokens to VQ-GAN decoder to obtain the generated CXR images. We train the LLM with the AdamW (Loshchilov and Hutter, 2018) optimizer. The learning rate is initialized as 5 × 10−6 and the total training epoch is 5. The per GPU batch size is set to 24. The hyper-parameters κ0, κ1, κ2, κ3, κ4 are set as 200, 10, 1000, 20, and 5, respectively. In CXR-to-report generation, we extract keywords for each adaptive patch and use the keywords with the top 10 patch-word similarities as hints of LLM. In addition, we use 5 keypatches to guide the LLM in report-to-CXR generation. All the experiments are conducted on 8 Nvidia A100 40GB GPUs. There is no overlap of patients among different subsets. We discard the medical reports with fewer than 3 tokens from both datasets. All the CXR images with different sizes are resized to 256× 256 pixels.

heading: Future Work
text: In our future endeavors, we intend to expand the scope of our methodology by applying the proposed AdaMatch-Cyclic to multi-domain scenarios, thereby mitigating this limitation and enhancing the versatility of our approach. Acknowledge
This work was supported by the Hong Kong Research Grants Council (RGC) General Research Fund under Grant 14220622, Innovation and Technology Commission Innovation and Technology Fund ITS/229/22, and the National Natural Science Foundation of China under Grant 82261138629