Report-Concept Textual-Prompt Learning for Enhancing X-ray Diagnosis

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 16:03:22

Abstract: Despite significant advances in image-text medical visual language modeling, the high cost of fine-grained annotation of images to align radiology reports has led current approaches to focus primarily on semantic alignment between the image and the full report, neglecting the critical diagnostic information contained in the text. This is insufficient in medical scenarios demanding high explainability. To address this problem, in this paper, we introduce radiology reports as images in prompt learning. Specifically, we extract key clinical concepts, lesion locations, and positive labels from easily accessible radiology reports and combine them with an external medical knowledge base to form fine-grained self-supervised signals. Moreover, we propose a novel Report-Concept Textual-Prompt Learning (RC-TPL), which aligns radiology reports at multiple levels. In the inference phase, report-level and concept-level prompts provide rich global and local semantic understanding for X-ray images. Extensive experiments on X-ray image datasets demonstrate the superior performance of our approach with respect to various baselines, especially in the presence of scarce imaging data. Our study not only significantly improves the accuracy of data-constrained medical X-ray diagnosis, but also demonstrates how the integration of domain-specific conceptual knowledge can enhance the explainability of medical image analysis. The implementation code will be publicly available.

Main paper: 
 introduction: 
 Deep learning models have demonstrated remarkable potential in the analysis of chest X-rays (CXR), such as disease classification [9] . However, the training of these models relies on a large number of annotated images, which limits their application in practical settings [21] . Pre-trained language models, such as BERT [17] , have provided new perspectives for medical vision and language pretraining [24, 37] . Recently, large-scale visual-language pre-training … Atelectasis suspected in the left upper lobe, minimal pleural effusion in the left lower lobe, and a mass shadow observed in the right middle lobe… (VLP) models that utilize naturally paired image-text contrastive learning, such as CLIP [24] and BLIP [18] , have made significant progress in visual tasks. Nevertheless, transforming foundational vision-language pre-training models into actual clinical applications remains challenging.First, the limited availability of medical imaging data presents a challenge for language models to comprehend free-form reports [3] . Unlike natural images that can be annotated with relative ease, medical image annotation requires verification by experienced domain experts, such as clinical physicians, resulting in high thresholds and costs. This leads to a scarcity of annotated medical images, while an abundance of unpaired radiology reports remain underutilized. Although MedCLIP [33] learns from these unpaired texts, there is still a necessity for image involvement. The majority of current medical VLP models merely pair CXR images directly with raw radiology reports, as shown in Figure 1 (a), neglecting the modeling of fine-grained relationships between the semantic content in images and the concepts in reports. Despite efforts to denoise radiology reports and integrate multi-source medical knowledge into pre-training [5, 20, 34] , overcoming the limitations of coarsegrained knowledge and reliance on image data remains elusive. Thirdly, explainability is critical. Linking diagnostic outcomes with visual evidence is crucial for aiding radiologists' comprehension of the system and building trust between patients and technology.To address the above challenges, we advocate learning from prompts by treating radiology reports as medical images. This bypasses the need for time-consuming and laborious image annotation. It is feasible as in pre-trained visual-linguistic models [14] where an image encoder and a text encoder encode images and text into a shared space. For a given image and its radiology report, the visual features produced by the image encoder will be close to the text features of the radiology report produced by the text encoder. This has also been demonstrated on generalized visual multi-classification tasks [8] . Second, when target task data are constrained, prompt learning as a representative parametric learning paradigm has emerged and served as an effective way to adapt VLP models to downstream tasks. For example, CoOp [40] learns from annotated images. DualCoOp [27] learns from annotated images by training positive and negative cue pairs with partially labeled images.Radiology reports usually contain information from two perspectives, anatomy and pathology, as in Figure 1 (b). Pathologic information is the clinical concept of the chest X-ray examination, e.g., Mass. Anatomical structure describes the anatomical structure and location, e.g., right middle lobe. A recent work, MedKLIP [34] , provides a promising approach for decomposing textual information into anatomical and pathological dimensions, but their approach does not have explainable analysis. In this paper, we extract key anatomical and pathological information, including clinical concepts, lesion locations, and positive labels, from easily accessible radiology reports and combine them with an external medical knowledge base to form fine-grained self-supervised signals. Furthermore, we propose a novel Report-Concept Textual-Prompt Learning (RC-TPL), which aligns radiology reports at multiple levels. In the inference phase, report-level and concept-level prompts provide rich global and local semantic understanding of X-ray images. Note that although these prompts are learned only from textual descriptions, they can be easily used to classify entire images as well as image blocks during testing (see Figure 1 (b) ).To sum up, the contributions of this work include.• We propose to use radiology reports as medical images in prompt learning to adapt medical VLP models to X-ray image diagnosis. Radiology reports are easily accessible and their class labels can be directly exported compared to images. Compared to prompting from images, our method is less affected by the problems of image data limitations and labeling restrictions.  
related work 2.1 medical vision-language models: 
 Large-scale pre-trained visual language models (VLMs) have been widely applied across various general domains [18, 19] . For instance, the CLIP [24] was trained on a dataset of 400 million image-text pairs collected from the internet, resulting in an effective alignment of visual and textual encoders. However, due to the complexity of medical reports and the scarcity of large-scale medical image-text datasets, pre-trained visual language models in the medical domain are still being explored. To utilize the medical image-text data, the contrastive learning framework has also been extensively adopted [11, 32] . Many studies utilize precisely matched medical image and text data for contrastive learning, such as ConVIRT [36] . In contrast, MedCLIP [33] introduced a novel approach by decoupling the strict pairing relationship between images and texts, which not only expanded the range of available training data but also reduced the risk of false negatives. Additionally, some strategies incorporate medical knowledge as part of the model input or as guidance during the model training process [4] , such as MedKLIP [34] , KAD [35] , Med-UniC [30] , DeV-iDe [20] . Although these methods have facilitated the integration of knowledge in the training of VLMs, there is still a lack of effective means to integrate the fine-grained discriminative knowledge required in the diagnostic process.Prompt learning developed from the field of natural language processing which is motivated by the use of pre-trained language models as a knowledge base from which valuable information can be elicited for downstream tasks using prompt templates [38] . Recently, prompt learning has become a parameter-efficient way to transfer pre-trained knowledge to downstream tasks in data-limited environments [6, 15] . CoOp [40] learns prompts using a few annotated images per category in the target dataset. CoCoOp [39] further improves upon CoOp by formulating prompts in an imageconditional manner. CoCoOp [39] achieves impressive zero-shot inference performance for previously unseen categories under lowresource conditions. In the medical domain, Qin et al. [23] developed a method for automatically generating medical prompts to enhance the knowledge transfer capability of pre-trained visual language models for medical object detection. MedPrompt [38] enables lowcost medical image classification through prompt learning. Most related to our work is XCoOp [2] , which leverages medical knowledge by aligning the semantics of images and learning fine-grained knowledge through trainable prompts.Despite the significant performance achieved by existing prompt learning methods on downstream tasks, they typically require images and some class labels as inputs to learn prompts, making them less applicable in the medical context where image annotation is costly. Other research has developed methods that avoid the need for images in the prompt learning process [8, 41] . Guo et al. proposed TaI [8] , which treats text as image learning prompts. Inspired by TaI, we utilize readily available radiology report descriptions to learn prompts. After training, the prompts learned in text can be easily applied to test images. . . .:Element-wise Sum :Concatenate  
methodology: 
 In this section, we detail our proposed RC-TPL, including concept extraction and knowledge injection, definition of report-level and concept-level prompts, training and inference. As shown in Figure 2 .3.1.1 Concept Extractor. To obtain sufficient clinical concept information for the identification of X-ray images, we must ensure that the collected reports contain detailed descriptions of diseases. The content of the reports needs to cover all clinical concepts as comprehensively as possible to represent all categories of diseases. To guarantee reproducibility, we use existing radiology reports from public X-ray image datasets (e.g., MIMIC-CXR [16] ). It is important to note that although each radiology report is paired with the corresponding image and human-annotated labels, we only utilize the radiology reports themselves and do not use information from images and labels during the training process.For a target X-ray dataset D that has a clinical concept set C = {𝑐 1 , 𝑐 2 , 𝑐 3 , . . . , 𝑐 𝐿 }, Where 𝐿 denotes the number of concepts and 𝑐 𝑖 denotes particular concept name like "effusion", "mass", etc. . Given a radiology report T with multiple sentences, we follow [34] employ the named entity recognition method Rad-Graph [13] to extract (Concept, Location, Positive) triplets from the radiology reports, as shown in the top of figture 2. Here, "Concept" refers to clinical pathological observation emerge from CXR image (e.g., "Mass"). "Location" indicates the specific anatomical body part related to the clinical observation, such as "left lower lobe." Additionally, there is a "Positive" label to indicate whether the clinical observation reported is positive (present), negative (not present), or uncertain result, which will be used as a supervisory signal for subsequent concept-level semantic alignment. Based on this, we can obtain a set of triplets to describe the fine-grained information from the radiology reports:T concepts = (c 𝑖 , l 𝑖 , pos 𝑖 ), 𝑖 ∈ N (1)where N represents the total number of concepts contained in one radiology report. We only search for radiology reports that contain at least one concept name in C.To achieve fine-grained semantic alignment, we enhance the comprehensibility of the extracted triplets by connecting them with external medical knowledge. Specifically, we query publicly accessible medical knowledge bases (such as Wikipedia1 ) or utilize large language models (such as ChatGPT After extensive large-scale image-text contrastive pre-training, textual features have achieved complete consistency with image features of the same semantics [8] . Therefore, radiology text features used to describe specific diagnostic categories also exhibit classifiable recognizability. Based on this alignment of visual-language (VL) representations, we propose a novel perspective that assume the necessity of images for prompt learning, we suggest using textual features describing specific diagnostic categories as substitutes for image features. Furthermore, medical diagnoses often rely on various clinical concepts (symptoms) observable within specific local regions of images. Considering that different clinical concepts may correspond to different subregions of medical images, we propose Report-Concept Textual-Prompt Learning (i.e., RC-TPL), which uses two sets of prompts to process report-level (i.e., coarsest-grained level) and concept-level (i.e., finer-grained level) features in two parallel branches, respectively. Specific details are shown in Figure 2 .Report-level Prompt. Following [8] , a report-level is defined as:EQUATION)Where 𝑖 is the concept index, 𝑐 𝑖 denotes word embedding of the 𝑖-th concept name. For 𝑗 ∈ {1, 2, . . . , 𝑀 }, 𝒑 𝑗 is a learnable word embedding whose dimension is the same as the dimension of normal word embeddings in the vocabulary. Then the 𝒕 𝑅 𝑖 is fed to a copy of the text encoder Enc 𝑇 of MedCLIP to generate report-level prompt embeddings for each concept.EQUATION3.2.2 Concept-level Prompt. In general, global features are sufficient for common image classification because the target object is usually prominent in the picture. However, in X-ray images, there exists information at the anatomical and pathological levels. Therefore, it motivates us to explore fine-grained features. We propose concept-level prompt definitions as follows:EQUATIONWhere 𝒑 ′ 𝑗 is a learnable word embedding that concatenated with word embedding 𝑐 𝑖 of the 𝑖-th concept to obtain the concept prompt. Similarity, the 𝒕 𝐶 𝑖 is fed to a copy of the text encoder Enc 𝑇 of MedCLIP to generate concept-level prompt embeddings for each concept.EQUATION3.2.3 Report and concept description embedding. For radiology reports, we similarly utilize the text encoder in MedCLIP to encode and then take the embedding of the [𝐶𝐿𝑆] token. For each concept, a series of sentence descriptions obtained after external medical knowledge injection, we encode them separately. Report text embedding and concepts description embedding define as:EQUATIONwhere 𝒓 denotes a complete radiology report. 𝒉 ∈ R 𝐷 are the extracted global text features. 𝑯 ∈ R 𝐿×𝐷 are the extracted concept description features.Just like in previous methods [40] , the prompts are learned by maximizing the probability of classifying each concept description into its ground-truth class. Then, the report-level and concept-level similarities are computed by:EQUATIONwhere 𝒖 denotes either language feature 𝒉 in training or visual feature 𝒇 in testing of image 𝒙, and 𝑼 denotes 𝑯 or the flattened dense image features 𝑭 coordinately. Information in local branch 𝑷 is aggregated in a spatially weighted manner:EQUATIONwhere 𝜏 𝑐 accommodates the extent of focusing on a specific location.Following [8] , We adopt the ranking loss [7] to measure the discrepancy between classification scores and ground-truth labels extracted from report for each concept. Specifically, L 𝑟𝑒𝑝𝑜𝑟𝑡 and L 𝑙𝑜𝑐𝑎𝑙 are formulated as follows:EQUATIONwhere 𝒑 and 𝒑 ′ are report-levle and aggregated concept-level similarities described in above. 𝑚 is the margin controlling how much higher the similarity score with the positive classes is than with the negative classes. During training, we minimize the overall objective L = L 𝑟𝑒𝑝𝑜𝑟𝑡 + L 𝑙𝑜𝑐𝑎𝑙 with frozen text encoders, by optimizing the global and local prompts.At inference time, given a test image, we can directly infer the existence of certain concepts/disease, and ground their visual evidence.Given an X-ray image scan X ∈ R 𝐻 ×𝑊 ×3 , we can compute the visual features 𝒇 and flattened dense image features 𝑭 with a visual backbone {𝑓 , 𝐹 } = Enc 𝐼 (X) (10) where 𝒇 ∈ R 𝐷 and 𝑭 ∈ R 𝐿 𝐼 × 𝐷, and 𝐿 𝐼 = 𝐻 × 𝑊 denotes the flattened spatial dimension of visual feature. Then, the report 𝒑 and concept 𝒑 ′ similarities are computed and combined to obtain the final classification score. COVIDx CXR-3 and COVID Rural. [22, 28] aim to evaluate on diagnosing COVID-19. COVIDx CXR-2 contains 29,986 X-rays with the COVID-19 classification labels. We used it as a classification dataset. In addition, we use the COVID Rural dataset for COVID-19 segmentation, which contains more than 200 chest radiographs with segmentation masks. RSNA Pneumonia. [26] is a chest X-ray image dataset publicly provided by the NIH, which consists more than 260k chest X-rays with corresponding pneumonia opacity masks. It can be used as both a classification and grounding task. SIIM Pneumothorax. is a more than 12k frontal chest radiograph dataset with pneumothorax mask collected by the Society for Imaging Informatics in Medicine and the American College of Radiology. Similar to the RSNA pneumonia dataset, it can be used as both a classification and grounding task. 
experiment 4.1 experimental setup: 
 4.1.3 Baselines. We evaluate our method against a range of stateof-the-art (SOTA) methods for medical VLP, including ConVIRT (2022) [36] , GLORIA (2021) [11] , MedCLIP-ViT (2022) [33] , BioViL (2022) [3] , CheXzero (2022) [29] MedKLIP (2023) [34] , MedPrompt-ViT (2024) [38] , MeDSLIP (2024) [5] .ConVIRT trains two modality-specific encoders by bidirectional contrastive loss to learn visual representations. GLoRIA utilizes both global and fine-grained features for medical VLP. MedCLIP-ViT decouple images and texts for multimodal contrastive learning, and we choose ViT as the image encoder. BioViL proposes a radiology-specific text encoder for the subsequent classical pipeline of VLP. CheXzero retrains one CLIP model with a corpus of the medical domain. MedKLIP designs one novel entity extraction and transition module to inject domain-specific knowledge into the process of VLP. MedPrompt-ViT uses a weakly supervised prompt learning method to automatically generate medical prompts. MeDSLIP establishes vision-language fine-grained alignments via disentangling visual and textual representations. Pointing Game and Dice are used for evaluating the grounding performance. In specific, we extract the region with max response in the output heat-map, for one instance, if the region hit the groundtruth mask, it is considered a positive prediction, otherwise negative. Finally, accuracy can be calculated as the pointing game score. We report the maximal Dice score for each model. Precision and Recall refer to the detection Precision and Recall. For medical, in some hard cases, especially for the zero-shot setting, Dice may be too strict to reflect the performance difference. Precision and recall scores can compensate for these. Precision@𝐾 is used for measure the performance of various models in Image-Text retrieval task.4.1.5 Implementation. We adopt MedCLIP [33] with ResNet-50 [10] as the image encoder, and use the MedCLIP with BioClinical-BERT [1] as the text encoder. During training, the parameters of the two encoders are kept frozen, and only learnable prompts are optimized. Our learnable prompts are shared between classes across all datasets. Following [8] , we used Gaussian noise sampled from N (0.02) to initialize the value of each prompt parameter. The length of both the report prompts and the conceptual prompts are set to 𝑀 = 12, while longer sequences bring negligible improvement.We pre-train on the MIMIC-CXR dataset involves a batch size of 64, an AdamW optimizer with learning rates 2𝑒 -5, and a cosine scheduler. And for all datasets, the number of training epochs is set to 20 on 4×V100 GPUs.We conduct zero-shot image classification evaluation on four datasets: ChestX-ray14, CheXpert, COVID, and RSNA. Through matching the encoded image embedding and the embeddings of learned report-level and concept-level prompts for each disease class to achieve zero-shot prediction. We illustrate the results in Table 1 . We analyze and discuss separately the categories that were seen in the pre-training and the categories of chest diseases that were not seen.Seen Diseases. The pre-training dataset MIMIC-CXR includes 14 disease labels. The specific categories are consistent with the disease categories included in CheXpert, as listed in Figure 3 (right). On both the CheXpert and RSNA pneumonia datasets, their diseases were seen in the pre-training text. Despite the fact that the images were collected by different hospitals at the time, our model improves the AUC from 0.83 to 0.85 on CheXpert, and from 0.86 to 0.89 on the RSNA dataset. For the ChestX-ray14 dataset, which has half of the disease labels not included in MIMIC-CXR, our model improves the AUC from 0.78 to 0.83. This suggests that our approach can better handle multi-center, multi-disease data distribution in medical.Unseen Diseases. We are considering a strict set for open-set classification, in particular, we utilize Covid-19 to evaluate the model. Covid is a new disease that only appeared in 2019, and the MIMIC-CXR reports collected in 2015 do not include any Covid entities, so it requires the model to have the ability to diagnose truly unseen diseases. As shown in Table 1 , it is difficult to make a correct diagnosis with existing methods that rely only on the name of the disease. In contrast, methods that introduce medical Table 1 : Comparison with other state-of-the-art methods on zero-shot classification task. AUC and ACC scores are reported. For ChestX-ray14 and CheXpert, the metrics all refer to the macro average on the all diseases. * represents partial overlap of disease categories seen in the pre-training data, and * * represents disease categories that did not seen in the pre-training data.ChestX-ray14 knowledge including MedKLIP, MeDSLIP and our method can substantially improve the model performance. Our method improves the introduction of fine-grained concept descriptions that can be generalized to new chest diseases, since images of Covid-19 also show similar pathological features such as "Lung opacity". Compared to state-of-the-art methods that do not incorporate medical knowledge, our method significantly improves the performance of AUC from 0.68 to 0.77 and ACC from 0.67 to 0.76, suggesting that the introduction of fine-grained concept medical knowledge is crucial for unseen diseases classification. This is also confirmed on unseen diseases in the ChestX-ray14 dataset, as shown in the disease classification performance highlighted in red font in Figure 3 (left).4.2.2 Zero-shot Region Grounding. Beyond plain diagnosis, explainability is equally crucial in healthcare to improve the reliability and trustworthiness of machine learning systems. Here, we consider providing explainability by grounding anomalies in predictions and compare it with existing methods to demonstrate the effectiveness of our proposed concept prompt learning for explainability. Similarly, we split the diseases into seen and unseen ones, depending on whether their names have peared in the medical reports. Specifically, "Pneumonia" and "Pneumothorax" are treated as seen, and "Covid-19" is treated as unseen.Seen Diseases. Tabel 2 and Table 3 shows the results on RSNA Pneumonia dataset and SIIM-ACR Pneumothorax dataset respectively. Follow [34] , we only consider pointing game, recall, and precision, as the pneumothorax region tends to be thin and narrow, and grounding is more challenging. Our approach leads to 2 , our proposed model outperforms existing methods in all metrics, such as increasing the fixed-point game score from 0.88 to 0.90. Overrall, our method can achieve better performance than previous methods.Unseen Diseases. As shown in Table 4 , we also conducted a zero-shot grounding experiment on the unseen disease Covid-19. Our model shows consistent improvements across all metrics, e.g., increasing the pointing game score from 0.54 to 0.62. Our model with knowledge-enhanced concept-level prompt embedding facilitates the iamge encoder to learn potential evidence related to the disease, and therefore produces more explainable representations compared to previous methods. To better validate the effectiveness of our proposed method in text senmantic alignment. We followed the experimental setup in MedCLIP [33] and evaluated the semantic richness of the representations learned by the model through image-text retrieval tasks in CheXpert testing split of five disease categories, including Atelectasis, Cardioomegaly, Edema, Pleural, and Effusion. Due to the lack of publicly available report data for CheXpert, we used the MIMIC-CXR dataset to generate reports/sentences. We sampled 200 sentences for each of the five classes present in the CheXpert-5x200 dataset. This generated 1000 images and 1000 sentences as the retrieval dataset. The results are shown in Table 5 . Our method achieved better performance, which proves that it effectively provides the semantic information required for retrieving text. 1 . The result in Figure 4 (left) shows that the model achieves the best performance when 𝑀 = 12. When the prompt length is too long, both the training efficiency and the computation budget will be increased. Further, we performed ablation experiments on Report-level and Concept-level at 𝑀=12. From Figure 4 (left), we can conclude that our recept-level can achieve better results by injecting external medical knowledge. 4.3.2 Knowledge injection compare. We further explored different knowledge injection approaches in the preparation of the concept description phase. Specifically, we used five concept knowledge injection settings, including without knowledge (W/O KG), general knowledge (i.e., a fixed format for knowledge without a specific clinical concept, e.g., a photo of [concept name].) , medical knowledge generated based on Large Language Models (LLMs) (LLMs KG), medical knowledge without anatomical location based on clinical knowledge base (KB KG W/O position), and clinical knowledge base knowledge with anatomical location (KB KG). Similarly, the previous three knowledge injection settings splice anatomical location information. Specifically, we take the unseen Covid-19 dataset as an example. Figures 4(middle) and 4 (right) show that without external medical knowledge or general knowledge leads to degraded model performance. Secondly, without incorporating the disease concept in the anatomical location information of the image brings a degradation to the model performance , especially in the grounding task. Finally, we observed that clinical knowledge generated with LLMs is also comparable to specialized knowledge bases. The comprehension capabilities of LLMs can be utilized for more research extensions in the future.To verify that different prompts do reflect different image attributes, we visualize the image contents corresponding to different prompts using Grad-CAM [25] .Specifically, given a test image, a feature map 𝑭 is obtained by image encoder. Concept-levle prompt and each disease category 𝑖 are concatenated and then passed through the text encoder to obtain concept prompt embedding 𝑬 𝑅 𝑖 . 𝑭 and𝑬 𝑅 𝑖 then used to calculate similarity 𝑷 , which is adopted to highlight the corresponding image contents using Grad-CAM.In Figure 5 , the image contents in different columns correspond to the activation regions of different disease categories. As can be seen in Figure 5 , for the same X-ray image, concept prompts with different disease category do reflect different regions in the image, indicating the validity and diversity of the learned prompts.To verify whether the learned concept prompts indeed reflect image attributes with high-level semantics, we visualize the content of the same disease category (e.g., mass) on different images in in Figure 6 . It can be seen that our method can focus on different anatomical locations of the same disease category on different images. This suggests that the concept prompts effectively learns key attributes that can be generalized across images, thereby improving the performance of X-ray diagnostics.  
conclusion: 
 In this paper, we propose a new perspective on prompt learning that treats radiology reports as X-ray images, which learns prompts from the discriminative features of concept descriptions.. Compared to previous prompt learning methods trained with images, our method benefits from the easy accessibility of scalable content-rich radiology reports, which enables prompt learning for visual diagnostic tasks even without downstream X-ray data. Report-level and concept-level prompts are introduced to leverage global and finegrained features for better X-ray diagnosis ability. Our approach outperforms other prompt learning methods while retaining the inherent explainability of visual and textual interpretation. Extensive experiments and explainability analyses on variety of datasets show that our method achieves both excellent performance and explainability. 
 
 Appendix: Wikipedia https://en.wikipedia.org/wiki/ChatGPT https://chat.openai.com/ 


Decision: Overall, all reviewers are satisfied with the response given by the authors, and are glad to see that the quality of the paper has been improved substantially.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel method to use radiology reports as medical images for cue learning, aiming to solve the problem of high cost of fine-grained annotation and improve the interpretability of medical image analysis. By extracting key clinical concepts, lesion locations, and positive labels from radiology reports and combining them with external medical knowledge bases to form fine-grained self-supervised signals, this paper proposes a Report-Concept Text Prompt Learning (RC-TPL) method. Experiments on multiple X-ray image datasets show that RC-TPL performs well under data-scarce conditions, significantly improves diagnostic accuracy, and demonstrates how the integration of domain-specific conceptual knowledge enhances the interpretability of medical image analysis.
strength_weakness: strength_weakness: Innovation: This paper proposes an innovative method of treating radiology reports as images for prompt learning, bypassing the high-cost image annotation problem.

Interpretability: Through multi-granularity prompt learning, the model not only improves diagnostic accuracy, but also enhances the interpretability of diagnostic results, which is particularly important in the medical field.

Detailed experiments: This article conducted extensive experiments on multiple data sets and verified the effectiveness of the method through a variety of evaluation indicators. The results are convincing.

Comparative analysis: A variety of state-of-the-art medical visual language pre-training methods are compared, and the superiority of this method is demonstrated in detail.
limitations: limitations: Insufficient explanation of external knowledge bases: Although this article mentions incorporating external medical knowledge bases, the discussion of knowledge base selection, quality, and coverage is not detailed enough. Furthermore, the insufficient presentation of visualization results from different knowledge bases makes it difficult to assess the specific role of various knowledge bases in improving model performance and interpretability. 

Versatility: Although the method has been validated on multiple X-ray data sets, it is not yet clear whether it is applicable to other medical images (such as MRI, CT).
paper_summary: paper_summary: To avoid relying on high-cost fine-grained annotations while focusing on critical diagnostic information in reports, this paper introduces radiology reports as images into rapid learning. Key clinical concepts, lesion locations, and positive labels are extracted from easily accessible radiology reports and combined with external medical knowledge bases to form fine-grained self-supervised signals. A novel Report-Concept Textual-Prompt Learning (RC-TPL) method is proposed, aligning radiology reports at multiple levels and achieving state-of-the-art performance across various downstream tasks.
strength_weakness: strength_weakness: 1. This paper identifies the challenges of high-granularity alignment between radiology images and reports, emphasizing that existing methods have not considered the critical diagnostic information within reports, an issue that has not been previously addressed.

2. The proposed method introduces radiology reports as prompts in learning, demonstrating both innovation and rationale. By extracting key information from reports and integrating it with external knowledge bases to create fine-grained annotations, the approach effectively enhances cross-modal alignment performance.

3. Comprehensive experiments, including comparisons with state-of-the-art methods and ablation studies, provide evidence for the effectiveness of the proposed method.
limitations: limitations: 1. The implementation details of the Clinical Knowledge Injector should be provided.
2. Ablation studies with different backbones should be included.
3. Figure 4 should be represented more effectively.
paper_summary: paper_summary: The authors propose to replace X-ray images with medical reports to conduct prompt learning, which aims to address the problem of data limitation. To achieve this, a knowledge injection method and a Report-Concept Textual-Prompt Learning approach are introduced. Experimental results verify the effectiveness of the proposed model for improving X-ray Diagnosis.
strength_weakness: strength_weakness: This paper is well-written, with clear descriptions that make it easy for readers to understand the ideas, methods, and results. The authors have conducted necessary experiments to demonstrate the effectiveness.
limitations: limitations: 1. The paper lacks innovation; the method of extracting fine-grained medical knowledge from reports is quite common. Additionally, the approach of using reports instead of images for training is very similar to [1]. The report-prompt and concept-prompt in this paper are almost identical to the global-prompt and local-prompt in [1], and even the figures are drawn in a similar style.

[1] Guo Z, Dong B, Ji Z, et al. Texts as images in prompt tuning for multi-label image recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 2808-2817.

2. One major concern: Unlike CLIP for natural image-text pairs, MedCLIP, despite improving the correlation between medical images and reports, could be limited by the limited scale of the medical data. So, it seems not as effective as CLIP to align the embeddings of the two modalities in the same semantic space (maybe for multi-modal medical data, the learned spaces are close, but still have a huge gap). Therefore, directly replacing X-ray images with reports might not be a reasonable approach. It is recommended to qualitatively or quantitatively verify the similarity between image embeddings and report embeddings to support your idea.

3. The definition of module name in Figure 2 and the Method section is inconsistent. For example, the Concept Extractor in 3.1.1 and the Clinical Knowledge Injector in 3.1.2 do not have corresponding names in the figure. Additionally, there are numerous typos, such as the double “.” in line 908 and the repeated sentences in lines 29-31.
paper_summary: paper_summary: This paper proposes a novel Report-Concept Textual-Prompt Learning (RC-TPL) framework for enhancing X-ray diagnosis by leveraging radiology reports instead of labeled images. The key idea is to extract clinical concepts, lesion locations, and positive labels from radiology reports and combine them with an external medical knowledge base to form fine-grained self-supervised signals. The RC-TPL model aligns radiology reports at both report-level and concept-level, providing rich global and local semantic understanding for X-ray images during inference.
strength_weakness: strength_weakness: 1. The proposed method addresses the challenge of limited availability of annotated medical imaging data by utilizing readily accessible radiology reports for prompt learning.
2. The integration of domain-specific conceptual knowledge from radiology reports and external medical knowledge bases enhances the explainability of medical image analysis.
3. The RC-TPL framework aligns radiology reports at multiple granularity levels (report-level and concept-level), enabling both global and local semantic understanding of X-ray images.
4. The experimental results demonstrate superior performance compared to various baselines, especially in data-constrained scenarios.
limitations: limitations: 1. The extraction of clinical concepts relies on the Rad-Graph method, which may have limitations in accurately identifying all relevant concepts. How does the quality of concept extraction affect the overall performance of RC-TPL?
2. The integration of external medical knowledge is described briefly. More details could be provided on how the knowledge is queried and incorporated into the concept descriptions. Is there a systematic way to ensure the quality and relevance of the injected knowledge?
3. In Equation 6, the concept description embedding $H$ is defined as $H = {h'{i}^{[CLS]}}^N{i=1}$. However, it is unclear how the dimension of $H$ becomes $\mathbb{R}^{L\times D}$ as mentioned later. Please clarify the dimensionality of $H$.
4. The aggregation of local branch information in Equation 8 uses a softmax-like operation. Is there any ablation study to justify the choice of this aggregation method compared to other alternatives?
5. The paper mentions combining the scores from report-level and concept-level branches for the final classification. However, the specific combination method is not provided. Is it a simple averaging, weighted sum, or some other approach?
6. The experimental setup and datasets used for evaluation could be described in more detail to enhance reproducibility.