Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding
Authors: Zhongyi Shui
Jianpeng Zhang
Weiwei Cao
Sinuo Wang
Ruizhe Guo
Le Lu
Lin Yang
Xianghua Ye
Tingbo Liang
Qi Zhang
Ling Zhang
Venue: ICLR 2025 Spotlight (2025)
Keywords: Vision-language model
fine-grained alignment
large-scale pre-training
CT image
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Artificial intelligence (AI) shows great potential in assisting radiologists to improve the efficiency and accuracy of medical image interpretation and diagnosis. However, a versatile AI model requires large-scale data and comprehensive annotations, which are often impractical in medical settings. Recent studies leverage radiology reports as a naturally high-quality supervision for medical images, using contrastive language-image pre-training (CLIP) to develop language-informed models for radiological image interpretation. Nonetheless, these approaches typically contrast entire images with reports, neglecting the local associations between imaging regions and report sentences, which may undermine model performance and interoperability. In this paper, we propose a fine-grained vision-language model (fVLM) for anatomy-level CT image interpretation. Specifically, we explicitly match anatomical regions of CT images with corresponding descriptions in radiology reports and perform contrastive pre-training for each anatomy individually. Fine-grained alignment, however, faces considerable false-negative challenges, mainly from the abundance of anatomy-level healthy samples and similarly diseased abnormalities, leading to ambiguous patient-level pairings. To tackle this issue, we propose identifying false negatives of both normal and abnormal samples and calibrating contrastive learning from patient-level to disease-aware pairing. We curated the largest CT dataset to date, comprising imaging and report data from 69,086 patients, and conducted a comprehensive evaluation of 54 major and important disease (including several most deadly cancers) diagnosis tasks across 15 main anatomies. Experimental results demonstrate the substantial potential of fVLM in versatile medical image interpretation. In the zero-shot classification task, we achieved an average AUC of 81.3% on 54 diagnosis tasks, surpassing CLIP and supervised methods by 12.9% and 8.0%, respectively. Additionally, on the publicly available CT-RATE and Rad-ChestCT benchmarks, our fVLM outperformed the current state-of-the-art methods with absolute AUC gains of 7.4% and 4.8%, respectively.

Published as a conference paper at ICLR 2025
LARGE-SCALE AND FINE-GRAINED VISION- LANGUAGE PRE-TRAINING FOR ENHANCED CT IMAGE UNDERSTANDING
Zhongyi Shui Jianpeng Zhang 1.355 Weiwei Cao Sinuo Wang!
Ruizhe Guo 3
Le Lu Lin Yang"
Xianghua Ye
Tingbo Liang?
Qi Zhang?
Ling Zhang'
DAMO Academy, Alibaba Group
The First Affiliated Hospital of College of Medicine, Zhejiang University, China ZZejiann University, China
*Westlake University, China "Hupan Lab, 310023, China jianpeng. zhang @maill com
ABSTRACT
Artificial intelligence (AI) shows great potential in assisting radiologists to im- prove the efficiency and accuracy of medical image interpretation and diagnosis. However, a versatile Al model requires large-scale data and comprehensive anno- tations, which often impractical in medical settings. Recent studies leverage radiology reports as a naturally high-quality supervision for medical images, us- ing contrastive language-image pre-training (CLIP) develop language-informed models for radiological image interpretation. Nonetheless, these approaches typ- ically contrast entire images with reports, neglecting the local associations be- tween imaging regions and report sentences, which may undermine model per- formance and interoperability. In this paper, we propose a fine- fine-grained vision- language model (fVL.) for anatomy-level CT image interpretation. Specifi- cally, we explicitly match anatomical regions of CT images with corresponding descriptions in radiology reports and perform contrastive pre-training for each anatomy individually. Fine-grained alignment, however, faces considerable false- negative challenges, mainly from the abundance of anatomy- healthy ples and similarly diseased abnormalities, leading to ambiguous patient-level pair- ings. To tackle this issue, propose identifying false negatives of both normal and abnormal samples and calibrating contrastive learning from patient-level to disease-aware pairing. We curated the largest CT dataset to date, comprising imaging and report data from 69,086 patients, and conducted comprehensive evaluation of 54 major and important disease (including several most deadly can- cers) diagnosis tasks across 15 main anatomies. Experimental results demon- strate the substantial potential of fVLM in versatile medical image interpreta- tion. In the zero-shot classification task, we achieved an average AUC of 81.3% on 54 diagnosis tasks, surpassing CLIP and supervised methods by 12.9% and 8.0%. respectively. Additionally, ChestCT benchmarks, our IVLM outperformed the current state-of-the-art art meth- ods with absolute AUC gains of 7.4% and 4.8%, respectively. Code is available htaps:///3ttubb.com a bbaa dame a academy/ fvim
the publicly available CT-RATE and Rad-
INTRODUCTION
Medical image interpretation is critically important yet exceptionally burdensome task in clinical workflows, particularly when dealing with 3D imaging scans [dare al. (2022). Radiologists are required to examine hundreds of slices across dozens of anatomies meticulously Blankemeier et al. (2024). As a result, there a growing demand for versatile and reliable AI to assist the auto- mated interpretation of medical images for wide range of diagnostic needs. Supervised learning is a prominent strategy for automating this process, demonstrating remarkable success natural scene
"Correspondence Jianpeng Zhang. The work done during Zhongyi's internship DAMO Academy

---

Published as a conference paper at ICLR 2025
OurMEM
nommal marphshgy skgn fullesss the boly and tul regions The perpaorente (lanet curnuncinng g body indeteett
Improvtan: Adde porcreestb with rennna per pencoeati
61 CT image
(b) Dlegnostic report
[c] Veual activation map of CUP
(6) Visuall octivation map of our NUM
(4)
Figure 1: Comparative analysis of vanilla VLM (CLIP) and fine-grained VLM (IVLM). (a.b) A representative CT slice and its corresponding radiological report. (c.d) Visual activation maps generated by CLIP and (VLM respectively, illustrating regions of interest for pancreatitis diagnosis. (e) Quantitative comparison of AUC scores across 54 disease diagnosis tasks in 15 anatomies.
images, such as ImageNet Deng et al. (009)). In the medical domain, specific disease category infor- mation must be precisely defined in advance, necessitating extensive annotations from specialized annotators Isensee et al. (2021); Wang et al. (2023); Zhang et al.] (2023a); Guo et al. (024). Unlike natural images, medical images encompass a complex variety of conditions, making challeng- ing to fulfill all clinical diagnostic requirements through predefined one-hot label space Liu et al. (202c)). Furthermore, the labor-intensive annotation constitutes an additional burden on doctors outside of their regular duties. These challenges make it particularly difficult to apply supervised learning methodologies effectively within the medical field.
Recently, Vision Language Models (VLMs) Zhang et al. (022) Tiu al.
Lin et al. (023))
Wt al. (023) Blankemeier et al. (024) have gained considerable attention, presenting a promis- ing alternative supervised learning paradigms. The fundamental concept involves supervising model training directly through diagnostic reports, thus eliminating the need for specific disease cat- egory labels Cao et al. (2024). Radiology reports are highly condensed recordings of the diagnostic process, meticulously documenting the evaluations conducted by at least one experienced radiolo- gist. During this evaluation, they can reference patient history and clinical information, resulting in text-based annotation. Current VLMs predominantly employ global contrastive learning, wherein embeddings of entire images and reports from the same patient are brought closer together, while those from different patients are pushed apart Bai al. (222)); Hamamci et al. (2024). However, this global contrast inherently coarse-grainn gaanse-gaaind, overlooking local similarities or disparities between anatomical regions and report sentences. Pulling certain anatomical regions closer to unrelated text or vice versa may result in mislcading alignment, making challenging to align complex medical images and reports within a unified representation space. As illustrated in Fig. [ (c), the attention map of the CLIP Radford et al. (202T). a vanilla coarse-grained VLM, is visualized when execut- ing certain diagnostic tasks. The result demonstrates that such global alignment mechanism can readily induce the model to focus on the regions that are not relevant to the diagnosis, potentially compromising its performance and interpretability.
In this paper, we propose a fine-grained vasion-language model (fVLM) for automated CT image interpretation. This model moves beyond the traditional global image-text contrastive learning pipeline, enabling anatomy-level fine- ine-ggrined alignment between CT scans and reports. Our moti- vation arises from the fact that diagnostic reports typically document clinically significant abnormal findings in various organs or body structures in the CT images per anatomy level, thus establishing an intrinsic fine-grained vision-language correspondence between any text-described finding and its image location. Specifically, we perform anatomical-level decomposition and matching for both the images and reports, followed by fine- -raigrrinddd aligmment of the matched visual embeddings and the corresponding report embeddings of the same anatomy. This explicit matching alleviates the misalignment issues associated with global contrastive learning and enhances the interpretability of VLMs, as illustrated in Fig. I (d). Moreover, fine-grained alignment encounters significant chal-
2

---

Published as a conference paper at ICLR 2025
lenges related to false negatives, primarily arising from the prevalence of anatomy-leve healthy samples and similar abnormalities across different diseases, which could result in ambiguous pair- ings at the patient level. We introduce a simple yet effective method to identify and manage the mas- sive false negatives from both normal and abnormal samples, and advocate for a shift in contrastive learning from broad patient-level pairing a more nuanced disease-aware pairing approach.
Due to privacy concerns and the scarcity of quality medical data, the limited availability of vision- language data has been one of the most significant bottlenecks for medical VLMs. To overcome this limitation, we have curated the largest CT dataset to date, named MedVL-CT69K, which includes 272,124 CT scans from 69,086 unique patients and their corresponding diagnostic reports. On this extensive dataset, our fVVMM has demonstrated outstanding zero-shot diagnostic capabilities. achiev- ing an average AUC of 81.3% across 54 disease diagnosis tasks, surpassing the competing CLIP model by 12.9% (see Fig. D (e)) and the supervised baseline by 8.0%. Moreover, on the publicly available CT-RATE and Rad- Ead-ChestCTI datasets, our fVLM outperforms the state-of-the- th-rth-rrt approach by 7.4% and 4.8% absolute AUC value gains, respectively. Beyond diagnostic tasks, the model also exhibits remarkable proficiency in downstream report-generation tasks. Our key contributions are summarized as follows:
1. We propose scalable and annotation- vision-language model, IVVM,, for CT image in- terpretation, which demonstrates strong scaling capabilities to meet a wide range of clinical diagnostic needs.
2. We address the vision-language misalignment issues of VLMMs by employing fine-grained anatomy-level contrastive learning framework.
3. We introduce dual false negative reduction module to alleviate the adverse effects of false negatives in both normal and abnormal samples.
4. Extensive experiments on a large-scale in-house dataset as well as two public benchmarks demonstrate the advantages of IVLM over the state-of-the-an counterparts.
RELATED WORK
2.1 MEDICAL VISION-LANGUACE PRE-TRAINING
Existing medical vision- language pre-training (Med-VLP) methods primarily focus on 2D images depicting a single body part, notably chest X-rays (CXR). Most of them learn transferable represen- tations by aligning the medical scans and corresponding reports with contrastive loss Zhang et al.
2022) Boecking et al. (2022): Tiu et al.
(2022) Huang et al. (2023): Zhou et al.] (2023): Zhang (2023b); Bannur et al. (2023) g Cheng et al. (2023): ) Liu
al. 2023bb; Lin et al. (202)) Liu et al.
et al. 2023ar Lin et al. (003); Sun et al. (2024); Lu et al. (2024); Christensen et al. (2024). In
ZhDnghan particular, MedKLIP Wu et al. (2023) and KAD Zhang et al. a (202cc) utilize medical domain knowl-
edge enhance the textual information extraction, thereby improving the contextual understanding of radiology reports. Imitate Liu et al. 20236) derives multi-level visual features from CXR im- ages and separately aligns these features with descriptive and conclusive text in hierarchical medical reports. Given the paucity of paired image-text data in the medical domain, several studies have investigated data-efficient Med-VLP. Notably, MedCLIP Wang al. (222b) and PTUnifier Chen et al. (2023) use unpaired CXR images and reports for multimodal pre-training Pairaug Xie et al. (2024) designs a pairwise augmentation approach that scales up the training data by manipulat- ing existing image-report pairs or generating entirely new cases. Beyond inspecting single body part, recent studies have expanded the scope of VLP to encompass broader anatomical structures
within high-detail 3D CT images Cao et al. (2024); Hamamci et al. (2024) Bai et al. (2024); Lin et al. 2024|; Blankemeier et al. (024). enabling more comprehensive diagnostic support in clinical practice. Specifically, BIUD Cao al. 2024) and CT-CLIP Hamamci et al. (2024) align chest CT volumes and radiology reports. Merlin Blankemeier et al. (2024) focuses on abdomen scenarios and incorporates structured electronic health record (EHR) data as additional supervision.
While existing Med-VLP studies have demonstrated decent performance, they predominantly em- play a global alignment scheme that contrasts entire images and reports Zhang et al. (2022); Tiu et al. (202)), overlooking the local similarities or disparities between image patches and report pieces. This oversight result in misalignment problem Müller et al. (2022), constraining the
3

---

Published as a conference paper at ICLR 2025
Original reports
Anatomy-leval decomposed reports
Findings. The liver demonstrates normal postion, size.
(Fiedings) The lisst demorstrates normal position, sies, marphology Diffuse decreased denstty is observed in lver porenchyma. A small, roune, non enhanding low densily lcsion with well-defined margins visible in the right lobe of the liver. No diation of intrahopatic or extrohepatic bile ducts is noted. (Impresioon) Fatty live: Small in the right labe of the live.
marpholagy:
decressed duraity
the parenchyma. smat, round, non-enhancing low-
darsity lasion right labe of the liver. No dilation intrahepatic or
defined margins viäble
extrahepat' bile ducts noted. The kidneys are normal In
povtion and marpholagy; the right kodney shows punctate
Segmentation
high density shadows, with no enhancing small cystic
lasiars in both kidneys. The porereas normal sice and
with enhancing small cystic lnitars both kidneys (bmpresaion) Small calcuit the right denney small both kidneys.
(Aindings) Punctate high-denstty shadows the right sidney,
denelty. with abnormal enhancement fod. There
dilation
duct. slightly hipo
observed on preater curveture
gastric wall, indistinct margins the stomach wall messuring approcimately 14mm in diametec.
(Rlddings) Normal sae and denstry with no abnormal er hanntmment foci. No dilation of the pancreatic duct.
Qwen
Impression. Fatty fiver. Small cys in the right loba of the fiver. There are small calculi the right Aldney and small epes both kidnnys. Nodule the graater cunvature side the stomoch; consideration of gastrointastinal
(Findings) siinnly tapo dense natule is observed on the the curveture side the well with indiviinet margins approsmaaelyy 14mm diameter. side of
(Impression) Nadule consideration of pastrointestinal stromal suggested.
tumor suggested.
Figure 2: Illustration of CT anatomy parsing (left) and diagnostic report decomposition (right).
model to a coarse- granseegreied understanding and limiting its capacity to capture fine-grained, clinically relevant details.
2.2 FINE-GRAINED ALIGNMENT IN MED-VLP
To address the misalignment challenge, GLoRIA Huang et al. 202T). LoVT Müller et al. (2022) and MGCA Wang et al. (2022a) integrate global contrastive learning with a local alignment tech- nique. They leverage a cross-attention mechanism to implicitly learn fine-grained correspondences between image regions and report sentences within each sample. However, while this implicit local alignment has demonstrated effectiveness for 2D CXR data, we argue that its applicability to 3D CT volumes may be limited due the dramatically higher data complexity. Specifically, compared to 2D CXR images that involve only a few anatomical anatomies Li et al. (2024b), 3D CT scans typ- ically encompass hundreds of anatomical structures and provide detailed, volumetric views of the human body Waaserthal et al. (2023). This increased imaging complexity enables a deeper analysis of intricate medical conditions while concurrently yielding more extensive and nuanced radiology reports that delineate wide-ranging anatomical features and clinical findings Ddare et al. (002);; Blankemeier et al. (2024). Given these distinctions, the endeavor to learn local alignments implic- itly, which is already prone to be sensitive to hyper-parameters and difficult to train Müller et al. 2022), becomes exceedingly intractable in CT scenarios.
3 METHOD
3.1 DATA PRE-PROCESSING
Anatomy parsing. We utilize Totalsegmentator to generate detailed anatomical structure masks for 104 regions within CT scans Wasserthal et al. (2023). encompassing organs, bones, muscles, and vessels, as illustrated in Fig. 2 Subsequently, we group these 104 regions into 36 major anatomies to align with the granularity of descriptions in clinical reports, as detailed in Appendix Tab. x This grouping is necessary because CT diagnosis reports often lack precise localization of the lesion ar- cas Li et al. (2024a). For instance, the lung is segmented into five distinct lobes in Totalsegmentator Wasserthal et al. 2023). while report might merely state "lung inflammation" without specifying which lobe affected. This ambiguity presents a significant challenge in precisely extracting corre- sponding diagnostic descriptions for each lobe from the report. Furthermore, even when the lesion locations are reported in some cases, the probability of anomalies occurring at a specific fine-grained anatomical site (i.e., right middle lobe) is considerably low, leading to overwhelming imbalance between normal and abnormal samples for that anatomical structure. As a result, most bati-batches may consist entirely of normal samples, which may skew the training process and impair the model's 's diagnostic capability. Overall, anatomical grouping entails a tra among analytical granularity, image-text consistency, and data balance.
4

---

Published as a conference paper at ICLR 2025
anatomy-wise queries
Liver
(a) Visual encoding
Via
Proj & Norm
Image Encoder
Colon
anatomy Man.myy--iiii Wil. Sult
V116
(VTT)
Image
Norm
Attention
Norm
Bladder
VLN
(c) Fine-grained VLP
(b) Textual encoding
batch of images 4, 1, la and reports R ".,
Liver
Multippe round-shaped lowrdensity. nom-
Rg,
entlonting lesions are abserved
Ti,1
Encoding
BPod & Nerm
Colon
R116
Calefication-iie highdersiity observed the wells the rectum and sigmoid oolon.
Encoder
T2,16
V4.1
1,1
8.1
V4
T12
18
(BERT)
Ten
Bladder
not thickning, na abnormal densities soen within the luman.
REN
Vea
Vaa
Paired report R₃
E
Lg
Figure 3: Framework of IVLM. (a) Visual encoding. We input a CT volume I into the image encoder and extract corresponding visual tokens for each anatomy. We then append an anatomy- specific query token to the extracted visual tokens of each anatomy. These query tokens are sub- sequently updated through self-attention,, constituting the visual representations of their respective anatomies. N is the number of anatomies. (b) Textual encoding. We decompose the paired re- port R2 into anatomy-wise descriptions and feed them separately into the text decoder to obtain anatomy-speciio textual representation. (c) Fine-grained VLP. We perform local alignment for each individual anatomy across different CT scans. Ly denotes the contrastive loss computed for the j-th anatomy.
Report decomposition. As depicted in Fig. 2 we decompose raw CT diagnostic reports accord- ing the grouped anatomies. To reduce the complexity, we employ a divide-and-conquer strategy, executing the decomposition process for the findings and impression sections of each report in- dependently, followed by an integration of extracted anatomy-level descriptions. Our approach delineated in the following three steps. First, we design a prompt (see Appendix Fig. 6 and employ the I.IM, Qwen 2.5 Bai et (2023), identify all anatomies mentioned in both sections. Notably, we found that when one section lacks explicit references to some anatomies but instead mentions their anatomical sub-structures or uses medical terminology as referents, the LLM may fail recog- nize these anatomies due to insufficient domain knowledge. To mitigate these potential omissions, we employ a complementary utring-matchin strategy. For instance, the inclusion of terms such "jejunum". "ileum", "duodenum" in the section will prompt the recognition of "small intes- tine". Second, we use the LLM to extract amatomy-specine descriptions from both sections, with the prompt detailed in Appendix Fig. Lastly, a simple post-processing is performed to integrate the anatomy-level descriptions extracted from these two sections. Specifically, for each anatomy mentioned in both sections, we concatenate the extracted findings content with its corresponding impression description. In instances where the anatomy appears in only one section, we supplement the absent component with "null" string before concatenation. If one anatomy is not mentioned in either section, we default its description to "(anatomy) shows no significant abnormalities." based on established clinical practice.
3.2 FINE-GRAINED CONTRASTIVE PRE-TRAINING
Our approach is grounded in the CLIP architecture Radford et al. (2021). which aligns visual and linguistic modalities through contrastive learning of positive and negative pairs Following Bai et al. 2024 [Co et al. (2024): Lu et al. (2024), we adopt vision transformer (ViT) Dosovitskiy et al.
2020 and BERT
Devlin al. (2018) as the image and text encoder, respectively. Given CT
5

---

Published as a conference paper at ICLR 2025
volume 1, E RIXDxHxW where D, H and W represent the inter-slice, spatial height and width dimensions respectively, the vision encoder transforms the input into a compact visual embedding Fi E Rxxdxxxxx For each anatomy, we utilize its segmentation mask M,, E {0, 10 where O represents the background and I denotes the foreground, to guide the construction of anatomy- specific visual representations. Specifically, we begin by partitioning Mij into non-overlapping x W. Each patch spatially corresponds we locate the patches that contain foreground elements of M,, and extract their associated tokens as the visual descriptors of the j-th anatomy. Next, we append a learnable anatomy-wist query token to these extracted tokens and update it via self-attention layer. Finally, the updated query token fed into linear projection layer followed by 1.2-normalization to generate anatomy-wise visual representation V.j. Given the irregular sizes of CT images between patients, we employ RandomCrop to facilitate the construction of mini-batches. It is important to note that anatomies truncated by the cropping operation will be overlooked to maintain the integrity of anatomical visual content during contrastive alignment. The absence of this visual information could include critical diagnostic cues, potentially resulting in alignment failures.
patches of size 1
a visual token in F(. Then,
Given the image's associated report R4, we decompose it into discrete descriptions Ræ. for each anatomy, detailed in Sec. 3.1 Then, we employ the text encoder transform Rag into anatomy-specifii textual embeddings Tig- For a mini-batch of images (J,,... TB} and reports (R, R, calculate the softmax-normalized image-to-text and text-to-image similarity as:
e e(TT.).u.s)/
120 Pij.A
12 Pig.k
ZK=1 (V..:
ビ== (T..ee
where j denotes anatomy index, Nj is the number of structurally complete samples for the j-th anatomy after RandomCrop, (a, b) refers to the cosine similarity between vectors a and b, T is a learnable temperature parameter. The total loss is computed as:
T N
Lac = 2
22 P H p()) (H
in which i' the number of anatomy categories, and H cross-entropy loss. 2 and y denote ground-truth one-hot similarity, where negative pairs have a probability of O and the positive pair has probability of 1.
3.3 REDUCING FALSE NEGATIVES IN IMAGE-REPORT PAIRS
The core of contrastive- based VLP lies in instance-level pairing, which brings together the vision and language modalities of the same instance while distancing different instances. However, there are often complex semantic relationships between different instances (patients) in medical contexts Hamamci et al. 2024p. For example, patients diagnosed as normal are semantically consistent and abnormal samples with the same pathologies also exhibit high semantic similarities. These seman- tically similar samples constitute false negatives when they co-occur within the same mini-batee during contrastive pre-training, and inadvertently increasing their distances could degrade the diag- nostic accuracy of medical VI.Ms. To address this issue, we propose a dual false negative reduction (FNR) approach that goes beyond instance-level pairing and pursues a more comprehensive under- standing of the semantic landscape in medical imaging.
When performing global contrastive learning between entire images and reports Hamamci et al. 2024.. patient sample is diagnosed as normal only if all scanned anatomies are free of abnormal- ities. Under this definition, the proportion of normal samples is notably low (e.g., 0.2% in MedVL- CT69K). However, in our fine- grained framework, the number of normal cases increases substan- tially (refer to Appendix Fig. X due to the more granular definition of normality at the anatomy level: although CT examination reveals abnormalities in specific anatomical structures, those unaffected anatomies can still be considered normal based on established clinical protocols. This substantial increase in normal samples leads to a proliferation of false negatives in our fine-grained contrastive learning framework. Moreover, in contrast to the relatively fixed template-style descriptions for en- tirely normal images Cao et al. (2024). normal cases of each anatomy. As result, how to identify and cope with these massive normal sam- ples poses critical challenge in unlocking the full potential of our method. We address this issue by
observe considerable variability in the descriptions for

---

Published as a conference paper at ICLR 2025
(Findings) The pancreas is swollen, with patchy fluid accumulation visible around No dilation the pancreatic diutt observed. (Impression) Acute pancreatitis considered.
Man (Findings) The structure of the pancreas is clearly displayed with normal density and uniform enhencement. No abvious dilation of the pancreatic duct observed.
(Findings) The pancreas is in a normal position with usual density. There apparent abnormal enhancement foci, and no dilation of the pancreatic duct is observed.
(Findings) The pantreas is enlarged, with blurred surrounding fat spaces. faint patchy shadows fluid density areas are visible around it. (Impression) Acute pancrestitis with peripancreatic effuslon suggested.
Taxe
FVLM-1
322
Row-wise
Dun
1..0
11.00
norm
1000
FVLM-2
Va12
2:24
bas
buth 1..0
0007
0
0
0
Predicted similarity
Final label
Default label
Correct false negatives among normal samples
Estimated simliaiity by ca teazhing
Figure 4: Illustration of the proposed dual false negative reduction approach. Vi,12 and T222 repre- sent sanerics-specific visual and textual features from the i-th sample, respectively. On one hand, we identify those samples not mentioned in the impression section of reports as normal and cor- rect the corresponding labels for these semantically consistent samples to 1 in the label matrix. On the other hand, we further incorporate the estimated image-text similarities into the label matrix, aiming to capture potential semantic relationships between different samples and thereby enhance the model's 's semantic comprehension. Notably, to mitigate error accumulation in this process, propose a co-teaching strategy, wherein two fVLMs are trained alternately and the image-text simi- larities employed by one model are estimated from the other.
leveraging the inherently hierarchical structure of CT reports Hamamci et al. (202g:: Blankemeier et al. (2024). Specifically, the findings section of report outlines all observations derived from the image, including the appearance of anatomical structures, any abnormalities such as masses or lesions, and the conditions of various anatomical components. Meanwhile, the impression section consolidates the abnormal observations into a concise summary, offering standard diagnostic con- clusions and highlighting potential discases. Based on this prior, we empirically annotate anatomies e not mentioned in the impression section as normal. We then correct v.y,k and y to if the i-th and k-th patients are both normal in terms of the j-th anatomy. Moreover, to stabilize model training, normalize y and v so that their sums equal 1.
Furthermore, due to the narrowed abnormality space from the entire body to specific anatomical regions, the abnormal samples in our fine-grained framework typically exhibit higher semantic sim- ilarity compared to those in global contrastive learning methods. For instance, although two patients exhibit significant overall differences, their pancreas may manifest the same pathology as described as follows: R1,12: "The pancreas is swollen, with patchy fluid accumulation visible around it. Acute pancreatitis is considered" and R2,12: "The pancreas is enlarged, and fluid density shadows are visible around it. Acule pancreatitis with peripancreatic fluid collection is suggested." Here, the sub- script 12 denotes the index of the pancreas in all involved anatomies. In this scenario, pushing away the visual features Vi, 12 from the textual features T2, 12 is unreasonable and potentially compromises the model's capability in pancreatitis diagnosis. To accommodate this heightened inter-sample sim- ilarity, propose bootstrapping strategy that utilizes the similarity scores pd 2 and 2 predicted by the model itself to dynamically correct the target label during contrastive pre-training. However, these predicted similarity scores may be biased. Further incorporating them into model training could cause error accumulation and ultimately result in significant performance degradation. To tackle this, we propose a co-teaching training framework that alternately trains two fVLMss, where the image-text similarity scores predicted by one model are used to correct the contrastive learning target of the other model:
y 2t = ay 126 + (1 a)p 22 y22 = =yy" 220 + (1 - a)p 12i
(3)
y = ay 22 + a)p
2i 22 =ay a + (1 a a))p"
Here [0, 1] is free parameter and we empirically set it 0.5 in this work. 22 and 2 are predicted image-to-text and exx-to-image similarities from another model. To reduce the risk of concurrent emror arising from both models for the same image-text pair, enhance their diversity
7

---

Published as a conference paper at ICLR 2025
Table 1: Zero-shot performance comparison on the MedVL-CT69K dataset. The best and second- best zero-shot results are highlighted in bold and underlined.
Method
AUC
ACC
Spec
Sens
F1
Prec
Baseline
73.3
69.1
76.2
62.0
79.4
17.6
Supervised
CT- VocabFine
76.7
72.2
76.1
68.2
81.6
20.3
CT-LiPro
76.5
70.9
76.8
65.1
81.3
19.3
CLIP
Radford et al.
(2021)
68.4
66.7
68.0
65.5
76.0
18.0
LOVI
Müller
(222
69.4
65.4
60.1
70.8
70.9
15.2
MGCA Wang et al. (2022a)
70.1
66.4
64.5
68.3
73.9
16.0
Imitate Liu al. 2023b)
70.6
67.9
66.6
69.2
75.4
17.9
Zero-shot
ASG Li al. (2024a)
70.1
67.7
67.4
68.0
75.9
18.8
CT-GLIP Lin ef al. al. ((024)
69.3
66.9
63.1
70.6
74.2
18.1
BIUD Cao et al. (2024)
71.4
69.2
69.0
69.3
76.6
18.7
Merlin Blankemeter et al. (2024
71.9
69.5
69.7
69.2
77.0
18.1
Ours
81.3
76.2
76.5
75.8
82.2
21.1
by employing different model initialization, data iteration sequences and augmentations. Fig. exemplifies the calculation of the final labels.
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Dataset. In this study, we curate MedVL-CT69K, a large-scale CT dataset comprising 272,124 CT scans from 69,086 unique patients and their associated reports. Each patient consists of non- contrast CT scan and contrast-enhanced CT scans, which include one more of the following phases: arterial, venous, and delayed. We randomly split the dataset into training, validation and test sets of 64,476, 1,151, and 3,459 patients, respectively. The validation and test sets are annotated with 36 and 54 diseases by expert radiologists. A detailed distribution of these diseases provided in Appendix Tab. 9 and Tab. TO Additionally, we conduct experiments on two benchmarks, CT-RATI
Hamamci et al (024p and Rad-ChestCTT Draclos et al. (202I). Following Hamamci et al. 2024). train IVLM
the training set of CT-RATE and use its test set and the whole Rad-Chestr
dataset for internal and external evaluations, respectively. The details regarding these two datasets
be found in Hamamci et al. (2024) and Draelos al. (2021).
Evaluation metrics. We compare the performance of different pre-training methods on zero-shot abnormality detection and downstream report-generation tasks. For the zero-shot experiments, fol- lowing Hamamci et al. (2024), we adopt the area under the ROC curve (AUC), balanced accuracy (ACC), specificity (Spec), sensibility (Spec), precision (Prec) and weighted F1 F1-score the metrics. For the report-generation task, we employ both diagnostic metrics and natural language genera- tion metrics for model evaluation. To facilitate the calculation of diagnostic metries, we develop high-performing text classifier to identify abnormalities in generated radiology reports. A detailed exposition of the classifier's training and evaluation provided in Appendix A.I
Implementation details are available in Appendix A.2
4.2 ZERO-SHOT ABNORMALITY DETECTION
Through the extensive MedVL-CTT6K dataset, we compare the zero-shot abnormality detection performance of different methods on 54 diseases across 15 anatomies. The results are presented in Tab. It can be seen that our method outperforms all counterparts by a large margin. Specifically, suppresses CLIP Radford et al. 2 by 12.9 points on AUC and 9.5 points on ACC. Furthermore, compared to the second-best competitor, Merlin Blankemeier et al. 2024 our method achieves absolute gains of 9.4 points AUC and 6.7 points ACC. Notably, observe that LOVT Müller jet al. (2022) and MGCA Wang et al. (2022a) exhibit marginal performance improvements over CL.P which underscores the significant limitations of implicit local alignment methodologies in CT
8

---

Published as a conference paper at ICLR 2025
Table Performance comparison the CT-RATE and Rad-ChestCT benchmarks. Here, CT-CLIP refers to the CLIP model trained on the CT-RATE dataset, as named in the original paper. The best and second-best zero-shot results are highlighted in bold and underlined.
Supervised CT CocaabiFin
Zero-shot
Dataset
Metric
Baseline
CT-LiPro
CT-CLIP
BIUD
Merlin
Ours
AUC
60.3
75.0
75.1
70.4
71.3
72.8
77.8
ACC
58.1
69.2
67.6
65.1
68.1
67.2
71.8
Internal
F1
63.2
72.8
71.4
69.1
71.6
70.9
75.1
validation
(CT-RATE)
Prec
24.0
34.2
33.1
30.6
33.8
33.7
37.9
Spec
68.6
66.8
71.7
Sens
67.3
70.1
72.8
AUC
$4.1
64.9
64.7
63.2
62.9
64.4
68.0
ACC
53.9
62.2
62.5
60.6
61.9
64.7
External
58.7
66.5
66.8
64.8
65.2
66.3
68.8
validation
Prec
28.7
35.9
35.3
33.9
33.7
34.8
37.4
Spec
60.2
61.7
64.6
Sens
59.6
61.0
64.6
Table 3: Performance comparison on the downstream
task using the MedVL- CT69K dataset. 'IN" means initialization with ImageNet supervised weights, while all other meth- ods are trained with our dataset. "SP" denotes the supervised baseline model.
Encoder
Init
ACC
GREEN
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
ROUGE-L
CIDEr
IN
51.3
34,0
47.4
32.2
25.5
21.2
28.1
44.3
10.6
SP
55.8
25.9
48.3
26.3
18.0
12.8
30.8
40.6
6.6
MАE
50.7
21.6
49.0
27.1
18.6
13.1
30.5
41.6
6.1
Frozen
CLIP
S7.3
33.4
49.7
28.8
20.7
15.5
31.0
42.2
9.6
BIUD
58.4
33.7
47.1
30.4
23.4
29.1
44.2
13.9
Merlin
49.5
31.4
23.9
30.0
43.8
14.3
Ours
61.5
37.2
50.7
32,2
24.5
19.6
31.3
45.1
14.9
IN
58.0
33.4
49.4
29.8
21.9
16.9
30.4
43.6
10.0
SP
60.6
35.5
49,9
30.7
22.9
17.9
30.6
43.4
11.7
MAE
54.4
29.4
49.0
27.1
18.6
15.1
30.5
42.5
8.8
Fine-
tuning
62.0
49.9
31.9
30.7
44.7
BIUD
62.6
38.8
50.2
31.7
24.0
30.9
44.7
13.8
Merlin
63.0
39.2
50.7
33.3
25.8
20.9
31.1
46.0
17.2
Ours
64.5
40.2
52.2
34.5
26.8
21.9
31.6
46.4
17.1
imaging scenarios. We enumerate the detection performance of our model for each abnormality in
Appendix Tab. 11
Tab. exhibits the performance comparison with the current state-of-the-an method (i.e., CT-CLIP) on the CT-RATE and Rad-ChestCT benchmarks. In the zero-shot setting, our method demonstrates significant improvements over CT-CLIP, achieving absolute AUC gains of 7.4% and 4.8% in the internal and external evaluations, respectively. Notably, the zero-shot performance of our model even outperforms outcomes of CT- VocabFine and CT-LiPro that are both derived from CT-CLIP through supervised fine-tuning. To be specific, in the internal test set, approach exceeds CT- VocabFine and CT-LiPro by 2.3 and 3.7 points on Fl-score. In the external test set, it surpasses these two models by 2.3 and 2.0 points F1-score.
To further assess the clinical utility of our method, we conduct a reader study to compare our method with three board-certifed radiologists. Please refer to Appendix A.3 for detailed results and discus- sions.
4.3 RADIOLOGY REPORT GENERATION
To assess the transfer abilities VLMs, we conduct experiments on the downstream task of ra- diology report generation using the MedVL-CT69K dataset. For these experiments, integrate each pre-trained image encoder with a BERT-base text decoder for whole report generation. The generation process is optimized using the language modeling loss Devlin al. (2018).

---

Published as a conference paper at ICLR 2025
Table 4: Effect of proposed modules. CLIP serves as the baseline.
0.8
Avemge Soore
0.7
FGA
FNCN
CoT
AUC
ACC
70.9
69.3
0.6
76.0
74.0
CLIP: AUC 0.4230
78.7
75.3
"
77.5
74.6
0.5
10
20
50
100
79.8
75.9
Percent of training data (%)
Figure Scaling laws of CLIP and our method.
Tab. presents the experimental results in both frozen and fine-tuning protocols, where the frozen protocol keeps the pre-trained image encoder fixed, while the fine-tuning protocol allows the en- tire model to be updated during training. demonstrates that vision-language pre-trained models outperform those with purely visual pre-training, underscoring the benefits of aligning visual and textual features into unified representation space for the report generation task. Notably, in the frozen regime, our method significantly outperforms CLIP by 4.2 points on ACC and 3.8 points on GREEN Ostmeier et al. 2024). While the performance gap attenuates in the fine-tuning protocol, method still surpasses CLIP by clear margin, achieving 2.5-point improvement on ACC and and 2.6-point improvement on GREEN. Although the results have demonstrated the superiority of approach, argue that directly employing the fin-grrines alignment model for whole report generation may not unleash full power due to the granularity mismatch issue. We will explore a potentially more effective strategy of generating anatomy-wise diagnostic reports in our future work.
4.4 ANALYSIS OF OUR FRAMEWORK
Ablation study, We investigate the impact of three modules on the performance of fVLM on the val- idation set of MedVL-CT69K, including fine grained alignment (FGA), false negatives correction between normals (FNCN) and co-teaching strategy (CoT). As shown Tab. $ each enhancement component contributes the improvement of the model's performance. Notably, the FGA and FNCN contribute the largest performance gains. The combination of them leads to overall im- provement of 7.8 points on AUC and 6.0 points on ACC. Furthermore, in Appendix A.4 we demon- strate that applying CoT to correct contrastive learning labels yields superior results compared to using either the training model or the momentum model.
Scaling law, In Fig. 5 we compute the data scaling law curves to assess how the performance of CLIP and our method improves as the volume of training data increases. can be seen that our approach consistently outperforms CLIP across multiple data scales, exhibiting superior data efficiency.
Visualization analysis. The visualization results and discussions can be found in Appendix A.5
CONCLUSION
In this paper, we have presented FVLM, fine- sie-ggaaied vision- language pre-training method for CT data. Our proposed methodology explicitly aligns discrete anatomical structures CT scans with their corresponding descriptions in diagnostic reports, thereby addressing the misalignment issue of CLIP and its existing variants that contrast entire images and reports. Extensive experiments, includ- ing quantitative abnormality detection and report generation tasks as well as qualitative visualization analysis, demonstrate the superiority of IVLM.
Limitations and future work. The implementation of our fine-grained alignment methodology necessitates localizing anatomical structures in CT images and decomposing diagnostic reports into anatomy-wil -iste sub-descriptions This data processing step entails additional resource consumption and time commitment. For future work, we plan to investigate anatomy-wist wise report generation to fully unleash the potential of FVLM on this application.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. Jianpeng Zhang supported by the Zhejiang Province Postdoctoral Re- search Excellence Funding Program (ZI2024032). This work was also supported by the Zhejiang Province "Pioneer Eagle X"" Research and Development Initiative (2024C03043).
REFERENCES
Fan Bai, Yuxin Du, Tiejun Huang, Max Q. H. Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16600, 2023.
Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Perez-Garchs Maximilian Ilse, Daniel C Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, et al. Learning to exploit temporal structure for biomedical vision-language processing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15016-15027, 2023.
Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck. Eduardo Reis, Cesar Truyts, et al. Merlin: A vision language foundation model for 3d computed tomography. arXiv preprint 205.:22006.0651.. 2024.
Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanic Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez- Valle, et al. Making the most of text semantics improve biomedical vision-language processing. In Euro- pean conference on computer vision, pp. 1-21. Springer, 2022.
Weiwei Cao, Jianpeng Zhang, Yingda Xia, Tony CW Mok, Zi Li, Xianghua Ye, Le Lu, Jian Zheng, Yuxing Tang, and Ling Zhang. Bootstrapping chest ct image understanding by distilling knowl- edge from X-ray expert models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11238-11247, 2024.
Zhihong Chen, Shizhe Diao, Benyou Wang, Guanbin Li, and Xiang Wan. Towards unifying medical vision-and-language pre-training via soft prompts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23403-23413 2023.
Pujin Cheng, Li Lin, Junyan Lyu, Yijin Huang, Wenhan Luo, and Xiaoying Tang. Prior: Prototype representation from medical images and reports. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21361-213711 2023.
Matthew Christensen, Milos Vukadinovic, Neal Yuan, and David Ouyang. Vision-language foun- dation model for cchocardiogram interpretation. Nature Medicine, pp. 1-8, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248 248-255. leee, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom,, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
11

---

Published as a conference paper at ICLR 2025
Rachel Lea Draclos, David Dov, Maciej A Mazurowski, Joseph Y Lo. Ricardo Henao, Geoffrey D Rubin, and Lawrence Carin. Machine- learnenn-------aeeeee based multiple abnormality prediction with large- scale chest computed tomography volumes. Medical image analysis, 67:101857, 2021.
Heng Guo, Jianfeng Zhang. Jiaxing Huang, Tony CW Mok, Dazhou Guo, Ke Yan, Le Lu, Dakai Jin, and Minfeng Xu. Towards a comprehensive, efficient and promptable anatomic structure segmentation model using 3d whole-body et scans. arXiv preprint arXiv:2403.15063, 2024.
Ibrahim Ethem Hamamei, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Bastian Wittmann, Enis Simsar, Mehmet Simsar, et al. A foundation model utilizing chest ct volumes and radiology reports for supervised-level zero-shot detection of abnormalities. arXiv preprint arXiv:2403.17834 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Shih- Chih-Chhng Huang, Liyue Shen, Matthew P. Lungren, and Serena Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3942- 3951, October 2021.
Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual- language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):2307-2316 6, 2023.
Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu- net: self-coonig method for deep learning- methods, 18(2):203-211, 2021.
bearning-based based biomedical image segmentation. Nature
Qingqiu Li, Xiaohan Yan, Jilan Xu, Runtian Yuan, Yuejie Zhang, Rui Feng, Quanli Shen, Xiaobo Zhang, and Shujun Wang. Anatomical structure-guided medical vision-language pre-training. arXiv preprint arXi::2403.09294, 2024a.
Shiyu Li, Pengehong Qiao, Lin Wang, Munan Ning, Li Yuan, Yefeng Zheng, and Jie Chen. An organ-aware diagnosis framework for radiology report generation. IEEE Transactions on Medical Imaging, 2024b.
Jingyang Lin, Yingda Xia, Jianpeng Zhang, Ke Yan, Le Lu, Jiebo Luo, and Ling Zhang. Ct-glip: 3d grounded language-image pretraining with scans and radiology reports for full-body scenarios. arXiv preprint arXivv2404.15211 2024.
Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pme-clip: Contrastive language-image pre-training using biomedical documents. In In- lernational Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 525-536. Springer, 2023.
Che Liu, Sibo Cheng, Chen Chen, Mengyun Qiao, Weitong Zhang, Anand Shah, Wenjia Bai, and Rossella Arcucci. M-flag: Medical lasion-language pre-training with frozen language models and latent space geometry optimization. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 637-647. Springer, 2023a.
Che Liu, Sibo Cheng, Miaojing Shi, Anand Shah, Wenjia Bai, and Rossella Arcucci. Imitate: Clinical prior guided hierarchical vision-language pre-training. arXiv preprint arxiv:23 2023b.
Jie Liu, Yixiao Zhang, Jic-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmenta- tion and tumor detection. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 21152-21164 166,, 2023c.
Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-language foundation model for computational pathology. Nature Medicine, 30(3):863-874, 2024.
12

---

Published as a conference paper at ICLR 2025
Philip Müller, Georgios Kaissis, Congyu Zou, and Daniel Rueckert. Joint learning of localized representations from medical images and reports. In European Conference on Computer Vision, pp. 685-701- Springer, 2022.
Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Ame Edward Michalson, Michael Moseley, Curtis Langlotz, Akshay S Chaudhari, et al. Green: Generative radiology report evaluation and error notation. arXiv preprint arXiv:2405.03592 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Yuxuan Sun, Chenglu Zhu, Sunyi Zheng, Kai Zhang, Lin Sun, Zhongyi Shui, Yunlong Zhang, Honglin Li, and Lin Yang. Pathasst: A generative foundation ai assistant towards artificial gen- eral intelligence of pathology. In Proceedings of the AAAI Conference on Artificial Intelligence. 2024.
volume 38, pp.
Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar. Expert- level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature Biomedical Engineering, 6(12):1399-1406 2022.
Amar Udare, Minu Agarwal, Kiret Dhindsa, Amer Alaref, Michael Patlas, Abdullah Alabousi, Yoan K Kagoma, and Christian B van der Pol. Radiologist productivity analytics: factors im- pacting abdominal pelvic et exam reporting times. Journal of Digital Imaging, pp. 1-11, 2022.
Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti, and Lequan Yu. Multi-granularigg cross-modal alignment for generalized medical visual representation learning. Advances in Neural Information Processing Systems, 35:33536-33549, 2022a.
Haoyu Wang, Sizheng Guo, Jin Ye, Zhongying Deng, Junlong Cheng, Tianbin Li, Jianpin Chen, Yanzhou Su, Ziyan Huang, Yiqing Shen, Bin Fu, Shaoting Zhang, Junjun He, and Yu Qiao. Sam- med3d, 2023.
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medelip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163, 2022b.
Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang. et al. Totalseg- mentator: robust segmentation of 104 anatomic structures in images. Radiology: Artificial Intelligence, 5(5), 2023.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowl- edge enhanced language-image pre-training for x-ray diagnosis. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 21372-21383, October 2023.
Yutong Xie, Qi Chen, Sinuo Wang, Minh-Son To, Iris Lee, Ee Win Khoo, Kerolos Hendy, Daniel Koh, Yong Xia, and Qi Wu. Pairaug: What can augmented image-text pairs do for radiology? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11652-11661, 2024.
Janpeng Zhang, Xianghua Ye, Jianfeng Zhang, Yuxing Tang, Minfeng Xu, Jianfei Guo, Xin Chen, Zaiyi Liu, Jingren Zhou, Le Lu, et al. Parse and recall: Towards accurate lung nodule malig- nancy prediction like radiologists. International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 199-209. Springer, 2023a.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre- ston, Rajesh Rao, Mu Wei. Naveen Valluri, et al. Biomedclip: multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.009/5. 2023b.
13

---

Published as a conference paper at ICLR 2025
Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhancee visual-language pre-training on chest radiology images. Nature Communications, 14(1):4542, 2023c.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis Langlotz. Con- trastive learning of medical visual representations from paired images and text. In Machine Learn- ing for Healthcare Conference, pp. 2-25. PMLR, 2022.
Hong- Yu Zhou, Chenyu Lian, Liansheng Wang, and Yizhou Yu. Advancing radiograph representa- tion learning with masked record modeling. arXiv preprint arXiv:2301.13155 2023.
14


Reviews:
[{"review_id": "ZDMV9wHsb7", "reviewer": "Reviewer_PfQv", "summary": "The paper proposes a fine-grained vision-language model (fVLM) to improve CT image interpretation by associating specific anatomical regions in CT scans with corresponding descriptions in radiology reports, addressing limitations of global image-report contrastive methods. It introduces a large dataset, MedVL-CT69K, encompassing 272,124 CT scans and achieves state-of-the-art performance. Key contributions include anatomy-level contrastive learning and a method to handle false negatives.", "questions": "1. What's the authors reproducibility statement? Will the dataset, code and weights be released? Especially publishing the dataset would be of great value for the community.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Fine-Grained Anatomy-Level Alignment: The paper introduces a novel, fine-grained vision-language model (fVLM) that aligns anatomical regions in CT images with corresponding report descriptions.\n2. Large and Comprehensive Dataset: Experiments were conducted on a range of datasets including the largest CT dataset to date (MedVL-CT69K)\n3. Effectively tackling shortcomings of their method by introducing the dual false-negative reduction approach.", "weaknesses": "1. Experiments are incomplete: Table 1 doesn't include a performance evaluation of the methods from Table 3, namely CT-CLIP, CT-VocabFine, CT-LiPro. Table 3 doesn't include performance evaluation of the methods from Table 1. In both cases it's not argued why those experiments were not conducted. For Table 1, and Table 2 it's also unclear how the 2D approaches were adapted for 3D and how their evaluation was then done. Additionally, the evaluation metrics in Table 1 and Table 3 differ. Why? Furthermore, the performance comparison in Table 2 should also include comparisons to other CT-CLIP approaches, not only natural image counterparts (he it's again not clear how the 2D models were adapted to 3D).  \n2. Report generation uses only a single NLU metric. Other NLU metrics such as ROUGE would be interesting. Furthermore, using the GREEN metric[1] or RadFact[2] for evaluation would allow assessing clinical relevance.\n3. T-SNE visualization: An (additional) comparison of the embeddings to CT-CLIP would be be interesting to see if they have the same clustering behavior. \n4. Ablations study on masking missing: The masking approach is not well investigate and no ablation studies are done. One intuitive baseline would be the anatomical cropping of the image. What's the advantage of the masking approach? \n5. The authors approach only works for CT scans, since it relies on anatomy segmentations from TotalSegmentator. Furthermore, only on the classes which are available in TotalSegmentator.\n\n[1] Ostmeier, S., Xu, J., Chen, Z., Varma, M., Blankemeier, L., Bluethgen, C., ... & Delbrouck, J. B. (2024). GREEN: Generative Radiology Report Evaluation and Error Notation. arXiv preprint arXiv:2405.03595.\n\n[2] Bannur, S., Bouzid, K., Castro, D. C., Schwaighofer, A., Bond-Taylor, S., Ilse, M., ... & Hyland, S. L. (2024). MAIRA-2: Grounded Radiology Report Generation. arXiv preprint arXiv:2406.04449.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "znvdC2yFhF", "reviewer": "Reviewer_oBhB", "summary": "This work discusses the fine-grained alignment issue for vision-language clip-like pretraining for CT and reports. To improve the granularity of alignment the authors propose to pre-segment key anatomical structures in CT using a public segmentation model (Totalsegmenter), and to pre-segment descriptions corresponding to individual anatomical structures in the free-text report using an open-access LLM (Qwen). This allows fine-grained contrastive learning at the anatomical structure level. Considerations are made to avoid pushing representations of the same organ/condition away. Improved performances are shown for zero-shot abnormality detection compared with several existing approaches.", "questions": "A high-level question: The capability of open-source image segmentation tools (totalsegmenter and/or SAM family) and open-source LLM advance fiercely -- does it imply that the utility of (un-/weakly-supervised) vision-language pretraining for CT will gradually phase out? One may directly construct semantic labels from text reports using strong enough LLM tools and turn the problem back to large scale supervised learning. I am curious to hear the authors’ insight.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of increasing the granularity of contrastive learning by data pre-segmentation using public tools is neat, simple, and straightforward. \n\nThe paper is written with sufficient clarity, where the insights behind each design and the implementations details are well-presented. \n\nImproved performance on zero-shot abnormality detection is shown compared with some existing works.", "weaknesses": "For a comprehensive assessment of report generation some essential scores are missing, such as BLEU 1-3, ROUGE-L, and METEOR. Also, comparisons with vision encoders from peer vision-language pre-training for CT works may be needed. This is my major concern as the authors have claimed improved performance on report generation task.\n\nDespite improved granularity, attributing sentences of reports `mentioning` a structure to the image feature of that structure may sometimes be misleading especially when manual verification/correction is not accessible. E.g., in practice pancreatitis is often associated with the entire abdomen instead of the pancreas alone and neighboring structures. The authors are encouraged to comment on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ND1hzw9hUg", "reviewer": "Reviewer_wu1u", "summary": "The authors present an vision-language model for  anatomy-aware medical report generation, adressing the task of false negative reduction.  A large (private) dataset is curated and good performances on public datasets are presented.", "questions": "* Image Encoder Setup: The authors mention using a ViT base model with MAE pretraining on ImageNet-1K and a patch size of 16 x 16 x 32 (referenced in A.2). However, it’s unclear how they process 3D volumes with 3D patch embeddings through a 2D ViT model. Although they provide details on the training procedure, the overall model architecture remains vague. Given the lack of source code, a clearer description of the model's structure is crucial. Can the authors improve?\n* Comparison with 2D CLIP Methods: The comparisons made with 2D CLIP methods in Table 1 are not fully explained. Specifically, it's unclear how they adapted these methods to a 3D context and how the training was conducted. Please clarify.\n* Evaluation with CT-CLIP: In Table 3, the comparison with CT-CLIP models is ambiguous. Did the authors fine-tune a pre-trained model (based on their in-house data) on CT-RATE, or was there a different approach taken?\n* t-SNE Consistency: To enhance the consistency of the embedding spaces shown in Fig. 7, the authors could consider setting a random seed before calculating the t-SNE embeddings.\n* HU Value Clipping: The authors chose to clip HU values between -300 and 400, which raises a question: Does this range cover all abnormalities? For instance, lung-related abnormalities are typically closer to -1000 (representing air), while bone-related abnormalities can reach values close to 1000. Since they utilized both non-contrast and contrast-enhanced CT scans (arterial, venous, and delayed phases), did they apply this same HU preprocessing across all scans? Additionally, did the authors use this range for the CT-RATE training ablations (which is different than the original CT-RATE paper)? Please clarify.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The study addresses an interesting multi-modal learning problem from the medical domain\n* The presented false negative reduction is a meaningful integration to vision-language model report generation\n* The results are promising indicating the relevance of this addition", "weaknesses": "* The methodology is not quite clear\n* There is no codebase, preproduction may be a challenge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f1SJToGPFK", "reviewer": "Reviewer_5y7k", "summary": "The paper presents fVLM, a fine-grained vision-language model for CT image interpretation. The key innovation is to perform anatomy-level contrastive learning between CT scans and radiology reports. The authors introduce a dual false negative reduction approach and curate a large dataset (MedVL-CT69K) with 69,086 patients. The model achieves strong zero-shot performance across 54 disease diagnosis tasks and outperforms existing methods on public benchmarks.", "questions": "1. Will the MedVL-CT69K dataset or the pre-trained model checkpoint be open-sourced? Additionally, why are there different numbers of diseases annotated in the validation (36) and test (54) sets? \n2. While Figure 7 shows more compact embedding clusters for positive cases compared to CLIP, there seems to be limited separation between positive and negative cases. \n3. It would be great if the radar plot could link with positive case numbers and normal/abnormal ratios.  This would help readers better understand the model's performance across different conditions and data distributions. Also more analysis on long-tailed disease.\n4. The scaling law curve is interesting, 272,124 CT scans from 69,086 unique patients are the largest CT dataset till now, but still far away from data saturation. Do you do the data selection (try to include more abnormal patients) when collecting the dataset? Will the scaling law related to data distribution?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Anatomy-level fine-grained alignment for the complex report generation task/ diagnosis task is a great idea. \n2. The paper collects large-scale datasets with expert annotations, which is time-consuming and expensive in the medical domain.\n2. The paper is well-structured, with extensive results (including ablation studies and visualizations) showing clear improvements over baselines.", "weaknesses": "The evaluation metrics for the report generation task are limited to basic metrics (BLEU-4) and diagnostic accuracy. The paper should include medical-specific metrics like RadCliQ, RaTESCore, or GREEN that better capture clinical relevance and factual consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)