Towards Scalable Language-Image Pre-training for 3D Medical Imaging
Authors: Chenhui Zhao
Yiwei Lyu
Asadur Zaman Chowdury
Edward S Harake
Akhil Kondepudi
Akshay T Rao
Xinhai Hou
Honglak Lee
Todd C Hollon
Venue: Accepted by TMLR ()

Abstract: The scalability of current language-image pre-training for 3D medical imaging, such as CT and MRI, is constrained by the need for radiologists to manually curate raw clinical studies. In this work, we pioneer pre-training directly on uncurated studies, which both aligns more closely with the clinical workflow and provides a natural path to scalability. However, the unique structure of such data presents new challenges for existing model architectures, which were originally designed for 2D slices or single 3D scans. To address this, we introduce a novel hierarchical attention mechanism inspired by the intrinsic hierarchy of radiology data: slice, scan, and study. We denote our framework as Hierarchical attention for Language-Image Pre-training (HLIP). Trained on 220K studies with 3.13 million scans for brain MRI and 240K studies with 1.44 million scans for head CT, HLIP achieves state-of-the-art performance, e.g., +10.5% balanced ACC on the proposed publicly available brain MRI benchmark Pub-Brain-5; +8.3% and +1.7% macro AUC on head CT benchmarks CQ500 and RSNA, respectively. HLIP also exhibits strong generalizability on existing 3D medical language-image pre-training benchmarks, e.g., +4.3% macro AUC on the Rad-ChestCT benchmark when pre-trained on CT-RATE. These results demonstrate that, with HLIP, directly pre-training on uncurated clinical datasets is a scalable and effective direction for language-image pre-training in 3D medical imaging. The code is available at https://github.com/zch0414/hlip

Published in Transactions on Machine Learning Research (02/2026)
Towards Scalable Language-Image Pre-training for 3D Medical Imaging
Chenhui Zhao University of Michigan
chuishao@humich.co edu
Yiwci Lyu University of Michigan
ylucilguueumicch.eee edu
Asadur Chowdury University of Michigan
achowdur@umicce edu
Edward Harake University of Michigan
charakej@umich. edu
Akhil Kondepudi University of Michigan
akbill:@huicch.co edu
Akshay Rao University of Michigan
akshagro@umichh. edu
Xinhai Hou University of Michigan
Finhaih@uuiide edu
Honglak Lee University of Michigan
honglak@umiich. edu
Todd Hollon University of Michigan
tocho edu
Reviewed on OpenReview: https: openreview. net/ forun? id- Wallf BBBBAA
Abstract
The scalability of current langunge-image pre-training for 3D medical imaging, such CT and MRI, is constrained by the need for radiologists to manually curate raw clinical studies. In this work, we pioneer pre-training directly on uncurated studies, which both aligns more closely with the clinical workflow and provides a natural path to scalability. However, the unique structure such data presents new challenges for existing model architectures, which were originally designed for 2D slices or single 3D scans. To address this, we introduce novel hierarchical attention mechanism inspired by the intrinsic hierarchy of radiology data: slice, scan, and study. We denote our framework as Hierarchical attention for Language-Lmage Pre-training (HLIP). Trained on 220K studies with 3.13 million scans for brain MRI and 240K studies with 1.44 million scans for head CT, HLIP achieves state-of-the-art performance, e.g., +10.5% balanced ACC on the propased publicly available brain MRI benchmark Pub- Brain-5; +8.3% and +1.7% macro AUC on head CT benchmarks CQ500 and RSNA, respectively. HLIP also exhibits strong generalizability on existing 3D medical languag--mage pre- pre-training benchmarks, e.g., +4.3% macro AUC the Rad-ChestCT benchmark when pre-trained CT-RATE. These results demonstrate that, with HLIP, directly pre-training uncurated clinical datasets is scalable and effective direction for language-image pro-training in 3D medical imaging. The code is available at https: //gith// comm/cch0144/hii 4/hlip

---

Published in Transactions on Machine Learning Research (02/2026)
Uncurated Radiology Study for Patient
Brain Disease Zero-Shot Classification Performance
Others
HLIP
Annotation Bottleneck
for Data Curation
Data Scale f
-10%
Balanced Accuracy (%)
60
Biomedical CLIP SOTA
Effeciency f
Curate
Modeling
50
Performance
47.9
40
Uncurated Datasels
Segmented
selection
2D image
selection
Curated Dutasets
>220K
VLM 69K
20
Meria INX CTRATEE MBD 42K
Vision Encoder
HLIP Encoder
VII
ViTi
VII
HLIP
Radiology Report Supervision
Radiology Report Supervision
(a) Previous works and HLIP
(b) Naive study modeling strategy and HLIP
Figure 1: Illustration of (a) an uncurated study for patient. While previous work has relied annotation and curation, HLIP enables language-image pro-training directly on uncurated data. (b) Despite training on large-scale domain-specific datasets, naively modeling the uncurated study with a vanilla ViT, e-g,, by randomly selecting a scan at each training step", encoding scans independently before study aggregation or directly encoding the entire study?, yields performance only comparable to the SOTA trained on PubMed corpus, whereas HLIP outperforms these by large margin.
1 Introduction
Language--uupervised pre-training is well-suited for radiology, where each study comprises medical images paired with a corresponding radiologist's report. This natural alignment between visual and textual information has motivated the adaptation of language-image pre-training methods, such as CLIP (Zhang et al., 2022a; Radford et al., 2021), to learn clinically mcaningful radiology representations. CLIP-based models are especially notable for their clinical utility: they demonstrate strong zero-shot transfer performance tasks (Hamanci et al., 2024; Blankemeier et al., 2024), and their encoders consistently improve performance on multimodal learning benchmarks (Bai et al., 2024; Shui et al., 2025). In the domain of 2D modical imaging, such as chest X-rays, language pre-training is a key driver for integrating computer vision into clinical workflows (Boecking et al., 2022; Wang al., 2022b; Tiu et al., 2022).
diagnost
However, languago-image pre-training in 3D medical imaging has yet reach the scale performance demonstrated in 2D modalities. For instance, chest X-ray CLIP models have been trained corpus (Wang et al., 2022b; Tiu et al., 2022; You et al., 2023), achieving human-level performance on multiple diagnostic tasks (Tiu et al., 2022). In contrast, progress in 3D medical imaging remains limited, with existing models underperforming relative to their 2D counterparts (Hamamci et al., 2024; Bai et al., 2024). We attribute the performance this gap to two primary factors: data annotation that constrains the training scale, and architectural limitations arising from the complexity of 3D modical imaging.
500K
Computed tomography (CT) and magnetic resonance imaging (MRI) generate 3D volumetric images across various anatomical regions, including the brain, chest, and abdomen. As illustrated in Figure standard MRI study typically includes several sequences (c.g., T1-weighted, T2-weighted, and FLAIR), each contributing distinct diagnostic information. Similarly, CT sludies oflen include scans acquired with varying orientations or scanner settings within the same study. To perform language-inan image pre-training for such data, common stratogy is to curate datasets by having radiologists manually select a representative scan or slice from each study, as shown in Figure I(a) (Blankemeier al., 2024; Hamamci al., 2024; Bai et al., 2024; Shui al., 2025), which presents significant barrier to the scalability. In contrast, pre-training on uncurated studies
2

---

Published in Transactions on Machine Learning Research (02/2026)
aligns more closely with real-world practice and readily expands the data scale, it imposes no additional burden radiologists. Despite this, the unique structure of such data presents new challenges for current. visual encoders, which were originally designed for 2D images or single 3D scans (Dosovitskiy et al., 2020; Liu et al., 2021; Ryali et al., 2023). As shown in Figure (bb), even trained on a large-scale domain-specific dataset, naively encoding uncurated studies with the Vision Transformer (ViT) (Dosovitskiy el al., 2020) results only comparable zero-shot transferability to the state-of-the-art (SOTA) biomedical CLIP (Nie et al., 2025) trained on the PubMed corpus. Particularly, encoding the entire study can produce tokens on the order of 10 which both incurs substantial computational overhead and limits performance (Barbero et al., 2024).
In this work, we first address the key barriers to scaling language-image pre-training for 3D medical imaging by pioneering the use of uncurated studies. Second, to effectively extract features from such data, we introduce a novel hierarchical attention mechanism leverages the natural hierarchy of radiology data: slice, scan, and study. We name this framework Hierarchical attention for Language-Image Pre-training (HLIP). Unlike architectural designs such as Swin (Liu et al., 2021), MVIT (Fan al., 2021; Li et al., 2022b), and Hiera (Ryali et al., 2023), HLIP leverages the inherent data st struuueee to define the attention scope, making it. suitable for uncurated studies that contain multiple 3D scans. Compared to window attention that captures only local features, slice or scan attention can capture all diagnostic features while also providing constructive priors for learning study representations. Moreover, slice and scan attention is already much lighter than study attention, and such minimal adaptation of the original ViT remains orthogonal to flash attention (Dao et al., 2022) and patch dropout (Li et al., 2023), further reducing the computational burden.
Trained on our health system, HLIP outperforms the SOTA (Nie al., 2025) on the proposed publicly available brain MRI benchmark, Pub-Brain-5, by 10.5% balanced ACC; and surpasses the head CT foundation model (Zhu et al., 2025) by 8.3% and 1.7% macro AUC the CQ500 (Chilamkurthy et al., 2018) and RSNA (Flanders et al., 2020) benchmarks, respectively. HLIP also demonstrates strong generalizability on the curated 3D medical language-inan image pre-training benchmark CT-RATE (Hamamci et al., 2024), which contains only one scan per study, outperforming the SOTA (Lin et al., 2023) by 4.3% macro AUC on the external evaluation Rad-ChestCT (Draelos el al., 2021). Our paper makes the following contributions:
We introduce HLIP, an effective and scalable language-image pre-training framework for uncurated 3D medical imaging, that leverages a novel hierarchical attention mechanism derived from the natural structure of radiology data.
We conduct the sargest-scale training for 3D medical imaging to date, using 220K studies with 3.13 million scans for brain MRI and 240K studics with 1.44 million scans for head CT.
We demonstrate the state-of-the-art performance on multiple benchmarks spanning diverse modalities and anatomical regions, including brain MRI, head CT and chest CT.
We release the following assets to the public: a brain MRI benchmark for zero-shot classification, an effective language-image pre-training implementation for 3D medical imaging, the pre-training recipe, and model checkpoints.
2 Related Work
2.1 Image Pre-training in Radiology
has been developed for 2D and 3D medical imaging, facilitated by public datasets such as CheXpert (Irvin et al., 2019), MIMIC-CXR (Johnson et al., 2019), and CT-RATE (Hamamci et al., 2024). In 2D imaging, techniques such as local alignment (Huang et al., 2021; Wang et al., 2022a; Müller et al., 2022), knowledge enhancement (Wang et al., 2022b; Wu et al., 2023), and longitudinal analysis (Bannur et al., 2023) have been explored. CheXZero (Tiu et al., 2022) demonstrates strong empirical results comparable to those of human experts in the domain of chest X-rays. BiomedCLIP (Zhang et al., 2023a) and ConceptCLIP (Nie et al., 2025) have been trained on more than 15 million sample pairs, achicving strong performance across modalities, including radiology. In addition, other works (Boecking et al., 2022; You et al., 2023; Zhang et al., 2023b) have also contributed significant insights and achieved impressive performance.
3

---

Published in Transactions on Machine Learning Research (02/2026)
3D medical imaging offers more comprehensive view of anatomical structures. However, due to its computational cost. and limited data availability, several studies (Cao et al., 2024; Wang et al., 2024; He et al., 2024) focus on bridging the domain gap between 2D and 3D. For example, UniMedl (He et al., 2024) learns a shared feature space for both 2D and 3D modalities and demonstrates improvements across both. BIUD (Cao et al., 2024) distills 3D representations from a well-trained 2D model (Tiu et al., 2022), significantly improving data efficiency.
More recently, aided by advances in hardware and the availability of public datasets (Hamamci et al., 20244,, several studies (Hamamci et al., 2024; Bai et al., 2024; Shui et al., 2025; Lai et al., 2025) have performed hinguag-imaage pre-training on 3D imaging, surpassing methods (Cao et al., 2024) that still rely on 2D representations. Specifically, CT-CLIP (Hamanci et al., 2024) has been pre-trained on 20,000 paired chest CT scans and reports. M3D (Bai et al., 2024) explores a versatile multi-modal large language model designed for universal 3D medical imaging analysis. fVLM (Shui al., 2025) proposes a fino-grained pre-training framework that relies on segmentation to perform organ alignment. However, while these methods have demonstrated promising performance on various tasks such as zero-shot abnormality detection and report. generation, several factors limit their training scalability and real-world applicability. For example, CT-CLIP and M3D are constrained to curated studies containing only single imaging scan. Such studies rely heavily on human annotation and also fail to reflect real-world scenarios, where studies typically include multiple scans. IVLM further depends on a segmentation model, which introduces bias from segmentation quality and largely limits its scalability and generalizability. Moreover, these methods suffer from inefficient modeling of 3D medical imaging, leading to small batch sizes (e.g., 48), which are insufficient for effective language-image pre-training (Radford et al., 2021).
2.2 Efficient Language-Imas Pre-training
is crucial, as it directly affects the number of sample pairs seen during training and contrasted per batch, two key factors of model capacity. Existing hicrarchical architectures (Liu et al., 2021; Hamamci et al., 2024) have been widely adopted to improve efficiency. However, as analyzed in Appendix E.2, these models may be less efficient than commonly assumed in the context of 3D medical imaging, even when compared with the original expensive for
VIT. From the architecture perspective, components such as relative position embedding 3D inputs and is not compatible with recent advancements like flash attention (Dao et al., 2022). From the training perspective, FLIP (Li et al., 2023) demonstrates a favorable trade-off by randomll removing 50% of tokens during training. However, models that require a fixed activation shape (Liu et al., 2021; Li et al., 2022b;a; Ryali et al., 2023) cannot benefit from this strategy. Moreover, many works in radiology (Matsoukas et al., 2022; Tiu et al., 2022: Zhao & Shen, 2024; Shui et al., 2025) have demonstrated the benefits of the universal feature learned by MAE (He et al., 2022). Therefore, modeling 3D medical imaging with minimal adaptation of the original VIT (Dosovitskiy et al., 2020) is promising research direction and stands to benefit from recent advancements such as flash attention, patch dropout, and pre-trained models.
3 Method
Our goal is perform language-inage pre-training on uncurated studies. Given a study S E HRMxIXDXHXW where M denotes the number of single-channel 3D scans, and D, H, and W represent the depth, height, and width dimensions of each scan, respectively. Following the VIT (Dosovitskiy et al., 2020), each scan is divided into a grid of non-overlapping volumes of size (2) и h W). These volumes are projected into visual tokens F. E RR-x where N = M x d x h x w is the total number of tokens and e denoles the number of channels. As N can be on the order of 104, computing self-attention over all tokens throughout the VIT backbone is prohibitive memory and also limits performance (Barbero et al., 2024). We introduce a novel hierarchical attention mechanism guided by the inherent data hierarchy of uncurated studies, enabling lighter self-attention while introducing effective priors derived from this hierarchy: slice, scan, and study.
4

---

Published in Transactions on Machine Learning Research (02/2020)
(1) Rudiology Data Hierarchy and Hierarchical Atfention
Study
Study Attention
Radivlogs Stady
(b) HLIP: Hicrarchical attention Language-imago Pre-training
Contrastive
Loss
Visual Encoder
Text Encoder
Scan
Scan Attention
Scom
study layor
soan alin layar
layer
Radiology Report
Adiacent
Wiffrcmm SSee
Slices
Slice Attention
Tokens
Uncurated Radiology Study
Figure 2: Illustration of (a) the radiology data hierarchy for a single patient, including the study, single sean, and adjacent slices. Our hierarchical attention mechanism mirrors this hicrarchy and computes self-attention independently within each level. (b) Our HLIP framework incorporates a visual encoder that performs attention at different levels. In practice, lightweight slice scan attention with a few study attention layers suffices to extract features from the full study.
3.1 Hierarchical Attention Mechanism
The radiology data exhibits an inherent hicrarchical strueture for each patient, and our proposed hicrarchical attention mechanism mirrors this structure. As illustrated in Figure 2, the data for an individual patient. ient. comprises three levels:
The study contains M imaging scans, whose modalities and acquisition planes are selected by a radiologist. based on the clinical context. Collectively, these M scans contain all the visual information required for radiologic diagnosis.
The scan contains D slices. While single scan conveys only partial context of the full study, still captures the complete extent of the target pathology.
The adjacent slices contains D consecutive slices within a single scan. Although it conveys only partial information about the target pathology, the slices can capture focal diagnostic features.
We explore a simple hierarchical attention mechanism grounded in this structure. Specifically, as shown in Figure 2, we compute self-attention independently within each hicrarchical lovel:
Study attention computes single sclf-attention operation over all N tokens in study. The I/O complexity (Aggarwal & Vitter, 1988) of the study altention is Q(N2 + N x e) (Dao el al., 2022), detailed in Appendix E.
Scan attention computes M independent self-attention operations, each over h tokens within a single scan. The I/O complexity of the scan attention is 2 N² + x c).
Slice attention computes M x d independent self-attention operations, each over h x tokens within a N²
group of adjacent slices. The T/O complexity of the slice attention O Mxa + N x c).
5

---

Published in Transactions on Machine Learning Research (02/2026)
The proposed hierarchical attention mechanism contrasts with existing methods that modify attention computation using convolution (Fan et al., 2021; Li et al., 2022b), pooling (Ryali et al., 2023), or shifted windows (Liu et al., 2021), all of which require either regular activation shapes or attention masks. Our mechanism relics solely on a simple reshape operation, as there is no overlap between different scans or groups of adjacent slices', allowing greater flexibility for uncurated studies that comprise multiple scans. As shown in Figure 2, each layer in the ViT can perform attention at any level. In practice, we evenly divide the ViT backbone into four subsets of layers (e.g., three layers per subset for the 12-layer ViT-B) and apply study attention only to the last layer of each subset, while the remaining layers perform the lighter scan slice attention. Moreover, this intentionally simple design also enables seamless integration with recent efficiency advances such as patch dropout (Li et al., 2023) and flash attention (Dao et al., 2022).
3.2 Implementation
Model. Our visual encoder is a MAE pre-trained VIT-B (Hle et al., 2022). The input is rescaled to [O, and normalized with MAE's averaged meon and std. Compared to the original ViT designed for 2D RGB images, our vision encoder only has three differences: (1) each scan is divided into 3D volumes instead of 2D patches; (2) the positional embedding encodes 3D spatial coordinates within each scan, along with an additional dimension to distinguish different scans; The cls token is minimally adapted to propagate information across layers that perform different hierarchies of attention.
For (1), which employs convolution layer (He et al., 2022), we adopt average inflation initialization (Zhang et. al., 2022b). Specifically, we first sum the 2D weights along the channel dimension to accommodate a single-channel input, then replicate them & times to construct the 3D weights, and finally scale by a factor of $ We set the token size as (8,16,16) 16) by default. For (2), we tile the pre-trained 2D positional embedding Pance E Rxxuxe with a 1D sinusoidal positional embedding for slices, Pstice E TRdxe, and another for scans, Peean E MM-Maa xc_ Mmor indicates the maximun number of scans our model can process without interpolation. During training, to construct a batch, we randomly sample M scans along with M positional embeddings from Pecan, which forms the final positional embedding E R xdxxxxxxxx We set Max==40 and M=10 by default. For (3), the cls token may lose gradient continuity across different hicrarchics. To address this issue while maintaining the efficiency of our architecture, we propagate the cls loken using a combination of cloning and averaging. For example, when transitioning from the study to scan attention layer, we distribute the cls token across M scans by cloning. In the reverse direction, we aggregate information from M cls tokens into a single cls token by averaging them. The transition between other hierarchies (e.g., study and slice or scan and slice) follows the same procedure. Empirically, we find this stralegy to be sufficient, performing better than alternatives such as weighted average or global pooling at the end of the encoder.
For uncurated datasets, we use scan attention with 4 evenly distributed study attentions. A curated datasct with only one scan per study corresponds to the special case of M=1 in our scenario. For such datasets, we employ slice attention with 4 evenly distributed scan attentions; We use PubMedBERT (Gu et al., 2020) our text encoder. Model configurations are provided in Appendix B.
Datasets. Given that HLIIP enables pre-training on real-world, clinical studies, we collect two datasets within our health system, namely BrainMR1220K and HeadCT940K. BrainMRI220K contains 220,993 MRI studies. We hold out 992 studies for hyperparameter tuning using the retrieval task. In the training split, the number of scans per study ranges from 1 162, with the third quartile at 17; the number of slices per scan ranges from 5 to 500, with the third quartile at 80. HeadCT240K contains 244,253 CT studies for training and 998 held-out studies. The nunber of scans per study ranges fro 1 to 71, with the third quartile at 6; the number of slices per scan ranges from 5 to 500, with the third quartile at 110. More details about these two datasets are provided in Appendix A.
With the original attention layer unchanged, study attention takes the input of size (B, M x d d w, c); scan attention takes the input of size (B M, x x w. c); and slice attention takes the input size (B x M x d, h x wr c), where B denotes the batch size.
6

---

Published in Transactions on Machine Learning Research (02/2020)
Preprocessing. We do not standardize the orientation spacing. Instead, consistently align the depth dimension with the through-plane axis of each scan, and then resize it to fixed shape of (48, 224, 224). As this differs from prior practice (Blankemeier et al., 2024; Hamamci et al., 2024; Shui et al., 2025), we provide a discussion in Appendix A.3. For brain MRI, we apply 10.5,99 5,99.5) percentile clipping to the intensity values. For head CT, we expand each scan into three separale scans, with Hounsfield Unit (HU) values truncated to [20] for soft tissue, [-20, 180] for blood vessels, and [-800, 2000] for bone.
Pre-training. Our implementation builds upon OpenCLIP (Cherti et al., 2023) and FLIP (Li et al., 2023). We apply 25% patch dropout by default as regularization and acceleration. We do not apply additional augmentation or unmasked fine-tuning, Training 20 epochs on our uncurated datasets takes ~1 day with a batch size of 256 on 8 L40 GPUs. More details are provided in Appendix B.2.
4 Experiments
We apply HLIP to brain MRI, head CT, and chest CT. For brain MRI, we construct new benchmark for zero-shot classification, using public datasets (Dufumier et al., 2022; Liu et al., 2023; Baid et al., 2021; LaBella et al., 2023; Moawad et al., 2024; Kazerooni et al., 2024; Rudie et al., 2024; Link et al., 2024). For head CT and chest CT, we follow ovaluation protocols from previous work (Hamamci et al., 2024; Shui et al,, 2025; Zhu et al., 2025). Experiments on our uncurated brain MRI and head CT datasets underscore the importance of effective modeling and demonstrate superior performance of HLIP over current foundation models (Zhang et al., 2023a; Nie al., 2025; Zhu et al., 2025; Yang et al., 2024). Experiments on curated chest CT datasets (Draelos et al., 2021; Hamamei et al., 2024) further isolate the effectiveness of the proposed hierarchical attention mechanism. Finally, we conduct comprehensive ablation and analysis.
4.1 Brain MRI
Pub-Brain-5. To the best of our knowledge, no publicly available benchmark exists for zero-shot transfer evaluation brain MRT tasks? To this end, we construct benchmark, named Pub- Pub-BBainn55, based on existing publicly available brain MRI datasets. Pub-Brain-5 comprises 18,343 studies drawn from Open- BHB (Dufumier et al., 2022), the Stroke dataset (Liu et al., 2023), BraTS 2023 (Baid et al., 2021; LaBella et al., 2023; Moawad et al., 2024; Kazerooni et al., 2024), NYU-Mets (Link et al., 2024), and UCSF-Mets (Rudie et al., 2024). It spans five classes: healthy (3,984), acute stroke (2,871), glioma (1,614), meningioma (1,141), and metastasis (8,733). For each study, the number of scans ranges from 1 to 14. We also construct a subset of Pub-Brain-5, namely Pub-Brain-Ba original segmentation ground truth. Pub-Brain-5-GT comprises 8,944 studies covering the same five classes: healthy (3,984), acute stroke (2,372), glioma (1,350), meningioma (1,000), and metastasis (238). We evaluate three zero-shot tasks: (i) binary anomaly detection (ie., distinguishing pathological from healthy studies); (ii) three-way tumor classification; and (iii) five-way disease classification.
which contains centan-con slice annotations derived from the
Baselines, We first evaluate the zero-shot transferability of two biomedical CLIP models on our benchmark: CIIM (Zhang et al., 2023a) and ConCeptCI.IP (Nie et al., 2025). Since both models require 2D inputs, we generate study predictions by applying average pooting to the outputs across all slices. We acknowledge that slices without lesions introduce noise for these 2D baselines; however, this is an intrinsic limitation of such models. We support these two baselines on Bab-Brain-S-GT, where predictions are made only on lesion-containing slices, while HLIP continues to take the raw study as input. In addition, we evaluate Prima (Lyu et al., 2026), which represents the equae + hierarchicul transformer architecture family (Hamanci et al., 2024; Lyu et al., 2026), as well as a vanilla ViT that encodes the entire study.
Implementation Details. We use the prompt TThss brain MRI shows: fdisease]." to perform zero-shot inference (Radford et al., 2021) with biomedical CLIP models. HLIP and the vanilla ViT are trained on BrainMRI220K as described in Section 3.2, while Prima (Lyu et al., 2026) is re-trained the same dataset. using the training recipe provided in the original paper. These three models employ the prompt "This MRI study shows: fdiscose. during zero-shot transfer.
BrainMD (Wang et al., 2024) is not publicly available due to an unexpected privacy palicy.
7

---

Published in Transactions on Machine Learning Research (02/2026)
Table Results zero-shot classification on Pub-Brain-6 and Pub-Brain-5-G7 We report balanced accuracy, with the best results highlighted in bold and the second-best result highlighted in underline. *annotationn indicates only predicting on lesion-containing slices which are manually annotated.
Anomaly Detection
Method
Tumor
Disease
Stroke
Glioma
Meningioma
Metastasis
menn
Pub-Brain-S
BioMedCLIP
64.7
87.8
63.6
59.8
69.0
50.4
31.5
(Zhang et al., 2023a)
ConCept
66.8
91.9
57.7
67.9
71.1
35.7
30.9
(Nie et al., 2025)
Prima
78.8
89.3
70.8
64.7
75.9
42.8
31.6
(Lyu al,, 2026)
VIT (our impl)
72.8
93.4
72.9
63.1
75.6
45.7
43.4
HLIP (ours)
91.5
89.2
79.2
78.1
84.5
63.3
63.9
Labb-Brain---GT
BiomedCLII
66.7
88.2
63.8
74.6
73.3
46.2
33.1
aamootatonn
86.4
94.1
75.8
82.8
45.7
45.3
ConCeptCLIP
69.6
92.1
57.8
69.5
72.3
35.2
31.6
tammotation
93.6
97.8
70.8
76.8
84.8
39.4
50.8
Prima
61.2
81.0
87.7
53.4
70.8
45.9
31.4
VIT (our impl)
76.7
93.5
58.2
58.2
71.7
42.1
43.5
HLIIP (ours)
95.0
89.2
79.6
73.4
84.3
54.8
61.3
Results. Given both benchmarks are imbalanced, we report balanced accuracy (ACC) in Table 1. On -rb-BBrin-5,, HLIP demonstrates superior performance over all baseline models, outperforming the second-best baseline by 20.5% ACC in disease classification. On Pub-Brain-5-GT, we observe a substantial performance boost for two biomedical CLIP models (Zhang et al., 2023a; Nie et al., 2025) when predictions are restricted to lesion-containingg slices, underscoring the fairness of our evaluation for these models even on uncurated studies. Notably, in disease classification, ConceptCLIP (Nie al., 2025) achieves 50.8% ACC, outperforming Prima (Lyu et al., 2026) and the vanilla VIT trained our large-scale domain-specific datasets, indicating that merely scaling up data is not sufficient solution for language-image pre-training in 3D medical imaging. On the other hand, HLIP achieves the new SOTA 61.3% ACC, attributable to the effectiveness of the proposed hierarchical attention mechanism.
HLIP can occasionally lag behind BiomedCLIP (Zhang et al., 2023a) and ConceptCLIP (Nie et al., 2025), particularly in the detection of brain glioma. However, we note that datasets like BraTS (Baid et al., 2021) are likely included in their pre-training corpora, as both models are trained on figure-caption pairs from PubMed. HLIP, the other hand, does not encounter any instances from public datasets during training. The skull-strippin used in BraTS actually exacerbates the distribution shift. Moreover, HLIP makes predictions directly from the original radiology study, offering greater flexibility for real-world applications.
4.2 Head CT
Implementation Details. We apply HLIP to HeadCT240K, as described in Section 3.2. For lincar-probe evaluation, we benchmark HLIP on CQ500 (Chilamkurthy et al., 2018) and RSNA (Flanders et al., 2020), exactly following the train-tee test split and hyperparameters of FM-HeadCT (Zhu et al., 2025). For zero-shot evaluation, we evaluate HLIP on the full CQ500 and RSNA datasets and compare it with the vanilla VIT also trained on the same HeadCT240K "ataset We employ the prompt "This CT study shows: {disease)." during zero-shot transfer.
Reeeent CLIP models for 3D medical imaging, such as Merlin (Blankemeier al., 2024) M3D (Bai et al., 2024), excluded from baselines, as they perform no better than random guessing in aero-shat evaluation on CQ500 and RSNA.
8

---

Published in Transactions on Machine Learning Research (02/2020)
CQ500
CQ500 Zere-Shot
RSNA
Henerraag
RSVA
Lincur-Probe
Mase Fdertt
0030
Mereryaaaaa Sreaa ()
Maaie Elleet
Linear-Prod
Zoro-Shot
91
(tres
Moddle Dhaas
K1
TO
811
30
15
60
54
59
200 empai pink
78
06
20
80
GenglCT PALScCECT HLIPoero
(SDI))
(EDRE
GoogleCT
Sabdural
ERLEP pwart
Figure 3: Results of linear-probe and zero-shot evaluations on the CQ500 (Chilamkurthy et al., 2018) and RSNA (Flanders et al., 2020) datasets. We report AUC for each class. In linear-probe, red represents Google CT (Yang al., 2024); green represents FM-HeadCT (Zhu al., 2025); and blue represents our HLIP. In zero-shot evaluation, red represents the vanilla. ViT and blue represents our HLIP.
Results. Consistent with FM-HeadCT (Zhu et al., 2025), we report the area under the ROC curve (AUC) for each class in Figure 3. In linear-probe evaluation, HLIP demonstrates superior performance over FM- HeadCT (Zhu et al., 2025), which is pre-trained on 361,663 studies using DINOv2 (Oquab et al., 2023), and the Google-CT (Yang et al., 2024) foundation model. Specifically, on the CQ500 dataset (Chilamkurthy et al., 2018), HLIP outperforms both FM-HeadCTT and Google-CT with macro AUC improvements of 8.3% and 12.2%, respectively; on the RSNA dataset (Flanders al., 2020), HLIP achieves macro AUC improvements of 1.7% and 5.8% over the these baselines, respectively. Outperforming current foundation models demonstrates the clinical significance of HLIP. In zero-shot evaluation, HLIP consistently outperforms the vanilla ViT, achieving macro AUC improvements of 10.0% on CQ500 and 2.4% on RSNA, further demonstrating the effectiveness of the proposed hierarchical attention mechanism.
4.3
Chest CT
Implementation Details. Following CT-CLIP (Hamamci et al., 2024) and fVLM (Shui et al., 2025), we apply HLIP to the CT-RATE (Hamamci et al., 2024) training set, then perform internal validation on its test split and external validation on the full Rad-ChestCT dataset (Draclos et al., 2021). The implementation details differ from those Section 3.2. Following prior works (Cao el al., 2024; Shui el al., 2025), we standardize each scan to a spacing of (3mm, 1mm, 1mm), truncate HU values to [-1150, 350], construct mini- batches with center crop of size (112, 336, 336), and apply token size of (8, 24,24,2). We also use (Boecking et al., 2022) as the text encoder. Other than the aforementioned modifications, all remaining settings are exactly the same as those described in Section 3.2. HLIIP is trained for 20 epochs without patch dropout. Using a batch size of 512 on 4 A40 GPUs, training completes within 6 hours, significantly faster than prior works (Hamamei et al., 2024; Shui et al., 2025), which require several days even with more advanced computational resources. More details are provided in Appendix B.1.
Results. Consistent with IVLM (Shui et al., 2025), we report the area under ROC curve (AUC); balanced accuracy (ACC); weighted Fi-score (F1); recall; and precision for multi-label zero-shot classification in Table 2. When pre-trained with original reports, HLIP outperforms second-best models by 4.9% AUC internal and 7.9% AUC on external validation. When pre-trained with reports summarized by large language models (c.g., Qwen (Bai et al., 2023), as used by IVLM), HLIP outperforms IVLM by 0.9% and 3.7% AUC on internal and external validation, respectively. These results demonstrate that, compared with other components such as the VAE (Van Den Oord et al., 2017) tokenizer (used by CT-CLIP) and anatomical-guided fine-grained alignment (used by FVLM), the proposed hierarchical attention more effective and generalizable adaptation for language-image pre-training in 3D medical imaging.
9

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Results multi-label zero-shot evaluation on the CT-RATE (Hamamci et al., 2024) test split for internal validation and the Rad-ChestC7 (Draclos et al., 2021) for external validation. The best results are highlighted in bold.
Internal Validation (CT-RATE)
External Validation (Rad-ChestCTT
Method
AUC
ACC
F1
Precision
Recall
AUC
ACC
F1
Precision
Recall
Supervised by Original Reports
CT-CLIP
73.3
66.9
70.8
32.6
63.3
59.9
64.7
34.1
(Hamamci al., 2024)
BIUD (Cao et al., 2024)
71.3
68.1
71.6
33.8
67.3
62.9
60.6
65.2
33.7
59.6
Merlin
72.8
67.2
70.9
33.7
70.1
64.4
61.9
66.3
34.8
61.0
(Blankemeier et al., 2024)
HLIP (ours)
77.7
71.4
74.7
37.9
73.0
72.3
68.4
72.1
40.4
66.7
Supervised by Qwen (Bai al,, 2023) Summarized Reports
IVLM (Shui et al,, 2025)
77.8
71.8
75.1
37.9
72.8
68.0
64.7
68.8
37.4
64.6
HLIP (ours)
78.7
72.4
75.5
38.4
74.1
71.7
67.7
71.4
39.8
66.9
65
65
ACC (%
55
55
55
60
45
45
45
55
none
avg
learned
first 4
evenly last 4
6
12
64
128
256
(a)
als token prop. strategy
(b) study attn. location
(c) # study attn. layers
(d) batch size
69.00
68.5
69.0
68.5
68.00
66.0
ACC
67.5
67.5
8 67.5
67.0
66.0
66.5
67.0
66.0
none
avg
learned
first eventy last
2
12
128
256
512
(e) cls token prop. strategy
(f) scan attn. location
(g) # scan attn. layers
(h) batch size
Figure 4: Ablation study on the fub-Brain-5 (a)-(d) and Tad-ChestCTT (Draelos et al., 2021) dataset (e)-(h).
We report balanced accuracy (ACC) on both datasets.
4.4 Ablation Study
In Figure 4, we present an ablation study on the key components of our visual encoder, including the cls token propagating strategy (Figure 4a and de); the location of dense allention (Figure 4b and 4f); and the number of dense attention layers (Figure 4e and 4c). We additionaly investigate the batch size in Figure 4d and 4h, factor overlooked in previous work (Hamamci et al., 2024; Shui et al., 2025).
CLS token. When propagating the els loken across different atlention levels, in addition to the default averaging setting, we also consider two alternative scenarios: (i) global average pooling at the end of the visual encoder (none); and (ii) applying weighted averaging using an attention pooling (learned). As shown in Figures 4a and 4e, aggregating the cls token by averaging proves to be the most effective approach. This result demonstrates that more complex components not necessary to achieve better performance.
10

---

Published in Transactions on Machine Learning Research (02/2020)
Chest CT
Brain MRI
Pericardial effusion
Pulmonary fibrosis
Glioma
Meningioma
Z
Z+8
Z
Z+8
FLAIR
Tiotcontss
FLAIR
TiWteontrast
Clinical scan
Activation
map
Figure 5: Qualitative results of zero-shot diagnosis on the Rad-Chest CTTt- (Draelos et al., 2021) and BraTS (Baid et al., 2021; LaBella et al., 2023) datasets. The first row shows the original clinical scans with pathologic regions outlined (dashed red). The second row shows activation maps (Chefer et al., 2021). HLIP identifies the pathologic regions across multiple groups of adjacent chest CT slides (left) and brain MRI scans (right).
Location of dense attention layers. In Figure 4b and 4f, we consider alternative locations of the global attention layer (i.e., study attention for uncurated datasets and scan attention for curated datasets). Compared with distributions such as placing all global attention layers at the front or all at the end, we find that evenly distributed global attention layers yield the best performance. This is because they enable the model to capture global information at multiple feature levels.
Number of dense attention layers. In Figure 4e and 4e, we investigate the number of global attention layers, where 0 and 12 correspond to two implementations of vanilla VIT. We find that 4 global attention layers are sufficient to capture global information, outperforming both 0 layers, which lack global information, and 12 layers, which lack the constructive priors provided by slice or scan attention.
Batch size. Previous work (Hamamei et al., 2024; Shui et al., 2025) used only small batch sizes (e.g-, 48) during the pre-training, In Figure 4d and 4h, we investigate this factor, which has been widely recognized important in language-image pre-training, in the context of 3D medical imaging. We find that language-image pre-training in 3D medical imaging still benefits from larger batch sizes, underscoring the importance of memory-friendly designs such as our hierarchical attention mechanism.
4.5 Visualizations
In Figure 5, we visualize the activation maps of HLIP, following Chefer et al. (2021) in the zero-shot sctting on Rad-Chest.CT (Draelos et al., 2021) and BraTS (Baid et al., 2021; La LaBella el al., 2023). Although HLIP mainly employs lightweight local attention layers, these visualization results demonstrate the effectiveness of few global attention layers in enabling HLIP to attend to visual features globally. Specifically, for chest CT (curated, single scan), HLIP attends to pathologic regions across different slices, e.g., slice Z and Z+8, by leveraging scan attention. For brain MRI (uncurated, multiple scans), HL.IP attends to pathologic regions across different scan types, e.g,, FLAIR and tcottaat,,,, by leveraging study attention. This demonstrates the offectiveness HLIP in modcling the hicrarchical structure of 3D medical imaging.
4.6 Clinical Translation of HLIP
93.5
90
We conduct 1-year, health system-scale, compre- hensive evaluation
AUC (%)
93.0
prospective set of neuro- logical studies. The datascts include -23K brain MRI studies covering 52 diagnoses and ~15K head CT studies covering 83 diagnoses. We report macro AUC in Figure 6a and 6b. HLIP consistently outper- forms VIT. Results for each diagnosis are presented in Figures 11 and 12 in Appendix C.
89
92.5
92.0
88
ViT
HLIP
VIT
HLIP
(a) 52 brain MRI diagnoses
(b) 83 head CT diagnoses
Figure 6: Prospective testing results on (a) 52 brain MRI diagnoses and (b) 83 head CT diagnoses.
11

---

Published in Transactions on Machine Learning Research (02/2026)
5 Conclusion
In this work, we pioncer language-inage pre-training directly on uncurated 3D medical imaging studies containing multiple scans, together with a novel hierarchical attention mechanism that effectively extracts visual features. These two contributions form HLIP, scalable and effective pre-training framework for 3D medical imaging. Despite its simplicity in both concept and practice, HLIP achieves state-of-the-art performance on multiple benchmarks across diverse modalities and anatomical regions, including brain MRI, head CT, and chest. CT. These results demonstrate that directly pre-training on uncurated clinical datasets is scalable and effective paradigm for language-image pre-training in 3D medical imaging. We hope work facilitates large-scale language-image pre-training in other health systems and inspires future machine learning research on scalable approaches for real-world clinical data.
12

---

Published in Transactions on Machine Learning Research (02/2026)
References
Alok Aggarwal and S Vitter, Jeffrey. The input/output complexity of sorting and related problems. Commu- nications of the ACM, (10):1111611127 1988.
Fan Bai, Yuxin Du, Tiejun Huang, Max Q-H Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models. Xiv preprinl arXiv:2404.000 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXim:2309.16000 2023.
Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Cramer, Felipe C Kitamura, Sarthak Pati, et al. The rsna-asnr-miccal brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 2021.
Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos. Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific data, 4(1):1-13, 2017.
Shruthi Banmur, Stephanie Hyland, Qianchu Liu, Fernando Perez-Garcia, Maximilian Ilse, Daniel C Castro, Benedikt Boccking, Harshita Sharma, Kenza Bouzid, Anja Thieme, et al. Learning to exploit temporal structure for biomedical vision-language processing. In Proceedings of the IEEE/CVE Conference Computer Vision and Paltern Recognition, pp. 15016-15027, 2023.
Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João Madeira Araújo, Oleksandr Vitvitskyi, Razvan Pascamu, and Petar Veličković. Transformers need glasses! information over-squashing in language tasks. Advances in Neural Information Processing Systems, 37:98111-98142, 2024.
Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyls, et al. Merlin: A vision language foundation model for 3d computed tomography. arXiv preprint arXiv:2406.0653 2024.
Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Valaeez-Valle, et al. Making the most of text semantics to improve biomedical vision language processing. In European conference on computer vision, pp. 1-21. Springer, 2022.
Weiwei Cao, Jianpeng Zhang, Yingda Xia, Tony CW Mok, Zi Li, Xianghua Ye, Le Lu, Jian Zheng, Yuxing Tang, and Ling Zhang. Bootstrapping chest et image understanding by distilling knowledge from x-ray expert models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11238 11247, 2024.
M Jorge Cardoso, Wenqi Li, Richard Brown, Nic Ma, Eric Kerfoot, Yiheng Wang, Benjamin Murrey, Andriy Myronenko, Can Zhao, Dong Yang, et al. Monai: An open-source framework for deep learning in healthcare. XrXiv preprint arXiv:2211.00 1.021011011,,, 2022.
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 397-406, 2021.
Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEFE/CVF conference computer vision and pattern recognition, pp. 2818 2829, 2023.
Sasank Chilamkuthy, Rohit Ghosh, Swetha Tanamala, Mustafa Biviji, Norbert G Campeau, Vasantha Kumar Venugopal, Vidur Mahajan, Pooja Rao, and Prashant Warier. Deep learning algorithms for detection of critical findings in head et scans: retrospective study. The Lancel, 392(10112) 2396, 2018.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-effciient exact attention with io-awareness. Advances in neural informalion, processing systems, 35:16344-16338 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold, S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Rachel Lea Draelos, David Dov, Maciej A Mazurowski, Joseph Y Lo. Ricardo Henao, Geoffrey D Rubin, and Lawrence Carin. Seahhine-lerning-basseee multiple abnormality prediction with large-scale chest computed tomography volumes. Medical image analysis, 67:101857, 2021.
Benoit Dufumier, Antoine Grigis, Julie Victor, Corentin Ambroise, Vincent Frouin, and Edouard Duchesnay. Openbhb: a large-scale multi-site brain mri data-set for age prediction and debiasing, Newrolmage, 263: 119637, 2022.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6824-6835, 2021.
Adam E Flanders, Luciano M Prevedello, George Shih, Safwan S Halabi, Jayashree Kalpathy- Robyn Ball, John T Mongan, Anouk Stein, Felipe C Kitamura, Matthew P Lungren, et al. Construction of a machine learning dataset through collaboration: the rsna 2019 brain et hemorrhage challenge. Radiology: Artificial Intelligence, 293)211,,, 2020.
Yu Gu, Robert Timn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing, 2020.
Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Omer Faruk Durugol, Bastian Wittmann, Tamaz Amiranashvili, et al. Developing generalist. foundation models from multimodal datasel for 3d computed lomography. arXiv preprint Xix::2400. 1 7834, 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference computer vision and pottern recognition, pp. 16000 16009, 2022.
Xiaoxuan He, Yifan Yang, Xinyang Jiang, Xufang Luo, Haoji Hu, Siyun Zhao, Dongsheng Li, Yuqing Yang, and Lili Qiu. Unified medical image pro-training in language-guides common semantic space. In European Conference on Computer Vision, pp. 123-12 139. Springer, 2024.
Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3942-3951, 2021.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Neus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590 597, 2019.
Fabian Isensee, Paul Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: a self-configurig method for deep learning -bsed biomedical image segmentation. Nature methods, 18(2): 203-211, 2021.
Alistair EW Johnson, Tom Pollard, Scth Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger Mark, and Steven Horng. Mimic-cxr, do-identified publicly available database of chest radiographs with free-text reports. Scientific dala, 6(1):317, 2019.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Samuel Joutard, Maximilian Pictsch, and Raphael Prevost. Hyperspace: Hypernetworks for spacing-adaptive image segmentation. In International Conference on Medical Image Compuling and Compuler-Assin Assisted Intervention, pp. 339 349. Springer, 2024.
Anahita Fathi Kazerooni, Nastaran Khalili, Xinyang Liu, Debanjan Haldar, Zhifan Jiang, Syed Muhammed Anwar, Jake Albrocht, Maruf Adewole, Udunna Anazodo, Hannah Anderson, et al. The brain tumor segmentation (brats) challenge 2023: focus on pediatrics (obu--onneelldig----------ee ArXiv, pp. MeXiv-2305) 2024.
brats-peds).
Dominic LaBella, Maruf Adewole, Michelle Baid, Timothy Bergquist, Radhika Bhalerao, Sully Chen, Verena Chung, Gian-Marco Conte, Farouk Dako, James Eddy, Ivan Ezhov, Devon Godfrey, Fathi Hilal, Ariana Familiar, Keyvan Farahani, Juan Eugenio Iglesias, Zhifan Jiang, Elaine Johanson, Anahita Fathi Kazerooni, Collin Kent, John Kirkpatrick, Florian Kofler, Koen Van Leemput, Hongwei Bran Li, Xinyang Liu, Aria Mahtabfar, Shan McBurney- McLean, Zeke Meier, Ahmed W Moawad, John Mongan, Pierre Nedelec, Maxence Pajot, Marie Pirand, Arif Rashid, Zachary Reitman, Russell Takeshi Shinohara, Yury Velichko, Chunhao Wang, Pranav Warman, Walter Wiggins, Mariam Aboian, Jake Albrecht, Udunna Anazodo, Spyridon Bakas, Adam Flanders, Anastasia Janas, Goldey Khanna, Marius George Linguraru, Bjoern Menze, Ayman Nada, Andreas M Rauschocker, Jeff Rudie, Nourel Hoda Tahon, Javier Villameva-Meyyrr, Benedikt Wiestler, and Evan Calabrese. The asnr-miccai brain tumor segmentation (brats) challenge 2023: Intracranial meningioma, 2023.
Talissa Altes, Syed Muhammad Anwar, Uijwal
Ryan
Haoran Lai, Zihang Jiang, Qingsong Yao, Rongsheng Wang, Zhiyang He, Xiaodong Tao, Wei Wei, Weifu Lv, and S Kevin Zhou. Bridged semantic alignment for zero-shot 3d medical image diagnosis. arXiv preprint wrXiv:2501.0366, 2025.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pp. 280-296. Springer, 2022a.
Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4804-4814 2022b.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-iman -maage pre-training via masking. In Proceedings of the IEEE/CVE conference on computer vision and patlern recognition, pp. 23390 23400, 2023.
Katherine E Link, Zane Schmurman, Chris Liu, Young Joon Kwon, Lavender Yao Jiang, Mustafa Nasir-Moin, Sean Neifert, Juan Diego Alzate, Kenneth Bernstein, Tanxia Qu, et al. Longitudinal deep neural networks for assessing metastatic brain cancer on large open benchmark. Nature Communicctionns, 15(1):8170, 2024.
Chin-Fu Liu, Richard Leigh, Brenda Johnson, Victor Urrutia, Johnny Hsu, Xin Xu, Xin Li, Susumu Mori, Argye E Hillis, and Andreia V Faria. A large public dataset of annotated clinical mris and metadata of patients with acute stroke. Scientific Data 10(1):548, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012 10022, 2021.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint
ar www:::666.000000 2016.
Tya Loshchilov and Frank Hutter. Decoupled woight docay regularization. arXi preprint arXie:1111.05100 2017.
Yiwei Lyu, Samir Harake, Asadur Chowdury, Soumyanil Banerjec, Rachel Gologorsky, Shixuan Liu, Anna- Katharina Meissner, Akshay Rao, Chenhui Zhao, Akhil Kondepudi, et al. Learning neuroimaging models from health system-scale data. Nature Biomedical Engineering, pp. 1-13, 2026.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Noture Communicaations, 15(1):654, 2024.
Christos Matsoukas, Johan Fredin Haslum, Moein Sorkhei, Magnus Söderberg, and Kevin Smith. What makes transfer learning work for medical images: Feature reuse & other factors. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9225 9234, 2022.
Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy- Crameer Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE transactios on medical imaging, 34(10):1993 2024, 2014.
Ahmed W Moawad, Anastasia Janas, Uijwal Baid, Divya Ramakrishnan, Rachit Saluja, Nader Ashraf, Nazanin Maleki, Leon Jekel, Nikolay Yordanov, Pascal Fehringer, et al. The brain tumor segmentation- metastases (brats-mets) challenge 2023: Brain metastasis segmentation on pre-treatment mri. ArXiv, pp. 2024.
Philip Mullrr, Georgios Kaissis, Congyu Zou, and Daniel Rueckert. Joint learning of localized representations from medical images and reports. In European conference on computer vision, pp. 685 701. Springer, 2022.
Yuxiang Nie, Sunan He, Yequan Bie, Yihui Wang, Zhixun Chen, Shu Yang, and Hao Chen. Conceptelip: Towards trustworthy medical ai via concept-enhanced contrastive langange-image pre-training, arXiv preprint arXie:2001. 15579, 2025.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Danicl Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07198, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PmLR, 2021.
Jeffrey D Rudie, Rachit Saluja, David A Weiss, Pierre Nedelec, Evan Calabrese, John B Colby, Benjamin Laguna, John Mongan, Steve Braunstein, Christopher P Hess, et al. The university of california san francisco brain metastascs stercotactic radiosurgery (uesf-bmsr) mri datasct. Radiology: Artificial Intelligence, 6(2): e230126, 2024.
Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Holfman, et al. Hiera: A hierarchical vision transformer without the 2023.
In International conference on machine learning, pp.
PMLR,
Zhongyi Shui, Jianpeng Zhang, Weiwei Cao, Sinuo Wang, Ruizhe Guo, Le Lu, Lin Yang. Xianghua Ye, Tingbo Liang, Qi Zhang, et al. Large-scale and fine-grained vision-language pre-training for enhanced ct image understanding. arXiv preprint arXiv:2501.1455 14548, 2025.
Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar. Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature biomedical engineering, ((12):1399-1400) 2022.
Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No' zero-shot" " without exponential data: Pretraining concept frequency determines multimodal model performance. In The eiigtt- Annual Conference on Neural Information Processing Systems, 2024.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances neural information processing systems, 30, 2017.
Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti, and Loquan Yu. Multi-gramularity cross- modal alignment for generalized medical visual representation learning. Advances in Neural Information Processing Systems, 03333366335444 2022a.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Yuli Wang, Yuwei Dai, Craig Jones, Haris Sair, Jinglai Shen, Nicolas Loizou, Wen-Chi Hsu, Maliha Imami, Zhicheng Jiao, Paul Zhang, et al. Enhancing vision-language models for medical imaging: bridging the 3d gap with innovative slice selection. Advances in Neural Information Processing Systems, 37:99947-99998 2024.
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medelip: Contrastive learning from unpaired medical images and lext. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2022, pp. 3876, 2022b.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21372- 213383, 2023.
Lin Yang, Shawn Xu, Andrew Sellergren, Timo Kohlberger, Yuchen Zhou, Ira Klena, Atilla Kiraly, Faruk Ahmed, Farhad Hormozdiari, Tiam Jaroensri, et al. Advancing multimodal medical capabilities of gemini. arXiv preprint ar 4rx: 2024.
Kihyun You, Jawook Gu, Jiyeon Ham, Beomhee Park, Jiho Kim, Eun K Hong, Woonhyuk Back, and Byungseok Roh. Cxr-clip: Toward large scale chest x-ray language-image pre-training. In International Conference Medical Image Computing and Computer-Assisten Intervention, pp. 101-111. Springer, 2023.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jasprect Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint XXi:23333.00915 2023a.
Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhanced visual-language pre-training on chest radiology images. Nature Communications, 14(1):4542, 2023b.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive lcarning of medical visual representations from paired images and text. In Machine learning for healthcare conference, pp. 2-25. PMLR, 2022a.
Yuhui Zhang, Shih-Cheng Huang, Zhengping Zhou, Matthew P Lungren, and Serena Yeung. Adapting pre-trained vision transformers from 2d to 3d through weight inflation improves medical image segmentation. In Machine Learning for Health, pp. 391-404. PMLR, 2022b.
Chenhui Zhao and Liyue Shen. Part-aware personalized segment anything model for patient-specific segmen- (ation. arXiv preprint ar Xiv:24403 05433, 2024.
Weicheng Zhu, Haoxu Huang, Huanze Tang, Rushabh Musthyala, Boyang Yu, Long Chen, Emilio Vega, Thomas O'Donnell, Seena Dehkharghani, Jennifer Frontera, et al. 3d foundation model for generalizable disease detection in head computed tomography. ar preprint or Xiv 02779, 2025.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Appendix
A: Dataset
B: Implementation
C: Prospective Evaluation
D: ical Analysis
Complexity
F: Discussion
A Dataset
As mentioned in Section 3.2, this section provides additional details on the two datasets we collected: MRT220K and HeadCT240K. For other datasets, such as CT-RATE (Hamamci et al., 2024), Rad- ChestCT (Draclos et al., 2021), BraTS 2023 (Menze et al., 2014; Bakas al., 2017; Baid al., 2021; LaBella et al., 2023; Moawad et al., 2024; Kazerooni al., 2024), NYU-Mets (Link et al., 2024), UCSF-Mets (Rudie et al., 2024), RSNA (Flanders et al., 2020), and CQ500 (Chilamkurthy et al., 2018), refer readers to their original publications. Additionally, we include discussion of our preproccssing strategy.
A.1 BrainMRI220K
$0,000
scans miltin total)
Studies (220K in total)
70,000
600,000
scans (million)
1.75
500,000
50,000
40,000
300,000
30,000
20,000
200,000
100,000
50
125
150
200
Addal
Conoral
Sugita
# scars per study (binned by
slices per scan (binned by 5)
scans per acquisition plane
(a) # scams per study
(b) slices per scan
(c) # scams per acqusition plane
Figure 7: Statistic results on BrainMRR220K.
10a
104
studies
10a
10a
None
Cyst
Oendly Weller Hellom allen
Dysienessts Corpus Callouum
Hattrctapla
Brain Absscets ets Inoophaliiis
Multiple Scienois
Joursraccoo i
Genngghrynggee
Mtulltary Aderoms
Syrinx-
Brain Hemmitioon
Cephollcceles
Curdbral Atrophy
Malfommtioo
Edema
Encephalomat mappalommaiii
Masa Mass Effect
Midine Shift
catheter
Cramictomy
Resection Cavlly
Brain Cantusion
Dmuuee Noonal Imtury Intury
Subsurss Hemetoms
Glooma
High Gade Gloma
low Since Bloms
tomptomn
Maninglomaa
Proaa Tumor
schwannrom
trainstem Gloms
Ependdymmn
Germ Call Tumar
Plecyttc Astrorytoins
intorral carial memarrhage
Intra intraeezzeuut for Homormaa
cubarachheld hemorrahage
Letge Vessel Stoke-
Boyamoys
artertovenous Diseaa
Wossel Diosast
Anouryam
Intoorarial Hypderssion
Vertriculoom eeccoorrepp egply
Arccinaid
Raihbas Claft
Small Smrll
Diagnosis
Figure 8: Diagnosis distribution on BrainMRT220K..
18

---

Published in Transactions on Machine Learning Research (02/2026)
The distributions of the number of scans per study, slices per scan, and scans per acquisition view in the BrainMR1220K dataset are shown in Figure 7. Specifically, the number of scans per study ranges from 1 to 162, with the third quartile at 17. In total, there are 3.13 million scans from 220,001 MRI studies. All scans are acquired with through-plane spacing equal to or smaller than 4mm. The number of slices per scan ranges from 5 500, with the third quartile al 80. Each sean is originally acquired from different view, with the ratio between axial, coronal, and sagittal views being ::1:2. Morcover, we show the distribution of 52 brain MRI diagnoses in Figure 8, based on keyword statistics extracted from the reports.
A.2 HeadCT240K
scans (1. 1. million total)
studies 1240K tetal)
0.8
100,000
scans (million)
120,000
80,000
0.6
100, 100,000
80,000
60.000
40,000
20,000
20
30
50
100
2000
300
Aabl
Corcnal
Sagital
scans per study (binned by 5)
# slices per scan (binned by 5)
scans per acquisition plane
(a) # scams per study
(b) slices per scan
(c) # scams per acqusilion plane
Figure 9: Statistic results on HeadCT240K.
10a
104
10a
102
aree Ein
SUSEY (S eiqie
ainool linn 70r
Alinn Cecoo musus
Sino
122 th 12
Ueit plo brouuhh (ouquas
sol Mills 200 spal peau
you Shlarrmaae jeaquaa
theass
poeunn pereee ageuren
injers
seenns (eass muss
lunvee
Diagnosis
Figure 10: Diagnosis distribution on HeadCT240K.
The distributions of the number of scans per study, slices per scan, and scans per acquisition view in the HcadCT240K dataset are shown in Figure 9. Specifically, the number of scans per study ranges from 1 to 71, with the third quartile at 6. In total, there are 1.44 million scans from 244,253 CT studies. All seans are acquired with through-plane spacing equal to or smaller than 4mm. The number of slices per scan ranges from to 500, with the third quartile at 110. Each scan is originally acquired from a different view, with the ratio between axial, coronal, and sagittal views being 8:3:3. Moreover, we show the distribution of 83 head CT diagnoses in Figure 10, based on keyword statistics extracted from the reports.
19

---

Published in Transactions on Machine Learning Research (02/2026)
A.3 Discussion
To the best of our knowledge, this is the first work to handle 3D medical datasets at the scale of Brain- MRI220K and HcadCT240K, which comprise millions of scans. Our preprocessing strategy, introduced in Section 3.2, follows standard practices where possible, but also diverges due to the scale and diversity of our dataset. Specifically, percentile clipping for brain MRI and HU value truncation for head CT follow standard practices (Ma et al., 2024; Zhu et al., 2025). For head CT, we do not concatenate scans with different HU truncations along the channel dimension, as our method is capable of processing multiple scans simultaneously. However, our strategy regarding orientation and spacing may differ from established preprocessing pipelines (Isensee et al., 2021; Blankemeier et al., 2024; Hamamci et al., 2024; Shui et al., 2025).
Orientation. CT and MRI scans are acquired using different planes, including axial, coronal, and sagittal. Isotropic data are rare in routine radiology studies; the through-plane spacing is typically larger than the in-plane spacing. In the context of batch construction, where all samples in batch must share the same shape, standardizing the orientation can lead to downsampling along the in-plane axes and upsampling along the through-plane axis. Upsampling in pixel space does not introduce new information, whereas downsampling results in information loss. From the perspective of data augmentation, the orientation can change when 3D random rotation is applied (Isensee et al., 2021). So, why do we slandardize the orientation at the beginning then augment during training? Instead, we consistently align the first dimension (depth) with the through-plane axis. Please note that the in-plane orientation is still standardized as RP, RI, or PI. We treat the diversity of acquisition planes (as shown in Figure 7c and Figure 9c) in our dataset as form of natural data augmentation.
Spacing. Standardizing spacing across the entire dataset is a common practice in medical image segmentation to ensure that convolution filters interpret anatomical structures consistently (Isensee et al., 2021). However, HyperSpace (Joutard et al., 2024) demonstrates that, with appropriate spacing augmentation, model can learn spacing-invariant features with sufficient spacing augmentation. Given the diversity of spacing settings (as shown in Figure 7b and Figure 9b) in our dataset, we believe naturally occupies sufficient spacing augmentation for the model to learn ppacing-invarian first resize each volume to 246, 256), and then apply center crop to obtain (48,224, 224). Here, 48 represents the median nunber of slices, which does not substantially distort the original scan.
features. Therefore, when constructing batch, we
M scans per study. As the number of scans per study varies, we randomly sample M=10 scans per study at each training step to construct a batch. An equal number of positional embeddings also sampled for these M scans from a total of Mmax= positional embeddings at cach step. This means the model is exposod to Mmax different positions over the course of training, while observing M positions at each step. This is analogous to dropout strategy, with a scan dropout rate of 1 M_ During evaluation, the model is able to process studies containing up to Maax scans without the need for interpolation.
B Implementation
In this section, we summarize the model configuration described in Section 3.2 and provide details of the pre-training setup referenced in both Section 3.2 and Section 4.3, For linear probing evaluation head CT, we refer readers to FM-HeadCT (Zhu et al., 2025).
B.1 Chest CT
The model architecture and pre-training configuration for chest CT are summarized in Table 3 and Table 4, respectively. These configurations closely follows prior works such as CT-CLIP (Hamamci et al., 2024), OpenCLIP (Cherti et al., 2023), and BiomedCLIP (Zhang et al., 2023a). In the pre-training configuration, we report the base learning rate corresponding to a batch size of 64. During training, we follow the linear learning rate scaling rule (Li et al., 2023): Ir = base_lt batch size
64
The discases used for zero-shot evaluation are listed in Table 5. This setup is consistent with fVLM (Shui et al., 2025) and all methods reported in Table 2.
20

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Model (Chest CT)
Table 4: Pre-training (Chest CT)
config
value
config
value
image precision
float32
projection
linear 512
window mean, std report
[-1150,350]
embed dim
[0.449. 0, 0.26] original
Visual Encoder slice attn index scan attn index imput size
ViT-B (Ronowntelly at,, 2020)
(0,1;3, :;1;3,4;67;;9,00 6,,5,8, 11) (112, 336, 336) (8, 24, 24)
optimizer 31, 32 base Ir weight decay Ir schedule warmup steps numerical precision
AdamW (Loshchtlov & Hutter, 2017)
0.9, 0.98 (Radford es ol., 2021)
1e-5
patch size
palch dropout
0.0
cosine (Loskschilov Butter, 2016)
Text Encoder
ARRBERTT (Boocking et ol, 2023)
47 (Shai et ai., 20263
context length
512 (faamamel al,, 2024)
amp
Table 5: Evaluation (Chest CT)
CT-RATE (ffammmett ah,, 2024)
Rad-ChestCT (Drselos 2021)
Emphysema
emphysema
Atelectasis
atelectasis
Lung nodule
nodule
Lung opacity
opacity
Pulmonary fibrotic sequela
fibrosis
Pleural effusion
pleural_effusic
Mosaic attenuation pattern
Peribronchial thickening
bronchial __all_ wwwl_whwileeeeeeeeaeaaaaaaaeeece thickening
Consolidation Bronchiectasis
consolidation
bronchiectasis
Interlobular septal thickening
poptaltthickning
Cardiomegaly
cardiomegaly
Pericardial effusion
pericardial_effie _ffuisiin
Coronary artery wall calcification
calcification
Hiatal hernia
hernia
Arterial wall caleification
calcification
B.2
Brain MRI/Head CT
Table 6: Model (Brain MRI/Head CT)
Table 7: Pro-training (Brain MRI/Heat CT)
config
value
config projection
value
image precision truncate mean, std report.
uint8
linear 512
(0.05,0.95)
embed dim
[0.449, 226]]
Visual Encoder scan attn index study attn index
VIT-B (Dusovitally 2020)
Prima (Lora al., 2026)
(0,1;3,4;6 4;1;3,;;6,;;,,00 (2,5, 8, 11) (48, 224, 224) (8, 16, 16)
optimizer 31, 32 base weight decay Ir schedule warmup steps numerical precision
AdamW (Loshchillo & Hutter, 2017)
0.9, 0.95 (ti et 2023)
input size
1e4 0.2
patch size
patch dropout
0.25
cosine (Loshhcldto & Horter, 2016)
Text Encoder
PubMedBERT (Gu al., 2020)
2000
context length
256 (Zhang et al,, 20235))
amp
The model architecture and pre-training configuration for brain MRI/hcad CT are summarized in Table 6 and Table 7, respectively. We process the reports using the method proposed in Prima (Lyu el al., 2026), which leverages a large language model to summarize the original radiologist reports and reduce biases related to writing style, grammar, and wording habits. Given the rapid development of modern language models, we do not consider this fully automatic text processing applied existing radiologist reports be equivalent the data curation pipeline the visual side, which relies radiologists and imposes additional burdens.
21

---

Published in Transactions on Machine Learning Research (02/2026)
C
Prospective Evaluation
100
100
1
Mithe Mton
Hary
Cricel
30
10
000
100
VIT (mAUCC - 92.51)
(mADC 93.24)
Figure 11: Prospective evaluation on 52 brain MRI diagnoses.
Figure 12: Prospective evaluation on 83 head CT diagnoses.
We present the detailed results of the prospective evaluation conducted within our health system for 52 brain MRI diagnoses and 83 head CT diagnoses in Figure 11 and Figure 12, respectively. We report linear-probe AUC for HLIP (bluc) and the vanilla VIT (red). HLIP demonstrate consistent improvement.
22

---

Published in Transactions on Machine Learning Research (02/2026)
D Statistical Analysis
We investigate statistical metrics for HLIP and the original VIT on the Pub-Brrin-5 dataset. First, using the zero-shot prompt reported in the paper, "This brain MRI shows: fdisease]"", we find that the p-value between the two models less than 0.0001. We then perform inference using four additional prompis: 'This brain MRI shows: likely {disease).', TTis brain MRI shows: {disease} identified.", This broin MRI shows: suggesting {diseaseJ.', and "This brain MRI shows: compotible with {disease]." Under the live prompts considered, the 5-way disease classification performance is 63.4_3.8 for HLIP and 40.6+1.0 for ViT. This result demonstrates clear benefit of the HLIP methodology over the original ViT.
Table 8: Results five prompts.
of zero-shot classification on Pub-Brain-5. We report the balanced accuracy of HLIP under
Anomaly Detection
Tumor
Disease
Stroke
Glioma
Meningioma
Metastasis
mean
Pub-Brain-B
86.9 + 6.0
92.4 + 4.8
76.1 + 3.4
84.9 + 4.9
85.4 + 3.7
62.3 + 1.4
63.4 + 3.8
Pub- Brain-5-GT
90.4 + 5.2
92.5 + 5.0
76.3 3.3
77.4 6.9
84.1 3.7
53.8 = 0.9
59.3 + 3.7
In Table 8, we additionally report the full results of HLIP under this sctting on the Fub-Brain-5 and Pub-Brain-5-GT benchmark, for reference in future work.
E
Complexity
In this section, we analyze the compulational efficiency of the proposed hierarchical attention mechanism from both theoretical and practical perspectives.
E.1 Analysis
Given a sequence length of N, model dimension of the matrices Q, K.V. E Nxe. The output O E Nxe of standard self-attention is computed as:
S = K E ENNN P P= ==softmaz($) E R NxN. O = PV
(1)
where S and P require 2(N memory access; Q, K. V. O requires Q(N x c) memory access. Therefore, the I/O complexity for a slandard self-attention over tokens is:
Q(N + X c)
(2)
The S requires Q(N² x c) multiply-add operations; P requires ?(N?)) multiply-add operations; O requires 2(N2 x c) multiply-add operations. Therefore the compute complexity for a standard solf-attention over tokens is:
Ω(N2 x c)
(3)
Study Attention. In our scenario, the sequence length of a study is = M x d w. The study attention is standard self-attention over all N tokens. Therefore the I/O complexity and the compute complexity of the study attention is Q(N2 + N e) and Ω(N2 c), respectively, as shown in Equation 2 and Equation 3.
The scan attention is M standard self-attention operations, each over M tokens. Therefore X2 replace with M7 in Equation and Equation 3 resulting in 2 N + Nxe) and S f x c) for 1/O and compute complexity of each self-attention operations, therefore the total I/O and compute complexity of the N² N N x c) and Q N² W x c), respectively.
Scan Attention.
N
scan attention are O
Slice Attention. The slice attention is M x d standard self-attention operations, each over Mxd tokens. Therefore, similar to the scan attention the total 1/O and compute complexity of the slice attention are N² 2 Mxa + N x e) and l Mxa c), respectively.
N
Na
23

---

Published in Transactions on Machine Learning Research (02/2026)
E.2 Experiments
We conduct experiments comparing ViT (Dosovilskiy et al., 2020), the 3D Swin Transformer imple- mented in MONAI (Cardoso et al., 2022), and our HLIP visual encoder. Since Swin is not com- patible with multi-scan inputs, the experiment is conducted on a single-channel 3D input of shape (B,1,224, 224, 224), where B=1 denotes the batch size. Both ViT and HLIP use a ViT-Base architecture with patch_size= size= (16, 116,6,, 16,16). The Swin Transformer is configured with dim=128, patch_size= size=(4, 4,4,, window
and 8 Following the original Swin Trans- former (Liu et al., 2021) while matching the model size.
Lindow_siz=-(i sii_=size== 7 (,7,,)), 7), (2,th=(2,,,12)
Table 9: Practical Complexity.
model size (M) throughput (img/s; t-better)
memory (G; J=better)
VIT
88.2
9.0
6.6
Swin
87.9
9.9
8.0
HLIP (ours) 88.2
17.5
4.1
All experiments are conducted single A40 GPU. We report the model size, throughput and training memory for each model. To measure throughput, we include 100 warm-up iterations and compute the average over 1000 forward passes. As shown in Table 9, Swin is slightly faster than the original ViT but lags behind our HLIP visual encoder. Swin also consumes more memory during training due to the large codebook of relative position embeddings constructed for 3D inputs. Moreover, Swin not compatible with flash attention, which has been integrated into PyTorch 2.0 and supports wide range of GPUs.
F Discussion
In this work, we present HLIP, scalable language image pre-training framework for 3D medical imaging, For health systems that have accumulated large volumes of data over the past decades, HLIP offers new direction for learning transferable representations. In this section, we provide additional discussion, including both the limitations and future directions for readers interested in pursuing this line of research.
Study Curation. In this work, we collect all studies from our health system, resulting in an imbalanced pre-training dataset. Building findings from the natural image domain, believe that developing systematic approach to construct more balanced pro-training dataset is an important direction for future work. Please note that this curation should occur al the study level and therefore does not impose additional burden on radiologists, unlike selecting representative scans or slices.
Zero-shot Transferability. We observe that zero-shot transferability does not always correlate with the number of training si uddes, in contrast to the findings of Udandarao el al. (2024). For example, although keyword search yields more patients with meningioma or metastasis than with glioma, zero-shot performance glioma is substantially higher. We hypothesize that this discrepancy arises from fundamental differences between natural image and medical imaging datasets. For natural image datasets such as ImageNet, the relationship between the number of positive instances in the training corpus and zero-shot performance is approximately log-linear. This observation implicitly assumes a small or near-zero Bayes error rate, for example, when differentiating boats from dogs. In contrast, in medical imaging, diagnosing meningioma and metastasis is challenging for multifaceted reasons, including lesion size, radiographic features, and anatomical location, which Icads to a higher error rate compared glioma.
Scale. Even at its current scale of 200K studies, the largest to date, our dataset remains far smaller than training scales in the natural image domain, which typically involve billions of samples. Although collected from neuro-radiology system, we find that other organ systems such as the spine, abdomen, and knce occasionally appear in our dataset, and the current. model demonstrates promising zero-shot transferability to these organs as well. Therefore, we believe there is no barrier to training a model on more comprehensive data, inchuding different organ systems (e.g., brain, cardiac, chest, abdomen) and modalitics (e.g., CT and MRI). Since no human labeling burden is involved, we expect the training scale to expand millions of samples in the near future.
24

---

Published in Transactions on Machine Learning Research (02/2026)
As the dataset scales further, especially when encompassing multiple organ systems and modalities, the underlying patterns will become more complex. Therefore, we believe that larger batch sizes will be necessary. Although our hierarchical attention mechanism introduces no additional computational burden, we rely on both flash attention and gradient checkpointing to achieve a batch size of 256 on 8 L40 GPUs. Further increasing the batch size will require either gradient accumulation strategy or additional computational resources.
Moreover, as the dataset scales up, larger models may prove beneficial but will also require additional computational resources.
Vision-Language Model. Although current vision language models achieve impressive performance even on medical lasks, we believe these general-purpose models will reach a bottleneck when sufficient. specialist. medical data are not available on the internet. On the other hand, as with other Sunguage-supervsser visual encoders, HLIP's visual encoder could facilitate the development of vision language models under current frameworks. Therefore, we expect to radiology specialist vision language model that can operate on entire studies in the near future.
Dataset Release Policy. The assets introduced in this work, including new brain MRI benchmark for zero-shot classification, an effective language-image pre-training implementation for 3D medical imaging, the pre-training recipe, and the model checkpoints, will be fully publicly available. However, we are unable to release the BrainMR1220K and HeadCT240K datasets alongside this work due to privacy constraints associated with neuroimaging data. We plan to release a synthetic dataset in the future, where the generative model is trained large-scale dataset.
25


Reviews:
[{"review_id": "2lmMtrkjvj", "reviewer": "Reviewer_NGWG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes HLIP (Hierarchical attention for Language-Image Pre-training), a language–image pre-training approach tailored to real-world 3D radiology studies, such as brain MRIs and CT scans. The paper is motivated by the fact that prior 3D medical language–image pre-training often depends on curated data (single standardized scan per study, careful selection, extensive data cleaning), which is expensive and limits scale. HLIP instead aims to pre-train directly on uncurated clinical studies using a heirarchical attention mechanism that computes attention on three levels of granularity:\n1. Slice-level attention: attention within groups of adjacent slices (lowest-level local context).\n2. Scan-level attention: attention within each scan (mid-level context).\n3. Study-level attention: occasional global attention across the whole study (cross-scan context). \n\nThe paper then performs training using two new uncurated datasets:\nBrainMRI220K: ~221k MRI studies\nHeadCT240K: ~244k CT studies\n\nThis paper also proposes a new benchmark, Pub-Brain-5, for zero-shot transfer evaluation.\n\nStrengths\n- I find the heirarchical attention mechanism to be a strong contribution, as it allows for a more nuanced approach to CLIP-style training. It allows for a larger batch size of training without the need to equally scale resources. It also demonstrates improved performance over contemporary methods, such as fVLM and CT-CLIP. \n- Large scale data training strongly positions this work's contributions. \n\nWeaknesses/questions\n- When comparing performance of chest CT models, I am unclear if the training on CT-RATE takes place for a model checkpoint that is already trained on the HeadCT dataset or if training takes place from scratch from a randomly initialized model. If it is the case that a pretrained checkpoint is used, I am unclear the extent to which the model architecture itself is improving performance, and to what extent it is just a case of the model having seen more data, albeit of a different location. \n- Was any preprocessing applied to the reports themselves in your two new datasets, or were raw reports used as is?\n- In practice, DICOM series boundaries can be noisy, incomplete, or inconsistent, especially when working across institutions. Reporting styles may also differ. Were any efforts made to make training more robust to such cases? Do you think it would make a difference?", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes I believe they are\n1. Large scale training is conducted using dataset sizes greater than existing methods. \n2. Strong competing baselines are compared against in both head and chest settings. \n3. Ablation studies confirm that the reported configuration of the model provides the strongest results.", "audience_yn": "Yes", "audience_explanation": "Yes I believe so. I believe the techniques, model, and benchmark introduced in this paper would definitely be useful for people working in this space, particularly for downstream tasks such as vision language modelling for medical images and segmentation tasks.", "requested_changes": "Please provide clarity on the weakness and questions listed above.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ny2uhz3rUX", "reviewer": "Reviewer_jasq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary** \n\nHLIP introduces a simple hierarchical attention schedule (slice, scan, study) that lets a ViT-style encoder pre-train directly on uncurated, multi-scan radiology studies. Trained at health-system scale (BrainMRI220K, HeadCT240K), it achieves SOTA zero-shot and linear-probe transfer performance on public benchmarks (Pub-Brain-5/5-GT, CQ500, RSNA, CT-RATE, Rad-ChestCT) and strong prospective results on large internal cohorts. The paper also contributes the Pub-Brain-5/5-GT evaluation sets, reports clear efficiency gains over ViT and Swin, and provides ablations on attention placement, class-token propagation, and batching.\n\n**Strengths** \n\n1. Clear problem setting and novelty: modeling uncurated, multi-scan studies and aligning the attention hierarchy with the radiology data hierarchy is reasonable and practically motivated. \n\n2. Strong empirical results: the proposed model achieves consistent gains across diverse modalities (brain MRI, head CT, chest CT) in zero-shot and linear-probe settings, e.g., notably +10.5% balanced accuracy on Pub-Brain-5 disease classification and AUC gains on CQ500 and RSNA over strong head-CT baselines. \n\n3. Scalability and efficiency: memory advantages over Swin and compatibility with flash attention and patch dropout are demonstrated. Ablations are informative. \n\n4. Practical relevance: results on an internal prospective set, and qualitative activation maps that localize pathologies across slices and scans, strengthen translational value.\n\n**Weaknesses**\n\n1. Single-Source Data: The training data was collected from a single health system. It may introduce biases related to specific scanner manufacturers, imaging protocols, or patient demographics, which could limit the model's generalizability.\n\n2. The authors observe that zero-shot transferability does not always correlate with the number of training studies for a given diagnosis. This discrepancy is not fully explained.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. The paper states concrete dataset sizes, model configs, and training recipes. It also includes ablations and shows consistent gains across Pub-Brain-5/5-GT, CQ500/RSNA, and CT-RATE/Rad-ChestCT\n\n2. Convincing evidence: Improvements are large (e.g., 8.3% macro AUC on CQ500) and align with the proposed mechanism that hierarchical attention reduces attention cost while preserving global context intermittently.", "audience_yn": "Yes", "audience_explanation": "3D Medical Imaging pretraining is a important topics for real-world applications.", "requested_changes": "See weakness 2.", "broader_impact_concerns": "None.", "additional_comments": "None.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0ShWWrqGCO", "reviewer": "Reviewer_TJLi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary:\nThe authors' contribution to the community is three-fold: a) they introduce a lightweight attention mechanism tailored to the inherent structure of the data, b) they present a scalable pre-training approach that does not rely on human-annotated, curated data, and c) they provide a zero-shot dataset for brain MRI disease classification. The authors evaluate their approach on both uncurated and curated data, including a zero-shot experiment and a real-world case study.\n\n### Strengths:\n\n**(S1 - clarity and quality wrt problem setting).** The introduction and related work sections are very well written. They clearly define the general problem of pretraining in 3D medical imaging and discuss the limitations of existing SOTA methods. This effectively sets the stage for the authors’ contributions, as the need for methods that can handle uncurated data becomes evident, and the motivation for a more lightweight attention mechanism is well justified given the large number of tokens and samples involved.\n\n**(S2 - description of the input data format and the attention mechanism).** I highly appreciate the first paragraph of Section 3 and Section 3.1. The data object \"study\" is well introduced, with all dimensions clearly explained. The descriptions of \"volume\" elements and \"visual tokens\" make the input format for training and evaluation easy to understand. The explanation of the different dimensions further aids in grasping the data structure and how information is shared across the various attention mechanism variants.\n\n**(S3 - relevance and novelty of the attention mechanism variants proposed).** The proposed lightweight attention mechanism is both novel and relevant. It is well motivated: replacing full attention with a more efficient version reduces computational overhead, and the included inductive bias is grounded in the data's structure (consider a) a whole data sample, b) scan-wise, or c) share information across tokens within slides close to each other). I also appreciate the discussion of similarities and differences with sliding-window attention.\n\n**(S4 - strong results).** The proposed method outperforms baselines which may indicat its effectiveness. However, see (W2) and (Questions).\n\n### Weaknesses:\n\n**(W1 - clarity wrt to pre-training).** If understood correctly, a) the proposed model leverages uncurated (i.e., non-annotated) data, and b) pretraining is performed via contrastive learning by matching text and image representations. However, it is unclear what exactly serves as input to the text encoder. The manuscript lacks sufficient detail in this regard, which is critical since pretraining on uncurated data is a major selling point of the work.\n\n**(W2 - missing error bars).** All results are reported without error bars or statistical tests. This significantly weakens the strength of the reported findings, as the observed differences may be due to random variation. The authors should include error bars and appropriate significance testing.\n\n**(W3 - selection of attention-mechanism hierarchy).** The authors describe mixing blocks with different attention hierarchies, which is reasonable and reminiscent of mixed-architecture LLMs using SSM and transformer blocks. However, it would be helpful to provide intuition on when to choose scan-wise vs. adjacent-slice-wise attention. How was this decision made? Was it treated purely as a hyperparameter? More insight would be valuable.\n\n### Questions:\n* See W1 and W3.\n* The model is pretrained on two in-house datasets, BrainMRI220K and HeadCT240K. Will these datasets be released? If not, reproducibility is limited, and the authors should consider using public datasets.\n\n### Very minor comment:\nmax, mae, scan, slice, and any other abbreviations or terms appearing in formulas should be written in non-italic font.", "claims_and_evidence_yn": "No", "claims_explanation": "Unfortunately, the manuscript lacks details on how the preprocessing was performed, making it difficult to assess how the central research question — training from uncurated data — was addressed. In addition, the effectiveness of the proposed approach cannot be verified due to the absence of statistical tests and error bars.", "audience_yn": "Yes", "audience_explanation": "Medical imaging and machine learning methods tailored to it are of interest to the community. The authors clearly articulate the limitations of current state-of-the-art approaches, and the proposed lightweight attention mechanism appears both reasonable and interesting.", "requested_changes": "- Clarify the pre-training procedure, including data inputs and methodology.\n- Add error bars and statistical tests to all reported results in tables and figures.\n- If not already planned, open-source the datasets used for pre-training or consider using publicly available alternatives.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)