Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models
Authors: Ulzee An
Moonseong Jeong
Simon Austin Lee
Aditya Gorla
Yuzhe Yang
Sriram Sankararaman
Venue: ICML 2025 spotlightposter (2025)
Keywords: Embedding
Volumes
Foundation model
Random projection
Compression
MRI
CT
Primary area: applications->health_medicine
TL;DR: Raptor (Random Planar Tensor Reduction) leverages a pretrained image model (DINOv2-L) to create and compress visual tokens in 3D, and achieves state-of-the-art performance across 10 medical volume tasks without the need for costly training.

Abstract: Current challenges in developing foundational models for volumetric imaging data, such as magnetic resonance imaging (MRI), stem from the computational complexity of state-of-the-art architectures in high dimensions and curating sufficiently large datasets of volumes.
To address these challenges, we introduce Raptor (Random Planar Tensor Reduction), a train-free method for generating semantically rich embeddings for volumetric data. Raptor leverages a frozen 2D foundation model, pretrained on natural images, to extract visual tokens from individual cross-sections of medical volumes. These tokens are then spatially compressed using random projections, significantly reducing computational complexity while retaining rich semantic information. Extensive experiments on 10 diverse medical volume tasks verify the superior performance of Raptor over state-of-the-art methods, including those pretrained exclusively on medical volumes (+3% SuPreM, +6% MISFM, +10% Merlin, +13% VoCo, and +14% SLIViT), while entirely bypassing the need for costly training. Our results highlight Raptor's effectiveness and versatility as a foundation for advancing deep learning-based methods for medical volumes (code: github.com/sriramlab/raptor).

Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models
Ulzee An"
Moonseong Jeong" 1 Simon A. Leee
Aditya Gorla 23
Yuzhe Yang" 12
Sriram Sankararaman 124
Abstract
Adrenal
Fracture
.926
Current challenges in developing foundational models for volumetric imaging data, such mag- netic resonance imaging (MRI), stem from the computational complexity of training state-of-the- art architectures in high dimensions and curat- ing sufficiently large datasets of volumes. To address these challenges, we introduce Raptor (Random Planar Tensor Reduction), a train-free method for generating semantically rich embed- dings for volumetric data. Raptor leverages a frozen 2D foundation model, pretrained on nat- ural images, to extract visual tokens from indi- vidual cross-sections of medical volumes. These tokens are then spatially compressed using ran- dom projections, significantly reducing compu- tational complexity while retaining semantic in- formation. Extensive experiments on ten diverse medical volume tasks verify the superior perfor- mance of Raptor over state-of-the-arr methods, including those pretrained exclusively on medical volumes (+3% SuPreM, +6% MISFM, +10% Merlin, +13% VoCo, and +14% SLIViT), while entirely bypassing the need for costly training. Our results highlight the effectiveness and ver- satility of Raptor as foundation for advancing deep learning- based methods for medical volumes (code: ggithub.com/rriaalll//raaaa
Vessel
Nodule
966
929
ptor
Synapse
943
[hapiter
999
Organ
.95
389
CC-CCI
Brain IDPs
620
.711
CTRG-C
CTRG-8
Merlin
SUVIT
SuPreM
Raptor
Figure I. Comparisons between Raptor and state-of-the-art methods on diverse medical volume tasks. Raptor achieves superior performance across classification (AUROC1) and regres- sion (mean r21, indicated with +) tasks while remaining entirely train rrin-free.
human-computer interaction (Jiang et al., 2023), and au- tonomous systems (Huang et al., 2025). However, learning from volumetric modalities, represented 3-dimensional voxels, meshes or point clouds, continues to pose logistical challenges compared with 2-D images. For instance, adapt- ing convolutional (Yang et al., 2021) transformer architec- tures designed for images (Dosovitskiy et al., 2021) to vol- umes remains computationally challenging as both classes of architectures incur cubic or higher order costs to learn or infer with volumes. Overcoming these hurdles generally re- quires the development of novel training objectives, efficient architectures, or access to large-scale compute infrastruc- ture (though such infrastructure remains out of reach in most research settings). Furthermore, significant challenges remain in acquiring large-scale 3D training datasets rele- vant to the task of interest: the largest 3D medical datasets (160K volumes, Wu et 2024b) remain several orders of magnitude smaller than the 2D image datasets (1.2B images, Oquab et al. 2023) that have allowed researchers scale large image models for current state-of-ther aht 2D vision applications. As result, few works to date have experi- mentally demonstrated the benefits of scaling large models for medical volumes (Zhu et al., 2023; Ma et al., 2024; Wu et al., 2024a), while large models for natural images and their capabilities are well-established (Radford et al., 2021;
1. Introduction
Understanding volumetric data is essential in variety of applications, including healthcare (Milletari et al., 2016),
"Equal contribution 'Computer Science Department, Univer- sity of California, Los Angeles 'Department Computational Medicine, David Geffen School of Medicine, University of Cali- fornaa, Los Angeles ' BBoinnormatics Interdepartmenttal Program, University of California, Los Angeles 'Department of Human Ge- neties, University of California, Los Angeles. Correspondence to: Ulzec An <ulzcee@ es. ucla.edu>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Random Planar Tensor Reduction
Caron al., 2021; Zhou et al., 2022; Saharia et al., 2022; Rombach al., 2021; Ranftl et al., 2022; Kirillov et al., 2023; Liu et al., 2024a; Oquab et al., 2023).
Raptor is model-agnostic it is designed to seamlessly integrate with and benefit. from future advancements in image foundation models.
In this work we propose extracting semantically rich, low- dimensional embeddings from large 3D volumes by lever- aging foundation models trained on generic 2D images without requiring training on the volumes. The straight- forward approach involving a direct application of an image foundation model (DINOv2-L, Oquab et al. 2023) to infer tokens over orthogonal cross-sections of a volume, however, leads to embeddings that are substantially larger than the volumes. To this end, we introduce Raptor: Random Planar Tensor Reduction, a train-free technique that drastically compresses the dimensionality of features over volumes by using random projections. Random projection is a class of scalable stochastic dimensionality reduction methods that map high-dimensional data into a lower-dimensional space by multiplying with a random matrix, with well-studied theoretical/empirrcal guarantees such as approximate preser- vation of pairwise distances and low-rank signals (Xie et al., 2017). We use random projections to compress tokens in- ferred by frozen image foundation model over orthogonal cross-sections of volume and then reduce major spatial dimensions to significantly compress the final representa- tion. Our method reliably yields embeddings that have ~ 99% smaller footprint than that of raw voxels (256', compressed).
2. Related Works
Volumetric data span modalities such as point clouds (Cae- sar et al., 2020), meshes (Chang et al., 2015), and voxels (Yang et al., 2023). Each modality presents unique computa- tional challenges; for instance, point clouds are not arranged in regular grids and have no per-point features such as color that are common for voxels (Sajid et al., 2025).
Raptor produces embeddings of voxels with cross sec- tions, frequently obtained using magnetic resonance imag- ing (MRI) or computational tomography (CT). Below, we review works that learn useful representations from such medical volumes for downstream tasks.
State-of-the-art architectures. Many works have pro- posed adapting popular convolutional architectures, such as ResNet (He et al., 2016), for 3-dimensional data (Ning et al., 2019; Ebrahimi et al., 2020; Qayyum et al., 2021; Yang et al., 2021; Turnbull, 2022; Xue & Abhayaratne, 2023; Blankemeier al., 2024). The success of the vision trans- former (VIT) with shifted windows (Swin, Dosovitskiy et al. 2021; Liu et al. 2021) has inspired many recent works to build on ViT as the backbone network for 3D imaging tasks (Hatamizadch et al., 2021; Wasserthal et al., 2023; Li et al., 2024; Cox et al., 2024; Wu et al., 2024b). However, trans- former architectures incur a heavy computational cost in 3D due self-attention': subsequent works have proposed alternate architectures utilizing convolutional steps (Choy et al., 2019; Lai et al., 2024; Avram et al., 2024) or using forms of sub-quadratic attention (Liu et al., 2024b; Shaker et al., 2024; Xing et al., 2024; Dao, 2024). Large-scale pretraining for medical volumes. Innova- tions in self-supervised learning (SSL) and the acquisition of web-scale datasets have enabled the development of foun- dational image (2D) models that demonstrate downstream capabilities that surpass domain-specific supervised models (Radford et al., 2021; Caron et al., 2021; Zhou et al., 2022; Saharia et al., 2022: Rombach et al., 2021; Ranftl et al., 2022; Kirillov et al., 2023; Liu et al., 2024a; Oquab et al., 2023). Several works have sought to replicate their success by collecting large datasets of medical volumes (3D) and scaling state-of-the-arr methods.
Despite its substantial size reduction, we show that Raptor embeddings retain volumetric information by demonstrating higher accuracies on downstream tasks than current state- of-the-art (SOTA) models, including those that leverage extensive pretraining (Figure 1). We demonstrate these ca- pabilities without fine-tuning the underlying vision model or fitting any additional 3D architecture (i.e., train-free), mak- ing Raptor a particularly appealing choice in data-scarce and cempute-constrained settings that are common in biomedi- cal applications. Our work enables researchers with limited computational resources to utilize powerful pretrained mod- els for -igheniienni volumetric analyses, thus broaden- ing access to lar---aaninnbaasse 3D methods and fostering more inclusive innovation in machine learning.
Our contributions can be summarized as follows:
Raptor is highly data-efficient - it the first method that achieves state-of-the -frt---rr performance without requiring access to large datasets of medical volumes.
Raptor embeddings are train-free for costly 3-dimensional model training, significantly re- ducing computational overhead.
we eliminate the need
Li et al. 2024 released suite of models including Swin- UNETR, SegResNer, and ResNet trained on 5K CT vol- umes, collectively termed SuPreM, that are proposed to serve as foundation models given the pretraining process that involves 673K annotations over multiple organs. Simi-
Raptor embeddings are scalable they are sub-cubic with respect input volume size, ensuring efficiency even for high-resolution medical volumes.
Liu et al. 2021 utilized 8x V100 with batch size I per GPU
2

---

Random Planar Tensor Reduction
Table Current otate-of-the-art models that leverage pretraining with medical volumes. state scale of each method's medical volume pretraining data, its target medical domain, and its backbone architecture ("we note that Raptor is agnostic to any downstream domain, medical or not).
METHODS
MEDICAL PRETRAINING DATA
DOMAIN
ARCHITECTURE
SLIVIT (AVRAM ET AL., 2024)
14M IMAGES (IMAGENET)
OPTICAL
2D CONV+3D TRANSFORMER
+ 108K OCT IMAGES
SUPREM (LI ET AL.., 2024)
5K CT VOLUMES
GENERAL
3D SWINUNETR, 3D RESNET
MERLIN (BLANKEMEIER ET AL.. 2024)
15K VOLUMES
CHEST
3D RESNET
MISFM (WANG ET AL., 2023)
110K CT VOLUMES
GENERAL
3D CONV+3D TRANSFORMER
VoCo (WU ET AL., 2024A)
160K CT VOLUMES
GENERAL
3D SWINUNETR
RAPTOR (OURS)
NONE
GENERAL"
2D VIT + 3D COMPRESSION
(USES 2D FOUNDATION MODEL)
main feature of large-scale pretrained models is their ability to infer powerful latent representation of the target modality (i.e. embedding) that encompass features that generalize to many unseen downstream tasks (c.g. CLIP, Radford et al. 2021). In spite of ongoing efforts, few works qualify as be- ing capable of generalizing to arbitrary medical volumes due to (1) the limited availability (in terms of size and access) of the 3D datasets and (2) the computational cost associated with scaling current approaches in 3D. To overcome these challenges, we propose a new paradigm to compute rich embeddings of volumes without training any foundational models.
Table 2. Sizes of comparable medical volume models. We com- pare the parameter count of cach medical volume model and the size of their latents. The latent was determined as the smallest bottleneck layer without pooling,
METHODS
PARAM. COUNT
LATENT SIZE
SLIVIT
48.4M
768 64 x x
SUPREM
5.1M
128 x 12 X
12
MERLIN
124.7M
2048 x 14 X 7 x 7
MISFM
46.2M
100 x 16 x 16 x 16
VoCo
294.9M
3072 x 3 x 3
RAPTOR
304.4M
3 x 100 16 x 16
(OURS)
(DINOv2-L)
(K 100)
3. Raptor
larly, Wang et al. 2023 proposes MISFM, medical image segmentation foundation model with a hybrid convolutional- transformer architecture trained on 110K scans containing multiple organs. Wu et al. 2024a proposes training a Swin- UNETR model with foundational capabilities called VoCo, based on weakly-annotate dataset of 160K CT volumes that span multiple organs. This work takes advantage of geometric SSL approach that leads to ttate-of-the-art capa- bilities on several downstream semantic segmentation tasks, outperforming SimCLR (Chen et al., 2020), Minkowski (Choy et al., 2019), UniMISS (Xie et al., 2024). Despite performance improvements, training these models still de- mands substantial computational r
3.1. Overview of the approach
An outline of our approach is visualized in Figure 2. The core idea behind Raptor is to leverage a pretrained 2D im- age foundational model to encode the semantics of the 3D volumes in three different axial views. The resulting high- dimensional tensor is then aggregated over the axial views and low-rank approximated with random projections. We obtain the final Raptor embedding by flattening the three projections of the volumes. There is no training involved in any of the steps in our method.
3.2. 3D Inference using 2D foundation model
Other related works develop methods that pretrained specific CT modalities (e.g., chest) but still can be useful for general volumetric datasets when finetuned. Such ap- proaches include Merlin (chest CT, Blankemeier et al. 2024) and SLIVIT (optical CT, Avram et al. 2024). These methods leverage pretraining with up to 100K volumes in their medical domains. We summarize notable prior works in Table 2.
Pretrained 2D foundation model. We use the current state- of-the-art image foundation model as a domain agnostic feature extractor given 2-dimensional views of voxels. We utilize the latest DINOv2 model (Oquab et al., 2023) check- point the time of writing'. DINOv2 consists of image encoder, which is vision transformer (VIT) trained curated dataset of 142M images (LVD-142M) and 1.2B ad- ditional images crawled on the web. The curated image dataset is reported to contain ImageNet-22k,, the training split of ImageNet-Ik, Google Landmarks and additional
General-purpose embeddings for medical volumes. A
"Wu et al. 2023 utilized Hx00 GPUs for undisclosed amount of time, while Li et al. 2024 discloses the use of 8 A 100 GPUs for at least? 7 days.
http://aaagnnaaacccoocccooo/oon/
3

---

Random Planar Tensor Reduction
Embodaice
40 Tonso
Flatten
Dense Volume (e.g. 256")
3-axis
DINO VIT image encoding
Planar reduction
Random projections
Raptor (76k)
sampling
(K=100)
Figure Flowchart visualizing the computation of Raptor embeddings from medical volume. Raptor leverages a pretrained 2D image foundational model encode the semantics of the 3D volumes in three different axial views. The resulting high-dimensional tensor low-rank approximated with random projections and aggregated. We obtain the final Raptor embedding by flattening the projections of the volumes. There is no training involved in any of the steps in our method (high-dimensional tensors not drawn to scalc).
datasets containing natural images. The crawled images are described to have been identified via URL,, making it unlikely for to contain the training or test set domain- propose
3.3. Planar reduction using random projections
We describe the core contribution of Raptor, which is the efficient low-rank approximation of the high-dimensional tensor inferred by the DINOv2-L ViT through mean-pooling across the slice embeddings and random projections. This approach requires no further training of any parametric models, avoiding the need for additional GPU compute.
specific datasets (such as Medical MNIST). using the largest model (DINOv2-L) with 304M parameters (we validate this choice against other foundation models ex- perimentally in A. We note that our approach is agnostic to the underlying choice of the image foundation model, and can benefit from continued development of better models. In A.3, we further demonstrate how features in the medi- cal domain can detected using a general purpose image encoder.
Given volume E DxDDx we specify all its slices (2 (2-dimensional images) in three orthogonal directions as S E DR3xDx(DxD), where S,, i E {1, 21,2,3) 3) corresponds to D slices of width and height x D observed in one the three directions (Figure A. l). We use pretrained VIT d(.) that operates on images to obtain the intermediate represen- tation z = concat (S:)] E 33xDxdxpp where token size 15153.
3-axis volume sampling. We apply DINOv2-L across 3D volumes along all three axes (axial, coronal, and sagittal referring to top-down, front-back, and left-right directions; Figure A.I) of the volumes to capture cross-sectional fea- tures without relying on a 3-dimensional model. Given a volume of dimensions 2563, we apply the visual encoder of DINOv2-L to 256 total slices. The ViT defines patch size T (typically T 16), resulting in initial raw representation of 3 x 256 x 1024 16 16 x 201M val- ues, where 16 x 16 is the number of patches given slices of width and height 256 x 256 of the volume, and 1024 is the dimensionality of the patch embeddings.
d and number of patches p D/T are fixed by the VIT. We first reduce by averaging the direction in which the slices were observed; repeating this for three views results in a tensor of size 3 x d x p². Then, for each of the p² patches, we obtain a low-rank approximation of the spa- tially reduced tokens through random projections. For a number of projections K, we sample a projection matrix R R xd where REI N N(00) 1) and K d. Projecting the tokens with R results in tensor of size 3 x K x p² The final Raptor embedding is generated by flattening this tensor into vector 33kp"
Using the ViT to parse volumes in a dense fashion introduces computational challenges, as the inferred latent tensor is far greater than the size of the original image (in contrast to e.g., VAEs Kingma& & Welling 2014). For instance, storing every tokenized slice of a volume requires 383MB of space using single precision floats (36TB+ + for a dataset of 100K volumes), which 127x x larger than the storage size of the 256 cube in unsigned integers (~ 3MB compressed with gzip. ~ 300GB for dataset of 100k volumes).
1 R Σ Zij
: flatten concat 15153
where j indexes the tensor in žiy corresponding to slice in particular axis i. A more detailed formulation of Raptor is described in Appendices A.5 and A.6. As described, the size of Raptor embeddings scale as O(p² K), which is sub-cubic in the dimension of the input volume, allowing us to store and use the representations for downstream tasks with ease.

---

Random Planar Tensor Reduction
Train pams
Val (e54)
Test g0m)
Raptor
Embeddings
Figure 3. Triangulation of a feature from lower dimension. Visualization of the intuition behind triaxial sampling. Features reduced to orthogonal lower dimensions can be used to triangulate features in the original space (c.g. in 2D or 3D).
Fine-tuning (8%))
Vooc P9h
Test
Classification
Regression
Figure 4. Making downstream predictions using Raptor. Raptor was used to obtain embeddings of medical volumes with no train- ing by using frozen image foundation model. For downstream tasks, embeddings of training and validation volumes were used to obtain the best lightweight predictor.
The quantities in typical settings are D = 256, = 1024, 7 = 16, and p = 256/16 16 16 (using DINOv2-L), resulting in Raptor embeddings of size 768 x K based on the choice of K. We infer two different Raptor embeddings on all datasets: Raptor using K 100 and Raptor-base (Raptor-B) using K = 10 (determined experimentally, Sec- tions 4.5 and 5.1), the latter of which is 10x more compact.
4. Experiments
4.1. Experimental settings
Effectiveness random projections. Random projection is underpinned by the Liinsson-inddestrruuss (JL) lemma (Dasgupta & Gupta, 2003), which states that distances be- tween points (e.g.. patch embeddings) are preserved up small distortion factor (1 = =) with high probability when mapped into TR& (see Appendix A.5 for full proof). Empir- ically, we find that modest K = 10 preserves features that are comparable to stte-off-hhe-at methods while reducing each slice embedding by up to 50x or more (Section 5.1).
Datasets. We benchmark Raptor on ten datasets that span classification and regression tasks for medical volumes. We summarize the statistics for each dataset in Table A.5.
3D Medical MNIST (Yang et al., 2023). The Medical MNIST (MedMNIST) dataset was proposed as a standard- ized benchmark for classifying medical imaging modali- ties. The 3D category of McdMNIST compiles six sepa- rate benchmarks based on prior works (Bilic et al., 2023; Armato et al., 2011: I; Jin et al., 2020; Yang et al., 2020). Each benchmark presents a classification task, where the number of classes can vary from two 11. The number of volumes available for training is generally low ($ 1.3k) unlike the 2D datasets in the same benchmark (S 165k), highlighting the relative data-scarcity of 3D data. The imaging technology underlying the volumes is also di- verse, ranging from computed tomography (CT), mag- netic resonance imaging (MRT), and electron microscopy (EM).
We choose random projection over other dimensionality reduction methods for reasons. First, it allows for more direct theoretical treatment (Appendix A.6). Sec- ond, given N number of volumes, the time complexity of Raptor is (D + K)) (A.7). In comparison, a full PCA to reduce to the same low-rank dimensions would be df)"d2N), which more costly. A probabilistic PCA may be used achieve a comparable complexity of ODDpdNN + d2),), but is iterative process that is slower in practice. Tensor factorization methods also pose computational challenges: they require the formulation of a factorization structure for the tensors, and generally have worse time complexity (more discussion in Appendix A.8). Our method can process volumes of size 256 in approxi- mately ~ 6.5s per volume with an 11GB RTX 2080 Ti.
CC-CCI (Zhang et al., 2020). We explore the China Consortium for Chest CT Image Investigation dataset which presents a classification task between three classes over 2, 471 chest CT seans.
CTRG (Tang et al., 2024). We explore classification tasks over brain MRIs and abdominal CT scans available in the CTRG dataset. total of 6, 000 brain MRIs were available with 3 multi-class labels (CTRG-B). A total of 1, 804 chest CTs were available with 4 multi-class labels (CTRG-C).
Orthogonal planar reduction. The reduction hinges on the observation that substantial information from volumes can be preserved given their cross sections from multiple viewpoints (Figure 3). For instance, feature specific to the left or right side of the brain may be lost from a sagittal profile, but can be recovered when also given the axial profile.
UKBB Brain MRIs (Bycroft et al., 2018). We examine 1, 476 MRIs from the UK Biobank that were available at the time of writing, in conjunction with 162 quantitative
5

---

Random Planar Tensor Reduction
Table 3D Medical MNIST volume classification benchmark. We compare Raptor and state-of of the-art art methods on six datasets from the 3D Medical MNIST benchmark. We report AUROC and raw accuracy (ACC1) (best scores in bold. second best underlined). The methods are grouped by training strategies from scratch, pretrained on medical datasets, and Raptor.
ORGAN
NODULE
FRACTURE
ADRENAL
VESSEL
SYNAPSE
METHODS
AUC
ACC
AUC
ACC
AUC
ACC
AUC
ACC
AUC
ACC
AUC
ACC
RESNET
0.995
0.918
0.886
0.860
0.759
0.487
0.869
0.835
0.932
0.915
0.889
0.853
MAE
0.982
0.800
0.820
0.828
0.600
0.481
0.710
0.752
0.607
0.880
0.560
0.737
MISFM
0.989
0.833
0.886
0.855
0.689
0.537
0.868
0.665
0.932
0.894
0.918
0.307
SUPREM
0.999
0.968
0.891
0.848
0.645
0.492
0.906
0.869
0.964
0.929
0.907
0.879
SLIVIT
0.997
0.946
0.920
0.868
0.656
0.475
0.846
0.789
0.710
0.880
0.541
0.270
VoCo
0.992
0.870
0.797
0.836
0.699
0,535
0.913
0.872
0.799
0.880
0.844
0,830
MERLIN
0.976
0.766
0.809
0.861
0.691
0.549
0.836
0.801
0.870
0.879
0.833
0.825
RAPTOR-B
0.998
0.958
0.904
0.8.58
0.647
0.501
0.930
0.858
0.945
0.919
0.922
0.894
RAPTOR
0.999
0.961
0.929
0.870
0.677
0.502
0.926
0.845
0.966
0.922
0.943
0.911
-eaging-derived phenotypes (IDPs) in the dataset. We categorize IDPs into ten broad categories according to the major regions of the brain and use them as regression targets.
Table 4. Classification results on the CC-CCIT, CTRG-C, and CTRG-B datasets. We compare Raptor and state-of-the-art meth- ods on other publicly available medical volume datascts with an- notations. We report AUROCT and raw accuracy (ACC1) (best scores in bold, second best underlined scoring follows MedM- NIST standard).
Baselines. We evaluate a 3D ResNet-50 trained from scratch, which was top performer among baselines in the original 3D MedMNIST benchmark (Yang et al., 2023). We additionally fit a 3D ViT on each dataset. To improve the performance of the ViT on the small datasets, we perform masked autoencoder training (MAE) on each training set first, and then fine-tune the model on each downstream task.
CC-CCH
CTRG-C AUC
CTRG-B |ACC
METHODS
AUC
ACC
ACC|
ACC
RESNET
0.933
0.792
0.580
0.721
0.592
0.791
MAE
0.937
0.813
0.529
0.275
0.601
0.800
MISFM
0.975
0.835
0.548
0.690
0.651
0.643
Pretrained models. We fine-tune several state-of-the-art methods that underwent extensive pretraining for medical volumes, summarized in Table 2. The methods include SLIVIT (Avram et al., 2024), SuPreM (Li et al., 2024), Merlin (Blankemeier et al., 2024), MISFM (Wang et al., 2023), and VoCo-L (Wu et al., 2024a). Collectively, our baselines have been demonstrated in prior works to outper- form existing pretraining strategies such as SimCLR (Chen et al., 2020), and standard architectures for voxels such as Minkowski (Choy et al., 2019), standard SwinUNETR (Hatamizadeh et al., 2021) and UniMiSS (Xie et al., 2024). For models which do not have a classification or regression head, we add single linear layer above their latent space to enable the downstream tasks.
SUPREM
0.988
0.940
0.613
0.759
0.717
0.830
SLIVIT
0.986
0.906
0.571
0.719
0.656 0.834
VoCo
0.881
0.733
0.526
0.750
0.636 0.802
MERLIN
0.933
0.815
0.567
0.750
0.591
0.833
RAPTOR-B
0.992
0.954
0.600
0.754
0.685
0.842
RAPTOR
0.997
0.955
0.620
0.767
0.711 0.842
(consistent with Yang et al. 2023), and use Pearson's 722 for all regression datasets. For all methods, we reserve the validation set of the datasets for hyperparameter tuning, and report all final accuracies on the test set. For the 3D Medical MNIST dataset, we use the predetermined data splits from the benchmarks, allowing a direct comparison with prior and future works. For all other datasets without the standard splits, we use random training, validation, test splits of 60%/20%/20%.
To perform downstream tasks using Raptor embeddings, we train logistic regression models with L.2 regularization (tuning penalties of 0.01, 0.1, 1.0, 10.0, and 100.0 based on the validation set) for classification tasks", and MLPs (tuned up to 3 layers based on the validation set) under an MSE loss (described further in Appendix A.2) for regression tasks. This workflow is visualized in Figure 4.
4.2. Main results: Classification
Raptor achieves strong performance across classification tasks (3D MedMNIST, CC-CCII, and CTRG; Tables 3 & 4), evaluated using AUROC and accuracy as in the MedMNIST benchmark (Yang et al., 2023). It obtains the highest accu- racy in 6 of 9 datasets and ranks among the top two on 8 of (by AUC). When combined, Raptor and Raptor-B achieve the best-reported AUCs on five 3D MedMNIST categories.
Evaluation method. We measure AUROC (AUC) and accuracy (ACC) to benchmark all classification datasets
*I *Loiistic Regression scikit-leam
6

---

Random Planar Tensor Reduction
Table UKBB Brain IDP regression benchmark. We compare Raptor with state-of of the art methods on predicting quantitative traits
derived from ten different groups of regions from the brain (best scores in bold, second best underlined).
WHITEM
GRAYM
CEREB
AMYG
HIPPO
CORTEX
GYRUS
PALL
CAUD
ThAl
AvG.
METHODS
2²
RESNET
0.417
0.562
0.193
0.072
0.108
0.125
0.099
0.055
0.162
0.134
0.193
MAE
0.036
0.045
0.072
0.036
0.040
0.043
0.032
0.012
0.037
0.036
0.039
MISFM
0.418
0.624
0.276
0.089
0.145
0.236
0.209
0.087
0.166
0.164
0,242
SUPREM
0.646
0.696
0.330
0.109
0.163
0.275
0.256
0.067
0.255
0.195
0.299
SLIVIT
0.474
0.694
0,258
0.134
0.190
0.268
0.213
0.053
0.192
0.174
0.265
VoCo
0.225
0.375
0.189
0.071
0.113
0.059
0.048
0.043
0.060
0.075
0.126
MERLIN
0.622
0.734
0.335
0.127
0.180
0.313
0.269
0.093
0.247
0.210
0.313
RAPTOR-B
0.614
0.742
0.398
0.185
0.247
0.355
0.314
0.116
0.331
0.258
0.356
RAPTOR
0.681
0.777
0.437
0.170
0.262
0.404
0.340
0.142
0.381
0.300
0.389
Models that leverage pretraining (MISFM, SuPreM, SLIVIT, VoCo, Merlin, Raptor) generally score higher than those trained from scratch (3D ResNet and MAE). Across all clas- sification datasets, Raptor on average improved metrics by +2% over SuPreM, +4% over MISFM, +9% over Merlin, +10% over VoCo, and + +33 over SLIVIT.
10-
0.
AUROC
0.5
4.3. Main results: Regression
Merlin
SuPreM
- VoCo
We evaluate regression performance using y2 across 10 brain regions from UK Biobank Brain MRI volumes and their sociated IDPs (Table 5). We observe that Raptor shows strong predictive accuracy volumetric traits, achieving the highest ,2 scores 9 out of 10 regions. Raptor-B ranks second overall, outperforming Raptor in one region (Amygdala). Merlin and SuPreM follow as the next-best performers. Interestingly, the ResNet baseline bests several pretrained models, suggesting that current volumetric pre- training strategies may not generalize well between imaging modalities such CT and MRI. Models generally perform better on broader volumetric measures (e.g., white matter, gray matter) than on more localized regions (e.g.. Hippocam- pus). Compared to other methods, Raptor improves average performance by +24% over Merlin, +300% over SuPreM, +477 over SLIVIT, +61% % over MISFM, and +100% relative to the remaining baselines. Full performance com- parisons are visualized in Figure 1, with additional metrics provided in Appendix 10
MISFM
SLMT
10
200
500
1230
Training samples (log scale)
Figure 5. Prediction accuracy with limited training data. We measure the effect of limiting the number of training samples between 10 500 the final test accuracy the Synapse dataset. Shaded area represents 95% CI.
(0.729) of the performance achieved with the full 1,230 samples (0.943). With 100 samples, Raptor obtains 88% (0.838) of the full-sample performance. Results for addi- tional datasets show similar trends (Figure A.2).
4.5. Efficiency of the embeddings
We find that Raptor-B demonstrates exceptional capabilities despite its representation being 90% smaller than Raptor and 97% smaller than that of SuPreM (next-best, Figure 6). In classification, Raptor-B matches SuPreM in average ac- curacy, and surpasses in regression. We also highlight the efficiency of Raptor with 100 projections it consistently average while being smaller,
4.4. Effectiveness in cata-constrained settings
We observe that Raptor can generalize effectively to down- stream tasks that have limited number of samples a common scenario in the medical domain (Figure 5). We measure the AUROC of predictions using a single linear layer on top of Raptor embeddings given a limited number of samples (10, 50, 100, 200, 500) in the 3D MedMNIST Synapse dataset, which originally contains 1, 230 samples for training. Even with 10 samples, Raptor obtains 77%
outperforms all baselines including the feature map inferred by VoCo (3072 x 39), Our results highlight the potential of Raptor for compress- ing and sharing medical volumes, with the flexibility to adjust K as needed. As demonstrated in Section 5.1, Raptor achieves vate-of-the-and performance, while Raptor- B offers

---

Random Planar Tensor Reduction
Classification
Regression
0.40
Table 6. Reliability random projections. We vary the number of random projections for Raptor embeddings, and examine the variance the average AUC on the MedMNIST dataset.
0.86
0.35
AUROC
0.84
0.62
6et u
uift
256'
0.30
urit
64
256 uirtt
0.25
X
0.20
METHOD
K
SEED 1
SEED 2
SEED 3
STD.
0.78
0.15
RAPTOR
0.818
0.817
0.793
0.0116
0.76
198
15455 memory (log2 scale)
0.890
0.860
0.864
0.0133
Size memory llog2 scale)
10
0.866
0.896
0.876
0.0127
Raptor
SuPreM MISFM
Merlin VoCo
SLIVIT
100
0.901
0.899
0.900
0.0008
Raptor-B
X
150
0.898
0.897
0.897
0.0004
Figure 6. Overall accuracy of each method in the context of their embedding sizes. Average performance of cach method on the classification and regression datasets according to the size of the latent space inferred by each method (single precision floats).
Table 7. Effeet of using fewer views volumes. We vary which orthogonal views to parse when computing Raptor embeddings, and compare their average performance on the 3D McdMNIST dataset.
substantial space savings with only marginal decrease in performance.
METHOD
VIEWPOINT
AUC
ACC
RAPTOR
(A)XIAL
0.887
0,780
(C)ORONAL
0.862
0.814
5. Ablation Studies and Analyses
(S)AGITTAL
0.881
0.806
A.C
0.893
0.826
C,S
0.900
0.812
We quantitatively validate our embedding strategy by testing against alternative approaches. For all ablations, we report the average score the 3D MedMNIST dataset to assess their impact.
A,S
0.884
0.822
A,C,S
0.901
0.838
5.1. Reliability of random projections
5.3. Limitation on detectable features
We evaluate the sensitivity of our approach to the number of random projections used to compress the DINOv2-L latent representations. Specifically. we test K values of 1,5, 10, 100 and 150 across three different random seeds (Table 6). As expected, smaller K results in lower accuracy and higher variance across seeds. However, performance plateaus after K 100, with negligible variation between seeds (standard deviation < 0.001), indicating that random projections yield stable embeddings at each run. These results suggest that the embedding size can be substantially reduced with only minor tradeoff in stability, making the approach well-suited for resource-constoe constrained settings. In Table A.3, we further show the robustness of Raptor across larger range of seeds (integers from 0 9), showing consistent performance in up to 10 independent runs.
To explore the spatial sensitivity and resolution limits of our method, we conduct two controlled simulations using MNIST digits inserted into medical volumes from the 3D MedMNIST Nodule dataset.
In the Location task, we embed the same MNIST digit into two volumes, with varying lateral pixel (px) distances apart (64px to 8px), and test whether Raptor can distinguish the two.
In the Size task, we randomly insert an MNIST digit of varying sizes (from 64px to 8px) into volumes from the 3D MedMNIST dataset, and test whether Raptor can detect the presence of the digit.
For both tasks, we train binary classifiers on top of the Raptor embeddings. As expected, classification becomes in- creasingly difficult as spatial resolution decreases (Table 8).
5.2. Using multiple viewpoints
To assess the importance of multi-view processing, we com- pare models trained on fewer views (one or two) instead of all three (Table 7). We find that incorporating multiple views always leads to better classification. The largest per- formance gain is observed when increasing from one to two views, with a smaller but positive gain when using all three. This confirms the benefit of aggregating information from multiple viewpoints when parsing 3D volumes.
In the location task. Raptor achieves AUC 0.8 even at Spx resolution, highlighting the high spatial specificity of our embeddings. In the size task, performance is near perfect at the largest scale (> 0.9 AUC and ACC for 64px), but drops sharply for smaller digits (c.g., AUC ~ 0.5 at 16px), reflect- ing the limits of detection at low spatial scales. We further explore whether the choice of digit can lead to variability in the simulation results Table A.4.
8

---

Random Planar Tensor Reduction
X
X
scalable and generalizable solution for resource-constrainee settings, particularly in domains where annotated volumetric datasets are limited and computational efficiency is critical. Extending our framework beyond healthcare applications is a promising direction to explore in future work.
Same location?
Shape present?
Despite these advancements, certain limitations remain. For instance, modest performance on select datasets (e.g. Frac- ture3D) suggests that domain-specific priors or refining the axial sampling strategy could further improve downstream results. In supplementary analysis (Figure A.3), we identify contributing factors to such performance drops, pointing to immediate next steps for improvement. Nonetheless, the re- sults establish Raptor as a novel and impactful contribution to the intersection of foundation models and dense volumet- ric learning, offering a computationally efficient alternative to traditional 3D learning paradigms. These findings hold promise for broader applications, including multimodal in- tegration and large-scale analysis of medical data.
Figure 7. Examples of the "location" and "size" detection simu- lated tasks. The "location" task asks whether digit appears in an expected location in volume. The "size" task asks whether digit of varying sizes present a volume.
Table Results of "location" and "size" simulated bench- marks. We perform simulations that place MNIST digits at vary- ing locations and sizes in 3D MedMNIST volumes, and pose bi- nary classification tasks to detect the digits with the predctermined
settings.
METHOD
TASK
RESOLUTION
AUC
ACC
RAPTOR
LoC.
64PX
0.941
0.886
32PX
0.892
0.822
16PX
0.912
0.855
Impact Statement
8PX
0.783
0.701
We introduce Raptor, paradigm paradigm-changing framework that dramatically reduces the computational and data require- ments for high-dimensional medical volume analysis. By combining pretrained 2D foundation models with efficient random projection techniques, Raptor makes state-of of of-the- art volumetric analysis more accessible to researchers with limited computational resources. Our method not only im- proves performance across a wide range of medical imaging tasks but also enables scalable and cost-effective deploy- sssorrce-oonsrrainee settings. By
RAPTOR SIZE
64PX
0.988
0.932
32PX
0.758
0.682
16PX
0.520
0.507
8PX
0.474
0.476
6. Discussion
Raptor achieves state-of-the-art performance across range of medical imaging tasks, while compressing volumes into a representation space that is significantly smaller than those used in prior work. On average, Raptor outperforms all competing methods, achieving 3% accuracy gain the next best approach (SuPreM), despite using embedding that is 2.9x smaller. With further ten-fold compression, Raptor-B matches SuPreM's performance while reducing embedding size by 28.8x. Our results underscore the effi- cacy Raptor as paradigm-shifting framework for volu- metric data analysis that overcomes major challenges in the field by leveraging pretrained 2D foundation models.
ment in data-scarce lowering technical and computational barriers, Raptor has the potential to broaden participation in machine learning research and drive innovation in volumetric data analysis across healthcare and beyond.
Acknowledgments We thank Eric Liu, Jeffrey Chiang, Richard Border, and Oren Avram for their feedback and discussions throughout this project. This research was conducted using the UK Biobank Resource under applica- tion 331277. We thank the participants of UK Biobank for making this work possible. This research was sup- ported by grants from the National Institutes of Health (NIH) and the National Science Foundation (NSF). The work was supported by NIH grant R35GM153406 and NSF grant ZAREER-1943499 SL is supported by the Warren Alpert Fellowship. AG supported in part by NIH grant ROI M0IM330088 and ROI MH122255...
Raptor is an original approach for bridging the gap between widely studied 2D foundation models and 3D volumetric tasks a domain-agnostic manner. Unlike existing methods that rely on task-specific pretraining (e.g.. VoCo, Merlin) or complex architectural modifications, our approach bypasses these inefficiencies through projection- and reduction- based pipeline. A key strength of Raptor lies in its flexibility: the 2D foundational model it relies on can be seamlessly upgraded better models emerge.
This work provides both theoretical and empirical justifica- tions for our design choices, rather than relying on heuristics or purely practical considerations. We position Raptor as

---

Random Planar Tensor Reduction
References
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference machine learning. pp. 1597-1607. PMLR, 2020.
Armato III, S. G., McLennan, G., Bidaut, L., MrNytt-Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffmman, E. A., et al. The lung image database consortium (lide) and image database resource initiative (idri): completed reference database of lung nodules on scans. Medical physics, 38(2):915-931 2011.
Choy, C., Gwak, J., and Savarese, S. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3075-3084, 2019.
Avram, O., Durmus, B., Rakoez, N., Corradetti, G., An, U., Nittala, M. G., Terway, P., Rudas, A., Chen, Z. J., Wakatsuki, Y., et al. Accurate prediction of disease-risk factors from volumetric medical scans by deep vision model pre-trained with 2d scans. Nature Biomedical Engineering, pp. 1-14, 2024.
Cox, J., Liu, P., Stolte, E., Yang. Y., Liu, K., See, K. B., Ju. H., and Fang, R. Brainsegfounder: towards 3d foundation models for neuroimage segmentation. Medical Image Analysis, 97:103301, 2024.
Dao, T. FlashAttention-2:: Faster attention with better paral-
lelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
Bilic, P., Christ, P., Li, H. B., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Szeskin, A., Jacobs, C., Mamani, G. E. H., Chartrand, G., et al. The liver tumor segmentation bench- mark (lits). Medical Image Analysis, 84:102680, 2023.
Dasgupta, S. and Gupta, A. An elementary proof a
theorem johnson and lindenstrauss. Random Structures & Algorithms, 22(1):20-24)) 2003.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021. URL https:// openreview net / forum? dd= dd=iccbaNNTy...
Blankemeier, L., Cohen, J. P., Kumar, A., Veen, D. V.,
Gardezi, S. J. S., Paschali, M., Chen, Z., Delbrouck, J.- B., Reis, E., Truyts, C., Bluethgen, C., Jensen, M. E. K., Ostmeier, S., Varma, M., Valanarasu, J. M. J., Fang. Z., Huo, Z., Nabulsi, Z., Ardila, D., Weng, W.-H., Junior, E. A., Ahuja, N., Fries, J., Shah, N. H., Johnston, A., Boutin, R. D., Wentland, A., Langlotz, C. P., Hom, J., Gatidis, S., and Chaudhari, A. S. Merlin: A Vision Lan- guage Foundation Model for 3D Computed Tomography, June 2024. URL http::ttpp//... arxiv .rgg abs///////// 2406. 06512. 2. arXiv:2406.06511 [cs].
Ebrahimi, A., Luo, S., and Chiong, R. Introducing transfer learning 3d resnet- 18 for alzheimer's disease detection on mri images. In 2020 35th international conference on image and vision computing New Zealand (IVCNZ), pp. 1-6. IEEE, 2020.
Bycroft, C., Freeman, C., Petkova, D., Band, G., Elliott, L. T., Sharp, K., Motyer, A., Vukcevic, D., Delaneau, O., O'Coneell, J., et al. The uk biobank resource with deep phenotyping and genomic data. Nature, 562(7726): 203-209, 2018.
El Banani, M., Raj, A., Maninis, K.-K., Kar, A., Li, Y., Rubinstein, M., Sun, D., Guibas, L., Johnson, J., and Jam- pani, V. Probing the 3d awareness of visual foundation models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21795- 21806, 2024.
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E.,, Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: multimodal dataset for autonomous driving- In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11633 2020.
Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H. R., and Xu, D. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In Inter- national MICCAI brainlesion workshop, pp. 272-284. Springer, 2021.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leam- ing for image recognition. In Proceedings of the IEFE conference computer vision and pattern recognition. pp. 770-777 778, 2016.
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. Shapenet: An information 3d model repository. arXiv preprint Iix 2015.
Huang, Z., Zhang, J., and Ohn-Bar, E. Neural volumet- ric world models for autonomous driving. In European Conference on Computer Vision, pp. 193-213. Springer, 2025.
10

---

Random Planar Tensor Reduction
Jiang, B., Chen, X., Liu, W., Yu, J., Yu, G., and Chen, T. Motiongpt: Human motion as foreign language. Advances in Neural Information Processing Systems, 36: 20067-20079, 2023.
Milletari, F., Navab, N., and Ahmadi, S.-A. V-net: Fully con- volutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 565-5555777 5511. Ieee, 2016.
Jin, L.,, Yang, J. Kuang, K., Ni, B., Gao, Y., Sun, Y., Gao, P., Ma, W., Tan, M., Kang, H., al. Deep-leariig--ssiste detection and segmentation of rib fractures from ct scans: Development and validation of fracnet. EBioMedicine, 62, 2020.
Moor, M. Huang, Q.. Wu, S., Yasunaga, M., Zakka, C., Dalmia, Y., Reis, E. P., Rajpurkar, P., and Leskovec, J. Med- fled-flamingo: A multimodal medical few-shot learner. July 2023. URL https: :s:t:ss//// wtt.a.owwgggaaaaaaaaaaaa.....laa / aaaabb///33 2307. 15189. arXiv:2307.1518..
Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Ning, J., Zhao, H., Lan, L., Sun, P., and Feng, Y. A computer-aided detection system for the detection of lung nodules based on 3d-resnet. Applied Sciences, 9(24); 5544, 2019.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C.,, Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 4015-4026, 2023.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint urXiv:2304.07199. 2023.
Lai, H., Jiang, Z., Yao, Q., Wang, R., He, Z., Tao, X., Wei, W., Lv, W., and Zhou, S. K. E3d-gpt: Enhanced 3d visual foundation for medical vision-lame vision-language model. arXiv preprint ar Xix:v22410.1 44200, 2024.
Qayyum, A., Benzinou, A., Mazher, M., Abdel-Nasser, M., and Puig, D. Automatic segmentation of head and neck (h&n) primary tumors in pet and ct images using 3d- inception- resnett model. In 3D Head and Neck Tumor Segmentation in PET/CT Challenge, pp. 58-67. Springer, 2021.
Li, C., Wong, C., Zhang. S., Usuyama, N., Liu, H., Yang. J., Naumann, T., Poon, H., and Gao, Llava-med med: Training large minguage-and-viste one day. arXiv preprint arXiv:2336600880 2023.
assistant for biomedicine in
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748 8763. PMLR, 2021.
Li, W., Qu, C., Chen, X., Bassi, P. R., Shi, Y., Lai, Y., Yu,
Q. Xue, H.. Chen, Y., Lin, X., et al. Abdomenatlas: A large-scale, detailed-annotated, & multi-center dataset for efficient transfer learning and open algorithmic bench- marking. Medical Image Analysis, 97:103285, 2024.
Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., and
Koltun, V. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-datase transfer. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 44(3), 2022.
Liu, H., Li, C., Wu, Q.. and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024a.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models, 2021.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Saharia, C., Chan, W., Saxena, S., Li, L.,, Whang. J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36479-36494, 2022.
Liu, Z., Xu, H., Woicik, A., Shapiro, L. G., Blazes, M., Wu, Y., Lee, C. S., Lee, A. Y., and Wang, S. Octcube: a 3d foundation model for optical coherence tomogra- phy that improves cross-dataset, cross-disease, cross- device and mooss-modality analysis. arXiv preprint arXiv:2408.11 12227, 2024b.
Sajid, M., Razzaq Malik, K., Ur Rehman, A., Safdar Ma- lik, T., Alajmi, M., Haider Khan, A., Haider, A., and Hussen, S. Leveraging two-dimensional pre-trained vi- sion transformers for three-dimensional model generation via masked autoencoders. Scientific Reports, 15(1):3164, 2025.
Ma, J., He, Y., Li, F., Han, L., You, C., and Wang, B. Seg- ment anything in medical images. Nature Communica- tions, 15(1):654, 2024.

---

Random Planar Tensor Reduction
Shaker, A. M., Maaz, M., Rasheed, H., Khan, S., Yang, M.- H., and Khan, F. S. Unetr++: delving into efficient and accurate 3d medical image segmentation. IEEE Transac- tions on Medical Imaging, 2024.
Yang, J., Huang, X., He, Y., Xu, J., Yang, C., Xu, G., and
Ni, B. Reinventing 2d convolutions for 3d images. IEEE Journal of Biomedical and Health Informatics, 25(8): 3009-3018, 2021.
Tang, Y., Yang, H., Zhang, L., and Yuan, Y. Work like doctor: Unifying scan localizer and dynamic generator for automated computed tomography report generation. Expert Systems with Applications, 237:121442. 2024.
Yang. Shi, R., Wei, D., Liu, Z., Zhao, L.., Ke, B., Pfister,
H., and Ni, B. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
Turnbull, R. Using 3d resnet for detecting the presence and severity covid-19 from ct scans. In European Conference on Computer Vision, pp. 663-676. Springer, 2022.
Yang. X., Xia, D., Kin, T., and Igarashi, T. Intra: 3d intracra- nial aneurysm dataset for deep learning. In Proceedings of the IEEE/CVF Conference Computer Vision and 2020.
Pattern Recognition, pp.
Wang, G., Wu, J., Luo, X., Liu, X., Li, K., and Zhang, S. Mis-fm: 3d medical image segmentation using foun- dation models pretrained a large-scale unannotated dataset. arXiv preprint arXiv:2306. 16925, 2023.
Zhang, K., Liu, X., Shen, J., Li, Z., Sang. Y., Wu, X., Zha, Y., Liang, W., Wang, C., Wang, K., al. Clinically applicable ai system for accurate diagnosis, quantitative measurements, and prognosis of covid-19 pneumonia using computed tomography, Cell, 18366):1433-1433, 2020.
Wasserthal, J.. Breit, H.-C., Meyer, M. T., Pradella, M., Hinck, D., Sauter, A. W., Heye, T., Boll, D. T., Cyriac, J., Yang, S., et al. Totalsegmentator: robust segmenta- tion of 104 anatomic structures in et images. Radiology: Artificial Intelligence, 5(5), 2023.
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022.
Wu, B., Xiao, Q., Liu, S., Yin, L., Pechenizkiy, M., Mocanu, D. C., Van Keulen, M., and Mocanu, E2enet: Dynamic sparse feature fusion for accurate and efficient 3d medical image segmentation. NeurlPS, 2023.
Zhu, Z., Ma, X., Chen, Y., Deng, Z., Huang, S., and Li, Q. 3d-vista: Pre-trained transformer for 3d vision and text alignment. ICCV, 2023.
Wu, L., Zhuang, J., and Chen, H. Large-scale 3d medical image pre-training with geometric context priors. arXiv preprint arXiv:24/00 00....0.09900 2024a.
Wu, L., Zhuang, J.. and Chen, H. VoCo: A Eiffeeivet-fffective Volume Contrastive Learning Framework for 3D Medical Image Analysis, April 2024b. URL http:/ htap://arxivv www.ii..og//abb.. 17300. arXiv:2402.17300 [eess].
Xie, H., Li, J., and Xue, H. A survey of dimensionality reduction techniques based on random projection. arXiv preprint arXiv: (00v://66.477,1 2017.
Xie, Y., Zhang, J., Xia, Y., and Wu, Q. Unimiss+: Universal medical self-supervised learning from cross-dimensional unpaired data. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
Xing, Z., Ye, T., Yang, Y., Liu, G., and Zhu, L. Segmamba: Long-range sequential modeling mamba for 3d medi- cal image segmentation. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention, pp. 578-588. Springer, 2024.
Xue, S. and Abhayaratne, C. Region-of offinerrsst aware 3d resnet for classification of covid-19 19 chest computerised tomography seans. IEEE Access, 11:28856-2855 28872, 2023.
12

---

Random Planar Tensor Reduction
A. Appendix
A.I. Using alternate image foundation models in Raptor
Table A.I. Effectiveness of other 2D ViT encoders
METHOD
ENCODER
AUC
ACC
RAPTOR
SAM
0.850
0.800
CLIP
0.870
0.813
LLAVA- MED
0.869
0.812
MEDSAM
0.872
0.808
DINOv2-L
0.907
0.835
As our approach agnostic to the choice of the image foundation model, we explore CLIP (Radford et al. 2021; also utilized in Moor et al. 2023), the encoder of SAM (Kirillov al., 2023), the encoder of Llava-Med (Li al., 2023), and the encoder of MedSAM (Ma al., 2024). Of the explored options, embeddings obtained using DINOv2 lead to the highest accuracies in 3D MedMNIST (Table A. We also find that embeddings based on the McdSAM encoder do not lead to significantly higher accuracies than those based on SAM, despite MedSAM being directly trained on medical imaging modalities. Overall, we note that our findings are roughly in line with El Banani et al. (2024) and Oquab et al. (2023), which determined that DINOv2 currently yields the most informative features for general-purport downstream tasks.
A.2. Multi-Layer Perceptron for downstream
We train a multi-layer perceptron (MLP) to predict brain imaging-derivee phenotypes (IDPs) in the UK Biobank. The configuration of the MLP is described below for the case of K=100 Raptor embeddings. We use Py to implement and train the MLP module. Training proceeded up to 50 epochs using the mean-squaree error loss. The model weights are checkpointed only when accuracy improved on the validation split of each dataset.
MLP
(compare_bn): BatchNormld (199)
(model): : Sequential
(0): sinear 256)
(1): BatchNermldd(56)
(2): : ReLU()
(3) : linear (256, 256)
(4): ()
(5): linear (256, 162)
Listing 3-layer multi-layer perceptron
A.3. Quantifying how much medical features can be captured using DINOv2-L
We explore one way of quantifying how effectively DINOv2-L extracts features related to the medical domain. The high-level intuition that the semantically meaningful features of medical volumes are still part of the universal image features (edges, textures, shapes), which a broad 2D encoder has already learned from large-scale, diverse data. Raptor uses generic image foundation models like DINOv2-L. to capture any meaningful image features and performs low rank approximations retain only the relevant ones. This is in contrast many existing medical image/volume models that try to learn such mappings from scratch fine-tune general model be biased towards them.
To illustrate this point, we projected both generic (ImageNet) and medical (3D MedMNIST) embeddings onto the principal components derived from the generic dataset. In the table below, we calculate the total variance explained as we add more PCs (Table A.2). Although the medical data do not align with the very top principal directions, the variance the medical dataset increases as more components are added (eventually matching the generic variance), demonstrating that medical image features lie within the general image feature space.
13

---

Random Planar Tensor Reduction
Table A.2. Effectiveness DINOv2-L in the medical domain. Here we compare how much of variance explained two datasets' DINOv2-L embeddings, we add more principal components (PCs) from the generic image embeddings. As can be seen below, with more generic image embedding PCs, variance the medical dataset embeddings can be explained in near full.
#PCs
General Explained Var.
Medical Explained Var.
Ratio (Medical/Ginneral)
100
0.613
0.225
0.367
200
0.765
0.397
0.518
500
0.926
0.734
0.793
1000
0,999
0,994
0.996
Table A.3. Reliability random projections. We vary number random projections for Raptor embeddings, and examine the variance in the average AUC on the MedMNIST dataset.
METHOD
K
SEED
2
3
4
5
6
7
8
10
AVG.
STD.
RAPTOR
1
0.818
0.817
0.793
0.823
0.831
0.818
0.831
0.814
0.801
0.831
0.818
0.0121
5
0.890
0.860
0.864
0.869
0.869
0.877
0.868
0.858
0.872
0.866
0.869
0.0086
10
0,866
0.896
0.876
0.875
0.882
0.879
0.871
0.876
0.877
0.886
0.878
0.0078
100
0.901
0.899
0,900
0.898
0.905
0.891
0.899
0.898
0.900
0.898
0.899
0.0034
150
0.898
0.897
0.897
0.894
0.898
0.903
0.902
0.905
0.902
0.904
0.900
0.0034
A.4. MRI Anatomical Terminology
Figure A.I. Brain Anatomy, Axial, Sagittal, and Coronal views brain MRI volume (source: http://ccsc.cddu/necd/nooooggg
Table A.4. Variability in simulated settings under specific digits. We repeat the "Size" simulation experiment given specific digits (0. 1, and 8) found in the MNIST dataset.
Method
Size
Avg.
Std.
Raptor
64px
0.798
0.678
0.753
0,743
0.049
32px
0.580
0.559
0.575
0.571
0.009
16px
0.552
0.514
0.517
0.527
0.017
8px
0.506
0.509
0.508
0.508
0.001
14

---

Random Planar Tensor Reduction
10
+- Repter
Markn
0.7-
SuProM
VoCo
0.5
-x- MEPM
-4- SLNV
0.5
AUROC
0.6
AUROC
0.5
0.3
0.2
6.1
SUVT
50
100
200
500
1230
10
100
200
500
2413
10
1.00
200
500
1204
Training samples (log scale)
Training samples
Training samples (log scale)
(log scale)
Figure A.2. Prediction accuracy with limited training data. We measure the effect of limiting the number of training samples between 10 500 the final test accuracy the Synapse, CC-CCTI and UKBB (White Matter regression) datasets. Shaded represents 95% CI.
A.5. Formal Guarantees on Random Projections
To give a more principled view of our random projection step, we restate version of the Johnson-Lindensee (JL) lemma (Dasgupta & Gupta, 2003) and show how it applies in our multi-view volumetric setting.
Full Proof of the Johnson-Lindenstrane Lemma Let X1, X,,..., Xn, E Rd be arbitrary vectors, and fix 0 < 6 < 1. Let R E R xd be a random projection matrix whose entries are drawn i..d. from a sub-Gaussian distribution (e.g., Gaussian) with zero mean and variance x² chosen so that T.
[ = ||z|| for any z E Rdd. We show that K is on the order of
- log(n), then with high probability for all pairs j.
(1 - ||x - x <
Rx; Rx 11 S (1++) |x xy113.
Single-Pair Analysis. Consider single pair (i,j) and let X; Xj. The vector Rz has i.i.d. sub- Gaussian components with mean 0 and variance the order of || K. particular, if has Gaussian entries .N(0,, 1/K), then each (re, z) (0, [)) and Rz|| follows scaled xK) distribution with expected value ||z||2-
I: Concentration Inequality. Sub-Gaussie concentration (or standard x2 tail bounds) implies that for any >> 0, there
exist positive constants C1, so that
Pr| Pr|
|| 2 e ||]] s a exp(exec 8² K).
Hence, [Rz]2 is within (1 + 2 with high probability for one pair (i,j).
III: Union Bound over All Pairs. There are (%) < n² /2 distinct pairs (i,j). A union bound over these pairs requires
n² ci exp(-c2e*
(-22 2)) 1,
which rearranges
K -2 Cog()) for some constant C. This guarantees that with high probability, for all i + j.
(1
||x; - x11 < Rxx Rx,1 (1 +) ||x - xx,1
This completes the proof that random projections preserve pairwise distances among (xi...., up (1 E) distortion.
15

---

Random Planar Tensor Reduction
A.6. Mathematical formulation of Raptor
Assume DxDx be a 3D volumetric sean. For each planar direction (defined along the axes 2, y, 2), we get D slices S {Sa 1 S.a...., Sap J. Sy. Sz. We use pretrained ViT (e.g.. DINOv2-L) encoder that takes each slice 8ij to generate feature vector Ztz:
6(-): DxD
D - R
In practice, the encoder outputs a token-wise embedding for a grid of patches, but we may flatten it to a m dimensional vector (m = dy". where d is the latent dimension of token embedding, p² is the number of patches), These feature vectors are semantically rich, meaning they encode information that is reflective of real-world semantics, such as contours of similar objects, distributions of certain features, etc. Ideally we would like leverage all embeddings of the slices for all three planar views; however this is computationally prohibitive:
(ø(S.), ((S,),
((S)), o(S))) E Rx3xxpp
Instead, Raptor aggregates the slice-level token embeddings and low-rank approximates them for each planar view (note that due to the linearity of random projection and mean aggregation, the two operations can be done in any order). Reducing the planar view by mean-pooling the slices gives
D Zij. where E v.
2 D j=1
Define E Rxdxd Random Projection matrix, whose entries are sampled from the standard normal.
u = R2; Kv' K kdd, Rat ~ 10,
Concatenating the three low-rank approximated planar embeddings gives the final Raptor embedding:
4 ((x) = concat (u,r, By Uz] E k3Kp²
Or alternatively:
D
((x) concat
1 R Σ (Bx,) ), R
Σ D
1 (BB ) R D
Σ ((,,,)
D
Because (with the exception of ((-)) all our mappings are linear, we can analyze the pairwise distances in different scales: (1) the pairwise distances between two volumes, and (2) the pairwise distances between two classes (in downstream task of classification).
First, we formulate how Raptor is able to preserve the difference in the semantic embeddings of two distinct volumes the three orthogonal axes. Let aa Th be two 3D volumetric scans. Their high-dimensional semantic embeddings are given as:
((xa) = (0( (SS)) ((S)), d(x;) = (o(S)), (S (SS)), slxa), ((xr)) E 3xxDx
For each axial view, the difference given as (taking the Frobenius norm)
[(((z))= ((xr)= - o(St))l
-
D
- (55))) }
((s -
The Raptor embeddings of the two volumes are given as:
1 D
D
4 (x.) = concat
Σ
RO(sE,)
Roiss
)
Σ RØ(8
E RRpp
D
83
D
1=1
j=1
j=1
Φ (2%)
concat
1
Σ
R
Σ
R b R
R
1 23)
E 3kp?
D
D
J=1
1=1
j=1
16

---

Where S, = EkE As is the partial sum of the distances. Geometrically, this means that at each slice level, A, deviates from Sj-1 by at most arccos (az). This ensures that the slices (and their embeddings due to the Lipschitz bound) are globally smooth, and there are no abrupt (negative) changes in the differences at each slice level. Now we derive the bounds Random Planar Tensor Reduction
and the distance between the along the axis is:
(((x))= (b(0))))
1
Σ Ro( - -E Ro(
-
1 D
R ((66) - -60 Σ
-
Define A, = ((5 (11ss,)-00 - (( as the difference in the slice-wise DINOv2-L embeddings. Then we may rewrite the distances in the raw embeddings and Raptor embeddings for one of the axial views as
D Σ
draw
Σ DJ Raptor
1 D
RA
If A, are aligned in the same direction, then there exists a trivial mapping between drow and dapporr but if there are mis-alignments (i.e., Aj and Ay are in opposite directions), then that is no longer the case. Given that we are working with 3D volumetric scans of real objects that have certain properties, we make additional assumption that the slices are "smooth". Because DINOv2-L. is a (locally) Lipschitz mapping, smoothness in the raw volumes also mean the embeddings of the slices are smooth. Assume that for some aj > 0 values (again taking the la norms in flattened form or Frobenius norm in matrix form), the following holds:
AJ S 2 a |ا j=
for daptor by induction. In the base case we have:
S = A1
| = |
For the j-th slice,
Sj+1 S3+ + Ajt1,
|| = ||S - || 212 + 25} Aj+1
2 ||S, || 2 + 2a, ||5 ||ر
2 (||)) + 0g ||4
(since 0 < <a 1))
+ ||5 2 || + a, ||د+
Which we can unroll to the j = D slice:
D
D
||S
Σ
A
> Σ ag ||ا
j=l
j=l
Σ a | 2
(since aj ||4,1 2 0)
1=1
Σ |1,,1
Ommn
0₁ 2 Omin
1=1
Giving us lower bound in aggregating the Ay:
D
Σ
A
2
amin
(({(99)
<(S))|F -rmnndraw
/=1
17

---

Random Planar Tensor Reduction
Applying random projection gives the distance in the Raptor embeddings of the two volumes. Therefore we get the final lower bound in the distances:
D
2 (1 (1-6) D
D
(1 (1-8) D
( ((Db)=ll =
R
Σ A
minnn
Σ ||,11
Aminnraw
D
Similarly, we can derive the upper bound of the distance in the Raptor embeddings.
Σ 4
|ارا|
(triangle inequality)
VD
|
(Cauchy-Schwarte inequality)
Giving a similar bound as before with respect to the raw Raptor embedding distance. The final bounds on both ends are:
(1- amindraw S dRaptor
(1 + draw VD
D
In short, under the assumption that aj > O, the Raptor difference does not vanish unless there no difference in the raw embeddings of all the slices between volumes, nor does it blow up there is large distinction in the original embedding space. This means that if there are any differences between the two volumes in the semantically DINOv2-L space, these differences will be preserved in embeddings. The presence of upper- and lower-bounds of daptor a function of draw suggests that it serves as fuzzy mapping of the distance in the raw embedding space.
Finally, we show that Raptor preserves class separability in the original DINOv2-L space. For a binary classification dataset, assume there are classes A, B, whose cluster centers are given by μA PB. The cluster centers (in the DINOv2-L space) are set apart by I| pB 2 B. Then the distance between the cluster centers are given by
(1-E) ammn 3 cB
(((())) (pp))
1 2
Where >> is some constant. That is, the cluster centers remain ((B))-separated even after aggregation and RP, thereby guaranteeing class separability. Empirically, we find that this generally the case for the datasets we have benchmarked on, as Raptor generally exhibits superior performance in classification tasks.
However, when there is violation in our assumptions (e.g., there is no slice-level differences in the embedding space between two volumes or such differences are not "smooth"), the lower-bound of the Raptor distance is no longer true (the upper bound still holds), and it may vanish to zero. We conjecture that this may be the case for datasets where Raptor performs relatively poorly (e.g., 3D FNAS:;reMNIST); in Figure A.3, we estimate the ay values of volume pairs for the 3D MedMNIST datasets to bolster our point. As seen the figure, there are lot of aj in the axial view of the Fracture dataset. Intuitively, what this means is that some portion of daptor will be canceled out as we average over the slices, thereby reducing the distance between two semantically distinct volumes. For other datasets where this conjecture that Raptor well approximates the distances in the raw embedding space, and results in superior downstream performance.
significant,
18

---

Random Planar Tensor Reduction
Histogram of aj estimates for 3D MedMNIST dataset
fracture
adrenal
vessel
2750
1000
030
Lawest 5%;
2500
Lowest 5%
Lowest 5%
Axtal: 00.118
Axial: 0.000
Axial:
300
Sagittal: 0.035
1250
Coronal: 0.000
Sagittal: 0.00C
1000
200
750
500
100
250
View
-0.50 -.:0--25
6.00
0.25
2.50
0.75 1:00
-0
-0.50-0.25 0.00 0.25
0.50 0.75 3.00 R)
-0.50-0:25 0.00
0.25 0.50 0.25 1.00
Axial
Coronal Sagittal
nodule
organ
synapse
140
220
170
Lawest 5%
Axial: 0.177
300
Lowest E%
Axtal: 0.152
Axial: 0.115
Lowest 5%c
100
Coranal: 0.138
Cononal: 0.159
Coronel: 0.046
Sagittal: 0.132
Sagittal: 0.204
Segittel: 0.111
60
60-
60
60
40-
20
20-
0.00
0.75
0
-0.50-0.25
0.50
0-1
-0.50-077 nno ais aso 0.75
9
Figure A.3. Histogram of a, estimates for pairs of volumes in each 3D MedMNIST dataset. The a, estimates for the fracture dataset slightly violates our core assumption alignment (a, 0), especially in the Axial view. This means that the Raptor distance dRaptor no longer a good proxy for dra.. as some slice-level embedding differences between volumes are canceled out in dRapttor The lowest 5% quantile values of o; is annotated as well. In many datasets, there are peaks of az O, mostly due to the empty slices the volumes.
A.7. Runtime complexity of Raptor
We compute Raptor embeddings by taking the average of patch-level tokens inferred by a pretrained ViT in three orthogonal directions, then applying random projections to further compress the semantic information preserved in the tokens. Given E Dxdxp', the intermediate tensor inferred by applying the ViT to every observed image, we first apply the average step. For a dataset with N volumes, reducing the sampling direction of each tensor (the dimension of size D) requires iteration over the tensor, incurring a cost of O(p² D(p2DdN.. The resulting intermediate tensor is of size 3 x d x p². Then, the projection step with random matrix R which projects down the token dimension (of size d) incurs cost of OppdKKN) In total, the cost is Dy22Dd + N2 Oly K)).
A.8. Comparison to Alternate Compression Methods
Why not just use PCA? While PCA can also reduce dimensionality, it requires (1) data-dependent fitting, (2) storing the eigenvector matrix for all new data, and (3) does not come with distance-preservation guarantees for all points in general. PCA preserves variance in directions of largest eigenvalues but might discard subtle signals that are crucial for certain tasks. In contrast, random projection has uniform guarantees over all distances and does not require any optimization.
Why not train a smaller parametric layer? Another option is to learn a small linear or MLP-based projection on top of the 256-d tokens. This typically requires a labeled set or at least a self-supervised objective, incurring additional training time and potential overfitting (especially in resource-scarce domains). By contrast, random projection is train-free and general-purposs
Why not use nonlinear aggregation? A simple linear aggregation method allows for many "nice" properties, such as Lipschitz continuity and commutativity; not to reiterate the similar argument against training nonlinear aggregator.
Why is it necessary to aggregate over the slices? Because we use random projection to embed the local geometry of the tokens, it doesn't i matter whether we aggregate over the slices or patches (or any similar ways), since with high probability, the pairwise distances of the volumes will be preserved (under certain assumptions, e.g., Appendix A.6).
19

---

Random Planar Tensor Reduction
A.9. Dataset Characteristics
Table A.5. Summary of benchmarked datasets. For classification tasks, the balances of the classes are also reported.
DATASET
SIZF
TASK
CLASSES
6
ORGAN
1.742
CLS
11
11%
11%
11% %
11%
10%
10%
10%
4%
3%
NODULE
1,663
CLS
2
79%
20%
FRACTURE
1,584
CLS
3
43%
37%
19%
ADRENAL
1,370
CLS
2
76%
23%
VESSEL
1,908
CLS
2
88%
11%
SYNAPSE
1.759
CLS
26%
CC-CCII
2,471
CLS
3
41 %
36%
21%
CTRG-C
1,804
CLS (MULTI)
4
%
87%
67%
29%
CTRG-B
6,000
CLS (MULTI)
3
92%
61 %
13%
WHITE MATTER
1,476
REG
4
GRAY MATTER
1,476
REG
6
CEREBELLUM
1,476
REG
28
AMYGDALA
1,476
REG
8
HIPPOCAMPUS
1,476
REG
8
CORTEX
1,476
REG
34
GYRUS
1,476
REG
50
PALLIDUM
1,476
REG
8
CAUDATE
1,476
REG
8
THALAMUS
1,476
REG
8
20

---

Random Planar Tensor Reduction
A.10. Additional Results
Table A.6. AUPR (precision-recall) for all classification benchmarks.
METHODS
ORGAN
NODULE
FRACTURE
ADRENAL
VESSEI.
SYNAPSE
CC-CCI
CTRG-C CTRG-B
MISFM
0.940
0.850
0.546
0.853
0.880
0.873
0.934
0.658
0.659
SUPREM
0.994
0.841
0.484
0.883
0.910
0.892
0.978
0.704
0.711
VoCo
0.943
0.780
0.544
0.884
0.656
0.831
0.811
0.634
0.648
MERLIN
0.845
0.784
0.522
0.803
0.718
0.821
0.874
0.679
0.647
RAPTOR
0.994
0.884
0.503
0.889
0.906
0.929
0.990
0.704
0.688
Table A.7. (Micro averaged) AUPR (precision-recall) for all classification benchmarks.
METHODS
ORGAN
NODULE FRACTURE
ADRENAL
VESSEL
SYNAPSE
CC-CCII CTRG-C CTRG-B
MISFM
0.922
0.937
0.509
0.761
0.970
0.536
0.913
0.776
0.819
SUPREM
0.995
0.942
0.487
0.940
0.986
0.925
0.983
0.846
0.914
VoCo
0.955
0.893
0.543
0.943
0.937
0.890
0.776
0.821
0,898
MERLIN
0.865
0.888
0.575
0.895
0.960
0.887
0,876
0.824
0.893
RAPTOR
0.994
0.956
0.552
0.886
0.983
0.959
0.990
0,846
0.917
Table A.8. (Micro averaged) AUROC scores for all classification benchmarks.
METHODS
ORGAN
NODULE FRACTURE
ADRENAL,
VESSEL
SYNAPSE
CC-CCII CTRG-C
CTRG-B
MISFM
0.988
0.936
0.708
0.749
0.972
0.442
0.945
0.720
0.765
SUPREM
0.999
0.941
0.682
0.937
0.986
0.929
0.990
0.802
0.902
VoCo
0.994
0.903
0.716
0.940
0.944
0.893
0.867
0.780
0.886
MERLIN
0.982
0.905
0.731
0.901
0.959
0.889
0.934
0.794
0.884
RAPTOR
0.999
0.954
0,709
0.900
0.982
0.960
0.995
0,803
0.904
21


Reviews:
[{"review_id": "6BmfuDm3UW", "reviewer": "Reviewer_28uY", "summary": "This paper presents a random projection-based strategy for generating embeddings from volumetric data. The approach leverages pre-trained 2D foundation models without requiring additional re-training or fine-tuning. The proposed embedding construction method is computationally efficient, and experiments conducted on ten datasets across multiple downstream tasks demonstrate strong performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and constructive comments, as well as their positive assessment of our method’s clarity and novelty. As noted by the reviewer, Raptor introduces a paradigm that demonstrates many advantages beyond existing works, and we verify our claims with a wide range of empirical results.\n\n> **Average score over 10 brain regions:**\n\nFor better clarity, we have now added a column for average performance on the regression tasks, reproduced below. Overall, Raptor most accurately predicts physiological measures given brain MRIs, and Raptor-B comes in as the second best (best score in bold, second best italicized).\n\n| Methods      | WhiteM   | GreyM    | Cereb   | Amyg    | Hippo   | Cortex  | Gyrus   | Pall    | Caud    | Thal    | Avg.   |\n|--------------|----------|----------|---------|---------|---------|---------|---------|---------|---------|---------|--------|\n|  $r^2$                                                                                     \n| ResNet       | 0.417    | 0.562    | 0.193   | 0.072   | 0.108   | 0.125   | 0.099   | 0.055   | 0.162   | 0.134   | 0.193  |\n| MAE          | 0.036    | 0.045    | 0.072   | 0.036   | 0.040   | 0.043   | 0.032   | 0.012   | 0.037   | 0.036   | 0.039  |\n| MISFM        | 0.418    | 0.624    | 0.276   | 0.089   | 0.145   | 0.236   | 0.209   | 0.087   | 0.166   | 0.164   | 0.242  |\n| SuPreM       | _0.646_  | 0.696    | 0.330   | 0.109   | 0.163   | 0.275   | 0.256   | 0.067   | 0.255   | 0.195   | 0.299  |\n| SLIViT       | 0.474    | 0.694    | 0.258   | 0.134   | 0.190   | 0.268   | 0.213   | 0.053   | 0.192   | 0.174   | 0.265  |\n| VoCo         | 0.225    | 0.375    | 0.189   | 0.071   | 0.113   | 0.059   | 0.048   | 0.043   | 0.060   | 0.075   | 0.126  |\n| Merlin       | 0.622    | 0.734    | 0.335   | 0.127   | 0.180   | 0.313   | 0.269   | 0.093   | 0.247   | 0.210   | 0.313  |\n| Raptor-B       | 0.614    | _0.742_  | _0.398_ | **0.185**| _0.247_ | _0.355_ | _0.314_ | _0.116_ | _0.331_ | _0.258_ | _0.356_ |\n| Raptor    | **0.681**| **0.777**| **0.437**| _0.170_ | **0.262**| **0.404**| **0.340**| **0.142**| **0.381**| **0.300**| **0.389**|\n\nWe again appreciate the reviewer for their words of support for our work. In addition to the improved table, we hope that the additional analyses we shared with the other reviewers have further demonstrated the capabilities of our method, and if deemed so, humbly request that the reviewer consider raising the score as a further vote of confidence.\n\n_(Below is in response to a point raised by reviewer zwUQ; we have moved this here due to the space limit, but we think that reviewer 28uY would also be interested to know.)_\n> **More evidence for effectiveness with limited training samples:**\n\nWe agree that a single dataset is insufficient to support our point. Hence we conducted the same experiments (varying training set size) for the CCCC-II dataset (classification) and the white matter category of the UKBB dataset (regression). As shown in the table below, while some methods eventually catch up as the sample size grows, Raptor maintains a clear advantage in low-data regimes for both tasks, underscoring our original claim of data efficiency (best score in bold, second best italicized).\n\n**UKBB White Matter ($r^2$)**\n| Sub | 10 | 50 | 100 | 200 | 500 | 1104 |\n|-|-|-|-|-|-|-|\n| SLIViT | _0.070_ | 0.155 | 0.206 | 0.241 | 0.437 | 0.474 |\n| VoCo | 0.068 | 0.048 | 0.091 | 0.099 | 0.178 | 0.225 |\n| Merlin | 0.028 | 0.123 | 0.314 | 0.177 | _0.629_ | 0.622 |\n| MISFM | 0.056 | 0.106 | 0.104 | 0.208 | 0.330 | 0.418 |\n| SuPreM | 0.059 | _0.305_ | _0.396_ | _0.557_ | 0.593 | _0.646_ |\n| **Raptor** | **0.193** | **0.414** | **0.446** | **0.588** | **0.634** | **0.681** |\n\n**CCCC-II (AUROC)**\n| Sub | 10 | 50 | 100 | 200 | 500 | 2413 |\n|-|-|-|-|-|-|-|\n| SLIViT | 0.483 | _0.861_ | _0.914_ | 0.936 | 0.956 | 0.986 |\n| VoCo | _0.638_ | 0.797 | 0.819 | 0.817 | 0.861 | 0.879 |\n| Merlin | 0.509 | 0.499 | 0.483 | 0.492 | 0.484 | 0.927 |\n| MISFM | 0.494 | 0.821 | 0.826 | 0.900 | 0.965 | 0.975 |\n| SuPreM | 0.631 | 0.821 | 0.906 | _0.939_ | _0.965_ | _0.988_ |\n| **Raptor** | **0.706** | **0.917** | **0.939** | **0.955** | **0.982** | **0.997** |", "claims_and_evidence": "1. Semantically meaningful embeddings for volumetric data can be obtained from 2D foundation models without extra training. \n\nThe analytical results and empirical experiment support this claim.", "methods_and_evaluation_criteria": "The paper evaluates the proposed method against comprehensive baseline models, including 3D ResNet, ViT, and MAE, as well as pre-trained models such as SLIViT, SuPreM, Merlin, MISFM, and VoCo-L. The approach is validated on both classification and regression tasks, demonstrating its effectiveness across multiple benchmarks.", "theoretical_claims": "I reviewed the computational complexity and found no issues. However, I did not verify the theoretical analysis provided in the Appendix.", "experimental_designs_or_analyses": "I reviewed the experiment settings and results and found them to be fair and thorough. The results are sound and effectively support the analysis and conclusions.", "relation_to_broader_scientific_literature": "The paper addresses a key challenge in extracting volumetric embeddings for biomedical images. By eliminating the need for re-training, the approach offers efficiency and flexibility. Future improvements could be achieved by leveraging more advanced 2D foundation models from general computer vision fields.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: \n\nThe paper is well-organized and clearly written, making it easy to follow and understand.\n\nThe novelty is well-highlighted, and the experimental results effectively support the claims.", "other_comments_or_suggestions": "In Table 4, reporting the average performance across the 10 regions for each method would improve clarity and readability.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YmavfdxfoC", "reviewer": "Reviewer_zwUQ", "summary": "This paper proposed a framework to leverage the pretrained large 2D encoder for 3D medical image analysis (i.e., classification and regression). By applying random projection to feature embeddings encoded from 2D slices taken of three orientations (i.e., sagittal, coronal, and axial) using DINOv2-L and concatenating an MLP, it provides a fine-tuning-free way to leverage 2D encoder pretiraned on 2D natural images. Compared with several encoders pretrained on large-scale medical image datasets, the Raptor outperformed them in both classification and regression tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed assessment of our manuscript. We appreciate that they viewed our experiments as supporting the utility of our method, and we agree that we could support some of our claims with additional experiments. We address each of the points raised below.\n\n> **Meaning of scalable in our work:**\n\nWe use the term “scalable” to highlight some key aspects of Raptor:\n1. It can process high-resolution 3D medical volumes with only a single pass through a fixed 2D encoder, avoiding the heavy computational cost of purely 3D models.\n2. The framework can readily scale to larger or more diverse medical datasets and reuse the embeddings for multiple downstream tasks, as there is no training involved.\n3. The inference step can be easily parallelized since we only require a frozen foundational model and random projections.\n\n> **More evidence for effectiveness with limited training samples:**\n\n_Due to the word limit, we have moved the subset results to our response to reviewer 28uY._\n\n\n> **Why not sample from N(0, 1/k):**\n\nIn terms of choice of noise for the random projections, we proceeded with the numpy default which was N(0, 1). Quantitatively we don’t expect a notable difference choosing a variance in this range, as the embeddings are eventually standardized in the fine-tuning step.\n\n> **Exploration of medical VLMs:**\n\nWhile Llava-Med and Med-Flamingo are strong multimodal methods in their respective domains, their encoders process 2D data, making direct comparison on volumetric (3D) tasks ambiguous (unlike Merlin, which was designed for 3D). Nonetheless, we tried substituting their image encoders into our pipeline in place of DINO: Raptor-LVM and Raptor-CLIP (note that Med-Flamingo leverages OpenAI’s CLIP as the image encoder, which we have previously benchmarked as an alternative to DINO).\n\n\n(AUROC shown)\n| Methods       | Organ | Nodule | Fracture | Adrenal | Vessel | Synapse |\n|-|-|-|-|-|-|-|\n| Raptor-CLIP    |        0.994 | 0.869 | 0.669 | 0.906 | 0.936 | 0.849 |\n| Raptor-LVM     | 0.996     | 0.888      | 0.632        | 0.904       | 0.941      | 0.851        |\n| Raptor-B        | _0.998_   | 0.904      | 0.647        | **0.930**   | 0.945      | _0.922_      |\n| **Raptor**     | **0.999** | **0.929**  | 0.677        | _0.926_     | **0.966**  | **0.943**    |\n\nThese alternatives show competitive performance but do not surpass DINO (our current proposed approach).\n> **Parameter number:**\n\nWe would be happy to include this information in our final version.\n\n\n| Methods | \\# Param | Latent|\n|-|-|-|\n| SLIViT | 48.4M | $768\\times64\\times8\\times8$ |\n| SuPreM |  62.1M | $128\\times12\\times12\\times12$  |\n| Merlin | 124.7M | $2048\\times14\\times7\\times7$ |\n| MISFM | 46.2M | $100\\times16\\times16\\times16$ |\n| VoCo | 294.9M | $3072\\times3\\times3\\times3$ |\n| **Raptor (Ours)** | 304.4M (DINOv2-L) | $3\\times100\\times16\\times16$ (K=100) |\n\n> **Plot upper bound in scaling figures:**\n\nWe appreciate this suggestion, and plan to revise our figures accordingly.\n\n> **Can accuracy be improved by removing problematic views:**\n\nWe appreciate this interesting suggestion. We explored whether skipping a view might improve FractureMNIST performance (either AC, CS, or AS, instead of ACS). However, results show no improvement: AC (0.657), CS (0.654), and AS (0.660), all perform below the default ACS (0.677). Intuitively, removing an entire view discards potentially useful information, even if that view is somewhat degenerate.\n\nAn alternative solution would be to partition slices within a problematic view. For instance, instead of aggregating all 64 slices at once, we could split them into two groups of 32 slices each and perform Raptor, then concatenate. Although this could mitigate partial cancellation effects, it increases embedding size. We find this to be a promising extension to Raptor in both practical deployment (handle slice-wise misalignment) and theoretical analysis (quantify error bounds under partitioning). We are actively exploring how best to balance these tradeoffs.\n\n\n> **Why MAE performs bad:**\n\nThe choice of architecture for the MAE was a transformer with 3-dimensional positional encodings. Despite our efforts to tune the model, we suspect that the datasets are simply too small (to learn e.g. effective patch embeddings and relationships between positional encodings).\n\n> **VQA Captioning:**\n\nBased on the strong performances that we observed (+ segmentation, in response to reviewer BpWD), we do hypothesize that Raptor embeddings would be capable of captioning. In this work, we focused on thoroughly verifying several properties of Raptor, and hope to evaluate its novel use cases in the future.\n\nWe appreciate the insightful questions raised by the reviewer, and hope that the additional information we have provided further supports the potential of our method. If our response has sufficiently resolved current questions regarding our work, we would appreciate an increase in our score.", "claims_and_evidence": "In four claimed contributions (lines 90-102), the second and forth claims are well-supported. \nFor the first claim about data efficiency, it was only supported by a single example in Figure 4. It would be more convincing if the data efficiency of the model is further evaluated on one of CC-CCII, CTRG-C, CTRG-B in Table 3, and one of the regression tasks in Table 4.  \nI found the third claim regarding scalability a bit confusing and wish the authors could clarify a bit during the rebuttal.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense.", "theoretical_claims": "I checked for correctness. I have a question about the sampling of random matrix for projection. \n\nIn lines 697-698, the random matrix for random projection was sampled from $\\mathcal{N}(0,1)$; I wonder why not sample from $\\mathcal{N}(0,1/k)$, as the default option in [sklearn package](https://scikit-learn.org/stable/modules/generated/sklearn.random_projection.GaussianRandomProjection.html#sklearn.random_projection.GaussianRandomProjection).", "experimental_designs_or_analyses": "I think the overall experimental designs and analyses are sound and valid. I wonder why the encoder of other multimodal medical foundation models except Merlin, such as Llava-med [1] and Med-flamingo[2], is not used for comparison. \n\n1. Li, Chunyuan, et al. \"Llava-med: Training a large language-and-vision assistant for biomedicine in one day.\" Advances in Neural Information Processing Systems 36 (2023): 28541-28564.\n\n2. Moor, Michael, et al. \"Med-flamingo: a multimodal medical few-shot learner.\" Machine Learning for Health (ML4H). PMLR, 2023.", "relation_to_broader_scientific_literature": "There are many existing studies exploring training foundation models/encoders for medical images from scratch or fine-tuning existing models/encoders. This paper proposes a novel method to leverage a pre-trained 2D large encoder for volumetric medical image analysis without fine-tuning, which is a considerable development compared to previous studies.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The presentation is good, but some parts can be further improved. Please refer to Other Comments Or Suggestions and Questions For Authors sections.", "other_comments_or_suggestions": "1. Please consider supplying the parameter number for the encoder and embedding dimension in Table 1, which will help the reader better understand the memory footprint in Figure 5. \n2. Please consider plotting the upper-bound results in Figure 4 (i.e., training on the full synapse dataset).", "questions_for_authors": "1. Lines 822-832 gave a reasonable explanation of why Raptor has worse performance on 3D FractureMNIST. From Figure 8, it seems most of the negative $\\alpha_i$ exists in axial only. I wonder, if you run Raptor on Coronal and Sagittal views (combined or separately, similar to experiments in Table 6), would that improve the performance of Raptor in the Fracture dataset? \n2. Is there any potential explanation for why MAE performed so badly in the regression task in Tabel 4? \n3. It seems the setting of Raptor would generally work for global-level image understanding (e.g., classification, Visual Question Answering, captioning) and not work for dense-prediction tasks (e.g., segmentation). I wonder if the authors have explored the use of randomly projected embedding for VQA and captioning. \n4. I'm not so sure if I correctly understand the term 'scalable' in the third claimed contribution (lines 96-98). Can you please elaborate on that?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mo1ryXq54U", "reviewer": "Reviewer_BpWD", "summary": "This paper introduces Raptor (Random Planar Tensor Reduction), a train-free method for generating semantically rich embeddings for volumetric data. Raptor leverages a frozen 2D foundation model, pretrained on natural images, to extract visual tokens from individual cross-sections of medical volumes. These tokens are then spatially compressed using random projections, significantly reducing computational complexity while retaining rich semantic information. Extensive experiments are carried out on multiple medical image datasets, and the experimental results are analyzed in detail, and the experimental results are excellent. The superiority and efficiency of the proposed method are verified.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's keen observations and suggestions. Thanks to their recommendations, we were able to further uncover Raptor's capability in a challenging task (segmentation) and are excited for future directions.\n\n> **statistical tests (e.g. p-values) … especially for small sample settings**\n\nWe acknowledge the value of statistical tests for clarifying performance differences and intend to include in a future revision. Given the strong performance of existing methods, we did not observe any significant improvement over the next-best method under the bonferroni-corrected threshold (p < 0.05/19, reflecting 19 total datasets). However, we did observe some modestly significant improvements (p < 0.05). Among the regression datasets, we observed such improvements in four of the datasets (4/10). We did not observe any such improvements for the classification datasets.\n\nIn terms of the subset experiments (varying training set size from 10~500), we similarly found modestly significant improvements (p < 0.05) in 3/5 settings for the UKBB white matter dataset, 1/5 in CCCC-II, and 2/5 in Synapse.\n\n> **Is there any bias per digit in the simulations?**\n\nWe appreciate the suggestion to control for digit shape in our simulations. Accordingly, we fixed the digit to either 0, 1, or 8 and generated simulations for each case for benchmark. We observed that the choice of digit can slightly influence the results. Here we demonstrate these results with 300 generated training samples each, as the simulation and evaluation process is time consuming end-to-end.\n| Method       | Size  | 0   | 1   | 8    | Avg.  | Std.   |\n|--------------|-------|-------|-------|-------|--------|--------|\n| **Raptor**    | 64px  | 0.798 | 0.678 | 0.753 | 0.743  | 0.049  |\n| | 32px  | 0.580 | 0.559 | 0.575 | 0.571  | 0.009  |\n| | 16px  | 0.552 | 0.514 | 0.517 | 0.527  | 0.017  |\n| | 8px   | 0.506 | 0.509 | 0.508 | 0.508  | 0.001  |\n\n\n> **Segmentation**\n\nWe concur with the reviewer’s point that more downstream tasks need to be attempted. We have now conducted additional experiments on the Medical Segmentation Decathlon (https://www.nature.com/articles/s41467-022-30695-9), focusing on four tasks: hippocampus, spleen, colon, and hepatic vessel segmentation. Each task presents unique challenges (e.g., limited contrast for hippocampus, class imbalance for hepatic vessels). For Raptor, we trained a 2-layer convolutional head to learn a transformation of the embeddings to the segmentation.\n\n| Task           | Dataset Size (Train/Val/Test) | Model   | IoU   | Dice Score |\n|-|-|-|-|-|\n| Hippocampus  | 182 / 39 / 39 | Raptor  | 0.607 | 0.719|\n| | | MedSAM  | 0.575 | 0.615|\n| Spleen         | 28 / 6 / 7  | Raptor  | 0.592 | 0.657|\n|  |  | MedSAM  | 0.960 | 0.979       |\n| Colon          | 88 / 18 / 20 | Raptor  | 0.597 | 0.597       |\n| |  | MedSAM  | 0.841 | 0.906       |\n| Hepatic Vessel | 212 / 45 / 46   | Raptor  | 0.387 | 0.431       |\n|  | | MedSAM  | 0.387 | 0.428 |\n\nAs shown here, Raptor achieves competitive results compared to MedSAM (a dedicated segmentation model). Notably, on the Hippocampus dataset, Raptor surpasses MedSAM’s Dice (0.719 vs. 0.615) and IoU (0.607 vs. 0.575), and performs similarly for hepatic vessels. MedSAM, however, excels on spleen and colon, which have fewer training samples. Although our primary goal was to obtain general-purpose embeddings, these early results suggest Raptor can also serve as a reasonable foundation for volumetric segmentation, without requiring large‐scale 3D training.\n\n> **Figure 1 clarity**\n\nWe agree Figure 1 could be improved in clarity and will make the requested adjustment given the opportunity.\n\n> **Would orthogonal feature processing compromise inter-slice contextual features**\n\nWe appreciate this important point raised by the reviewer. We provide an intuition how the resulting Raptor embedding can still retain volumetric signals. For a raptor embedding $e$, a tensor of dimensions $3\\times100\\times16\\times16$, the coordinate $(x,y,z)$ in the volume can be represented by three slices—one per axis—yielding tokens indexed by, e.g., $e[0,:,x,y]$, $e[1,:,y,z]$, and $e[2,:,x,z]$. Each patch within these slices is mapped into the K-dimensional embedding space; subsequent MLP layers can then fuse these partial 2D perspectives into a more coherent 3D representation. Our quantitative results on datasets like UKBB suggest that Raptor’s multi-view scheme does effectively capture enough high-level volumetric information for robust performance.\n\n> **Clearer workflow description on using Raptor embeddings**\n\nThis is an important suggestion and we plan on including a diagram and a detailed description of the Raptor pipeline for downstream tasks.\n\nWe thank the reviewer for the insightful questions, and hope our response addresses any remaining concerns. If so, we would greatly appreciate it if the reviewer could reflect these clarifications by raising their score of our work.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "Appendix A.3 restates a version of the Johnson-Lindenstrauss (JL) lemma (Dasgupta & Gupta, 2003) and demonstrates its applicability in the proposed multi-view volumetric framework. No issues were identified in the proof or its adaptation to this setting.\nAppendix A.4 provides the mathematical formulation of Raptor and proves that the model’s effectiveness is guaranteed under the assumption that the slices are \"smooth\". No logical gaps or technical flaws were detected in the derivations.", "experimental_designs_or_analyses": "The comparative experiments in Section 4.2 (Main Results: Classification) and Section 4.3 (Main Results: Regression) were thoroughly reviewed. For classification tasks, the comparisons against baselines include the metrics AUC and Accuracy (ACC), while regression tasks utilize the R² score. Additionally, the evaluation of varying training data sizes on the Synapse dataset in Section 4.4 and the impact of different embedding sizes across methods in Section 4.5 were examined. The overall design of these comparative experiments is methodologically sound, with clear and logically structured comparisons. Regarding ablation studies, the sensitivity analysis to the number of random projections K in Section 5.1, the influence of viewpoint numbers in Section 5.2, and the exploration of captured feature ranges in Section 5.3 were assessed. The ablation settings are well-defined and systematically address critical components of the proposed method.\nHowever, two aspects could be further clarified:\n1. Statistical Validation in Regression Analysis: While the r² scores in Section 4.3 provide useful insights, it would be valuable to supplement these results with statistical tests (e.g. p-values) to enhance the interpretability of performance differences, especially in small-sample settings.\n2. Using MNIST numbers with different shapes (e.g., “1” vs. “8”) may introduce bias. To better isolate the impact of scale variation, the authors might consider controlling for shape differences (e.g., using a single digit class like “0”) or exploring class-specific error patterns (e.g., through confusion matrices).", "relation_to_broader_scientific_literature": "The authors position their work within the context of image foundation models (specifically DINOv2-L, Oquab et al. 2023). Their key innovation—compressing tokens inferred from frozen image foundation models via random projections on orthogonal cross-sections of volumetric data—addresses two limitations of prior work: high computational costs (Hatamizadeh et al., 2021; Wasserthal et al., 2023; Li et al., 2024; Cox et al., 2024; Wu et al., 2024b) and the limited scale of 3D medical datasets (160K volumes, Wu et al., 2024b), which remain orders of magnitude smaller than 2D image datasets (1.2B images, Oquab et al., 2023). This study is the first to integrate scalable 2D foundation model priors with computationally efficient 3D medical analysis, offering a pathway to leverage large-scale 2D pretraining for volumetric tasks.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "strength:\nThe training-free design of this work significantly reduces model construction costs, while achieving robust performance on an 11GB RTX 2080 Ti.\nWeakness:\nAlthough the authors claim that they demonstrate their approaches in 10 tasks, it only contains two simple tasks, classification and regression. I would suggest the authors to demonstrate the proposed method on more fundamental and challenging tasks (like segmentation).", "other_comments_or_suggestions": "In Figure 1, if the metrics for regression tasks (r²↑, indicated with +) represent mean values, this should be explicitly stated to avoid potential misinterpretation.", "questions_for_authors": "1) Could the feature extraction approach that processes orthogonal cross-sections independently potentially compromise inter-slice contextual features in 3D medical volumes?\n2) The application of Raptor’s features to classification and regression downstream tasks could be further clarified. To enhance methodological transparency, it would be beneficial to explicitly illustrate this process in a workflow diagram, demonstrating how the extracted features are integrated into task-specific heads or decision pipelines.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dTXHn8yjir", "reviewer": "Reviewer_EKbX", "summary": "The paper introduces a novel design that enables the application of 2D foundation models to 3D data tasks without requiring additional pretraining. The proposed method significantly reduces the size of feature maps, and demonstrates strong performance even in scenarios with limited training samples. Additionally, the authors provide detailed theoretical descriptions and ablation studies to support their design choices.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We also agree with the reviewer that our novel approach has several implications for the field, as it would make the computation of embeddings for volumes much faster while maintaining the ability to perform downstream tasks. We also appreciate the reviewer’s evaluation of the theoretical background of our approach.\n\n> **Why DINO works for medical data:**\n\nThe high-level intuition is that the semantically meaningful features of medical volumes are still part of the universal image features (edges, textures, shapes), which a broad 2D encoder has already learned from large‐scale, diverse data. RAPTOR uses generic image foundation models like DINO to capture any meaningful image features and performs low rank approximations to retain only the relevant ones. This is in contrast to many existing medical image/volume models that try to learn such mappings from scratch or fine-tune a general model to be biased towards them.\n\nTo illustrate, we projected both generic (ImageNet) and medical (MedMNIST) embeddings onto the principal components derived from the generic dataset. In the table below, we calculate the total variance explained as we add more PC’s. Although the medical data do not align with the very top principal directions, they catch up as more components are added (eventually matching the generic variance). This demonstrates that medical image features lie within the general “image” space.\n\n| #PCs | General Explained Var | Medical Explained Var | ratio(Medical/General) |\n|-|-|-|-|\n|  100 |     0.613   |     0.225   |       0.367    |\n|  200 |     0.765   |     0.397   |       0.518    |\n|  500 |     0.926   |     0.734   |       0.793    |\n| 1000 |     0.999   |     0.994   |       0.996    |\n\n\n> **Why we don’t use another method to aggregate slices:**\n\nWe explored the simplest approach—summing and random projection—because it is computationally lightweight and maintains near‐orthogonal signals from the 2D tokens without additional training (also note that averaging/summing is identical in expectation to random projection into dimension 1). More structured dimension‐reduction techniques may introduce significantly higher computational or memory costs, especially for high‐resolution volumes. We agree that specialized subspace methods are an interesting direction for future work, particularly if they exploit known volumetric structures. As an example, please see our response to reviewer zwUQ on partitioned embedding.\n\n> **Time taken to run Raptor:**\n\nWe appreciate the reviewer’s interest in a more detailed efficiency comparison. In the table below, we provide approximate times for fine-tuning, inference, and one-time embedding extraction across Raptor, SuPreM, and Merlin when training on the Organ dataset (971 volumes). While Raptor does incur a one-time cost to produce embeddings, its subsequent fine-tuning is minimal (logistic regression or a small MLP) and can often run on CPU. This contrasts with end-to-end 3D methods (SuPreM, Merlin) which must either be entirely trained or extensively fine-tuned, typically on a GPU with longer runtimes.\n\n\n| Method | Medical Pre-training | GPU | Embedding (per vol) | Fine-tuning | Inference (per vol) |\n|-|-|-|-|-|-|\n| Raptor | None (use Image FM) | RTX2080Ti | $4.2$ secs | $1.3$ mins | $<0.1$ secs  |\n| SuPreM | Days | A100 | - | $70.0$ mins | $<0.1$ secs |\n| Merlin | Days | A100 | - |  $63.0$ mins |  $<0.1$ secs |\n\n\n> **Stability of random projections:**\n\nWe also tested more seeds (up to 10) for the 3D MedMNIST dataset to assess the stability of random projection. In the table below, we observe consistently low standard deviations, suggesting that random projections remain highly reliable in practice:\n\n| Method       | K   | Seed 1 | 2     | 3     | 4     | 5     | 6     | 7     | 8     | 9     | 10    | Avg.  | Std.   |\n|--------------|-----|--------|-------|-------|-------|-------|-------|-------|-------|-------|-------|--------|--------|\n| **Raptor**    | 1   | 0.818  | 0.817 | 0.793 | 0.823 | 0.831 | 0.818 | 0.831 | 0.814 | 0.801 | 0.831 | 0.818  | 0.0121 |\n|              | 5   | 0.890  | 0.860 | 0.864 | 0.869 | 0.869 | 0.877 | 0.868 | 0.858 | 0.872 | 0.866 | 0.869  | 0.0086 |\n|              | 10  | 0.866  | 0.896 | 0.876 | 0.875 | 0.882 | 0.879 | 0.871 | 0.876 | 0.877 | 0.886 | 0.878  | 0.0078 |\n|              | 100 | 0.901  | 0.899 | 0.900 | 0.898 | 0.905 | 0.891 | 0.899 | 0.898 | 0.900 | 0.898 | 0.899  | 0.0034 |\n|              | 150 | 0.898  | 0.897 | 0.897 | 0.894 | 0.898 | 0.903 | 0.902 | 0.905 | 0.902 | 0.904 | 0.900  | 0.0034 |\n\n\nWe hope that the additional analysis provides better insight into the capabilities of our method. If these clarifications have addressed any lingering concerns, we would sincerely appreciate if the reviewer would consider raising their score.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Yes", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\nThe method provide a new direction for 3D medical image classification in a training-free manner.\n\nWeakness:\n1. The DINO series models do not be trained on the medical images. Why they can perform well? The authors should give more analysis and explanation.\n2. Spatial dimensionality reduction method: The authors employ random projection for feature dimensionality reduction. However, in Section 3.3, they perform an averaging operation along the observing direction on the intermediate representation z, reducing the corresponding spatial dimension. Why random projection or other dimensionality reduction techniques are not used for this step as well?\n3. Computational efficiency comparison: The proposed method requires processing slices from three directions using a 2D foundation model. The authors only report inference time for the proposed method, but it would be more informative to compare inference speeds with other 3D methods.\n4. Reliability of random projections: The authors evaluate the reliability of random projections using three random seeds. However, this may not be sufficient. A larger number of random seeds should be tested, and error bars should be included to better illustrate the impact of randomness on performance.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This manuscript introduces Raptor (Random Planar Tensor Reduction), a novel training-free method for generating semantically meaningful embeddings from volumetric medical data using pre-trained 2D foundation models. The approach leverages random projections to compress visual tokens extracted from cross-sectional views, enabling efficient processing of 3D medical volumes without requiring additional training or fine-tuning. The work demonstrates strong performance across multiple classification and regression tasks on ten different datasets, with particular effectiveness in limited-data scenarios. The reviewers acknowledged several strengths, including the method's computational efficiency, theoretical foundations supported by detailed mathematical analysis, and comprehensive empirical validation. While initial scores were mixed (two weak accepts and one accept), the authors' thorough rebuttal addressing concerns about statistical validation, computational comparisons, and additional segmentation experiments led one reviewer to raise their recommendation to accept. Some limitations noted by reviewers included questions about the reliability of random projections and the need for clearer workflow descriptions for downstream tasks. The authors' responses provided additional experimental evidence showing consistent performance across multiple random seeds and detailed explanations of the integration with task-specific heads. The discussion around segmentation capabilities generated particular interest, though reviewers differed in their assessment of its significance to the paper's core contributions.
Decision: Accept (spotlight poster)