Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
Authors: Ibrahim Ethem Hamamci
Sezgin Er
Suprosanna Shit
Hadrien Reynaud
Dong Yang
Pengfei Guo
Marc Edgar
Daguang Xu
Bernhard Kainz
Bjoern Menze
Venue: NeurIPS 2025 poster ()
Keywords: 3D Medical Imaging
Vision-Language Modeling
Radiology Report Generation
Text-Conditional 3D CT Synthesis
Volumetric Tokenization
Primary area: machine_learning_for_sciences

Abstract: Recent progress in vision-language modeling for 3D medical imaging has been fueled by large-scale computed tomography (CT) corpora with paired free-text reports, stronger architectures, and powerful pretrained models. This has enabled applications such as automated report generation and text-conditioned 3D image synthesis. Yet, current approaches struggle with high-resolution, long-sequence volumes: contrastive pretraining often yields vision encoders that are misaligned with clinical language, and slice-wise tokenization blurs fine anatomy, reducing diagnostic performance on downstream tasks. We introduce BTB3D (Better Tokens for Better 3D), a causal convolutional encoder-decoder that unifies 2D and 3D training and inference while producing compact, frequency-aware volumetric tokens. A three-stage training curriculum enables (i) local reconstruction, (ii) overlapping-window tiling, and (iii) long-context decoder refinement, during which the model learns from short slice excerpts yet generalizes to scans exceeding $300$ slices without additional memory overhead. BTB3D sets a new state-of-the-art on two key tasks: it improves BLEU scores and increases clinical F1 by 40\% over CT2Rep, CT-CHAT, and Merlin for report generation; and it reduces FID by 75\% and halves FVD compared to GenerateCT and MedSyn for text-to-CT synthesis, producing anatomically consistent $512\times512\times241$ volumes. These results confirm that precise three-dimensional tokenization, rather than larger language backbones alone, is essential for scalable vision-language modeling in 3D medical imaging. The codebase is available at: https://github.com/ibrahimethemhamamci/BTB3D

Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
Ibrahim Ethem Hamameil:*
Sezgin Er!-
Suprosanna Shit!
Hadrien Reynaud
Dong Yang²
Pengfei Guo
Mare Edgar
Daguang Xu²
Bernhard Kainz
Bjoern Menze*
University of Zurich
NVIDIA
Imperial College London
FAU Brlangen-Nürnber,
Abstract
Recent progress in vision-language modeling for 3D medical imaging has been fueled by large-scale computed tomography (CT) corpora with paired free-text reports, stronger architectures, and powerful pretrained models. This has enabled applications such automated report generation and rext-conditioned 3D image synthesis. Yet, current approaches struggle with high-resolution, long-sequence volumes: contrastive pretraining often yields vision encoders that are misaligned with clinical language, and slice-wise tokenization blurs fine anatomy, reducing diagnostic performance on downstream tasks. We introduce BTB3D (Better Tokens for Better 3D), a causal convolutional encoder-decoder that unifies 2D and 3D train- ing and inference while producing compact, frequency-awan volumetric tokens. A three-stage training curriculum enables (i) local reconstruction, (ii) overlapping- window tiling, and (iii) long-context decoder refinement, during which the model learns from short slice excerpts yet generalizes to scans exceeding 300 slices with- out additional memory overhead. BTB3D sets new state-of-ther
on two key tasks: it improves BLEU scores and increases clinical F1 by 40% over CT2Rep, CT-CHAT, and Merlin for report generation; and it reduces FID by 75% and halves FVD compared to GenerateCT and MedSyn for text-to-CT synthesis, producing anatomically consistent 512 x 512 x 241 volumes. These results confirm that pre- cise three-dimensional tokenization, rather than larger language backbones alone, is essential for scalable vision-language modeling 3D medical imaging. The code- base is available at: petpp://githule wtw.thuu.coomm/eee wtt.:cccmccaaaaee
1 Introduction
Three-dimensional (3D) medical images, such as CT, provide rich volumetric view of anatomy, offering significantly more detail than radiographs 11. This makes them well-suited for vision- language models (VLMs), which can automate radiology report generation and enable text-guidee rext-guided volume synthesis for data augmentation, education, and planning [2]. Recent progress in 3D VLMs, driven by public datasets with paired reports (especially in chest CT) and advances in vision encoders and large language models (LLMs), has opened new clinical possibilities B 6 However, these systems still face major challenges: 3D volumes often consist of hundreds of slices, introducing long- sequence modeling challenges 8 Moreover, the limited availability paired 3D data prevents robust
Authors may list their names first in their CVs. Correspondence: ibrahim. hamamci @uzh. ch
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

training 19]. As result, 3D VLMs struggle to capture fine- tine-griindd clinical details and maintain spatial coherence when tasked with generating detailed reports or high-fidelity 3D volumes [T, [2]
Why do 3D VLMs lag behind their 2D counterparts in report generation? A typical pipeline combines pretrained 3D vision encoder with an L.M for report generation (3). While LLMSs have improved significantly through large-scale pretraining, the vision side remains a bottleneck (TT] Most state- of-the-art methods rely on vision encoders pretrained with contrastive objectives [3 [4| However, this approach faces two critical limitations in the 3D medical context. First, contrastive learning assumes that only paired image-text samples are semantically aligned, while unpaired samples are unrelated [15]. In radiology, this assumption often does not hold: multiple reports can describe the same conditions differently due to variations language, clinical focus, or radiologist style [16]. Penalizing such unmatched pairs may therefore degrade the model's 's understanding of medical semantics. Second, CLIP--tyle training requires large batch sizes for stability [17]. High-resolution 3D volumes are memory memory-intensive, often forcing trade-off's in batch size or model depth, resulting in underpowered vision encoders [14. Prior work has noted that key findings (e.g., small nodules or subtle textures) may be lost when compressing
entire 3D scan into a single vector using weak models trained contrastively [3]. Another major challenge is the lack of interoperability between 2D and 3D representations 3. Most 2D medical VLMs cannot directly process volumetric input and instead require projecting 3D scans into 2D slices [8 19 This modality gap limits the transferability of pretrained 2D models to 3D, particularly important given the scarcity of 3D image-tex text datasets.
A similar bottleneck affects text- cotdicnniiional 3D medical image generation. Prior methods rely on mooder-decode networks pretrained in a self-supervised manner to reconstruct volumes [1]] How- ever, training an encoder-decoder capable of reconstructing long, high-resolution CT sequences remains an open challenge. Thus, existing methods adopt cascaded generation frameworks, resulting in spatial discontinuities and reduced realism, or rely on lightweight encoder-decoder architectures that fail to capture nuanced context (20). Consequently, current models struggle to generate anatomi- cally consistent, high-quality 3D volumes from text. Hence, we need encoder-decoder networks for 3D VLMs that (1) scale to long volumetric sequences without losing critical details, (2) bridge the 2D/3D divide through unified representations, and (3) decode high-resolution 3D volumes from these representations. Improved encoders would enable more effective alignment between CT scans and textual descriptions for report generation, while better decoders enhance text-to-image generation.
To address these challenges, we introduce BTB3D (Better Tokens for Better 3D), a novel encoder- decoder framework that advances VLMs in 3D medical imaging through improved tokenization, reconstruction, and training strategies. Our core contribution is a causal convolutional encoder- decoder that learns a compact sequence of volumetric tokens for each CT. We adopt a quantized latent space for efficient 3D tokenization and reconstruction (21] Causal 3D convolutions enable the model to process scans slice by slice (analogous to a temporal sequence), allowing scalability to arbitrarily long scans and compatibility with pretrained 2D features, serving modalities. For downstream tasks, the proposed architecture can be combined with modules such as transformer decoder for report generation or diffusion model for volume synthesis. Crucially, we introduce a novel three-stage training strategy that progressively adapts the encoder-decoder to longer input contexts, enabling robust training even under compute- limited conditions.
bridge between 2D and 3D
2 Related works
Radiology report generation from 3D medical images. The recent availability of 3D medical datasets paired with corresponding reports (such as CT-RATE) has enabled significant progress in this domain (3]. The first framework for radiology report generation from 3D scans, CT2Rep, did not leverage any pretrained vision or language models [00] Due limited training data and the need to generate long, meaningful reports, CT2Rep employed relational memory module to retrieve relevant past reports for coherent generation, inspired by prior work (22]. Subsequent models, such as CT-CHAT and Merlin, improved upon this by incorporating pretrained components [1] These approaches followed contrastive strategy to align image and text features during vision encoder pretraining [5] A more recent model, fVLM, pretrained its 3D vision encoder, using more clinically relevant contrastive learning approach [4]. However, we do not include fVLM in our comparisons, neither its report generation codebase nor model weights have been available.

---

8.
Zeroppdd
Inverse
363r3 Cauaal Conv
Wanert (VV)))
3D Haar Wavelet (WV-1)
Grouphicm SLU
Zoro padi
Zero pad
Deptiviosppeee
Upsampler
0TESO Decoder (OGC))
Decooder
BTB3D Encoder
Pre ReaBlocks
Looksp Quericatur (Q)
%
3x363 Causal Conv
b.
X
x
BTB3D
(G())
Figure 1: (a) Stage 1: A 9-slice subvolume is compressed via wavelet transform, then encoded causally using two stride-2 temporal convolutions. The encoder processes the input strictly causally by prepending zero-padded slices. Tokens decoded back to the wavelet domain by symmetric causal decoder. (b) Stage 2: To scale to long CT volumes, we introduce overlapping temporal tiling (retaining only the second token from each window) to follows the same scheme but trains only the decoder to refine long-range anatomical reconstruction.
consistent representation. Stage
Text-conditional 3D medical image generation. The first framework for text-conditioned 3D volume generation, GenerateCT, introduced cascaded pipeline [12. Due computational limita- tions, it trained a 3D encoder-decodes network on low-resolution CT scans, followed by 2D diffusion steps for upsampling (23). While this cascaded approach enabled high-resolution and long- -equeecee generation, the volumes suffered from poor interslice consistency. MedSyn addressed this limitation with 3D architecture designed to improve spatial coherence; however, its lightweight design reduced image fidelity (20) Other 3D CT scan generation methods, such as MAISI, exist, but we do not include them in our comparisons, as they are not designed for text-conditional generation 24]
Methodology
Due to the computational challenges, our goal is to pretrain short sequences while enabling scala- bility to full volumes. Contrastive learning is inappropriate because paired reports typically describe the entire scan, and training cropped partial slices introduces semantic misalignment they are difficult to adapt for 3D image synthesis pretraining, generalizing effectively to longer sequences. Prior work has shown that convolutional networks outperform transformer- basereed---aedddd ones in video reconstruction (26.. Thus, we adopt a 3D CNN-based backbone (27). Given the limited 3D medical datasets, design model support both 2D and 3D, enabling unified training across modalities, which motivates our causal network (28).
[13]. Also, [25]. Thus, we propose a reconstruction-
However, training convolutional model on short sequences introduces two key challenges: (1) When performing inference in a single pass over the full CT, reconstruction quality tends to degrade toward the latter slices; or (2) When performing inference in short chunks, inconsistencies arise at window

---

boundaries. To address them, we introduce a three-stage training strategy that progressively scales while preserving temporal consistency. The resulting model can reconstruct long sequences with high fidelity and generalizes well across downstream tasks. Below, we detail the architecture and training.
3.1 Model architecture
BTB3D is 3D convolutional encoder-decoder network with discrete latent codebook [29]. We design for CT scans using causal convolutional mechanisms along the temporal (axial) dimension and wavelet-based compression for efficient and scalable input representation, shown in Figure
3D Haar wavelet compression. Given volumetric CT scan DD apply 3D Haar wavelet transform W-) to obtain multi-channel representation W(x) E R x x8 following prior work in video generation that showed wavelet decompositions improve reconstruction quality [30
32]. This transformation reduces resolution by a factor of two along each axis while retaining essential frequency information. Each non-overlapping 2 x 2 x 2 voxel block yields 8 subband coefficients: one low-frequency and seven high-frequency components. These frequency- aware channels encode both coarse and fine anatomical features, improving representation learning. Although the channel dimension increases from to 8, the size of volumes is reduced by factor of yielding substantial compression for the input. This reduces memory and computation costs, enabling deeper or wider networks and making the model scalable to long. high-resolution CT volumes. For 2D slices (D 1), the Haar transform along the z-axis produces a low- and feeeffrey pair, with the latter often near
near-zero zero due to the absence of temporal variation. Nonetheless, the same 3D transform applies to both 2D and 3D inputs, ensuring architectural consistency across training modes.
The encoder E takes a wavelet- transformddff CT sequence ((x) E DD XC and outputs a latent representation E R " xH'''"'' XW' xd, consists of residual blocks with factorized 3D convolutions that decouple spatial and temporal processing. Fach block applies 1 x & k spatial convolution (sagittalccoronal followed by a k x 1 1 temporal convolution (axial), with causal padding of (k-1) zero slices in the past and none in the future. This ensures that the encoder at index É only attends to slices S t, enabling strictly causal encoding. As a result, the first token is computed from the first slice alone, and future leakage is prevented in all subsequent tokens. This causal design supports unified 2D (e.g., single-slice) and 3D volume-Ievel training while preserving axial consistency, beneficial for downstream autoregressive tasks [33 35
Encoder with causal 3D convolutions.
Figure Ta Ialusuussssess the model architecture and causal axial compression. Downsampling is performed using strided convolutions interleaved with residual blocks. the 8x8x8 configuration, two stride-2 convolutions along each axis yield a 4x reduction spatially and temporally. Combined with the initial 2x wavelet downsampling [31]. this results in an effective 8x compression. The 16x16x 16x8 8 variant adds extra spatial stride-2 layer, achieving 16x spatial and 8x temporal compression. Each variant presents a trade-off: the 8 setting preserves more spatial detail for tasks like segmentation or volume synthesis, while the 162x8 x8 variant offers higher compression for memory-constrained settings or tasks focused on global semantics, such as classification or radiology report generation.
Lookup-free quantization and decoder. The encoder output y (()()) E ER D''W' x is transformed into a discrete latent representation using lookup-free free quantizatiinn BI. The number of quantization codes is K 2d, and each feature vector at spatial position is binarized independently across dimensions by computing the sign of each element, resulting in binary vectors b E E 1)
These vectors are then packed into integers, forming the token map 2 E {O..... K - 1) D' xw" This approach removes the need for a codebook or embedding lookup during training and inference, significantly improving speed and memory efficiency, critical for large 3D volumes. To prevent code collapse (e.g., overuse of a limited set of tokens), we include an entropy regularization term Lentrops that encourages uniform usage of all K codes. Prior work in discrete representation learning 36 26
H''"".
has shown this regularization improves both reconstruction and token diversity. The decoder G. which mirrors the encoder's structure as shown in Figure Ta employs transposed convolutions and
residual upsampling blocks to reconstruct the wavelet-domain volume W Wxr). Like the encoder, the decoder is also causal: each slice reconstructed using only previously decoded slices, preserving autoregressivity [3.. The decoder accepts either the packed integer tokens or their binary expansions.
The final volume à is obtained by applying W-1, the inverse 3D Haar wavelet transform, to Wxr).

---

Training objectives. The model is trained to reconstruct high-resolution 3D CT sequences using combination of three objectives: reconstruction loss, adversarial loss, and quantization loss.
The reconstruction loss free encourages the decoder to accurately recover the input volume:
NV
1 N
Lree Pa [| 2 2111
where è is the reconstructed volume and N is the number of voxels. We use the & norm, which promotes sharper reconstructions and preserves fine anatomical detail better than & 38
The adversarial loss Lade improves perceptual realism by encouraging indistinguishabilit::
N log D(::):
Ladv
E2 log D(
N
where D is a 3D discriminator trained to distinguish real from generated CT volumes. We apply this
supervision directly in the CT domain rather than the wavelet domain, stabilizing training [39 36
The quantizaaion loss Lug enforces commitment to discrete representations:
Laq
E [|g[y] ell² + 3 |yy sg[e]12]
where is the discrete embedding (from codebook binary quantization), and sgl:] denotes the stop-gras graddentt operator 40 The overall training objective is a weighted sum:
Lree + Andy + Lvg.
where Amdv controls the influence of adversarial supervision. We omit perceptual losses (e.g., VGG- based features) due to the mismatch between natural RGB images and grayscale medical ones 41 42
Prior work has shown that these losses can degrade performance in medical image reconstruction
3.2 Three-stage training
Stage 1: Short-volume pretraining. We first train the model on single 2D slices or short 9-slice subvolumes. The entire model (encoder, quantizer, and decoder) is optimized end-to-end:
min ((a19,, where -1.9 = W- (C(((((((((()))))))
(5)
E,G
This phase helps the model learn local spatial and
structures.
Stage 2: Overlapping temporal tiling. We continue training both the encoder and decoder using overlapping short subsequences instead of full-volume encoding. To scale long volumes while preserving temporal consistency, we adopt overlapping window strategy. Specifically:
1. Encode T1:9 = =|, 2/. Keep both tokens.
2. Encode T9:17 = 21. 22. Discard 2f (covers only slice 9); keep 2 (covers 29-T17).
3. Encode T17:25 21. 22. Discard 2/: keep 23. Repeat this pattern until the end of the CT.
In each 9-slice window, we discard the first token and retain the second, which encodes information from all slices (except the first window in which take both tokens). This overlapping strategy promotes temporal consistency. Letting 2f denote the i-th token from the t-th window, we have:
1st window:
[z] 22 E(W (21.9))
(6)
2nd window:
[2] 2
E(W (29.17)) keep only 2f
(7)
T-th window:
E(W (HIDD-))D) = keep only z7
(8)
where 7 - 1)/8), and D is the sequence length of the partial CT volume that fits into memory this second stage training. As shown in Figure Tb the final latent sequence
[=] 22, 2....,,
This allows for efficient training on longer sequences than Stage 1, thanks lower memory footprint compared to one-shot encoding. The decoder processes all tokens in single forward pass.
5

---

Stage 3: Long-sequence decoder fine-tuning, This stage mimics Stage 2, but we freeze the encoder E and the codebook, and fine-tune only the decoder G. The training objective is:
min Er, [C (IL.D. fc
subject to E and codebook frozen,
(10)
where fG are the parameters of the decoder G. This step enhances the decoder's 's capacity to model long-range anatomical dependencies by training it to reconstruct CT without modifying the encoder.
Inference. BTB3D supports two inference strategies for reconstructing full-length 3D CT volumes: one-shot and tiled inference, offering flexibility based memory constraints and desired consistency. In one-shot inference, the entire volume x is encoded and decoded in a single pass:
f W (G(E(W
(11)
This is efficient when memory allows full-volume processing but it may degrade on long sequences. In riled inference, we follow the overlapping tokenization scheme from Stagg 2 to ensure spatial coherence over long sequences. The input volume is split into overlapping 9-slice windows, and:
|=] 2 EW (I a = W (G((=) 2) 22 22
(12)
This strategy mirrors the training and ensures consistent reconstructions with lower memory use. We adopt tiled inference for downstream experiments due to its robustness and improved consistency.
Experiments
We evaluate the effectiveness of our model and three-stage training through reconstruction and two downstream tasks (report generation and text-to-CT synthesis). Our experiments address three key questions: (1) Does the three-stage training improve reconstruction over naive end-to-end training? (2) Can our tokenization enhance vision-language tasks like report generation? (3) How does BTB3I) compare state-of-the-art models in synthesizing high-resolution CT from clinical prompts?
4.1 Three-stage training performance
Table 1: Reconstruction metrics after each stage of the stree-stage training. We report full-volume reconstruc- tion performance each stage for two models with compression rates of 8x8x x and 16x 16x8.
We assess BTB3D's three-stage training via reconstruction metrics and ablations, demonstrating its scalability from short subvolumes full-resolution 3D chest CT's. The strategy progressively (1) learns local spatial and short-range temporal features, (2) extends to longer sequences using over- lapping tiling, and (3) enhances global co- herence through decoder-only fine-tuning. This enables strong representations for downstream tasks such as report and text- to-CT generation (Sections 4.2 and
Stage
C. Rate
PSNR 1
SSIM 1
MSE +
Stage I
83
9.350
0.206
0.117
Stage 2
83
23.980
0.697
0.005
Stage 3
83
28.166
0.760
0.001
Stage 1
162 x8
11.067
0.353
0.079
Stage 2
162 x62x8
23.808
0.700
0.005
Stage 3
162x8
26.750
0.749
0.002
Experimental results. Table reports reconstruction performance across training stages 44 45
Stage 1, trained on short subvolumes, captures local structure but fails at long-range consistency.
Stage 2 introduces overlapping tiling, yielding the largest improvement: PSNR increases by over 14 dB. SSIM triples, and MSE drops an order of magnitude. Stage 3 refines inter-slice fidelity through decoder tuning, with modest but consistent gains. Figure 2 visualizes reconstruction quality (8 x variant) across planes. Stage outputs are blurry and lose structure beyond slices. Stage 2 restores coherent anatomy across distant slices. Stage 3 sharpens fine boundaries such as fissures and vessels.
Dataset and implementation. Aligned with the baselines, we use 25,692 chest CT scans from 21,304 patients in CT-RATE (the largest public 3D medical dataset with paired reports) [3]. The training set includes 20,000 patients and the test set includes 1,304. Volumes are converted to For Stages I and 2, we use raw volumes; for Stage 3, volumes are resampled to 0.75 X 0.75 x 1.5 mm and cropped/padded to
Hounsfield Units (HU) and clipped to [-1000, 1000) [46 47
512 x 512 x 241. Training is conducted on 64 NVIDIA H100 GPUs using DDP and mixed precision.

---

Stage
Stage
Stage
Ground Truth
Astal
Coronal
Sagittal
Figure 2: Reconstruction results of our 8x8x8 model across axial, coronal, and sagittal planes after each training stage. Progressive improvements demonstrate the effectiveness of our three-stage strategy, with Stage providing the largest gains in anatomical fidelity and inter-slice consistency.
In Stage 1, the batch size is 8 for 9-slice subvolumes and 40 for single slices. In Stages 2 and 3, we use a batch size of I (with 201 and 241 slices, respectively). The discriminator is reinitialized at each stage. We use Adam [48] with a learning rate of 1c-5, B3 0.99, and 1c-8, along with gradient clipping (threshold: 0.5) 49]. LFQ uses a token dimension of (codebook size: 262,144) 31. We train Stage for 150k iterations, Stage 2 for 60k, and Stage 3 for 50k.
4.2 Radiology report generation from 3D chest CT
We evaluate BTB3D on report generation for 3D CT. Each volume is encoded into sequence of latent tokens v E Txxd by the BTB3D encoder. For the 16 x 16 8 model, we 18- dimensional tokens; for the 8 x x 8 one, the token count quadrupled and the embeddings are reduced to 72 via merging. A linear layer maps these tokens the input space of LLaMA 3.1-8B [50), which autoregressively generates reports following the CT-CHAT setup 33. We adopt the same configuration and pretrained LLM as CT-CHAT ensure fair comparison and isolate the impact of our tokenization.
Table 2: Evaluation on Rad-ChestCT proves strong
out-of-distribution generalization. As only binary labels are available, text-based based metrics are omitted.
Model
F1 f
Precison t Recall 1
CT2Rep
0.133
0.299
0.139
Merlin
0.182
0.271
0.149
CT-CHAT
0.182
0.382
0.171
Ours-8
0.192
0.269
0.165
Ours- 16
0.266
0.272
0.329
Prior methods rely on lightweight encoders and contrastive pretraining, limiting tok- enization quality to capture relevant details for report generation. In contrast, our pretraining (with architectural and training advances) produces richer representations. We benchmark BTB3D against three state-of of-te-of-thearr report generation models. CT2Rep [10] uses CT-ViT encoder and GPT-
Baselines.
7

---

Table 3: Report generation performance on CT-RATE. BTB3D outperforms prior methods in both metric types, with the higher compression variant achieving the highest and BLEU scores.
Clinical Accuracy
Natural Language Generation
Model
F1
P
R
CRG
B1
B2
Bg
B1
B
M
CT2Rep
0.160
0.435
0.128
0.359
0.372
0.292
0.243
0.213
0.280
0.197
Merlin
0.160
0.295
0.112
0.352
0.231
0.163
0.124
0.099
0.154
0.148
CT-CHAT
0.184
0.450
0.158
0.368
0.373
0.284
0.231
0.198
0.272
0.215
Ours-8
0.187
0.260
0.150
0.357
0.411
0.307
0.245
0.215
0.295
0.220
Ours-16
0.258
0.260
0.260
0.370
0.439
0.320
0.248
0.213
0.305
0.223
Table 4:
Text-conditional CT generation results. BTB3D with lower compression outperforms previous methods across all metrics, showing superior consistency, image quality, and text alignment.
FID
FVD +
CLIP Score 1
Model
Axial
Sagittal
Coronal
Mean
CT-Net
I3D
Text-Img
Img-Img
GenerateCT
10.416
10.365
7.754
9.512
7.659
1512.5
23.625
84,287
MedSyn
14.963
12.115
10.698
12.592
13.927
725.81
23.571
84.153
Ours-8
2.479
2.166
2.062
2.236
3.955
325.51
24.270
88.352
Ours-16
7.077
4.226
3.729
5.011
4.020
429.34
23.322
84.957
style decoder with memory modules. CT-CHAT 3 aligns CLIP-pretrained vision encoder with LLaMA [5I]. Merlin [I]] employs an 13D-ResNet backbone pretrained with masked and contrastive objectives and attaches decoder similar to CT-CHAT. We use official weights for CT2Rep and CT-CHAT, both trained on CTERATE, As Merlin was originally trained on private data and does not release weights, we retrain it on CT-RATE using the official codebase.
Experimental results. Table reports clinical accuracy (F1, precision, recall, CRG [22) and language quality (BLEU, METEOR [53] (4]) CT-RATE. Merlin achieves high precision but low recall, tending to under-report findings, while CT2Rep and CT-CHAT show high recall but lower precision, often hallucinating abnormalities. Our method achieves a better balance, reflected in superior F1 and CRG scores. The 16 x 16 x 8 variant achieves the highest FI, with 40% relative improvement over CT-CHAT, confirming the effectiveness our volumetric tokenization. Compared to CT-CHAT, BLEU-1 improves by 18% and BLEU-mean by 12%. Table 2 presents results on RadChestCT. Our 16 16 x model again achieves the highest Fl score, 46% relative improvement over the best basclines. Recall also improves substantially, indicating strong generalization to out-of- distribution data. All clinical metrics are computed using the official CT-RATE report classifier.
Dataset and implementation. We use the CT-RATE dataset for report generation training, with the findings and impression sections used as in prior baselines. Chest CT scans are resampled to a voxel spacing of 0.75 0.75 1.5 mm and padded or cropped to uniform size of 512 x 512 241. For external validation, we use RadChestCT (55), which contains multi-label annotations but no text reports. Models trained for 40,000 iterations using DeepSpeed ZeRO-3 40 NVIDIA H100 GPUs We use the Adam W optimizer with a learning rate of 2 x 10-5 and a warm-up ratio of 0.03.
Each GPU processes one sample, yielding an effective global batch size of 40. For LoRA, we set r = 64, a = 128 for the 83 variant, and r = 128, a 256 for the 162 x8 variant.
4.3 Text-conditional 3D chest CT generation
We evaluate BTB3D's encoder-decoder on generating 3D chest CT scans from free-text prompts, assessing the realism, anatomical coherence, and text alignment of its latent representations. For generation, we use transformer-bag model with cross-attention layers, similar to GenerateCT [2]

---

MedSyn
GenerateCT
Ours (16x16x8)
Ours (8x8x8)
Axial
Coronal
Segiitall
Figure 3: Example generations for clinical prompt: "Chest CT scan of 63 63-year-old male: No findings compatible with pneumonia were detected. Mild sequelae changes are observed in both lungs. Atherosclerotic changes noted, with slight increase in the calibration of vascular structures in the mediastinum. Hepatosteatosis. Hiatal hernia... We show one representative slice per anatomical plane. Ground-truth volumes omitted, following standard practice in text-to-image generation.
Baselines. GenerateCT [12 adopts cascaded design res-uusotution 3D generation followed by 2D diffusion upsampling). which leads to spatial inconsistencies. This approach is motivated by the computational constraints of training high-capacity 3D encoder-decoder directly high-resolution volumes. MedSyn (20) employs a unified 3D transformer with windowed attention to improve inter-slice consistency but suffers from lower fidelity due to its lightweight architecture.
Experimental results. Table reports FID (per view and mean), FVD (with CT-Net (55) and I3D backbones), and CLIP-based alignment scores. Since CT-Net is trained on 3D chest CT volumes, provides more relevant assessment than I3D. BTB3D with lower compression achieves substantial improvements: mean FID drops from 9.51 (GenerateCT) to 2.24 (a 76.5% reduction). FVDer-Net improves by 48.3%, confirming better spatiotemporal realism, while CLIP alignment also slightly increases. The higher-compression BTB3D variant still outperforms GenerateCT and MedSyn across all metrics, though with smaller margins, highlighting a trade-off between compression and generation quality. These results underscore the effectiveness of our encoder-decodee network for high-fidelity. text-aligned 3D medical image synthesis. Figure illustrates qualitative differences: our model generates sharper volumes with clearer anatomical structures and better alignment to the clinical prompt, while MedSyn and GenerateCT outputs appear blurrier or contain inter-slice artifacts.
Dataset and implementation. We use CT-RATE, with CTs resampled as in Section Prompts are generated as: "Chest CT scan of (agel-year-old (sex]: [impression]" following GenerateCT. 12-layer transformer (1024 hidden size, heads), trained using flow matching loss (56). We apply [7, 7, 7 windowed self-attention and [2, 2, patching. Prompts are H100 GPUs for 1500 epochs, using Adam W with a learning rate of 10th The batch size is 4 for the 8 variant and for 162 x8.
The generation model
encoded using the 1. 1-ase model [57]. Training conducted

---

Conclusion
We introduced BTB3D, framework that advances vision- language modeling for 3D medical imaging. It unifies 2D and 3D processing through causal convolutional encoder-decoder and compact volumetric tokenization. Our three-stage training scales learning from local patterns to full-volume anatomical coherence, addressing memory and resolution bottlenecks. BTB3D achieves state-of- the-art results both radiology report generation and text-conditioned 3D CT synthesis. Notably, the 8x8 variant excels in fine-grained tasks such as text-to-CT synthesis, while the 16x 16x8 variant is better suited for memory--cnstrainee We believe BTB3D is significant step toward scalable and clinically meaningful vision-languag modeling in 3D medical imaging, and we expect its open open-source release to catalyze further research.
settings and high-level tasks like report generation.
Acknowledgments and Disclosure of Funding
high-perrformnce computing resources were provided by the Erlangen National High Performance Computing Center (NHR @ FAU) at Benatric--lexande---nivveeee
ärlangen-Nürnber (FAU), under the NHR projects b143dc and bl 80dc. NHR is funded by federal and Bavarian state authorities, and NHR @ FAU hardware is partially funded by the German Research Foundation (DFG) 0000-4407110 19683. Additional support was received by the ERC- project MIA-NORMAL 101083647, DFG 512819079, and by the state of Bavaria (HTA). The authors also express sincere gratitude to the Gelmuu-Horten Foundation for their generous support, which made this work possible.
References
[1] Cynthia H McCollough and Prabhakar Shantha Rajiah. Milestones ct: past, present, and future. Radiology, 309(1) ):e200803, 2023.
[2] Shaoting Zhang and Dimitris Metaxas. On the challenges and perspectives of foundation models for medical image analysis. Medical image analysis, 91:102996 2024.
[3] Ibrahim Ethem Hamamei, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Omer Faruk Durugol, Bastian Wittmann, Tamaz Amiranashvili, et al. Developing generalist foundation models from multimodal dataset for 3d computed tomography. arXiv preprint arXivv2403.17833 2024.
[4] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEF/CVF winter conference on applications of computer vision, pages 574-584, 2022.
[5] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-ana aad-uaggg-ad-viiioon assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541-28564, 2023.
[6] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.. 13971, 2023.
[7] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang. et al. Internvideo2: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision, pages 396-416. Springer, 2024.
[8] Sarthak Pati, Siddhesh P Thakur, ibrahim Ethem Hamameı, Ujjwal Baid, Bhakti Baheti, Megh Bhalerao, Orhun Güley, Sofia Mouchtaris, David Lang, Spyridon Thermos, et al. Gandlf: the generally nuanced deep learning framework for scalable end-to-end clinical workflows. Communications Engineering, 2(1):23, 2023.
[9] Fui Jin Hwang, Jin Mo Goo, and Chang Min Park. Ai applications for thoracic imaging: considerations for best practice. Radiology, 314(2))e240650 2025.
10

---

[10] Ibrahim Ethem Hamamci, Sezgin Er, and Bjoern Menze. C2rep: Automated radiology report generation for 3d medical imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 476-486. Springer, 2024.
[11] Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis. Cesar Truyts, et al. Merlin: A vision language foundation model for 3d computed tomography. Research Square, pages 3s-3, 2024.
[12] Ibrahim Ethem Hamamei, Sezgin Er, Anjany Sekuboyina, Enis Simsar, Alperen Tezcan, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Furkan Almas, Irem Dogan, Muhammed Furkan Dasdelen, al. Generatect: Text-conditional generation of 3d chest volumes. In European Conference on Computer Vision, pages 126-143. Springer, 2024.
[13] Zihao Zhao, Yuxiao Liu, Han Wu, Mei Wang, Yonghao Li, Sheng Wang, Lin Teng, Disheng Liu, Zhiming Cui, Qian Wang. et al. Clip in medical imaging: A survey, Medical Image Analysis, page 103551, 2025.
[14] Zhongyi Shui, Jianpeng Zhang, Weiwei Cao, Sinuo Wang, Ruizhe Guo, Le Lu, Ling Zhang, Tingbo Liang. Lin Yang, Xianghua Ye, al. Large-scale and fine-grained vision-language pre-training for enhanced ct image understanding. In The Thirteenth International Conference on Learning Representations, 2025.
[15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 7748-8763. PmLR, 2021.
[16] Katharina Jeblick, Balthasar Schachtner, Jakob Dexl, Andreas Mittermeier, Anna Theresa Stüber, Johanna Topalis, Tobias Weber, Philipp Wesp, Bastian Oliver Sabel, Jens Ricke, et al. Chatgpt makes medicine easy to swallow: an exploratory case study on simplified radiology reports. European radiology, 34(5):2817-2825 2024.
[17] Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar. Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature biomedical engineering, 6(12):1399-1406, 2022.
[18] Tim Tanida, Philip Müller, Georgios Kaissis, and Daniel Rueckert. Interactive and explainable region-guided radiology report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7433-7442, 2023.
[19] Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai. Nejm Ai, A33:3102200 2024.
[20] Yanwu Xu, Li Sun, Wei Peng. Shuyue Jia, Katelyn Morrison, Adam Perer, Afrooz Zandifar, Shyam Visweswaran, Motahhare Eslami, and Kayhan Batmanghelich. Medsyn: Text- fextdd anatomy aware synthesis of high-fideliyy 3d et images. IEEE Transactions on Medical Imaging, 2024.
[21] Jiahui Zhang, Fangneng Zhan, Christian Theobalt, and Shijian Lu. Regularized vector quantiza- tion for tokenized image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18476, 2023.
[22] Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. Generating radiology reports via memory-driven transformer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1439-1449, 2020.
[23] Shady
and Raja Giryes. Udpm: Upsampling diffusion probabilistic models. Advances in Neural Information Processing Systems, 37:27616-2764, 2024.
11

---

[24] Pengfei Guo, Can Zhao, Dong Yang, Ziyue Xu, Vishwesh Nath, Yucheng Tang, Benjamin Simon, Mason Belue Stephanie Harmon. Baris Turkbey, et al. Maisi: Medical ai for synthetic imaging. In 2025 JEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4430 4441. IEEE, 2025.
[25] Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8850-8860, 2024.
[26] Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen. Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander Hauptmann, et al. Language model beats diffusioontokeenzzer is key to visual generation. In The Twelfth International Conference on Learning Representations, 2024.
[27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances neural information processing systems, 25, 2012.
[28] Weiyang Liu, Zhen Liu, Liam Paull, Adrian Weller, and Bernhard Schölkopf. Structural causal 3d reconstruction. In European Conference on Computer Vision, pages 140-159. Springer, 2022.
[29] Aidan Scannell, Mohammadreza Nakhaeinezhadfard, Kalle Kujanpää, Yi Zhao, Kevin Sebastian Luck, Amo Solin, and Joni Pajarinen. Discrete codebook world models for continuous control. In The Thirteenth International Conference Learning Representations, 2025.
[30] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint wtXXv:25011035722 2025.
[31] Albert Tseng, Qingyao Sun, David Hou, and Christopher M De Sa. Qtip: Quantization with trellises and incoherence processing. Advances in Neural Information Processing Systems, 37:59597-59620, 2024.
[32] Ting Yao, Yingwei Pan, Yehao Li, Chong-Wal Wah Ngo, and Tao Mei. Wave-vit: Unifying wavelet and transformers for visual representation learning. In European conference on computer vision, pages 328-345. Springer, 2022.
[33] Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In International Conference on Machine Learning, pages 1771-1779. PMLR. 2017.
[34] Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. arXiv preprint arXiv: 0266444 2019.
[35] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In European conference computer vision, pages 17-33. Springer, 2022.
[36] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. 2021 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), volume 10, 2020.
[37] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents, 2022.
[38] Hang Zhao, Orazio Gallo, luri Frosio, and Jan Kautz. Loss functions for image restoration with neural networks. IEEE Transactions on computational imaging, 3(1):47-57, 2016.
[39] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[40] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
12

---

[41] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken. Alykhan Tejani, Johannes Totz, Zehan Wang, al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4681 4681-4690, 2017.
[42] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[43] Tao Zhou, Huazhu Fu, Geng Chen, Jianbing Shen, and Ling Shao. Hi-net: hybrid-fusion network for multi-modal mr image synthesis. IEEE transactions on medical imaging, 39())2772-22811, 2020.
[44] Alain Hore and Djemel Ziou. Image quality metries: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366-2369, IEEE, 2010.
[45] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, (3):6000 612, 2004.
[46] Tami D DenOtter and Johanna Schubert. Hounsfield unit, 2019.
[47] Ramit Lamba, John P McGahan, Michael T Corwin, Chin-Shang Li, Tien Tran, J Anthony Seibert, and John M Boone. Ct hounsfield numbers of soft tissues on unenhanced abdominal ct scans: variability between two different manufacturers' mdet scanners. AJR. American journal of roentgenology, 20315) 2014.
[48] Kingma Diederik. Adam: A method for stochastic optimization. 2014.
[49] Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794 803. PMLR, 2018.
[50] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 20:1v:2407:21173, 2024.
[51] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34999 2023.
[52] Ibrahim Ethem Hamamci, Sezgin Er, Suprosanna Shit, Hadrien Reynaud, Bernhard Kainz, and Bjoemn Menze. Crg score: A distribution-awanl clinical metric for radiology report generation. In Medical Imaging with Deep Learning- Scarning-Short Pupers, 2025.
[53] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[54] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation andfor summarization, pages 65-72.
2005.
[55] Rachel Lea Draelos, David Dov, Maciej A Mazurowski, Y Lo, Ricardo Henao, Geof- frey D Rubin, and Lawrence Carin. Machine-learnine lew-leennnnnbseeee multiple abnormality prediction with large-scale chest computed tomography volumes. Medical image analysis, 67: 67:101857, 2021.
[56] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden and Saining Xie. Sit: Exploring flow and diffusion- generative models with scalable interpolant transformers. In European Conference Computer Vision, pages 23-40. Springer, 2024.
[57] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
13

---

[58] M Jorge Cardoso, Wenqi Li, Richard Brown, Nic Ma, Eric Kerfoot, Yiheng Wang, Benjamin Murrey, Andriy Myronenko, Can Zhao, Dong Yang, al. Monai: An open-source framework for deep learning in healthcare, arXiv preprint (rxXv:2211.00200)) 2022.
[59] Xueyan Mei, Zelong Liu, Philip Robson, Brett Marinelli, Minggian Huang, Amish Doshi, Adam Jacobi, Chendi Cao, Katherine E. Link, Thomas Yang. Ying Wang. Hayit Greenspan, Timothy Deyer, Zahi A. Fayad, and Yang Yang. Radimagenet: An open radiologic deep learning research dataset for effective transfer learning, Radiology: Artificial Intelligence, 0(a))e210315, O.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state two main claims: (1) BTB3D achieves state-of-the-ard performance on both radiology report generation and text- conditional 3D CT synthesis, and (2) its improvements stem from a unified causal encoder- decoder architecture and three-stage training strategy. These claims are thoroughly validated Section through reconstruction analysis (Section 4.1), report generation bench- marks (Section 4.2), and CT synthesis metrics (Section 4.3). Quantitative improvements (such as 40% Fl gain over CT-CHAT for report generation and a 76.5% FID reduction over GenerateCT) directly support the claims. The paper also discusses architectural trade-off's and generalization, aligning the claims with the demonstrated scope and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses several limitations throughout and includes dedicated section the Appendix. Specifically, we acknowledge that BTB3D is evaluated only on chest CT's due to the lack of large-scale paired datasets for other anatomies and modalities. Also, our method requires substantial computational resources for training, which may limit accessibility, Finally, although we evaluate performance using both clinical and generative metrics, further validation through expert reader studies and prospective clinical trials is necessary to assess real-world safety and efficacy.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
15

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper focused on an empirical vision- -aaguane modeling framework for 3D medical imaging and does not contain theoretical results, assumptions, or proofs.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed methodology in Section 3 and implementation details in Section 4. Our experiments use the publicly available CT-RATE dataset for training and in-domain evaluation, and RadChestCT for out-of-distribution validation. We release all training and validation scripts, along with pretrained model weights, ensure full reproducibility of the results and support verification of all main claims.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
16

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We will open-source our codebase, including all training, validation, and evaluation scripts. The repository will contain detailed instructions on setting up the environment, accessing and preprocessing the CT-RATE and RadChestCT datasets, and reproducing all experiments reported in the paper. Pretrained weights for BTB3D and the generation models will also be provided to ensure faithful replication of our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
guuides/CooeSuumissttss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
ctips.cc/public//gge gwtgg/uddeesee/ http://iii..ggiaeeee/////aaaaaas
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and evaluation details thoroughly described in Section 4, including dataset splits, preprocessing steps, model configurations. optimizer settings, batch sizes, learning rates, training schedules, and hardware setup. We also specify stage-wise training parameters and evaluation protocols for each task, ensuring the experimental setup is fully transparent and reproducible.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the substantial computational cost of training large 3D models on high- resolution CT volumes,
did not conduct repeated runs or report statistical significance metrics such as error bars or confidence intervals. Our focus was on demonstrating consistent performance gains across multiple benchmarks using standardized evaluation protocols. We acknowledge this as limitation and will consider statistical analysis in future work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We detail the hardware and software setup in Section Batch sizes, memory constraints, and stage-wise wise training configurations are specified, allowing estimation of training time and resource requirements. This enables reproducibility and helps assess the computational cost of our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
18

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppee///ourrsss.cccc/////s//ts
Answer: [Yes]
Justification: We have carefully reviewed the NeurlPS Code of Ethics and ensured that all aspects of our research (including data usage, model development, evaluation, and reporting) fully comply with its principles. The datasets used are publicly available, de-identified, and ethically sourced, and we maintain transparency and reproducibility throughout the work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We provide a discussion of both potential positive and negative societal impacts in the Appendix. On the positive side, BTB3D has the potential to improve diagnostic accuracy, reduce radiologist workload, and enable better access to training tools through high-quality synthetic CT data. On the negative side, we acknowledge potential risks such as misuse generative models for medical fraud or reconstruction sensitive information from improperly anonymized data. We emphasize the importance of proper anonymization and include cautionary notes regarding ethical deployment and data governance. Guidelines:
Justification:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes]
Question:
19

---

Justification: While we rely solely on publicly available datasets, we acknowledge the potential misuse of generative models. To mitigate this, access to our pretrained models is gated: researchers must apply for access to the model weights, and requests are manually reviewed to ensure responsible use. This safeguard helps prevent fraudulent or unethical deployment of our generation models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets used are publicly available datasets, and we comply with their respective licenses (We cite the original papers and follow license terms). Specifically:
CT-RATE dataset: Released under the CC BY-NC-SA 4.0 license.
RadChestCT dataset: Released under the CC BY-NC-ND 4.0 license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release all new assets (including code, pretrained models, and training scripts) under the CC BY 4.0 license. The GitH-u repository includes comprehensive documentation covering usage instructions, training configurations, dependencies, and licensing terms to facilitate reproducibility and reuse.
Guidelines:
The answer NA means that the paper does not release new assets.
20

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve human subjects and therefore does not require IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were used only writing and editing assistance, not as part of the core methodology.
Guidelines:
21

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, non-standard components.
Please refer to our LLM policy (https://neuripg /ttp://pp//cc//offfeeeee//////// for what should or should not be described.
22

---

A Technical Appendices and Supplementary Material
In this supplementary document, we provide additional insights and supporting materials for our main paper. We begin by outlining the key limitations of our work and discussing the broader societal and clinical implications of the BTB3D framework, particularly with regard its capabilities in radiology report generation and coxt-iondtio 3D CT volume synthesis. We then present further experimental details that expand upon the methodology described in the main text. Additional qualitative and quantitative results are also included to further validate our findings. All code, pretrained model weights, and instructions to reproduce our experiments will be made openly available in our GitHub repository to promote transparency, reproducibility, and further research in the field.
A.1 Limitations and Broader Impacts
Limitations. While our BTB3D framework significantly improves tokenization and decoding for 3D medical vision- lasion-language modeling (especially for 3D chest CT scans), several limitations remain. First, our current framework does not explicitly model clinical reasoning or uncertainty, both of which are crucial for real-world deployment in clinical settings. Incorporating modules for uncertainty estimation or causal inference remains an open challenge for future studies.
Second, the scope of our experiments is limited to 3D chest CT scans, primarily due to the lack of large-scale, publicly available paired datasets (with reports) for other anatomical regions or modalities (e.g., MRI, PET). While BTB3D is designed to generalize across 2D and 3D inputs, we have not yet validated its transferability to other clinical domains. Future efforts should extend the architecture to whole-body imaging and explore zero-shot or few-shot generalization across organs.
Third, although BTB3D supports both 2D and 3D training modes, we used only 2D slices extracted from 3D volumes in CT-RATE to remain consistent with baselines and ensure fair comparisons. As such, we have not demonstrated its transfer capabilities (e-g., transfer from pretrained 2D) models or joint training with 2D paired datasets), which limits the evaluation of BTB3D's capabilities.
Fourth, statistical analysis is limited due to the high computational cost of 3D training. We did not perform repeated runs or report confidence intervals. Thus, while our results demonstrate strong and consistent performance across multiple benchmarks, statistical significance remains to be validated.
Lastly, although we evaluate both clinical and generative metries, BTB3D has not yet been assessed in real clinical workflows. Validation through expert reader studies or prospective trials is necessary to fully establish its utility, reliability, and safety in decision-making environments.
Broader impacts. BTB3D offers promising benefits for both clinical and research communities. High-quality radiology report generation from 3D CT scans can reduce reporting delays, alleviate radiologist burnout, and improve documentation consistency, particularly in high- sogumm settings such as emergency departments or large-scale screening programs. Moreover, BTB3D's ability to synthesize realistic, anatomically plausible CT scans from text opens new avenues for training, simulation, and rare disease modeling. In educational settings, synthetic CT scans can used to create diverse training sets for radiology students and enhance learning outcomes through exposure to rare or atypical cases. In research, BTB3D may support data augmentation, helping mitigate class imbalance in supervised learning tasks and enabling pretraining in low-resource environments.
However, the generative capabilities of the BTB3D framework also present potential risks. Synthetic scans, if not properly labeled or constrained, may be misused in regulatory or insurance contexts, or exploited for fraudulent purposes. Additionally, there is a non-trivial risk of inadvertently replicating identifiable patient anatomy, even when training data is anonymized. While our work uses only pub- licly available, fully de-identified datasets (e.g.. CT-RATE), developers must ensure strict compliance with data governance and anonymization protocols when deploying similar models.
Finally, the substantial computational cost of training such models may exacerbate disparities in access to advanced medical AI. Ensuring open-source availability and supporting low-resource inference are critical to democratizing BTB3D's benefits. Continued oversight and interdisciplinary dialogue will be essential to ensuring the responsible deployment of generative models in healthcare.
23

---

A.2 Additional Experimental Details
Three-stage training performance. To further evaluate the impact of our progressive training strategy, we qualitatively assess the reconstruction performance of BTB3D across the three stages of training in Figure 4 The figure presents axial, coronal, and sagittal slices reconstructed by our two model variants (16x 16x8 and 8x8x8) 8) at each stage and compares them to the ground truth.
In Stage 1, the model is trained solely small, non-overlapping 3D volumes and 2D slices. This initialization allows the model to learn basic volumetric structure and inter-slice continuity but suffers from block artifacts and poor spatial coherence, especially in regions with high anatomical complexity, As seen in the leftmost columns of Figure 4 both variants at this stage produce noisy and blurry reconstructions with limited detail and sharpness. Stage 2 introduces overlapping windows during training, which significantly improves anatomical consistency across slices and helps reduce the block discontinuities learned in Stage This stage yields the largest qualitative leap in fidelity, with more well-defined boundaries of lung lobes, airways, and soft tissue structures. The improvements are most noticeable in the axial and sagittal views, where inter-slice alignment and smoothness are critical for clinical interpretabilityy. Stage 3 refines the decoder with high-resolution patches while keeping the encoder part frozen. This final stage enhances local detail, texture realism, and structural sharpness without sacrificing the global consistency achieved in Stage 2. Notably, lung fissures, pleural contours, and fine vascular structures become visibly clearer, indicating that the decoder has learned to reconstruct complex anatomical regions with higher fidelity and resolution.
Comparing the two BTB3D variants, the 8x8x8 model consistently achieves superior anatomical fidelity and inter-slice coherence in reconstruction, as expected due to its lower compression rate and higher capacity. In contrast, the 16x 16x8 variant (while more compressed) proves more effective for language-drin -riieen tasks such as report generation, where coarse global structure suffices and memory efficiency is critical. Our stage training pipeline plays a pivotal role for both models: first captures global structure, then progressively refines local detail, enabling accurate and high-resolution 3D reconstructions from compact tokens. This staged optimization bridges the gap between token efficiency and clinical utility, facilitating both high-fidelity text-conditional CT generation and precise report generation, underscoring BTB3D's versatility in multimodal 3D medical image understanding.
Radiology report generation from 3D chest CT. We comprehensively evaluate BTB3D's capabil- ity in generating accurate and clinically coherent radiology reports from volumetric chest CT scans. As described in the main paper, each CT volume is first compressed into a compact sequence of frequency-aware 3D tokens by our encoder, and these tokens are then passed to a pretrained LLM (LLaMA 3.1 8B) for report generation via a linear projection layer. To ensure fairness, we use the official weights for CT2Rep and CT-CHAT, both trained on the same CT-RATE dataset. Since Merlin was originally trained available, we retrained Merlin on CT-RATE using its official codebase and default hyperparameters.
private dataset and neither its weights nor training data publicly
Our evaluation includes both internal (CT-RATE test set) and external (RAD-ChestCC) benchmarks. The results Table 5 show that BTB3D (particularly the 16x 16x 8 variant) outperforms prior methods in average abnormality level Fl scores, achieving higher clinical precision in most of the findings. This trend holds in Table 6 where our BTB3D framework demonstrates strong out-of- distribution generalization, with the 16x 16x8 variant yielding 46% relative Fl improvement over CT-CHAT (the previous state-of-the-ard method). This highlights the robustness of our tokenization and training pipeline, even when evaluated on unseen institutional distributions.
Figure offerss qualitative comparison across models, illustrating report generation for the same CT scan. BTB3D's 's reports more faithfully reproduce key clinical details from the ground truth, particularly with the higher compression (16x 16x8) variant, supporting the notion that coarser representations, while less suitable for pixel-level synthesis, may better capture global semantics for language modeling. Meanwhile, the 8x8x8 variant provides more spatially detailed reconstructions but slightly underperforms text generation metrics, suggesting trade-off between compression depth and semantic abstraction. In all comparisons, BTB3D demonstrates a compelling advantage by combining precise volumetric tokenization with scalable training strategy, ultimately allowing both -rained anatomical reconstruction and clinically relevant report generation.
24

---

Text-conditional 3D chest CT generation. To evaluate the generative capabilities of BTB3D, we benchmark its performance on synthesizing realistic and anatomically coherent 3D chest CT volumes from free-text clinical prompts. As shown in Figure 6 BTB3D generates sharper, more anatomically faithful volumes compared MedSyn and GenerateCT, with improved inter-slice consistency and alignment to prompt semantics. For fair comparison, we use the official pretrained weights for both MedSyn and GenerateCT, which were trained on the same modality (3D chest CT), ensuring that differences in performance stem from architectural and training innovations.
Quantitative results reported in the main paper (Table 4) show that our BTB3D framework with the lower compression variant (8x x 8x8) achieves the best overall performance across all generative metrics. Specifically, it reduces the mean Fréchet Inception Distance (FID) from 9.51 (GenerateCT) and 12.59 (MedSyn) to just 2.24, 76.5% improvement, demonstrating superior fidelity. We compute FII using the FIDMetric from the MONAI library (58), leveraging a Badlmmageett-rerraiing ResNet50 backbone [59], which is better suited for grayscale radiology images than traditional Inception networks. Following standard practice established by GenerateCT and MAISI (24), FID) is calculated on the central 40% of slices (in each anatomical plane) across 100 randomly selected volumes per method, reducing boundary noise and focusing on clinically relevant regions.
In terms of temporal and anatomical realism, Fréchet Video Distance (FVD) is computed using both CT-Net (specialized for 3D chest CTs) [55] and I3D (trained on RGB videos). BTB3D again significantly outperforms prior work, halving the FVD compared to GenerateCT. To assess semantic alignment between text prompts and generated volumes, we compute CLIP scores using the CLIPScore implementation from Torchmetrics. We follow GenerateCTTss s protocol: axial slices are resized 224 224 and converted to pseudo-RGh by repeating the single intensity channel. Using clip-vit-base-pate
we observe that BTB3D achieves the highest text-image alignment score (24.27), suggesting it captures fine semantic cues better than prior methods.
Interestingly, we note a compression-qualliy trade-off: while the 8x8x8 variant excels in fine- grained reconstruction and text-conditional volume synthesis, the more compact 16x 16x8 variant still surpasses existing baselines and may be preferable in memory-constrainee or latency-sensitive settings. Together, these results confirm that BTB3D's volumetric tokenization and three-stage training pipeline offer a significant leap forward in text-condiie cext-conditioal 3D medical image generation, bridging the gap between semantic understanding and pixel-level anatomical coherence.
25

---

Stage
Stage
Slage
Ground
Axtal
Coronal
16
Sagittal
Axial
8x8xx
Coronal
Sagittal
Figure 4: Qualitative reconstruction results across axial, coronal, and sagittal planes for two BTB3D variants: 16x 16x8 (top) and 8x8x8 (bottom). The figure shows outputs after Stage 1 ishort-volume training), Stage 2 wverlapping-windes training), and Stage 3 (decoder refinement), compared to the ground truth. The progressive improvements highlight the effectiveness of our three-stage training strategy. Stage 2 yields the largest gain in anatomical fidelity and inter-slice consistency, while Stage 3 further sharpens structural details such as lung fissures and vascular boundaries.
26

---

Table 5: Abnormality-based Fl scores on CT-RATE along with the prevalence ratio of each abnormal- ity in the test set. BTB3D consistently achieves the highest clinical accuracy across categories, with the cigher-compression variant (16x 16x8) demonstrating superior overall performance.
Abnormality
Ratio
16
Ours-8
CT-CHAT
Merlin
CT2Rep
Medical material
0.103
0.142
0.120
0.006
0.057
0.000
Arterial wall calcification
0.285
0.414
0.273
0.451
0.262
0.322
Cardiomegaly
0.107
0.305
0.207
0.123
0.176
0.013
Pericardial effusion
0.074
0.095
0.095
0.009
0.060
0.000
Coronary artery wall cale.
0.252
0.403
0.260
0.412
0.235
0.335
Hiatal hernia
0.137
0.164
0.118
0.207
0.110
0.074
Lymphadenopathy
0.260
0.358
0.209
0.069
0,227
0.013
Emphysema
0.197
0.196
0.155
0.391
0.216
0.198
Atelectasis
0.235
0.269
0.242
0.341
0.199
0.323
Lung nodule
0.448
0.427
0.397
0.443
0.290
0.029
Lung opacity
0.390
0.408
0.382
0.266
0.312
0.557
Pulmonary fibrotic sequela
0.273
0.318
0.211
0.069
0.117
0.104
Pleural effusion
0.124
0.308
0.199
0.173
0.183
0.341
Mosaic attenuation pattern
0.083
0.183
0.094
0.064
0.076
0.198
Peribronchial thickening
0.117
0.125
0.043
0.000
0,054
0.099
Consolidation
0.191
0.259
0.185
0.120
0.174
0,236
Bronchiectasis
0.109
0.126
0.094
0.091
0.075
0.013
Interlobular septal thick.
0.082
0.135
0.087
0.075
0.065
0.032
Mean
0.193
0.258
0.187
0.184
0.160
0.160
Table 6: Abnormality-win wise F1 scores on the RAD-ChestCT dataset (external test set), along with the prevalence ratio of each abnormality. BTB3D demonstrates strong generalization performance, particularly with the 16x 16x8 8 variant, achieving the highest F1 score across most categories.
Abnormality
Ratio
Ours-16
Ours-8
CT-CHAT
Merlin
CT2Rep
Medical material
0.327
0.235
0.154
0.000
0.107
0.000
Calcification
0.706
0.671
0.406
0.567
0.359
0.434
Cardiomegaly
0.109
0.187
0.151
0.181
0.031
0.010
Pericardial effusion
0.155
0.193
0.101
0.007
0.047
0.089
Hiatal hernia
0.117
0.149
0.087
0.149
0.136
0.173
Lymphadenopathy
0.165
0.252
0.227
0.121
0.176
0.000
Emphysema
0.273
0.201
0.172
0,412
0.242
0.142
Atelectasis
0.298
0.350
0.265
0.387
0.194
0.153
Lung nodule
0.802
0.424
0.408
0.721
0.495
0.068
Lung opacity
0.539
0.542
0.389
0.140
0.394
0.560
Pulmonary fibrotic sequela
0.132
0.204
0.139
0.026
0.174
0.157
Pleural effusion
0.200
0.290
0.199
0.043
0.135
0.032
Peribronchial thickening
0.085
0.117
0.064
0.000
0.095
0.039
Consolidation
0.139
0.217
0.145
0.108
0.149
0.264
Bronchiectasis
0.158
0.128
0.084
0.034
0.122
0.000
Interlobular septal thick.
0.069
0.091
0.075
0.008
0.057
0.000
Mean
0.267
0.266
0.192
0.182
0.182
0.133
27

---

Findings: Trachea and both main bronchi open. No occlustre pathology was detected in the traches and both main bronchi There are linear stolectasis the right lung middle lobe and medial lobo anterior segment, and left lobe Engular segment. Minimal emphysemetous changes
observed in both lunga, There is millimetrie nodule In upper lobe of the right lung. No mass infirratvv lasion was detected both lunga. Ao far can be observed: Heart contour and size are normal No pieural or pertcardial offasion was detected, The widths of the mediastinal main vascular structures are normal. No pathologically enlarged lymph nodes were detected in the mediastinum and hilar regions. No pathological wall thickness increase was observed in the esophague within the sections. No upper abdominal free fludd-clllacia pathological dimansions were detacted. In the upper abdominal organs within the sections, there no masa with distinggishabll borders as far as can be observed within the borders of non-enhanced CT. Thoradie vertebral corpus heights, algnments and densities are normal. There are osteophytes the vertebral corpus corners. impression: Minimal emphycemstous changes in both lungs. Linear stelectasis in both lungs. Millimetric nodule in the upper lobe of the right ling.
Ground Truth
was detected in the sactions. No enlarged lymph nodea in
Mediastinal main vascular structures, heart contour, size normal Thoradic aorta diameter normal. not observed. Thoracic esophageal calibration normal and significant tumoral wall thickaning welanged lymph nodes in prevascular, pre-garatrachee pra-paratracheel, subcarinal or bilateral hilar-axilarry pathological dimensions were detected. When acamined in the lung parenchyma window, Minimal paraseptal amphysomatous changes are observed In the right lung lower lobs superior segmant postarolataral. Linear stolectasis changes are abserved in the right lung middle and upper lobo anterfor segment. There a nodule with a diameter pneumonia both lungs. Upper abdominal organs included in the
Findings: Trachea, both main bronchi
Pericardial effusion-thickoni
detected. No
endarged
Ours (16x16x8)
mm in the
lobe of
the right lung. There was no finding compatible with picural effusion, pnsumothorax sections are normal. No space-occupying lesion was detected in the liver that entered the cross-sectioaa area. Bilateral adrenal glands were normal and no spece- occupying leston was detected. Mild deg Segeneratvee changes are observed in the bone structure entering the esaminattion area. Vertebral corpus hiights are preserved.
Impression: Minimal paraseptal emphysematous changes in the right lung lower lobe superior segment posterolateral, linear subpleural atelectasis changes in the right kung middle and upper lobe anterior segment, nonspecific nodule in the upper lobe of the right lung: no fincings in favor of pneumonts weree detected.
main branchi open. occlusive pathology was detected trachee both bronchi. left calcific nodules interiobular changes observed both is minimal pericardial effusion. The widthe the mediaetinal main vascular structures normal. There pothologically enlarged hmmh nodes pathological wall thicknees increase in the esophagus within the sections. No abdominal free fluid-
Findings Trachea
evaluated
Ours (6x8x8)
stelectasis both evalusted optimally because
Infitrative lasion detected both lungs. Mediastinna be observed: Heart contour and
structures
There
mediassiimm and hiler regions. There collection was delected the sections, No enter ged yymph nodes pathological dimensions were delected, No frectures or dylic-eetructive lestons were abserved the bone structures within the poctions.
Impression: empayssmatuus changes in both lungs. Locally linear atelectasis in both lungs. Mnimal pericarcial effusion.
Findings Trachea both main bronahi
No occlustve pathology detected millmmerrc nonspecific nodules both
tracheo both main bronchL There minimal
both lungs. Mediastinal structuree cannot evaluated optimely because contrest material not given. As far can be observed Heart contour and size are normal. No
emphyeematous
infiltratvv lesion was detected both
CT-CHAT
pleural paricardial effusion was detected. The widths of the mediastinal main vascular stractures are normal. No pathologically enlarged lymph nodes
wore detected in the modiastinum and hilar rogions. No pathological wall thickness increase was observed in the esophagus within the sections. No upper
abdominal free ffuld-collection was detected In the sections. No enlarged lymph nodes in pathological dimensions were detected. in the upper abdominal
organs within the sections, there is no mass with distingulshable barders as far as It can be observed within the borders of non-enhanced CT. Thoracls
vertebral corpue help nightt, alignments and deneities are normal, Intervertebral disc distances are preserved. The neural forwmina are open. No lytoodestrudivve lealona were detected in the bone structures es within the sections.
Impression: Minimal emphysematios changes in both lungs. Millimatric nodules in both lungs.
Findings Trachea, both bronchi open. Mediactinal main vascular structures, heart contour,
diameter
esophagus callbration enlarged hymph nodes prevascular, pre-paratracheal, subcarinal bilateral hilar- axillary pathological dimenslons detected. When examined the
effusioon thickeniingg
thickening detected. No
Merlin
parenchyma window, Acretion of both lung parenchyma included the sections normal, No the crosss nectional area. Bilatenil adrenal glands were normall and spece- oocupyngguggggooooooo lesion detected, Bone structures the study srea netural.
and no nodular infibrative lesion detected the lung parenchyms, Pleural effusionn
thickening was detected Upper abdominel
lesion detected the that entered
Vertabral corpus heights preserved.
Impression: Thoracic CT examination within normal limits.
Findings: Trachea and both main branchi are open. No ooclusive pathology was detected in the trachea and both main bronchi There is minimal
bronchiestasis the central parts of both lungs. Gocasionally, linear atolectasis was observed both lungs. There are emphysematous changes in both
lungs. In the right lung, there are millimetric nodules with ground-glass aress around some of them When evalusted together with the patient's primary
disesse. these appearances were primerily evaluated favor meiastases. It is recommented that the palient be evalusted logether with previous exeminations, any There is no miss infitrative lesion both lungs. Mediestinnl sinucturee carnot be evalaited optimelly because contrast material not givan. As can observed, heart larger normal. There no pleural or parieardial offusion. There millmmttic atheroma plaques the aorts. No pathologically enlarged lymph nodes were detected sections. upper abdominal free collection detected
CT2Rep
thickness increase the sections. enlarged hymph in pathological dimenstons bone structures within sections. Although described appeerances cannot thought that presenca primary disease could indicate
mediastinum hilse regions.
no pathological
esophagus were There sometimes millmetric hypodense they vary Investigation recommanded.
characterived
these appearances. Further
Figure 5: Example of radiology report generation for the same 3D chest CT scan using our BTB3D method (both 16x 16x8 and 8x8x8 variants) compared to baseline models (CT-CHAT, Merlin, CT2Rep) and the ground truth report. Key phrases from the ground truth are highlighted and matched across model outputs using consistent colors to indicate alignment. Our BTB3D framework. especially the higher compression rate variant, produces more detailed, clinically relevant, and accurate radiology reports, showing superior coverage of anatomical structures and abnormalities.
28

---

MedSyn
GenerateCT
Ours (16x16x8)
Ours (8x8xB)
Axdel
Coronal
Sagittal
Chest CT scan of 32 years old male: consolidative area the posterior segment of the right upper lobe.
MedSyn
GenerateCT
(16x16x8)
(8x8x8)
Adal
Coronal
Sagittal
Chest CT scan of 31 years old male: Findings within normal limits.
Figure 6: Text-conditioned CT generation results for two clinical prompts using MedSyn, GenerateCT, and our BTB3D models. For each case, show representative slice per anatomical plane. The lower-compression variant of BTB3D produces the most consistent volumes, demonstrating superior alignment with the prompt. Prompts and corresponding anatomical regions highlighted using color overlays. Ground-truth volumes are omitted, following standard practice in generative tasks.
29


Reviews:
[{"review_id": "LNs8WDaIXy", "reviewer": "Reviewer_Ks7t", "summary": "This paper introduces BTB3D, a causal convolutional encoder-decoder framework for vision-language modeling in 3D medical imaging. To generate compact and semantically meaningful volumetric tokens, the model combines wavelet-based compression, causal 3D convolutions, and lookup-free quantization. To address limitations of contrastive pretraining misaligned with clinical semantics, BTB3D adopts a three-stage training strategy: (1) short-volume pretraining, (2) overlapping temporal tiling, and (3) long-context decoder refinement. Experimental results demonstrate that BTB3D achieves state-of-the-art performance on both radiology report generation and text-conditioned CT synthesis tasks.", "questions": "1. Could the authors discuss existing efforts in unifying 2D and 3D modeling in medical imaging, and how BTB3D differs from or builds upon them?\n\n2. Could the authors compare BTB3D’s tokens against alternative tokenization methods, such as VQ-VAE, CLIP-based features, or anatomy-aware tokens, to better support the \"better token\" claim?\n\n3. Could the authors provide ablation studies comparing report generation and text-to-CT synthesis performance between single-stage and three-stage training? This would more directly validate the proposed training strategy for the intended applications.", "limitations": "The paper includes a limitations discussion, noting the chest CT-only scope, computational requirements, and lack of clinical user studies.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n\n-\tThe model achieves strong performance in both report generation and image synthesis tasks, outperforming state-of-the-art baselines in F1, BLEU, and FID/FVD metrics.\n-\tThe causal convolutional design and wavelet-based compression allow training and inference on long CT sequences under limited memory.\n-\tThe authors provide detailed implementation and training setup and claim to release code and pretrained weights, which supports future use and benchmarking.\n\n**Weaknesses**:\n\n-\t**Incomplete related work discussion**: While the paper emphasizes the novelty of unifying 2D and 3D training and inference while producing compact, frequency-aware volumetric tokens, it does not adequately discuss prior work in 2D–3D unification in the medical imaging domain. Relevant approaches, such as [1] [2], are not reviewed. In addition, the related work section lacks coverage of tokenization techniques like [3-5]. This weakens the contextual positioning of the contribution.\n\n-\t**Insufficient validation of the proposed training strategy**: The effectiveness of the proposed three-stage training is primarily validated via reconstruction metrics (e.g., PSNR, SSIM). However, the downstream tasks of interest (i.e., radiology report generation and text-to-CT synthesis) depend on semantic alignment rather than pixel-wise fidelity. It is unclear whether better reconstruction directly leads to better language generation or synthesis. The lack of direct comparison on downstream task performance across different training stages weakens the claim that the three-stage training benefits the end tasks.\n\n-\t**Lack of evidence for the “better token” claim**: While the tokenization is central to the paper’s claims, there is no direct comparison to other tokenization strategies (e.g., VQ-VAE, CLIP embeddings) to validate its advantage.\n\n[1] Xie, Yutong, et al. \"Unimiss: Universal medical self-supervised learning via breaking dimensionality barrier.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] He, Xiaoxuan, et al. \"Unified Medical Image Pre-training in Language-Guided Common Semantic Space.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[3] Zhang, Jiahui, et al. \"Regularized vector quantization for tokenized image synthesis.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[4] Tseng, Albert, et al. \"Qtip: Quantization with trellises and incoherence processing.\" Advances in Neural Information Processing Systems 37 (2024): 59597-59620.\n\n[5] Yao, Ting, et al. \"Wave-vit: Unifying wavelet and transformers for visual representation learning.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.", "was_revised": true, "final_justification": "I recommend the paper for acceptance because the authors have adequately addressed my concerns regarding: (1) how BTB3D relates to prior efforts on unifying 2D and 3D modeling in medical imaging; (2) the empirical support for the claim of using “better tokens” through comparisons with alternative tokenization strategies; and (3) the effectiveness of the proposed three-stage training strategy, validated through additional ablation studies on downstream tasks like report generation. Their rebuttal was thoughtful and thorough, and I believe the revised version will make a meaningful contribution to the field.", "rebuttal": "## Reviewer Ks7t\n\nWe sincerely thank the reviewer for the thoughtful and constructive feedback, as well as for recognizing the strengths of our work, including the strong performance across tasks, the novel causal and frequency-aware tokenization strategy, the three-stage training scheme, and our detailed implementation and open-sourcing commitment.\n\nBelow we provide detailed point-by-point responses to the reviewer’s concerns.\n\n---\n\n### **1. Incomplete Related Work Discussion**  \n> *\"Could the authors discuss existing efforts in unifying 2D and 3D modeling in medical imaging, and how BTB3D differs from or builds upon them?\"*  \n> *\"The related work section lacks coverage of tokenization techniques like [3–5]. This weakens the contextual positioning of the contribution.\"*\n\nThank you for highlighting this important point. We agree that the related work section can be improved to better ground our contributions. In our original submission, we focused primarily on two task-specific areas that we aimed to improve:  \n- **Radiology report generation from 3D medical images**, and  \n- **Text-conditional 3D medical image generation**\n\nThis focus was motivated by our goal to advance these downstream applications specifically in the context of **3D chest CT**, a domain that only recently became accessible at scale thanks to the release of large multimodal datasets such as **CT-RATE [6]** and **Merlin [7]**. While the studies [1,2] suggested by the reviewer indeed explore 2D–3D unification, they were proposed before the availability of such datasets and do not directly target downstream vision-language tasks in 3D medical imaging.\n\nNonetheless, we fully agree that two additional sections would enrich the related work, particularly if placed before the discussion of downstream tasks (i.e., report generation and text-to-CT generation), to better contextualize our contributions:\n- **Tokenization strategies in computer vision**: We will discuss strategies such as regularized vector quantization [3], QTip [4], and Wave-ViT [5], and explain how BTB3D adapts these concepts to the 3D CT domain, which involves long volumetric sequences and significant computational constraints.\n- **Unified 2D–3D pretraining in medical imaging**: This section will include discussions of Unimiss [1] and Unified Medical Image Pretraining [2]. Compared to these, BTB3D not only supports 2D/3D unification, but also scales to long, high-resolution 3D volumes and demonstrates strong performance on downstream VLM tasks such as report generation and text-to-volume synthesis. BTB3D’s key contribution is not only unification, but also scalability to long sequences, which is not addressed by these prior works.\n\nWe will incorporate these additions and citations in the revised paper to strengthen the contextual grounding. We thank the reviewer for this helpful suggestion.\n\n---\n\n### **2. Insufficient Validation of the Three-Stage Training Strategy**  \n> *\"Could the authors provide ablation studies comparing report generation and text-to-CT synthesis performance between single-stage and three-stage training?\"*\n\nThank you for this valuable suggestion. We initially validated the three-stage training strategy through **reconstruction metrics** (PSNR, SSIM, MSE) and qualitative reconstructions (see Figure 2 in the main paper), assuming that improvements in fidelity would translate into better downstream performance.\n\nHowever, we agree that this assumption should be tested explicitly, particularly since downstream tasks (such as **report generation**) rely on **semantic alignment** rather than pixel-wise similarity.\n\nTo this end, we conducted an ablation study evaluating **report generation performance** at each training stage using the BTB3D 16×16×8 model. The results below were computed on our internal validation set using F1 score, CRG score, and BLEU mean:\n\n| **Stage**     | **F1 ↑** | **CRG ↑** | **BLEU ↑** |\n|---------------|----------|-----------|------------|\n| Stage 1       | 0.154    | 0.349     | 0.176      |\n| Stage 2 & 3   | **0.258** | **0.370** | **0.305**  |\n\nImportantly, since Stage 3 only refines the decoder and does **not affect the encoder**, it has no impact on report generation, which relies on encoder features. Stage 3 is specifically designed to improve reconstruction quality and text-to-CT generation fidelity. Therefore, report generation performance remains unchanged between Stage 2 and Stage 3.\n\nThe majority of the gains in report generation come from our **temporal overlapping tiling strategy introduced in Stage 2**, which enforces long-range consistency and better captures volumetric semantic context, essential for generating clinically plausible reports.\n\nThese results confirm that our progressive training scheme (especially the temporal tiling) significantly enhances VLM performance. We will include these results and their implications in detail in the final version of the paper. We thank the reviewer for this valuable suggestion.\n\n---\n\n### **3. Lack of Evidence for the “Better Token” Claim**  \n> *\"Could the authors compare BTB3D’s tokens against alternative tokenization methods, such as VQ-VAE, CLIP-based features, or anatomy-aware tokens?\"*\n\nWe appreciate this constructive suggestion. In fact, our baselines **already incorporate diverse tokenization strategies**, enabling indirect yet meaningful comparisons:\n- **CT2Rep** employs a CapPa-style encoder [8].\n- **Merlin** and **CT-CHAT** rely on CLIP-like contrastive embeddings.\n- **GenerateCT** and **MedSyn** use VQ-VAE/VQ-GAN tokenizers.\n\nSince all models are trained on the **same CT-RATE dataset**, the observed performance gains of BTB3D (especially in **Tables 2–4**) can largely be attributed to its **superior tokenization approach**. For example, **CT-CHAT** and **BTB3D** use the **same LLM backbone (LLaMA 3.1-8B)**, with the **only difference** being the tokenization strategy. BTB3D outperforms CT-CHAT significantly, which underscores the effectiveness of our causal, frequency-aware 3D tokenization and three-stage training scheme.\n\nSpecifically:\n- In **report generation**, BTB3D shows clear improvements in F1 and BLEU scores over CT-CHAT and Merlin, suggesting more effective language-guided representations.\n- In **text-to-CT synthesis**, BTB3D significantly outperforms GenerateCT and MedSyn in **FID**, **FVD**, and **CLIP scores**, confirming that its tokenization enhances both realism and semantic alignment.\n\nWe agree that these comparisons were not emphasized clearly enough in the initial submission. In the revised version, we will:\n- Clarify that **tokenization is the only varying component** in models like BTB3D and CT-CHAT.\n- Explicitly link BTB3D’s improvements to its **discrete, causal, volumetric tokenization and three-stage training scheme**.\n- Include **t-SNE visualizations** of the learned token spaces (omitted here due to rebuttal policy) comparing BTB3D with other models. Preliminary results show that BTB3D produces more coherent and semantically clustered token embeddings. We believe these visualizations are particularly important: if we claim that BTB3D generates better tokens for 3D computed tomography, it is essential to provide empirical evidence (such as improved structure and interpretability in the embedding space) to substantiate this claim.\n\n---\n\n### **Limitations**\n\nWe sincerely thank the reviewer for acknowledging our limitations section, described in detail in Appendix A.1, including the current restriction to chest CT, the absence of clinical user studies, and the computational demands of training.\n\n---\n\n**References:**  \n[1] Xie et al., ECCV 2022. *Unimiss: Universal medical self-supervised learning via breaking dimensionality barrier*  \n[2] He et al., ECCV 2024. *Unified Medical Image Pre-training in Language-Guided Common Semantic Space*  \n[3] Zhang et al., CVPR 2023. *Regularized vector quantization for tokenized image synthesis*  \n[4] Tseng et al., NeurIPS 2024. *QTip: Quantization with trellises and incoherence processing*  \n[5] Yao et al., ECCV 2022. *Wave-ViT: Unifying wavelet and transformers for visual representation learning*  \n[6] Hamamci et al., arXiv 2024. *Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography*  \n[7] Blankemeier et al., arXiv 2024. *Merlin: A Vision Language Foundation Model for 3D Computed Tomography*  \n[8] Tschannen et al., NeurIPS 2024. *Image Captioners Are Scalable Vision Learners Too*  \n\n---\n\nWe again thank the reviewer for the detailed and insightful feedback. We believe these suggestions will substantially improve the clarity and positioning of our work."}, {"review_id": "hyqL3GDevx", "reviewer": "Reviewer_PJbc", "summary": "The authors propose BTB3D, a 3d convolutional encoder-decoder that converts CT scans into a sequence of discrete and frequency aware tokens. There are primarily 4 components to their methodology - a Haar wavelet processor that shrinks the volume while accumulating channels with frequency information, a causal 3d CNN, a lookup free binary quantization that improves speed and memory efficiency, and a three-stage training setup. The training objectives basically (1) enable local slice reconstruction through short-volume pretraining, (2) overlapping temporal tiling that enforces consistency across slices of the volume, and (3) decoder-only fine tuning on the full scans. The resulting tokens are evaluated on the task of report generation and text-to-CT synthesis. The results show that BTB3D beats the baselines such as CT-CHAT, Merlin and Generate-CT across various metrics, highlighting the strengths of the proposed method", "questions": "1. Is there a quick way to show/discuss generalization beyond chest CT. What changes are required for a different anatomy or modality - slice window, increasing channel dimension etc. Showing transfer to other body regions or modality would raise the significance of the method\n2. Does changing the tiling length affect downstream metrics? I wonder if variable slice windows based on which slice range the anatomy is usually located in could affect reconstruction \n3. A few examples of generated reports vs. those from the baselines such as merlin or CT-Chat would be useful to see, especially since the clinical accuracy metrics (P/R) of the baselines are better than those from BTB3D", "limitations": "Yes, the authors note the limitations in the checklist", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The writing is generally clear, concise, and the figures are informative \n2. Each component of the encoder-decoder model is clearly explained. The causal 3d CNN design enables a single model to operate on 2d slices or full 3d volumes. The novelty of this work is in the three stage training pipeline, which solves learning over longer CT sequences through overlapping tiling and decoder only fine tuning. As Table 1 shows, stage 2 and 3 enable dramatic improvement in reconstruction metrics. \n3. The same discrete tokens derived from the pretrained model are used by an (1) LLM for report generation, (2) a transformer decoder with cross attention, for CT synthesis. This demonstrates that the tokenizer is versatile and agnostic to the downstream head. \n4. The report generation performance on CT-RATE across NLG metrics is higher than that of the baselines. Similarly, the CT generation metrics of BTB3D is better than generateCT and MedSyn across the board. \n5. On the implementation side, dataset and training details are clearly provided along with resource requirements and tradeoffs between various components in the model in terms of speed and memory\n\nWeaknesses:\n1. The novelty lies in the three stage training pipeline, while the encoder decoder components themselves are an application of prior work (related work is mentioned in the paper under each subsection in methods) to the CT dataset. The title of the paper “BTB3D: Advancing VLM in 3D Medical Imaging” suggests the applicability of the method across other modalities in medical imaging, which is an overstatement. There is no experiment to show the usage of the method on MRI data, for example. The usage of a fixed 9-slice window, and the Haar wavelet transform seem to be well suited and chosen for chest CT data - there is no evidence this holds for other modalities and no experiments are provided on how to tune it to other modalities. The title of the paper has to match the empirical scope shown in it.\n2. The authors note the above limitation in the checklist and attribute it to the lack of scaled image-report paired datasets. However, none of the pretraining stages, which is the core contribution, require paired reports. Only the downstream tasks of report gen would be a blocker. Datasets such as BraTS and fastMRI are widely used in the medical imaging community.", "was_revised": true, "final_justification": "The authors provide detailed clarification to the questions I had. Particularly, the experiments on MRI data established cross -modality performance and shows promise without fine-tuning. I've read the other reviews and responses and doing another detailed pass over the paper including supplementary Figure 2., I am happy to increase my rating to 4.", "rebuttal": "## Reviewer PJbc\n\nWe sincerely thank the reviewer for the thoughtful and constructive review. We especially appreciate your detailed emphasis on the **clarity of our writing**, the **novelty and effectiveness of the three-stage training**, and the **strong performance across both report generation and text-to-CT synthesis**. Your recognition of the **modular versatility of our token representations** and the **transparency of our implementation** is deeply encouraging.\n\nBelow we provide point-by-point responses to your comments and suggestions:\n\n---\n\n### **1. Scope Overstatement in Title & Generalization Beyond Chest CT**\n> *“The title ... suggests the applicability ... across other modalities ... which is an overstatement ... there is no experiment to show ... on MRI data ... usage of a fixed 9-slice window and Haar transform seem CT-specific.”*  \n> *“None of the pretraining stages require paired reports ... datasets such as BraTS and fastMRI are widely used.”*  \n> *“Is there a quick way to show/discuss generalization beyond chest CT?”*\n\nWe thank the reviewer for this valuable observation. You are absolutely right, our **experiments are currently limited to chest CT**, as stated in Limitations section (Appendix A.1) and throughout the main paper, due to the lack of large-scale, report-paired datasets in other modalities. While we have noted this limitation throughout the paper, we acknowledge that the title may overstate the generality of our approach. The phrase 3D medical imaging can be interpreted more broadly than our current empirical scope.\n\nWhile paired reports are not required for the pretraining stages, our **main motivation** was to enable **vision-language modeling (VLM)** in 3D medical imaging (specifically, **report generation** and **text-to-volume synthesis**) both of which require report-paired data. Unfortunately, datasets such as **BraTS [1]**, **fastMRI [2]**, and **FOMO-MRI [3]** do not provide such supervision and are therefore unsuitable for these tasks.\n\nThat said, we agree with the reviewer that our **main contribution lies in the three-stage training strategy**, and that its effectiveness **can and should be tested across modalities**. To address this, we conducted additional experiments to evaluate the generalizability of our three-stage pretraining pipeline on:\n- **FOMO-MRI (T1-weighted brain MRIs, n=8857) [3]**\n- **TotalSegmentator full-body CT scans (n=1228) [4]**\n\nWe used our pretrained BTB3D encoder-decoder **without any fine-tuning**, and evaluated reconstruction quality across stages using PSNR, SSIM, and MSE. Results are shown below:\n\n### Brain MRI (FOMO-MRI, T1-only)\n\n| **Stage**  | **PSNR ↑** (16×16×8) | **SSIM ↑** (16×16×8) | **MSE ↓** (16×16×8) | **PSNR ↑** (8×8×8) | **SSIM ↑** (8×8×8) | **MSE ↓** (8×8×8) |\n|------------|----------------------|----------------------|---------------------|--------------------|--------------------|-------------------|\n| Stage 1    | 12.60                | 0.409                | 0.223               | 8.05               | 0.074              | 0.627             |\n| Stage 2    | 24.95                | 0.825                | 0.013               | 31.14              | 0.928              | 0.004            |\n| Stage 3    | **27.00**            | **0.889**            | **0.008**           | **31.27**          | **0.931**          | **0.003**        |\n\n---\n### Full-Body CT (TotalSegmentator)\n\n| **Stage**  | **PSNR ↑** (16×16×8) | **SSIM ↑** (16×16×8) | **MSE ↓** (16×16×8) | **PSNR ↑** (8×8×8) | **SSIM ↑** (8×8×8) | **MSE ↓** (8×8×8) |\n|------------|----------------------|----------------------|---------------------|--------------------|--------------------|-------------------|\n| Stage 1    |  11.43  | 0.197  | 0.297 | 8.86  | 0.039  | 0.529             |\n| Stage 2    | 15.48  | 0.371  | 0.126               | 18.19  | 0.460  | 0.075 |\n| Stage 3    |**17.17** | **0.429**  | **0.071**           | **19.51** | **0.498**  | **0.058** |\n\n\n---\n\nThese results confirm that our **progressive three-stage training strategy consistently improves performance**, especially through **Stage 2's overlapping tiling** (which is our main contribution). While these are not full VLM evaluations, they offer promising signals that BTB3D can generalize across modalities and body regions.\n\nWe also acknowledge that further **adaptations may be necessary** for full extension to VLM tasks in other modalities. For example:\n- Multi-series MRI (e.g., T1/T2/FLAIR) may require an updated input channel strategy.\n- The Haar wavelet transform may not be optimal in densely structured anatomical domains such as brain or abdominal MRI. Ablation studies will be required to explore this further.\n\nWe hope that these additional results offer meaningful insight and will help inform future applications of BTB3D to other modalities.\n\n---\n\n### **2. Impact of Tiling Length**\n> *“Does changing the tiling length affect downstream metrics?”*\n\nThank you for raising this important question, we have also considered this in our internal discussions. In our current design, we tile with **overlapping windows sharing only one slice**, which provides a reasonable balance between **temporal consistency** and **computational cost**.\n\nIncreasing the overlap (e.g., to 3 slices) could:\n- Enhance temporal continuity further;\n- Improve capture of localized structures (e.g., the diaphragm or cardiac apex).\n\nHowever, doing so would **significantly increase memory usage and training time**, which are already considerable with our 3-stage setup. Given that we already significantly outperform all baselines, we did not increase overlap in this version. Nevertheless, we believe your suggestion is highly valid and worthy of future investigation if needed, especially as computational resources become more accessible.\n\n---\n\n### **3. Examples of Generated Reports**\n> *“A few examples of generated reports vs. baselines ... would be useful to see.”*\n\nWe completely agree with the reviewer. Generated reports are an important qualitative indicator. As such, we already included **side-by-side comparisons** in **Supplementary Figure 2**, showing:\n- The **ground-truth report**,\n- Reports generated by **BTB3D, CT-CHAT, CT2Rep, and Merlin**, with consistent color-coding by abnormality.\n\nWe acknowledge that placing them in the appendix may have reduced their visibility, and we will consider bringing selected examples into the main paper in the revised version.\n\n---\n\n**References:**  \n[1] Menze et al., IEEE TMI 2015. *The Multimodal Brain Tumor Image Segmentation Benchmark (BraTS)*  \n[2] Zbontar et al., arXiv 2018. *fastMRI: An Open Dataset and Benchmarks for Accelerated MRI*  \n[3] Munk et al., arXiv 2025. *A Large‑Scale Heterogeneous 3D Magnetic Resonance Brain Imaging Dataset for Self‑Supervised Learning (FOMO‑MRI)*  \n[4] Wasserthal et al., Radiology: Artificial Intelligence 2023. *TotalSegmentator: Robust Segmentation of 104 Anatomical Structures in CT Images*  \n\n---\n\nOnce again, we thank the reviewer for their evaluation. Your comments prompted us to strengthen our **cross-modality validation** and clarify our implementation choices. We hope these revisions demonstrate BTB3D’s potential more clearly and make a strong case for its contributions to vision-language modeling in 3D medical imaging."}, {"review_id": "qO6PpUNnQD", "reviewer": "Reviewer_b6g3", "summary": "This paper introduces BTB3D (Better Tokens for Better 3D), a encoder-decoder framework for vision-language modeling in 3D medical imaging. The causal convolutional architecture processes CT scans slice-by-slice while maintaining temporal consistency, combined with wavelet-based compression and lookup-free quantization. The authors propose a three-stage training curriculum that progressively scales from short 9-slice subvolumes to full 300+ slice CT scans, addressing the computational challenges of training on high-resolution 3D medical data. The framework demonstrates good performance on two key tasks: radiology report generation (and text-conditioned CT synthesis.", "questions": "- Have you conducted any qualitative evaluation with radiologists? what features does the model fail to reconstruct properly etc?\n- Whats the performance of other similar tokenziers e.g cosmos's CV/DV tokenizers on this task?", "limitations": "- clear dataset bias, nonetheless, this is understandable given the requirements for the data and the nature of medical datasets.\n- no clear clinical evaluation, beyond downstream use and upstream reconstruction metrics.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper clearly articulates why contrastive learning fails for 3D medical VLMs (semantic alignment issues, memory constraints) and proposes a reconstruction-based alternative that addresses these limitations systematically.\n- They develop a good 2D/3D causal convolution based tokenizer, with good temporal consistency similar to Cosmos tokenizers\n- Novel three-stage curriculum effectively bridges the gap between computational constraints during training and the need to process full-length volumes at inference.\n- The paper evaluates on multiple downstream tasks with strong results.\n\nWeaknesses:\n- While the three-stage training is evaluated, other design choices lack thorough ablation.\n-  Minor: Despite addressing efficiency, training still requires 64 H100 GPUs. This limits accessibility.\n- No clinical validation with experts.\n- No comaprisim with non-medical tokenizers", "was_revised": true, "final_justification": "I'm inclined to improve the score of the paper to 5, giving the additional human eval, and robust rebuttal to the concerns raised.", "rebuttal": "## Reviewer b6g3\n\nWe sincerely thank the reviewer for the thoughtful and constructive review. We especially appreciate your emphasis on the key strengths of our paper:\n- The detailed critique of contrastive learning limitations in 3D medical vision-language models;\n- The development of a causal convolution-based tokenizer with strong temporal consistency;\n- The introduction of our novel three-stage training curriculum for full-volume CT processing;\n- The comprehensive evaluations across both report generation and text-to-CT synthesis tasks.\n\nWe are also grateful for your understanding of our dataset limitations, which stem from the current scarcity of publicly available large-scale paired datasets in medical imaging.\n\nBelow we provide a point-by-point response to your concerns and suggestions:\n\n---\n\n### **1. Other Design Choices Lack Thorough Ablation**\n\nWe acknowledge that additional ablations for architectural components would strengthen our findings. In this work, we prioritized a **comprehensive ablation of our main contribution: the three-stage training strategy**, as shown in **Table 1 and Figure 2** of the main paper.\n\nDue to the **high computational cost** (around 1 week with 64× H100 GPUs), we were unable to thoroughly ablate every individual design choice. Our goal was to achieve strong performance in demanding multimodal tasks such as **report generation** and **text-to-CT synthesis**, which require long-range spatial-temporal modeling and high semantic fidelity.\n\nNonetheless, we recognize the importance of additional ablations and appreciate the reviewer’s suggestion.\n\n---\n\n### **2. High Training Cost Limits Accessibility**\n\nYou are absolutely right that BTB3D is computationally intensive during training. Our current setup requires approximately **one week of training on 64× H100 GPUs**.\n\nHowever, we designed BTB3D to optimize for **clinical-scale downstream performance**, not training efficiency. Thanks to our **three-stage curriculum and overlapping tiling**, **inference is significantly more affordable**:\n- **Radiology report generation** for a full 3D CT volume takes **<18 seconds** on a single H100 GPU;\n- **Text-to-CT generation** takes **<12 seconds** on a single H100 GPU.\n\nThese results are highly valuable in the 3D medical imaging field. For context, one of the baselines, **GenerateCT**, reports an inference time of **184 seconds** per scan on an A100 80GB GPU, compared to BTB3D’s **<12 seconds** on a single H100.\n\nWhile clinical deployment still requires further steps, these results demonstrate that **BTB3D is a promising direction for scalable, low-latency 3D medical VLMs**.\n\n---\n\n### **3. No Clinical Validation with Experts**\n\nWe strongly agree that **metrics alone are insufficient** to assess clinical usability or realism in report generation and CT synthesis.\n\nAs part of this revision, we conducted **clinical expert evaluations** on both tasks:\n\n#### **Radiology Report Generation Evaluation:**\n\nWe selected a **subset of 150 CT scans** from the CT-RATE test set for expert evaluation. Since the **full test set contains over 3,000 scans**, evaluating all was infeasible.\n\nTo ensure fairness and diversity (rather than random sampling), we adopted a **stratified sampling strategy based on LLaMA Score**, using an LLM-based automated evaluation metric introduced in the CT-RATE benchmark [3]. This metric scores each generated report from 0–10 based on clinical plausibility, using a pretrained LLM as an evaluator. We calculated LLaMA scores across the entire CT-RATE test set using our BTB3D 16×16×8 variant.\n\nWe then sampled 50 reports from each LLaMA score tier: **Low** (0–3), **Medium** (4–6), **High** (7–10)\n\nThese 150 reports were assessed by a **board-certified radiologist**, who scored each on a scale of 0–10 based on **diagnostic relevance**, **semantic accuracy**, and **clinical coherence**.\n\n---\n\n|                | CT2Rep | Merlin | CT-CHAT | BTB3D |\n|----------------|--------|--------|---------|--------|\n| **Expert Score**| 2.74   | 3.82   | 4.52    | **5.46** |\n\n---\n\nThese results further show that **BTB3D sets a new state-of-the-art for radiology report generation in 3D CT**. However, our radiologist emphasized that **even the best models remain far from clinical deployment**.\n\nSpecifically, the expert noted that current models can **identify gross abnormalities** but **struggle with nuanced details**, such as:\n- Accurate **counts** of findings (e.g., number of nodules),\n- Precise **measurements** (e.g., sizes),\n- Consistent description of **pathological features** (e.g., shape, margins, density).\n\nThis confirms that while BTB3D leads among existing approaches, **substantial progress is still needed for reliable clinical use**.\n\n---\n\n#### **Text-to-CT Generation Evaluation:**\n\nUsing the same 150 prompts, we generated chest CT scans using **BTB3D, GenerateCT, and MedSyn**. The same radiologist was asked to evaluate:\n- The **visual realism** of each 3D volume,\n- The **alignment** between the generated scan and the conditioning text.\n\n|          | GenerateCT | MedSyn | BTB3D |\n|---------------|------------|--------|--------|\n| **Realism**       | 2.90       | 2.47   | **6.58** |\n| **Alignment**     | 3.74       | 3.66   | **4.12** |\n\nBTB3D achieved the highest scores in both realism and alignment, further validating its ability to model high-fidelity semantic structure from text. However, as noted by the expert, text-to-CT alignment remains an area for improvement, as the generated volumes were not always well-aligned with the conditioning prompts.\n\n*We used the 16×16×8 variant of BTB3D for report generation and 8×8×8 variant for text-to-CT synthesis.*\n\n---\n\n### **4. No Comparison with Non-Medical Tokenizers (e.g., Cosmos)**\n\nThank you for highlighting this important point.\n\nWe also explored using **Cosmos CV/DV tokenizers** (which are state-of-the-art in video modeling). However, we encountered several barriers:\n\n- **Domain mismatch**: Cosmos models are pretrained on **RGB video**, whereas CT scans are **grayscale medical volumes**.\n- **Sequence length limitations**: Cosmos is designed for sequences up to **121 frames**, while CT-RATE scans can exceed **600 slices**.\n- **Training code unavailability**: Public repositories only support **inference**, making it infeasible to fine-tune Cosmos on CT datasets.\n\nIn our initial evaluations, Cosmos models performed **worse** on:\n- **Reconstruction quality**,\n- **Report generation accuracy**,\n- **Text-conditioned CT generation fidelity**.\n\nWe attribute this gap primarily to **training domain mismatch**, not architecture flaws. As **fine-tuning was not feasible**, we refrained from including Cosmos in the main paper to avoid unfair comparisons.\n\nThat said, we agree with the reviewer and documenting these findings in the supplementary will provide valuable context. We will add a section comparing Cosmos and BTB3D in the revised appendix.\n\n---\n\n### **5. Formatting and Presentation Issues**\n\nWe appreciate your helpful suggestions to improve readability and clarity:\n\n- **Figure 2**: We will revise the figure by adding **zoomed-in insets** and adjusting **contrast levels** to better highlight reconstruction differences between stages. We will also clarify cases where Stage 2 outperforms Stage 3 on specific slices.\n- **Discriminator architecture**: We appreciate this comment, the adversarial discriminator was very briefly mentioned in Section 4.1 but lacked full details. We will update the **Methods section** to include complete architectural specifications (layers, kernel sizes, etc.) for reproducibility.\n\n---\n\nOnce again, we thank the reviewer for your fair and insightful feedback. Your comments helped us strengthen both the **experimental evaluation** and **clinical relevance** of BTB3D, and we hope the additional clarifications address your concerns."}, {"review_id": "oi68NAFJbP", "reviewer": "Reviewer_kRtj", "summary": "This paper proposes a multimodal long-context pre-training framework for medical image-text tasks, MedCoT, aimed at addressing the issues of input redundancy and information loss caused by “text redundancy” and “high-resolution images” in medical image-text scenarios. By introducing a text compression mechanism based on medical ontologies and a prompting method using abstract visual tokens, the authors effectively enhance image-text alignment capabilities and context utilization efficiency. The method achieves outstanding performance across multiple medical multimodal tasks (question-answering, localization, report generation, etc.), particularly demonstrating significant advantages under long context input conditions, thereby showcasing its practicality and the rationality of its model design.", "questions": "1. What are the algorithmic details and robustness of token-aware text compression? This module combines entity structure and importance scoring to compress long texts. It is recommended that the authors further explain the training method of the scoring mechanism (is it weakly supervised? Is it prone to overfitting?) Additionally, its transferability across different data domains (e.g., radiology vs. pathology) should be addressed.\n\n2. Adaptability of the compression strategy to inputs of different lengths? Have tests been conducted to examine how the model's compression and performance vary across different input text lengths (e.g., report excerpts vs. full text) or visual resolutions (e.g., 512×512 vs. 1024×1024)?\n\n3. Is the learning method for abstract tokens in image compression sufficiently stable? Are the abstract tokens in Compressed Visual Prompting fixed learnable parameters, or are they dynamically generated based on visual content? Is there a risk of collapse or semantic confusion? It is recommended to add learning curves or analyze their stability.\n\n4. Are there conflicts in the multi-task setting? The authors uniformly optimize multiple tasks (question-answering, generation, localization, etc.) in MedCoT. Are there task interference or weight conflict issues? Are there explicit task adaptation mechanisms (e.g., adapters, conditional routing) used? If not, why can stable performance be maintained across tasks?\n\n5. Does compression affect clinical semantic accuracy? During the compression of report text or image embeddings, is there information loss for certain fine-grained diagnostic descriptions (e.g., “small nodule in the right middle lung”)? Has the semantic retention rate before and after compression been evaluated using metrics like BLEU or BERTScore?\n\n6. The scope of generalization validation remains limited, with the current model primarily validated on datasets like MIMIC-CXR and PathVQA. Have you attempted to transfer MedCoT to more complex non-X-ray data (e.g., MRI or CT)? Does it have the capability to adapt to multilingual medical reports?\n\n7. Are the training resources and costs manageable? While achieving efficiency improvements through compression, multiple modules (such as scorers and visual abstraction tokens) have been introduced. Is the overall model training and inference resource usage lighter than that of LLaVA or BioViL? Please quantify the parameter scale and inference time.\n\n8. How does it differ from existing general-purpose long-context models? Has MedCoT been compared with current mainstream long-context VL frameworks (such as CoLT5, LongViT, Fuyu-8B, etc.), particularly in terms of robustness and output quality when the context length exceeds 2048?", "limitations": "The authors briefly mention the applicability and potential scalability of the MedCoT model in the appendix, but the following points should be further refined:\n1. The impact of compression strategies on small-sample tasks. Compression may cause performance fluctuations in scenarios with extremely limited samples;\n2. Potential ethical and bias propagation risks. If critical medical descriptions are inadvertently lost during compression, it may affect downstream diagnostic judgments, so explainability and traceability should be strengthened;\n3. Cross-modal generalization capability. The current validation is primarily for chest X-ray scenarios, and the effectiveness in MRI, CT, or pathology images has not yet been demonstrated. It is recommended to include this in future plans.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Advantages:\n\n1. Tailored to the characteristics of medical text-image data (“long text + high-resolution images”), the framework proposes a specially designed token compression scheme with strong practical applicability.\n2. Text compression combines knowledge-driven (medical ontology) and data-driven (importance scoring) approaches, while image compression employs pre-trained abstract tokens for guidance, balancing theoretical and engineering design considerations.\n3. The method covers four major medical VL tasks and has been validated on multiple benchmarks (MIMIC-CXR, PathVQA, MedPix, etc.), demonstrating robust multi-task generalization capabilities and advantages in handling long contexts.  \n\nDisadvantages:\n\n1. The compression module details are somewhat simplified, particularly regarding the specific algorithm implementation and interpretability of how text is selected based on ontology and importance scoring.\n2. The ablation experiment dimensions are somewhat limited. While the effects of different components are demonstrated, more granular analysis is not conducted on token quantity changes, compression rate adjustments, or text-image adversarial robustness.\n3. The discussion on the generalization of visual abstract tokens is insufficient. The paper does not delve into whether abstract tokens can be applied to new modalities (such as MRI/PET) or images of different resolutions.", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer kRtj\n\nWe sincerely thank the reviewer for their time. However, **this review appears to have been mistakenly assigned to our submission and does not discuss our work**.\n\nOur paper, **\"Better Tokens for Better 3D (BTB3D): Advancing Vision-Language Modeling in 3D Medical Imaging\"**, focuses specifically on:\n- Improving 3D vision-language modeling for **chest CT** scans,\n- Introducing a **causal convolutional tokenizer** and a **three-stage curriculum** for scalable long-sequence processing,\n- Enabling key downstream tasks such as **radiology report generation** and **text-to-CT synthesis** in 3D,\n- Addressing the unique challenges of **3D medical imaging**.\n\nIn contrast, the review discusses a different paper, named **MedCoT**, focused on:\n- A **text and image compression framework** for multimodal medical tasks,\n- Experiments conducted primarily on **2D X-ray datasets** such as MIMIC-CXR and PathVQA,\n- Use of **medical ontology-based text compression** and **abstract visual prompting**,\n- Tasks including **question-answering, localization, and multi-task learning**.\n\nWe are confident that this is a mismatched review, likely intended for a different submission in the reviewer’s stack. We understand that such mismatches can occur in large-scale reviewing pipelines. We kindly ask the program chairs and the reviewer to reassess and either reassign or disregard this review, as appropriate."}]

Author rebuttal:

Meta-review: The paper proposes a causal convolutional encoder-decoder method that combines 2D and 3D training with inference for 3D vision language modelling. Furthermore, a three stage curriculum learning strategy is introduced to overcome the limitations of contrastive learning misalignment with clinical semantics.

Some notable strengths of the submission include:

- the model obtains strong performance on report generation and image synthesis tasks
- the paper is well-written and in the methodology, the core components are explained in detail and clarity
- the motivation for contrastive learning struggles for 3D medical VLMs
- a novel three stage curriculum learning strategy
- propose a adequate 2D/3D causal convolution based tokenizer

Among the concerns highlighted by reviewers, the important ones are:

- the related work is missing citations of some relevant works
- the validation of 3-stage curriculum learning strategy needs more analyses
- novelty mainly lies in 3-stage learning strategy while the rest of the design resembles with prior work
- some design choices are not well ablated 
- there is mention of clinical validation with experts

After the author-reviewer discussion period, majority of the reviewers expressed their satisfaction with the responses of authors on the concerns. For example, the concerns of better positioning in the literature for 2D and 3D medical imaging, empirical validation of 3D tokens, the experiments on MRI data, improved validation of 3-state learning strategy, and clinical experts validation has been resolved. To this end, the AC decides to recommend acceptance of the paper. Authors are recommended to include important reviewers comments in the final version.
Decision: Accept (poster)