Title: Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Medical Imaging

Vision-Language Modeling

Radiology Report Generation

Text-Conditional 3D CT Synthesis

Volumetric Tokenization
Abstract: Recent progress in vision-language modeling for 3D medical imaging has been fueled by large-scale computed tomography (CT) corpora with paired free-text reports, stronger architectures, and powerful pretrained models. This has enabled applications such as automated report generation and text-conditioned 3D image synthesis. Yet, current approaches struggle with high-resolution, long-sequence volumes: contrastive pretraining often yields vision encoders that are misaligned with clinical language, and slice-wise tokenization blurs fine anatomy, reducing diagnostic performance on downstream tasks. We introduce BTB3D (Better Tokens for Better 3D), a causal convolutional encoder-decoder that unifies 2D and 3D training and inference while producing compact, frequency-aware volumetric tokens. A three-stage training curriculum enables (i) local reconstruction, (ii) overlapping-window tiling, and (iii) long-context decoder refinement, during which the model learns from short slice excerpts yet generalizes to scans exceeding $300$ slices without additional memory overhead. BTB3D sets a new state-of-the-art on two key tasks: it improves BLEU scores and increases clinical F1 by 40\% over CT2Rep, CT-CHAT, and Merlin for report generation; and it reduces FID by 75\% and halves FVD compared to GenerateCT and MedSyn for text-to-CT synthesis, producing anatomically consistent $512\times512\times241$ volumes. These results confirm that precise three-dimensional tokenization, rather than larger language backbones alone, is essential for scalable vision-language modeling in 3D medical imaging. The codebase is available at: https://github.com/ibrahimethemhamamci/BTB3D


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multimodal long-context pre-training framework for medical image-text tasks, MedCoT, aimed at addressing the issues of input redundancy and information loss caused by “text redundancy” and “high-resolution images” in medical image-text scenarios. By introducing a text compression mechanism based on medical ontologies and a prompting method using abstract visual tokens, the authors effectively enhance image-text alignment capabilities and context utilization efficiency. The method achieves outstanding performance across multiple medical multimodal tasks (question-answering, localization, report generation, etc.), particularly demonstrating significant advantages under long context input conditions, thereby showcasing its practicality and the rationality of its model design.\n\nSTRENGTHS AND WEAKNESSES:\nAdvantages:\n\n1. Tailored to the characteristics of medical text-image data (“long text + high-resolution images”), the framework proposes a specially designed token compression scheme with strong practical applicability.\n2. Text compression combines knowledge-driven (medical ontology) and data-driven (importance scoring) approaches, while image compression employs pre-trained abstract tokens for guidance, balancing theoretical and engineering design considerations.\n3. The method covers four major medical VL tasks and has been validated on multiple benchmarks (MIMIC-CXR, PathVQA, MedPix, etc.), demonstrating robust multi-task generalization capabilities and advantages in handling long contexts.  \n\nDisadvantages:\n\n1. The compression module details are somewhat simplified, particularly regarding the specific algorithm implementation and interpretability of how text is selected based on ontology and importance scoring.\n2. The ablation experiment dimensions are somewhat limited. While the effects of different components are demonstrated, more granular analysis is not conducted on token quantity changes, compression rate adjustments, or text-image adversarial robustness.\n3. The discussion on the generalization of visual abstract tokens is insufficient. The paper does not delve into whether abstract tokens can be applied to new modalities (such as MRI/PET) or images of different resolutions.\n\nQUESTIONS:\n1. What are the algorithmic details and robustness of token-aware text compression? This module combines entity structure and importance scoring to compress long texts. It is recommended that the authors further explain the training method of the scoring mechanism (is it weakly supervised? Is it prone to overfitting?) Additionally, its transferability across different data domains (e.g., radiology vs. pathology) should be addressed.\n\n2. Adaptability of the compression strategy to inputs of different lengths? Have tests been conducted to examine how the model's compression and performance vary across different input text lengths (e.g., report excerpts vs. full text) or visual resolutions (e.g., 512×512 vs. 1024×1024)?\n\n3. Is the learning method for abstract tokens in image compression sufficiently stable? Are the abstract tokens in Compressed Visual Prompting fixed learnable parameters, or are they dynamically generated based on visual content? Is there a risk of collapse or semantic confusion? It is recommended to add learning curves or analyze their stability.\n\n4. Are there conflicts in the multi-task setting? The authors uniformly optimize multiple tasks (question-answering, generation, localization, etc.) in MedCoT. Are there task interference or weight conflict issues? Are there explicit task adaptation mechanisms (e.g., adapters, conditional routing) used? If not, why can stable performance be maintained across tasks?\n\n5. Does compression affect clinical semantic accuracy? During the compression of report text or image embeddings, is there information loss for certain fine-grained diagnostic descriptions (e.g., “small nodule in the right middle lung”)? Has the semantic retention rate before and after compression been evaluated using metrics like BLEU or BERTScore?\n\n6. The scope of generalization validation remains limited, with the current model primarily validated on datasets like MIMIC-CXR and PathVQA. Have you attempted to transfer MedCoT to more complex non-X-ray data (e.g., MRI or CT)? Does it have the capability to adapt to multilingual medical reports?\n\n7. Are the training resources and costs manageable? While achieving efficiency improvements through compression, multiple modules (such as scorers and visual abstraction tokens) have been introduced. Is the overall model training and inference resource usage lighter than that of LLaVA or BioViL? Please quantify the parameter scale and inference time.\n\n8. How does it differ from existing general-purpose long-context models? Has MedCoT been compared with current mainstream long-context VL frameworks (such as CoLT5, LongViT, Fuyu-8B, etc.), particularly in terms of robustness and output quality when the context length exceeds 2048?\n\nLIMITATIONS:\nThe authors briefly mention the applicability and potential scalability of the MedCoT model in the appendix, but the following points should be further refined:\n1. The impact of compression strategies on small-sample tasks. Compression may cause performance fluctuations in scenarios with extremely limited samples;\n2. Potential ethical and bias propagation risks. If critical medical descriptions are inadvertently lost during compression, it may affect downstream diagnostic judgments, so explainability and traceability should be strengthened;\n3. Cross-modal generalization capability. The current validation is primarily for chest X-ray scenarios, and the effectiveness in MRI, CT, or pathology images has not yet been demonstrated. It is recommended to include this in future plans.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces BTB3D (Better Tokens for Better 3D), a encoder-decoder framework for vision-language modeling in 3D medical imaging. The causal convolutional architecture processes CT scans slice-by-slice while maintaining temporal consistency, combined with wavelet-based compression and lookup-free quantization. The authors propose a three-stage training curriculum that progressively scales from short 9-slice subvolumes to full 300+ slice CT scans, addressing the computational challenges of training on high-resolution 3D medical data. The framework demonstrates good performance on two key tasks: radiology report generation (and text-conditioned CT synthesis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper clearly articulates why contrastive learning fails for 3D medical VLMs (semantic alignment issues, memory constraints) and proposes a reconstruction-based alternative that addresses these limitations systematically.\n- They develop a good 2D/3D causal convolution based tokenizer, with good temporal consistency similar to Cosmos tokenizers\n- Novel three-stage curriculum effectively bridges the gap between computational constraints during training and the need to process full-length volumes at inference.\n- The paper evaluates on multiple downstream tasks with strong results.\n\nWeaknesses:\n- While the three-stage training is evaluated, other design choices lack thorough ablation.\n-  Minor: Despite addressing efficiency, training still requires 64 H100 GPUs. This limits accessibility.\n- No clinical validation with experts.\n- No comaprisim with non-medical tokenizers\n\nQUESTIONS:\n- Have you conducted any qualitative evaluation with radiologists? what features does the model fail to reconstruct properly etc?\n- Whats the performance of other similar tokenziers e.g cosmos's CV/DV tokenizers on this task?\n\nLIMITATIONS:\n- clear dataset bias, nonetheless, this is understandable given the requirements for the data and the nature of medical datasets.\n- no clear clinical evaluation, beyond downstream use and upstream reconstruction metrics.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose BTB3D, a 3d convolutional encoder-decoder that converts CT scans into a sequence of discrete and frequency aware tokens. There are primarily 4 components to their methodology - a Haar wavelet processor that shrinks the volume while accumulating channels with frequency information, a causal 3d CNN, a lookup free binary quantization that improves speed and memory efficiency, and a three-stage training setup. The training objectives basically (1) enable local slice reconstruction through short-volume pretraining, (2) overlapping temporal tiling that enforces consistency across slices of the volume, and (3) decoder-only fine tuning on the full scans. The resulting tokens are evaluated on the task of report generation and text-to-CT synthesis. The results show that BTB3D beats the baselines such as CT-CHAT, Merlin and Generate-CT across various metrics, highlighting the strengths of the proposed method\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The writing is generally clear, concise, and the figures are informative \n2. Each component of the encoder-decoder model is clearly explained. The causal 3d CNN design enables a single model to operate on 2d slices or full 3d volumes. The novelty of this work is in the three stage training pipeline, which solves learning over longer CT sequences through overlapping tiling and decoder only fine tuning. As Table 1 shows, stage 2 and 3 enable dramatic improvement in reconstruction metrics. \n3. The same discrete tokens derived from the pretrained model are used by an (1) LLM for report generation, (2) a transformer decoder with cross attention, for CT synthesis. This demonstrates that the tokenizer is versatile and agnostic to the downstream head. \n4. The report generation performance on CT-RATE across NLG metrics is higher than that of the baselines. Similarly, the CT generation metrics of BTB3D is better than generateCT and MedSyn across the board. \n5. On the implementation side, dataset and training details are clearly provided along with resource requirements and tradeoffs between various components in the model in terms of speed and memory\n\nWeaknesses:\n1. The novelty lies in the three stage training pipeline, while the encoder decoder components themselves are an application of prior work (related work is mentioned in the paper under each subsection in methods) to the CT dataset. The title of the paper “BTB3D: Advancing VLM in 3D Medical Imaging” suggests the applicability of the method across other modalities in medical imaging, which is an overstatement. There is no experiment to show the usage of the method on MRI data, for example. The usage of a fixed 9-slice window, and the Haar wavelet transform seem to be well suited and chosen for chest CT data - there is no evidence this holds for other modalities and no experiments are provided on how to tune it to other modalities. The title of the paper has to match the empirical scope shown in it.\n2. The authors note the above limitation in the checklist and attribute it to the lack of scaled image-report paired datasets. However, none of the pretraining stages, which is the core contribution, require paired reports. Only the downstream tasks of report gen would be a blocker. Datasets such as BraTS and fastMRI are widely used in the medical imaging community.\n\nQUESTIONS:\n1. Is there a quick way to show/discuss generalization beyond chest CT. What changes are required for a different anatomy or modality - slice window, increasing channel dimension etc. Showing transfer to other body regions or modality would raise the significance of the method\n2. Does changing the tiling length affect downstream metrics? I wonder if variable slice windows based on which slice range the anatomy is usually located in could affect reconstruction \n3. A few examples of generated reports vs. those from the baselines such as merlin or CT-Chat would be useful to see, especially since the clinical accuracy metrics (P/R) of the baselines are better than those from BTB3D\n\nLIMITATIONS:\nYes, the authors note the limitations in the checklist", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces BTB3D, a causal convolutional encoder-decoder framework for vision-language modeling in 3D medical imaging. To generate compact and semantically meaningful volumetric tokens, the model combines wavelet-based compression, causal 3D convolutions, and lookup-free quantization. To address limitations of contrastive pretraining misaligned with clinical semantics, BTB3D adopts a three-stage training strategy: (1) short-volume pretraining, (2) overlapping temporal tiling, and (3) long-context decoder refinement. Experimental results demonstrate that BTB3D achieves state-of-the-art performance on both radiology report generation and text-conditioned CT synthesis tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n-\tThe model achieves strong performance in both report generation and image synthesis tasks, outperforming state-of-the-art baselines in F1, BLEU, and FID/FVD metrics.\n-\tThe causal convolutional design and wavelet-based compression allow training and inference on long CT sequences under limited memory.\n-\tThe authors provide detailed implementation and training setup and claim to release code and pretrained weights, which supports future use and benchmarking.\n\n**Weaknesses**:\n\n-\t**Incomplete related work discussion**: While the paper emphasizes the novelty of unifying 2D and 3D training and inference while producing compact, frequency-aware volumetric tokens, it does not adequately discuss prior work in 2D–3D unification in the medical imaging domain. Relevant approaches, such as [1] [2], are not reviewed. In addition, the related work section lacks coverage of tokenization techniques like [3-5]. This weakens the contextual positioning of the contribution.\n\n-\t**Insufficient validation of the proposed training strategy**: The effectiveness of the proposed three-stage training is primarily validated via reconstruction metrics (e.g., PSNR, SSIM). However, the downstream tasks of interest (i.e., radiology report generation and text-to-CT synthesis) depend on semantic alignment rather than pixel-wise fidelity. It is unclear whether better reconstruction directly leads to better language generation or synthesis. The lack of direct comparison on downstream task performance across different training stages weakens the claim that the three-stage training benefits the end tasks.\n\n-\t**Lack of evidence for the “better token” claim**: While the tokenization is central to the paper’s claims, there is no direct comparison to other tokenization strategies (e.g., VQ-VAE, CLIP embeddings) to validate its advantage.\n\n[1] Xie, Yutong, et al. \"Unimiss: Universal medical self-supervised learning via breaking dimensionality barrier.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] He, Xiaoxuan, et al. \"Unified Medical Image Pre-training in Language-Guided Common Semantic Space.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[3] Zhang, Jiahui, et al. \"Regularized vector quantization for tokenized image synthesis.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[4] Tseng, Albert, et al. \"Qtip: Quantization with trellises and incoherence processing.\" Advances in Neural Information Processing Systems 37 (2024): 59597-59620.\n\n[5] Yao, Ting, et al. \"Wave-vit: Unifying wavelet and transformers for visual representation learning.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.\n\nQUESTIONS:\n1. Could the authors discuss existing efforts in unifying 2D and 3D modeling in medical imaging, and how BTB3D differs from or builds upon them?\n\n2. Could the authors compare BTB3D’s tokens against alternative tokenization methods, such as VQ-VAE, CLIP-based features, or anatomy-aware tokens, to better support the \"better token\" claim?\n\n3. Could the authors provide ablation studies comparing report generation and text-to-CT synthesis performance between single-stage and three-stage training? This would more directly validate the proposed training strategy for the intended applications.\n\nLIMITATIONS:\nThe paper includes a limitations discussion, noting the chest CT-only scope, computational requirements, and lack of clinical user studies.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-12"}