OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis
Authors: Tianwei Lin
Zhongwei Qiu
Wenqiao Zhang
Jiang Liu
Yihan Xie
Mingjian Gao
Zhenxuan Fan
Zhaocheng Li
Sijing Li
Zhongle Xie
Peng LU
Yueting Zhuang
Ling Zhang
Beng Chin Ooi
Yingda Xia
Venue: ICLR 2026 Poster (2026)
Keywords: Medical foundation model
Vision-language modeling
CT
VLM
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Computed Tomography (CT) is one of the most widely used and diagnostically information-dense imaging modalities, covering critical organs such as the heart, lungs, liver, and colon. Clinical interpretation relies on both slice-driven local features (e.g., sub-centimeter nodules, lesion boundaries) and volume-driven spatial representations (e.g., tumor infiltration, inter-organ anatomical relations). 
However, existing Large Vision–Language Models (LVLMs) remain fragmented in CT slice versus volumetric understanding: slice-driven LVLMs show strong generalization but lack cross-slice spatial consistency, while volume-driven LVLMs explicitly capture volumetric semantics but suffer from coarse granularity and poor compatibility with slice inputs. The absence of a unified modeling paradigm constitutes a major bottleneck for the clinical translation of medical LVLMs. 
We present OmniCT, a powerful unified slice–volume LVLM for CT scenarios, which makes three contributions: 
(i) Spatial Consistency Enhancement (SCE): volumetric slice composition combined with tri-axial positional embedding that introduces volumetric consistency, and an MoE hybrid projection enables efficient slice–volume adaptation; 
(ii) Organ-level Semantic Enhancement (OSE): segmentation and ROI localization explicitly align anatomical regions, emphasizing lesion- and organ-level semantics; 
(iii) MedEval-CT: the largest slice–volume CT dataset and hybrid benchmark integrates comprehensive metrics for unified evaluation. 
OmniCT consistently outperforms existing methods with a substantial margin across diverse clinical tasks and satisfies both micro-level detail sensitivity and macro-level spatial reasoning. More importantly, it establishes a new paradigm for cross-modal medical imaging understanding. Our project is available at https://github.com/ZJU4HealthCare/OmniCT.

Published as a conference paper at ICLR 2026
OMNICT: TOWARDS A UNIFIED VOLUME LVLM FOR COMPREHENSIVE CT ANALYSIS
Tianwei Lin Lin Zhongwei 2., Wenqiao Zhang' Jiang Liu', Yihan Xie', Mingjian Gao', Zhenxuan Fan', Zhaocheng Li', Sijing Li Zhongle Xie', Peng Lu Yueting Zhuang'. Ling Zhang'. Beng Chin Ooi Yingda Xia? 'Zhejiang University, 2DAMO Academy, Alibaba Group, Hupaan Lab [lintw, wenqiaazhangg edu. cn (qiuzhongwei. qzw) @ wwabbabbiinng Anc.com
ABSTRACT
Computed Tomography (CT) is one of the most widely used and diagnostically information-dense imaging modalities, covering critical organs such as the heart, lungs, liver, and colon. Clinical interpretation relies on both slice-driven local fea- tures (e.g., sub-centimeter nodules, lesion boundaries) and volume-driven spatial representations (e.g., tumor infiltration, inter-organ anatomical relations). How- ever, existing Large Vision-Language Models (LVLMs) remain fragmented in CT slice versus volumetric understanding: slice-driven LVLMs show strong general- ization but lack cross-slice spatial consistency, while volume-driven LVLMs ex- plicitly capture volumetric semantics but suffer from coarse granularity and poor compatibility with slice inputs. The absence of unified modeling paradigm constitutes a major bottleneck for the clinical translation of medical LVLMs. We present OmniCT, powerful unified slice-volume LVLM for CT scenarios, which makes three contributions: (i) Spatial Consistency Enhancement (SCE): volumetric slice composition combined with tri-axial positional embedding that introduces volumetric consistency. and an MoE hybrid projection enables efficient volume adaptation; (fi) Organ-level Semantic Enhancement (OSE): seg- mentation and ROI localization explicitly align anatomical regions, emphasizing lesion- and organ-level semantics; (iii) MedEval-CT: the largest slice-volume CT dataset and hybrid benchmark integrates comprehensive metrics for unified evaluation. OmniCT consistently outperforms existing methods with a substantial margin across diverse clinical tasks and satisfies both micro-level detail sensi- tivity and macro-level spatial reasoning. More importantly, it establishes a new paradigm for cross-modal medical imaging understanding. Our project avail- able at /ttps::/gttthhiiee com ZJJUHea TeateellhhCaar thCare/ i
INTRODUCTION
Large LanguaeL Models (LVLMs) have become a cornerstone of multi-modal artificial intel- ligence, demonstrating strong cross-modal representation and reasoming capabilities in both image understanding (Qiu al. 2023 Li al. 2024a Zhu et al. 2025 Bai al. (2055) and video per- ception (Lin et al.) 2023a Li et al. 2024a Zhang et al. 2025a Yuan et al. 2025). Benefiting from large-scale pre-training and modality alignment, LVLMs achieve remarkable performance in open- domain tasks (Qiu al. 2022, Yue et al.) 2024 Fu et al. (025), excelling in both generation and reasoning. These advances establish LVLMs as a universal paradigm for unified vision-language modeling, where the joint modeling of 2D and 3D modalities has emerged
key design principle.
In recent years, the potential of LVLMs in medical imaging has received increasing attention, with exploration in radiological imaging being particularly notable (Wu et al.) 2025 Xu et al.) (025). However, most existing methods are tailored to process either CT slices (Chen et al. [022aa]
et al. 2025) or volumetric data (Bai et al. 2024 Hamamci al.) 2024c), with limited focus
"Equal contributions. The work was done during Tianwei's internship at DAMO Academy. Cooresponning author.

---

Published as a conference paper at ICLR 2026
(a) MedEval- CT-Dataeet
(b) OmniCT Archilecture
SLAKE (c) Performance
derive Bate
Cauld wrile the radiology rxport
Slice-driven
Benchmark
Capaie
natt
Lingshu
1%
42%
Report Hideersr
11.5%
Laderee-rm thas
Organ-lavel Semantie Enhancament
MeDD-attedd
Spatial Consistency Enhancement
Question Types
Slice-Volume Distribution
CT-CHAT
GAR
52.4%
MID
Owerd 5-VL
21.9%
CRO
CAN redio
Large Language Model
OPTS
19.9%
6.1%
RadFM
34.5%
gland has increased nnd has
thyyold
CEMATE
OMniCT
All
heteregencons appearnnce.
Volume-driven Benchmark
Clinical Typea
Organ Typea
The thyroid Jobe...
Figure (a) is the statistics of the proposed MedEval-CT-Daatset (b) describes the simplified architecture of proposed OMniCT. (c) shows that OmniCT consistently surpasses all baselines on both slice-driven and volume-driven CT benchmarks.
cooperative processing. Slice-driven models leverage large-scale 2D pre-training to achieve strong vision-languag alignment and perform well in tasks such as lesion detection and radiology report description, yet they fail to capture cross-slice spatial consistency. In contrast, volume-driven models explicitly model voxel-level spatial structures, offering advantages in holistic spatial representation and organ-level reasoning. Nevertheless, these models often lack sensitivity to fine- une-grained abnor- malities and boundary morphology, and their architectures are difficult to adapt to slice-level tasks, thereby limiting their upplicability across diverse medical scenarios. This persistent dichotomy be- tween slice and volume modeling constitutes major bottleneck in the development of medical LVL.M..
Among various medical imaging modalities, CT is one of the most widely used and dense in in- formation, with hundreds of million performed each year globally. CT can cover critical organs such as the heart, lungs. liver, and colon, and is widely applied in essential tasks including dis- ease screening (Hu et al. 2025). lesion assessment (Li et al. 2025b; Shui et al.) 2025), and tumor staging (Bassi et al.) [025 Qiu et al. 2025). Its diagnostic process relies on both slice-level local imaging cues, such as sub-centimeter pulmonary nodules or hepatic lesion boundaries, and volume- level spatial-topological representations, such as tumor infiltration ranges or inter-organ anatomical relationships. Modeling along single dimension alone cannot meet these dual requirements. There- fore, integrating the complementary strengths of 2D and 3D modeling within a unified framework is not only central scientific challenge for CT understanding but also an inevitable step toward the clinical translation of medical LVLMs.
We propose OmniCT (see Fig. D. powerful unified volume LVLM for CT-centric under- standing, which preserves the cross modal alignment and generalization strengths of 2D models while integrating the spatial structural awareness of 3D models. To bridge the modality gap between slice and volume representations, we introduce a Spatial Consistency Enhancement (SCE) strat- egy. Unlike generic LVLMs that rely on frame sampling or key-frame stacking strategies (Xu al. 2024 Li et al. 22224b Huang et al. (024), SCE performs Volumetric Slice Composition (VSC) by structurally combining adjacent slices along the channel dimension into locally consistent volumet- ric units, thereby retaining contextual spatial transitions. It further incorporates Tri-axial Posi- tional Encoding (TPE), which injects 3D positional encodings into visual representations to enable volumetric awareness while maintaining compatibility with bascd inputs. In addition, MoE Hybrid Projection (MHP) dynamically aligns slice and volume features within a shared representa- tion space, ensuring semantic unification with the Large Language Models (LLMs). Overall, SCE injects robust 2D/3D spatial priors while achieving a balance between efficiency and adaptability.
In clinical diagnosis, image interpretation is performed at the organ level, where observations and lesion localization are conducted within this scope (Shui et al. 2025). Building on this clinical requirement, propose Organ-level Semantic Enhancement (OSE). OSE performs task- aus-gdidedd anatomical region localization, explicitly projecting critical organ regions into the token representa- tion space and fusing them with global visual context, thereby embedding organ-centric semantics
2

---

Published as a conference paper at ICLR 2026
ou touts
Orean-level Semantic Enhancement (OSE)
AgB
Volume Token
Large Language
Slice Token
Segmenter
Align Expert
MoE Hybird Projection
Querying <maskas/cbbor> for ieetiliable organs
Aligned Token
Volume Expert
Slice Expert
Encodar
Vision
Organ
Vialon Encoder
Vislon Encoder
Text Token
84
Sn
Sequence
v
Whether the <lungs> show abnormailies?
Volumetric Slice Composittin (VSC)
Mask/
Figure 2: The architecture of OmniCT, unified slice-volume LVLM paradigm.
into the representation. It then applies an adaptive aggregation to compress long-sequence represen- tations: this mechanism preserves overall information coverage while adaptively magnifying smaller organ regions and compressing larger ones, thus highlighting the most diagnostically relevant struc- tures. In this way, OSE explicitly incorporates region priors with high task-relevant semantic load while improving the relevance and interpretability of models in clinical tasks.
Existing medical benchmarks (Hu et al. 2024, Yue et al. 2024, Yamagishi et al. 2025) often adopt multi-modality designs to evaluate the general capability of LVLMs, yet they fall short in task alignment and clinical representativeness for CT interpretation. To address this gap, we in- troduce MedEval-CT, the first holistic evaluation framework dedicated CT images. At the data level, MedEEvll-CC Soald-CT-Daaasse consolidates 1.7M slice-driven and volume-driven VQA samples across 7 clinical task types, establishing the largest CT resource date (Fig. (a)). At the benchmark level, organizes hybrid evaluations along clinical problem types and organ distribu- tions. At the toolkit level, MedEval-CT CCEEEal----aacooyy standardizes input handling, feature construction, and multi-dimennione metrics, supporting statistical, semantic, and LLM-based evaluations. Col- lectively, MedEval-CT institutionalizes fairness and comparability medical LVLM evaluation, while providing scalable foundation for larger and more complex clinical scenarios.
MedEval-CT-BBl
Experimental results on multiple CT-centric benchmarks show that OmniCT achieves substantial improvements over existing methods, as illustrated by the radar chart in Fig. (c), validating the effectiveness of proposed unified slice-volume modeling paradigm. Our main contributions are:
Bridges the gap between slice and volume represen- tations, injecting 3D spatial priors while retaining the efficiency of 2D alignment.
Unified LVLM Paradigm for CT Imaging:
Representation Enhancements: We design SCE and OSE to bridge slice-volume gaps and em- bed organ-centric semanties, yielding spatially coherent and clinically meaningful representations.
MedEval-CT: Establishes the first holistic evaluation suite for CT imaging, augmented with 1.7M multimodal VQA samples, enabling fair, comparable, and scalable assessment medical LVLMs.
Substantial Performances and Strong Bascline: OmniCT outperforms all medical LVLMs and general LVLMs with a significant margin across multiple slice- and volume-driven CT benchmarks, establishing strong baseline for future research towards clinical medical LVLMs.
METHODOLOGY
We propose OmniCT, unified voice-volume LVLM for CT-centric understanding (Fig. 2. Unlike prior medical LVLMs restricted to either 2D slices or 3D volumes, OmniCT incorporates SCE and OSE enable comprehensive CT representation.
3

---

Published as a conference paper at ICLR 2026
2.1 SPATIAL CONSISTENCY ENHANCEMENT
To bridge the representational gap between slices and volumes, we propose Spatial Consistency Enhancement (SCE) module, which injects volumetric priors into LLM while remaining com- patible with slice-driven approaches. SCE leverages Volumetric Slice Composition (VSC), Tri-Axial Positional Embedding (TPE), and MoE Hybrid Projection (MHP) to unify 2D slices and 3D volumes into the LLM space, enabling localized spatial perception, spatial position encoding, and seamless alignment of slice/volume representations within the L.M space, respectively.
Volumetric Slice Composition. For a 3D CT volume V E RD where D, H. W rep- resent the dimensions along the z, y, and x directions, respectively, VSC structurally concate- nates adjacent slices along the axis to construct locally consistent volumetric units: & Concal (oncal(V3:-a (V3--2, V3--1, 1, V3s) for i I,...: [D/3], where å; E R33xXW represents a reassem- bled unit that preserves cross-slice spatial transitions. For independent 2D slice inputs S {s1...., S, ), 8t E IR xXxxXW we simply replicate s; along the channel axis to construct 8,. In this way, both 2D slices and 3D volumes are unified as a series of reassembled units $ = {&:li E [1, and 8, has a size of 3 H W, where 3 is the channel number.
Tri-Axial Positional Embedding. Through volumetric slice composition, 2D slices or 3D volume are transposed into unified units S of size N₂ x 3 x H x W, which are processed by a vision encoder (. 0.) with parameters 0g to obtain patch-level visual tokens F:
F= 2 ( 0.) = {o. (.)
0.)3 RN. xd.
Here, = H H'
and w' W denote the spatial dimensions before flattening the patch features, and &
K the patch size for tokenization is x K K. N, represents the number of unified reassembled units and can be regarded as new depth dimension of reassembled units. To summarize, N, reassembled units are as inputs to generate N, 1 x H' x W" tokens with a dimension of d,.
To explicitly inject global volumetric awareness, we construct sinusoidal positional encodings P == {PN,, pH ,PW') } along the depth N.. height Hi' and width W' dimensions of the reassembled units. This yields tokens Z enriched with 3D positional priors:
== P F pÑv pH' pW', .Z ((d,
where @ denotes concatenating tokens with positional encodings along the feature dimension.
MoE Hybrid Projection. To mitigate token explosion and reduce redundancy in the visual token representation of volumetric units for native volume input, we first perform a loken-level unshuffle m tokens into more representations
operation on Z. This operation clusters spatially adjacent while preserving spatial relationships, resulting in newly generated token representations 2;
- = =2), 2 N. /m) /m)x x [dd, +d, +d, +d=)xm])
(3)
where 2/ denotes the token-level unshuffle operation, with m for slice inputs to preserve original resolution. Subsequently, we employ a slice-volumg Mixture of Experts Hybrid Projection (MHP) (.(- 0p) to align features with the LLM's representation space, formally expressed as:
サ= 0p- (W., W., W )) Wahare o (W.2. 1 + W 1volume)
where (-) denotes the GELU activation function, and 1 and 1 are binary indicator fune- tions that represent routing conditions for the slice and volume features, respectively (1 if the condi- tion is satisfied, and 0 otherwise). The final tokens JF has size of L xd,, where L = N, H' W represents the total number of tokens, and y denotes the output feature dimension of MHP, which takes an input feature dimension of (d, + dg + dyy + dz) x m²?
Overall, the above SCE process generates unified CT tokens that are compatible with both slices and volumes, while embedding spatial positional awareness. These unified tokens are subsequently projected into the LLM representation space via MHP, serving as the input tokens for the LLM.
2.2 ORGAN-LEVEL SEMANTIC ENHANCEMENT
CT images are typically represented as high-resolution 3D volumes, often consisting of more than 150 axial slices with an in-plane resolution of 512 x 512, whereas lesions usually occupy only
4

---

Published as a conference paper at ICLR 2026
small and localized region. To enable clinically practical LVLMs capable of identifying abnormal features within such high-dimensional data, we introduce an Organ-level Semantic Enhancement (OSE) module within our unified framework, which consists of three components: anatomical re- gion localization, semantic feature aggregation, and context fusion.
Anatomical region localization. Given the visual token representation f E R.xd produced by SCE, we perform region-wise selection based on spatial priors of the target organ The organ mask is denoted as M, E RDxxXW including 117 anatomical structures, which is generated by TotalSegmentor (Wasserthal et al. 2023). This mask of D H W is mapped to the token size by leveraging the scaling relationship between pixels and vision tokens, resulting in the organ organ-specific subset: F. F[Mol, where [Mo] denotes mask-based indexing for token selection. Fo represents the selected tokens of size Lo d, for organ by the organ mask M..
Adaptive organ-level feature aggregation. Since different organs exhibit significant variation in scale and token length, directly concatenating them with text tokens lead severe length im- balance. To address this issue, we design a died-dimensional discriminative aggregation function Agg(.), which compresses F, into unified size:
fo (Fg) E Rleeddn Fo E RR..xdd,
(5)
where Le denotes the fixed number of aggregated tokens compressed from L.- This aggregation not only reduces token redundancy but also introduces "magnification effect" for small organs, enhancing fine-grained lesion features. Simultaneously, it applies a "compression effect" to large organs or global regions, effectively minimizing redundancy and preserving essential information.
Finally, the organ-level aggregated representation fo is concatenated with the global visual tokens fF to generate global-local vision tokens FOSE: FOSE = fol. and combined with text tokens as input to the LLM backbone, forming semantically enhanced multimodal representation.
Overall, OSE enhances discriminative capability at the local (organ) level while maintaining contex- tual consistency the global level, thus delivering more relevant and interpretable representations for downstream clinical reasoning tasks.
2.3 TRAINING STRATEGY
After applying Spatial Consistency Enhancement and Organ-level Semantic Enhancement, ob- tain enhanced medical visual features FOsE E [Ld+ Meanwhile, the text query Q {gI.... .m is embedded with text embedding matrix dx(-)).):
= ==((0)))) E gmdda
The two modalities are concatenated into unified input T= T= E) which fed into the LLM to model the conditional probability distribution. The overall optimization objective is formulated as minimizing the autoregressive cross-entropy loss:
Ny log P(y
0 2=[(0p) [0p). -1111 (0p, Oum), Instruction Tuning Stage.
Pretraining Stage,
min B(TTEND
Mee T:0)
(6)
0
3 DATASET
Current medical benchmarks predominantly emphasize broad multi cooti--mooall capabilities, yet fall short capturing the domain-specific demands of CT-based clinical interpretation. To bridge this gap, we introduce MedEval-CT, the first holistic evaluation framework for CT understanding, structured along three complementary dimensions: Datasets (MedEval- -T-aaasee),, Benchmarks (MedFval- CT CT-Bench), and Tools (Data Orchestration Engine and MedEval- CT CT-Factory).
3.1 MEDEVAL-CT
MedEval-CT-Dattsset, We introduce BodEval--CT-attasee the largest unified CT imaging resource to date, comprising over 1.7 million VQA samples from 170,280 independent 3D volumes and
5

---

Published as a conference paper at ICLR 2026
(g) Valddd------Beehhee
atorrs
langs
HeArT
(b) MediEval-CT- Beeeaal----e--------- Benchhhh--olumee Volume
alhers
Hears
(c) Data Engineering
VGA Quality pool
Modeilyy VOA Oua [Sroat
MedEval-CT Dateee!
Liver
volle
58%
1344%
spine
4.4%
Liver vpllo
BB
Quadlted Carpus Pool
Images & Vieual QA Pairs
21.8%
1.%
Powe trachen
18.7%
35.0%
Corpus Selector
Integrity Verifier
Po Kidwey
11.1%
31.%
Kalnes
ModEval-CT :Bench
Muli Chaace Qussston
Pronpt Templase
42
Semantic Reflinr
Stumac
MCD the medical where paneechyma changeo primartly cbeanved A. Apeec of lunea Bsew of lunge C. Mectastrum D. Fleualccoiity
Clivcal Task feod Organ: jood
beswel
Pamn
Figure 3: (a) and (b) illustrate the data distribution of MedEval-CT--eene levels, respectively, encompassing both the clinical- based categorization (4 types: GIR, MAI, All, and CRD) and the organ-level distribution (13 organs). (c) presents the data engineering pipeline.
at the slice and volume
327,063 standalone 2D slices. The two subsets are collected from distinct, non-overlapping sources, ensuring that the 2D slices are neither extracted nor sampled from the 3D volumes. To fully leverage the rich spatial structure and clinical semantics of volumetric CT data, each 3D volume is anno- tated with multiple VQA instances covering diverse diagnostic perspectives, whereas the 2D subset mainly supports slice-level interpretation with typically one question per slice. Overall, the dataset enables comprehensive evaluation of both 2D clinical understanding and 3D volumetric perception, accounting for 16.3% and 83.7% of the data, respectively, As shown in Fig. 3 the dataset is sys- tematically partitioned across three dimensions: task types, clinical categories, and organs, enabling multi-faceted evaluation of LVLMs. For task types, it spans seven medical VQA scenarios, from structured to open-ended tasks: (1) Free-form QA (38.0%) (2) Multiple Choice (18.0%) (3) Short Answers (13.7%) (4) Report Generation (11.5% 5%) (5) Caption (7.2%) (6) Dialogue (6.2%) and (7) Judgment (5.5%). Clinical categories reflect increasing difficulty, progressing from basic anatomical recognition to expert-level reasoning: (1) General Imaging Recognition (GIR, 21.9%) (2) Medical
Abnormality Identification (MAL, 23.6%) (3) Advanced Imaging Interpretation (AII, 34.5%) and (4) Clinical Reasoning and Decision (CRD, 19.9%). Organ-wise, it covers lungs (32.4%), vessels (8.2%), heart (6.6%), liver (4.9%), kidneys (4.3%), and additional regions like spine, trachea, and esophagus, ensuring robust anatomical diversity. Overall, MedEval-CT surpasses existing datasets in scale and granularity, providing high-resolution distributions across tasks, clinical expertise, and organs to advance the development of LVLMs for CT imaging. Data sources and other details are presented in Table and Appendix E
Based on the MedEval-CT--atteee we further construct MedEval-CT. Bench, the first systematic hybrid benchmark tailored for slice-volume CT. Its design emphasizes task-organ dual balance: on the one hand, we perform stratified sampling across different clinical problem types (GIR, MAI, AlI, and CRD), ensuring full task uask-spetrumm coverage from lowel interpretation to high-level reasoning; tion, strengthening core organs (heart, lungs, liver, kidneys, etc.) while retaining long-tail structures (spine, trachea, esophagus, etc.), thereby guaranteeing fairness and comparability in clinical. To further improve clinical semantic fidelity, we propose clinicl-ggraulaan test questions to more fine-grained clinical level and adds more confounding answer options while maintaining their diagnostic intent, ensuring they better reflect the variations encountered in real- world diagnostic scenarios. In summary, MedEval-CT-Bench represents significant advancements in task hierarchy, organ-level balance, and clinical authenticity, offering a more rigorous and de- manding benchmark for CT understanding evaluation.
SodEval--T-Benele
the other hand, we maintain balanced organ representa-
rewriting. which refines
Data Orchestration Engine. We introduce a Data Orchestration Engine to support the construc- tion of MedEval-CT. The engine comprises four complementary modules that collaborate across key stages, forming a self-consistent medical knowledge pipeline. It enables end-to-end capabilities for large-scale sampling. clinical consistency verification, structured task mapping, and semantic refinement: (i) Corpus Selector: Combines LVLM capabilities with rule-based constraints to filter CT samples from multi-source imaging datasets, ensuring representativeness across modality (2D slice/3D volume), anatomy (heart, lungs, liver, etc.), resolution, and image quality. (ii) Integrity Leverages multi-modal reasoning and rule-based checks, supplemented by a 10% manual audit, to guarantee alignment between images and texts in modality. organ semantics, and pairing consistency. (iii) Task Mapper: Maps qualified samples to four clinical task categories and thirteen organ classes, ensuring balanced task complexity and anatomical coverage in MedEvval-CT-Bent
Verifier:
6

---

Published as a conference paper at ICLR 2026
(iv) Semantic Refiner: Rewrites test questions under clinical context, introducing synonymous phrasing, terminology variations, and subtle confounding options to generate semantically simi- lar but more discriminative multiple-choice items, thereby enhancing the benchmark's ability to evaluate clinical reasoning. Overall, the engine constructs a large-scale yet distribution-balance MedFFal-CTT-Datase while ensuring that MedEvaa--CT-Bench achieves reliability in terms of task hierarchy, organ balance, and clinical authenticity. Details are provided in the Appendix F
MeedEval-CT-Factooe We introduce MedEval-CT-Factory, an institutionalized evaluation factory designed to address the heterogeneity of inputs, features, and outputs in medical LVLMs. At the input level, the Factory standardizes diverse CT data formats, including DICOM, NIfTI, arrays, and slice sequences, enabling seamless 2D/3D processing. At the feature level. it unifies model inputs (single images, multi-slice sequences, videos, or volumes) via frame sampling, resampling, and projection strategies. At the output level, it provides multi-layer evaluation protocol, ranging from statistical metrics (BLEU (Papineni et al.) 2002), ROUGE (Lin 2004)), to semantic metrics (BERTScore (Zhang et al. 2019), embedding similarity (Zhang et al. 2025b)), and further to LLM- based evaluation simulating clinical reasoning. Overall, MedEval-CT- Faccory streamlines complex engineering workflows into standardized framework, ensuring comparability across models. Serv- ing as the fourth pillar of the MedEval-CT paradigm alongside the Dataset, Bench, and Data Engine, it is designed toolbox enhance both the efficiency and fairness of LVLM evaluation in the CT domain. (The framework and details of BodEFval-CT--actovy are shown in Appendix G
4 EXPERIMENTS
Baseline Comparisons. To comprehensively evaluate the performance of OmniCT against exist- ing open-source medical LVLMs as well as strong general purpose LVLMs, we select diverse set of baseline models that systematically cover both the general-to-medien spectrum and the 2D- to-3D setting. For the 2D slice-based benchmurks, we include general-purpose LVLMs such as In- VL3 (Zhu et al. 2023), Qwen2.5-VI. (Bai et al.) (025b)), and GPT-5 (Wang et al., (025)) together with representative indicalldooaai models including HealthGPT (Lin et al. (2025). HuatuoGPT- (Chen et al. 2024b), MedGemma-44-B
(Sellergren et al. (2025), MedVLM-RI-
2B (Pan et al. 2025). Lingshu (Xu et al. 2025). and RadFM (Wu et al., (022). For the 3D volume-based benchmarks, we compare OmniCT with strong general LVLMs (MiniCPM-V-44 (Yu ct al. 2025), Qwen2,5-VL. and GPT-5) as well as specialized 3D medical LVLMs, including M3D- LaMed (Bai et al. 2024), CT-CHAT (Hamamci al.] 2024a), and RadFM. The Data Details, Model Details, and Implementation Details are shown in Appendix
4.1 MAIN EXPERIMENTS
Slice-driven Understanding. We systemat- ically evaluate OmniCT on four mainstream VQA benchmarks. As shown in the Ta- ble 2 medical LVLMs (c.g., HuatuoGPT-V. Owen2.5, MedGemma) demonstrate relatively strong medical semantic understanding in cer- tain tasks but remain limited in overall perfor- mance, often encountering bottlenecks on com-
Table 1: Ablation analysis of OMniCT.
Public Bench.
MedEval-CTT -ennch
SCE
OSE
2D
3D
Organ
Task
Avg.
78.68
62. 17
76.51
78.41
77.62
80.14
63.68
76.79
78.69
78.166
80.74
65.37
77.02
79.42
78.62
81.45
66.15
78.24
80.27
79.62
plex tasks. In contrast, general LVLMs achieve competitive or even superior results on some bench- marks, reflecting their strengths in language reasoning, but lack adaptation CT images. For com- parison, RadFM, although capable of handling both slice and volume inputs, achieves the weakest performance across all slice benchmarks, with an average score of only 32.12. failing to meet the demands of fine-grained CT tasks. Under the same evaluation protocol, OmniCT consistently sur- passes existing models both 3B and 7B scales: the 7B version achieves an average score of 81.45, exceeding the second-best model Lingshu by more than +11.01. These results demonstrate the ro- bustness and comprehensiveness of OmniCT on slice-driven tasks.
Volume-driven Understanding. As shown in Table 3 we further assess OmniCT on M3D, CT- RATE, and 3D-RAD to evaluate its volumetric perception capability for CT volume. Results show that existing volume-driven medical LVLMs (e.g.. M3D-LaMed-77/4B CT-CHAT) achieve strong performance on specific subtasks for example, CT-CHAT reaches 86.46 on CT-RATE multi- choice but their overall averages remain below 36, highlighting limitations in coverage and stabil-
7

---

Published as a conference paper at ICLR 2026
Table 2: The comparison of OmniCT with other LVLMs on 2D CT benchmarks.
SLAKE
VQA-RAD
OmniMedVQA
RadFig- -QQ
Model
#Params
Avg.
Close
Open
Close
Open
Task1
Tack2
Fasy
Medium
Med-LVLM (Slice-centric
HealthGPT
74.74
56.33
71.88
33.45
57.36
54.20
70.81
71.22
72.90
62.54
HuatuoGiPT---Hees V-Qwen2.5
7B
72.68
44.19
72.92
35.68
71.07
83.07
76.56
73.76
71.74
66.85
MedGemma-4BBFT
4B
68.04
53.95
$6.25
33.55
61.42
67.00
64.59
65.40
64.20
59.38
MedVLMM-R1-BB
2B
$9.90
67.37
55.50
54.51
55.56
Lingshn
7B
80,93
74.23
75.00
34.62
68.02
69.97
77.51
78.48
75.22
70.44
General-LVV..
InternVL3
8B
73.20
60.88
69.79
34.79
63.96
71.78
68.42
70.46
68.55
64.65
Qwen2.5.VL
8B
69.59
47.86
69.59
35.54
62.94
65.92
65.07
69.45
67.10
61.45
GPT-5
78.35
45.86
70.83
41.05
67.00
69.10
80.86
78.90
81.74
68.19
Med-I.VIM
(Multi-ge Solutti-ggadaarityy
RadFM
14B
51,03
43.88
53.12
20.29
30,97
28.29
23.92
19.75
17.83
32.12
OmniCT (Ours)
3B
77.84
85.32
70.83
30.01
97.46
97.25
79.43
82.03
79.13
77.71
OmniCT (Ours)
7B
85.05
87.20
76.04
36,32
97.97
98.70
82.30
85.82
83.62
81.45
ity. General LVLMs also exhibit strong cross-modal generalization in certain volume tasks, with GPT-5 leading multiple subtasks on 3D-RAD; however, their performance is highly inconsistent and lacks domain adaptation to CT volume. By contrast, OmniCT achieves clear advantages both 3B and 7B scales: the 3B version reaches 87.38 on CT-RATE multi-choice with an average of 63.48, while the 7B version achieves 85.69 on the LTD task of 3D-RAD, pushing its overall average to signifiiiyy outperforming all compared models. In addition, considering the signifi- cant importance of CT report generation, we performed 18-class abnormality label prediction for the report generation task on CT-RATE using fine-tuned RadBERT (Yan et al. 2022). The results show that OmniCT outperforms most volume-driven CT models and previous unified models (see Table [6). and performs similarly to models specifically designed for CT volume report genera- tion (Hamamci et al., 2024b; Di Piazza et al. 2025). This validates the superiority of OmniCT in 3D spatial modeling and cross-tast consistency. Across both slice-driven and volume-driven bench- marks, OmniCT demonstrates stable and comprehensive superiority at different scales, highlighting its holistic perception of spatial-semane features in CT volume understanding tasks.
4.2 ABLATION ANALYSIS
We conduct a systematic ablation study on the proposed SCE and OSE modules across multiple pub- lic 2D/3D CT benchmarks and our MedEval-CT-Benceh Notably, we keep the VSC and the MHP fixed throughout all experiments, since they serve as fundamental mechanisms for coupling 2D slice understanding with 3D volumetric perception. Under this prerequisite design, we further analyze the contribution of SCE and OSE. Results are summarized in [ On the 2D slice benchmarks, the baseline achieves an average score of 79.38; introducing SCE alone improves performance to 80. 14, while adding OSE alone yields 80.74. When both are combined, the performance further increases to 81.45, achieving the best results. On the 3D volume benchmarks, the baseline starts at 62. 17: adding SCE improves it to 63.68, while adding OSE alone boosts it 65.37. The complete model OmniCT
combining both modules reaches the highest score of 66.15. On the MedEval-CT--ennde consistently outperforms the baseline with the addition of the SCE and OSE The improvements in both organ-level and clinical-level tasks further validate the effectiveness of these two modules. Overall, both SCE and OSE contribute significantly to performance gains, with even stronger effects observed on volume-driven tasks, demonstrating the effectiveness and complementarity of the pro- posed enhancements. Moreover, MHP is a crucial component for synergizing 2D and 3D modalities. We provide further analysis of its beneficial role in cross-modal generalization in Appendix H.2
4.3 IN-DEPTH STUDY
(i) Performance Advantages of Mixed Data Training- As shown in Figure 4 (a), OmniCT con- sistently achieves the best performance across different proportions of mixed data, demonstrating its strong adaptability to cross cross-modal modeling. OmniCT exhibits strong performance even under langl--modllity training. We attribute this behavior the combination of unified single-tower mantic space and the MHP, which together enable projection patterns learned from slices to extend naturally to volumes, and symmetrically allow volume-trained representations transfer back to
8

---

Published as a conference paper at ICLR 2026
Table 3: The comparison of OmniCT with other LVLMS on 3D CT benchmarks.
M3D
CT-RATE
3D-RAD
Model
#Params
Avg.
Cup
Close
Open
choice
Malu
Clinical
Entity
Report
1..
A.D.
E.D.
STD.
LTD.
Med-LVL.n
(eotume-centric))
M3D-LaMed-7B
7B
24.79
75.78
56.09
47.44
18.15
16.18
16.85
16.71
18.00
25.47
24.17
30.88
M3D-LaMed-41
4B
46,30
75.08
53.83
59.29
13.66
13.46
17.60
17.49
40.25
25.40
24.31
35.15
CT-CHAT
8B
21.21
35.88
21.81
86.46
49.95
46.76
31.56
29.98
45.33
12.95
13.68
35.97
General-IVVM
MiniCPM-V-4.5
9B
18.44
43.20
26.89
69.21
26.21
23.21
28.03
29. 29.80
30.98
12.70
16.32
29.54
Qwen2.5-VL.
8B
22.62
48.64
28.99
61.34
37.51
26.84
30.51
30.60
41.28
9.19
13.05
31.87
GPT3
21.66
50.36
33.60
64.27
32.98
67.00
59:07
77.97
Med-L.VI.
Miuluargruuulrritt
RadFM
22,62
30.39
19.82
63.93
19.52
17.92
23.25
24.67
29.20
42.99
30.77
OmniCT (Ours)
27.75
81.24
62.
87.38
63,43
51.67
52.02
84.75
64.43
72.05
63.48
OmniCT (Ours)
7B
26.61
83.84
63.88
89.80
63.99
$2.48
53.68
51.97
87.77
67.91
85.69
66.15
slice. A more detailed analysis of this mechanism is provided in Appendix H.2 (vi). We conduct balanced sampling to verify that OmniCT can facilitate effective knowledge fusion between 2D and 3D modalities through mixed training.
(ii) 2D Encoders vs. 3D Encoders. Given the inherent differences in design objectives and input modes, directly applying 3D encoders to 2D inputs often requires artificial adaptations such as depth replication, which compromises the fairness of comparison. Therefore, we conduct evaluations in native 3D settings. As shown in Figure 4 (b), even though M3D-CLIP (Bai et al.) 2024) is pretrained with contrastive learning on the M3D dataset, it does not exhibit a clear advantage over 2D encoders such as DINOv3 (245 tokens) (Siméoni et al., 2025) and SigLIP (405 tokens) (Zhai et al. 2023), despite using the largest number of visual tokens (512). These results indicate that, at this stage, 2D encoders not only provide a more natural compatibility with both 2D and 3D inputs but also demonstrate stronger generalization across tasks, organs, and modalities. To assess the generality of this finding, we additionally evaluate several recent native 3D encoders (Wan et al. 2025 Wang et al. 2023) under the same protocol; the results are provided in Table 4 It is worth noting that we are not claiming that 2D features can fully represent 3D volumes. Instead, we offer a more measured assessment: at the current stage, through structured reorganization and volume-level embedding, more generalizable 2D encoders can robustly carry 3D spatial information. This design does not collapse dimensionality; rather, it retains spatial structures and relationships that remain interpretable from 3D perspective on top of a 2D semantic backbone.
(iii) Organ- and Task-level Performance Analysis. On the organ level (Figure (c)), OmniCT consistently outperforms baselines across the chest, transition zone, and abdomen. The advantage particularly striking for anatomically challenging small organs such the pancreas and esopha- gus, where most existing LVLMs suffer severe performance degradation. This highlights OmniCT's ability to capture fine- mne-grained organ semantics and boundary cues, effectively filling critical blind spot of prior models in handling complex anatomical structures. On the task level (Figure 4 (d)), performance shows a clear gradient with respect to clinical difficulty: while most models display a significant gap between low-level anatomical recognition and high-level reasoning, OmniCT main- tains consistently strong results across all levels, substantially narrowing this gap. This stability demonstrates that OmniCT not only enhance local anatomical discriminability but also reinforce consistency in clinical reasoning.
RELATED WORK
Slice-driven Medical LVLMs. With the success of Large Language Models in general lan- guage (Liu et al., 2024 Yang et al. 2025) and vision (Qiu et al. 2020 Bai et al. 2025a), early explorations focused on adapting general LVLM paradigms to the medical domain, such as LLaVA- Med (Li et al. 2023) and Med-Flamingo (Moor et al. (003), which leveraged medical image-text pairs and instruction data to enable initial medical capabilities. Subsequently. series of more general- general-purposs purpose medical LVLMs emerged, including RadFM (Wu et al., (025)), BiomedGPT (Luo et al. 2023), HuatuoGPT- Gi-iuiiooT---iionnno (Chen et al., 2024a), and Lingshu (Xu et al. 2025). These mod- els advanced the field through large-scale data curation (Bansal et al. 2024). iissonin-eenhaced
training strategies (Pan et al. 2025 Xu et al. 2025), multi-task generalization (Jiang et al.) 2024).

---

Published as a conference paper at ICLR 2026
Only 20 duL
OnS d0
Misod 80
OPEET Ranch
Model jit (Medrild ModBerma, interni,, Owend Vi, HeathoPtt GPT., Lagots]
Berchrank
Sencteaak..gg
Mode jrt (MenGMMays Oiees2 arrid, criChstt MIDLaleesis, MIDualled-49
0.8
30 Perfomenn 30 Perormnno
-0.6
M0T catoo 201 cr CHAT
+0.4
Lings
storrads
Aveen
cntore
30%
100%
211
30%
100% MeCC31M
OmnectT
Chesz
Transiton
Abdomon
(6)
Figure 4: (a) Comparison of OmniCT with 2D/3D LVLMs on 2D/3D benchmarks using 30%, 100% training data of 2D, 3D, and mixed 2D/3D. (b) The study of using 3D vision encoder, 2D vision encoders by different pre-training ways. (c) Per-organ performance heatmap of 2D/3D models and OmniCT on 2D/3D MedEval-CT-Bench. (d) Performance heatmaps by clinical task category and bar charts comparing performance with clinical knowledge requirements across task categories.
Table 4: 2D 3D Encoder Comparison.
Encoder
Token Budget Ratio
Plane
Phase
Organ
Abnormality
Location
Avg.
M3D-CLIP
1.26x
99.0
84.8
77.1
78.2
63.1
80.4
VideoMAEv2
0.97x
91.3
74.7
76.9
75.3
62.8
76.2
Wan2.1-VAN
1442x
97.9
76.4
77.1
74.7
63.9
78.0
DINOv3
0.61x
99.5
77.8
78.9
65.3
81.9
Sigl.ip
1.00x
99.5
90.2
78.4
79.2
67.4
82.9
and domain-specite knowledge integration (Sellergren et al. 2025 Recently, models such CXR- LLaVA (Lee al. 2025) and EyccareGPT Li et al. 2002a)) have demonstrated stronger adaptability and diagnostic value in moodality-specinfe and specialty 2023). Nevertheless, despite substantial progress in data scale, architectural design, and task versity, slice-driven medical LVLMs remain constrained by their reliance planar inputs, limiting their ability to capture the spatial consistency and cross-slice dependencies essential for CT under- standing.
tasks (Xie al.) 2025 Hao al.
Volume-driven Medical LVLMs. To overcome the limitations of 2D modeling, research has in- creasingly turned 3D volumetric imaging, employing dedicated datasets, 3D encoders, and cross- modal alignment modules to strengthen spatial modeling in clinical tasks (Wu et al. 2025). M3D- LaMed (Bai et al., 2024) established a comprehensive evaluation system across multiple volumetric medical tasks, while CT-CHAT (Hamamci et al. (024c) introduced paired chest CT data and an architecture tailored for fine-gr fine-grained analysis and dialog-based interaction. At the methodological level, Med-2F3 (Shi et al.] 2024) combined 2D and 3D encoders and enhanced reasoning consis- tency through dynamic cross-slice scoring, whereas Med3DInsight (Chen et al. 2024c) aligned a 3D encoder with a 2D LVLM, achieving strong performance in both segmentation and classifica- tion. Nevertheless, the lack of a unified clinical evaluation framework and efficient voiue-volume collaboration mechanisms continues limit adaptability and scalability.
CONCLUSION
We propose OmniCT, unified slice-volume LVLM for CT analysis. Through the proposed SCE and OSE modules, OmniCT achieves spatially coherent and clinically grounded representations, leading OmniCT to realize new state-of-the-at performances on multiple benchmarks. We further propose MedEval-CT, a unified, fair, and comprehensive evaluation framework for 2D/3D CT analy- sis. Detailed evaluations reveal that existing general-pure purpose and medical LVLMs exhibit significant performance biases across clinical tasks for different organs. In contrast, OmniCT demonstrates ceptional capability with balanced performance across all organs, which will encourage LVLMs to focus enhancing clinical capabilities for various organs in the CT domain.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work has been supported in part by the NSFC (No. 62436007), the China Postdoctoral Science Foundation under Grant Number 2024M752794, the ZJNSF (No. LZ25F020004), the Key Research and Development Projects in Zhejiang Province (No. 2025C01 128, 2025C01030, 2025C02156), Zhejiang University Education Foundation Qizhen Scholar Foundation.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. No human subjects or animal experimentation were involved in this study. All medical imaging datasets used in experiments, including SLAKE, VQA-RAD, OmniMedVQA, RadFig-VQA, M3D, CT-RATE, and 3D-RAD, are publicly available under relevant research licenses and comply with usage guidelines, ensuring no violation of privacy or ethical standards. No personally identifiable information (PII) was included, and all CT data were pre rre-processed (e.g.. windowing, resampling, and anonymization) before use. We took special care to avoid any misuse of data and to ensure that all evaluations were conducted fairly and transparently. Our contributions focus solely on methodological and benchmarking innovations, without clinical or diagnostic decision- maisiinnmmaiing implications.
REPRODUCIBILITY STATEMENT
We have made every effort ensure the reproducibility of our results. All codes and datasets will be released in an open repository with detailed documentation. The experimental setup, including model architectures, training hyperparameters, optimization strategies, is described in full detail in the main text and appendix. In addition, we rely on multiple widely used public datasets (e.g., SLAKE, VQA-RAD, M3D, CT-RATE) to facilitate verification and cossscomparioo.. Our pro- posed Data Orchestration Engine ensures consistent data preprocessing and evaluation, further im- proving reproducibility across different models. We believe these measures will enable the commu- nity to replicate our work, benchmark future models fairly, and extend the development of medical I.V.Ms CT understanding.
REFERENCES
Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-lorat arXiv preprint arXiv:2503.01743. 2025.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmid,, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 0tte:2333.087744 2023.
Anthropic AI. The claude 3 model family: Opus, sonnet, haiku. 2024. URL https://api. Comanticschor wwwpihhoooooor...ooooooooooo wwwpoorgr//oooooooooooooooe ID:
(T: 268232499
Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Serhonero-Candela
Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, et al. Health- bench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08875. 2025.
Fan Bai, Yuxin Du, Tiejun Huang, Max Q-H Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models. arXiv preprint arXiv:2404.00558, 2024.
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng. Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang. Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li. Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu. Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lx, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren,
11

---

Published as a conference paper at ICLR 2026
Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qi- uyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang. Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Hu- men Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report. arXiv preprint arXiv:2511.2166 2025a
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025b.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization,, pp. 65-72, 2005.
Hritik Bansal, Daniel Israel, Siyan Zhao, Shufan Li, Tung Nguyen, and Aditya Grover. Med- max: Mixed-modal instruction tuning for training biomedical assistants. arXiv preprint arXiv:2412. 126661, 2024.
Pedro RAS Bassi, Mehmet Can Yavuz, Kang Wang, Xiaoxi Chen, Wenxuan Li, Sergio Decherchi, Andrea Cavalli, Yang Yang, Alan Yuille, and Zongwei Zhou. Radgpt: Constructing 3d image-text tumor datasets. arXiv preprint arXiv:2501.04678 2025.
Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xi- dong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, et al. Huatuogpt-vision, towards injecting medi- cal visual knowledge into multimodal Ilms at scale. arXiv preprinl urXiv:2406. /9280, 2024a.
Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. Huatuogpt- vision, towards injecting medical visual knowledge into multimodal Ilms at scale, 2024b. URL
https::/arxiis wwg.ooogg//ooooo.ooo 2406. 19280
Qiuhui Chen, Huping Ye, and Yi Hong. Med3dinsight: Enhancing 3d medical image understanding with 2d multi-modal large language models. arXiv preprint drXiv:2403.05144 2024c.
Theo Di Piazza, Carole Lazarus, Olivier Nempont, and Loic Boussel. Ct-agrg: Automated annormallit--uuidee report generation from 3d chest volumes. In 2025 IEEE 22nd Interna- tional Symposium on Biomedical Imaging (ISBI), pp. 01-05. IEEE, 2025.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang. al. Video-mme: The first-ever comprehensive evalu- ation benchmark of multi-modal Iims in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24108-24115 2025.
Xiaotang Gai, Jiaxiang Liu, Yichen Li, Zijie Meng, Jian Wu, and Zuozhu Liu. 3d-rad: A compre- hensive 3d radiology med-vqa dataset with multi-temporal analysis and diverse diagnostic tasks. arXiv preprint arXiv:2506. 10.66.11111 1/100...1...,... 11711172200 2025.
Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Bastian Wittmann, Enis Simsar, Mehmet Simsar, et al. A foundation model utilizing chest et volumes and radiology reports for supervised-level zero-shot detection of abnormalities. CoRR, 2024a.
Ibrahim Ethem Hamamci, Sezgin Er, and Bjoern Menze. Ct2rep: Automated radiology report gen- eration for 3d medical imaging. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 476-486. Springer, 2024b.
Ibrahim Ethem Hamamei, Sezgin Er, Chenyu Wang. Furkan Almas, Ayse Gulnihan Simsek, Sev- val Nil Esirgun, Irem Doga, Omer Faruk Durugol, Weicheng Dai, Murong Xu, et al. Developing generalist foundation models from a multimodal dataset for 3d computed tomography. arXiv preprint 243.:2443.77833 2024c.
Jing Hao, Yuxuan Fan, Yanpeng Sun, Kaixin Guo, Lizhuo Lin, Jinrong Yang, Qi Yong H Ai, Lun M Wong, Hao Tang, and Kuo Feng Hung. Towards better dental ai: A multimodal benchmark and instruction dataset for panoramic x-ray analysis. arXiv preprint arXiv:2509.0922 2025.
12

---

Published as a conference paper at ICLR 2026
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang. Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, arXiv e-prints, pp. arXiv-2501 2025.
Can Hu, Yingda Xia, Zhilin Zheng, Mengxuan Cao, Guoliang Zheng, Shangqi Chen, Jiancheng Sun, Wujie Chen, Qi Zheng, Siwei Pan, et al. Ai-based large-scale screening of gastric cancer from noncontrast imaging. Nature Medicine, pp. 1-9, 2025.
Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvga: A new large-scale comprehensive evaluation benchmark for medical Ivim. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22170-22183, 2024.
De-An Huang. Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed caapporalllocaizztion assistant. In European Confer- ence on Computer Vision, pp. 202-218. Springer, 2024.
Songtao Jiang, Tuo Zheng, Yan Zhang, Yeying Jin, Li Yuan, and Zuozhu Liu. Med-moe: Mix- ture of dopain-specific experts for lightweight medical vision-language models. arXiv preprint arXiv:2404. 10237. 2024.
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman, dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Seowoo Lee, Jiwon Youn, Hyungjin Kim, Mansu Kim, and Soon Ho Yoon. Cxr-llava: a multimodal large language model for interpreting chest x-ray images. European Radiology, pp. 1-13, 2025.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang. Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and as- sistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36: 28541-28564 2023.
Sijing Li, Tianwei Lin, Lingshuai Lin, Wenqiao Zhang, Jiang Liu, Xiaoda Yang, Juncheng Li, Yucheng He, Xiaohui Song, Jun Xiao, et al. Eyecaregpt: Boosting comprehensive ophthalmol- ogy understanding with tailored dataset, benchmark and model. arXiv preprint arXiv:2504. 13650, 2025a.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid; An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323-340, Springer, 2024b.
Yingtai Li, Haoran Lai, Xiaoqian Zhou, Shuai Ming, Wenxin Ma, Wei Wei, and S. Kevin Zhou. More performant and scalable: Rethinking contrastive vision-language pre-training of radiology in the LLM era. In proceedings of Medical Image Computing and Computer Assisted Intervention MICCAI 2025, volume LNCS 15966. Springer Nature Switzerland, September 2025b.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122. 2023a.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang. Mengze Li, Xiaohui Song, et al. Healthgpt: A medical large vision- languae model for unifying comprehension and generation via heterogeneous knowledge adaptation. arXiv preprint arXiv:2502.09838, 2025.
Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-clip: Contrastive language-imen image pre-training using biomedical documents. In In- ternational Conference on Medical Image Computing and Computer- Assisted Intervention, pp- 525-536. Springer, 2023b.
13

---

Published as a conference paper at ICLR 2026
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. 2024.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang. and Xiao-Ming Wu. Slake: A semantically- labeled snowledge-enannee dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pp. 1650-1654. IEFE, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.
arXiv preprint
arXiv:1711.05101. 2017.
Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv preprint arXiv:2308. 09442, 2023.
Vishal Monga and Brian L Evans. Perceptual image hashing via feature points: performance evalu- ation and tradcoff's. IEEE transactions Image Processing, 15(11):3452-346) -465523556,, 2006.
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovee, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar, Med-flan led-ffamingoo; multimodal medical few- shot learner. In Machine Learning for Health (ML4H), pp. 353-367. PMLR, 2023.
OpenAl. Gpt-5 model documentation.
cctps://opent com. htai:/zh-Haa-S-onn ndex/
Lotrouucing--pt-- 5/| 2025. Accessed: 2025-08-07.
Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvlm-rl: Incentivizing medical reasoning capability of vision- language models (vims) via reinforcement learning. In International Conference on Medical Image Computing and Computer-Assister Intervention, pp. 337-347. Springer, 2025.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Zhongwei Qiu, Kai Qiu, Jianlong Fu, and Dongmei Fu. Dgen: Dynamic graph convolutional net- work for efficient multi-person pose estimation. In Proceedings of the AAAI Conference Arti- ficial Intelligence, volume 34, pp. 11924-11999 11931, 2020.
Zhongwei Qiu, Huan Yang. Jianlong Fu, and Dongmei Fu. Learning spatiotemporal frequency- transformer for compressed video super-resolution. European conference on computer vision, pp. 257-273. Springer, 2022.
Zhongwei Qiu, Qiansheng Yang, Jian Wang. Haocheng Feng. Junyu Han, Errui Ding, Chang Xu, Dongmei Fu, and Jingdong Wang. Psvt: End-to-end multi-person 3d pose and shape estimation with progressive video transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2023.
Zhongwei Qiu, Hanqing Chao, Tiancheng Lin, Wanxing Chang, Zijiang Yang. Wenpei Jiao, Yixuan Shen, Yunshuo Zhang, Yelin Yang, Wenbin Liu, al. Bridging local inductive bias and long- range dependencies with pixel-mamba for end-to-end whole slide image analysis. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 18778-227444 2025.
Ronan L.M. Rocov2-radiology (revision 5d66908), 2024. URL bataaets//elorri//ooocc radi llogy
co/
Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilities of gemini models in medicine. arXiv preprint 4rXiv:2404. 18416, 2024.
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu. Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint arXiv:2507.05201, 2025.
14

---

Published as a conference paper at ICLR 2026
Yiming Shi, Xun Zhu, Ying Hu, Chenyi Guo, Miao Li, and Ji Wu. Med-2e3: A 2d-enhanced 3d medical multimodal large language model. arXiv preprint arXi::2411.12883, 2024.
Zhongyi Shui, Jianpeng Zhang, Weiwei Cao, Sinuo Wang, Ruizhe Guo, Le Lu, Lin Yang, Xianghua Ye, Tingbo Liang, Qi Zhang, et al. Large-scale and fine-grained lasounla pre-training for enhanced ct image understanding. arXiv preprint 0rXiv:2501.1454., 2025.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, al. Dinov3. arXiv preprint ar Xiv:25008 10104, 2025.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vicillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2, 2024.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20. 203003222344... 2025.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14549-14560, June 2023.
Shansong Wang, Mingzhe Hu, Qiang Li, Mojtaba Safari, and Xiaofeng Yang. Capabilities of gpt-5 on multimodal medical reasoning. arXiv preprint arXiv:2508.08224 2025.
Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang. al. Totalseg- mentator robust segmentation of 104 anatomic structures in ct images. Radiology: Artificial Intelligence, 5(5)x2230024, 2023.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Hui Hui, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. Nature Communi- cations, 16(1):7866 2025.
Yihan Xie, Sijing Li, Tianwei Lin, Zhuonan Wang, Chenglin Yang. Yu Zhong, Wenqiao Zhang, Haoyuan Li, Hao Jiang, Fengda Zhang, et al. Heartcare suite: Multi dilti-dimeesional understanding of ecg with raw multi-lead signal modeling. arXiv preprint 20Xxv:2500605833, 2025.
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng. and Jiashi Feng. Pllava: Parameter- free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404. 16994, 2024.
Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang. Cheng- hao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foun- dation model for unified multimodal medical understanding and reasoning. arXiv preprint 1xXiv:2506.00444, 2025.
Yosuke Yamagishi, Shouhei Hanaoka, Yuta Nakamura, Tomohiro Kikuchi, Akinobu Shimizu, Take- haru Yoshikawa, and Osamu Abe. Radfig-vqa: A www---imaninimmmmliiiiee radiology benchmark for evaluating vision-language models in clinical practice. In MICCA12025 Joint Springer pro- ceedings: 1st Workshop on Multimodal Large Language Models (MLLMs) in Clinical Practice. Springer, 2025. in press.
An Yan, Julian McAuley, Xing Lu, Jiang Du, Eric Y Chang, Amilcare Gentili, and Chun-Nan Hsu. Radbert: adapting transformer-baseee language models to radiology. Radiology: Artificial Intelligence, 4(4))e210258 2022.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338 2025.
15

---

Published as a conference paper at ICLR 2026
Tianyu Yu, Zefan Wang, Chongyi Wang, Fuwei Huang, Wenshuo Ma, Zhihui He, Tianchi Cai. Weize Chen, Yuxiang Huang, Yuanqian Zhao, Bokai Xu, Junbo Cui, Yingjig Xu, Liqing Ruan, Luoyuan Zhang, Hanyu Liu, Jingkun Tang, Hongyuan Liu, Qining Guo, Wenhao Hu, Bingxiang He, Jie Zhou, Jie Cai, Ji Qi, Zonghao Guo, Chi Chen, Guoyang Zeng. Yuxuan Li, Ganqu Cui, Ning Ding, Xu Han, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Minicpm-v 4.5: Cooking efficient milms via architecture, data, and training recipe, 2025. 18154
Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, et al. Videorefer suite: Advancing spatial-temporal object understanding with video 1lm. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18970-18980. 2025.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yuho Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.022 02813, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986, 2023.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13/00 2025a.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre- ston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915. 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint
2019.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang. Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advanc- ing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05170 2025b.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
16

---

Published as a conference paper at ICLR 2026
A
APPENDIX
Section B
LLM usage statement.
Section
Notation Table.
Section
Extended Related Work.
Section
B
Implementation Details.
Section
Data Orchestration Engine.
Section
G
Mechanism of
Section
Supplementary experiments.
B LLM USAGE STATEMENT
In this work, primarily employ LLMs in the following two aspects: (i) Data construction: We use LLMs for data annotation, cleaning, filtering, and rewriting, thereby building MedEval-CTT-Datase and MedEval-CT-Bench. (ii) Manuscript polishing: We leverage LLMs to review the grammar and improve the clarity and accuracy of the manuscript, ensuring that our methodology is properly and comprehensively presented. (iii) Model evaluation: We include several advanced LLMs baseline methods for comparison. For open- open-weight models, we download the publicly released checkpoints and conduct inference locally. For closed-source models, we perform evaluation through their of- ficial APIs. All baseline results are obtained under unified evaluation protocol to ensure fairness and comparability.
C NOTATION TABLE
To provide a comprehensive overview of the notations used throughout the paper, we present summary of key symbols and their corresponding definitions in Table 5 This table serves as a convenient quick reference, covering the main variables and operators involved in formulation. We hope that this notation list facilitates the understanding of our methodology and improves the readability of the paper by enabling readers easily recall the meaning of each symbol.
D EXTENDED RELATED WORK
In recent times, the release multiple multi-modal Large Language Models (MLLMs) has driven innovations in vision-language fusion, long temporal sequence processing, and scenario adaptabil- ity (Hong et al. 2025 Team et al. 2025). laying solid foundation for cross-domain applications. Prominent foundation models such Qwen2.5-VI (Bai al. 2025b). GPT-4o (Achiam et al. 2023), Claude 3.5 (AI 2024), Intern (Zhu et al. 2025). and the latest GPT-5 (Wang et al. (2025) have continuously advanced in multi-modal understanding, long- long-sequence processing, multi-task learning, and vertical domains like healthcare (Arora et al. 2025). demonstrating exceptional poten- tial. These advancements are primarily driven by high-quality data curation and iterative algorithmic optimization. However, foundation models, maintaining a balance between general-purpose ca- pabilities and domain-specific expertise remains a significant challenge.
E IMPLEMENTATION DETAILS
Data Details. For 2D slice evaluation, we construct test sets based on SLAKE (Liu et al.) 2021 VQA-RAD (Lau et al. 2018), OmniMedVQA (Hu et al., (024). and RadFig YYmmagishi et al, 2025), where all samples are systematically filtered by the data engine (Section to retain only high-quality CT VQA data. For 3D volume evaluation, we adopt existing benchmarks including M3D (Bai al. 2024 CT-RATE (Hamamci et al.] 2024a), and 3D-RAD (Gai al. 2025) to cover the full spectrum of CT volumetric scenarios. Regarding evaluation metrics, Accuracy is used for closed-end and multiple-choice tasks, while open-ended QA is assessed by weighted
17

---

Published as a conference paper at ICLR 2026
combination of BLEU (Papineni et al. 2002). ROUGE (Lin 2004), Token-F1 (Saab et al.
2024).
and BERTSccoe (Zhang et al. 2019), balancing lexical matching with semantic alignment to achieve multi-level quality measurement. For data pre-processing, all CT volumes with preserved metadata are windowed to [-1000, 1000) and resampled to 32 x 384 x 384.
Model Details and Implementation Details. We use siglip-so400m-ppiie 2att00000-atth1--33388 14-384 (Zaii al. 2023) as the vision encoder and Qwen2.5 [Team] 2024) as the backbone LLM, with Adam (Loshchilov & Hutter 2017) as the optimizer. During pretraining, only the MoE Hybrid Projection is updated to perform mross-modal alignment, with learning rate of 2 x 10-2, In the vision instruction tuning stage, both the projection layer and LLM parameters are optimized, with the learning rate reduced to 5 x 10-S to ensure stable convergence. All experiments are trained with a global batch size of 256 and warmup-cosine learning rate scheduler. Unless otherwise specified, experiments are conducted under the 7B parameter scale. The specific hyperparameter settings can be found in Table 6
F DATA ORCHESTRATION ENGINE
The construction of MedEval-CT is powered by the proposed Data Orchestration Engine, which integrates four complementary modules. Specifically, the Corpus Selector and Integrity Verifier are implemented with Qwen2.5-VL-72B (Bai et al. (025b), while the Task Mapper and Seman- tic Refiner leverage Qwen3-237B-A3B Yangg et al. 2025), thereby exploiting the complementary strengths of different models in large-scale data filtering and semantic refinement. The specific prompt designs for each module can be found in Fig. 5
To ensure the reliability and independence of MedEval-CT, we proactively conduct a systematic audit of potential data overlap during its construction. For the slice datasets, we note that RO- COv2 (Ronan I.MM., 2022),, PubMedVision (Chen et al., 2024b), ILA (Li et al. 2023), ) and RadFig- VQA (Yamagishi al. 2025) originate from PMC-OA (Lin et al., 2023b). Although these four datasets follow their own automated curation pipelines, we further apply a two-stage dedupli- cation strategy within them: perceptual image hashing (pHash) (Monga & Evans 2006 is used to cluster visually similar images, followed by BiomedCLIP (Zhang et al. (203) feature matching to remove image-text pairs with high semantic similarity. For the volume datasets, we strictly adhere to the official splits of M3D (Bai et al. 2024), CT-RATE (Hamamci et al. (024a)), and 3D-RAD GGai al. 2025); even in cases where datasets share underlying CT volumes, we avoid any cross-dataset training or evaluation. Throughout the pipeline, we retain only
and high-quality CT scans, filtering out blurry, artifact-heavy, or low-resolution samples. These measures allow MedEval-CT to maintain strict separation in data sourcing, partitioning, and deduplication, effec- tively minimizing the risks of training-testing contamination and data leakage.
To further address the concern that using Qwen-family models in both the data pipeline and the LLM base model might introduce family-specific bias or circularity, we additionally instantiate OmniCT with different LLM backbone, Phi-4-mini (Abouelenin et al. 2025). while keeping the training data and optimization protocol unchanged. As shown in Table 7 OmniCT with Phi-4-mini achieves performance that is highly comparable to, and several benchmarks slightly better than, the Qwen2.5-3B variant across both slice-driven (SLAKE, VQA-RAD, OmniMedVQA, RadFig- VQA) and volume-driven (M3D, CT-RATE, 3D-RAD, sistency indicates that the observed gains mainly stem from the proposed unified framework itself rather than from any base model-specifie preference or bias induced by the models used in the data construction pipeline.
(IIEEvvl--T-Bench) benchmarks. This con-
G
MECHANISM OF MEDEVAL-CT--FCTTO
The logical structure of CedEval-CT-Faache is illustrated in Figure 6 This section explains its design motivations and module responsibilities from framework-level perspective.
Unified Processing of Heterogeneous Formats. MedEval-CT-Faatory begins at the input level, where commonly used medical imaging formats standardized. Medical data often come in di- verse forms such as DICOM, NIFTI, NRRD, 3D arrays, RGB slices, and slice sequences, which dif- fer significantly in metadata organization, spatial resolution, and storage layouts. The Factory maps
18

---

Published as a conference paper at ICLR 2026
Given medical Image and corresponding text input, determine whether this Image- text pair a volid CT dataset. Evaluste the following aspects:
medical multimodal quality control
You are a Medical Data Integrity Veriher, The samples you receive have already passed an Initial hitoring step (Corpue Selector) and are considered possible CT imaga-tact ttxt pains. Your role is to conduct a stricter integrity audit, enauring the data auffccently reliable and standardized for training high quality medical muttimodal models.
1. Modality Check:
Confrmm whether the image is a CT scan (computed tomography). X-ray, MRI, ultrasound, natural
Evallate each sample from the following perspectives:
Exclude
images
1.. Data Completeness:
photographs.
Verify that the corrupted presented coherent siice volume, with
Image Quality Assessment:
- Detect any noise, motion artifacts, distortions, color
Check that
description sernantically complete, garbled, nonsensical.
shifts/ pow.do-colooiiiti or incomplete silces that degrade diagnostic quality.
Consistency:
Verify grayscale intensity resemblee standard features
Ensure the Image and remain consistent in details (e.g..
folose to Hounsfield Unit range, without unmatuurl colors or abnormal
anstoricaal focus, scan type, common terminology..
contrest).
Flag subtle mismatches (e.g. text mentions contrest agent but the Image does not show contrast features).
a. Text-image Consistency:
Check whether the text matches the Image (e.g. should mention
3. Format and Standordizaziin:
CT
CT content).
Confirm that the image adheres to basic medical imaging
Flag cases where the text is unrelated, mismatched, or inaccurate.
conventions Igrayscale representation, consistent orientation, no artifcial coloring).
Overall Usability:
Mark as Valid only the image CT scan, the quality is
-Ensur the text clear, medically relevant, and not casual or irelevent.
acceptable, and text relevant to the image.
wrong modility, low quality, text-Image mismatch).
Otherwise,
as Invalid briefly explain the (e.g.,
vague descriptions.
Potential Flaws and Risks:
Idontify latent issues such mild artifacts, unusual labeling,
Output format:
Judgge whether these undermine usability sample.
Validity: Valid/ / Invalid
Judgment:
Resson: short explanation
Corpus Selector
Maak as High-Qualtty / Complete Valid only no sarious flows are detected.
Identify primary organ anatomical region mentioned the question,
Otherwiss, mark as Risky/ / Low-Qualitt and briefly explain the lssue,
and map it to one categories
Output format:
lungs
Integrity Check: Complete & Valid / Risky
heart
-easson short explanation
3. liver
5. kidneys
spleen
Classify the question into of the folloving clinical task
pancrees
categories:
atomach
A. General Imaging Recognition
bowel (small_owwe duodenum, colon)
-asic tasks: modality identihcation (CT/MRI/Ulrraound) body
head Ibrain, skull)
region classification, organ count.
10. galibladder
Medical Abnormality identifiattin
adrensll.go gland
Entry-level tasks: presence of abnormality, lesion, tumor,
esophagus
mass, abnormal density.
14. thyroid_gland
trachee
C. Advanced Imaging Interpretation
15. urinary. bladder
- Intermediate tasks: measuring tumor size, segmentation of organ/ /leslnn, nadule detection, structural assessment.
16. prostate
17. vossals
D. Clinical Reasoning and Decision
18. muscles
Advanced tasks: diagnostic reasoning, malignancy judgment, staging (e.g. TNM), clinical interpretation from imaging.
19. spine
20. rbs__su jos_stemum
21. upper, axtrrmiiiis
Rules:
23. others Inot included above, unclear)
22. lower, axtremities
the question about basic attributes - A about presence/absenee abnormality/ http://eeioo//oorrearr/// about quantitative analysis, segmentation, or structural
Rules:
multiple organs mentioned, choese the most rolevant/primary
organ.
Teatures - C
question only refers modality or general region
about diegnosie, reasoning, staging
specihi organ) choose others.
If the question is vague or unclear choose 23, others.
Output format:
Output format:
task class: [A-D]
or organ_class: [1-23)
with rewriting medical Q8A problems into standardized four-option multiple-choice questions. Your goal preserve the medical logic while increasing clinical challenge through dell-deslgged distractors.
medical data engineer
inputs:
original_question: the original question original answer: the correct enswer
Steps:
Interpret Context
Ensure the rewritten question is medically accurate and consistent with the original.
2. Rewrite Question
Convert into cancise, clear four-option multipee-haiice question.
Generate Distractors
Provide three distractors
Relevant: connected to the medical context but incorrect.
-Pluusible: challenging enough to mialead weaker models, yet identifible es wrong with proper reasoning. Professional: use accurate medical terminology, avoid trivial errors.
Output format:
JSON:
"question": "<Rewiitton question=",
"options": ('A, <Option A>", "B. <Option B=", "C, <Option C=", "D. <Option D=*1
"correct_anewnn anewer": *CCorrect option, equivalent originel_answer answer in clinical semanticss"
Rules:
Exactly one correct answer, three distractors.
Correct answer must match original answer.
Randomize correct answer position.
Distractors may similar organs, diseases, imaging findings Increase diffcultyy
Sertento Befine
Figure 5: Prompt template of data orchestration engine for generating MedEvallCT..
19

---

Published as a conference paper at ICLR 2026
these heterogeneous imputs into unified representation through designated loading rules, enabling subsequent modules to perform slice-volume unified processing without relying on format -ormat-specific operations.
Lightweight but Unified Feature Construction. Building on the standardized inputs, the Fac- tory provides lightweight yet flexible feature construction layer. Instead of enforcing any model- specific preprocessing pipeline, it offers general-purpose mechanisms such as frame sampling, slice aggregation, 2D-3D projection, and resampling, allowing various LVLMs to interface with the eval- uation workflow in consistent manner.
Multi-dimensional Evaluation Protocols. At the output level, MedEval-CT--actoon inte- grates multiple evaluation strategies to ac- commodate the diversity of outputs produced by medical LVLMs. Rather than impos- ing rigid scoring pipeline, it provides composable and extensible evaluation space: (i) statistical metrics (BLEU (Papineni et al.
spap
Avclss
Feiture Engineering
Engies
MedEval-CT-Factory
2002). ROUGE Lin 2004). METEOR (Baner- jee & Lavie 2005 for measuring surface-
officenc
Falrness
level textual alignment; (ii) semantic metrics
(BERTScore (Zhang et al. 2019). embedding similarity (Zhang et al. 2025b)) for assessing semantic correspondence; and (iii) LLM-based evaluation for simulating clinical reasoning, of- fering more qualitative judgments aligned with medical scenarios. Users may flexibly select appropriate evaluation layers according to task requirements without being restricted to sin- gle metric.
Statirical Metrica
Metrics
S
Lvaddigg
Maiinn
Figure 6: MedEval-CT-Factor enables standard- ized preprocessing and fair, consistent evaluation of medical LVLMs across benchmarks.
Overall, the Factory provides a structured, extensible, and model-agnostic framework for conducting consistent and reproducible CT LVLM evaluation. Although not all modules are used in every experiment, its modular design offers room for future extensions.
H SUPPLEMENTARY EXPERIMENTS.
H.1 MEDEVAL-CT-BENCC
Across both MedEval-CTT-Bench-20 (Table 9) and MedEEvl--CT-Bench-3) (Table TO.. OmniCT consistently achieves the highest overall performance, with averages of 79.80 and 77.63, respec- tively, surpassing strong baselines such as GPT-5-mini (OpenAl) 2025). Lingshu, CT-CHAT, and M3D-LaMed. It demonstrates robust gains across diverse organs (e.g., liver, kidneys, heart, spine) and task levels (GIR, MAI, AII, CRD), excelling particularly in advanced interpretation and rea- soning. These results highlight the effectiveness of our unified volue- paradigm in delivering stable, cross-task generalization and comprehensive CT understanding.
H.2 SUPPLEMENTARY ABLATION.
(i Analysis of Cross-Modal Generalization. To understand the source of OmniCT's cross-modal generalization, we analyze the roles of (i) the unified single-tower semantic space and (ii) the MoE Hybrid Projection (MHP). The single-tower backbone embeds 2D slices and 3D volumes into a shared semantic neighborhood, preventing the semantic drift commonly observed in dual-encoder designs. MHP further learns a modality-adaptive mapping from visual tokens to the LLM space, al- lowing the projection behavior learned from 2D slices to transfer effectively to 3D representations, and vice versa. To disentangle the contributions of the two components, we compared a dual-tower without MHP configuration against the single-tower with MHP under the same training setup, and observed a substantial degradation in cross-modat generalization. The results are reported in Ta- ble 1T Therefore, these components form coherent mechanism that supports cross-modal
20

---

Published as a conference paper at ICLR 2026
transfer: unified semanties provide a common representational anchor, and MHP supplies the flexi- bility needed to align slice- and Allume-based tokens under a unified LVLM interface.
(ii) I-SNE Visualization of MoE Hybrid Projection. To further examine whether the two experts in the MHP module learn distinguishable token transformations, we project their output embeddings into 2D space using t-SNE. As shown in Fig. 7 the features routed to the 2D expert and the 3D) expert form two clearly separated clusters. This separation emerges without any explicit supervi- sion enforeing modality-specifie behavior; instead, it arises from the structural differences in the inputs (e.g.. voxelized tokens with VSC/TPE for 3D vs. planar tokens for 2D) and their decoupled optimization paths before entering the shared semantic space. The visualization supports that the two experts encode modality-deeendeee transformations, serving the intended role of normalizing heterogeneous inputs before alignment with the L.L..
(iii) Analysis of Organ-level Semantic Enhancement. The OSE module leverages organ segmen- tation as structural regional prior rather than a supervision target. The segmentation masks indicate organ regions with high semantic load for typical CT-based reasoning, from which OSE aggregates a compact set of discriminative tokens, while all global tokens are preserved in the feature stream. In this way, OSE explicitly strengthens organ-level semantics without sacrificing global context. Since the module relies on organ-level structural consistency instead of pixel-level boundary fitting, the high stability of TotalSegmentor in thoracoabdominal CT (average Dice 94.3% (Wasserthal et al. ) is well suited for providing such regional cues. To evaluate the effectiveness of OSE and to rule potential bias introduced by the segmentation model, designed three alternative strate- gies: (i) removing ROI regions, (ii) random ROI pooling, and (iii) directly concatenating native ROI tokens. As shown in Table 12 removing ROIs yields the expected performance drop; random pool- ing brings limited gains mainly due to weak alignment effects arising from repeated visual tokens; and direct concatenation of native ROI tokens produces variable-length sequences that prevent stable semantic compression and offer no performance benefit. In contrast, OSE's s exed-dimensionp adap- tive aggregation preserves global information coverage while emphasizing diagnostically critical regions, making it better suited to the structured requirements of medical image analysis.
(iv) Ablation of Adaptive Feature Aggregation. We further examined the impact of different 2D/3D aggregation token settings (m2D, MaD) model performance (Table The results show that moderate aggregation (e.g., mgn = 81, MaD = 90) consistently improves both 2D and 3D performance compared to the baseline without OSE. As the aggregation context continues to grow, the gains diminish and eventually decline, likely due to excessive semantic overlap with global features that disperses the model's effective visual attention. Overall, these observations indicate that an appropriately sized set of aggregated tokens can effectively enhance organ-level semantics, increase the information density of visual tokens, and maintain favorable balance between accuracy and computational cost.
(v) Robustness of MedEval-CT-Bende to Answer Leakage To reduce the risk that models exploit language artifacts instead of visual evidence, MedEval-CT-Beee is multiple-choice questions are constructed with cannicllgraanlarigg refiner that rewrites prompts using synonymous expressions, refines clinical wording, and injects stronger distractor options. This preserves the underlying diag- nostic intent while weakening template-like phrasing, simple co-occurrence patterns, and answer- position biases. We further conduct two stress tests on MedEval-CT--ennch (i) an mismatch setting, where questions are randomly paired with incorrect CT scans/volumes, and (ii) noise substitution setting, where images are replaced by noise. As shown in Table 14] both 2D (6-way choice, random 23 x1..7%) and 3D (4-way choice, random Rx25% accuracies drop sharply toward near-random levels under mismatch/noise, while remaining high with normal inputs.
(vi) Unified Representation Gains, To further examine the fcasibility and utility of using a 2D encoder as the semantic backbone for incorporating 3D spatial cues, we conduct balanced sub- sampling study across slice-driven and volume-driven data. Specifically, we perform controlled ab- lations using 25%, 50%, and 100% of the available samples for each modality (results in Table 15). Across all settings, joint training consistently yields measurable performance gains. These results indicate that, under the current scale of available pretraining resources, 2D encoders exhibit more mature semantic generalization and thus serve as a reliable representational anchor for constructing 3D inputs. With structured spatial injection, the unified representation acquires effective volumetric awareness, enabling synergistic improvements across both slice- and volume-level tasks.
21

---

Published as a conference paper at ICLR 2026
20 expert features export reatures
-80
-00
-10
-20
60
Figure 7: t-SNE plot showing distinct clusters of 2D and 3D expert features after MHP module.
M3D-LaMed-7B
Please caption this scan with findings.
the left hepatic lobe with portal venous phase washout. The is diffusely infiltrating the liver and there is evidence of both portal vein and IVC invasion enhancing tumor thrombus. Tumor thrombus extends into the right atrium, Background features cirrhosis and portal hypertension, including ascites, porto-systemic varices, splenomegaly. Cavernous transformation portal vein likely reflects element of chronic thrombus/ocolusion addition the recent tumor in vein.
Large heterogeneous arterial enhancing
Ground Truth
Evidence of portal hypertension: splenomegaly well as coronary and para-esophageal collaterals. The liver volume redistribution and subtle liver surface irregularity are favor cirrhotic changes. Multiple focal cystic dilations of intrahepatic bile ducts visible, prominent
right lobe, some them show the central dot sign in favor of Caroli disease. Several small renal stones and a calcified gallstone are other findings.
GPT-5
Non-contrast axial head CT: The cerebral hemispheres symmetric with preserved gray-white matter differentiation and normal sulcation. The ventricles and basal cisterns normal in size and configuration, and there is no midline shift. No acute intracranial hemorrhage, mass effect, extra-axial fluid collection, or territorial infarct is identified. The calvarium is intact without fracture. The visualized orbits appear symmetric, and the paranasal sinuses and mastoid air cells clear. Overall appearance within normal limits for acute non-contrast head CT.
OmniCT
Evidence of portal hypertension present, Including splenomegaly cirrhotic changes with irregularity and volume redistribution. Multiple intrahepatic bile duct cystic dilations noted, some demonstrating the central dot sign consistent with Caroli disease. Additional findings include a calcified gallstone and several small renal stones.
ascites. The liver shows
pronounced in the right lobe,
Figure 8: case study comparing CT findings generated by different medical LVLMs with the clinical ground truth.
H.3 CASE STUDY
In Figure g the Ground Truth highlights portal hypertension, cirrhotic changes, intrahepatic bile duct cystic dilatations with the central dot sign, and additional renal and gallstones. M3D-LaMed- 7B, while mentioning hepatic and portal abnormalities, overemphasizes tumor invasion with in- correct findings, and GPT-5 completely deviates with irrelevant head CT descriptions. In con- trast, OmniCT accurately captures the key clinical features-porta hypertension, splenomegaly, mild ascites, cirrhosis, Caroli discase signs, and urinary/gallstones closslllllyyyyyyeeeee matching the Ground Truth with clinically coherent language, demonstrating its superior spatial- semanic consistency in chest-abdominal CT interpretation.
22

---

Published as a conference paper at ICLR 2026
Table 5: Notations used throughout this paper.
Notation
Description
V E R
3D CT volume with depth D, height H, and width W j-th 2D slice extracted from the 3D volume Independent 2D slice input
Vj E gxxx
RIXHXW
;.1,...,88] .n)
Collection independent 2D slice inputs Reassembled volumetric unit
8. = Concat (V3i V Vyc)
8 = 18: E [1,n]] No
Set of all reassembled slice units
Number of reassembled units (new depth dimension) Vision encoder with parameters Patch-level visual tokens extracted from
6. (.)) 0.)
F R. xw xdv
H' = H W' =
Spatial resolution of patch features after tokenization Patch size (stride along spatial dimensions) Dimension of visual tokens
K
du
P (PNo pi pw))
Sinusoidal positional encodings along depth, height, width Tokens enriched with tri-axial positional priors
2
d2y day dz
Feature dimensions wepth/heighh/width positional encodings Token-level unshuffle operation
Window size for unshuffle (m=1 for slice input)
2
Token representations after unshuflle
(- (p))
Slice-volume MoE Hybrid Projection function Parameters of MoE Hybrid Projection
0, = (W., We, W W
W., Wo, Wshare
Slice-specifee, volume-specific, and shared projection matrices
Lsiie 1vllume o())
Binary indicator functions for slice/volume routing
GELU activation function
GEBLXC
Projected tokens aligned with LLM space
L = N. H' W
Total number of projected tokens
d,
Output feature dimension of MoE Hybrid Projection
M. E DxHXW
Organ mask for organ o (from TotalSegmentor, 117 structures)
M. Fo Rdo
Organ mask mapped to token resolution Subset of tokens selected M.
Lo Agg(.)
Number tokens belonging organ o Organ-level feature aggregation function Aggregated tokens for organ o
fo E BLo
be
Fixed number of aggregated tokens after compression Global-local vision tokens after OSE fusion Input text query sequence
FOSE
Q = (q1,...,q9i 9m
(r(-10.)
Text embedding function with parameters 0g
E E R Rmxdn
Text token embeddings
T [oSE; E] (Bi,..., Fm)
Unified multimodal input sequence
Target output sequence
P(v Mマu; T:
Conditional probability distribution from LLM
D
Training data distribution
fmm
Parameters of the L.M backbone
23

---

Published as a conference paper at ICLR 2026
Table 6: Overview of the hyperparameter settings used for training OmniCT-3B and OmniCT-7B across two stages.
OmniCT-3B
OmniCT-7B
Hy Hyperparameter
Stage-1
Stago-2
Stage-1
Stage-2
Optimizer
AdamW
AdamW
AdamW
AdamW
Learning Rate of Adapter
2c-4
5c-5
2c-4
5e-5
Learning Rate LLM
5e-5
Se-5
Global Batch Size
256
256
256
256
Weight Decay
0
0
Scheduler
warm up-cosine
warm up-cosine
warm up-cosine
warm up-cosine
Warmup Ratio
0.03
0.03
0.03
0.03
Epoch
2
2
1
Max Sequence Length
2048
2048
2048
2048
Table 7: Performance of OmniCT with different LLM backbones on 2D and 3D CT benchmarks.
OmniMed VQA
RadFig -VQA
MedEval-CT
Base Model
#Params
SLAKE
VQA-RAD
M3D
CT-RATE
3D-RAD
-Bench
Owen2.5-3B
3B
81.6
50.4
97.4
80.2
57.1
67.5
75.9
Phi-4-mini 3.88
81.1
55.3
96.9
84.4
56.4
67.4
66.9
76.2
Table 8: Summary of the datasets included in MedEval-CT, with their task types, training/test sizes, sources, and licenses.
Dataset
Task Type
Train Size
Size
Source
License
Volume-driven Datasets
M3D-CAP
Report
116065
100
Radiopacdia
Apache-2.0
M3D-VQA
Multiple Choice
240929
5000
Radiopacdia
Apache-2.0
Short Answer
240929
5000
Report
93822
6076
Istanbul Medipol University.
CT-RATE<2
Free-form QA
693760
24149
Mega Hospital
CC-BY-NC-SA-4.0
Short Answer
9709
4692
Istanbul Medipol University,
3D-RAD
Judgment
100170
23472
Mega Hospital
CC-BY-NC-SA-4.0
Multiple Choice
26316
5746
Slice-driven Datasets
ROCOv2
Caption
18663
PMC-OA
Apache-2.0
Caption
113142
PuhMedVision
PMC-0A
Apache-2.0
Dialogue
112649
RadFig-VQA
Multiple Choice
46696
2084
PMC-OA
CC-BY-NC-SA-4.0
OmniMedVQA
Multiple Choice
14230
1579
Part of 73 datasets
CC BY & Apache-2.0
LLaVA-Med
Dialogue
10622
PMC-15M
Apache-2.0
MEDPIX-ClinQA
Caption
3895
MEDPIX 2.0
Apache-2.0
Dialogue
3895
Short Answer
1040
61
VQA-RAD
MEDPIX
CCO
Judgment
1248
96
Short VQA
2598
234
Medical Segmentation Decathlon, ChestXray- NIHCC, CHAOS
SLAKE
CC-BY-4.0
Judgment
2280
194
24

---

Published as a conference paper at ICLR 2026
Table 9;
Owen2.5-VL
Lingshu
HealthGPT
MedVLM-RI
GPT-S-mini
lungs
18.99
66.25
71.25
heart
69.76
65.85
5.39
77.07
68.29
61.95
66.83
76.47
80.39
liver
76.95
70.17
10.20
36.44
75.25
62.71
68.17
75.59
82.31
spleen
67.21
kidneys
72.48
75.19
9.34
80.62
65. 12
59. 30
60.47
78.29
82.10
pancrcas
72,00
66.00
8.08
$4.00
71.00
62.00
66.00
75.00
78.79
bowel
67.84
70.27
9.76
69.46
72.16
63.51
62.43
71.08
83.47
esophagus
58.33
58.33
8.57
58.33
69.44
50,00
52.78
63,89
80.00
vessels
63.88
70.98
7.57
76.47
69.80
$4.51
65.10
75.20
80.71
spine
66.67
70.00
others
72.25
78.75
9.27
85.25
76,00
73,00
76.50
72.25
86.47
GIR
67.75
72.16
13.95
75.41
66.36
64.97
67.75
72.56
70:70
All
75.62
09.71
9.73
76.76
65.90
62.22
59.05
69.90
86.83
CRD
69.14
71.11
6.95
78.33
74.67
60.23
67.82
79.15
81.78
Average
75.75
62.10
65.20
71.52
Table MedEval--TT-Bench--1
Mudel
MiniCPM-V.4.5
CTCHHA M3D-LaMMd.-Pi----a
M.DD-LaMMe--LJaa222777 GPT---miii
Ours
lungs
52.07
47.66
$2.07
54.27
62.26
53.72
67.22
heart
$4.45
59.16
51.83
58.64
63.87
54.45
86.91
55.39
spleen
$1.88
45.94
64.38
76.88
74.06
48.44
71.25
kidneys
47.87
41.60
$5.89
71.68
80.20
44.11
78.20
pancrens
41.38
40.52
56.47
76.72
70.69
43.53
75.00
stomach
36.92
40.65
62.62
68.22
71.50
$2.80
64.02
esphhagus
44.68
46.81
48.94
40.43
76.60
29.79
63.83
trachea
61.41
65.27
84.57
68.81
82.96
64.31
89.39
spine
45.98
52.87
77.01
67.82
72.41
$5.17
83.91
athers
51.63
48.12
61.15
68.42
$2.63
81.45
GIR
45.83
39.34
46.45
73.98
80.8
47.22
77.82
MAI
61.89
64.32
76.34
66.75
70.08
51.15
86.06
All
46.40
45.29
63.10
67.26
73.28
$1.06
77.18
CRD
49.47
48.00
56.01
63.92
64.45
58.12
72.78
Averago
49.72
31.17
Table 11: Ablation of MoE Hybrid Projection.
Training Strategy
Perf. 2D
Perf. 3D
Avg.
SigLip M3D-CLIP (wlo MHP) SigLip Siglip (w/ MHP)
34.57
30.58 48.61
32.58
55.30
51.96
Table 12: Ablation analysis of adaptive organ-level feature aggregation.
ROI Strategy
Perf. 2D
Perf.
Avg.
OSE
78.68
62.17
70.43
OSE w/ native ROI
78.37
62..44
70.31
OSE w/ random ROI
80.13
64.22
72.18
OSE w/ adaptive ROI
81.45
66.
73.80
Table 13: Ablation of the OSE aggregation ratios for 2D and 3D tokens. m&p and MyD denote the numbers of aggregated organ-level tokens for 2D slices and 3D volumes, respectively.
Perf. 2D
Perf. 3D
Avg.
0
78.68
62.17
70.43
40
80.66
63.81
72.24
90
$1.45
66.15
73.80
144
160
81.23
66.04
73.64
225
250
80.64
65.48
73.06
25

---

Published as a conference paper at ICLR 2026
Table 14: Organ-level accuracy on SoilEval-CT-Beeee under normal, image question mismatch, and noise settings
Organ
Normal
Mismatch
2D Noise
Normal
Mismatch
Noise
lungs
74.7
23.1
19.5
67.2
29.4
27.3
heart
80.4
24.0
14.9
86.9
31.8
20.1
liver
82.3
19.9
20.9
80.2
30.5
25.8
spicen
68.3
13.7
17.4
71.3
28.4
29.0
kidneys
82.1
25.1
21.1
78.2
22.3
22.6
pancreas
78.8
24.9
15.2
75.0
30.2
23.4
stomach
75.0
21.7
14.4
64.0
19.0
22.7
bowel
83.5
22.2
19.4
80.0
30.1
27.9
esophagus
80.0
20.3
18.0
63.8
28.6
29.2
trachea
73.1
18.0
20.6
89.4
21.8
24.1
vessels
80.7
23.9
19.8
84.5
30.1
28.6
spine
83.1
28.9
17.1
others
86.5
22.9
14.6
81.5
29.7
27.0
Table 15: Ablation study of unified representation gains. Compared 2D-only and 3D-only train- ing, mixed training consistently improves performance across all data scales while preserving the same 2D/3D ratio.
Training Strategy
Ratio
SLAKE
VQA-RAD
RadFig VQA
M3D-VQA
CT-RATE
3D-RAD
2D-Only
25%
70.6
62.5
77.2
3D-Only
25%
65.8
84.5
65.4
Mixed
25%
72.2
62.5
78.2
69.9
84,9
66.9
2D-Only
50%
75.8
66.7
79.4
3D-Only
50%
72.7
86.2
67.2
Mixed
50%
70.8
79.8
87.1
68.1
2D-Only
100%
81.0
72.3
78.2
3D-Only
100%
74.4
86.6
68.6
Mixed
100%
81.2
71.8
81.9
74.7
86.6
69.3
Table 16: Performance of 18 types of anomaly label prediction.
Model
Precision
Recall
RadFM
13.1
6.4
7.2
M3D-LaMed-78
8.1
2.5
3.5
M3D-LaMed-4B
16.5
8.4
9.6
CT-CHAT
24.3
38.8
27.2
CT2Rep
41.6
38.1
36.7
CT-AGRG
37.8
55.4
42.1
OmniCT
41.7
36.5
36.3
26


Reviews:
[{"review_id": "flUVootBvB", "reviewer": "Reviewer_RrqG", "summary": "This paper introduces OmniCT, a unified Large Vision-Language Model (LVLM) for CT interpretation that jointly models 2D slice-based and 3D volume-based inputs within one framework.\nThe key motivation is that current medical LVLMs are fragmented:\n- Slice-driven LVLMs (e.g., Med-VLM, LLaVA-Med) have good 2D generalization but lack spatial consistency across slices.\n- Volume-driven LVLMs (e.g., CT-CHAT, M3D-LaMed) capture 3D structures but are coarse-grained and not compatible with slice inputs.\n\nOmniCT aims to bridge this gap through three main innovations:\n1. Spatial Consistency Enhancement (SCE):\nIntroduces Volumetric Slice Composition, Tri-axial Positional Encoding, and a Mixture-of-Experts (MoE) hybrid projection to unify 2D and 3D visual tokens. This allows both slices and volumes to share a common latent space while maintaining volumetric spatial consistency.\n2. Organ-level Semantic Enhancement (OSE):\nInjects anatomical priors via organ segmentation and region-of-interest masking, followed by adaptive feature aggregation to emphasize small organs and reduce redundancy in large ones.\n3. MedEval-CT Benchmark:\nA large-scale dataset (1.7M samples) and evaluation suite unifying 2D and 3D CT tasks with multi-level metrics and a standardized evaluation toolkit.", "questions": "1. Methodological distinction:\nHow does the proposed SCE differ from Med-2E3’s dynamic cross-slice fusion or Med3DInsight’s encoder alignment? It will be great if there is an explicit mathematical or architectural comparison.\n2. Organ-level supervision:\nPurely curious. The OSE module depends on pre-computed segmentation masks (TotalSegmentor). Does this supervision introduce label bias? How does OmniCT perform if segmentation is noisy or missing? Is the aggregated token more than enough for fusion. Wondering if the performance will enhance while the aggregated tokens has a longer context length for concatenation?\n3. Cross-modality transfer:\nThe paper claims OmniCT trained on 2D can generalize to 3D tasks. Is this because of the tri-axial positional encoding or MoE routing? It will be great to provide an ablation to separate their contributions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is strongly motivated and addresses a real, clinically relevant gap in current medical LVLMs: \n1. Strong and clinically grounded motivation and \n2. Unified slice–volume modeling framework\n\nIt proposes a coherent architectural design (SCE + OSE + MoE projection) that enables shared representation learning between slice-driven and volume-driven inputs. the fragmentation between 2D slice-based and 3D volume-based CT understanding. Its unified slice–volume framework is conceptually coherent and technically well-designed, combining Spatial Consistency Enhancement (SCE), Organ-level Semantic Enhancement (OSE), and a Mixture-of-Experts hybrid projection to jointly model local detail and global volumetric context. The integration of tri-axial positional encoding and anatomy-guided ROI alignment contributes to more spatially consistent and semantically rich representations. Empirically, the work demonstrates high-quality experimentation and reporting, including detailed ablations, multi-task evaluations, and architecture analysis that isolate the effects of each proposed module. Finally, the paper is well-written and logically structured.", "weaknesses": "1. Conceptual novelty and differentiation:\nThe “unified slice-volume” design is valuable but not fundamentally new, similar goals have been pursued by Med-2E3, Med3DInsight, and hybrid 2D/3D LVLMs. Can the authors clarify more clearly articulate what OmniCT does differently (e.g., tri-axial positional encoding vs. cross-slice attention in Med-2E3, why tri-axial positional encoding is important? Is there any ablation for this?).\n\n2. Architectural complexity vs. simplicity:\nIn the MoE routing, only 2 expert model is used for 2D and 3D accordingly. However, it is difficult for us whether the 3D expert is really learning the 3D representations from tokens, same as 2D. Can you clarify more on how to well distinguish these two experts are learning distinctive representation to each other?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iYryK4NRzN", "reviewer": "Reviewer_xwuZ", "summary": "The authors propose OmniCT and MedEval-CT. OmniCT is a new architecture, aimed at allowing to combine slice and volumetric information in vision-encoders, allowing their LVLM to be applied to both, 3D and 2D data. Additionally, they introduce Spatial Consistency Enhancement (SCE) and Organ-level semantic enhancement (SCE).\nRegarding the MedEval-CT Dataset and Benchmark, they unify the existing 2D and 3D VLM benchmarks and unify their evaluation.", "questions": "Q1: MedEval-CT-Dataset: Where does this data come from? 1.7 million samples means what in Volumes/Slices?\nQ2: MedEval-CT-Bench: Why is a joint benchmark necessary? Why not just use the existing slice-wise benchmarks and the existing 3D benchmarks?\nQ3: Did the authors evaluate how LLMs are able to solve the VQA (in particular the multiple-choice) questions? There is work that shows that question design alone can leak information on the correct answers. \nQ4: Did you evaluate quantitative metrics? I.e. CT-RATE report generation has a RadBERT classifier to yield abnormality labels, which would circumvent the potential VQA question leakage.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The paper exceeds their baselines across many benchmarks, yielding performance improvements and pushing the state-of-the-art of medical image understanding. \nMoreover they introduce a way to leverage both, 2D and 3D medical images.", "weaknesses": "My main concern with the paper two-fold: \n\nFirstly and most-importantly, the motivation of the paper is unclear to me. The authors motivation originates largely from the fact that there exist 2D and 3D CT medical image datasets, however by default all CT images are 3D. Subsequently, i don't know (and the authors don't motivate well) why 2D slices are needed to be integrated. How would one get the 2D slice selection? If a professional is in the loop, why would I need the LVLM? This question puts the authors method as well as the unification of their benchmark in question.\n\nSecondly, clarity. The paper is squeezing a lot of content into the 9 pages, which leads to a lot of important context missing or being deferred somewhere else. E.g. details on the large dataset the authors propose (MedEval-CT-Dataset) are hard to comeby. Maybe it's a collection of all already publicly available datasets, maybe it's new data? All they provide is 1.7 million 2D slices. This glossing over important details is prevalent in the paper.\n \n\nOther minor concerns are:\n1. The idea of leveraging 2D encoders and packing slices into RGB channels is not novel with an old example being [1] and joint training existing as well in [2].\n2. The idea of leveraging organ-masks to aggregate embeddings is not new either, as it was done by fVLM before already [3].\n\n[1] Draelos, Rachel Lea, et al. \"Machine-learning-based multiple abnormality prediction with large-scale chest computed tomography volumes.\" Medical image analysis 67 (2021): 101857.\n[2] Xie, Yutong, et al. \"Unimiss: Universal medical self-supervised learning via breaking dimensionality barrier.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n[3] Shui, Zhongyi, et al. \"Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding.\" The Thirteenth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j3OFA2gRLl", "reviewer": "Reviewer_GKxu", "summary": "The authors propose OmniCT, a unified LVLM that handles both slice-level and volume-level CT understanding. This is achieved through two key modules: Spatial Consistency Enhancement (SCE) and Organ-level Semantic Enhancement (OSE). SCE comprises volumetric slice composition, tri-axial positional embeddings, and a slice/volume MoE hybrid projection. OSE utilizes organ masks (generated using TotalSegmentator) to localize and analyze 117 anatomical structures. Subsequently, adaptive token aggregation and fusion with global tokens are employed. Additionally, the authors introduce MedEval-CT, a comprehensive dataset, benchmark, and toolkit suite for CT. MedEval-CT boasts approximately 1.7 million slice/volume VQA samples across seven distinct task types, for organ- and task-balanced evaluation. Extensive experiments were conducted across various public 2D and 3D CT benchmarks, including SLAKE, VQA-RAD, OmniMedVQA, RadFig-VQA, M3D, CT-RATE, and 3D-RAD. The results demonstrate that OmniCT (3B/7B) outperforms prior general and medical LVLMs.", "questions": "- Could you provide a clear breakdown of MedEval-CT’s data sources, licenses, and how overlap with public evaluation benchmarks (e.g., SLAKE, VQA-RAD, CT-RATE) is prevented? Details on your deduplication or leakage-checking strategy would greatly strengthen the dataset’s credibility.\n\n- Could you report sensitivity analyses on the token unshuffle parameter m and its computational trade-offs? How does varying m impact memory usage, inference latency, and accuracy for 2D vs 3D settings?\n\n- Since OSE relies on TotalSegmentator masks, how robust is OmniCT to imperfect or noisy organ segmentations? Have you tested performance degradation under synthetic noise or partial-mask conditions?\n\n- The results suggest 2D encoders outperform 3D ones for volumetric reasoning. Could you add or discuss comparisons against stronger native-3D models (e.g., VideoMAE-3D, Uni3D-ViT) with matched token budgets to solidify this claim?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed SCE/OSE design directly addresses the gap between slice-driven detail sensitivity and volume-driven spatial reasoning, which are squarely motivated by clinical reading patterns.\n\n- OmniCT shows consistent improvements over some models across diverse 2D and 3D CT benchmarks, with well-documented ablations supporting the claims.\n\n- Table 1 (SCE/OSE ablation) and studies on mixed-data training, encoder choices (2D vs 3D), and organ/task-level heatmaps add useful insight and support the design claims. \n\n- Volumetric Slice Composition + Tri-Axial Positional Embedding adds volumetric awareness while keeping slice compatibility. The MoE Hybrid Projection is a pragmatic way to route slice vs volume tokens.", "weaknesses": "- While MedEval-CT is positioned as “largest” (~1.7M) and “holistic,” the data sources, licensing, and deduplication across training vs evaluation (and vs existing public benchmarks) are not detailed enough to rule out contamination/leakage.\n\n- The pipeline leans on large models (Qwen2.5-VL-72B, Qwen3-237B-A3B) for selection/mapping/refinement. This raises bias propagation questions and potential circularity if similar model families are used in training/eval. \n\n- The 2D-encoder-dominant findings are interesting, but the native-3D comparisons seem limited (e.g., M3D-CLIP vs DINOv3/SigLIP). Stronger baselines (recent 3D ViTs / 3D CLIP variants with tuning matched to your token budgets) would make the '2D encoders suffice' claim more convincing.\n\n- The presentation of the work, particularly the figures, could be enhanced by using more aesthetically pleasing boxes or eliminating the shadows to improve readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)