Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models
Authors: Chengyu Fang
Heng Guo
Zheng Jiang
Chunming He
Xiu Li
Minfeng Xu
Venue: ICLR 2026 Poster (2026)
Keywords: 3D Medical Image Analysis
Medical VQA
Medical VLM
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Photon is a variable-length 3D medical VQA framework with instruction-conditioned token scheduling and surrogate gradients, achieving adaptive acceleration and state-of-the-art performance.

Abstract: Multimodal large language models are promising for clinical visual question answering tasks, but scaling to 3D imaging is hindered by high computational costs. Prior methods often rely on 2D slices or fixed-length token compression, disrupting volumetric continuity and obscuring subtle findings. We present Photon, a framework that represents 3D medical volumes with token sequences of variable length.  Photon introduces instruction-conditioned token scheduling and surrogate gradient propagation to adaptively reduce tokens during both training and inference, which lowers computational cost while mitigating the attention dilution caused by redundant tokens. It incorporates a custom backpropagation rule with gradient restoration to enable differentiable optimization despite discrete token drop. To stabilize token compression and ensure reliable use of visual evidence, Photon further applies regularization objectives that mitigate language-only bias and improve reliability. Experiments on diverse medical visual question answering tasks show that Photon achieves state-of-the-art accuracy while reducing resource usage and accelerating both training and inference.

Published as a conference paper at ICLR 2026
PHOTON: SPEEDUP VOLUME UNDERSTANDING WITH EFFICIENT MULTIMODAL LARGE LANGUAGE MODELS
Chengyu Fang 1.2.a Heng G3
Zheng Jiang
Chunming He
Xiu Li
Minfeng Xu
Tsinghu University
DAMO Academy, Alibaba Group
Hupan Lab 4Duke University
ABSTRACT
Multimodal large language models promising for clinical visual question an- swering tasks, but scaling to 3D imaging is hindered by high computational costs. Prior methods often rely on 2D slices fixed-length token compression, disrupt- ing volumetric continuity and obscuring subtle findings. We present Photon framework that represents 3D medical volumes with token sequences of variable length. Photon introduces instruction-conditoon gradient propagation to adaptively reduce tokens during both training and inference, which lowers computational cost while mitigating the attention dilution caused by redundant tokens. It incorporates a custom backpropagation rule with gradient restoration to enable differentiable optimization despite discrete token drop. To stabilize token compression and ensure reliable use visual evidence, Photon fur- ther applies regularization objectives that mitigate language-only bias and improve reliability. Experiments on diverse medical visual question answering tasks show that Photon achieves state-of the-art-rrt accuracy while reducing resource usage and accelerating both training and inference.
token scheduling and surrogate
DOM
PHicOs DRINN ARS Hebicc
Orend SHU-A00: Tun
20 Volma
Undorstancing
Variable longth Visual
650
Lower GPU Memory Cost
Photon
Lawr nwrreable Token Scheclliin
21.5H
5.4x Speed Up
LTACC
ASUUU
AJOOUDE
Accelerating 3D Rediology intelligence.
Accelerata in Training Presee
Free
ExAcO
GIQLEU
2.8H
1.5H
1.9X Speed
GanucE
MID
Photor-3
Accelerate in Inference Phose
Bette Results
Deep Tumor//A
Better Results 3D-RAD
Figure 1: Photon is 3D-native framework that adaptively models medical volumes using variable- length tokens, accelerating both training and inference. It enables efficient clinical question answering by removing instruction-irred irrelevant tokens while maintaining better quantitative performance.
'Equal Contribution, Corresponding Author. This work done during Chengyu Fang and Zheng Jiang's internship at DAMO Academy, Alibaba Group. Email: chengyufang, thu gmail.com
Photon: the quantum particle of light nature and the fundamental component of X-rays in CT. Its speed, controllable energy deposition, and discrete interactions for precise measurement align well with idea.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent advances in multimodal large language models (MLLMs) open new directions for clinical question answering. Despite these advances, modeling entire 3D volumes remains challenging because of heavy memory and computational demands. To reduce cost, many systems adopt slice- based pipelines or select limited set of frames from CT MRI scans (Xu al., 2025; Chen et al., 2024a; Sellergren et al., 2025; Li et al., 2023a). Although such strategies shorten input sequences, they disrupt spatial continuity, eliminate volumetric detail, and introduce human bias in frame selection.
To preserve volumetric fidelity without manual frame selection, reducing visual tokens has become a promising strategy. In the general domain, approaches such as VisionZip (Yang et al., 2025d) and LLaVA-PruMerge (Shang et al., 2024) prune or merge tokens directly based on attention or similarity, achieving a speed-up in the inference time. However, these methods uniformly apply saliency signals without instruction awareness, making them prone to discarding subtle but clinically important patterns. Moreover, most employ fixed pruning ratios, limiting flexibility in instruction-conditin tasks. ATP-LLaVA (Ye et al., 2025) introduces learnable thresholds for adaptive pruning, but still retains soft masks during training, so computation and memory are not reduced until inference.
In the medical domain, recent studies have advanced MLLMs toward volumetric modeling by aligning 3D vision encoders with large language models (Wu et al., 2023; Bai et al., 2024; Hamamci et al., 2024b; Blankemeier et al., 2024). These models support joint reasoning over full 3D volumes and alleviate some limitations of 2D slice-based inputs. However, they often produce visual embeddings in few and fixed length, limiting high-resolution detail preservation and spatial interpretability.
To address these limitations, we propose Photon, a native 3D MILM framework to model 3D medical volumes with variable-length token sequences, preserving volumetric fidelity without slice sampling or fixed-length token compression. Photon integrates two complementary components: Instruction- conditioned Token Scheduling (ITS) and Surrogate Gradient Propagation (SGP). ITS adaptively reduces visual tokens by estimating instruction-wwar saliency and predicting instance-specific thresh- olds. SGP enables effective training by restoring gradients to retention probabilities, ensuring that token selection remains differentiable and instruction-
In the forward pass, Photon removes unimportant tokens together with their caches and positional encodings to lower computation. In the backward pass, straight-through surrogate mask restores gradients to the underlying retention probabilities, allowing optimization to reflect instruction-driven token importance. To further en- hance stability, Photon incorporates lightweight regularization objectives that mitigate language-only hallucination bias and improve reliability.
The main contributions of this work are summarized as follows:
(1) We present Photon, a medical multimodal large language model that directly operates on 3D scans and produces variable-length token sequences, preserving critical volumetric details without slice sampling or fixed token compression ratio.
(2) We introduce instruction-condiiioned Token Scheduling (ITS), which leverages instruction self- affinity and instruction-vins
interactions to adaptively regulate visual token retention.
(3) We propose Surrogate Gradient Propagation (SGP), a mechanism that combines discrete token reduction in the forward pass with surrogate gradient optimization in the backward pass, enabling differentiable training despite discrete token dropping.
(4) Photon significantly accelerates both training and inference compared to general-purpose MILMM, while achieving state-of-the-an performance on multiple medical visual question answering tasks.
2 RELATED WORK
Medical Vision-Language Models. Early medical VL.Ms, such as BiomedGPT (Luo et al., 2023) and CT2Rep (Hamamci et al., 2024a), primarily adopted CLIP-style alignment between 2D slices and text, but lacked scalability and capability of interactive reasoning. More recent MLLM models have also been adapted to the medical tasks, inspiring systems such as HealthGPT (Lin et al., 2025a), Med-R1 (Lai et al., 2025), and others (Li et al., 2023a: Sellergren et al., 2025; Pan et al., 2025). Most of these methods retain the Alice-based paradigm, which undermines volumetric context and risks spatial discontinuity. To enhance spatial understanding, volumetric MILMSs (Lee al., 2024) such
2

---

Published as a conference paper at ICLR 2026
Netting
Phone
Phaned::
MTT
Veen Qweed. LUN
Pacct Eis.
With Phison LLM
Visaan Eine.
Tince Laven
3D Eirodd
Vision Enc.
Photon LLM
Decoster
Rubb Peep
Pla Rlepi
Tolertzer
Suxsd perss...
* Photon
Figure 2: Photon's pipeline: Phase 1 aligns the visual embedding layer, and Phase finetunes all modules for task adaptation, learning token reduction threshold estimation through our backpropaga- tion strategy. Modules in right upper with black contour are not updated during training.
as RadFM (Wu et al., 2023) and M3D (Bai et al., 2024) align 3D vision encoders with language models, compress each scan into a fixed-length sequence of vision tokens. The sequence length typically small due to memory constraints, limiting their scalability to high-resolution inputs and weakening fine-grained spatial detail. OmniV-Med (Jiang et al., 2025) instead supports variable- length sequences, but its pruning strategy relies on L1 similarity of slice features, coarse criterion that can inadvertently remove subtle, yet clinically important pathologies.
Visual Token Reduction. To lower computational cost in large vision-language models, many works compress the visual representation at the vision-language interface or within vision encoder layers. Representative designs include Q-Former (Li et al., 2023b), clustering-based merging (Shang et al., 2024), and pooling modules (Li et al., 2024b; Yao et al., 2024), as well as internal pruning that removes less informative tokens inside the model (Chen al., 2024b). Although these approaches reduce token redundancy and improve efficiency, they ignore that different instructions require different amounts of information. In medical imaging, diverse instructions often focus on distinct anatomical regions represented by visual token sequences of varying lengths. Therefore, there is pressing need for adaptive token reduction strategies tailored to computationally intensive scenarios. IVTP (Huang et al., 2024), TransPrune (Li et al., 2025a) and some pioneering works (Han et al., 2024; Wang et al., 2024b; Xing et al., 2024; Zhang et al., 2025a;e; Dhouib al., 2025) sort the visual tokens by heuristic methods (Yang et al., 2025d; Alvar al., 2025; Zhang et al., 2025d; Yang et al., 2025b; Sun et al., 2025) to enhance the relevance between visual token reduction and instructions. These methods account for instruction- viiionn correlation but restricted fixed retention ratios or manually set thresholds, which risks discarding vital pathological cues in medical imaging. ATP-LLaVA (Ye al., 2025) introduces learnable thresholds enable adaptive pruning but applies pruning only at inference, while additional computational cost is still required during training.
3 METHODOLOGY
Photon is multimodal framework that integrates a 3D vision encoder with large language model to jointly process volumetric seans and clinical instructions (see Fig. 2). It directly models entire medical volumes by partitioning them into 3D patches and mapping each patch to token. The token sequence length is dynamically adjusted to balance spatial detail preservation against computational cost while mitigating the attention dilution caused by redundant tokens. In the forward computation, visual tokens are adaptively reduced according to their estimated importance and the predicted threshold derived from the visual summaries. In the backward computation, discrete selection remains differentiable through a straight straight-through! through path with masked gradient routing aligned to the active tokens, ensuring the same selection logic is honored in both training and inference.
3

---

Published as a conference paper at ICLR 2026
3.1 INPUT PROCESSING AND 3D VISION ENCODER
Given an input scan of size D x H x W, the volume is divided into non-overlapping 14 X 14 x 14 patches. Each patch is linearly embedded into a d-dimensional vector. To balance resolution and sequence length, spatial merging is applied only along (H, W) with stride S, while the depth D remains unchanged. The compact grid contains visual tokens with a length of:
N D. S:
(I)
where H' and W are the in-plane sizes after patchification. Let (1...., N₂) be the visual-token index set and Q = ....... N be the instruction-token index set. Our 3D vision encoder (ViT-based) V. and encodes these tokens into visual embeddings T in the
then handles the full visual-token same hidden dimension of Tixt that is encoded by a language tokenizer. Photon's LLM decoder part takes as input the following hybrid tokens:
T [To
via: TO J.
with positional indices following Qwen2.5-VL. VL (Bai et al., 2025). Tokens are processed by self- attention and feed-forward blocks. To improve efficiency under instructions, Surrogate Gradient Propagation (SGP) is applied at selected layer & E la of Photon's 's language modules.
3.2 INSTRUCTION-CONDITIONE TOKEN SCHEDULING
Token-level saliency scores provide a relative ranking of visual tokens within each sample, but do not specify how many tokens should be retained in total. A fixed retention ratio would treat all seans and instructions equally, ignoring differences complexity and clinical focus implied by instructions. To address this, we introduce Instruction-conddtioned Token Scheduling (ITS), mechanism that adapts the pruning threshold to each sample by conditioning on both the instruction and the token-level distribution. ITS integrates the Instruction-aware Saliency Estimation (ISE) rule with an Instance-aware Threshold Predictor (ITP), ensuring that token retention adapts to instruction demands while maintaining visual compactness.
Instruction-aware Saliency Estimation. To estimate the saliency of visual tokens under instruction guidance, we introduce ISE selection layer la which explicitly links token saliency to variations in the given instructions.
Let ge denote the query vector of the I-th instruction token, by the key vector of the 1'tth instruction token (t, t' E Q). and a the dimensions of the attention head. To prevent trivial self-matching, we exclude the diagonal (t' and define the centrality score as:
Σ F'EQ
(gt, ke) 0),
max
which measures the extent to which token is positively reinforced other instruction tokens, while ignoring negative contributions. The centralities are then normalized into weights:
Ct
Σ w. = EEQ
W'
Ereecr
where the tokens with consistently strong positive affinities obtain larger wy, thereby highlighting salient instruction tokens that contribute more to the results. Let k, denote the key vector of the j-th (j E V) visual token, we compute the weighted instruction-we
saliency score by:
(ge,kg) Kg)) Va
Pj norm(u,),
M Σ MEQ
Wi
Pj E [0,1],u, i-00, +00
(5)
where norm(-) denotes instance-wist min-max normalization. how strongly each visual token aligns with the most central instruction tokens, provides relative importance ranking that serves as the basis for adaptive thresholding.
The saliency score pj. reflecting
Instance-aware Thresholding Prediction. While saliency scores provide token-wise rankings, they do not determine the retention level per instance. A fixed keep ratio ignores variability scan complexity and instruction demands.
4

---

Published as a conference paper at ICLR 2026
To adapt retention, Photon introduces ITP to predict an instance-specific threshold conditioned on both the instruction and the distribution of visual saliency scores. According to the results of ISE, {u, jev is the raw logits and p {pjbie is the normalized scores. We construct feature vector from three complementary views:
[P()), p(u), T[
(6)
where V, ②. T summarize distributional shape, absolute scale, and compressed tails, respectively (see Appendix A.1. We combine these form a compact descriptor 2, A lightweight predictor then maps to a scalar threshold:
( (W1= + b,) + b2),
E [0, 1},
where (W1. b1) and (W2, b2) denote the weight- pairs of the hidden and output layers, Φ the GELU activation, and is the logistic function. The resulting 0 serves as smooth probabilistic threshold, which can directly compared with the normalized saliency py=
4
4 E 10.
where the qj denotes the retention probability and Tce is sufficiently small coefficient to make the continuous mask approximate to binary. Based on this sigmoid result, we split to get the token retention binary mask with:
Mj ifg; q3 > 0.5),
Tokens with M, = are hard dropped and removed entirely, including their cache, attention mask, and positional embedding. To enable differentiabilityy we replace M with a straight-through surrogate:
M sg(M) sg( + 9,
(10)
where sg(.) denotes the stop- hoopggradient operator, and q = { Jiev. Thus the forward path follows hard selection, while the backward path propagates gradients through probabilities. The surrogate gradients that govern the update dynamics are detailed in the subsequent forward-backward analysis.
3.3 SURROGATE GRADIENT PROPAGATION
Instructiion-condiiionee Token Scheduling (ITS) produces the surrogate mask M that determines which tokens are retained in the forward sequence. Surrogate Gradient Propagation (SGP) addresses the training challenge that arises from this discrete selection by restoring gradients to the retained tokens and by providing surrogate gradients for the retention probabilities derived from the threshold predictor. Specifically, the upstream gradient of the compressed sequence is scattered back to the original tokens as:
Oc =
-s( Oc OT.
M)
(11)
σT vis
where S(:; M) scatters gradients from the compressed sequence T vis back to their original positions in T This scattered path keeps decoder activations trainable for retained tokens. However, the threshold predictor receives useful optimization signal, since the hard selection operation blocks gradients with respect to its probabilities. To address this, we design a surrogate gradient on qj so that the predictor can be optimized under token pruning.
With token-level gradients available, we define an importance score that measures the task-driven contribution of each token. Salieney scores β5 and retention probabilities qj characterize relevance or selection tendency, but they do not reflect the effect of tokens on the loss. We therefore adopt a first-order approximation, where the variation in loss induced by removing token is estimated by the inner product between its activation and gradient (detailed in Appendix A.1.4):
n = ((TThs))
BC (TT.
(12)
Here ny reflects the contribution of token to the loss, with larger values indicating stronger influence. These scores are standardized and clipped:
n 1-1p(00 ② M)
2j max(Std(n ② M), Edd) 23 clip(e), -c, c)2
(13)
5

---

Published as a conference paper at ICLR 2026
where yu(-) and (td(.) denote the masked mean and standard deviation, eau prevents division by small variance, and bounds outliers. The standardized values are rescaled into (0, 1) through a smooth monotonic mapping (-)). The directional term is then defined as:
r = ()()),
d, = 0.5 sg(r,),
(14)
where (0) = 0.5 serves as the neutral point, so that d, quantifies the deviation from this midpoint. In parallel, a magnitude factor is computed by accumulating activation-gradies products across the feature dimension:
m
Σ (T (ivi)) 9jkl k=1
9j
T
ac
(15)
where 5t and Ojk are the k-tth components of (Ivis) and gy. These values are normalized within each row and clamped to a bounded interval:
m,
Sj clip
Mkrow (m M) Smins Smax
(16)
which yiclds scale factor that amplifies consistently strong activation cetivation-gradeent interactions while suppressing weak ones. Finally, combining direction and magnitude provides the surrogate gradient with respect to the relaxed probabilities:
Oc
= Bdj 8, max (1-x3) -;); Eeath.
qz),
(17)
dqj
where 3 balances directional and magnitude contributions, Esot provides lower bound on the gradient term (1 - 9j) to avoid vanishing when qj is close to 0 or 1, thereby ensuring stable updates. As result, tokens assessed as more informative are gradually reinforced toward retention, whereas less informative tokens are progressively suppressed, which guarantees that the pruning mechanism behaves as intended (see detailed analysis in Appendix A.1.3).
3.4
OBJECTIVE
Soft Retention Band. While forward-backwart learning already propagates supervision to guide pruning, unconstrained training may converge to degenerate cases keeping nearly all or pruning excessively. To avoid such extremes, we constraint the retention probabilities 9j by softly constraining their mean ratio within a band:
Ladd [[maa((,r Tmax) + max Tmin - -r)],
Σ 9)
(18)
JEV
This band-constrained penalty softly restricts the retention ratio to stable range, ensuring pruning remains effective without overriding the dynamics of the nawaard-bckweer process.
Robustness Regularizer. Another failure mode in multimodal learning is language-onf, hallucination bias, where answers rely on textual priors while ignoring the image. This risk is critical in medical tasks, since predictions may appear correct without visual grounding. To address it, the model is encouraged to produce higher predictive uncertainty whenever visual evidence insufficient. Given perturbed volume T (by masking, shuffling, etc.), the robustness loss is defined as:
Lrohust (H(o((()))))
H(p) = --Σn-vi p(v) log p(v).
(19)
During these perturbed cases, we drop other loss functions and optimize only Probust- This discour- ages overconfident predictions without valid visual grounding.
Flip Regularizer. To further reduce reliance on spurious pruning patterns, we introduce a mask-flip regularizer. With certain probability, the retention mask is inverted so that tokens originally reduced are retained and those originally retained are reduced. Under this adversarial mask, the model should not remain highly confident the correct answer; otherwise, it indicates over-reliance textual priors or shortcut cues. We penalize such cases by assigning a higher loss whenever the model still predicts the correct label with high confidence:
log 1 Pa(y x, M. + +
Laip - Exgrp
(20)

---

Published as a conference paper at ICLR 2026
Table Comparison of zero-shot and finetuned performance across six tasks in 3D-RAD benchmark. violet and indigo indicate the best and the second best. Stat. Temp. Diag. Static Temporal Diagnosis, Longit. Temp. Diag. Longitudinal Temporal Diagnosis.
Zero-shot
Finetuned
Tusk
Metrie
Qwen2.5-VL
RadFM
M3D-L2
M3D-P3
OuniV
Lingshu
M3D-L2
MSD-P3
3B
13B
7B
4B
1.58
7B
7B
4B
3B
Existence Detection
Accurncy
19.52
29.20
18.00
40.25
28.66
59.60
81.09
82.43
83.07
Stat. Temp Diag.
Accuracy
0.00
44.11
25.47
25.40
22.96
6.02
$1.20
49.30
$2.86
Longit. Temp. Ding.
Accuracy
0.29
42.99
24.17
24.31
24.23
12.13
74.17
77.01
BLEU
1.78
15.95
2.55
2.52
33.52
Medical Measurement
ROUGE
4.30
6.62
23.24
5.63
7.88
5.45
36,06
36.46
39.36
BERT Score
84.20
86.85
91.50
85.74
85.66
83.81
94.65
94,86
96.14
BLEU
3.51
13.48
10.69
16.31
16.42
5.34
31.28
39.66
$1.59
Image Observation
ROUGE
23.19
50.52
BERT Score
84.53
87.16
86,61
86.92
88.29
85.67
90.00
92.19
93.62
BLEU
2,93
11.00
9.10
15.06
13:47
3.71
25.25
33.28
42.33
Anomaly Detection
ROUGE
9.17
17.62
18.64
23.19
25.72
9:46
33.76
42.45
47.50
87.11
where (y v, "Ip) is the probability of the ground-truth label under the flipped mask and is small constant for numerical stability. This regularizer discourages the model from producing correct answers under inconsistent retention, thereby reinforcing reliance on the intended pruning structure.
Overall Objective. In training Phase 1, we only use the cross-entropy loss to supervise the alignment. In training Phase 2, the complete training objective combines faithful language modeling, stable token usage, robustness against language-only hallucination bias, and consistency under mask flipping:
= (CCE or Liand or Crobust) + Laiip
LCE = [Es,,) plog (y)
(21)
This innovative design ensures that Photon learns to generate clinically accurate text, maintains efficiency through adaptive token pruning, and grounds its reasoning volumetric evidence rather than spurious textual patterns.
4 EXPERIMENTS
All comparison baselines are initialized from models that have been pretrained or finetuned on large- scale medical datasets and then further finetuned following their respective protocols. In contrast, before task-specific finetuning, our method applies only lightweight alignment of the 3D vision patch embedding layer using volume-caption pairs while keeping the VIT backbone, MLP aligner, and LLM decoder frozen. Details of the training process (Appendix A.1.I), experimental setup (Appendix 1,2.1), and dataset information (Appendix A.2.2) are provided in the appendix.
4.1 COMPARATIVE EVALUATION
On 3D-RAD (Gai et al., 2025), Photon achieves the best finetuned results across all tasks in Table 1. The most notable gains appear in anomaly detection and image observation, where performance improves by about 14.0% compared with the best baseline. Medical measurement also shows a clear improvement of about 7.3%, which is critical since measurement accuracy directly underpins many downstream diagnostic decisions. Longitudinal temporal diagnosis rises by around 3.0%. These results highlight Photon's ability to retain volumetric details and adjust token selection based on instruction relevance, thereby enhancing the accuracy of both descriptive and quantitative clinical predictions. The consistent improvements across descriptive (e-g., anomaly detection, observation), quantitative (c.g., measurement), and reasoning oriented (c.g., temporal diagnosis) tasks indicate that Photon generalizes beyond a single evaluation type.
cessoning-oriented
On DeepTumorVQA (Chen al., 2025c), Photon also obtains the highest overall accuracy in both multi-choice and free-text settings in Table 2. The total average in multi-choice increases by about 3.6%, while in free-text the improvement is larger approximately 11.5%. The strongest gains occur measurement subtypes evaluated with MRA, where Photon improves by more than 35.3%, confirming its strength in handling numerical and quantitative queries. Visual reasoning subtypes,
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance on DeepTumorVQQ benchmark. Subtypes marked with indicate free- text numerical answers evaluated using MRA, higher is better. Meas. = Measurement, Recog. Recognition, Vis. Rsn. Visual Reasoning, Med. Rsn. = Medical Reasoning.
Multi-choire
Free-fest
Type Subtype
Rand Medis
MSD-12
M1D-P3
CT-CHAT
RuFFM
Photon
Mertin
M3D-122
M3D-P3
CT-CHAT
RaIFM
Phowon
7B
7B
4B
7B
130
3B
70
7B
48
7B
13B
3B
Mros
Iesion volume messuremen"
0.250
0.253
0,815
0.825
0,833
0,815
018225
0,079
0.085
0.079
0,875
0.112
0.265
organ HE mewarement
0.250
0.254
0.638
0.640
0.637
0.647
0.642
0.487
0.490
0.49]
0.513
0.488
0.777
volume messureeen
0.250
0.262
0.747
0.754
0.750
0.755
0.756
0.525
0.535
0.525
0.540
0.583
0.717
Average
0.250
0.256
0.733
0.740
0.740
0.739
0.741
0,361
0.370
0.366
0.379
0.434
0.587
colon lesion esistence
0.500
0.899
0,859
0.859
0.839
0.856
0.858
0.859
0.559
0.859
0.859
0.893
0.581
0.500
0.797
0.797
0.861
0.910
0.757
0.797
0.797
0.864
0917
Recog.
lesion esistence
0.500
0.510
0.501
0.717
0.515
0.400
0,402
0/64
0.500
0.574
0.574
0.886
0.574
0.574
0.574
0.805
liver lesion existence
0.524
0.517
0.524
0.652
0.521
0.524
0.524
0.524
0.662
0.610
pancreatic Jesion ctistence
0,500
0218
0.718
0718
0.800
0.505
0.718
0.18
0.718
0.718
0371
0.819
Average
0,500
0.661
0.664
0.661
0.664
0.789
0.811
0.664
0.665
0.660
0.663
0.812
0.796
adjacent orgin
0.333
0.217
0.565
0.609
0.000
0.609
0.696
0.174
0.174
0.304
0.304
0.435
0.304
inter-egment comporison
0.333
0.470
0.567
0.576
0.573
3.591
0.549
0.577
0.561
0.592
5.589
0446
0.359
kičney volume comparison
0.333
0.347
0.370
0.364
0.372
0.386
0,370
0,350
0.370
0.356
0.370
0.286
0.396
Iarges lovinn iitomation
0.353
0.317
0.541
0.539
0.544
0.555
0.539
0.536
0.544
D:548
0.542
0.521
0.571
largest lesion diameter'
0.250
0.263
0.778
0.783
0.781
0.766
0.776
0.162
0.209
0.213
0.269
0.232
0.310
largest lesion location
0.392
0.307
0.310
0.310
0.340
0.340
0.356
0.399
0.353
0.337
0.353
0.384
0.350
largest lesion slicee
0.250
0.241
0,672
0.684
0,672
0.564
0.700
0.524
0.535
0,510
0.513
0.672
0.725
Vix.
lasion connt location'
0.250
0.583
0,861
0.60
1
0.961
0
0.534
0.534
0.534
0.534
0.506
0.546
lesion counting"
0.328
0.455
0.781
0.784
0.796
0.750
0.985
0.000
0.000
0.000
0.000
0.001
0.917
lesion cotlier
0.500
0.521
0.507
0,549
0.451
0,483
0.555
0.451
0.535
0.535
0.577
0.521
0.634
liver lesion clustering
0,333
0.331
0,438
0.475
0.409
0.475
0.388
0.469
0,469
0.431
0.313
0.456
oegan aggregation"
0.250
0.560
0.667
0.569
0.586
0.574
0.621
0.798
organ enlargement
0.736
0.736
0.746
0.736
0,736
0.736
0.759
orgin HU diffienence
0.836
0.122
0.189
0.307
Average
0.335
0.616
0.627
0.625
0.653
0.408
0.420
9.428
0:439
0.530
lutty liver
0.333
0.318
01461
0.455
0,481
0.558
0.481
0.481
0.396
9.487
0.578
0.73
Med. Ren.
lesion type classification
0.500
0.865
0.865
0865
0.8K5
0.965
0.865
0.865
0.665
0.865
0.565
0.511
0.665
pancreatic cysa resectability
0,500
0.371
0.657
0.800
0,800
0.771
0.771
0,800
0.000
0,800
0.300
0.771
0.743
puncreatic lesion resectability
0.333
0.379
0,653
0.483
0,483
0.683
0,483
0,414
0.833
0:683
0.483
0.483
0.433
pancreatie steatosis
0.580
0.526
0.526
0.513
0,513
0.539
0.526
0.526
0.526
0.526
0.526
0.658
0.671
punercatic ramor staging
0.250
0.216
0.351
0.343
0.160
0.334
0.440
0.216
0.36
0.207
0.135
0.432
0-460
Average
0.403
0.465
0,557
0.560
0,555
0.584
0,611
0,550
0.562
0
0.549
0.629
0.666
Total Average
0.369
0.440
0,626
0.632
0.628
0,662
0.686
0.478
0.489
0,493
0.497
0.555
0.619
such as lesion counting and slice localization, also show large margins exceeding 20.7%, reflecting better spatial understanding. These large improvements in both measurement and reasoning tasks suggest that Photon is not only effective for recognition but also excels in tasks that require quantitative precision and spatial analysis. Taken together, the results from both 3D-RAD and DeepTumon V AQA confirm that Photon delivers consistent and meaningful benefits in multiple tasks, strengthening its reliability as a versatile medical vision- Janiuae system.
4.1.1 COMPARATIVE EVALUATION WITH TOKEN PRUNING METHODS
Most existing pruning methods for multimodal large language models do not provide case-specific adaptive retention based on rellaed visual content. Instead, they typically rely on fixed retention ratios. To fairly compare with such methods, we follow their configurations and evaluate them at three nominal retention levels (about 30%, 50%, and 70% of visual tokens), which correspond to keeping roughly 2.1K, 3.5K, and 4.9K vision tokens per sample in our 3D-RAD setup.
It is also important note that these methods were not designed to optimize training- acceleration, backpropagation stability, or overall training robustness. Several of them do not provide a complete training recipe. In the 3D medical setting, where the base model has limited prior knowledge and relies heavily on visual evidence, directly training with such pruning strategies can lead to premature removal of important tokens and collapse towards language-only bias. To avoid this issue and keep the comparison fair, we first train full Qwen2.5-VL VL model without pruning using the paradigms recommended in the respective papers. and then apply the pruning methods only at inference time using their official code. This isolates the effect of pruning at prediction time and avoids confounding factors from unstable training-
Most previous pruning methods, including VisionZip (Yang et al., 2025d) and HiPrune (Liu et al., 2025b), are not optimized for compatibility with FlashAutention in the visual encoder or require extra attention computation across all visual encoder layers. This leads to additional overhead and erodes the expected speed gains. In our setting, which has a large number of visual tokens and relatively short textual answers, these methods do not provide clear advantages in inference speed, HiPrune (Liu et al., 2025b) in particular slows down decoding despite keeping fewer tokens. By contrast, Photon couples dornaciio-cooditionnn pruning with FlashAttention-friee friendll implementation and a pruning
8

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of different token pruning strategies on 3D-RAD. E.D. Existence Detection,
S.T.D. = Static Temporal Diagnosis, L.T.D. = Longitudinal Temporal Diagnosis.
Methods
E.D.
S.T.D.
L.T.D
Medical Measurement
Image Obervvatio
Anomaly Detection
Infer Speed
Token Num
Aec.
Apc.
Acc.
BL.FU
ROUGE
RERT
BLFF
ROUGE
BERT
REFU
ROUGE
RERT
(Toki))
(Tokieme)
Q-
81.97
47.62
75.
37,00
38.57
96.04
52.72
56.48
93.63
42.01
47.35
91.93
2.30
7,0K
Visioniaa
82.00
47.39
75:42
37.04
38.09
96.00
52.51
56.36
95.63
41.95
47.39
91.93
2322
2.1K
VisionZip
81.99
47.74
75.93
37.11
38.68
96.07
52.72
$6.60
93.65
41.98
47.35
91.92
2.23
3.5K
VisianZip
82.01
47.40
37.07
52.57
42.01
47.39
91.93
4,9K
HiPrune
81.99
48.08
75.50
37.00
38.59
52.56
56.46
42.08
47.32
91.93
2.1K
$1.97
47.81
75.58
52.65
41.92
47.31
3.5K
HiPrune
82.02
47.19
52.57
47.40
4.9K
Phonos
83,017
77.01
37.74
42.53
47.50
4.12
Dynamic
Photon MAX
Dynamic
Question: Where is the pleurat effusion more prominent?
Quastion: Before now, long spacity showadl [1. 0, 1, 1) (Notw:: D absence,
presence. Lesron behawier from CT. How does lung opacity sppcar now compared to its sequence history?
Answer: Right sidle
Groundi-oe truth: Right sicle.
Timeline history
A. Refractory Lesion (Persistent or recurrent, now present)
Question: least cyets the kidney?
Resalved Lasion (Pravioualy present
alseen))
C. New Lesion (Absent previousily, now present)
Answer: Yee
Abnormallty (Ahways absent)
Ground truuth:
Question: What is the mean HU valuus of the pancress? A: 13.6 B: 16.7 C: 16 9.9
Rafrectory Lesion (Paraistent
recument, now present)
13.6
Ground brthh Refractory Lesion
Ground-truths A: 15.6
(Persiaent recurentt preseet))
Figure 3: Visualization Photon's results. White regions indicate reduced tokens, while purple boxes highlight retained areas that carry clinically relevant information for answering the questions.
schedule that is used during training. As shown in Table 3, Photon achieves higher accuracy all three 3D-RAD tasks and reaches up to 4. 4.12 tokens/s, improving both performance and inference efficiency under the same backbone.
4.2 VISUALIZATION ANALYSIS
Fig. 3 and Fig. 7 show qualitative results of Photon and its token reduction behavior. The examples demonstrate that the model is able to discard large portions non-essential context while preserving subtle but diagnostically important structures. For instance, when asked about pleural effusion, Photon selectively retains regions around the thoracic cavity, while in kidney cystic-related questions, it keeps the cystic kidney area. These cases indicate that pruning decisions adapt to the clinical focus of each question rather than being uniform. The visual results combined with the results from Table show that our token scheduling process reinforces the contribution of retained tokens, ensuring that relevant regions dominate the final prediction. As a result, Photon maintains high accuracy in various medical visual question answering tasks while reducing redundancy in visual tokens.
4.3 ABLATION STUDY
As shown in Table 4, Photon Phase 1, which finetunes only the modified 3D visual embedding layer, yields clear gains over the zero-shot initialization of Qwen2.5-VI. (Bai et al., 2025): BLEU improves by 70% on anomaly detection, 134% on image observation, and existence detection accuracy rises by 44%. These results indicate that re-aligning the adapted 3D patch embedding provides a stronger initialization for subsequent finetuning. To test whether adapting the full vision stack is advantageous, we continued training from the Phase checkpoint. However, even with a small learning rate, fully finetuning the vision encoder and MLP aligner (Vis. Ful. Ft.) led overfitting and degraded performance. As illustrated in Fig. 4, the model also lost instruction-folloiii following capability and often produced answers inconsistent with the given questions.

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study on 3D-RAD. Computational statistics include training speed (iterations/s), average retaining tokens. and inference speed (tokens/s). inference peak GPU memory. E.D. Existence Detection, S.T.D. = Static Temporal Diagnosis, L.T.D. Longitudinal Temporal Diagnosis.
Tiak
R.R.
S.T.n.
I.T.
Medical Meacurement
Image Observation
Anumaly Detection
Computational Matric
Act.
BLBL
RODGE BERT
BLEU
ROUTG
BLEU
ROLGE
DCRI
Tean
Trin lnk. Inter Spd
Inver Men.
Willen volitee Siniioe
Qwee2.5- VLL
19:52
0.00
0.29
1.78
4,30
$4.20
3.51
5.84
$4.53
2.95
9.17
$4.47
Vis, Ful.
0.00
9.01
94.52
90.43
Plonus Plose
28.10
8:41
10.90
4.81
9.00
85.35
5.95
17.00
$6.39
6.84
15.64
85.95
Alewmond vitk 3D dosturte
Q
81.97
47.62
75.36
38.57
96.04
52.72
91.93
7,00K
2600BB
wil Photce Plaae
$2.35
52.20
75.69
39.00
40.55
96.11
51.23
50.3 27
99.55
42.13
47.46
91.92
0.54
0.45%
4.09
0.2GB8
$2.09
47.21
75.43
40.29
41.59
96.34
52.18
50, 16
09.06
42.02
41.03
91.39
0.92
0.44
4.03
9.2GB
E- Eap
82.35
50.93
16.62
38,06
39:45
96.14
51.94
42,05
91.39
0.54
047X
9.2018
$2.97
82.71
73.85
37.01
18.51
96.06
52.11
56.20 26
09.64
41.90
47.03
91:57
0.85
0.19%
9.20B
I
wit Setiorliiiy
$3.34
57.87
77.89
37.41
18.75
96,05
51.78
55.59 54
93.67
41.97
47.46
41.41
0.5
0.5KK
4.08
9.20B
win Robrs
61.81
45.15
77.38
39.25
95.97
51.70
93.59
42.05
91:95
0.57
0.79%
9.2GB
wilk Fip Reg.
52.20
52.09
77.09
36.97
38.24
95.96
51.79
36.61
99.64
42,02
47.50
91.92
0.55
0.36%
4.12
9.2GB8
Plonas
$3.07
52.85
77.00
37.74
39.36
96.14
51.59
$6.66
93.02
42.33
47.50
91.95
0.55
0.39%
4.12
9.20i8
Paotss
$5.50
57.79
79.00
40.96
96.17
52.83
58.72
42.85
92.00
3.088
21,40B
Question: What device observed on the chest woll?
Ground-truth Pacemaker.
Qwen2 Qwen2.6-VLL S-VL: stethorcape is abserved on the chest wall in the image. the Photon Phase 1: device observed on the chest wall, which appears to be pacemaker.
Ful. diemitee mitxilic
Quastion: Which organ showss staatossis
Ground-truth: Liver.
steatcsis, which characterize by presence Vis. Ful. Pt.: Traches and both main branchi am open. No occlusov pathology was...
Qwan
Phase shows stestosis.
Figure 4: Visualization Results of base model and different visual alignment methods. Our method achieves alignment while avoiding mode collapse. Vis. Ful. Ft. Visual Modules Fully Finetuned.
In the task-related finetuning stage, compared with other chosen layers, Photon achieves the best trade- off by placing the module at the é = bn/4. attaining higher accuracy while maintaining efficiency. The importance of the core components is evident, removing ITS and SGP lowers static temporal diagnosis accuracy by more than 3%, increases the number of retained tokens, and slows training, confirming their necessity for both performance and efficiency. Regularization is also critical, since removing these terms not only reduces accuracy but also destabilizes training and undermines result reliability (Appendix A.2.3). Overall, Photon improves accuracy while substantially reducing computation. Compared with finetuned Qwen2.5-VV VL (Bai al., 2025), it consistently achieves better performance while reducing inference memory usage by about two-thirds and achieving over a five-fold speedup. Gains in inference speed are more modest due to the use of KV Cache, but remain steady.
Beyond the main Photon design, we offer an alternative option called Photon Max. It reuses the original visual patch embedding layer of Qwen2.5-VL (Bai et al., 2025) with our components, improving existence detection and static temporal diagnosis by 2.4% and 4.9%, respectively, while training about 25% faster than Qwen2.5-VL. with lower memory usage, thus serving more accuracy-oriented variant that remains more efficient than the original baseline.
5
CONCLUSION
We present Photon, an efficient MLLM framework for 3D medical volume understanding. Photon directly operates on 3D scans and integrates Instruction-conditionee Token Scheduling (ITS) with Surrogate Gradient Propagation (SGP) to achieve adaptive and consistent token reduction during both training and inference. ITS discretely selects informative tokens in the forward path and predicts adaptive sample-secificc thresholds from instruction- vision interactions. SGP ensures effective gradi- ent back-prok proppaaapoa in discrete token dropping scenario. Through this innovative design, Photon reduces computational and memory costs while preserving clinically critical information. Experi- ments on various medical visual question answering benchmarks demonstrate that Photon achieves state-of-the- performance with substantially fewer tokens and less computational overhead.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity- based visual token pruning for large multimodal models. In CVPR, pp. 9392-9401, 2025.
Fan Bai, Yuxin Du, Tiejun Huang, Max Q-H Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models. arXiv preprint arXiv:2404.00578, 2024. 3
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
4, 9, 10, 16
Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Merlin: A vision language foundation model for 3d computed tomography. Research Square, pp. Is-3, 2024. 2
Delbrouck, Eduardo Reis, Cesar Truyts, al.
Chubin Chen, Sujie Hu, Jiashu Zhu, Meiqi Wu, Jintao Chen, Yanxun Li, Nisha Huang, Chengyu Fang, Jiahong Wu, Xiangxiang Chu, et al. Taming preference mode collapse via directional decoupling alignment in diffusion reinforcement learning. arXiv preprint arXiv:2512.24146 2025a. 24
Chubin Chen, Jiashu Zhu, Xiaokun Feng, et al. -²-guidance: Stochastic self guidance for training- free enhancement of diffusion models. arXiv preprint arXiv:2508. /2880, 2025b. 24
Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, al. Towards injecting medical visual knowledge into multimodal Ilms at scale. EMNLP, pp. 7346-7370, 2024a.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In ECCV, pp. 19-35. Springer, 2024b.
Yixiong Chen, Wenjie Xiao, Pedro RAS Bassi, Xinze Zhou, Sezgin Er, Ibrahim Ethem Hamamci, Zongwei Zhou, and Alan Yuille. Are vision language models ready for clinical diagmosis? 3d medical benchmark for tumor-centric visual question answering. arXiv preprint arX1152505.78115. 2025c. 7, 18, 20, 21, 24
Tri Dao. FlashAttention-2:: Faster attention with better parallelism and work partitioning. In ICLR, 2024. 17
Mohamed Dhouib. Davide Buscaldi, Sonia Vanier, and Aymen Shabou. Pact: Pruning and clustering- based token reduction for faster visual language models. In CVPR, pp. 14582-14592, 2025.
Chengyu Fang, Chumming He, Fengyang Xiao, Yulun Zhang, Longxiang Tang, Yuelin Zhang, Kai Li, and Xiu Li. Real-world image dehazing with coherence-bases pseudo labeling and cooperative unfolding network. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.7 24
Chengyu Fang, Chunming He, Longxiang Tang. Yuelin Zhang, Chenyang Zhu, Yuqi Shen, Chubin Chen, Guoxia Xu, and Xiu Li. Integrating extra modality helps segmentor find camouflaged objects well. urXiv preprint arXiv:2502. 14471, 2025.
Xiaotang Gai, Jiaxiang Liu, Yichen Li, Zijie Meng, Jian Wu, and Zuozhu Liu. 3d-rad: A compre- hensive 3d radiology med-vqa dataset with multi-temporal analysis and diverse diagnostic tasks. arXiv preprint arXiv:2506.1114 1/177, 2025. 7, 18, 21, 23.
Ibrahim Ethem Hamamci, Sezgin Er, and Bjoern Menze. Ct2rep: Automated radiology report generation for 3d medical imaging. In MICCAI, pp. 476-486. Springer, 2024a. 2
Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Doga, Omer Faruk Durugol, Weicheng Dai, Murong Xu, et al. Developing generalist foundation models from a multimodal dataset for 3d computed tomography. arXiv preprint arXiv:2403. 17834, 2024b. 2, 18, 24
11

---

Published as a conference paper at ICLR 2026
Kai Han, Jianyuan Guo, Yehui Tang, Wei He, Enhua Wu, and Yunhe Wang. Free video-llm: Prompt- guided visual perception for efficient training- free video llms. arXiv preprint arXiv:2410.10441, 2024. 3
Chunming He, Rihan Zhang, Fengyang Xiao, Chenyu Fang, Longxiang Tang, and Yulun Zhang. Run: Reversible unfolding network for concealed object segmentation. ICML, 2025. 24
Kai Huang, Hao Zou, Ye Xi, BoChen Wang, Zhen Xie, and Liang Yu. Ivtp: Instruction-guide visual token pruning for large vision-language models. In European Conference Computer Vision, pp. 214-230. Springer, 2024. 3
Songtao Jiang, Yuan Wang, Sibo Song, Yan Zhang, Zijie Meng, Bohan Lei, Jian Wu, Jimeng Sun, and Zuozhu Liu. Omniv-med: Scaling medical vision-language model for universal visual understanding. arXiv preprint arXiv:2504. J4692, 2025.
Zheng Jiang, Heng Guo, Chengyu Fang, Changchen Xiao, Xinyang Hu, Lifeng Sun, and Minfeng Xu. Medvr: Annotation-free medical visual reasoning via agentic reinforcement learning. In The Fourteenth International Conference on Learning Representations, 2026. 24
Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-rl: Reinforcement learning for generalizable medical reasoning in vision-language models, arXiv preprint urXiv:2503.13933 2025. 2
Changsun Lee, Sangjoon Park, Cheong-Il II Shin, Woo Hee Choi, Hyun Jeong Park, Jeong Eun Lee, and Jong Chul Ye. Read like a radiologist: Efficient vision-language model for 3d medical imaging interpretation. arXiv preprint arXiv:2412.13555 2024.
Ao Li, Yuxiang Duan, Jinghui Zhang, Congbo Ma, Yutong Xie, Gustavo Carneiro, Mohammad Yaqub, and Hu Wang. Transprune: Token transition pruning for efficient large vision- lisoon-language model. arXiv preprint arXiv:2507.2066 2025a. 3
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama. Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and-visst andivision assistant for biomedicine in one day. In NeurlPS, volume 36, pp.
2023a. 2
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML. pp. 19730-19742. PMLR, 2023b. 3
Wenxuan Li, Chongyu Qu, Xiaoxi Chen, Pedro RAS Bassi, Yijia Shi, Yuxiang Lai, Qian Yu, Huimin Xue, Yixiong Chen, Xiaorui Lin, et al. Abdomenatlas: A large-scale, detailed-annotatee & multi-center dataset for efficient transfer learning and open algorithmic benchmarking. Medical Image Analysis, 97:103, 2024a. 18, 20, 24
Xingyuan Li, Yang Zou, Jinyuan Liu, Zhiying Jiang, Long Ma, Xin Fan, and Risheng Liu. From text to pixels: A context-aware semantic synergy solution for infrared and visible image fusion. arXiv preprint arXiv:2401.004211 2023c.
Xingyuan Li, Zirui Wang, Yang Zou, Zhixin Chen, Jun Ma, Zhiying Jiang, Long Ma, and Jinyuan Liu. Difiisr: A diffuusion model with gradient guidance for infrared image super-resolution. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7534-7544, 2025b. 24
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV, pp. 323-340. Springer, 2024b.
Tianwei Lin, Wenqiao Zhang. Sijing Li, Yugian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li. Xiaohui Song, et al. Healthgpt: A medical large vision-language model for unifying comprchension and generation via heterogeneous knowledge adaptation. arXiv preprint 0wX2::2502.098388 2025a.
Yunlong Lin, Zixu Lin, Kunjie Lin, Jinbin Bai, Panwang Pan, Chenxin Li. Haoyu Chen, Zhongdao Wang, Xinghao Ding, Wenbo Li, et al. Jarvisart: Liberating human artistic creativity via an intelligent photo retouching agent. arXiv preprint arXiv:2506. /76/2, 2025b. 24
12

---

Published as a conference paper at ICLR 2026
Yunlong Lin, Linqing Wang, Kunjie Lin, Zixu Lin, Kaixiong Gong. Wenbo Li, Bin Lin, Zhenxi Li, Shiyi Zhang, Yuyang Peng, et al. Jarvisevo: Towards a self-evolving photo editing agent with synergistic editor-evaluator optimization. arXiv preprint arXiv:2511.25002 2025c. 24
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang. and Xiao-Ming Wu. Slake: A semantically- weweeeeeeennaeeedd
labeled knowledge enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pp. 1650-1654. IEEE, 2021. 18,21
Che Liu, Zheng Jiang, Chengyu Fang, Heng Guo, Yan-Jie Zhou, Jiaqi Qu, Le Lu, and Minfeng Xu. M3ret: Unleashing zero-shot multimodal medical image retrieval via self-supervision. arXiv preprint arXiv:2509.00 200220099900000 01060, 2025a. 24
Jizhihui Liu, Feiyi Du, Guangdao Zhu, Niu Lian, Jun Li, and Bin Chen. Hiprune: Training- free visual token pruning via hierarchical attention in vision- sainual models. arXiv preprint arXiv:2508. 00553, 2025b. 8
Tianyi Liu, Haochuan Jiang, and Kaizhu Huang. Kmd: Koopman multi-modalig decomposition for generalized brain tumor segmentation under incomplete modalities. In Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 15663-15671. 2025c. 24
Tianyi Liu, Zhaorui Tan, Muyin Chen, Xi Yang, Haochuan Jiang, and Kaizhu Huang. Medmap: Promoting incomplete multi-modal brain tumor segmentation with alignment. IEEE Journal of Biomedical and Health Informatics, 2025d. 24
Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv preprint arXiv:2308, 09444, 2023." 2
Yue Ma, Zexuan Yan, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Boung-Yeung Shum, et al. Follow-your-emoj-fattee Towards efficient, fine-controllable, and expressive freestyle portrait animation. arXiv preprint arXiv:2509.1600 2025. 24
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.0644 1.06400, 2016. 17
Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvlm-rl: Incentivizing medical reasoning capability of vision- language models (vlms) via reinforcement learning. In MICCAI, 2025. 2
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 3505-3506, 2020. 17
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, al. Medgemma technical report. arXiv preprint arXiv:2507.0222 2025. 2
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. plave-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403. /5388, 2024. 3
Boyuan Sun, Jiaxing Zhao, Xihan Wei, and Qibin Hou. Llava-scissor: Token compression with semantic connected components for video 1lms. arXiv preprint 22XX::2200..2862,, 2025.
Guankun Wang, Long Bai, Junyi Wang, Kun Yuan, Zhen Li, Tianxu Jiang, Xiting He, Jinlin Wu, Zhen Chen, Zhen Lei, et al. Endochat: Grounded multimodal large language model for endoscopic surgery. Medical Image Analysis, pp. 103789, 2025a. 24
Guankun Wang, Han Xiao, Renrui Zhang, Huxin Gao, Long Bai, Xiaoxiao Yang, Zhen Li, Hongsheng Li, and Hongliang Ren. Copesd: A multi-leve surgical motion dataset for training large vision- language models to co- co-pilot endoscopic submucosal dissection. In Proceedings of the 33rd ACM International Conference Multimedia, pp. 12636- 12643, 2025b. 24
13

---

Published as a conference paper at ICLR 2026
Hongqiu Wang. Guang Yang, Shichen Zhang. Jing Qin, Yike Guo, Bo Xu, Yueming Jin, and Lei Zhu. Video- inteo-instuument synergistic network for referring video instrument segmentation in robotic surgery. IEEE Transactions on Medical Imaging, 2024a. 24
Hongqiu Wang, Wu Chen, Xiangde Luo, Zhaohu Xing, Lihao Liu, Jing Qin, Shaozhi Wu, and Lei Zhu. Toward fair and accurate cross-domain medical image segmentation: A vlm-driven active domain adaptation paradigm. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14102-24113 2025c. 24
Jiangshan Wang, Zeqiang Lai, Jiarui Chen, Jiayi Guo, Hang Guo, Xiu Li, Xiangyu Yue, and Chunchao Guo. Elastic diffusion transformer. arXiv preprini arXiv:2602. 139933 2026a. 24
Jiangshan Wang, Kang Zhao, Jiayi Guo, Jiayu Wang, Hang Guo, Chenyang Zhu, Xiangyu Yue, and Xiu Li. Precisecache: Precise feature eaching for efficient and high-fidelity video generation. In The Fourleenth International Conference Learning Representutions, 2026b. 24
Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao. and Liqiang Nie. Retake: Reducing temporal and knowledge redundancy for long video understanding. arXiv preprint arXiv:2412.20504 2024b.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang. and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. arXiv preprint arXiv:2308. 02463, 2023. 2.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9777-9786, 2021. 18, 22
Long Xing, Qidong Huang, Xiaoyi Dong. Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-langue langualanguage models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410. 1711:24400.12277, 2024. 3
Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506. 07044, 2025. 2
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint wrXi:22505.09388 2025a. 21
Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Chendi Li, Jinghua Yan, Yu Bai, Ponnuswamy Sadayappan, Xia Hu, et al. Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model. CVPR, pp. 19803- 19813, 2025b. 3
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In CVPR, pp. 10643, 2025c.
18
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In CVPR, pp. 19792- 19802, 2025d. 2, 3,8
Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou. Deco: Decoupling token compression from semantic abstraction in multimodal large language models. arXiv preprint arXiv:2405.20998 2024. 3
Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. Atp-llava: Adaptive token pruning for large vision language models. In CVPR, pp. 11972-24982. 2025. 2,
Suhao Yu, Haojin Wang, Juncheng Wu, Cihang Xie, and Yuyin Zhou. Medframeqa: A -autti-mmage medical vqa benchmark for clinical reasoning. arXiv preprint arXiv:2505. /6964, 2025. 18, 20, 21
14

---

Published as a conference paper at ICLR 2026
Ce Zhang, Kaixin Ma, Tianqing Fang. Wenhao Yu, Hongming Zhang, Zhisong Zhang, Yaqi Xie, Katia Sycara, Haitao Mi, and Dong Yu. Vscan: Rethinking visual token reduction for efficient large vision-language models. arXiv preprint arXiv:2505.22654, 2025a. 3
Jiajun Zhang, Jianke Zhang, Zeyu Cui, Jiaxi Yang. Lei Zhang, Binyuan Hui, Qiang Liu, Zilei Wang, Liang Wang, and Junyang Lin. Plotcraft: Pushing the limits of Ilms for complex and interactive data visualization. arXiv preprint arXiv:2511.00010, 2025b. 24
Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. Beyond text-visual attention: Exploiting visual cues for effective 2025c.
token pruning vlms. arXiv preprint arXiv:2412.018/3
Qizhe Zhang, Mengzhen Liu, Lichen Li, Ming Lu, Yuan Zhang. Junwen Pan, Qi She, and Shanghang Zhang. Beyond attention similarity: Maximizing conditional diversity for token pruning in mllms, arXiv preprint arXiv:2506. 10967, 2025d. 3
15

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 METHODOLOGY
A.1.1 TRAINING PROCESS
Unlike most previous 3D medical MLLM methods that use a three-stage pipeline (including vision pretraining, vision language alignment, and vision language instruction finetuning), design lighter and more efficient two-stage training strategy.
In Phase 1, we perform visual embedding alignment by training the 3D vision patch embedding layer of the visual encoder on volume-radiolog. report data with a learning rate of le-a under annealing. The ViT backbone retains its pretrained weights from Qwen2.5-VL. (Bai et al., 2025), and both the backbone and the MLP aligner are kept frozen to preserve the priors learned from large-scale natural images. The Qwen2.5-VL decoder is also frozen and serves only as the alignment target. We find that fully finetuning the vision encoder tends to cause overfitting and negatively affects the decoder's 's ability to follow instructions and interpret visual inputs. By updating only the 3D vision patch embedding layer, this stage improves efficiency, maintains generalization, and avoids the need to re-pretrain the visual encoder or re-align vision and language on large-scale medical data.
In Phase 2, based on the alignment obtained in Phase 1, we replace the Qwen2.5-VI. language decoder with the Photon language decoder that integrates our ITS and SGP modules, while retaining the backbone parameters reused from Qwen2.5-VL. This design allows alignment results and the pretrained capabilities of Qwen2.5-Vi VL without restarting pretraining from scratch. In this stage, unfreeze all modules and finetune them the task-specific training set for optimization with a learning rate of 1e-5 under a cosine annealing schedule.
directly inherit the
A.1.2 DETAILS OF THRESHOLD FEATURE CONSTRUCTION AND PRUNING
We detail the three feature blocks and the subsequent thresholding and selection.
Normalization for ranking. Given raw logits d {d, }evo,, we define the min-max normalized saliency scores:
d min de Pj m d2 min dę + This rescales logits to [0, 1] and preserves within samile ordering.
j E D()
Block 1: shape descriptor from normalized scores. From p = {pj} we compute:
V [μ(p), Std(p), min(p), max(p), Q1/4(p), med(p)) R capturing central tendency, spread, range, lower quartile, and median in scale-free manner. Block scale descriptor from raw logits. From we extract absolute-scal statistics:
i (d) [n(d). Std(d), min(d), max(d)] R
which retain level and variability on the original linear scale.
Block 3: compressed descriptor from signed log view. To summarize large magnitudes while preserving sign, use:
slog(r) = sign(x) log (1 |x]),
and define:
T(slog(d) [p(slog(d)), max(slog(d)) e IR?
This provides compressed indicators of overall level and dominant peaks.
A.1.3 GRADIENT DYNAMICS OF THE SURROGATE
We provide formal analysis of the surrogate gradient mechanism described in the main text. The token-level retention probability is defined as:
0-8
q
16

---

Published as a conference paper at ICLR 2026
where P denotes the normalized saliency score and 0 is the instance- instance-specific threshold predicted by ITP. The surrogate gradient with respect to q takes the form:
Oc dq
3 ss max(g(1 - 9), Esat),
d= 0.5 sg(r), r= (0)) 0.5,
where d is the directional term measuring deviation from the neutral point 0.5, 8 is the magnitude factor obtained from activation -rradient interactions, 3 is balancing coefficient, and the saturation term max - q), Eeat) } prevents vanishing gradients near the extremes, As in the main text, all updates restricted to visual positions, so that only active tokens contribute.
The partial derivatives of g are:
dg
dq
p-0 Tce
O,
1. Tee
o'
0,
00
Tce
0p
Tee
which leads to:
0C
0C
0q
ac
0C 0g
00
dq
00
0p
0g
0p
When the directional term satisfies d < O, we obtain Bg O. Consequently,
0c > 0 and ac 00
Op so
gradient descent decreases and increases p. This enlarges the margin p - 0, raises q, and thereby promotes token retention.
Bp 0, so gradient descent increases and decreases This reduces the margin p - 0, lowers and thereby increases the tendency toward pruning.
Conversely, when > O, we obtain ac > this case, 50 < and BC 24
Overall, the surrogate gradient dynamics remain consistent with the intended behavior: tokens assessed informative reinforced toward retention, while less informative tokens receive weaker gradients and are progressively pruned.
A.1.4 DERIVATION TOKEN IMPORTANCE VIA FIRST-ORDER APPROXIMATION
Let Th E RLd denote the visual hidden states at decoder layer f. Removing token ý is modeled as setting its hidden state (ie)) to zero while keeping all other tokens unchanged. Applying first-order Taylor expansion the training loss L around vis gives
C(TV vis (T)) ((T ((T
(OTE ),)
so that the predicted loss variation caused by discarding token j is
AC
((T
ac
7T
This matches the Taylor-based importance criterion proposed for filter and neuron pruning (Molchanov al., 2016). We therefore adopt the activation -agaiiigggaiinnn product token-level importance signal and define
((T
ac JT
"
This extends the classical Taylor criterion to multimodal token reduction with differentiable masking, and the resulting scores are standardized and clipped across active tokens for stable optimization.
A.2 EXPERIMENTS
A.2.1 EXPERIMENTAL SETUP
In our experiments, training was performed on 16 NVIDIA H20 GPUs with BF16 precision with Deepspeed ZeRO2 (Rasley et al., 2020), and inference was conducted on 8 NVIDIA H20 GPUs. We set Tce = 0.2 and configured SGP with éxd = x 10 10-3, (Smin- Smax) = (0.5, 2.0), 3 = 0.8. Esat 5 x 10 33 Tmin and Fmax set to 0.3 and 0.8 respectively. Unless otherwise noted, ITS and SGP are applied at = En/4- Optimization uses Adam W. and FlashAttention2 (Dao, 2024) is enabled in all attention layers.
17

---

Published as a conference paper at ICLR 2026
A.2.2 TASKS, DATASETS AND METRICS
In Phase 1 of training, the visual embedding alignment process was trained on the volume-captions pairs of CT-RATE (we use 47,141 out of 50,188 pairs, reconstructed from 25,692 volumes belonging to 21,304 patients for training) (Hamamci et al., 2024b), Abdomen 3.0 (we use 8,334 out of 9,262 pairs for training) (Li et al., 2024a), and our internal dataset (we use 27,577 pairs for training). To address the potential overlap of identical volumes between Phase and Phase 2, we adopt the Phase data partitioning scheme in Phase thereby preventing data leakage.
In Phase 2 of training, we finetune all model parameters using task-specific data, We employ two large-scale 3D CT VQA benchmarks: 3D-RAD (Gai et al., 2025) and DeepTumorVOQ (Chen et al., 2025c), which together span descriptive, recognition, and reasoning-orienter diagnostic scenarios under clinically relevant protocols. We elaborate on these two datasets as follows.
3D-RAD (Gai et al., 2025) is a large-scale 3D medical VQA benchmark built upon CT- RATE (Hamamci et al., 2024b), comprising 16,188 CT scans from 11,255 patients with striet train-test separation. It covers six tasks (anomaly detection, image observation, medical measurement, existence detection, static temporal diagnosis, and longitudinal temporal diagnosis), supporting both open- and closed-ended questions. QA pairs are generated from radiology reports and multi-label annotations using task-specific templates, and refined through an LLM with human verification pipeline with over 91% factual alignment. The final dataset provides 33,910 curated benchmark QA pairs and 136,195 training pairs, enabling comprehensive evaluation of multi-task and multi-temporal reasoning in clinically realistic settings.
(Chen et al., 2025c) is built from 9,262 abdominal CT volumes. The benchmark provides 355,962 training and 39,650 testing QA pairs across measurement, recognition, visual reasoning, and medical reasoning categories, covering both multiple-choice and free-text formats. Evaluation follows task-specific protocols, including accuracy for multiple-choice, exact match for free-text answers, mean relative accuracy (MRA) (Yang et al., 2025c) for numerical prediction, and CE for region-level, clinical correctness. This design ensures reliable assessment of both reasoning capability and clinical validity.
(Liu al., 2021) is large-scale bilingual medical visual question-answerte benchmark that features richly annotated radiology images and knowledge graph support. The dataset covers multiple imaging modalities and several body regions. QA pairs span question types such as imaging plane, modality, organ detection, abnormalities, and knowledge- grage- reasoning. The benchmark supports both closed-ended and open-ended question formats and designed for evaluating visual reasoning plus medical knowledge reasoning in a multilingual setting.
SLAKE
NEXT-QA (Xiao et al., 2021) is a video question answering benchmark designed to move beyond scene description toward explaining causal and temporal action relations. It contains 5,440 real-world videos and 52,044 human-annotated QA pairs, organized into three categories: causal questions that require identifying visible cause- -fffect relations, temporal questions that probe ordering of actions, and descriptive questions covering objects, attributes and events.
MedFrameQA (Yu al., 2025) is benchmark designed for multi-image medical visual question answering that reflects more realistic clinical workflows. MedFrameQA contains 2,851 clinically grounded VQA items constructed from 9,237 high-quality frames extracted from 3,420 instructional medical videos. Each question is paired with two to five temporally coherent frames that share a continuous diagnostic focus. The dataset spans nine human body systems and 43 organs across diverse imaging modalities, forming a challenging setting that requires models to integrate complementary evidence across multiple images rather than relying on single-frame cues.
We evaluate model performance using a combination of text generation and classification metrics. For free-text tasks, we adopt BLEU, ROUGE, and BERT Score to assess fluency, lexical overlap, and semantic similarity between generated and reference answers. BLEU captures n-gram precision, ROUGE emphasizes recall-oriented overlap, and BERT Score leverages contextual embeddings for semantic alignment. For categorical predictions such as existence detection and temporal diagnosis, Accuracy is reported. For numerical free-text answers, we further apply the Mean Relative Accuracy (MRA) (Yang al., 2025c) metric, which measures correctness under range of relative error thresholds, reflecting clinical tolerance in quantitative assessment. Together, these metrics provide a comprehensive evaluation across descriptive, reasoning, and measurement-orientee tasks.
18

---

Published as a conference paper at ICLR 2026
Crost
Figure 5: Visualization of retention band triggers (Eq. (18)) and the number of kept tokens. Left: training without robust regularization, middle: training without flip regularization, right: training with both regularizations. A lower activation frequency and smaller magnitude of the retention band indicate more stable pruning during training.
reduction is Warmup Stage
Wermup Stege
Reduction Stage:
Soft reduction is pertormed.
reduction is performed.
Myw Mo الrي
Avarage retertion rata bandss stabillze
029
Figure 6: Visualization of retention rate of training time. Our token reduction method exhibits an average retention rate that fluctuates within a reasonable range before steadily stabilizing, indicating convergence toward an appropriate per-sample threshold estimation.
A.2.3 ANALYSIS OF SOFT RETENTION BAND AND REGULARIZATION
The soft retention band designed safeguard mechanism rather than regular penalty. Under normal conditions. it has little impact performance because most retention probabilities lie within the expected range and are not penalized. Its main role emerges in extreme situations, where the predictor deviates sharply from a reasonable retention ratio. In such cases, the band introduces corrective pressure that pulls the retention ratio back into stable range, preventing collapse.
Fig. 5 compares the trigger visualization of the soft retention band and the number of kept tokens under different regularization settings. The instability mainly arises from the limited visual priors of the pretrained backbone in 3D medical domains. Without sufficient prior knowledge, the predictor struggles to identify clinically relevant regions and systematically underestimates the importance of visual tokens. As shown in the left plots, in the absence of the robust regularizer this bias is amplified: the predictor tends to discard visual tokens, leading to unstable and sometimes extreme fluctuations in the retention ratio that reflect insufficient grounding in image evidence. The middle plots show that when only the robust term is applied but the flip regularizer is removed, pruning becomes smoother but the model starts overfit spurious retention patterns, which undermines stability. By contrast, the right plots demonstrate that using both regularizers produces stable retention dynamics: the robust term suppresses neglect of visual content and constrains excessive pruning, while the flip term prevents shortcut reliance on fixed masks. Together, they enforce balanced reliance on both visual and textual signals, resulting smoother training and stronger visual grounding.
A.2.4 TOKEN REDUCTION ANALYSIS
Fig. illustrates the progression of the average token retention rate during training Phase In Warmup Stage no reduction is applied and the rate remains fixed at 1.0. In Warmup Stage 2, the retention rate also stays at 1.0, but soft masking is applied to tokens so that the predictor begins learning threshold estimation, providing gradual transition between full retention and hard pruning.
19

---

Published as a conference paper at ICLR 2026
Table 5: Clinical Metries of Kidney tumor recognition (sensitivity, specificity, and accuracy (%)).
Metrie
RadFM
M3D-P3
M3D-1.2
Merlin
CT-CHAT
Photon
Photon Bootstrap
Seusitivity
75.2
0.0
0.0
50.2
0.0
81.2# 1.7
81.7 [78.6, 84.7]
Specificity
98.6
100.0
100.0
51.2
100.0
95.24 0.1
95.3 [93.8, 96.6]
Accuracy
88.6
57.4
57.4
50.7
57.4
89.3+ 0.7
89.5 (87.9, 91.1]
Table 6: Photon performance on the AbdomenAAlas3.-ssayy OOD split of DeepTumorVOA. Results are reported for multi-choice and free-text variants (accuracy (%)).
Method
Avg. keep tok.
Mcus.
Recog.
Vis. Rsn.
Med. Rsn.
Overall
Photon OOD Multi-Choice
0.34K
0.745
0.599
0.636
0.677
0.648
Photon Pree-Text
0.40K
0.459
0.599
Once entering the Reduction Stage, hard pruning is applied: the retention rate first oscillates as the predictor adapts and then gradually stabilizes. This staged schedule prevents premature pruning and enables smooth shift from probability learning discrete token removal.
A.2.5 ANALYSIS IN CLINICAL METRICS
As noted in (Chen et al., 2025c), lesion recognition poses greater challenges for MLLMs than general VQA tasks, since reliable clinical application requires balanced sensitivity, specificity, and accuracy. Table 5 highlights the diversity of model behaviors: some approaches emphasize specificity at the expense of detection, while others show higher sensitivity but limited overall stability. These differences reflect the difficulty of consistently capturing subtle lesion signals in raw 3D volumes without explicit spatial supervision.
To account for randomness, repeat the experiment with ten independent seeds on under the same protocol. We report mean + standard deviation for medical metrics. Photon exceeds the strongest baseline under every seed, which indicates that the improvements are stable with respect initialization. For the main Photon results, we also estimate test-set uncertainty with 1000 bootstrap resamples of evaluation questions and report 95 percent confidence intervals. The intervals remain narrow, showing that the observed gains are not due to sampling noise. In addition, Photon achieves a more stable balance across the three metrics. maintains competitive accuracy while improving lesion detection rates, thereby reducing the likelihood of missed cases without compromising overall reliability. This balance across sensitivity, specificity, and accuracy suggests that Photon better aligns with clinical requirements compared with prior MLLM baselines.
A.2.6 ZERO-SHOT AND OUT-OF-DOMAIN EVALUATION
In this section we study whether Photon maintains robustness and efficiency when evaluated outside its training distribution. Here focus two complementary scenarios: (1) institution-level out-of- domain (OOD) evaluation on DeepTumorrVon VQA (Chen et al., 2025c) using the AbdomenAtlas33. 1llaas.0-bbsed (Li al., 2024a) split, and (2) fully training-free zero-shot setting on MedFrameQA (Yu al., 2025).
Institution-level OOD split on DeepTumorVQA. To evaluate robustness under distribution shift, we follow the AbdomenAtlas3.0 OOD split and redivide DeepTumor into training and test sets based on medical institutions, which also induces changes population, devices, and imaging protocols. We report results on free-text and multi-choice variants. Table 6 summarizes task-wise performance and the average number retained vision tokens per sample.
Compared with the in-domain results reported in Table 2, the OOD setting leads to moderate drop of approximately 0.038 and 0.049, respectively. This indicates that Photon preserves most of its accuracy under institution-level distribution shift while still retaining only 0.34K ~ 0.40K visual tokens on average, which confirms that the pruning mechanism does not destabilize cross-domain generalization.
Training-fre zero-shot setting on MedFrameQA. To further isolate the effect pruning and threshold estimation, evaluate training-free variant of Photon where the token scheduling module
20

---

Published as a conference paper at ICLR 2026
Table 7: Zero-shot results full MedFrameQA dataset. performed (accuracy (%)).
No additional medical fine-tuning is
Method
Avg. keep lok.
CT
MRI
X-ray
Other
Ultrasound
Overall
Qwen2.5-VL. VL. 3B Zero-Shot
7,00K
0.461
0.460
0.485
0.477
0.432
0.460
Qwen2.5-VI. 3B + ITS Zero-Shor
3.28K
0.459
0.456
0.488
0.470
0.426
0.457
attached to Qwen2.5-VL. 3B without any additional medical fine-tuning. We test this configuration in strict zero-shot setting on MedFrameQA and compare it with the plain Qwen2.5-VL. 3B backbone. Results across modalities and the average token keep rate are shown in Table 7.
The training-free Photon variant reduces the average token keep rate from 7.00K to 3.28K while maintaining almost identical zero-shot accuracy. Since the token scheduling module is attached as a plug-in and operates independently from the MILLM backhone parameters, these results indicate that the pruning and threshold estimation strategy preserves the backbone's zero-shot and generalization ability, even when no spec-specific fine-tuning is applied.
A.2.7 MORE ABLATION STUDY
Hyperparameters affecting pruning. We test the three hyperparameters that most directly control pruning behavior: the temperature Toe, the surrogate gradient weight 3, and the saturation threshold 'sa- As summarized in Table 8, changing these values mainly adjusts the average number of retained tokens, while task accuracy on all three 3D-RAD subtasks and the generative metrics remains stable. This supports the claim that our default configuration is already in a robust regime and that aggressive hyperparameter tuning is unnecessary in practice.
Robustness to noisy thresholds. To probe the robustness of the threshold predictor, we perturb the predicted thresholds at inference time with random multiplicative noise in |-10%, +10%]. The noisy variant in Table 8 shows only mild performance degradation, indicating that Photon does not rely on finely tuned thresholds and is tolerant to moderate estimation errors.
Scaling across backbones. Finally, we examine cross-model scaling. On Qwen2.5-VL, extending Photon from a 3B to a 7B backbone improves performance across tasks with similar retention ratios, and attaching Photon to Qwen3- VL (Yang et al., 2025a) 2B yields consistent pruning behavior without training collapse. Together, these results show that Photon is stable under hyperparameter changes, robust to noisy thresholds, and transferable across different backbone sizes and architectures.
A.3 EXTEND PHOTON TO MORE MODALITIES
scheduling across modalities. Instruction-condiiionn Token Scheduling intensities or fixed spatial coordinates. Instead, it predicts retention ratio from a normalized alignment distribution produced by the instruction-visual cross-attention. This distribution is normalized within each sample and depends the relative ordering of token importance, rather than absolute intensities or fixed noise patterns. Therefore, ITS can adapt to changes imaging centers, scanning protocols, slice numbers, orientations, spacing, and noise levels.
(ITS) does not operate
Results on MedFrameQA (Yu et al., 2025) and SLAKE (Liu et al., 2021). To test this property, we apply Photon to MedFrameeAA, which contains diverse modalities and variable frame counts, slice numbers, and resolutions from multiple sources. We randomly split MedFrameQA (Yu et al., 2025) into training and testing sets with 9 ratio, and compare the Qwen2.5- VL 3B backbone with and without Photon. As shown in Table 9, Photon reduces the average token keep rate from 1.0 to about 0.43 while slightly improving the overall accuracy, with consistent gains on CT, MRI, and X- X-ray and no degradation ultrasound. This suggests that the same ITS configuration that works for 3D-RAD (Gai et al., 2025) and DeepTumorVQU. (Chen et al., 2025c) also remains stable on a heterogencous multimodal benchmark, supporting the view that ITS learns task-guided token importance structure instead of a dataset-specific mask pattern.
We also study transfer to MRI data using the SLAKE (Liu et al., 2021) dataset. Here focus on based VQA in SLAKE (Liu et al., 2021) and again compare the non-pruned Qwen2.5-VL 3B baseline with our pruned variant. Photon cuts the average keep rate about 0.38 while keeping
21

---

Published as a conference paper at ICLR 2026
Table 8: Ablation on key hyperparameters and robustness of Photon. Toe controls the temperature for contrastive estimation; 3 balances surrogate gradient strength: eat defines the saturation suppression threshold. The last block tests robustness under noisy threshold perturbation and cross-modes scaling.
E.D.
S.T.D.
I.T.D.
Medical Measurement
Imsgee Observation
Anounaly Detection
Setting
Arg. Train Tok.
Acc.
Acc.
Acc.
BLEU
ROUGE
BERT
BLBU
ROUGE
BERT
RI FU
ROUGE
BERT
Effert (toquertuur for contratiiv estimation)
Tw
82.44
51,66
77.66
38.65
39.87
96.20
51.11
56.77
93.61
41.95
46.91
91.86
0.2KK
m0.20
83.07
52.86
77.01
37.74
39.36
96.14
51.99
56.66
95.02
42.33
47.50
91.96
0.39K
7w0.50
82.88
52.13
71.71
38.59
39.77
96.25
51.61
56.16
95.53
41.73
46.81
91.87
0.32K
Fffect of (surmggote gradient weight).
(4005
82,99
52.25
75.57
37.05
38.45
96.04
$1.60
56.12
93.64
41.76
47.18
91.90
0,46K
(-0.8
83.07
52.86
77.01
37.74
39.36
96.14
$1.99
56.66
93.62
42.33
47.50
91.96
0.30K
8=1.0
82,70
52.86
76.63
38.51
39.70
96.11
52.08
56.79
93.72
41.99
47.52
91.93
0.4KK
Fffect of (mituration supprassion thes shold).
82.87
51.53
76.78
37.09
38.29
96.00
52.14
56.72
93.66
42.20
47.22
91.97
0.45K
83.07
52.86
77.01
37.74
39.36
96.14
$1.99
56.66
95.62
42.33
47.50
91.96
0.39K
82.83
53.27
76.40
37.67
39.10
96.20
$1.27
56.11
93.65
41.78
47.08
91.91
0.61K
Robustnesy under ihreshuld permurtation.
Wio Noise
83.07
52.86
77.01
37.74
39.36
96.14
$1.99
56.66
93.62
42,33
47,50
91.96
|- 1-10%, +10%]
82.87
52.33
76.59
37.63
39,05
96.13
$1.43
56.72
93.61
42.15
47,38
91.93
Canss-modei walling: Plutoon with difforeed hackfoos and stons.
Photon (Q. 3B)
83.07
52.86
77.01
37.74
39.36
96.14
51.59
56.66
93.62
42.33
47.50
91.96
0.39K
Photon (Q25VV 7B)
83.50
53.90
76.03
40,70
41,90
96,20
53.34
57.65
93.77
42.66
48.14
91.99
044KK
Photon (QVVI. 2R)
82,28
52.21
75.03
37.65
39.22
95.84
50.99
55.30
93.40
41.30
46.96
91.84
0.37K
Table 9: Token retention and performance on MedFrameQA and SLAKE (accuracy).
MedFrameQA
SLAKE
Method
Avg.
CT
MRI
X-ray
Other
Ultrasound
Acc.
Avg. keep
Acc.
Match / Total
Qwen2.5V. 5V
1,000
0.744
0.745
0.893
0.667
0.486
0.727
1.000
0.8445
413/489
Ours 38
0.432
0.761
0.787
0.57
0.750
0.486
0.745
0.375
0.8425
412/489
accuracy almost unchanged, with only one fewer correct case out of 489. This indicates that the pruning strategy is not tied to CTT-specific characteristics and can be applied to other medical imaging modalities.
Extension to video reasoning. Photon can also be applied to video reasoning tasks. In this setting, the base Qwen2.5-VI. VL model is already pre-trained on large-scale visual data, so we simplify the training schedule: the two warmup stages and robustness regularization used for 3D medical training are removed, and the pruning module trained directly with the main task loss.
To demonstrate this, apply Photon to the NExT-QA (Xiao et al., 2021) dataset and compare it with the non non-pruned Qwen2.5-Vi 3B baseline. Table 10 shows that Photon reduces the average token keep rate by more than half while keeping the overall accuracy essentially unchanged. Across causal, descriptive, and temporal question types, the differences remain small, and several categories show slight improvements. This confirms that ITS and SGP can be used in video understanding settings with minimal modifications, providing substantial token reduction without sacrificing performance.
Across CT, MRI, multimodal medical frames, and videos, Photon consistently cuts visual tokens by more than half while keeping accuracy comparable to non non-pruned basclines. This shows that the iestruction-condiite pruning design is not restricted to a single imaging protocol task format and can generalize across diverse modalities and benchmarks.
A.3.1 MORE VISUALIZATION RESULTS AND FAILURE CASES.
Qualitative behavior and typical failures. Fig. 7 presents additional visualization results, including both correct predictions and failure cases. Photon successfully identifies major findings such as air bronchogram signs and kidney tumors, but limitations appear in more subtle scenarios. For example, when estimating the size nodule on the minor fissure, the model underestimates the measurement (4x3 vs. ground-truth 5x3 mm), which reflects the difficulty of precise quantitative reasoning
22

---

Published as a conference paper at ICLR 2026
Table 10: Token retention and performance on NEXT-QA (accuracy).
Methed
Arg. keep
Cassal-How
Causal-Wag
Desc: Count
Desc Loc
Quasion
Desc: Other
securaay
Temp-Now
Temp-
Temp. PPeeent
Qwen2.5VL SVL 3B
1,000
0,7545
0,7738
0,6429
0,9588
0,5533 0.3483
0,7760
0.7268
0. 0,7097
0,7729
Ous 3B
0.473
0,7502
0,7693
0,6522
0,9388
0,7631
0,7455
0,7527
0.7723
sign
Question: the largest kidney quality oyst tumor? A: Tumorr B: Cyst
the lung?
Answer: Air bronchogoa signs.
Answarr: Turnor
Ground anuth: Air brenchogram signa.
Ground-tzen -rrutrrttth Tumor
lung notvestbie?
Question: faxure?
Right lung. Ground truth: Right lung
Ground truth: ES3 mm
Figure 7: More visualization results and failure cases by Photon.
on small targets. In the lung visibility case, although the prediction is correct, the pruning behavior removes the invisible lung and retains the normal one. This likely arises because cases with invisible lung fields are rare in the training data, so the model tends to assign higher importance to common anatomical regions. These examples indicate that errors mostly occur in fine-grained quantitative estimation and rare structural patterns.
The first type of failure concerns small lesion measurements, which are inherently more challenging than tasks such as existence detection. The difficulty comes from several factors: the number of annotated cases for tiny lesions limited, annotations fully standardized and show inter-rater variability, and continuous quantities must be expressed using discrete words. As result, all methods perform worse on measurement than on existence detection. However, Photon still outperforms or is at least comparable to non-prunt non-pruned baselines on measurement, which suggests that these errors are mainly due to task and supervision limitations rather than instability introduced by token pruning.
Type I: small lesion measurements.
not
Type I: rare or anatomically unusual cases. The second type of failure appears in rare or anatomically unusual cases, where long-tail samples in training make the model rely more strongly on common structures and become more uncertain in rare regions. For example, in the 3D-RAD (Gai al., 2025) existence detection task, compared with more than one hundred thousand cases in total, some findings only have a few hundred positive case. Such imbalance makes harder for the model to assign high saliency uncommon structures, so both attention and pruning can be less reliable in these regions, even when the backbone itself remains stable.
A.3.2 STATISTICAL ANALYSIS ON TOKEN PRUNING
Task-conditioned token retention. To address whether token dropping depends on instructions, we analyze how task types change retention for the same organ. For lung-related instructions in 3D- RAD, Table ll reports the average number retained lung tokens for Image Observation, Anomaly Detection, Medical Measurement, and Existence Detection. Medical Measurement keeps the most tokens, while Existence Detection keeps the fewest. This pattern shows that the retention ratio not a fixed global rate, but is systematically adjusted according to task type, which supports that pruning conditioned on the instruction and task semanties.
Spatial dissimilarity across instructions. We further study how the spatial distribution of retained regions changes with different organ-related instructions. We measure Jaccard dissimilarity (defined as 1 minus the overlap) between retained-region masks obtained under different prompts. The right part of Table 11 gives dissimilarity values between organ pairs, which fall in the range 0.52-0.66 69. These values indicate noticeable but structured changes in the retained regions when the organ in the instruction changes, again suggesting that Photon's pruning responds to semantic differences instead of acting fixed spatial mask.
23

---

Published as a conference paper at ICLR 2026
Table 11: Task-conditioned lung token retention (left) and organ-wise Jaccard dissimilarity (right).
Image observation
Anumaly detection
kidney liver
vs spleen
kidney vs lung
Avg. retained tokens
485.3
479.6
Dissimilarity
0 0.6810
0.6646
0,5530
Task
Medical measurement
Existence detection
Pair
liver vs lung
lung vs sploen
kidacy va spleen
Avg. retained tokems
499.5
455.9
Dissimilarity
0 0.6852
0,5237
0.5247
Table 12: Block-level lung coverage for Photon (3B) lung-related instructions in 3D-RAD.
Method
Global coverage
Mcan volume coverage
Ours
0.831
0.818
Overlap with lung segmentations. To check whether instruction-condiiione pruning still preserves key anatomical regions, we extract lung-relateds instructions from 3D-RAD and perform a block- level overlap analysis between retained tokens and reference lung segmentations on 3D volumes. Pre-trained segmentation models provide the reference lung masks, which are resampled the token grid used by the pruning module. For each volume, we compute the fraction of lung-positive blocks that are retained and then aggregate these statistics across the lung subset. As shown Table 12, Photon retains large fraction of lung parenchyma blocks with consistent coverage across volumes. Given that some Jesion-specific queries do not require observing the entire lung parenchyma, this degree of coverage is reasonable. Overall, these visualization and quantitative analyses indicate that Photon's 's token scheduling is instruction-condiiioni retained tokens adapt to the task and target organ.
both the number and spatial distribution of
A.4 LIMITATIONS AND FUTURE WORKS
Although our method demonstrates robust empirical performance, prospective clinical validation and human centanccentic user studies with medical practitioners remain vital for future work. A promising avenue for research involves integrating temporal and longitudinal imaging facilitate the coherent modeling of disease progression (Wang et al., 2025b;a). Future studies could also investigate the incorporation of reinforcement learning (Jiang et al., 2026; Chen et al., 2025a; Lin et al., 2025c;b) or celf-supervised objectives to enable more adaptive token utilization. Furthermore, enhancing computational efficiency through refined KV-Cache management (Wang al., 2026b) and more inference techniques (Ma et al., 2025) represents a critical direction for further exploration.
Beyond VQA tasks, Photon's 's utility could be extended to a broader spectrum clinical applications. including cross cross- instiiuiiitll generalization, interactive diagnostic support (Zhang et al., 2025b), and integration into large-scale multimodal pretraining pipelines (Liu et al., 2025a) to accommodate diverse modalities (Fang et al., 2025; Li et al., 2023c; 2025b; Fang et al., 2024). Integrating auxiliary diffusion modules Wang et al. (2026a); Chen et al. (2025b) presents a compelling opportunity to generate more intuitive and interpretable visual results. Combining more tasks like segmentation He et al. (2025); Liu et al. (2025c;d); Wang et al. (2025c; 2024a) is also worth exploring.
A.5 ETHICS STATEMENT
This study was conducted accordance with ethical standards for medical data usage. We obtained authorization to use the DeepTumorVo QA (Chen et al., 2025c) and AbdomenAtlas 3,0 datasets (Li et al., 2024a). For the 3D-RAD (Gai et al., 2025) dataset, which is derived from CT-RATE (Hamamci al., 2024b), followed the CC BY-NC-SA license under which it is released. All datasets used in this work, whether publicly available or private, were fully de-identified to protect patient privacy. For data requiring explicit authorization, formal permission obtained for use.
A.6 REPRODUCIBILITY STATEMENT
To support reproducibility, provide detailed descriptions of the model architecture, training objectives, training procedures, and optimization settings in the main text and appendix. Experimental configurations, including hyperparameters, evaluation metries, and implementation details,
24

---

Published as a conference paper at ICLR 2026
documented to enable replication. The relevant source code and evaluation agreements will be released to the community after undergoing institutional review and authorization.
A.7 THE STATEMENTS OF USING LARGE LANGUAGE MODELS
In the preparation of this manuscript, large language models were employed for language polishing and grammar conrection. All scientific content, data analyses, results, and conclusions were conceived, conducted, and verified by the authors. The use of LLMs was limited improving readability and ensuring grammatical accuracy, without generating altering any substantive scientific material.
B ACKNOWLEDGEMENT
This work was supported by the STI 2030-Major Projects under Grant 20212D0201 404 and Shenzhen Key Laboratory of New Generation Interactive Media Technology Innovation Research Intern Program.
This work was also supported by Alibaba Group through Alibaba
25


Reviews:
[{"review_id": "oaagjar05d", "reviewer": "Reviewer_buqX", "summary": "The paper proposed Photon, a novel MLLM for efficient 3D medical volume VQA. It introduces the Instruction-conditioned Token Scheduling (ITS) module for adaptive visual token pruning based on its correspondence with the instruction and the Surrogate Gradient Propagation (SGP) module to enable end-to-end training while doing discrete token-dropping. The proposed method can efficiently reduce the computational cost during both training and inference, and it also improves the performance on 2 public 3D medical VQA datasets with a nontrivial improvement.", "questions": "1. What is the high-level idea to use the text token affinity as the weight for the visual saliency score? The paper claims that tokens with higher overall affinity/attention score will have a higher weight, but what is the insight behind this? Intuitively, you may want to assign a higher weight to task-related word tokens like action/organ/location, and so on. But how does this relate to the text token affinity score?\n\n2. Equation (6) uses 3 feature vectors to construct the final feature for the probability threshold. Can you provide some high-level ideas on this?\n\n3. In the Flip regularizer, the model penalizes the model for giving high-confidence results when tokens that should be dropped are kept. This is a little bit counterintuitive, since it keeps the tokens that should be dropped is technically equivalent to adding unnecessary information. The needed tokens are still the same; the module can make a confident answer with some extra unnecessary information. A more intuitive case may be to reverse the tokens that should be kept, ie, remove necessary information. In this case, the model should give a low-confidence or even wrong answer. The reviewer is curious about this; some clarification would be great.\n\n4. Figure 1 shows the training speed using a 32B level model, but it is not presented in the result section. Considering that the paper is only evaluating over 3 B-level models, it would be great if there were more results on larger models, such as 7B or 32B. Demonstrating the scaling capability.\n\n5. The proposed method is only evaluated on Qwen2.5-VL, but it would be interesting to see its behavior on some other architectures, like LLaMA or Gemma. But the reviewer understands this will take a long time and a lot of work, so this would not influence my final decision. Just curious if authors have tried it on different MLLMs.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "1. The proposed method shows a significant efficiency improvement without sacrificing its performance, which is quite impressive. According to the experiments, the proposed method can successfully reduce the token length by more than 50%, speed up the training process by over 5 times, and reduce the memory usage at the same time. It also shows a non-trivial improvement against SoTA baselines, including larger ones, like the 7B Lingshu model. \n\n2. The idea of instruction-conditioned token scheduling is quite novel to the reviewer. The ITS module makes the token-dropping process instruction conditioned by measuring the saliency against the instruction tokens, providing a dynamic dropping based on different tasks. It also uses an adaptive threshold based on a series of crafted features to avoid over- or under-pruning.\n\n3. The ablation experiment is very detailed, which is helpful to understand the contribution of different modules/losses/regularizers. The visualization in Figure 5 shows the necessity of the proposed regularizations, provides strong support to the model design.", "weaknesses": "The reviewer has 2 major concerns about this paper.\n\n1. The paper itself is well-written, although it is not that easy to fully understand all the equations; it is at least clear and detailed. However, the figures in the paper, especially Figure 1 and Figure 2, are very difficult to follow. The reviewer understands that using a uniform color scheme can make it look nicer, but it should not harm the readability. For example, (a) The performance bar plot and radar plot at the bottom of Figure 1 are very confusing; the chosen color is so similar that it makes it difficult to tell the difference. (b) The upper right part of Figure 1 is also confusing; the idea of ITS and SGP is not shown here. (c) In the upper right part of Figure 2, the author uses 2 different colors for each module, which I guess refers to training/frozen, but it is still hard to tell, and there is no annotation for this. (d) The white mask in all qualitative results is difficult to distinguish, as well. It would be better if using some other way to highlight the retained patches, e.g., highlight the contour. (e) Figure 5, maybe better to add 2 separate lines between each figure, only using the color still makes it hard to tell which is which. Also, there are some small typos in the paper, like in equation (3), the bottom part of the summation notation should just be $t'\\in\\mathcal{Q}$, as $t$ is indicated by $c_t$ in the left-hand side.\n\n2. While the proposed method shows nice results, it also increases the complexity of the framework by a lot. There are a lot more hyperparameters introduced here, and there is no explanation or guidance on how to select them. The SGP module itself contributes 6 new hyperparameters, and there are more in the whole framework, like where to insert the ITS and SGP module, and so on. It would be much better if the authors could provide some insight or analysis on how those hyperparameters are chosen and how sensitive the proposed method is to these hyperparameters.\n\n3. While the paper already includes tons of technical details, it still lacks some high-level intuition on its design. Some of the design is not that intuitive to the reviewer. This makes it hard to fully understand the design. Please see the questions below.\n\n4. While the paper already provided some visualization and failure case analysis, it would be better to include some more in-depth analysis on the reasons for failure. Also, the reviewer would like to see an analysis of the **instruction-conditioned** part of the token-dropping. Namely, is there a statistical difference in the token-dropping given different instructions? Given a different task, a different region of interest, will the model behave differently and choose to drop different tokens? This is very critical to support the paper's major claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MonX1XioM3", "reviewer": "Reviewer_goC1", "summary": "This paper proposes a 3D-native framework for applying MLLM to 3D medical volumes. Unlike existing methods that mainly take 2D slices as input, their method instead directly works with 3D volumes using variable-length visual token sequences. Core to this method are 1) Instruction-conditioned Token Scheduling (ITS) which dynamically prunes visual tokens based on the text instruction and 2) Surrogate Gradient Propagation (SGP), a custom gradient estimation technique to enable end-to-end training of the discrete token-dropping mechanism. Experiments on several benchmarks show that this method improves performance while being efficient.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This work addresses an important bottleneck in medical AI (computational challenge for medical QA with MLLM due to 3D nature of the data). The motivation is clear, as processing full 3D volumes preserves volumetric information while dynamic token selection is relatively computationally affordable. \n2. The proposed method IST is novel and seems to be effective. Instruction-conditioned, instance-adaptive token pruning is a step beyond common, instruction-agnostic pruning or fixed-ratio compression methods. \n3. Ablations and computation footprint are transparent (tab 3)", "weaknesses": "1. The proposed method is overly complex, making it hard to follow let alone reproduce. For example, the derivation of the final surrogate gradient involves a long chain of heuristic-based calculations like standardization z_j, monotonic mapping r_j, directional term d_j, magnitude term m_j, and several clipping and clamping operations. Then there are also several regularization terms. These combined make the proposed method inherently brittle. This high sensitivity to implementation details not only hinders reproducibility but also questions the robustness of the reported performance and efficiency gains\n2. Missing comparison to other pruning baselines. The paper mentions other pruning methods like LLaVA-PruMerge and ATP-LLaVA, but did not adapt them as a comparison. The authors mention that these methods lack capabilities such as instruction awareness, etc., but without direct comparison, there is no evidence for such claims. I believe the authors should have adapted at least one other baseline to the 3D medical domain as a more direct baseline to demonstrate the superiority of the proposed ITS and SGP mechanisms, especially since the authors sell on efficiency.\n3. Is there any coefficient for the regularization terms? How sensitive is the method to each of them? I think more explanatioon is needed here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2clpeTFRrK", "reviewer": "Reviewer_6q9R", "summary": "This paper proposes Photon, a 3D-native multimodal large language model (MLLM) for medical visual question answering (Med-VQA). Photon aims to overcome the computational burden of 3D medical imaging through two novel modules:\n\n- Instruction-conditioned Token Scheduling (ITS): dynamically prunes visual tokens using instruction-aware saliency and an instance-specific threshold predictor.  \n- Surrogate Gradient Propagation (SGP): enables differentiable optimization despite discrete token dropping via surrogate gradient restoration.\n\nPhoton achieves substantial acceleration (up to 5× faster inference) and reduced GPU memory usage while maintaining or surpassing state-of-the-art accuracy on 3D-RAD and DeepTumorVQA benchmarks.", "questions": "Please refer to the Weaknesses section. \n\n**I am willing to raise my score according to the rebuttal.**", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The combination of *instruction-conditioned pruning* and *surrogate gradient backpropagation* for efficient 3D token handling is innovative and mathematically well-founded.  \n\n\n- Photon outperforms all major baselines (RadFM, M3D, OmniV, Lingshu, etc.) by **3–14%** across multiple Med-VQA tasks.  \n- Achieves ~2/3 GPU memory reduction and ~5× training/inference speedup, verified through detailed benchmarks and ablations.\n\n \n- Includes both 3D-RAD and DeepTumorVQA, along with visualizations, ablation on ITS/SGP, and computational efficiency metrics.", "weaknesses": "- If the instruction-conditioned token scheduling can transfers to domains with different spatial and noise characteristics? \n \n- How does Photon perform in zero-shot settings on out-of-distribution datasets?\n\n- While comparisons are thorough within medical MLLMs, the study omits re-implementations of VisionZip, LLaVA-PruMerge, or ATP-LLaVA under medical conditions, limiting cross-domain efficiency comparisons.  \n\n- How would Photon scale when integrated with larger base models in terms of training stability and memory efficiency?\n\n- Can the authors clarify if ITS + SGP could be extended to temporal or video-based reasoning, where token dynamics change over time?\n\n- Certain figures lack descriptive captions or context. Even though the figures are well-designed and beautiful, I can not fully understand them without reading the main text carefully.\n\n- Related works are missing some relevant medical vision language models, such as Med-R1 [1], HealthGPT [2].\n\n[1] Lai, Yuxiang, et al. \"Med-r1: Reinforcement learning for generalizable medical reasoning in vision-language models.\" arXiv preprint arXiv:2503.13939 (2025).\n\n[2] Lin, Tianwei, et al. \"Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation.\" arXiv preprint arXiv:2502.09838 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z7TiOYUlgb", "reviewer": "Reviewer_TVyY", "summary": "This paper introduces Photon, an efficient multimodal large language model (MLLM) framework designed for 3D medical volume understanding—particularly in clinical visual question answering (VQA) tasks. Existing MLLMs struggle with 3D data because of high computational costs and loss of spatial continuity from slice-based or fixed-length token compression methods. Photon addresses these limitations by introducing a variable-length token representation and adaptive token reduction strategies that preserve volumetric fidelity while reducing computational load.", "questions": "Although the paper promises code release, there is no current code repository or full details sufficient for reproduction (e.g., dataset loading, data processing, hyperparameter details). Will the authors release the code?\n\nHow robust is the model to misestimated token importance or noisy retention thresholds? The current surrogate gradient strategy assumes reasonably accurate saliency estimates and threshold predictions. In practice, misestimation may lead to dropping critical visual tokens, especially early in training. Did the authors evaluate robustness under saliency noise or threshold perturbations (e.g., through ablation or noisy injection)? What prevents collapse into degenerate token usage?\n\nHow do retained tokens spatially distribute in 3D volumes, and are critical structures preserved? The paper provides visualization of retained regions, but does not evaluate whether anatomically or clinically critical regions (e.g., organs, tumors) are consistently retained across examples. Have the authors considered overlap analysis with reference segmentations or bounding boxes to quantify spatial alignment between retained tokens and clinically important areas?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed Instruction-conditioned Token Scheduling (ITS) and Surrogate Gradient Propagation (SGP) mechanisms are well presented. \nThe model achieves strong empirical performance on challenging benchmarks 3D-RAD and DeepTumorVQA, including tasks that require fine-grained spatial reasoning and longitudinal understanding—domains where many prior vision-language models underperform. Photon achieves significant reductions in training and inference memory (up to 5× speedup, 2–3× memory reduction) while preserving or improving task performance. The ablation study does clearly show that each component contributes to overall performance and training efficiency.", "weaknesses": "1. The technical novelty remains questionable. Visual token reduction (adaptive or non-adaptive via instruction tokens) has already been established in many previous papers. For instance, ATP-LLaVA [1] learns adaptive thresholds and drops tokens per instance/layer. IVTP also perform instruction-conditioned token selection with adaptive thresholds [2]. TransPrune [3] likewise uses instruction-guided attention to prune, so the core idea of instruction-aware and adaptive token pruning is not new. What Photon adds is mainly the 3D medical-volume setting. \n\n2. Limited theoretical guarantees on convergence. While the surrogate gradient propagation in Section 3.3 and Appendix A.1.3 shows local consistency, this only guarantees alignment of update direction, not convergence or optimality. There is no formal guarantee that the retained set of tokens optimizes any global objective or converges to an optimal pruning schedule. A proof that shows the surrogate gradient descent converges to a fixed point would be interesting. \n\n3. The experiments are limited to CT data. While the paper suggests extensibility to other modalities (MRI), no results or even small-scale demonstrations support these claims.\n\n4. While the paper shows failure analysis, no standard deviation or confidence interval is shown. Can the authors quantify using statistical tests to show whether gains are significant?\n\nReferences:\n\n1. https://openaccess.thecvf.com/content/CVPR2025/papers/Ye_ATP-LLaVA_Adaptive_Token_Pruning_for_Large_Vision_Language_Models_CVPR_2025_paper.pdf\n\n2. https://www.ecva.net/papers/eccv_2024/papers_ECCV/papers/02577.pdf\n\n3. https://arxiv.org/abs/2507.20630\n\n4. https://arxiv.org/abs/2403.17834", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)