Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs
Authors: Yongyi Su
Haojie Zhang
Shijie Li
Nanqing Liu
Jingyi Liao
Junyi Pan
Yuan Liu
Xiaofen Xing
Chong Sun
Chen Li
Nancy F. Chen
Shuicheng YAN
Xulei Yang
Xun Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Model
Referring Expression Comprehension
Visual Reference Token
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce Patch-as-Decodable Token (PaDT), a unified paradigm that enables MLLMs to directly predict Visual Reference Token for multi-modal understanding and reasoning.

Abstract: Multimodal large language models (MLLMs) have advanced rapidly in recent years. However, existing approaches for vision tasks often rely on indirect representations, such as generating coordinates as text for detection, which limits performance and prevents dense prediction tasks like segmentation. To overcome these challenges, we introduce Patch-as-Decodable Token (PaDT), a unified paradigm that enables MLLMs to directly generate both textual and diverse visual outputs. Central to PaDT are Visual Reference Tokens (VRTs), derived from visual patch embeddings of query images and interleaved seamlessly with LLM's output textual tokens. A lightweight decoder then transforms LLM's outputs into detection, segmentation, and grounding predictions. Unlike prior methods, PaDT processes VRTs independently at each forward pass and dynamically expands the embedding table, thus improving localization and differentiation among similar objects. We further tailor a training strategy for PaDT by randomly selecting VRTs for supervised fine-tuning and introducing a robust per-token cross-entropy loss. Our empirical studies across four visual perception and understanding tasks suggest PaDT consistently achieving state-of-the-art performance, even compared with significantly larger MLLM models. The code is available at https://github.com/Gorilla-Lab-SCUT/PaDT.

Published as a conference paper at ICLR 2026
PATCH-AS-DECODABLL-TOOOS TOWARDS UNIFIED MULTI-MODAL VISION TASKS IN MLLMS
Yongyi Su
Haojie Zhang 1.
Shijie Li
Nanqing Liu
Jingyi Liao Junyi Pan
Yuan Liu!
Xiaofen Xingè Xun Xu 2*
Chong Sun
Chen Lis
Nancy Chen? Shuicheng Yan
Xulei Yang²
South China University of Technology
2 Institute for Infocomm Research (I²R), STTAR
3 WeChat Vision, Tencent Inc.
4 Yunnan Normal University
Nanyang Technological University
6 National University of Singapore
ABSTRACT
Multimodal large language models (MLLMs) have advanced rapidly in recent years. However, existing approaches for vision tasks often rely on indirect rep- resentations, such as generating coordinates as text for detection, which limits performance and prevents dense prediction tasks like segmentation. To over- come these challenges, we introduce Patch-as- ---b-----ee Token (PaDT), unified paradigm that enables MILMs to directly generate both textual and diverse visual outputs. Central to PaDT are Visual Reference Tokens (VRTs), derived from vi- sual patch embeddings of query images and interleaved seamlessly with LLM's output textual tokens. A lightweight decoder then transforms LLM's outputs into detection, segmentation, and grounding predictions. Unlike prior methods, PaDTT processes VRTs independently at each forward pass and dynamically ex- pands the embedding table, thus improving localization and differentiation among similar objects. We further tailor training strategy for PaDT by randomly select- ing VRTs for supervised fine-tuning and introducing robust per-token cross- entropy loss. Our empirical studies across four visual perception and under- standing tasks suggest PaDT consistently achieving state-of-the the-art performance, even compared with significantly larger MLLM models. The code is available https://githuu com Gori
la Lab a SCUT/PaD)
INTRODUCTION
Fine-grained image perception and understanding, which aim to associate specific image regions with contextual information, such as semantic or instance, is a fundamental task in computer vision and serves cornerstone for numerous applications. Classical vision models (Ren et al. 2015 Redmon el al. 2016 Carion et al. 2020) remain state-of-the-ard for pure detection and segmentation tasks, but they lack flexible language interaction and understanding, thus prohibiting open vocab- ulary oriented visual reasoning tasks. At an earlier stage, inspired by CLIP (Radford et al.] 2021)) many vision- lasion-language detectors such as GLIP (Li et al.) (2022b) and Grounding DINO (Ren et al., 2023) incorporate language information to detect arbitrary classes. However, these methods remain vision-centric backbones augmented with language, and thus struggle to handle more complex tex- tual descriptions and are limited to structured output.
Recent advances have led to powerful multi-modal large language models (MLLMs) (Alayrac et al, 2022 Li et al. 2023 Liu et al. 2024c: Bai et al. 2025 Zhu et al.) 2025) that couple vision encoders with Large Language Models(LLMs) Pretrained on massive multimdall datasets, these models
encode rich prior knowledge and provide a strong foundation for visual perception and understand-
ing, as illustrated in Fig. [
To conform with the textual output space of LLMs, most existing
Coorrsponddncce to <xx__xun (aw-star.eddd .dd.tar.edu.sgg>>>>>.... and SihijShije@@ aashjie@a-ttarr.duu....... This work was done during Yongyi Su's visit to I'R and Haojie Zhang's 's intern in WeChat Vision.

---

Published as a conference paper at ICLR 2026
Open Vocabulary Detection
Referring Expression Comprehension
Referring Image Captioning
Query. Plesse canefully check the image: and
Query: Please carefully check the image end detect the object this sentence descrbbes: "blus
Query, Please describe this image.
detect the fallowing objects:
I'person",
Response:
set dining table (=
"bicycle'... .., "toothbrush"
pointed on it'.
=) includes
Response: In this mmge, there "person' (Sa
=). cup soup
). and
2a M! | D. sports ball" (332C)) and
Response: The *blue dish with cats painted
sandwich with sesome Seed bun
"fennis rocket" (Ml
on if refers to MOSKR in this image.
A wine gloss (90=
and bottle of
souce: (X RD are also present. The table
B
&
creating on elegant dining setup.
arranged with nepkins and coasters.
L
OU
oppears be served a restouront or
formal dining environment.
Image with visual refarence
with bounding
Image with vanal reference tokens
with bounding
Referring Expression Segmentation
Query: Please corefully check the image and detect the object sentence describes silver suv with block rails".
TEF
Mentioned Objects: Dining Table!
Solod:
Soup:
Sandwich:
The "silver suv with black roof rails' refers to 2 = in this image.
Responde:
Image with visual reference tokens
Image with segmentation mask
Wine Glass
Sauce:
Figure 1: Illustration of unified visual/textual token prodiction for MLLM powered visual perception and understanding.
2025 serialize detected regions into bound- ing box coordinates, expressed textual form, such as [11, 42, While straightforward, this strategy introduces several challenges. First, output formats are often inconsistent across samples even under the same prompt, as illustrated in Fig. a), thereby increasing the difficulty of parsing and structured output. Second, numerical coordinate representations provide precise spatial descrip- tions but lack semantic alignment between textual and visual modalities, as shown in Fig. 2(b). This inherent misalignment can lead to repetition hallucination between coordinate and actual visual targets (Jiang et al. 2024b). Moreover, since numerical coordinate representations are mapped into discrete textual tokens, single coordinate value may be split into several unrelated tokens, as shown in Fig. (b)). These discontinuous coordinate tokens can hinder prediction accuracy, e.g., fragmented numbers.
MLLMs (Liu et al. 2025a Bai et al. 2025| Zhu et al.
In this work, we introduce a unified paradigm, Brcch-a--eeodable Token (PaDT), which enables MLLMs to directly generate both textual and diverse visual targets a unified yet flexible way. For this purpose, we propose the Visual Reference Tokens (VRTs), which can be seamlessly inter- leaved with LLM's output textual tokens. VRTs are generated by the proposed Dynamic Embedding Module, adapted directly from the original visual patch embeddings. In this way, they occur in feature space consistent with the original LLM, while each VRT explicitly corresponds to a specific image patch within the query image. Thus, VRTs can be naturally interpreted within the LLMs feature space, allowing detected objects to represented by multiple VRTs in a fine-grained man- ner. Based on this design, PaDT owns the inherent ability to predict diverse visual outputs, e.g. semantic masks and bounding boxes. Specifically, MLLMs only need to predict a subset of VRTs, which are then decoded into the final structured visual outputs by lightweight decoder. A prior art (Ma et al. 2025) attempted to empower LLMS to output image patch tokens, discretized by a global codebook, to represent the target within the image. However, this approach remains limited in flexibility and generality due to maintaining a global codebook. First, there is a risk of predicting visual tokens that do not appear in the query image. Moreover, the decoded visual token does not have unique correspondence in the query image, thus risking misalignment between predicted visual tokens and query image tokens, e.g., confusion between similar objects in the image. In contrast, PaDT processes VRTs independently at each forward pass, making it more efficient. By maintaining a high-level feature space aligned with that of L.LMs and preserving unique positional information for each image region. PaDT ensures coherent predictions illustrated in Fig. (c... Moreover, as shown in Fig. d), VRT predictions over objects exhibit great spatial continuity.
To enable PaDT to achieve strong performance, we design an effective fine-tuning strategy and propose a robust per-token cross cross-entropy loss tailored for the proposed visual reference token, which stabilizes training and mitigates overfitting. Notably, our 3B model surpasses the previous of the-art by 19.0 mAP on COCO detection and achieves an average accuracy of 93.6 on the referring expression comprehension (REC) task, outperforming the much larger 78B Intern VL.3 model.
2

---

Published as a conference paper at ICLR 2026
image, zebras (6.36.
Low Activated in image
High Related with ab object
(0.69. 0.56.
0.85, grazing grossy
field. wildebeest (10422 0.55. 0.59, (.67)) present floot type.
In image stop sign (1248, o 396, 977) attached wooden pole (1250, 472, 444]). pole appears be waathered
Please
Pheae cardfuly
int type, (0-H/W)
destoct the objject Uhis suntinns discriles Ulas wilh cummins jou
diseribune
"ythe
with
In this image, giraffe toy (bboxc 35, 84, 279, 236) stands to the left. Next to it, 343, 410)
The "ythe
(bbox: 204,
5.9% Iobed To with coumone
redors
<lvrt 220
201
<ojort 2015-
2
Unexpected Format
Consteloton: 9.
Condidates are
Consiibalca
< [ot__
Clvrt 240 In this Image.
(ert 200) 1wL_500
(d) Heat Map of VRT Candidates
(a) Inconsistent Output
(b) TAM of Qwen2.5-VL
(c) TAM of Ours
(a) Previous methods yield inconsistent output formats due free-form representations even under the same prompt. (b) Token Activation Map (TAM) (Li al., 2025) reveals semantic relationship between textual box representations and textual/visual information, w while converting continuous numbers into discrete tokens further introduces discontinuities. (c) With PaDT denoting objects with VRTs, semantic align- ment is preserved and the output becomes more unified and natural. (d) The heatmap of <VRT -27>> further demonstrates continuous and cejectt-onssstent predictions within the input image.
Figure 2:
The main contributions of this work can be summarized as follows:
We introduce unified paradigm, Patch-as-Decodable Token (PaDT), which enables MLLMs to directly generate both textual and diverse visual targets in a unified yet flex- ible way. With the proposed Visual Reference Token (VRT), method achieves superior performance across diverse fine-grained image perception and understanding.
We propose lightweight yet robust VRT-based decoder, termed the PaDT Decoder. Given the generated VRTss, it can uniformly decode diverse fine-graine structured visual outputs, such as segmentation masks and bounding boxes.
We propose effective fine-tuning strategy together with a robust per-token cross-entropy loss. PaDT achieves the state-of-th -het-----te-rtttt performance on a wide range of visual perception and understanding tasks. The effectiveness validated beyond perception tasks but also customized image captioning task.
RELATED WORK
Multimodal Large Language Models. With the rapid development of large language models, mul-
timodal LLMs (MLLMs) have emerged as powerful systems for rision-langusee reasoning (Alayrac)
ct al.
2022; Achiam et al.; 2023 Liu et al. [023 Zhu et al. [023 Zhang et al.) [2024a] Lian et al. Bai et al. 2025). Early mil milestones such CL.P (Radford al. 2021) and ALIGN (Jia al.,
2025
2021
demonstrated the effectiveness of large-scale contrastive pretraining for joint vision-text repre- sentations. BLIP-2 (Li et al. 2022a) further improved alignment through the Q-former design. More recently, instruction-tunce MLLMs including LLaVA (Liu et al. (2023) and MiniGPT-4 (Zhu et al.) 2023 leverage multimodal instruction data, yielding strong performance in open-ended visual ques- tion answering and reasoning. Building on these foundations, subsequent works extend capabilities heshetoreoolution image understanding (e.g.. LLaVA-Next (Liu al. 2024c). LLaVA-UHD (Guo)
et al. 2024)), diverse instruction sets (Ye et al. 2023 multi-image (Jiang et al.) 2024a Li et al.) 2024 and video inputs (Lin et al., 2023a Chen et al. (022a)), as well as new pretraining objec- tives and architectural designs (Fang et al. 2023 Wang et al. (023b). Collectively, these advances establish MLLMs as versatile
models for multimodal reasoning.
MLLMs for Visual Perception & Understanding. Despite their broad capabilities, general- purpose MLLMs remain limited in fine-grained perception tasks. This stems largely from vision encoders reliance fixed patch grids (Dehghani et al., 2203 Fang et al. 2023] Wang et al. 2023b), which often blur local details and impair tasks such as object localization, counting, or OCR. To mitigate this, adaptive tiling strategies, such as Vi patch dropping and AnyRes (Luo al.
3

---

Published as a conference paper at ICLR 2026
Model Pipeline
4. Light Light-Weigh Weight PaDT Decoder
Obj
Hoddnn Festures
Obj
Hivzen Festures
Bbox Mask
Score
VRTS
vR1e
Tioo May Athenhor Block
Uperrpping
Light- Weight PaDT
Patch Features
way ittertoon
Tndex
Decoder
Attertion Block
M
mask lngit
VRT Hidden Peotures
Bbox Taken
(cr,
Visuil Reference Takens (VRT)
3. PaDT T Head
Text
There are 2 "bear"
in this image.
WWest Pror
Weights
Visual Prototypes
Patch-as- Delii-as-Decoaable-Toooo Token (PaD Head
Y H We
reture
Dynomic Embedding Module
Tokee
Backbone
Voud Reforerze
(A41)
Prtdyy "akne
2p Na fally
Projector fo
Dynamic Embedding Module
Vision Encoder f
objeensi l'person" beer""
Runkk Foutarne
Axp
Vauu Reference Pratohpps
Figure 3: The framework of PaDT model.
Chen et al. 2024b Liu et al. 2024a). allow flexible handling of variable-resolution image tiles, leading to improved spatial resolution. Another line work explores reinforcement learning to enhance perception and reasoning, exemplified by VLM-RI (Shen et al. 2025). Visual-RFT (Liu ct al. 2025b), VisRL (Chen et al. (025), and Seg-R1 YYou Wu 2025). These approaches achieve better generalization and emergent capabilities such segmentation and grounding. Prior works have primarily relied on reinforcement learning (Chen et al. 2025) or instruction tuning (Jiang et al.] 2024b) strengthen visual reasoning, yet the potential of leveraging learned queries anchors for visual perception remains underexplored. Moreover, designing a unified architecture that seamlessly accommodates diverse vision tasks continues be an open challenge.
2023
Unified Visual Tokenization. A complementary research direction focuses on unifying visual and linguistic representations through multi-granular tokenization. At the region level, methods con-
vert object boxes or masks into geometric tokens (Chen et al. 2023b; Xuan et al.
2024 Peng
al. 2023; You et al. (023) or learnable proxies (Zhang al. 2024b] Yuan et 2024 Chen
et al. 2023a; Rasheed et al. 2024). often grounded by detectors or SAM (Kirillov et al.) 2023),
thereby enabling more precise vision-language grounding. At the patch level, models such as Emu series (Sun et al. 2023) and LaVIT (Jin et al. (2024) treat CI.IP-derived patch features vi- sual vocabularies for denser alignment. Recent works further introduce autoregressive quantization of image patches (Team] 2024: Sun al., (202), discretizing pixels into visual sentences sup- port efficient cross-modal modeling, with even finer-grained tokenization explored in (Ma et al.) 2025). While these approaches approximate linguistic structures via region, instance, or pixel to- kens, deeper semantic integration between vision and language still limited. To address this, we propose dynamic multimodal token space that enables close correspondence between language tokens and visual patches under a unified autoregressive modeling paradigm.
METHODOLOGY
3.1 REVISITING MULTIMODAL LARGE LANGUAGE MODELS
Multimodal Large Language Model (MLLM) augments Large Language Model (LLM) with a visual encoder, enabling it perform not only general-purpose purpose reasoning but also visual percep- tion (Alayrac et al. 2022 Liu et al. 2024c Bai et al., 2025). Given an image f X3 (4, - Lm)) the MILMM autoregressively generates output sequence
and text sequence

---

Published as a conference paper at ICLR 2026
Y (y1,.... ve). An image encoder fer typically a Vision Transformer (ViT) Dosovviskky al. 2020). partitionns into N non-overlapping patches [Pr]n=r which are subsequently encoded into embeddings F. fo(1) E I RN A projector f, then aligns dimensions and downsam- ples, yielding Faaach = f2(F,) E R'x, For instance, Qwen2.5.VL. adopts neighor patch merging in the 2D patch space, resulting in N' 1 A.N. The image embeddings are then fused with the text embeddings Etext E R to form a hybrid textual-visual representation Z = [F [Patchh P Here, Etext IR' VreatXd denotes the text embedding table that maps each text token to its corresponding feature vector. The resulting multimodal representation Z is subse- quently fed into At timestep I, the hidden state h, produces the next-token distribution:
measformer-basee LLM (Alayrac et al.) 2022 Liu al. 2024c
Bai et al. 2025
p(ue T, yer) = softmax (Wtezt ht):
with Wtent E R ViexeeXe xd denoting classifier weights.
Limitations of Text-based Vision Prediction. Current MLLMs are restricted accepting textu- alvisual representations as input and producing only textual outputs, owing to their compatibility with the underlying LLM architecture. This limitation is suboptimal for structured vision tasks such as object detection and image segmentation. Specifically, current MLLMs (e.g.. Qwen2.5-VL. (Bai al., 2025), VL3 (Zhu et al., 2025)) serialize visual targets into strings at output side. This leads to two major issues. First, outputs vary in format (absolute vs. normalized coordinates, JSON. style vs. free-form), complicating parsing and structured output, as shown Fig. aaa Second, numerical coordinate representations are mapped into discrete textual tokens which are generated digit by digit (e.g.. "489" "4, 8, 9*). This disrupts numerical continuity and may hinder prediction accuracy (Fig. 2(b)). More importantly, while this numerical representation effectively describes spatial information precisely, it lacks semantic information, which is crucial for image understand- ing tasks. This inherent mismatch, revealed through token activation analyses (Li et al., (025) as illustrated in Fig. 2bb). can lead to errors such as repetition or hallucination in dense prediction tasks @Jiang al. 2024b).
3.2 VISUAL REFERENCE TOKEN
(PaDT) framework, which introduces Visual Refer- ence Tokens (VRTs), a unified tokenization scheme that embeds visual patches directly as decod- able tokens within the autoregressive generation process. PaDT extends conventional MLLMs with three key components: (1) Dynamic Embedding Module augments the textual vocabulary codebook with visual patches, specific VRTs, at each forward pass, yielding multi-modal codebook. (2) With this multi-modal codebook and the proposed PaDT' Head, VRTs become both embeddable at the input side and decodable at the output side, resulting in a unified and natural format. (3) A lightweight PaDT Decoder is proposed to convert variable VRTs into diverse visual representations, such as bounding boxes and masks, enabling downstream tasks including detection, segmentation, and grounding. This further enhances both the robustness and flexibility of the proposed method.
We propose the Batch-as-Decodabiie
3.2.1 UNIFIED MULTI-MODAL FORMAT WITH VRTS
A core challenge to ensure that VRTs can by interpretable by LLMs, being both embeddable in the input space and decodable in the output space. Prior work, e.g., ClawMachine (Ma et al.) 2025 relies on pretrained discrete visual tokenizers (Jin et al. 2024). It inserts the entire codebook, which contains a massive number of tokens, into the LLM embedding table and forces the LLM to map its high-level semantic feature space to tokens representing low-level image patches. Thus, this method is limited by (i) a fixed dataset-level codebook expansion which contains massive tokens that ignore patch-specific cues such as spatial location, and (ii) ambiguity arising from the lack of high-level semantics when visually similar patches from different objects maybe mapped to the same token.
Dynamic Multi-Modal Codebook Expansion. To avoid the above limitations, rather than intro- ducing a standalone codebook, we reuse the extracted visual tokens from the input image, which already preserve rich semantic information. Since each visual token explicitly corresponds to an image patch, at each forward pass only the tokens from the current query image are dynamically ex- panded into the original textual codebook, instead of memorizing all possible visual patterns through fixed codebook. Specifically, in the proposed D Dynamic Embedding Module, original patch features
5

---

Published as a conference paper at ICLR 2026
Patch E R. N' are projected by lightweight module fop into visual reference prototypes Pres-
Sop consists of LayerNorm and a low-rank linear projection. These prototypes are then concate- nated with text embeddings to form dynamic embedding table as,
Edyn = Piezt: Prefl:
Pref = Sep (Patcc))))) E R N'xd
(2)
Unified Input and Output Format. With the above Multi-Modal Codebook, both textual and visual information can be input and output in unified way. On the input side, query image tokens are indexed in the Multi-Modal Codebook and converted into the corresponding VRTs, which are then embedded into the textual input to the LLM. Since VRTs are adapted from the original image tokens, they share a feature space that is similar to the LLMs representation space, which simplifies training compared to ClawMachine (Ma et al.) 2025). On the output side, to enable the original textual classifier to output expanded indices, the PaDT Head is proposed to augment the classifier with /r yielding
W Weati Pres] E (Veca
This joint design allows VRTs to be embedded as inputs and decoded as outputs, enabling the model to insert patch-level references directly into the autoregressive sequence. Building this, we propose a robust strategy that represents detected objects with several (but not all) VRTs placed on them, and then decodes fine-grained representations such as bounding boxes or masks through the lightweight PaDT Decoder introduced below. This strategy is shown to be more robust and effective in our experiments. Template examples for each vision task are provided in Appendix A.2
3.2.2 LIGHT-WEIGHT PADT DECODER
Considering that only several VRTs a de- tected object are predicted, a visual decoder is needed to convert these predicted VRTs into task-specific outputs. For this purpose, we in- troduce a lightweight vision task decoder, im- plemented as a stack of three two-way atten- tion blocks (Fig. 4b).. The decoder takes as input the hidden features of predicted VRTs from the final LLM layer. These features are grouped into object queries, where each group corresponds
Senerated
Outpor Query
Cutput
9: Yesr
S Viret
Grass-Attention
Split
1k
VRTS
Text Token
VRTs
Obj
Obj 2
Cress Attentior.
predict
prodict
Self-artention
sequence of VRTs separated by intervening text tokens (Fig [a)). To enable task-specific decoding, we inject three learn- able tokens, bounding box, mask, and score to- kens, into each group of object queries. After passing through the three attention blocks, each task token is projected into its respective output space, producing bounding boxes, segmentation masks, and confidence scores.
Gbj
0b12 Query (o)
2
BIM
Query
Inpur
Trput Patch Peotures
(6)
Figure 4: Illustration for PaDT decoder.
3.2.3 TRAINING STRATEGY
Robust Per-token Cross-Entropy Loss. For the autoregressive output of the MLLM, we adopt the standard supervised fine-tuning paradigm with a per-token cross-entropy loss:
LCE =
Σ log I, yet) = log softmaxcr ht),
(4)
where it denotes the ground-truth token at step I, h is the hidden state, and Wt projects to the token vocabulary.
propose to randomly sample Nerr foreground tokens for each forward pass. This sampling strategy increases the diversity of supervision and prevents the model from overfitting a fixed set of tokens, thereby improving generalization. To implement this, we introduce foreground mask M E {0, 1)7x,", where M,,,. = 1 indicates that token n at step was not selected. For such tokens, we suppress their contribution to the loss by masking their logits:
Unlike prior work that uses all foreground visual tokens supervision MMaetal. 2025).
1 Wt. he:
le -00 if M 1.
(5)

---

Published as a conference paper at ICLR 2026
This effectively removes the masked tokens from the softmax normalization, ensuring they nei- ther rewarded nor penalized. The resulting robust cross- coss-yy loss is:
CCE
log soflmaxcr
(6)
By combining random sampling with masked supervision, this objective improves robustness and encourages the model explore diverse valid visual references during training.
Task-specific Losses For structured outputs from vision task decoder, we adopt task-specific ob- jectives i.e. Loboz, Fmssk and Licore following (Kamath et al., 2021 Kirillov al. (2023). More
implemented details about the task-specific losses are given in the Appendix A.4 The final training objective of PaDT is
L LCB
probust + Libox + famask + Lecore-
EXPERIMENT
Tasks and Datasets. We evaluate PaDT across diverse set of visual perception & understanding
tasks. Specifically, we consider: (i) referring expression comprehension and referring expression
segmentation on RefCOCO, RefCOCO+, and RefCOCOg (Mao et al. 2016 Yu et al.) 2016); (ii) open-vocabulary detection on COCO 2017 (Lin et al. 2014]; and (ii)) referring image captioning (RIC), for which construct new benchmark by re-annotating COCO with visionlanguage model (VLM) supervision. Further dataset details are provided Appendix A.1
Architecture and Training Details. We adopt Qwen2.5-VI. (Bai et al. 2025 as the base model and conduct experiments with both 3B and 7B variants to evaluate scalability. Based on existing dataset annotations, each training step randomly sample Nyr visual reference tokens from the foreground mask of each target to construct the ground-truth MLLM sequence. If segmentation masks unavailable, VRTs instead sampled within the bounding box. The ground-truth token templates are provided in Appendix
A.2 Training is performed on a single node with eight 96GB 10-5 and apply gradient checkpointing together with bf oatl6 mixed precision for memory efficiency. FlashAttention-
GPUs, using batch size of 16 per GPU. We the learning rate
2 (Do 2023) is further employed accelerate attention computation.
Table Results of referring expression comprehension task on RefCOCO/+/g datasets.
RefCOCO
RefCOCO+
RefCOCOg
Model Name
Param.
Overall
val
test-A
test-B
val
test-A
test-B B
val
test
Grounding- DINO-L (Liu 2024d]
Bouuddin-DINO-L...
90.6
88.2
82.8
89.0
75.9
86.1
87.0
86.6
UNINEXT-H (Ln et 20236
92.6
85.2
89.6
79.8
88.7
89.4
ONE-PEACE Wang
2023a
92.6
89.3
88.8
92.2
83.2
89.2
89.3
89.9
InternVL.3 Zhu et al.
2025'
IB
85.8
90.1
81.7
76.6
84.1
69.2
82.8
82.6
81.6
Intem Zhu 2025
2B
89.8
92.6
86.4
84.0
89.2
76.5
87.6
87.2
86.7
Qwen2.5-Vis [Bai al. 2025
3B
89.1
91.7
84.0
82.4
88.0
74.1
85.7
85.0
Qwen2.5-VL (SF1. Shen
2025
3B
88.7
82.3
VLM-R1 (Shen 2025
3B
90.1
92.3
85.2
84.2
89.4
76.8
86.8
PaDT (Ours)
3B
88.5
92.4
83.5
88.5
PaDTT Pro (Ours)
3B
96.0
95.5
95.0
91.8
94.8
88.4
93.6
94.0
93.6
Shikra Chen et
2023b
7B
87.0
90.6
80.2
81.6
87.4
72.1
82.3
82.2
82.9
Ferret You et al.
2123
7B
87.5
91.4
82.5
80.8
87.4
73.1
83.9
84.8
83.9
Ferret-VZ Zhang ct al.
2024a
7B
92.8
94.7
88.7
87.4
92.8
79.4
89.4
89.3
TextHaw
al. 2024
7B
87.6
86.2
90.0
80.4
88.1
ClawMachinee [Ma al 2025
7B
89.7
86.9
84.4
88.9
78.0
87.1
Qwen2.5-VL. (Bai
7B
85.4
94.2
89.1
76.9
87.2
86.6
VL. Zhu 2025
8B
92.5
88.2
92.5
90.0
PaDTT (Ours)
7B
93.1
90.4
88.8
92.8
83.2
88.2
88.8
PaDT Pro (Ours)
7B
96.6
97.4
95.6
92.8
95.2
89.4
94.6
94.2
94.5
Ferret You et al. 2023
13B
89.5
92.4
84.4
82.8
88.1
75.2
86.3
Ferret-VZ Zhang ct al.
2024a
13B
92.6
95.0
88.9
87.4
92.1
81.4
89.4
90.0
89.6
Intern ZZuu 2025
14B
92.0
94.4
87.4
92.1
81.5
89.3
Cog Grounding GrgguddnG Wang al.
2024
17B
92.8
89.0
88.7
92.9
83.4
89.8
90.8
90.3
Zhu VL.
78B
93.4
95.4
90.1
93.8
85.3
91.5
91.4

---

Published as a conference paper at ICLR 2026
Multi-Task Scalability. Joint training across tasks consistently improves performance, indicating strong cross-task generalization. To evaluate multi-task performance and analyze how performance scales with the number of tasks, we train PaDT jointly across all benchmarks, i.c., RefCOCO/+/g, COCO, and RIC, resulting an enhanced multi-task variant denoted as PaDT Pro. Unlike task- specific PaITT models, PaDT Pro seamlessly switch between tasks by simply altering the prompt.
4.1 VISUAL PERCEPTION UNDERSTANDING TASKS
Referring Expression Comprehension. The Referring Expression Comprehension (REC) task evaluates MLLMs ability to localize objects given natural language descriptions, where predic- tion considered correct its IoU with the ground-truth box exceeds 50%. As shown in Tab. PaDT and PaDT Pro achieve state-of-the-ar performance both 3B and 7B scales. In particular, PaDT Pro (3B) obtains 96.0/95.5/955 on RefCOCO, 91.8/94.8/88 on RefCOCO+, and 93.6/94.0 on RefCOCOg, surpassing all previous MILM methods. The overall average of PaDT Pro (3B) reaches 93.6, which is further boosted to 94.5 with the 7B model. Remarkably, both PaDT and PaDT Pro (3B) already outperform the much larger 78B Intern model. These results demon- strate the effectiveness of the visual reference token paradigm, which substantially aligns textual semantics with image patches and thereby improves the precision of object localization MLLMs.
Referring Expression Segmentation. Similar to REC, the Referring Expression Segmentation (RES) task evaluates an MLLMs ability to segment the target object mask given natural language description. We adopt cloU the evaluation metric, and results are reported in Tab. 2 Both PaDT and PaDT Pro achieve the best performance compared with existing methods, even against ap- proaches such Seg-RI and Text4Seg+SAM that leverage the powerful SAM segmentation model. With the lightweight PaDT decoder that translates unified visual reference tokens into segmenta-
Table 2: Results of referring expression segmentation task RefCOCO/+/g datasets.
RefCOCO
RefCOCO+ val
RefCOCOg
Model Name
Param.
Overall
val
testA
testB
testA testB
val
test
X-Decoder (Zou et al. 2023a
64.6
SEEM (Zou et 20230
65.7
Seg-RI You & Wu 2025
3B
69.9
76.0
64.9
59.1
66.8
50.9
67.3
67.9
65.4
PaDT (Ours)
77.4
74.7
72.7
75.1
69.3
70.5
71.1
73.4
PaDT Pro (Ours)
3B
81.3
81.5
82.2
77.6
79.4
76.3
78.1
78.5
79.4
LAVT et al. 2023
72.7
75.8
68.8
62.1
55.1
65.0
66.0
66.7
LISA Lai et al.
2024
7B
76.5
71.1
62.4
67.5
56.5
66.4
68.5
67.9
PixeILM Ken et 2024
7B
73.0
76.5
68.2
66.3
71.7
58.3
69.3
70.5
69.2
OMG- (Zhang et al
2024c
7B
75.6
77.7
71.2
65.6
69.7
58.9
70.7
70.2
70.0
Seg-R1 You & Wu 2025
7B
74.3
78.7
67.6
62.6
70.9
57.9
71.0
71.4
69.3
Text4Seg CRF (L.n
2025
71.3
73.7
69.6
65.9
70.4
61.9
69.3
69.3
68.9
Text4Seg SAM Lan al.
2025
78.0
80.9
74.6
71.6
77.3
66.0
74.8
74.7
74.7
PaDT (Ours)
7B
78.5
79.8
77.3
75.0
77.7
71.3
73.0
73.9
75.8
PaDT Pro (Ours)
7B
86.0
86.1
86.4
82.5
84.1
80.7
83.5
83.3
84.1
Table 3: Results of oocn-vacyyylrry detection task the whole COCO2017 validation set.
Model Name
Param.
AP AP@[50:95]
AP@50
AP@75
AR @R@[50:95]
AR@50
AR@75
VL. Zhu al. 2025
2B
6.9
11.2
7.0
14.9
20.8
15.6
Qwen2.5-Vi Bai
2025
3B
13.7
22.1
14.2
21.8
30.5
23.3
Qwen2.5-Vi VL-LIF (Sten
2025
3B
17.1
27.5
17.3
25.4
35.6
26.4
VLM-R1 (Shen al. 2025
3B
19.2
33.1
19.0
32.2
46.9
33.6
PaDT (Ours)
3B
34.0
51.2
35.8
38.5
56.1
40.4
PaDT Pro (Ours)
3B
38.2
54.9
40.5
43.9
60.6
46.4
Qwen2.5-VL Bai et al.
17.9
28.1
29.3
LLaVa-NeXT [Lau et al. 2024b
7B
1.3
LLaVa-One Vision
2024
2.2
1.1
4.1
3.2
In3erVL33 Zhu et al. 2025
18.2
28.0
29.7
PaDT (Ours)
36.5
53.8
38.4
41.5
PaDT Pro (Ours)
39.0
56.2
41.5
44.8
61.8
47.6
8

---

Published as a conference paper at ICLR 2026
Table Results of referring image captioning task RIC validation set.
Text Metries
Detection Metries
Model Name
Param.
CIDFr-D
Meteor ROUGE-I. BL.EU-4
GP
GR
LLa Vision (Li al.
2024
0.5B
0.058
0.088
0.185
0.052
5.2
0.5
VL3 (Zhu al. 2025
2B
0.315
0.230
0.374
0.284
42.4
18.2
Qwen2.5-Ve VL. (Bat ct al. 2025
3B
0,386
0.241
0.369
0.261
61.8
6.2
PaDT (Ours)
3B
1,450
0.304
0,501
0.467
81.6
45.4
PaDT Pro (Ours)
3B
1.412
0.300
0.495
0.458
82.3
45.1
LeT [Liu et al.
2024b
7B
0.262
0.200
0.335
0.178
54.3
10.6
LLaVa-One Vision (Li al.
2024
7B
0.172
0.207
0.330
0.182
32.5
10.2
Qwen2.5-VL (Bai ot 2025
7B
0,266
0.251
0,369
0.257
60.8
19.8
InternVL3 (Zuu al. 2025
8B
0.208
0.207
0.373
0,249
56.6
32.1
LLaVa-NeXT (Liu et al. 2024b
13B
0,283
0.212
0,347
0.172
55.7
6,2
PaDT (Ours)
7B
1.445
0.304
0,500
0.466
77.0
45.2
PaDT Pro (Ours)
7B
1.387
0.299
0.491
0.449
82.3
45,8
Table 5: The ablation study of the proposed components in PaDT.
Visual Reference Token
Training Strategy
REC
RES
using VRTs
Sep
Task Decoder
Crohust CF
VRTs Selection
RefCOCO val
RefCOCO val
88.7
PaDT Decoder
91.1
72.1
PaDT Decoder
92.0
75.2
PaDT Decoder
All VRTs
76.5
69.5
PaDT Decoder
All VRTs
49.1
19.8
PaDT Decoder
93.2
76.1
tion masks, our models consistently outperform prior baselines. Additional qualitative examples are provided in the Appendix A.8
Open-vocas voca-vocyyylary Detection.
This is fundamental visual perception task that evaluates MLLMs ability to perform semantic grounding. As shown Table most existing MLLMs struggle with this task, showing low precision and recall. For instance, Qwen2.5-VL (3B) achieves only 13.7 mAP, and VL3ernVL3 (8B) reaches 17.5 mAP on the COC02017 validation set. Our PaDT and PaDT Pro substantially advance the state of the art. PaDT Pro (3B) achieves 38.2 mAP, while the 7B variant further improves to 39.0 mAP, nearly doubling the performance of prior best methods. These gains highlight the effectiveness of visual reference tokens in strengthening semantic association and object localization.
Referring Image Captioning. To validate both the visual understanding and grounding ability, conduct experiments on our RIC dataset. As shown in Table 4 PaDT and PaDT Pro (3B) deliver strong improvements, reaching 1.45 CIDEr, 0.304 Meteor, 0.501 ROUGE-L, 0.467 BLEU-4, and top detection scores of 82.3% Greedy Precision (GP) and 45.1% Greedy Recall (GR). The 7B models further extend performance, with PaDT Pro (7B) maintaining competitive caption quality, i.. 1.39 CIDEr, while achieving the best detection-oriented scores, i.e. 82.3% GP, 45.8% GR. These results suggest that PaDT generates not only fluent captions, but also semantically precise ones grounded in localized visual content.
4.2 ABLATION EXPERIMENTS
Ablation study of Proposed Components in PaDT. We conduct detailed ablation studies Tab. using the 3B model with the following observations. i) The first row without VRTs corresponds to supervised fine-tuning on Qwen2.5-VL. directly predicting bounding box coordinates. By inte- grating VRTs with robust CE loss and random VRTs selection, we observe noticeable improvement in REC (detection task) and RES (segmentation task) being enabled. ii) We further notice that both projection module fep and robust CE loss necessary for achieving improved performance. iii) Al- ternative choice of including all foreground VRTs during training may even harm the performance, probably due bias towards high density regions.

---

Published as a conference paper at ICLR 2026
Figure 5: The illustrations of the mask generations. Table 6: Performance of using SAM2-L mask refiner with types of prompts.
point
box
mask
RefCOCOg val
70.5
69.9
V
74.1
74.0
74.9
76.3
Table 7: The generalization analysis and finetuming result of PaDT on COCO2017 validation set.
Model Name
Objects365
COCO2017
AP@(50:95]
AP@50
AP@75
AR@[50:95]
AR
AR@75
Qwen2.5
14.2
PaIYT (Zero Shot)
16.9
23.7
18.0
21.5
30.6
22.7
PaDTT (Task Specific)
34.0
51.2
35.8
38.5
56.1
40.4
PaIYT (Fine)Tuned)
36.5
52.2
38.8
41.3
57.4
43.6
Effectiveness of Mask Refinement with SAM2-L. We further analyze the compability PaDT output with segmentation foundation model, SAM2-L under three schemes. i) Given the VRTs generated by PaDT, we extract their coordinates as point prompts to SAM2-L, denoted as point. úi Using the bounding box and mask generated by PaDT, respectively, as prompt for SAM2-L. We explored different combinations with results Tab. First, we observe that using point prompt fails to improve upon PaDT, due the sparse prior information. However, both box and mask prompts are conducive to further improving the results under the help of SAM. Combining multiple prompts yields more significant improvement. Visualizations in Fig. cooorobbratee these findings. The results suggest the segmentation performance can be further enhanced with expert foundation model at the expense of additional inference cost.
Effectiveness of Pretraining and Task-specific Finetuning. To evaluate the generalization and data-scaling properties of the PaDT framework, we pretrain on Objects365 (Shao al. 2019)) and subsequently finetune on the COCO dataset. As shown in Tab. 7 PaDT exhibits stronger zero-shot performance than the Qwen2.5-VV VL base model, and finetuned version consistently outperforms direct training on task-specifie data.
5 CONCLUSION
In this work, we proposed Patch-as- -ethoa---e Token (PaDT), unified paradigm that equips MLLMs with the ability to generate both textual and visual outputs through Visual Reference Tokens (VRTs). By dynamically embedding VRTs into the LLM output space, PaDT ensures semantically coherent and visually grounded predictions, overcoming the inefficiency and misalignment issues of prior codebook-basee further introduced to enable visual perception and understanding tasks within PaDT. Extensive ex- periments across detection, segmentation, grounding, and captioning demonstrate state-of- siitee-ffthh-eee of-thh-art performance, highlighting directly predicting visual tokens as an effective and scalable paradigm toward general-purpose multimodal reasoning systems.
methods. A light-weight decoder and an effective training strategy
ACKNOWLEDGEMENT
This work supported in part by the Guangdong Provincial Key Research and Development Projects (2024B0101040004) the National Research Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: AISG-NMLP-2024-0044) the Guang- dong Provincial Key Laboratory of Human Digital Twin (2022B 1212010004) the NUS Start-up Grant (0010106-00-000 and the National Natural Science Foundation of China (62320106007). Any opinions, findings and conclusions or recommendations expressed in this material are those of author(s) and do not reflect the views of National Research Foundation, Singapore.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We affirm that all authors have read and adhered to the ICLR Code of Ethics. Our research does not involve human subjects, personally identifiable data, sensitive information. The datasets used are publicly available and cited appropriately. We have considered potential risks, including issues related fairness, privacy, and security, and have taken steps to mitigate any possible negative impact. No conflicts of interest or external sponsorship have influenced the work. We commit to respecting research integrity and legal compliance throughout the research process.
REPRODUCIBILITY STATEMENT
We are committed ensuring the reproducibility of our results. The main text provides detailed description of our proposed method and experimental setup, including all hyperparameters, datasets, and evaluation protocols. Additional results and dataset details are included in the appendix. We also provide the detailed process for constructing the Referring Image Caption dataset in the appendix. We will release of our implemented code and reproduction instructions to further support the reproducibility of our findings.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Techni- cal Report. arXiv preprint arXiv:2303. 08774, 2023.
Bean-Baptiste Alayrac, Giorgio Donato, and et al. Flamingo: a Visual Language Model for Few-Shot Learning. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. QW Technical Report. arXiv preprint arXiv:2502. 2025.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Detection with Transformers. European Conference on Computer Vision (ECCV), 2020.
Chi Chen, Ruoyu Qin, Fuwen Luo, Xiaoyue Mi, Peng Li, Maosong Sun, and Yang Liu. Position- Enhanced Visual Instruction Tuning for Multimodal Large Language Models. arXiv preprint arXiv:2308.134 113308113433 2023a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic. arXiv preprint arXiv:2306. /5/95, 2023b.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, al. Long Scaling Cong-Context Visual Language Models for Long Videos. arXiv preprint arXiv:2408. 2024a.
Zhangquan Chen, Xufang Luo, and Dongsheng Li. VisRL: Intention-Driven Driven Visual Perception via Reinforced Reasoning. arXiv preprint arXiv:2503.07523, 2025.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, al. Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguisie Tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14185-241999 2024b.
Tri Dao. F Faster Attention with Better Parallelism and Work Partitioning. arXiv preprini arXiv:2307.086/0 2023.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling Vision Transformers to 22 Billion Parameters, In International Conference on Machine Learning, pp. 7480-7512, PMLR, 2023.

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv preprint arXiv:2010. 1929, 2020.
Yuxin Fang, Wen Wang. Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang. Xinlong Wang, and Yue Cao. EVA: Exploring the Limits of Masked Visual Representation Learning at Scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19358-19366 2023.
Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. LLaVA-UHD: LMM Perceiving Any Aspect Ratio and High- Resolutonn Images. In European Conference on Computer Vision, pp. 390-406. Springer, 2024.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Language Representation Learn- ing With Noisy Text Supervision. In International Conference on Machine Leurning, pp. 4904- 4916. PMLR, 2021.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. MANTIS: Interleaved Multi-Image Instruction Tuning, arXiv preprint 2xxiv22455.01488 2024a.
Qing Jiang, Gen Luo, Yuqin Yang. Yuda Xiong, Yihao Chen, Zhaoyang Zeng, Tianhe Ren, and Lei Zhang. ChatRex: Taming Multimodal LLM for Joint Perception and Understanding. arXiv preprint arXiv:2411.18363, 2024b.
Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin CHEN, Chengru Song, dai meng, Di ZHANG, Wenwu Ou, Kun Gai, and Yadong MU. Unified language- vision pretraining in LLM with dynamic discrete visual tokenization. In The Twelfth International Conference on Learning Representations, 2024.
net / forum?
id=Fllt]JBBOg
Aishwarya Kamath, Mannat Singh, Marcus Rohrbach, Nicolas Carion, Vedanuj Sharma, Zhecan Peng, Chunjie Li, Yuwei Kim, Alaaeldin El-Nouby, Ali Hassani, and et al. MDETR - Modulated Detection for End-to-End Multi-Modal Understanding. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Courtney Rolland, Lucas Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, and et al. Segment Anything. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2023.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: Rea- soning Segmentation via Large Language Model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Mengcheng Lan, Chaofeng Chen, Yue Zhou, Jiaxing Xu, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Text4Seg: Reimagining Image Segmentation Text Generation. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openzeview. net. wwt./t/orrmd.ddaaaaae.....
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li. Ziwei Liu, al. LLavA-OneVision: Easy Visual Task Transfer. arXiv preprint arXiv:2408.0 03326, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language- -maal Pre- training for Unified Vision-Language Understanding and Generation. In International Conference on Machine Learning, pp. 12888-12900. PMLR, 2022a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP-2: Bootstrapping Language- Pre- Pre-training with Frozen Image Encoders and Large Language Models. In International Confer- on Machine Learning (ICMI.,, 2023.
12

---

Published as a conference paper at ICLR 2026
Xiang Li, Wenhai Zhang, Yue Cao, Han Hu, Jinhui Guo, Haibing Ling, and Wei Fudan. Grounded Language-Iman Pre-training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Yi Li, Hualiang Wang. Xinpeng Ding, Haonan Wang, and Xiaomeng Li. Token Activation Map to Visually Explain Multimodal LLMs, 2025. URL https://arxx: wtwpvvv.rr/bbb/2556
23270
Long Lian, Yifan Ding, Yunhao Ge. Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming- Yu Liu, Trevor Darrell, Adam Yala, and Yin Cui. Describe anything: Detailed localized image and video captioning. arXiv preprint arXiv:2504. 16072, 2025.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning. Peng Jin, and Li Yuan. Video-LLaVA: Learning United Visual Representation by Alignment Before Projection. arXiv preprint arXiv:2311.10122 2023a.
Fangjian Lin, Jianlong Yuan, Sitong Wu, Fan Wang, and Zhibin Wang. UniNeXt: Exploring A Unified Architecture for Vision Recognition. In Proceedings of the 31st ACM International Con- ference on Multimedia, pp. 3200-3208, 2023b.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitncck. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. Advances in Neural Information Processing Systems (NeurIPS), 36:34892-34916, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual In- struction Tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, January 2024b. URL https:
v1.
01 30 1lava Salavvanee next/
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, al. LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents. In European Conference Computer Vision, pp. 126-142. Springer, 2024c.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding DINO: Marrying DINO with Grounded Pre- ree-Trainngg for Open-Set Object Detection. European Conference on Computer Vision, pp. 38-55. Springer, 2024d.
Yuqi Liu, Tianyuan Qu, Zhisheng Zhong, Bohao Peng, Shu Liu, Bei Yu, and Jiaya Jia. Vision- Reasoner: Unified Visual Perception and Reasoning via Reinforcement Learning, arXiv preprint arXiv:2505. /2081, 2025a.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-RFT, Visual Reinforcement Fine- Tuning, arXiv preprint arXiv:2503.0177 01785, 2025b.
Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and Quick: Efficient Vision Languu-Langugge Instruction Tuning for Large Language Models. Advances in Neural Information Processing Systems, 36:29615-29661 2023.
Tianren Ma. Lingxi Xie, Yunjie Tian, Boyu Yang, and Qixiang Ye. ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https:///////...... httEp::///penzevie.. net Forrmmiiic TOtk
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and Comprehension of Unambiguous Object Descriptions. In Proceedings of the IFFF/CVF Conference Computer Vision and Pattern Recognition, pp. 11-20, 2016.
13

---

Published as a conference paper at ICLR 2026
Bo Peng, Dongdong Yu, et al. Kosmos-2: Grounding Multimodal Large Language Models to the World. arXiv preprint arXiv:2306. 148244 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning, pp. 8748-8763. PmLR, 2021.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. GLaMM: Pixel Grounding Large Multimodal Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13009-13018, 2024.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You Only Look Once: Unified, Real-Time Object Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time
Object Detection with Region Proposal Networks. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
Tianhe Ren, Jinyu Zhao, Shilong Yang, Feng Li, Xiang Li, and Han Hu, Grounding DINO: Marrying DINO with Grounded Pre- Pre-Training for Open-Set Object Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. PixelLM: Pixel Reasoning with Large Multimodal Model. In Proceedings of the IEEE/CVF 2024.
Conference on Computer Vision and Pattern Recognition, pp.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8430 8439, 2019.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. VLM-RI: A Stable and Generalizable RI-style Large Vision-Language Model. arXiv preprint 2501015 2025.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang. Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation. arXiv preprint 06655: 2024.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang. Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative Pretraining in Multimodality. arXiv preprint arXiv:2307.05222, 2023.
Chameleon Team. Chameleon: Mixed-Modal Early-Fusion Foundation Models. arXiv preprint 2000022005.0988, 2024.
Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities, arXiv preprint arXiv:2305. 11172, 2023a.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, al. CogVLM Visual Expert for Pretrained Language Models. Ad- vances in Neural Information Processing Systems, 17:121455-,,149, 2024.
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, al. Internlmage: Exploring Large-Scale Vision Foundation Mod- els with Deformable Convolutions. In Proceedings of the IEEE/CVF Conference on Computer 2023b.
Vision and Pattern Recognition, pp.
Shiyu Xuan, Qingpei Guo, Ming Yang, and Shiliang Zhang. Pink: Unveiling the Power of Refer- ential Comprehension for Multi-modal LLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13838-13848 2024.
14

---

Published as a conference paper at ICLR 2026
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mPLUG-OwL Modularization Empowers Large Language Models with Multimodality. arXiv preprint 2rX1v.230..1417, 2023.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In The Twelfth International Conference on Learning Representations, 2023.
Zuyao You and Zuxuan Wu. Seg-rl: Segmentation can be surprisingly simple with reinforcement learning, arXiv preprint arXiv:2506.22624, 2025.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling Context in Referring Expressions. In European Conference on Computer Vision, pp. 69-85. Springer, 2016.
Ya-Qi Yu. Minghui Liao, Jiwen Zhang, and Jihao Wu. TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens. arXiv preprint arXiv:2410.05261, 2024.
Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel Understanding with Visual Instruction Tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28202-28211 2024.
Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, and Yinfei Yang. Ferret-v2: An improved baseline for referring and grounding with large language models. In First Conference on Lan- guage Modeling, 2024a. /RLLatttss//ppeneeeeee..ct wttp:/eeeee/oomm//-/22------
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. GPT4Rol: Instruction Tuning Large Language Model on Region-of- In European Conference on Computer Vision, pp. 52-70. Springer, 2024b.
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shumping Ji, Chen Change Loy, and Shuicheng Yan. OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding. Advances in Neural Information Processing Systems, 37:71737-71767, 2024c.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhanc- ing Vision-Language Understanding with Advanced Large Language Models. arXiv preprint arXiv:2304. 10592, 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv preprint arXiv:2504.1044 2025.
Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized Decoding for Pixel, Image, and Language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15116-15127, 2023a.
Xueyan Zou, Jianwei Yang. Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment Everything Everywhere All at Once. Advances in Neural Information Processing Systems, 36: 19789-997882,, 2023b.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 REFERRING IMAGE CAPTIONING (RIC) DATASET
A.1.1 DATASET CONSTRUCTION
Image captioning is fundamental benchmark for evaluating the vision understanding ability of MLLMs. In the conventional setting, given an input image, the model generates pure textual de- scription that summarizes the main subject and its activity, trained large-scale image-text pairs. However, such descriptions provide little supervision regarding object-level grounding, making difficult to assess whether the model accurately captures the spatial locations of entities. To ad- dress this limitation, we re-annotate the COCO2017 dataset with more fine-grained annotations and propose Referring Image Captioning (RIC) dataset. Specifically, (prompt provided below) and feed each image together with its box-level annotations, instructing the model produce enriched captions that explicitly reference the corresponding box IDs. Examples of the resulting training samples are shown in Fig.
leverage Qwen2.5-V1L-22h
Prompt for generating RIC dataset
Prompt: <image"> Please describe the image, focusing on the main objects (instances) present. After mentioning an object in the caption, immediately append its instance ID(s) parentheses, using the format (<box_ie id: ID/> /)>>). You MUST ONLY use the instance IDs provided in the list below. Do NOT invent make up any IDs, If there multiple instance IDs for the same object, list all IDs within the same parentheses, separated by commas. For example: "A man (<box id: ID/>) is standing by table (<box id: ID/>). Two ovens (<box id: ID/>, <box id: ID/>) are visible." Here are optional instances in the image (id: ID, format: category: name, bhox: [x1, y1, x2, y2]):
200887, person, [389, 70, 498, 348]
(list all annotations within this query image.)
IMPORTANT: Only use the instance IDs listed above your caption. Do NOT invent or make up any new IDs. Generate a fluent English caption that includes the main objects and their instance IDs specified.Yor You don't have to include all instances from the list above, they optional: only describe the main elements of the image. The caption should be more than 100 words and should not contain any information irrelevant the image.
Qwen2.5-VL-728 (Example) A bustling kitchen scene features a person (<box id: 200887/>) working near a sink (<box_id: 2196309/>>, The dining table (<box_id: 119568/>) holds vari- ous cooking items, including a bowl (<booxid:: 713388/>)
bottle (<box_id: 82445/>), Two ovens (<hox id: 1125079/>. <box_id id: 2139366/ >> are visible in the background. Another person (<hox id: 1218137/>) appears partially on the left. The table also has cup (<hox id: 1883614/>) and knife (<box id: (93231/>). The kitchen is equipped with hanging pots and well-used workspace, emphasizing busy cooking environment.
A.2 TEMPLATE EXAMPLE FOR EACH VISION TASK
Here present the interaction templates used in PaDT framework, covering open-vocabulary detection, referring expression comprehension, and referring image captioning tasks. The prompts designed to be concise and efficient, allowing PaDT to naturally interleave textual tokens and VRT predictions during task training.
Open Vocabulary Detection Task
USER: <image>> Please carefully check the image and detect the following objects: [[target list)]. ASSISTANT: In this image, there are (number) "[category]" }" (<VRT 0> VRT 1><VRT 2> <VRT 3> VRT 4>, < VRT 5><VRT 6> VRT 7> VRT 8>< VRT 9>,,.. and
<RT
Referring Expression Comprehension Segmentation Task
USER: <'mmag>>>lease carefully check the image detect the object this sentence describes: "[question)" )".
ASSISTANT: The "[question]" refers to <VRT this image.
in
16

---

Published as a conference paper at ICLR 2026
A man (+O+) rides motorcycle (<1> on a dirt path. He is wearing red shirt, black shorts, and helmet. The motorcycle has rugged design with box. The background features lush greenery, large mountain partially covered mist, and a bridge spanning the scene, Another person (+2x) and a bicycle (<3+) are visible in the distance, adding to the rural setting. The atmosphere is serene, with natural elements dominating the landscape.
storage
woman (+O+) stands a kitchen, holding a cat (<1>). She with a bowl of oranges (+3>) and a bottle (+4+), A refrigerator (+5x) visible to her right. handbag (<6>) rests on the floor by her feet. The kitchen features wooden cabinets and stove with pan (s75). A potted plant (+B>) and clock (c9>) adorn the counter. The scene
A man (+O+) is riding bicycle (d>) on path
train
sink (<2>)
(+2>). The train, with red and white design, is positioned on behind the cyclist. stop sign (<3x) is visible to the right of the man, and overhead electrical lines run above the train. The set in well-lit area with some greenery, along the path. The overall atmosphere suggests calm, possibly suburban environment.
tracks
and homey.
A young girl (<O>) is eagerly eating from bowl (<1) on dining table (<2>). She uses a spoon (<3>) to scoop food, which appears to be melting. The table is covered with red-checkered tablecloth, In the background, an adult (<4>) sits with arms crossed, and a wine glass (<5>) is partially visible. The scene is illuminated by natural light, creating warm and casual dining atmosphere.
A woman (<O>) is eating from spoon (<1>) in kitchen. She holds a plate of food, including what appears to be broccoli. Behind her, there is an oven (<2>) and a microwave (<3>). A bottle (<4>) is visible on
the counter near the oven. The setting is a
<>>
<3>
home kitchen with light-colored cabinets and a tiled backsplash. The woman wearing glasses and blue top.
Figure 6: The training samples of RIC dataset. For each image, there are 3-5 captions, in which ask the MLLMs to refer to the object (via bounding box or visual reference tokens) following each object's subject.
Referring Image Captioning Task
USER: <image>>Plcan deseribe this image.
ASSISTANT: {object1]
{object2]
17

---

Published as a conference paper at ICLR 2026
A.3 PROMPT USED FOR COMPETING METHODS
To guide MILMs (e.g.. Qwen2.5-VI. (Bai et al. 2025), Vn. (Zhu et al., 2025), and the LLaVA series (Liu al., 2024ch) in predicting bounding box coordinates each task, we append a box- specific and format- -preaf- instruction to the task prompt, detailed below.
Open Vocabulary Detection Task (with box and format instruction)
USER: <image>>Plass carefully check the image and detect following objects: [target list)]- Output each detected target's bbox coordinates ISON format, For example, "jjson [["bhox 2d": [x1, yl, x2, y2], "label": "target name")) targets are detected in the image, simply respond with None.
Referring Expression Comprehension Segmentation Task (with format instruction) USER: <image>> PPecase carcfully check the image and detect the object this sentence describes: "(question)". Output the final answer in JSON format.
Referring Image Captioning Task (with box instruction) USER: <image>> >Please describe this image. You should include the corresponding bounding box of the objects within the sentence. For example, "In this image, cat ([x1, yl. x2, y2]) is sitting the wooden table ([x1, y1, x2, y2D,
A.4 THE FORMULA OF THE TASK-SPECIFIC LOSSES ON THE PADT DECODER OUTPUT
Let Bored E IR Lx4 denote predicted bounding boxes with ground truth Bst .Mrred R LxxHx predicted masks with ground truth Mst and Spred R2x1 predicted confidence scores with ground truth Sat, The Laboz Lmaak and Lecore objectives are:
Libon = T Σ Liou soppred,By + || BB ||1,
(8)
1 L T,
L
Lmaak
Σ
Carce( (Mprred ,M Σ
C rocal (Mpreed Mi*))
(9)
1 Σ ||Spred 1/
L Lecore
(10)
L
A.5
ADDITIONAL. ABLATION STUDY
A.5.1 TOKEN ACTIVATION MAP ANALYSIS
We provide additional Token Activation Map (TAM) visualizations, illustrated in Fig. compar- ing Qwen2.5-VL and the PaDT Pro 7B model, showing that visual reference tokens establish much stronger associations with target image patches than digit-by-digh coordinate predictions. These results further highlight the robust semantic alignment and precise object localization achieved by visual reference tokens.
A.5.2 ABLATION STUDY OF OTHER USED LOSSES
As shown in Table 8 we conduct ablations on the loss components Lsboz, Lmask, and Lscore- PaDT achieves the best average performance when all visual task losses are combined. In particular, removing the dynamic embedding module omitting any individual loss (Lmask, Coboz: L L consistently degrades performance on both referring expression comprehension and segmentation. Notably, using all components yields the highest accuracy (93.2% REC and 76.1 mask cloU) and the strongest multi-task ability, underscoring that each module and loss is essential and complementary for optimal task performance.
18

---

Published as a conference paper at ICLR 2026
Batter 2ns up on
Big One
Right Toilet
55 Bus
All of the Pizza d
Left
Qwen2 5- 78
Paoe
Plowe menieel rhack
Plose
dlckk datees the abjact this
ped Booes Talg
331, 430
990 400
T0ll the plan
PaDT Pro 7B
[ Plase clock aad Dotect sboaet
cluck
Plaas
Back
skouet
and
The 50
relry 1603
of the plaz of rnfere
FR 26
Figure More TAM visualizations of Qwen2.5-VL and our PaDT Pro 7B models.
Table 8: Ablation study of each individual components (with the analysis of additional losses).
Dynamic Embedding Module
Visual Task Loss Lisuue
Detection RefCOCO val (REC)
Segmentation RefCOCO (RES)
VRT
fup
Lreask
Cacore
val
88.7
V
91.1
27.5
72.1
V
91.7
32.3
78.0
92.7
24.4
75.2
V
76.1
A.5.3 ABLATION STUDY OF THE NUMBER OF SELECTED VRTs PER TARGET
We analyze how the number of selected visual patches per target impacts performance. As shown in Table 9 inereasing the number of patches from 5 steadily improves both bounding box accuracy and mask cloU across all datasets. The best results are obtained with 5 patches per target, while further increasing to patches yields diminishing or even negative returns. This indicates that moderate number of representative patches provides richer representations, whereas excessive patches introduce noise and redundancy, leading to unstable training PaDT.
We also investigate the case of using all foreground patches ground-truth VRTs during training. As shown Fig. X this configuration produces the worst results. Although the number of output VRTs increases, the PaDT decoder exhibits clear performance degradation. We attribute this to the redundancy (that makes the PaDT hard to predict all VRTs at the inference stage) and low resolution of patch-level features: when all foreground patches are used, the decoder is forced to decode trivial and overlapping regions, which prevents from learning accurate target boundaries and masks, especially when only limited number of VRTs are predicted at inference. Consequently, selecting a moderate number of informative patches proves more effective than training with all foreground patches.
A.5.4 ABLATION STUDY OF DIFFERENT SAMPLING STRATEGY
We present detailed comparison among different sampling strategies, including random sampling (18 patches), using all foreground patches, and border-aware sampling (four tokens from left, top, right and bottom boundaries). The results summarized in Table
19

---

Published as a conference paper at ICLR 2026
Table 9: Ablation study of the number of selected visual patches per target and different sampling strategy.
#Patches Target
1
3
5
ALL
Border-aware Sampling
Bbox A.c
92.4
93.2
93.2
92.6
49.1
92.1
RefCOCO val
Bhox Acc@0.75
82.7
86.1
87.1
85.9
15.5
Mask cloU
67.3
75.2
76.1
75.7
19.8
70.9
Bhox Acc@0.5
87.5
88.1
88.5
87.5
86.6
RefCOCO+ val
Bbox Aco@0.77
78.8
82.1
82.8
81.7
Mask cloU
63.7
71.4
72.7
71.6
66.9
Bhox Acc@0.5
88.1
88.2
88.2
86.8
87.0
RefCOCOg val
Bbox Acc@0.1)
78.7
80.7
81.1
79.9
Mask cloU
62.7
69.7
70.5
70.0
65.6
X PaDT trained with all foreground VRTs
PaDT trained with 5 random foreground VRTs
Please carefully check the image and detect the object this sentence describes: "white car on the leff"
Please carefully check the image and detect the object this sentence describes: "white
car on the left"
Please carefully
Please carefully check the image and detect the object this sentence describes: "front
check the image and detect the object this sentence describes: "front
bowl wcarrots in if'
bowl wcarrots it'
Please carefully check the image and detect the object this sentence describes: *tal suitcase".
Please carefully check the image and detect the object this sentence describes: "tall suitcasé".
Figure 8: Qualitative analysis between training PaDT with all foreground VRTs and 5 randomly selected foreground VRTs.
20

---

Published as a conference paper at ICLR 2026
We make the following key observations:
Using all foreground patches as ground-truth VRTs leads performance collapse. When all foreground patches are provided during training. the task decoder tends to overfit the ground-truth VRTs and relies heavily on the MILM's predicted VRTs during infer- ence. As the decoder simply learns to produce trivial bounding boxes masks that cover all foreground areas, no longer needs to truly understand object boundaries, thus failing to generalize.
Random sampling consistently benefits performance. As the number of randomly sam- pled patches increases from to 5, the performance consistently improves. The best results achieved with 5 randomly sampled patches, indicating that this strategy strikes bal- ance between coverage and model generalization.
Boundary-aware sampling underperforms random sampling. Sampling exclusively from the four boundaries (left, top, right, bottom) yields weaker results. We hypothesize that boundary patches often contain ambiguous semantics, especially when segmentation annotations are unavailable. This increases training difficulty and again makes the task decoder overly dependent MLLM's predicted boundary VRTs.
A.6 SCALABILITY
IMAGES
A.6.1
COMPATIBILITY WITH HIGH-RESOLUTION IMAGES: YES, FULLY SUPPORTED
PaDT is fully compatible with high-resolution images and supports native resolutions. Our PaDT framework inherits from Qwen2.5-VL, and just like Qwen2.5-VL, it supports image inputs their original resolution. For instance, our experiments, we did not perform any resizing operations on training images and directly use their native resolutions.
COCO dataset: multiple resolutions, such as 640 480, 480 640, 640 573, 500 333, etc. (Tab. 123.
Objects365 dataset: high-resolution images such as 1024 727, 4608 3072, 768 1024, 5152 3864, etc. (Tab. 7
Table 7 further shows the results of training PaDT on Objects365 (with highly dynamic high- resolution inputs) and transferring to COCO dataset. Both Zero-Shot and Fine-Tuned results (mAP50) outperform Qwen2.5-VI. models on the same scale.
A.6.2 ADDITIONAL COMPUTATION OVERHEAD INTRODUCED BY PADT: VERY Low
High-resolution images naturally introduce more visual tokens, leading to increased computation and GPU memory, which is inherent to all MLLM models, including PaDT, Qwen2.5-VI, and InternVL.3. Importantly, the additional overhead introduced by PaDT, compared Qwen2.5-VL, is negligible. We quantify these additional costs as follows:
1. Number of Visual Tokens/ Patches: For an input image size H W. the number of VRTs is: #VRTs = h x w, where h = round(H H/2),, w = round(W/28), This is identical to the patch extraction process used Qwen2.5-VI. and Intern Thus, PaDT does not introduce new resolution-dependerr costs beyond standard visual encoder usage.
Dynamic Embedding Table:
Qwen2. 5-VL-7B Text Emioeddings: Memory: 152,, 064 3584
PaDT Dynamic Embedding Table: Memory: (152, 064 hw) 3584
Additional memory: hw 3584
Increasing rate hw 152, 064
For 1024 x 1024 image:
21

---

Published as a conference paper at ICLR 2026
w round (1024 28) 37
Extra memory = 37 37 3584 2 Bytes (bfloat16) = XB
Increasing rate - 0. 009
(1.e., <1&)
Projection Module Supt Negligible Cost:
LayerNorm:
Memory: 3584
2
Two Linear Pro Projections (W (W_A, W_B):
Memory: 3584
64
This overhead is less than 0.02% of the LLM backbone parameters (3B), thus negligible.
No Extra Overhead the LLM Forward Pass: VRTs are treated identically to text tokens during embedding lookup. Once embedded, they are processed by the LLM backbone without any additional computation. No structural modification or auxiliary branch is added to the forward pass.
5. PaDT Head Overhead: Let H RI.3584 be the backbone output:
Qwen2.5-VL- 5-VL-7B:
FLOPs 3584 152, 064
PaDT:
FLOPs L 3584 (152, 064 hw)
Increasing ng Rate: hw / 152, 064 (- 0. 009 for a 1024 1024 image)
Again, the overhead remains less than 1% even for high-resolution images.
The decoder consists of only three 2-way attention modules, with 95M parameters, significantly smaller than the 37B LLM backhone. Moreover, all VRTs are de- coded in a single forward pass, no iterative decoding is required.
Lightweight Decoder Head:
Overall: PaDT preserves the inference speed and memory footprint of standard multimodal LLMs. The only resolution-dependeer cost comes from visual patch extraction, which inherent to all high-resolution MILMs.
A.6.3 PADT IS EVEN MORE EFFICIENT THAN QWEN2.5-VL
Although PaDT introduces negligible overhead, it is more efficient during inference and training. This is because PaDT represents object with fewer tokens:
Qwen2. . 5LV
[100, 200, 300, 400] ' '1, ror, ' 17 tokens
PaDT:
VRT 01 VRI VRT VRI 3 VRT
takens
During inference (autoregressive decoding). PaDT saves 12 forward passes per object. During training. it reduces 12 forward tokens per object. These savings greatly outweigh the small mem- ory / computation overhead analyzed above. Therefore, overall, PaDT is both more efficient and more effective than Qwen2.5-Vi More quantitative analysis is shown as Tab. I0 we benchmarked inference RefCOCO val set (averaged over 100 samples) and scaled the images by 2x (c.g.,
448 x 644 > 896 x 1288).
A.7 COMPARISON WITH Q VL USING DIFFERENT POST- TRAINING STRATEGY
We compare PaDT with Qwen2.5-VL under different post-training strategies (i.e., SFT GRPO) on the specific datasets. The results in Table II and Table 12 show that: PaDT consistently outperforms post-trained Qwen2.5-Vi 5- across both tasks; 2. PaDT achieves superior zero-shot per- formance; and 3. PaDT demonstrates stronger transferability, as pretraining on Objects365 followed by finetuning COCO yields better results than training COCO alone.
22

---

Published as a conference paper at ICLR 2026
Table 10:
The quantitative analysis of computation cost and memory allocation for different image
resolutions.
Model
Image Resolution
A Whole Generation Process
Single-Pass Forward
Sequence Length
Peek Memory Allocation
Owen2.5 VL.
1x
1.127
0.027
42.22
86 MB
(+0.007 s)
(-22.78)
Qwen2.5-VL
2x
1,373
0,832 8 0,016 (+0.0L4
42,96
9,470 MB
PaDT
2x
0.905 (-0.468
(-2.4(223352)
9,446 MB (-21 MB)
Table 11: The results on Referring Expression Comprehension (REC) task.
Model Name
Setting
RefCOCO val RefCOCO+ val RefCOCOg val
Qwen2.5-VL LL
Zero-Shot
89.1
82,4
85.2
Qwen2.5-Vi VL
SFT
88.7
82.3
86,0
Qwen2.5-VI.
GRPO
(Shen al. 2025
90.1
84.2
85.6
PaDT PaDT-PPo
SFT
93.2
88.5
88.2
SFT
96.0
91.8
93.6
Table 12: The results Open- Viee-Vocabullry Detection (OVD) task.
Model Name
Setting
mAP@ [50:95]
Qwen2.5-VI.
Zero-Shot
13.7
PaDT
Zero-Shot (Pretrained on Objects365)
16.9
Qwen2.5-VL
SFT
17.1
Qwen2.5-VL
GRPO Shen et al. 2025
19.2
PaDT
Task- SFT
34.0
PaDT
Objects365 + COCO SFT
36.5
PaDT-Pro
38.2
A.8
QUALITATIVE EVALUATION
A.8.1 OPEN VOCABULARY DETECTION ON COC02017 DATASET
Comparison with representative MLLMs. In this section, we present qualitative results for open- vocabulary detection on the COC02017 dataset, comparing PaDT against representative MLLMs. As shown in Fig. 9 several key observations can be made.
Higher recall. PaDT consistently detects larger number of objects in the scene, demon- strating stronger recall. This improvement stems from its ability to directly predict visual reference tokens (VRTs) that are anchored to image patches, enabling more reliable cover- age of relevant objects.
Robustness in cluttered scenes. Competing MILMs, which predict serialized bounding box coordinates, often struggle in scenes with many repetitive or similar- objects. Their predictions may miss valid instances or collapse onto a few candidates, whereas PaDT maintains distinct references to multiple targets.
Avoiding invalid outputs. Existing MILMS occasionally fail to produce valid detections, labeled as Error in Fig. 9 such cases, the models tend to generate repetitive text se- quences until reaching the maximum output length, i.c. 2048 tokens. PaDT avoids this failure mode by grounding predictions directly in visual tokens rather than relying solely on
serialization.
Overall, these qualitative comparisons reinforce the advantages of PaDT: directly predicting visual tokens not only improves recall but also enhances robustness and stability open- vocabuuuyy de- tection.
23

---

Published as a conference paper at ICLR 2026
Image
GT
P Pro 7B
InternVL3 8B
Qwen2 5-VL 78
VLM-RI
Error
Error
Error
Error
Error
Figure 9: Qualitative comparison on COC02017 open-vocabulary detection. We compare PaDT with representative MLLMs including Intern and Qwen2.5-VL. Competing models frequently fail to produce valid outputs, leading to Error cases repetitive text generation. In contrast, PaDT achieves higher recall and correctly identifies multiple objects, even in cluttered scenes with repeti- tive instances. These results highlight the benefit of directly predicting visual reference tokens over serialized bounding box coordinates.
24

---

Published as a conference paper at ICLR 2026
Visualization of PaDT results REC/RES and OVD tasks. In Fig. 10 qualitative examples generated by the proposed PaDT framework. For Referring Expression Com- prehension (REC) and Referring Expression Segmentation (RES), PaDT first parses the query and identifies the corresponding target within the image. As illustrated in the top-left subfigure of each example, PaDTT generates five visual reference tokens (VRTs), each directly correlated with specific image patches and thus easily localizable. These VRTs are subsequently passed into the PaDT decoder produce the corresponding bounding box and segmentation mask. The overall pipeline is simple yet effective. Compared to character-by beaeteerbb-chhaacteee coordinate generation, PaDT requires far fewer tokens (only five VRTs per target) while providing stronger semantic and spatial grounding with respect to the object.
present extensive
Similar observations are made the veen-Vocabulary Detection (OVD) task. Unlike REC/EES, OVD requires PaDTT to predict multiple targets along with their category labels. In our response template, both categories and VRTs are naturally interleaved within the output sequence, enabling efficient multimodal reasoning. This training strategy strengthens the semantic alignment between text and image patches, thereby improving both precision and recall in detection task.
A.8.2 REFERRING IMAGE CAPTIONING ON RIC DATASET
Comparison with representative MILMs. In this section, we present qualitative results for open- vocabulary detection on the Referring Image Captioning (RIC) dataset, comparing PaDT with rep- resentative MI.LMs, including Intern 8B and Qwen2.5-VI. 7B models. As shown in Fig. []] PaDT exhibits clear advantages in both bounding box accuracy and object recall. Detailed qualitative comparisons are provided in the figure, further demonstrating the effectiveness of leveraging visual reference tokens bridge between high-level text semanties and low-level object localization.
Visualization of PaDT results on RIC task. We further present qualitative examples generated by the proposed PaDT framework. As shown in Fig. [2] visual reference tokens are automatically generated alongside the subject, illustrating a natural interleaving between semantic text and image patches. This design further enhances object-level alignment between textual descriptions and visual elements, thereby strengthening the co-reasoning ability across text and image modalities.
25

---

Published as a conference paper at ICLR 2026
Referring Expression Comprehension Segmentation
User Please conefully the
filly detecct
senticnos describes" "a dank gray couci.
exatence describse: pizzo with Fresh mocronell aud fewled eherry
grey
referm
Assiatunt The to pazza with fresh.
vRR L_2xx<20 R2440R 2504 2H1RR12290 2297 the mmage
mozzerello and halved cherry
176 199.//9T 22EHART (0DWART 150 this Image.
User: Please carefully check the image and detect the object this sentence describes: "a food with spoon and napkin next to if.
healthy
Assistant: The "a bowl of healthy food with spoon and napkin to <VR 170+VRT 183xV8T 22
refers
13>VRT__22 Nte-VR___2=xVV11 2_222222_x255555 this image.
Open Vocabulary Detection
In this inage, there are "person" (+VRT VRT B6XURI 42HYRT ADHURT 19h
1OHURT_32HVRT -URT 66 URT 67-URR 4 89, ART 36-VRT JOHART 32HVRT JO-VRT 9.), "cup"
(AVR T_ 242 24 222H-VRT CORTRRTTTT 177HURT VRR 244h <VRT (VRT_96-URT BAT.T/E-TTTTTTTTTEEE WETHTTHTEHBTTHHEEEE 329-VRT_293.)
3 "for (AVRT 2ETT_22500 325HVRT
URT VOR SES-VVRTT7666VT 276 S7THVRTT27700 277--RT 325-VAT 3253,
URT 326HART 302 278 URT VRT 350 MAW.WRT-39990- 395--99TTTT MT 373-VR7 373x), "cokee
3EST 38THVRT VRT 363HVRT -...49HHWTR.T
"char"
(VAR 150 128HURT VISWVRT 350-VVT JO6 <VR JORTLOOERRRT.. 78HVRT TOXVRT 77-VRT_99,
17 3ONVAT
60-VRT 38HVRT 3T
197+VRT 197-URT
175-VRT
197-)
table"
296HURT 2IRHRRT 1S0-VRR 2175,
V2HVTTT 666VTTTRRRRRR.. 66SHVRT_65))
tht page theee one Sorple
AA-AAT Aov JI1 Terage
There are "donut"
282-VRT_2633VATT
263-VRT_22
4XVRT 334HVRT 318
HET abwarr awaee HeueE paoeer VeT 30- Ct Wo Sowder 25)
<VRT_36I 36D.VRT VTT.VRRTTTTTRR 327HVRT_BB
V-VRT 347-VRT_
<RRT_ 247 247-VRT_213--6 213+VRT_24
8HVRT 215+VRT SD-VRT__300- 230+
<RRT 192 19E-VRT, 2115
<VRT_ 1755VRT_2266
<VRT_ <VRT_105 VOT=VVTTJ
XVRTT_ 57-VRT__
<VRT_ 62*VRT 63=VRTT VRVTTT661VV 61*V
78__8-VRT___ (_-77) image.
Figure 10: Qualitative visualization of PaDT generated examples on referring expression compre- hension/segmentation tasks.
26

---

Published as a conference paper at ICLR 2026
PaDT Pro
InternVL- 8B
Qowen2.55 5- 7B
mage grop women somil coam ore was the
white jody
durte dorts lodking toeas
beree with pontaa por
s-teaa nen (C0) and chariiik
cliulyy beyond
wode Hactes detten ploa
a
during with detert suauiing dil
Cob dol coroora cindl the
here bortee and
and Oud we
tode looks Lat CDs
ceeciig and semerd mocke
Figure 11: Qualitative comparison on the Referring Image Captioning (RIC) dataset. We compare PaDT with representative MLLMs. including InternVL3 and Qwen2.5-VL. PaDT shows clear ad- vantages in both bounding box accuracy and object recall over competing methods.
Referning Image Captioning
(URT GVRTL62HVRT 8I-VRT 8O-VRTT. 2-VRT_62-) is lounging on the hood of black Mercedes- Benz oar 187-1187-VR 205-V1 169--V B4H4VRT 183+). The car's 's's shity surface reflects
(_000) narre read, with hesses
bloking the
(ART 199 So LU PEORR
105-0R
buildings greanery. surrounding orange and white fur, appears relaxed as gazes into the distance. Behind the cor. a fence and lush bushes add touch of nature to the urbon setting. The scane coptures peaceful moment residenticl area.
col' 170 wer 17o-et 17O I7--RT
are stoncing carryy
others nor
the fesce The road lied with weeden fence
on side trees the other,
shodol.
poccerul intrrctioo on between the voride and
in cesttrys de conromeent
(VRR VRTT 189 145-VR
person
EHRTL
145--RT_ 168-) is driving horse-
Nig rig The lettws Fiet
drown racing cart. The horse 222__-V-VTT... 153 176-VV 306HVRT (__2))))))))))))))))))(()) is face, weoring cart and white, and the driver wearing a helmet and a yellow and blue jacket. They ane on a dirt track with grossy creas and advertisements background. The scere suggests harness racing
with
bridle and hameess.
toke
2440)
the
oeling portilyy oiver og the
meales and white baket
The horse motion, pulling cort steadily.
ploced
may
appeors
white
bird
(<VRT_ [<VR__99<VRT 79x 79.-VRTJ34x-VRT J344VRTT <VRT J16NYRT 153+) perches gracefully on the roof of a car (vVRT (CVRT_227-VR 227HVRT, 229HVRT 20 THVRT 208.VRT 2262). car parked a lot building and trees in the background. The bird's reflection is visible the shiny surface. Another car (vRRT 166--RTTTT 161-VRT 161 is partially visible in the distance.
In the image, senene rund scene unfolde with
horse (VRT 296-000 L8boya 200 22000 obstructed weeden
foneground, partially Another
(VRT 236x4a 235 anovver
25HyRT 212-) grezes poscefully the
thind horse [VRT__186 1866 ver 20awier woo
bockground.
a natiic store buiding.
fovRT 1851) is visble further back neor
backdrap odding o landocape dork, moody
<VRT J6b_)6T__666
features woathered
scene coptures serene moment on outdoor setting, with the bird as the focal point.
the otmospheric quality the while the ush green grass contrests
beautifully with the sarthy tores of the orchitecture and arimals.
Figure 12: Qualitative visualization of PaDT generated examples on referring image captioning task.
27


Reviews:
[{"review_id": "yNOX4FYWPK", "reviewer": "Reviewer_npQb", "summary": "This paper addresses a core limitation in current Multimodal Large Language Models (MLLMs): their inability to produce fine-grained visual outputs like bounding boxes or segmentation masks directly. Existing methods typically force the model to generate coordinates as a string of text (e.g., [x1, y1, x2, y2]), which is inefficient, prone to formatting errors, and creates a semantic gap between the visual features and the textual output.\nTo solve this, the authors introduce Patch-as-Decodable Token (PaDT), a new paradigm that allows an MLLM to directly output visual tokens that act as pointers to image patches. The key contributions are:\n1. Visual Reference Tokens (VRTs): The core idea is to create special tokens derived from the visual patch embeddings of an input image. These VRTs are dynamically added to the LLM's vocabulary for each specific image.\n2. Unified Input-Output: This dynamic vocabulary allows the LLM to seamlessly interleave text and VRTs in its output. For example, to describe a cat, instead of generating coordinates, the model might generate: \"Here is the cat <VRT_123><VRT_124><VRT_125>\".\n3. Lightweight Decoder: A simple, lightweight decoder then takes these few predicted VRTs and translates them into structured visual outputs like bounding boxes or segmentation masks.\n4. SoTA Performance: Through a robust training strategy, the authors show that PaDT achieves outstanding results across a range of tasks, including referring expression comprehension, segmentation, and open-vocabulary detection. Notably, their 3B parameter model significantly outperforms much larger models (e.g., 78B InternVL3) that rely on text-based coordinate generation.", "questions": "The proposed method is well-designed and directly targets the clearly articulated limitations of prior work. Please also refer to the weaknesses and hopefully the authors could address them. The paper is technically good. I would like to raise the scores if the authors can address my questions properly. \n\nBesides, some other citations are important, but the authors might be missing: \n\n1. Zhang, Haotian. \"Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models.\" arXiv preprint arXiv:2404.07973 3 (2024): 21. \n2. Lian, Long, et al. \"Describe anything: Detailed localized image and video captioning.\" arXiv preprint arXiv:2504.16072 (2025).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Novel and Elegant Paradigm: The core idea of dynamically creating and predicting patch-level visual tokens (VRTs) is a significant conceptual advance over generating text-based coordinates. It creates a more natural and semantically coherent link between the model's reasoning and the visual content.\n2. Empirical Performance: The results are a standout feature of this paper. The PaDT models not only set a new state-of-the-art on multiple challenging benchmarks but do so with remarkable efficiency. The fact that their 3B model outperforms an 78B competitor (Table 1) is a powerful testament to the superiority of their approach. The near-doubling of performance on the COCO open-vocabulary detection task (Table 3) is particularly good.\n3. Unified and Flexible Framework: The PaDT method is not a one-trick pony. It provides a single, unified mechanism for handling diverse visual outputs, including bounding boxes and segmentation masks, across different tasks like referring comprehension and open-vocabulary detection. The lightweight decoder design adds to this flexibility.\n4. Clarity and High-Quality Presentation: The authors do a great job of explaining a complex idea in a simple and intuitive way. The paper is well-organized, and the figures are highly effective at conveying the core concepts and results.", "weaknesses": "Overall, I think the idea is good. I still have some questions regarding the method. \n\n1. Scalability to High-Resolution Images: The number of VRTs in the dynamic vocabulary is directly tied to the number of input image patches. The paper evaluates on standard resolutions, but it's unclear how the method's computational cost (especially memory for the dynamic vocabulary and classifier weights) would scale to very high-resolution images that are divided into thousands of patches. A discussion on this limitation would be welcome. \n2. The random sampling of 5 VRTs per object for training is shown to be very effective. What is the intuition behind this random strategy? Have you experimented with more deterministic or \"intelligent\" sampling methods, for example, using attention maps to select the most salient patches or explicitly selecting patches near object boundaries? \n3. The performance on open-vocabulary detection is particularly strong. Do you have a hypothesis for why the VRT approach is so much more effective than text-based coordinates for this specific task? \n4. To what extent is the significant performance improvement attributable to simply providing the lightweight decoder with a richer, multi-token representation of the visual target? In other words, is the primary benefit that VRTs provide more detailed visual information to the final prediction head, which prior methods lack, rather than a fundamental improvement in the LLM's spatial reasoning itself?\n5. It is better not to only focus on the fine-grained benchmarks, but also try the method on some global semantic tasks, e.g., MMBench, MMVet, to make sure the introduced modules will not hurt on the other important tasks we mostly care about, rather than the specific domains. This makes the whole method more general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Y0X7mdaTi", "reviewer": "Reviewer_Cmsv", "summary": "This paper introduces Patch-as-Decodable Token (PaDT), a unified paradigm for multimodal large language models (MLLMs) that enables direct generation of both textual and diverse visual outputs. The core innovation is the Visual Reference Token (VRT), derived from image patch embeddings and interleaved with LLM textual tokens. A lightweight decoder transforms these outputs into detection, segmentation, and grounding predictions. The method claims improved localization, semantic alignment, and flexibility over prior approaches that serialize visual outputs as text (e.g., bounding box coordinates). Extensive experiments on detection, segmentation, grounding, and captioning tasks demonstrate state-of-the-art performance, even compared to much larger generalist models.", "questions": "Refer to the above part.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The proposed way to bridge textual and visual outputs, allowing MLLMs to generate structured predictions (e.g., bounding boxes, masks) in a unified format. Similar idea was used in Ma et al. 2025, but there is some difference in codebook design. However, I am not sure whether the proposed method is limited to fixed input visual token length, if yes, it will be a limitation compared to Ma et al. 2025.\n2) PaDT achieves strong results across multiple tasks and datasets, outperforming larger models (e.g., InternVL3 78B) in detection and segmentation benchmarks. Extensive ablation analysis is also provided.", "weaknesses": "1) The paper does not sufficiently address how PaDT handles flexible visual token lengths or variable image resolutions. While adaptive tiling and dynamic codebook expansion are mentioned, it remains unclear whether the method can robustly generalize to images of arbitrary sizes or aspect ratios, especially in real-world scenarios.\n2) The comparison with generalist models like QWen and InternVL3 may not be entirely fair. These baselines are designed for broad multimodal tasks, whereas PaDT is specifically trained for detection/segmentation. The paper should clarify whether baselines were fine-tuned for the same tasks or if the comparison is strictly zero-shot. This distinction is crucial for interpreting the reported gains.\n3) What is the computational overhead of dynamic codebook expansion and VRT decoding compared to standard coordinate-based approaches? Is inference time affected for high-resolution images?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J7RYc0BoRM", "reviewer": "Reviewer_XCqy", "summary": "The paper introduces Patch-as-Decodable Token (PaDT), a unified framework that enables MLLMs to generate both textual and visual outputs. It proposes Visual Reference Tokens, which are learnable tokens derived from image patch embeddings, and they are treated as decodable elements within the LLM’s output sequence. The proposed architecture and training strategy consistently achieve state-of-the-art performance across benchmarks.", "questions": "1. Table 4 shows that PaDT Pro underperforms PaDT on the referring image captioning task. Could the authors elaborate on the possible reasons for this performance gap?\n2. How well does the dynamic vocabulary expansion scale when processing high-resolution images with thousands of patches?\n3. What is the memory and computational overhead associated with dynamically updating the embedding table at each forward pass?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed framework effectively unifies diverse visual tasks such as detection, segmentation, grounding, and captioning under a single paradigm. The introduced VRT module aligns more naturally with both textual and visual semantics than coordinate-based outputs, reducing inconsistencies and hallucinations. Overall, the approach demonstrates consistent performance improvements over existing models.", "weaknesses": "1. The multimodal codebook needs to be adjusted at each forward pass, which may complicate deployment and increase inference latency.\n2. The decoder still relies on task-specific components (e.g., bounding box, mask, and score tokens), limiting the framework’s full generality.\n3. The evaluation primarily focuses on single-image tasks; it would be valuable to assess the model’s generalization to video or multi-image scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)