ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generation
Authors: Lingfeng Wang
Hualing Lin
Senda Chen
Tao Wang
Changxu Cheng
Yangyang Zhong
Dong Zheng
Wuyue Zhao
Venue: NeurIPS 2025 poster ()
Keywords: multimodal generation
multimodal large language model
adaptive-length
Primary area: deep_learning
TL;DR: ALTo is a novel framework leveraging adaptive-length tokens for multimodal mask generation, based on MLLM

Abstract: While humans effortlessly draw visual objects and shapes by adaptively allocating attention based on their complexity, existing multimodal large language models (MLLMs) remain constrained by rigid token representations. Bridging this gap, we propose ALTo, an adaptive length tokenizer for autoregressive mask generation. To achieve this, a novel token length predictor is designed, along with a length regularization term and a differentiable token chunking strategy. We further build ALToLLM that seamlessly integrates ALTo into MLLM. Preferences on the trade-offs between mask quality and efficiency is implemented by group relative policy optimization (GRPO). Experiments demonstrate that ALToLLM achieves state-of-the-art performance with adaptive token cost on popular segmentation benchmarks. Code and models will be released.

ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generation
Lingfeng Wang" 1,20, Hualing Lin"? Senda Chen" 33 Tao Wang" Changxu Cheng"!
Yangyang Zhong", Dong Zheng 1.2 Wuyue Zhao
Uni-Ubi ZZhejiang University TTogjii University
"Equal contributions 'Corresponding author Work done during internship at Uni-Ubi
Syayafengzi. linhualing, zhongyangyang, ddzheng) edu cn, (sendachen586, ccx0127 (cx0127)_gmail) com, (vangtaomarvel, zhaohongyil ouniubi. com
Eupower Maitimodal Large Langunge Model with Adspthe-lorgth Mask Generation
Tokm Legt
Input Image
4 Tokens
Tukens
Tokens
ALT Oulput
Moon
To T,
< Jirt
ChantCimphonn Mountain
To I. AL Eac- Paadk
SA Statt <ALTs Bwb- I17 mans rivenr
Land
All Peoppl
SALTs To Jn <ALTs, Jaat- (l mank tolenu)
Figure 1: ALToLLM realizes adaptive-length mask token generation according object complexity.
Abstract
While humans effortlessly draw visual objects and shapes by adaptively allocating attention based on their complexity, existing multimodal large language models (MLLMs) remain constrained by rigid token representations. Bridging this gap, propose ALTo, an adaptive-length tokenizer for autoregressive mask generation. To achieve this, novel token length predictor is designed, along with length regularization term and differentiable token chunking strategy. We further build ALToLLLM that seamlessly integrates ALTo into MIL.M. Preferences the trade- offs between mask quality and efficiency implemented by group relative policy optimization (GRPO). Experiments demonstrate that ALToLLM achieves state-of- the-art performance with adaptive token cost on popular segmentation benchmarks. Code and models are released http://ggthbb..oommaavvvgggg///l
Introduction
Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in image and text understanding tasks. However, their generative abilities remain largely limited to text 5 6). Given the inherent differences between text and image modalities, introducing
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

lightweight image decoder into multimodal understanding models to enable image generation remains a significant challenge. To align with the next-token prediction paradigm of text generation. discretizing images using image tokenizers (e.g., VQGAN 7 has become a natural and effective approach. Within this framework, various visual modalities-inchunee including RGB images, segmentation masks, and depth maps- can be uniformly represented as "images", enabling unified modeling for both multimodal understanding and generation 8 9
Early image tokenizers typically represent images using fixed-length token sequences without con- sidering the inherent complexity of the images 10 Ii [2]. This fixed-length design may lead to insufficient representation for complex images while generating redundant tokens for simpler ones, resulting in resouree wastage and reduced efficiency. In contrast, humans can flexibly allocate attention based on the complexity of the task [13]. For example, segmenting complex shapes requires more attention and effort compared to simpler shapes.
Recent arts are dedicated to learning hierarchical and flexible tokens 14 15 [6] The representations become increasingly fine-grained as the number of tokens increases. Based on our observations, the number of tokens required to represent fine-grained edge shapes can vary drastically depending on their complexity.
In recent years, several studies [9] 20 have explored adaptive-length tokenization for image representations. The problem, however, is that they all determine adaptive lengths by relying on heuristic rules conditioned the input image, rather than allowing the model to decide on its own. Although this is feasible in image tokenization, it becomes impractical for image generation since the re- construction loss is unavailable. As a result, it becomes imperative to enable the model to autonomously determine the adaptive token length, specifically for MLLM scenar- ios that are expensive in computation, like MLLM- -based object segmentation.
Table I:
The flcxibility and autonomous adaptivity of different token representa- tion methods. Flexibility refers to hi- erarchical coarse-to- fine token represen- tation, while autonomous adaptivity de- notes spontaneous allocation of token numbers based on object complexity.
Token Representation Adaptivity
Flexibility
Autenomous
VAE 17
x
VQVAE
x
VOGAN
x
TiTok
FlexTok [4
To enable adaptive-length modeling for the specific task of mask image generation, propose ALTo, an Adaptive- Length Tokenizer designed for autoregressive mask gen- eration. We further develop ALTOLLM. As shown in Fig. [ ALToLLM is multimodal large language model (MLLM) that realizes instruction-based mask generation using adaptive-length mask tokens according to object complexity. At the core of ALTo is novel token length predictor (TLP) embedded within an Voooder-VQ-deoodee architecture. Given an input mask image, the ALTo encoder is responsible not only for generating discrete tokens but also for predicting the appropriate token sequence length via TLP. To support adaptive-length learning. we introduce a length regularization term and differentiable token chunking strategy. Together, these enable ALTo to effectively encode masks into variable-length token sequences. To evaluate the effec- tiveness of ALTo, we construct ALToLLM without any bells and whistles, making no modifications to the underlying LLM architecture or training paradigm. The model is trained using supervised fine-tuning and group relative policy optimization (GRPO) on referring image segmentation tasks. ALToLLM learns to adaptively insert an end-of-mask token (<ALTo End>> once sufficient mask tokens have been generated. Moreover, GRPO allows dynamic control over token length to balance mask quality and computational efficiency.
Emu3
15
Chameleon
18
HiMTok 16
ALIT 19
BlasticTok 20
ALT (ours)
In summary, the contributions are as follows:
We propose AL.To, an adaptive-length mask tokenizer that, for the first time, enables the model to autonomously determine the number of mask tokens based on the complexity of the input mask.
We develop ALTOLLM, which integrates ALTo into a multimodal large language model (MLLM), enabling adaptive mask token generation for object segmentation tasks. The number of generated tokens can vary from as few as 2 to as many as 32, with most cases around 17, allowing ALToLLM to balance quality and efficiency under different scenarios via GRPO.
Extensive experiments demonstrate that ALTo enables effective and efficient mask image re- construction, while ALToLLM achieves state-of-the- the- performance with adaptive token usage

---

across various object segmentation benchmarks, including referring expression segmentation and open-vocabulary segmentation.
2
Related work
Visual tokenizers play important roles in various visual tasks, such as image reconstruction T0 7
visual compression (20], and visual generation [21 22 & [6] VQVAE IO 23 and VQGAN[7 are
popular frameworks that encode images into discrete 2D tokens by vector quantization. BEiT
24
exploits visual tokens in masked image modeling. To reduce the redundancy in 2D space, TīTok
and SEED [12 produce ID sequence for image tokenization. Methods above typically use a rigid number of tokens to represent images, regardless of the complexity of the visual content. To have flexible tokenization, FlexTok [14] projects images into ID variable-length token sequences. ElasticTok [20] proposed an adaptive tokenizer for images and videos by dropping a random number of the latter tokens during training. ALIT [9 discretizes the images into flexible-length tokens by recurrent distillation until the reconstruction quality is good the maximum iterations met. HiMTok [66 learns ID hierarchical mask tokens to represent coarse to fine segmentation masks. However, these methods cannot decide adaptive number of tokens autonomously. Trials have been made by heuristic rules about image reconstruction quality [99 20), which increases computational overhead and becomes impossible for image generation tasks. Our proposed ALTo is both flexible
and adaptive, as illustrated in Table [
MLLM-based image segmentation methods primarily follow three paradigms 225 16 26 MLLM-
segmentation joint models such as LISA [27), GSVA [28), GLaMM (29), PixeILM [30, and PSALM
31,. create semantic-to- to-tttic-tt-pieeel pixel connections through LLM hidden states and rely additional seg- mentation modules. Text-based methods, including Text4Seg [25), LLaFS [32] and VistaLLM [33], represent masks text sequences (pixel classes polygon vertices), suffering from heuristic and inaccurate mask representation. Interestingly, segmentation masks could also be viewed as images so that we rethink image segmentation mask generation task g
g 34) HiMToo (16) applies the
idea by utilizing hierarchical mask tokenizer into LLMs. Going step further, ALT generates adaptive-length token sequences, which is efficient and effective.
Reinforcement learning (RL) has become increasingly important for enhancing vision-language models K5 36
38, (9.] Approaches like direct preference optimization [40] and proximal policy optimization 41 face challenges with data efficiency and reward stability. Group relative policy 42 has emerged promising alternative through its groupwise reward mechanism. Recent applications demonstrate GRPO's effectiveness across various vision-language tasks. Visual-RFT [4) combines GRPO with verifiable rewards for efficient model adaptation. Vision-RI [44] employs GRPO with progressive thinking suppression for complex reasoning. Seg- Zero 455 achieves zero-shot segmentation through pure RL. These works apply RL to text output, while we make for preference optimization on mask token output.
37
optimization (GRPO)
3 Methods
3.1
Overview
The proposed adaptive-length tokenizer (ALTo) represents object masks as token sequences whose lengths adapt to the complexity of objects autonomously. Simple objects (e.g., sphere) may require few tokens, while intricate structures (e.g., complicated shapes and multiple objects) may use up to 32 tokens. Built on this, AL..oLLM is introduced to perform instruction mask generation for referring image segmentation, as shown in Fig. 2 We design a multi-stage training recipe for ALTo and ALToLLM to learn flexible, adaptive, and effective mask representations and achieve strong segmentation performance, shown in Fig. 3
Inference. As illustrated in Fig. 2 ALToLLM takes as input the image and text by the popular ViT-projector-LLM. -LMM architecture [I] [6),, then autoregressively generates both text tokens and compact mask tokens of adaptive length. The mask tokens along with the pixel-encoded features are fed into the mask de-tokenizer to generate the final mask.
Training. As shown Fig. 3 the training recipe consists of three progressive stages. Stage We pretrain the mask tokenizer (MT) and mask de-tokenizer (MD) reconstruct complex masks
3

---

Input Text
Multimodal Large Language Model
Response
Spli the yellow pants worr by the in yellmny clothess segment the bug hold
Text Tokenizer
Text De- tokenizer
Suie, the segmentation result
is <Mask
by the person in gray elothes.
Semantic Encoder
Input Image
Output Mask
Mask De- tokenizer
Pixel Encoder
Query Token
Pixel Token
Semantic Token
Text Tohen
Mask Taken + Element-wise Add Concateunde
Figure 2: Architecture of the proposed ALTOLLM.
using variable-length tokens. Stage 1.5: We fine-tune the token length predictor (TLP) to enable adaptive tokenization. Stage 2: We leverage ALTo to generate both fixed-length and adaptive-length token labels, which are used to supervise ALToLLM. This equips the model with the basic ability to understand and generate both text and mask tokens. Stage 3: We employ GRPO [42] to further adjust specific preferences on trade-off between mask quality and token efficiency, We will introduce the details the following subsections.
3.2 ALTo
The adaptive-length tokenizer (ALTo) comprises three components: a mask tokenizer (MT), a mask de-tokenizer (MD) with a pixel encoder, and token length predictor (TLP), as shown in Fig. (a). Following HiMTook [16], MT utilizes a transformer encoder with 32 learnable latent tokens to extract information from the input mask and then discretized into 32 mask tokens via vector quantizer (VQ). To support variable-length tokenization, a random number of tail tokens are dropped during training, retaining only the leading tokens. MD is bidirectional transformer. Differently from HiMTok, MD takes as input the mask tokens and 256 pixel-encoded image features rather than learnable latent tokens. This provides fine-grained guidance for mask generation, inspired by UViM 47). In training stage 1, the reconstruction supervised by a mean squared error (MSE) loss Mask to pretrain MT and MD.
The novel TLP determines the optimal number of tokens for each mask. TLP leverages the CLS token feature Tas, which encodes global image features, together with the 32 mask token features E 3232xd to predict proper token length. Tel is used as a query an attention mechanism to evaluate the importance of each mask token. For each mask token T; gated key is generated by SwiGLU as k; = (W,T,) @ o (W,T,). The probability for each token be the stopping point is computed via scaled dot-product attention: = softmax(gest /vd). The predicted length is
32 computed as the mathematical expectation L. - P.p Pt.
Accordingly, the first L tokens are selected and sent to the MD, while the remaining tokens zero- padded, represented H O T, where H is binary mask defined H II[ Ż). However, such token chunking strategy is not differentiable, which prevents gradients from the mask de-tokenizer from flowing back the TLP. To address this, introduce differentiable token chunking strategy by considering the stopping probability distribution The probability that the i-th token is used is given by the cumulative probability P.
Ejes Pý. which indicates that the stop position is later than this token and provides soft version of token chunking. This inspires through estimator as Í (P- P.detach() H) O T. In this formulation, the predicted mask is then given by Mpred MD(T, Ximg).
apply
In stage 1.5, we use reconstruction loss Lmakk = MSE( Mpred, Mg) to optimize mask reconstruction, and a length regularization term Linngth XL to encourage shorter token sequences, balancing

---

(a) Stage 1: ALTo Pretraining
Stage 1,5: ALTo Adaptive-length Tuning
(c) Stage MLLM
Adaptive-length Tuning
Iepus Image
Aspar Mesk
Mavk
Anpart
Inpur
Inags
Lizca My Litexif,
Livarlig
Encoder
TLP
Scae M-v
TLP
MILMM
Hed Tokan
Token
Predicted Mask
MD
Madk Tokes
Himeniintte Add ② Corererae
IEE Postion
Poodact
Prouiited Mork
Prodicted Mash
(d) Stage 3: Group Relative Policy Optimization
Relative Policy Opeimization
Reward Computation
Group Sampling
Rewurd Computation
R
MLLM
...
Rai
<ALTo Stare
ane TA
Intersection Union
Objective
(-2r)
1954 H632
444
Penalyy
Fonnal
Figure 3: Training recipes for ALTo and ALTOLLM. (a) ALTo Pretraining: Joint training of mask tokenizer (MT) and de-tokenizer (MD); (b) Adaptive-length Prediction: Training only the token length predictor (TLP); (c) Multimodal Integration: Exclusive training of MLLM with frozen ALTo for language-awand adaptation: (d) Group Relative Policy Optimization: Reinforcement learning for MILM optimization. Input image in (b), (c) and (d) is processed identically to (a), omitted for visual clarity.
accuracy and efficiency while MT and MD are frozen. The final combined loss is CALA -Mask Lenngth Further details about the length supervision design are provided in the Appendix. A
3.3 ALToLLM
ALToLLM is built naturally on MLLM architecture, and learned by supervised fine-tuning (SFT) and group relative policy optimization (GRPO).
During SFT (stage 2), ALToLLM receives adaptive-length mask tokens provided by the frozen ALTo module, along with text and image tokens. ALToLLM is supervised using two objectives: cross- entropy loss Lee for next-token prediction across the multimodal sequence, and mask prediction accuracy loss Lmask. which combines binary cross-entropy loss and dice loss to ensure precise mask reconstruction. This dual-objective training enables ALToLLM effectively align textual and visual information, and to autoregressively generate both language and adaptive-length mask tokens, which shows the effectiveness of ALTo.
We employ GRPO in stage 3 to adjust trade-off preferences flexibly based on the model after stage 2.
1) Group sampling: For each input consisting of an image and text, sample multimodal responses from ALToLLM. valid i-th sample must contain the following token sequence, where L, denotes the adaptive length:
<ALTo_Start> <TOK, > <TOKL,> End>,
L, E (1,...., 32}
(1)
tokens
2)) Reward computation: The composite reward R; for the i-th sample consists of three components:
R2
IFfommr
IoU
abi
Reand
Rasemmy
Rensiancy
where Ryani is 1 if the sample strictly follows the format in Eq. and 0 otherwise. Racruracy is the intersection-over-unioo (IoU) score between the predicted and ground truth masks, with the predicted mask reconstructed by the MD using the adaptive mask tokens; it vanishes to 0 if any responses do
5

---

Figure 4: Examples from the Multi Multi-Target-AA1B BBB dataset.
Table 2: Performance comparison on gReíCOCO. We report cloU, gloU and average token length. FT indicates fine-tuning on referring expression data.
Method
val
testA gloU
testB
cloU
gloU
Longth
cloU
Length
cloU
glo
Length
LISA-7B
27
38.7
32.2
52.6
48.5
44.8
39.7
LISA-7B (FT)
61.8
61.6
58.8
GSVA-7B
28
61.7
63.3
69.2
70.1
60.3
61.3
GSVA-78 11AA-BBK0 (FT)
28
63.3
66.5
69.9
71.1
60.5
62.2
GroundHog-77H
[1
66.7
SAM4MLLM-81
[2
67.8
71.9
72.2
65.3
UniRES++ S
69.9
74.4
74.5
76.0
66.6
69.8
LMM
16
66.8
68.7
32
68.6
67.6
32
65.8
64.1
32
LMM -BB (IT)
16
70.4
72.1
32
74.9
73.5
32
72.0
71.7
32
ALToLLM-SB (FL)
74.8
77.6
32
78.5
78.7
76.4
76.7
32
ALTsLLM-SB (AL)
75.4
78.0
17.5
78.8
19.4
76.6
76.9
17.3
not conform to the correct format. Reticiency is linear penalty -aL; proportional to the token length L; E (1...., 32)), scaled by the trade-off off parameter a 2 0.
3) Relutive policy optimization: We optimize the policy using the clipped objective (G):
9 min
E
A,, clip
1, 1,1-,,1+30 A. 3 B3Dr.. / mret)
Foldd
Fold
where тe is the current policy being optimized (parameterized by 0). Told is the policy before the update (used for importance sampling), Net is the reference policy (typically the initial supervised policy), A, = (R,- Rmean)/ Rad is the normalized advantage computed within each group. DKL is the Kullback-Leible (KL) divergence enforcing policy stability, is the clip range (usually 1-0.)) controlling update aggressiveness, and |8 is the KL penalty coefficient balancing exploration and constraint.
4 Experiments
4.1 Experimental settings
Datasets. For stages and 1.5, construct the training and validation sets of Multi-Target-SS.. from the SAIB dataset by randomly selecting multiple masks from all annotations for each image. Examples from Multi-Target-S.. masks, facilitating the learning of expressive mask representations by ALTo. For stage 2, we used all HiMTok and Multi-Target-SAAI datasets for SFT. For Multi- Maruti-Target--ae SAIB, we input the bounding boxes of all targets as "<box> (O) D
are shown in Fig. This approach yields complex multi-target
</box>". To ensure that the model supports both fixed- length and adaptive-length prompts, we randomly assign half of the data to each prompt type, as detailed in the Appendix. [ For stage 3, we use Multi- Multi-Targge-SAAT SAIB, the RefCOCO series [48] [49].
and gRefCOCO (50) to maintain complex mask representation and language understanding during
RL.
Implementation details. ALTo processes input and reconstructs masks at 256 x 256 resolution. During training and inference, the MILM processes images at 448 x 448, while the pixel encoder encodes image at 1024 1024. In stage 1, MT and MD are initialized from TiTok-L-32 [I] with codebook size of 1024, and the pixel encoder is initialized from SAM-ViT-L. [54] In stage 1.5, the feature dimension TLP set to 1024, consistent with MT. The length penalty coefficient is set to 0.0001, 0.001, 0.01, or 0.1, among which 0.01 is found to be optimal in subsequent experiments and is chosen for later stages. In stage 2, ALTOLLM--SH
initialized from Intern [46]. Stage 3 trains the RL model based on the stage 2 checkpoint, with the length penalty set to le-2, 5c-3,

---

Month
Fersen
(4)
P
(c)
Figure 5: Examples from the constructed multi-class version of open-vocabulary segmentation
datasets. (a) ADE20K (A-150); (b) PASCAL Context59 (PC-59); (c) PASCAL VOC 20 (PAS-20).
3e-3, 2e-3, 1e-3, or le-4, which are compared in later experiments. The KL penalty set to le-3, We sample 12 group responses with temperature of and top-k of 10. Stages 1, 1.5, and 3 are trained on 8 A 100 GPUs (80GB each), and stage on 16x 100 GPUs. Training durations are: stage I for 2 days, stage 1.5 for 3 hours, stage for days, and stage for 5 hours.
4.2 Comparative results
We evaluate ALToLLM-88 on the following tasks, considering two variants in SFT: (1) fixed- length version (ALTOLLM-8B (FL.) using 32 mask tokens, and (2) adaptive-length version (ALTOLLM-SB (AL)) that dynamically generates 1-32 mask tokens.
Generalized referring expression segmentation with multiple targets. We evaluate ALToLLM-8B on the generalized referring expression segmentation task (gRefCOCO (50)) and achieve state- of-the-art performance, as shown in Table 2 This task requires language-guided segmentation with multi-target referring expressions, demonstrating our model's ability to learn complex mask representations. Compared to the fixed-lengee variant (ALToLLM-8B (FL)), the adaptive-length variant (ALToLLM-8B (AL)) achieves higher performance and significantly reduces average token length, indicating improved segmentation accuracy and token efficiency. The superior performance of the adaptive-length approach may be attributed to its ability to use only the necessary tokens for simple masks, avoiding the noise introduced by redundant tokens. We also compare the average generation time per sample of the two variants. As shown Table 3 the adaptive length variant achieves a consistently shorter generation time in all splits.
Table 3: Comparison of average gener- ation time per sample (in seconds) be- tween fixed-length and adaptive-length variants gRefCOCO. Generation time is measured on a single A100 GPU with batch size 1.
Referring expression segmentation. We further evalu- ate ALToLLM-8B referring expression segmentation, a single-target version of the generalized task. Experi- ments are conducted on three standard benchmarks: Re- fCOCO (88. RefCOCO+ [48.. and RefCOCOg [49] As shown in Table ALT achieves state-of-the-art
Method
val
lestA
testB
results across all datasets.
LL- (FL.
1.079
1.099
1.076
We evaluate ALToLLM-98 on RefCOCOm (53), a multi-granulan qulti-granularity
Multi-granularity segmentation.
ALTOLLM-SB (AL)
0.710
0.753
0.669
referring segmentation dataset containing both part-level and object-level referring expressions. As shown in Table 5 ALToLLM-SB (AL) achieves the best performance.
Multi-class open-vocabulary segmentation. To demonstrate our model's 's ability to segment multiple and complex targets in open-vocabulary scenarios, we construct a multi-class version of open- vocabulary segmentation datasets by randomly merging annotations from several classes, including ADE20K (A-150) (22, PASCAL Context59 (PC-59) (33, and PASCAL VOC 20 (PAS-20) (64, as
shown in Fig. 5 We reproduce the inference pipelines for LISA (27] and MrSA [56) for comparison.
As shown in Table G ALTOLLM-88 achieves state-of-the-art results.
4.3 Adaptive-length preference adjustment via reinforcement learning
By tuning the length penalty in the reward function, we can flexibly control the model's preference for adaptive token lengths while maintaining high loU within a few hundred training steps by GRPO. As the average adaptive length decreases, the generation entropy also decreases, indicating that later
7

---

Table 4: Performance comparison on RefCOcO, ReiCOCO+, and ReiCOCOg, We report cloU. FT indicates fine-tuning on referring expression data.
RefCOCO
RefCOCO+
RefCOCOg
Methods
val
testA
Testl
val
testA
lestl
val (U)
test (U)
Text4Seg-IntemWLL. V1.288
25
79.2
81.7
75.6
72.8
77.9
66.5
74.0
75.3
PolyFormer-
55
74.8
76.6
71.1
67.6
72.9
59.3
67.8
69.1
VistaLLM 7B
33
74.5
76.0
72.7
69.1
73.7
64.0
69.0
70,9
LISA 7B
27
74.1
76.5
71.1
62.4
67.4
56.5
6664
68.4
LISA 7B
74.9
72.3
65.1
70.8
58.1
67.9
70.6
PixeILM 7B
30
73.0
76.5
68.2
66.3
71.7
58.3
69.3
70.5
GSVA 7B 28
76.4
77.4
72.8
64.5
67.7
58.6
71.1
72.0
GSVA
77.2
73.5
72.7
73.3
PSALM [3]
83.6
84.7
81.6
72.9
75.5
70.1
73.8
74.4
GLaMM 29
79.5
83.2
76.9
72.6
78.7
64.6
74.2
74,9
GroundHog
78.5
74.6
SAM4MLLM-8 M 522
79.8
82.7
74.7
74.6
80.0
67.2
75.5
76.4
MFSA 7B 56
74.0
76.8
69.7
63.1
67.2
56.1
67.0
68.3
AnyRef
74.1
68.7
69.9
AnyRef 59
76.9
79.9
74.2
70.3
73.5
61.8
70.0
70.7
SegAgent LLaVA+SAM 558
79.2
81.4
75.7
71.5
76.7
65.4
74.8
74,9
SegAgent Owen++AM 58
78.0
75.5
65.8
74.5
SegAgent LLaVAA
5%
77.8
80.0
74.1
66.7
71.2
59.9
70.5
71.3
SegAgent Owenn+SClic [88
79.7
81.4
76.6
72.5
75.8
66.9
75.1
75.2
Seg-Zero-7B
76.2
73.6
LMM 8B 16
81.1
81.2
79.2
77.1
78.8
71.5
75.8
76.7
LMM (FT) 116
85.0
85.2
83.5
79.7
82.7
76.0
80.0
80.6
ALIOLLM-88 (FL)
84.9
81.4
83.8
77.9
80.7
A. (AI.)
85.8
86.6
84.7
81.3
83.8
77.0
80.6
81.4
Table 5: Performance comparison on RefCOCOm. We report mloU for part-level and object & part-level expressions. indicates results reproduced by using the official code and settings.
Methods
val
lestA
testB
Part
Obj & Part
Part
Obj & Part
Part
Obj & Part
X-Decoder 59
16.2
29.5
13.6
23.6
20.3
33.8
SEEM 60
16.1
29.4
23.4
20.4
33,9
UniRES
61
19.6
34.3
16.4
27.8
25.2
41.7
LISA-7B 27
21.3
34.3
18.5
28.6
25.7
40.1
GSVA-7B 28
11.4
23.1
9.2
19.2
16.8
28.2
GLaMM 22
21.4
29.5
26.9
41.1
MPSA-7B
22.4
35.5
30.1
27.1
LMM
23.4
37.3
20.7
31.5
28.3
45.0
M- (FI.)
25.5
39,2
22.6
33.3
30.3
46.7
ALTOLLM-8B (AL)
25.5
39.1
22.6
33.2
30.2
46.5
tokens are associated with higher uncertainty. This trend visualized in Fig. 6 To quantify the token savings achieved by adaptive-length tokens various IoU levels, we compare six models trained with different length penalties in stage 3, alongside fixed-length baseline from stage Validation conducted on Multi-Target--3 understanding. For zero-shot evaluation, we use the multi-class A-150 dataset, which not seen during stage 3 training. As shown in Fig. 7 adaptive-length models consistently save more than 10 tokens at the same loU level, and this advantage persists even zero-shot scenarios.
for complex mask representation and gRefCOCO [50) for language
4.4 Ablation Studies
Ablation on ALTo. We first verify the necessity of the pixel encoder for reconstructing complex masks. As shown in Table 7 removing the pixel encoder leads to substantial drop in reconstruction gloU on the Multi-Target-SAA
validation set, confirming that pixel-level visual features are essential for capturing fine-grained mask details. We also study the effect of different penalty coefficients during Stage 1.5 training on adaptive-length mask prediction, as illustrated in Fig. The results show that cocfficient of 0.01 strikes optimal balance: it maintains high mask quality (in terms of IoU) while enabling a diverse range of predicted token lengths, as evidenced by high standard deviation and entropy in output lengths. We therefore adopt this value for Stage training.
Ablation on ALTOLLM. To better understand the key components driving ALToLLM: perfor- mance gains, we conduct comprehensive ablation studies on gRefCOCO (50), with results summa- rized in Table 8 Our analysis reveals that ALToLLM's improvements in mask quality primarily stem

---

Table 6: Performance comparison on multi-class vocabulaay segmentation. We report gloU.
Table 7: Ablation study of the pixel encoder on the Multi-Target-SAA validation dataset. We report gloU.
Method
A-150
PC-59
PAS-20
LISA-7B 27
39.1
53,3
67.8
Method
Pixel Encoder
gloU
MPSA-7B [56
63.9
74.1
79.5
ALTo (Ours)
ALTALLM-8H (FL)
65.6
75.2
81.2
ALTo (Ours)
91.9
ALToLLM-8D (AL)
65.8
75.4
81.1
Figure 6: Metrics of sampled responses during GRPO training. (a) Average token length, (b) Average
IoU, (c) Generation entropy.
from two factors: (1) pretraining Multi-Targer-SAIII dataset containing scenes with multiple and structurally complex objects, and (2) the pixel encoder, which provides high-resolution visual cues that refine mask boundary reconstruction. Moreover, the adaptive-length setting SFT not only improves token
the average output length by approximately 50%-but also slightly enhances mask quality. This demonstrates that adaptive token generation effectively elimi- nates redundancy while preserving, and even improving, mask quality. Finally, GRPO contributes marginally to absolute performance metrics but helps the model learn effective trade-off between mask quality and token efficiency.
Conclusions
We present ALToLLM, an innovative framework that dynamically adapts the number of mask tokens according to object complexity. ALToLLM approaches mask tokens as a visual language system, where our ALTo intelligently determines the optimal token count for each object. Furthermore, by integrating ALTo with MILMs, the system can interpret linguistic descriptions and correspond- ingly adjust token allocation. Extensive experiments demonstrate state-of-the-arr performance on RefCOCO [48). RefCOCO+ [48), RefCOCOg [49), RefCOCOm [53], and gRefCOCO [50) bench- marks, validating our approach's 's effectiveness aligning linguistic expressions with adaptive mask tokenization.
While ALTo effectively handles most segmentation tasks with adaptive token lengths (1-32 tokens), two key directions merit further exploration. First, our approach requires multiple training stages, which increases the engineering complexity, A simpler training pipeline is desirable for future appli-
LINVY
Avonge 340
Atrrtge (01)
A
Figure 7: Comparison of token cost between fixed-length and adaptive-length models with different length preferences. FL denotes the fixed-length model from stage 2. AL denotes stage models trained with different length penalties (from left to right: le-2, 5e-3, 3e-3, 2e-3, 1e-3, le-4). (a) Multi Malti--Trree--aa SA SABB val, (b) gRefCOCO val, (c) Multi-Class A-150 (zero-shot).
3

---

Anergg Longh
Chnssaaioon Suding mILL
60001
Figure 8: Analysis of the penalty coefficient in stage 1.5. (a) Length metrics for different penalty coefficients. (b) Distribution of length metries for penalty coefficients 0.0001, 0.001, 0.01, and 0.1.
Table 8: Ablation study for ALToLLM on gRefCOCO validation dataset.
Components
Metrics
Multi Bargi-Target-AAIB Pixel Encoder
AL(SFT)
GRPO
cloU
gloU
Avg. Length
70.4
72.1
32.0
72.8
75.6
32.0
74.8
77.6
32.0
75.4
78.0
17.5
/
75.5
78.1
15.7
cations. Second, though our pipeline is designed to be modality-agnostic (treating mask tokenization as special case of image tokenization). our experiments currently focus on mask validation. Future work will extend ALTo RGB image tokenization and validate its effectiveness across more general vision tasks.
Acknowledgment
This research supported by the Key Scientific Research Program of Hangzhou Municipal Bureau of Science and Technology grants 2025SZD1A01.
References
[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 00:34892-34911 2023.
[2] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic -ingal-linguistic tasks. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 24185-2411 24198, 2024.
[3] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308, 12966, 2023.
[4] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level mllm your phone. arXiv preprint arXiv:2408. 01000, 2024.
[5] Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingehao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-v12: Mixture-of-experts vision- language models for advanced multimodal understanding. arXiv preprint arXiv:2412. 2024.
10

---

[6] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.211785 2024.
[7] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[8] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language. and multi-modal tasks. arXiv preprint WMXiv:2266008116. 2022.
[9] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11439-264555 2024.
[10] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
[11] Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37:128940-12866 2024.
[12] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.08041 2023.
[13] Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence.
Minds and machines, 17:391 1444,,44422200 2007.
[14] Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oguzhan Fatih Kar, Elmira Amirloo, Alaaeldin El-Nouby, Amir Zamir, and Afshin Dehghan. Flextok: Resampling images into 1d token sequences flexible length. arXiv preprint arXiv:2502.13966 2025.
[15] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li. Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint 24X.v:2409.18866 2024.
[16] Tao Wang, Changxu Cheng, Lingfeng Wang, Senda Chen, and Wuyue Zhao. Himtok: Learning hierarchical mask tokens for image segmentation with large multimodal model. arXiv preprint arXiv:2503. 13026, 2025.
[17] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1312.:1312.611 |40, 2013.
[18) Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song- Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models, Advances in Neural Information Processing Systems, 36:43447-43478 2023.
[19] Shivam Duggal, Phillip Isola, Antonio Torralba, and William T Freeman. Adaptive length image tokenization via recurrent allocation. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2024.
[20] Wilson Yan, Volodymyr Mnih, Aleksandra Faust, Matei Zaharia, Pieter Abbeel, and Hao Liu. Elastictok: Adaptive tokenization for image and video. arXiv preprint 4rX1v.2400.0836, 2024.
[21] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821-8831. Pmlr, 2021.
[22] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances neural information processing systems, 0884899-848666 2024.

---

[23] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2, Advances Neural Information Processing Systems, 32, 2019.
[24] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprint wtw.:::2006.02...... 2021.
[25] Mengcheng Lan, Chaofeng Chen, Yue Zhou, Jiaxing Xu, Yiping Ke, Xinjiang Wang. Litong Feng, and Wayne Zhang. Text4seg: Reimagining image segmentation as text generation. arXiv preprint arXiv:2410.09855, 2024.
[26] Hao Tang, Chenwei Xie, Haiyang Wang, Xiaoyi Bao, Tingyu Weng, Pandeng Li, Yun Zheng, and Liwei Wang. Ufo: A unified approach to fine- -neigrained visual perception via open-ended language interface. arXiv preprint wrXivv22503.1 342, 2025.
[27] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579-99999 977999999,44 2024.
[28] Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[29] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji. Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 13009-13000 13018, 2024.
[30] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 06374-26383 2024.
[31] Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi-modal model. In European Conference on Computer Vision, pages 74-91. Springer, 2024.
[32] Lanyun Zhu, Tianrun Chen, Deyi Ji, Jieping Ye, and Jun Liu. Llafs: When large language models meet few-shot segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3065-3075, 2024.
[33] Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag. Ser-Nam Lim, Nicolas Ballas, Qifan Wang, Rama Chellappa, and Amjad Almahairi. Jack of all tasks master of many: Design- ing general-purpon coarse-to-fine vision-language model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14076-14088, 2024.
[34] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35:25005- 25017. 2022.
[35] Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RI on incorrect synthetic data scales the efficiency of Ilm math reasoning by eight-fold. Advances in Neural Information Processing Systems, 37:43000 43031, 2024.
[36] Hao Ma, Tianyi Hu, Zhiqiang Pu. Liu Boyin, Xiaolin Ai, Yanyan Liang, and Min Chen, Coevolv- ing with the other you: Fine-tuning mmm with sequential cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 37:15497-15525, 2024.
[37] Yun Qu, Yuhang Jiang, Boyuan Wang. Yixiu Mao, Cheems Wang, Chang Liu, and Xiangyang Ji. Latent reward: Lim-empowered credit assignment in episodic reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 20095-20103, 2025.
[38] Jiaxiang Li, Siliang Zeng, Hoi-To Wai, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Getting more juice out of the sft data: Reward learning from human demonstration improves sft for Ilm alignment. Advances in Neural Information Processing Systems, 1222222-222211111 2024.
12

---

[39] Yuanzhao Zhai, Tingkai Yang, Kele Xu, Dawei Feng, Cheng Yang, Bo Ding, and Huaimin Wang. Enhancing Docision-making for Ilm agents via step-level q-value models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27161-27163 2025.
[40] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53741 2023.
[41] John Schulman, Filipp Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint
[42] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033000 2024.
[43] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint 2rXhv:2503.01785, 2025.
[44] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint wrXiv2503300679, 2025.
[45] Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: heasoning-chaii guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06550, 2025.
[46] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101, 2024.
[47] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai Jeremiah Harmsen, and Neil Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes. Advances in Neural Information Processing Systems, 35:26295-26308. 2022.
[48] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara I. Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pages 69-85. Springer, 2016
[49] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 11 11-20, 2016.
[50] Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23592-23601, 2023.
[51] Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Ground- hog:
Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14227-14238, 2024.
[52] Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu-Chiang Frank Wang, and Chu-Song Chen. Sam4mllm: Enhance molti-modal large language model for referring expression segmentation. In European Conference on Computer Vision, pages 323-340. Springer, 2024.
[53] Jing Liu, Wenxuan Wang, Yisi Zhang, Yepeng Tang, Xingjian He, Longteng Guo, Tongtian Yue, and Xinlong Wang. Towards unified referring expression segmentation across omni-level visual target granularities. arXiv preprint arXiv:2504.01 01554, 2025.
[54] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In Proceedings of the IEFE/CVF international conference on computer vision, pages 2023.
13

---

[55] Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 18653-18663, 2023.
[56] Donggon Jang. Yucheol Cho, Suin Lee, Taehyeon Kim, and Dae-Shit Kim. Mmr: A large-scale benchmark dataset for mulli-target and multi multi-granuie granularity reasoning segmentation. arXiv preprint 0rXiv:2503.13880 2025.
[57] Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun-Yan Yan He, Jin-Peng Lan, Bin Luo, and Xuansong Xie. Multi-modal instruction tuned Ilms with fine-grained visual perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13980-13990, 2024.
[58] Muzhi Zhu, Yuzhuo Tian, Hao Chen, Chunluan Zhou, Qingpei Guo, Yang Liu, Ming Yang. and Chunhua Shen. Segagent: Exploring pixel understanding capabilities in mllms by imitating human annotator trajectories. arXiv preprint arXiv:2503. 08625,, 2025.
[59] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, al. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15126, 2023.
[60] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36: 1979-- 2023.
[61] Wenxuan Wang, Tongtian Yue, Yisi Zhang, Longteng Guo, Xingjian He, Xinlong Wang, and Jing Liu. Unveiling parts beyond objects: Towards amer-granularit referring expression segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22998-13008 2024.
[62] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
[63] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gru Cho, Seong-Whar Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 891-898, 2014.
[64] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303-338, 2010.
14

---

A Details of TLP
The Adaptive-Length Tokenizer (ALTo) utilizes a token length predictor (TLP) to dynamically deter- mine the number of tokens required for each mask. In this section, provide further mathematical details and training insights to clarify the differentiable token chunking strategy.
The overall loss function consists of reconstruction loss and a length regularization term:
CALA
L + Length:
where Lenghh = AL penalizes longer token sequences. The gradient of the loss is given by
V CALA = mmask + V Lienghh
= Σ ma VT. + A Σ VPp
If we directly chunk the token sequence, such as Í HOT, we obtain Ví = 0 because H is con-differentiable and 7' is generated by frozen mask tokenizer. As result, the gradient of Cmask cannot be backpropagated to the TLP, and only the token length is optimized to be minimal.
To address this limitation, we introduce a differentiable token chunking strategy. Specifically, we Ej<t Pj to represent the probability that the i-th token is used, and p denotes the detached version of P. We then construct soft token chunking as T (P P H) O. T. In this way, we have VT VVP T. The overall gradient is given by
use the cumulative probability P.
V CALA- =V Lmask + V Llength
mmmask P.T. + Σ Npp
ØT.
Hmrusk T;
Vpj + A iVp;
A..
321
mLiuk
T3 + + Σ VPp
Pi
OT,
i5i
mL OT,
Vps
T₁ Ai)
55
for the best predict length f. = ki there is (Ejsk maan Tj Ak) < Ejst-1 87,
0Lme
T, + A(k 1))
87,
and (Ejsk @C T) + Ak) Ejss+1 of,
&Lact Tj + >(k + 1)) which means
01,
mLmask
2Lmask Tk
Te+1 < A <
0T
O7,
This form shows that the model is encouraged to select the minimal number of tokens that still achieve high reconstruction quality, as the regularization term acts as threshold for including additional tokens.
Intuitively, the model will only increase the predicted token length if the marginal gain in reconstruc- tion quality outweighs the regularization penalty. This mechanism is analogous to a reward-cost trade-off in reinforcement learning, where the "reward" for including an additional token must exceed the cost X.
B Prompt design
We prepared different prompt templates for instruction tuning adaptive-length and fixed-length segmentation. and are the templates for adaptive-length ALToLLMM tuning, and Tabs 1T and 12 are the templates for fixed-length version.
15

---

Table 9; Templates of instruction for adaptive- length segmentation.
Table 10: Templates of response for adaptive- length segmentation.
"Segmentt <ref> joteef> by adaptive length."
"The aduptive mask appears at
[]-"
"Create <ref>(I<frefs <<fees> by adaptive length."
"The adaptive mask
()."
"Generate mask for <refs (>hef>> <href>>
adaptive length."
the adaptive mask
"Do segmentation for <refai J<freef> by adaptive length:
"The adaptive mask is (1."
"Please give the mask for <ref>(><<te>>> by adaptive
"I can give the adaptive mask at ()."
length."
"Its aduptive mask locuted at ()."
"What the mask <refsi J-hteff> adaptive length?"
aduptive mask ()."
"Can you segment <ref>> |</ref> by adaptive longth?"
Table 11: Templates of instruction for fixed-
Table 12: Templates of response for fixed-length segmentation.
length segmentation.
"Segment (eff=()<lrrfs>
"The mask appears at ()."
"Create mask for <ref>(]<<tee <//res'
"The mask created as ()."
"Generate for <rel> "<hrets
generate
[)."
"Do segmeniation for <relli "<hrels"
"The mask is ()."
"Please give the mask for <ref> cchefss
"I can give the mask []."
"What the mask for <rof>| !=trefs>"
"Its mask located at ()."
"Can you segment <ref> (ref>()<hefix <cref>>
"Sure, ()."
C Results on reasoning segmentation
We evaluate ALToLLM-SB the ReasonSeg benchmark 27 which demands complex visual reasoning for accurate segmentation. As shown Table 13 our method achieves consistently superior performance in both cloU and gloU, demonstrating enhanced reasoning capabilities over the baseline.
Table 13: Reasoning segmentation results on ReasonSeg validation dataset.
Methods
cloU
gloU
LISA-7B 7B
27
46.0
44.4
LISA-7B (ft)
27
54.0
52.9
SAM4MLLM
52
60.4
58.4
HiMTok [6
67.0
60.7
ALTOLLM-SB
67.3
62.8
D Results on region understanding
Following the setup of GLaMM 229 we assess Region-Level Captioning on the RefCOCOg dataset 49 Our method outperforms GLaMM in both METEOR and CIDEr metrics, demon- strating enhanced ability to generate semantically accurate and descriptive captions for localized image regions.
16

---

Table 14: Region-level captioning results on ReiCOCOg.
Methods
METEOR
CIDEr
GLaMM
16.2 16.5
106.0
ALTOLLM-8B
110.1
E Application examples
Fig. and Fig. 10 illustrates examples of ALToLLM in practical applications. As demonstrated, ALToLLM effectively segments the target object referred to by the user by leveraging adaptive-length mask tokens.
woman on the Jeft
woman on the right
suitcase
bat
blanket
the flowers the table
tolonss
$ toleett
16 toleens
24 Telketss
32 tolcees
Figure 9: An example of complex scenarios
17

---

oo
The adaptive mask appears at ALTo Start
Please segment all the people
ALTo End|> (23 mask tokens)
00
The adaptive mask appears at <LALTo Start>>
Please segment the hat and the left woman
<LAT End|> (16 mask tokens)
The adaptive mask appears at <LTo -ALTo_Start(>>
Please segment the man balck
<|AL End|> (15 mask tokens)
adaptive appears <VALTo Start
Please segment the basilica
<LALTo End(> (13 mask tokens)
00
The adaptive mask appears
Please segment the river
<LTTTo Start?
<LTT End mask tokens)
Figure 10: Examples of ALToLLM with adaptive- length segmentation.
18

---

NeurIPS Paper Checklist
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], , [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1 (1-2 sentence) justification right after your answer (even for NA).
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The main claims the abstract and introduction are consistent with the contributions and scope described in the main body of the paper, including the introduction ALTo and ALToLLM, their adaptive-length capabilities, and the experimental results. See
Abstract and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
The limitations of our work are discussed in the Supplementary Material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
19

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not contain theoretical results or formal proofs; it is focused empirical methods and experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of datasets, model architectures, training procedures, and hyperparameters in Section (Experiments), enabling reproduction of the main results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
20

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code and data are not released at submission time but will be made available in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtpppblcccguueeeeeeeecepeppeeee hwwbliiccguddd//dddeibiiisssss
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All relevant training and testing details, including data splits, hyperparameters, and optimizer settings, are provided in Section (Experiments) and the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
21

---

Answer:
[No]
Justification: We do not report error bars or statistical significance because we set the temperature 0 during inference, resulting in deterministic outputs without randomness.
This approach is consistent with standard practice in related work.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the compute resources used (8xA 100 GPUs for stages 1, 1.5, and 3; 16xA 100 GPUs for stage and each GPU with 80 GB memory) and training time for each stage in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our research complies with the NeurIPS Code of Ethics. We use only publicly available datasets and do not involve sensitive or private data.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
22

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper does not discuss potential societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeetm of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work does not pose high risks for misuse; therefore, no special safeguards are necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us, should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
23

---

Justification: All third-party datasets and models used in this work are properly cited in the main text and references. We only use publicly available assets with open-source licenses, and we respect the license terms for each asset as described in the corresponding references and documentation.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, snnerswwthcode com/co//caaoooocoo con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
New assets will be released soon.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Our research does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve human subjects and does not require IRB approval.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non sta-standard components.
Please refer to our LLM policy (https https:://neurippe cc/Confereeces/ /LLM) for what should or should not be described.
25


Reviews:
[{"review_id": "U0OG2Z6his", "reviewer": "Reviewer_sfeq", "summary": "The paper modifies the HimTok approach to image segmentation in several ways, obtaining better IoU scores on multiple referring expression and open vocabulary segmentation datasets. \nHimTok originally uses multimodal LLMs to perform image segmentation by expanding the LLM vocabulary size with discrete mask tokens, training the multimodal LLM to predict mask tokens, and outputting segmentation masks based on the predicted mask tokens.\nThe proposed modifications include (1) adaptively using a variable number of tokens to predict a segmentation mask, (2) adding a pixel encoder to provide additional information to the mask de-tokenizer, (3) using GRPO to tune the multimodal LLM, and (4) freezing and finetuning different parts of the model during different stages. (e.g. freezing everything except the LLM versus only freezing the mask tokenizer during Stage 2)", "questions": "In addition to the limited ablation studies mentioned above, some details are unclear.\n1. What is the mask token codebook size? 1024 as in HimTok?\n2. The reward function seems to suggest that the composite reward is non-zero for an incorrectly formatted sample. Does this mean that the incorrectly formatted sample is still somehow given to the mask de-tokenizer for generation?\n\nAlso, at the end of the introduction, it is mentioned that for most cases the length is 16, and some examples are given in the first page and the appendix. I'm curious if the authors have a plot of the distribution of mask token sequence length? And are there any oddities where the model uses many tokens for a seemingly simple mask, or few tokens for a seemingly complex mask?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe idea of allowing the model to adaptively decide how many tokens to use to represent an image is interesting. \nThe final pipeline brings improvements to both accuracy and efficiency (when compared against the fixed-length pipeline). \nThe length penalty hyperparameters for the loss function in Stage 1.5 and Stage 3  are also reasonably well studied, showing the exact trade-off between IoU and average mask token sequence length in the final models after Stage 3, as well as advantages of using 0.01 as the length penalty in Stage 1.5.\n\n\nWeaknesses\n\nMy main concern is the lack of ablation studies: \nConsidering the high similarity to HimTok, it is crucial to investigate the effects of each modification to the pipeline. However, only the adaptive token length predictor and the pixel encoder are studied, and only to a limited degree. This leaves the reader uncertain where the gains over HimTok come from. For example, does ALToLLM still always outperform HimTok when no pixel encoders are used? What about when no GRPO is used? What if HimTok also only trained the LLM in Stage 2 training? Or vice versa, what if ALToLLM only freezes the mask tokenizer during Stage 2?", "was_revised": true, "final_justification": "The rebuttal has resolved my original concerns about the lack of ablation studies investigating the effects of each modification to the pipeline. I believe that evaluating the proposed method on mask generation is sufficient, and the additional results on image understanding and region understanding is a plus, showing potential applicability of the method to other tasks.\n\nHence, I have raised my score to Accept.", "rebuttal": "We sincerely thank the reviewers for their thoughtful feedback and for recognizing the merits of ALToLLM, particularly its adaptive token length mechanism, improved accuracy-efficiency trade-off, and insightful analysis of length regularization.\n\n> **W1: Lack of ablation studies.**\n\n**A:** Good suggestions. To more comprehensively illustrate the key factors behind ALToLLM's performance gains over HiMTok, we conducted additional ablation studies on the *gRefCOCO val* set, although Table 2 and Figure 7 have already presented part of the relevant results.\n\nWe analyze the following design choices:\n- **Pixel Encoder**: Introducing a pixel-level encoder to supply fine-grained visual features to the mask de-tokenizer.\n- **Tuning Mask De-tokenizer (MD)**: Whether the mask de-tokenizer is updated or kept frozen during Stage 2 training.\n- **Token Length Predictor (TLP)**: Predicting the number of mask tokens adaptively, enabling variable-length sequences.\n- **GRPO Training**: Applying GRPO in Stage 3 with a length penalty coefficient of $1 \\times 10^{-3}$ for sequence regularization.\n- **Multi-Target-SA1B used in Pretraining**: Leveraging our dataset containing complex and multi-object masks during Stage 1 pretraining.\n\nThe results are summarized below:\n\n| Method                                      | Multi-Target-SA1B | Pixel Encoder | Tuning MD | TLP  | GRPO  | cIoU  | gIoU  | Avg. Length |\n|--------------------------------------------|:-----------------:|:-------------:|:--------:|:-----:|:-----:|:-----:|:-----:|:-----------:|\n| HimTok                                     |                   |               |          |       |       | 70.4  | 72.1  |     32.0    |\n| ALToLLM                |         ✓         |               |          |       |       | 72.8  | 75.6  |     32.0    |\n| ALToLLM                |         ✓         |       ✓       |          |       |       | 74.8  | 77.6  |     32.0    |\n| ALToLLM                |         ✓         |       ✓       |    ✓     |       |       | 74.7  | 77.6  |     32.0    |\n| ALToLLM                |         ✓         |       ✓       |          |   ✓   |       | 75.4  | 78.0  |     17.5    |\n| ALToLLM                |         ✓         |       ✓       |          |   ✓   |   ✓   | **75.5**  | **78.1**  |     **15.7**    |\n\n> *W1.1: Where the gains over HimTok come from.*\n\n**A:** The ablation study reveals that the performance improvements of ALToLLM over HimTok primarily stem from two key components:  \n(1) **Multi-Target-SA1B Pretraining** boosts cIoU from 70.4 to 72.8, significantly enhancing the model’s ability to handle scenes with multiple and complex objects.  \n(2) The **Pixel Encoder** further increases cIoU by 2.0 points (to 74.8), thanks to the fine-grained visual feature that enables more accurate mask boundary reconstruction.  \n\nAdditionally, the **TLP** improves token efficiency—reducing average output length by around 50%—while still increasing cIoU to 75.4, indicating that adaptive token generation reduces redundancy without sacrificing quality.\n\n> *W1.2: Effect of GRPO*\n\n**A:** We acknowledge that GRPO contributes only marginally to absolute performance metrics (e.g., +0.1 cIoU). However, as discussed in **L170**, its primary value lies in **controllability**: GRPO offers a flexible mechanism to balance mask quality and token efficiency.  As shown in **Figure 7**, increasing the length penalty coefficient in GRPO training improves token efficiency at varying degrees of cost to mask quality. This tunability allows us to adapt the model to different application needs—prioritizing accuracy in high-fidelity scenarios or efficiency in bandwidth-constrained environments. Moreover, as demonstrated in **Figure 6**, this adaptation can be achieved within just a few hundred training steps, highlighting the efficiency and practicality of GRPO training in dynamically shaping model behavior.\n\n> *W1.3: Difference from HimTok in Stage 2 training*\n\n**A:** In Stage 2, unlike HimTok—which jointly fine-tunes both the mask de-tokenizer (MD) and the multimodal large language model (MLLM)—we only train the MLLM while keeping the MD frozen.  Our ablation shows this design achieves comparable performance, as MD has been already well-learned during Stage 1 and capable of accurately generating diverse masks. Notably, we observed that the public implementation of HimTok omits the original LayerNorm layer when it is integrated into LMM, which is inconsistent and necessitate the tuning of MD in Stage 2. In contrast, our architecture is consistent before and after integrating ALTo into LMM, which eliminates the need to update the MD in Stage 2. \n\n>  **Q1: Mask token codebook size clarification**\n\n**A:** The mask token codebook size is 1024, consistent with HimTok for fair comparison.\n\n>  **Q2: Rewards about incorrectly formatted sample**\n\n**A:** In our mask de-tokenizer step, any responses that do not conform to the correct format will raise an exception, and its corresponding IoU reward is set to 0. As a result, incorrectly formatted samples always receive lower composite rewards compared to correctly formatted ones. We have empirically observed the desired behavior with this design: during GRPO training, the average format reward consistently converges to 1 within approximately 20 steps, and thereafter the model rarely produces samples with formatting errors.\n\n>  **Q3: Provide a plot of the distribution of mask token sequence length**\n\n**A:** (1) In **Figure 8**, we present the distribution of mask token sequence lengths in Multi-Target-SA1B under different penalty coefficients during Stage 1.5. When the coefficient is set to 0.01, the token lengths are primarily concentrated between 20 and 30, which may be attributed to the more complicated masks in Multi-Target-SA1B. (2) To see the actual distribution on real benchmarks, we plotted the token length distribution of ALToLLM (SFT) on the *gRefCOCO val* set, where most samples fall within the 10–20 range. Due to rebuttal constraints, we present the distribution for lengths 10–20 in tabular form instead of images.\n\n| Token Length | 10  | 11  | 12  | 13  | 14  | 15  | 16  | 17  | 18  | 19  | 20  |\n|--------------|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|\n| Sample Number| 190 | 286 | 436 | 551 | 378 | 291 | 498 | 526 | 325 | 424 | 432 |"}, {"review_id": "Wr3K0wyBk6", "reviewer": "Reviewer_QGmw", "summary": "This paper introduces a novel adaptive-length tokenizer called ALTo. The contribution of ALTo is its ability to dynamically determine the number of mask tokens needed to represent visual objects based on their complexity, enabling efficient and flexible mask generation for segmentation tasks. At the core of ALTo is a Token Length Predictor (TLP), which predicts the optimal token length using global and local mask features, combined with a differentiable token chunking strategy and length regularization loss. This allows the model to encode segmentation masks into variable-length token sequences—ranging from 2 to 32 tokens—rather than using a fixed length, which often leads to over- or under-representation of content.\n\nBuilt upon this tokenizer, the authors develop ALToLLM, a multimodal large language model trained in four stages: (1) pretraining the mask tokenizer and de-tokenizer, (2) fine-tuning the TLP, (3) supervised fine-tuning of the MLLM with adaptive mask tokens, and (4) Group Relative Policy Optimization (GRPO) to flexibly balance segmentation accuracy (measured by IoU) and token efficiency via a composite reward. Experiments across standard benchmarks demonstrate that ALToLLM achieves state-of-the-art performance while reducing average token usage and inference time.", "questions": "- How exactly are the tail tokens dropped during training? It would improve clarity if the paper explained this procedure in more detail.\n- The explanation of the cumulative sum might be clearer if rewritten using the form $P_i = 1 - \\sum_{j < i} p_j$, to emphasize the contribution of earlier tokens to the stopping probability.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- **Well-designed and novel Token Length Predictor (TLP):** The training strategy for TLP is innovative and empirically validated across diverse benchmarks, demonstrating its effectiveness.\n- **Careful architectural thoughtfulness:** The attention to detail in TLP’s design (e.g., differentiable chunking, attention-based gating) reflects a high level of engineering insight.\n- **Strong empirical results:** The model consistently achieves state-of-the-art performance while improving token efficiency, showing practical utility of adaptive-length modeling.  \n&nbsp;  \n### Weaknesses\n- **Limited scope of application:** Although the proposed adaptive tokenizer has broader potential (e.g. image tokenization), the paper evaluates it solely on mask tokenization tasks. While this choice may be justified, it would be valuable to discuss why the method is especially suitable for masks and whether it can be generalized beyond this domain.\n- **Lack of baseline with fixed optimal length:** The paper does not compare against a fixed-length model set to the optimal length determined by the adaptive method—this makes it hard to isolate whether the gain comes from adaptivity or just better length calibration.", "was_revised": true, "final_justification": "I have read the authors' rebuttal carefully. Thank you for the detailed and thoughtful response. The clarifications adequately addressed the concerns raised in my initial review, and I have updated my score accordingly.", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful and constructive feedback on our work. Below, we address each point with detailed clarifications.\n\n>  **W1.1: The paper evaluates it solely on mask tokenization tasks. Why the method is especially suitable for masks?**\n\n**A:** Although the color space of masks is simple, it is non-trivial and essential to generate masks that not only ground object positions but also capture fine-grained shape details, as pointed in HiMTok [16].\nOur method is well-suited for mask generation tasks, given *the recent advances in the field* and *characteristics of the task*.\n\n(1) Recent studies like FlexTok [14], HiMTok [16], have shown that representing masks or images of varying complexity may require different numbers of tokens. For masks, simple shapes can be captured with just a few tokens, while complex ones demand more. Although HiMTok adopts a hierarchical (coarse-to-fine) token sequence, it lacks the ability to adaptively determine how many tokens are needed based on the shape complexity. Building on this insight, ALTo introduces a method that enables the model to autonomously decide the token sequence length in an adaptive manner.\n(2) The quality of generated masks can be quantitatively evaluated in a straightforward manner (e.g., IoU), and segmentation instructions typically correspond to deterministic target masks—unlike the ambiguity and diversity inherent in tasks like image generation. As the first to propose and focus on a model that autonomously adapts token sequence length, we chose to validate it on mask generation rigorously to minimize the influence of confounding factors during both training and inference, before extending to more complex domains.\n\n>  **W1.2: Whether it can be generalized beyond this domain?**\n\n**A:**  As highlighted in our paper (Introduction and references [8, 9]), both  image genration and segmentation masks can be represented within a unified tokenization framework. As demonstrated by prior works like HiMTok [16] (for masks) and FlexTok [14] (for RGB images), the coarse-to-fine token generation paradigm effectively handles both domains. Our current validation on segmentation establishes the method's effectiveness for mask generation. We fully acknowledge the potential for broader applications and plan to explore image tokenization in future work. \n\n>  **W2: Lack of baseline with fixed optimal length**\n\n**A:** In fact, **Figure 7** directly compares our adaptive-length model (ALToLLM after GRPO) against the fixed-length baseline (FL) set to the TLP-determined optimal lengths. The x-axis and y-axis show the IoU and average sequence length for each experimental setting, respectively. The blue points represent the adaptive-length models obtained with different GRPO penalty coefficients, while the red points represent the fixed-length baseline. When we compare models with the same average token length (i.e., at the same y-coordinate), the adaptive model consistently achieves higher IoU than its fixed-length counterpart, demonstrating that the performance gain stems from adaptivity itself, not merely from better length calibration. Furthermore, when comparing models with similar IoU (i.e., at the same x-coordinate), the results show that ALToLLM maintains comparable IoU while reducing token usage by more than 5 tokens per sample on average. This efficiency advantage holds consistently across all tested datasets, including zero-shot scenarios, confirming that dynamic length adaptation is a key factor underlying the superiority of our method.\n\n>  **Q1: The tail tokens' dropping approach**\n\n**A:** The procedure is carefully designed across different training stages. **Stage 1**: We randomly drop tail tokens (keeping 1-32 tokens) during training to learn hierarchical mask representation. **Stage 1.5**: We introduce the differentiable *Token Length Predictor (TLP)* which learns to optimally truncate tokens through the cumulative stopping probability. This allows the model to: (a) predict appropriate lengths based on content complexity, and (b) maintain *differentiability* for end-to-end training via our straight-through estimator. **Stage 2**: During training, we use the pretrained TLP to determine optimal sequence lengths and perform deterministic truncation. For inference, the model dynamically generates mask tokens until predicting the end token, eliminating the need for explicit token dropping.\n\n>  **Q2: Rewritten of fomulation**\n\n**A:** Thanks for this valuable and careful suggestion. We will revise the formulation to emphasize the contribution of earlier tokens to the stopping decision. We will carefully check all equations throughout the paper to ensure mathematical rigor and clarity."}, {"review_id": "BwBQVGBkFf", "reviewer": "Reviewer_bQXT", "summary": "This paper proposes an adaptive method, ALTo,  for generating adaptive length tokens for the task of image segmentation. The length of the token sequences depends on the difficulty of the segmented objects. In addition, ALTo can be plugged into the Multimodal Large Language Model (MLLM) through SFT and RL, enabling the task of referring segmentation. Through the comprehensive experiment, the paper demonstrates the efficiency of the adaptive length tokens compared with the baselines: Achieving high segmentation results while being fast compared with the fixed-length token mechanism.", "questions": "- Restricted only to segmentation: Can this approach be extended to other tasks like Image understanding and region understanding, similar to GLaMM[1]? What is the performance of the adaptive tokenization for the Reasoning Segmentation task? Please refer to Sections 4.4 and 4.7 for metrics which is not Segmentation.\n- Datasets used in the experiments: What is the difference in terms of problem setting between different types of datasets used in this paper? For example, what is the difference between gRefCOCO, RefCOCO, and RefCOCO+? The paper should include the difference between these types of datasets for the reader to understand.\n- Ablation study: This paper should include the ablation study about the importance of different stages to the final results. For example, how important of RL phase compared with the results of the SFT phase only? \n- In L153, what is $\\hat{P}$? It is never defined.\n- In L156, to encourage short token regularization, is $\\mathcal{L}_{length}$ should be $\\lambda L$ instead of $-\\lambda L$.\n- Why is the method used to compare with ALToLLM in Table 2 and Table 3 not consistent? The method list is not similar for both tables, making it harder to understand what this paper is trying to compare.\n- What is the main factor that makes the difference between Fix-length ALTo with Fix-length tokenization compared with HimTok in Table 2 and 4?\n\nReference:\n[1] Rasheed, Hanoona, et al. \"Glamm: Pixel grounding large multimodal model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "limitations": "See the question section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**: This paper tries to improve HimTok [1] in terms of the number of tokens for the problem of pixel understanding (segmentation) in the MMLLM model. The approach is to use an adaptive token strategy, which uses a different number of tokens for each segmentation task, depending on the difficulty of the segmentation. This approach is intuitive, and the experiment indicates the efficacy of their approach.\n\n**Weakness**: Most of the parts are borrowed from the HimTok work, and only token adaptation is the main contribution of this work. However, the computation of the token adaptation at some point (straight-through estimator in L153) is not clear. The changes in terms of the number of tokens should not affect the results of the baselines MMLLM in some evaluation metrics besides segmentation, e.g., segmentation reasoning and general image understanding, which is not reported in this paper. Further ablation study is required to indicate the efficacy of each component (training phase) with the overall performance. However, the proposed approach is interesting for reducing the number of tokens used in HimTok.\n\n[1] Wang, Tao, et al. \"Himtok: Learning hierarchical mask tokens for image segmentation with large multimodal model.\" arXiv preprint arXiv:2503.13026 (2025).", "was_revised": true, "final_justification": "Based on the rebuttal, the authors have addressed my concerns. Based on the range of the application of the adaptive-length token representation, I will increase the score to 4.", "rebuttal": "Thanks for the constructive comments on the intuitive, interesting and effective method.\n\n\n>  **W1: Most of the parts are borrowed from the HimTok work, and only token adaptation is the main contribution of this work**\n\n**A:** We would like to clarify that the proposed adaptive-length mask tokenizer and its integration into LMM -- resulting in ALToLLM -- are novel and non-trivial contributions, even though they are experimentally built upon the HiMTok architecture. The method introduces crucial innovations:\n- **The First Autonomous Adaptive-Length Mask Tokenizer**. As shown in **Table 1**, ALTo represents the first framework enabling models to dynamically determine visual mask token lengths (2-32) based on shape complexities.\n- **Adaptive-Length Token Generation in MLLM**.  ALToLLM enables adaptive mask token generation for (referring) image segmentation tasks through SFT and GRPO, with elaborately-designed learning objective and RL reward. We can balance the  quality and efficiency conviniently.\n\nThis has the potential to be applied to a broader range of visual scenarios in the future.\n\n>  **Q1: Extended to other tasks**\n\n**A:** We appreciate the reviewer's suggestion to evaluate broader capabilities. Our experiments demonstrate ALToLLM's versatility across multiple vision-language tasks.   \n\n(1) **Reasoning Segmentation**      \nWe evaluate our model on the *ReasonSeg* benchmark, which requires complex reasoning for segmentation. Our method achieves superior performance compared to HiMTok, indicating stronger reasoning capabilities. These improvements are consistent with findings on other segmentation benchmarks such as gRefCOCO, further validating the robustness of our approach.\n\n| Methods | cIoU | gIoU  |\n|---------|------|-------|\n| HiMTok  | 67.0 | 60.7  |\n| Ours    | **67.3** | **62.8**  |\n\n(2) **Region Understanding**        \nFollowing the setup of GLaMM, we assess Region-Level Captioning on the *RefCOCOg* dataset. Our method outperforms GLaMM in both METEOR and CIDEr metrics, demonstrating enhanced ability to generate semantically accurate and descriptive captions for localized image regions.\n| Methods| METEOR | CIDEr  |\n|--------|--------|--------|\n| GLaMM  | 16.2   | 106.0  |\n| Ours   | **16.5**   | **110.1**  |\n\n(3) **Image Understanding**     \nWe further evaluate image captioning performance on *NoCaps* and *Flickr30K*, following the protocol of GLaMM. Two settings are considered: (i) the original setting, and (ii) a fine-tuned setting (denoted with †) using a small number of COCO training samples provided by GLaMM.     \n- In the original setting, our model tends to generate more detailed and descriptive captions—often exceeding the length and specificity of ground-truth annotations—leading to higher SPICE scores but lower CIDEr values due to style mismatch. This reflects our model’s strong scene understanding and descriptive richness.  \n- In the fine-tuned setting (†), our model successfully adapts to the concise captioning style typical of these benchmarks. On NoCaps, we achieve a CIDEr score of **113.6**, surpassing GLaMM, while maintaining competitive SPICE performance. This adaptability underscores the flexibility and broad applicability of ALToLLM across different captioning styles and domains.\n\n| Methods | NoCap          |            | Flickr30k     |            |\n|---------|----------------|------------|---------------|------------|\n|         | CIDEr          | SPICE      | CIDEr         | SPICE      |\n| GLaMM   | 106.8          | 15.8       | 95.3          | 18.8       |\n| Ours    | 61.7           | **17.1**       | 57.0          | **20.6**       |\n| Ours†   | **113.6**          | 14.8       | 82.8          | 18.0       |\n\n\n\n>  **Q2: The difference in terms of problem setting between different types of datasets**\n\n**A:** Thank you for raising this important point. We clarify the key differences in the problem settings across the benchmark datasets:\n\n- **RefCOCO [48]** focuses on basic referring expressions that rely on spatial cues (e.g., *\"left woman\"*).\n- **RefCOCO+ [48]** prohibits the use of absolute spatial terms, requiring models to understand attribute-based descriptions (e.g., *\"woman in red\"*).\n- **RefCOCOg [49]** emphasizes unambiguous expressions to distinguish between visually similar objects (e.g., *\"woman who is touching her head\"*).\n- **gRefCOCO [50]** includes generalized referring expressions, covering single-object, **multi-object**, and empty references (e.g., *\"all women\"*).\n- **RefCOCOm [53]** introduces **part-level segmentation**, requiring fine-grained understanding of object parts (e.g., *\"woman's hat\"*).\n\nThis progression from simple to complex referring expressions systematically evaluates a model's ability to handle various linguistic and visual challenges. Among them, **gRefCOCO**, with its focus on multi-object references, is particularly relevant for assessing the effectiveness of our adaptive tokenization mechanism in complex, real-world scenes.\n\n> **Q3 & 7: Ablation studies contributing to the final results**\n\n**A:** Thank you for your interest in our ablation studies. Building upon the results on the *gRefCOCO val* set (**Table 2**), we further incorporate the GRPO results with a penalty coefficient of $1 \\times 10^{-3}$ (as shown in **Figure 7**). We perform more ablation experiments on the following components:\n- **Multi-Target-SA1B Pretraining**: Utilizing our curated dataset containing complex and multi-object masks during Stage 1 pretraining.\n- **Pixel Encoder**: Incorporating a pixel-level encoder to provide fine-grained visual features to the mask de-tokenizer.  \n\nThe results are summarized below:\n\n| Methods           |          |            |     | Metrics           | | |\n|:-----------------:|:-------------:|:--------:|:-------------:|:-----:|:-----:|:-----------:|\n| Multi-Target-SA1B | Pixel Encoder | Adaptive Length | GRPO Training | cIoU  | gIoU  | Avg. Length |\n|                   |               |              |         | 70.4  | 72.1  |     32.0    |\n|         ✓         |               |               |        | 72.8  | 75.6  |     32.0    |\n|         ✓         |       ✓       |                |       | 74.8  | 77.6  |     32.0    |\n|         ✓         |       ✓       |           ✓    |       | 75.4  | 78.0  |     17.5    |\n|         ✓         |       ✓       |           ✓    |✓              | **75.5**  | **78.1**  |     **15.7**    |\n\n> *Q3: RL phase compared with the results of the SFT phase*\n\n*A:* As shown in Figure 7, the performance gain from the GRPO phase compared to the SFT phase (**Table 2**) is relatively modest in terms of absolute metrics (e.g., +0.1 cIoU). However, as noted in L170, the main advantage of GRPO lies in its **flexibility**—it allows users to control the trade-off between mask quality and token efficiency.  \nBy adjusting the length penalty coefficient during GRPO training, we observe that **reducing the penalty leads to higher mask quality at the cost of increased token usage**, while increasing the penalty yields more compact outputs with slightly lower mask fidelity. This level of controllability enables the model to be adapted to different application scenarios, depending on whether users prioritize high-quality masks or efficient token generation.\n\n> *Q7: Main factors contributing to the performance gains over HimTok in Tables 2 and 4*\n\n*A:* The table above shows that the improvements are primarily attributed to two key components:    \n- (1) **Multi-Target-SA1B Pretraining**, which significantly enhances the model’s ability to generalize to complex scenes with multiple objects.  \n- (2) **Pixel Encoder**, which provides fine-grained visual representations that enable more accurate reconstruction of detailed mask boundaries. \n\nImportantly, the performance improvement on *gRefCOCO* (**Table 2**) is notably larger than that observed on *RefCOCO*, *RefCOCO+*, and *RefCOCOg* (**Table 4**). This indicates that our model excels particularly well in **multi-object settings** (see **Q2**), which aligns with the design goals of the Multi-Target-SA1B pretraining and the fine-grained visual modeling enabled by the pixel encoder.\n\n>  **Q4 & 5: Problems on writing (L153&156)**\n\n**A:** Thank you for your careful comments on the writing issues. We will thoroughly review the manuscript.\n- In L153, $\\hat{P}=P.detach()$, which is detached from the computation graph, so that $\\hat{T}$ is differentiable with respect to $P$.\n- In L156, thanks for pointing out this typo — it is indeed $\\mathcal{L}_{\\text{Length}} = \\lambda \\hat{L}$.\n\n>  **Q6: The method list is not similar for Table 2 and Table 3**\n\n**A:** Different dimensions are compared using different method list in Tables 2 and 3, respectively.\n- It is natural to compare performance metrics (e.g., cIoU and gIoU) with many related works (method list in Tab. 2), but the average token length is only meaningful for methods following HiMTok paradigm (i.e., discrete mask token generation). Thus we only compare ALToLLM-8B (AL) with ALToLLM-8B (FL) and HiMTok on average token length in **Table 2**.\n- To better illustrate the efficiency gains brought by the adaptive token mechanism, we also report the corresponding average inference time (**Table 3**). Since the LMM part of ALToLLM-8B (FL) follows the architecture design in HiMTok, it should have the same inference time.\n\n> **Q7: What is the main factor that makes the difference between Fix-length ALTo with Fix-length tokenization compared with HimTok in Table 2 and 4?**\n\n**A:** Please kindly refer to **Q3 & 7** above."}, {"review_id": "58Yk2XzL0L", "reviewer": "Reviewer_7PTq", "summary": "This paper introduces ALTo (Adaptive-Length Tokenizer) and ALToLLM, a framework that enables MLLMs to generate adaptive-length mask tokens for object segmentation. ALTo uses a Token Length Predictor (TLP) to determine the optimal number of tokens based on object complexity, combined with a length regularization term and differentiable token chunking. ALToLLM integrates this into an MLLM, using Group Relative Policy Optimization (GRPO) to balance mask quality and efficiency. Experiments show ALToLLM achieves state-of-the-art results on segmentation benchmarks while reducing token usage and generation time.", "questions": "Please see weaknesses.", "limitations": "The authors state in the checklist that \"The limitations of our work are discussed in the Supplementary Material.\" However, I could not find a corresponding section discussing the limitations in the appendix or supplementary material.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. ALTo allows the model to automatically decide the token length for mask generation, which sounds reasonable because the number of tokens required to represent fine-grained edge shapes can vary drastically depending on their complexity.\n2. This work utilizes GRPO to dynamic control over token length, which balances mask quality and computational efficiency.\n3. ALToLLM achieves state-of-the-art performance with adaptive token usage across various object segmentation benchmarks.\n\nWeaknesses:\n1. The paper does not provide a quantitative evaluation regarding the effectiveness of the proposed adaptive length tokenizer (ALTo); for example, how does ALTo handle extremely complex objects that might require more than 32 tokens?\n2. The paper claims that \"We employ GRPO to further adjust specific preferences on the trade-off between mask quality and token efficiency.\" However, it does not seem to provide a quantitative analysis of the impact of GRPO on mask quality and token efficiency.\n3. In Figure 1, the first three cases presented involve relatively simple segmentation scenarios, where increasing the number of mask tokens does not seem to have a significant effect. The authors should provide more complex scenarios to better demonstrate the effectiveness of the adaptive token approach.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed my concerns and I tend to accept this paper.", "rebuttal": "We thank the reviewer for insightful and constructive feedback on our work. \n\n>  **W1.1: Provide a quantitative evaluation**\n\n**A:** We clarify that the effectiveness of the proposed adaptive-length tokenizer (ALTo) is quantitatively evaluated on our MultiSA1B dataset, which contains complex shapes and multi-object masks, as shown in **Figure 8**. The evaluation includes multiple settings of the length penalty coefficient. With a coefficient of 0.01—which is the value selected for our final model—we achieve an average output length of 21.51 (standard deviation: 6.38) and a high average mask IoU of 0.938, demonstrating the strong effectiveness of our approach in balancing token efficiency and segmentation accuracy.\n\nBeyond the tokenizer itself, the advantages of our method are further supported by the state-of-the-art performance and significantly reduced token usage of ALToLLM-8B in **Table 2**.\n\n>  **W1.2: How does ALTo handle extremely complex objects that might require more than 32 tokens**\n\n**A:**  **Theoretical Basis**. The maximum token length of 32 in ALTo is a deliberate design choice, inspired by the finding in TiTok [11] that \"an image is worth 32 tokens,\" which demonstrates that a compact 1D sequence of 32 tokens can effectively represent complex visual information.  **Empirical Validation**. (1) Our extensive experiments, including those on the highly complex Multi-Target-SA1B dataset, have shown that 32 tokens are sufficient to achieve high-quality segmentation.  (2) As shown in Figure 8, our selected penalty coefficient (0.01) naturally leads to token lengths below 32 for most samples, with fewer than 0.5% of cases reaching the maximum. (3) As evidence in the response to W3, even for complex multi-object scenes, such as an image containing six distinct objects, our model could still mask all objects within 32 tokens. This design ensures a favorable balance between representational capacity and computational efficiency.\n\n>  **W2: Provide a quantitative analysis of the impact of GRPO**\n\n**A:** In fact, **Figure 7** illustrates how GRPO influences mask quality and token efficiency. The x-axis shows IoU, while the y-axis represents the average sequence length. Blue points correspond to ALToLLM variants trained with different GRPO penalty coefficients, whereas red points indicate the fixed-length baseline. When comparing models with similar IoU (i.e., at the same x-coordinate), ALToLLM achieves comparable segmentation performance while reducing token usage by more than 5 tokens per sample on average—demonstrating superior **mask quality and token efficiency**.\n\nFurthermore, as the length penalty coefficient in GRPO training decreases, mask quality improves at the expense of longer output sequences, confirming a controllable trade-off between accuracy and efficiency. This enables users to flexibly tailor the model’s behavior based on whether **mask quality or token efficiency** is prioritized in a specific application. As shown in **Figure 6**, this adaptation is achieved within just a few hundred training steps, supporting our claim in **L170** that GRPO provides **flexible** and efficient control over the model’s output characteristics.\n\n>  **W3: Provide more complex scenarios**\n\n**A:** We sincerely appreciate the reviewer's insightful feedback regarding the demonstration of ALTo's adaptive capabilities. While our initial **Figure 1** illustrated the progressive token allocation for objects with increasing edge complexity (from 4 tokens for a simple moon to 17 tokens for complex land boundaries), we fully agree that more sophisticated multi-object scenarios would better showcase our method's advantages. To comprehensively address this point, we analyze a complex scene with sequentially added objects. The results demonstrate ALTo's dynamic scaling of token allocation: beginning with 7 tokens for a single woman, the count appropriately increases to 30 tokens when incorporating fine details like flowers on a table. This systematic evaluation confirms our method's ability to automatically adjust token usage based on both object quantity and geometric complexity.\nDue to the NeurIPS rebuttal constraints, we present these findings in tabular form below and will include the complete visual analysis in the supplementary material for the final version. \n| Object Description       | Number of Mask Tokens | Number of Objects |\n|--------------------------|-----------------------|-------------------|\n| the woman on the left    | 7                     | 1                 |\n| + the woman on the right  | 18                    | 2                 |\n| + suitcase                | 20                    | 3                 |\n| + hat                     | 22                    | 4                 |\n| + blanket                 | 25                    | 5                 |\n| + flower on the table     | 30                    | 6                 |\n\n>  **Limitations**\n\n**A:** We sincerely apologize for this oversight in the supplementary material. The \"Limitations and Future Work\" section was inadvertently omitted during the final formatting stage. We will immediately add this section to the supplementary material.\n\n **Limitations and Future Work**\n\nWhile ALTo effectively handles most segmentation tasks with adaptive token lengths (1-32 tokens), two key directions merit further exploration.  First, our approach requires multiple training stages, which increases the engineering complexity. A simpler training pipeline is desirable for future applications. Second, though our pipeline is designed to be modality-agnostic (treating mask tokenization as a special case of image tokenization), our experiments currently focus on mask validation. Future work will extend ALTo to RGB image tokenization and validate its effectiveness across more general vision tasks."}]

Author rebuttal:

Meta-review: The submission address tokenization in image generation and multimodal frameworks, specifically the fact that the tokens often (but not always) conform to a rigid and fixed length structure, irrespective of the content. They propose a tokenizer that utilizes an adaptive length, employing a token length predictor, a length regularizer, and a token chunking technique. 

This is an important problem acknowledged by many in the community, with numerous parallel efforts underway. Despite that, all reviewers recognize that the submission has novel merits and vote for acceptance, though mostly with a borderline stance. The AC agrees and recommends acceptance.
Decision: Accept (poster)