Title: ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimodal generation

multimodal large language model

adaptive-length
Abstract: While humans effortlessly draw visual objects and shapes by adaptively allocating attention based on their complexity, existing multimodal large language models (MLLMs) remain constrained by rigid token representations. Bridging this gap, we propose ALTo, an adaptive length tokenizer for autoregressive mask generation. To achieve this, a novel token length predictor is designed, along with a length regularization term and a differentiable token chunking strategy. We further build ALToLLM that seamlessly integrates ALTo into MLLM. Preferences on the trade-offs between mask quality and efficiency is implemented by group relative policy optimization (GRPO). Experiments demonstrate that ALToLLM achieves state-of-the-art performance with adaptive token cost on popular segmentation benchmarks. Code and models will be released.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ALTo (Adaptive-Length Tokenizer) and ALToLLM, a framework that enables MLLMs to generate adaptive-length mask tokens for object segmentation. ALTo uses a Token Length Predictor (TLP) to determine the optimal number of tokens based on object complexity, combined with a length regularization term and differentiable token chunking. ALToLLM integrates this into an MLLM, using Group Relative Policy Optimization (GRPO) to balance mask quality and efficiency. Experiments show ALToLLM achieves state-of-the-art results on segmentation benchmarks while reducing token usage and generation time.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. ALTo allows the model to automatically decide the token length for mask generation, which sounds reasonable because the number of tokens required to represent fine-grained edge shapes can vary drastically depending on their complexity.\n2. This work utilizes GRPO to dynamic control over token length, which balances mask quality and computational efficiency.\n3. ALToLLM achieves state-of-the-art performance with adaptive token usage across various object segmentation benchmarks.\n\nWeaknesses:\n1. The paper does not provide a quantitative evaluation regarding the effectiveness of the proposed adaptive length tokenizer (ALTo); for example, how does ALTo handle extremely complex objects that might require more than 32 tokens?\n2. The paper claims that \"We employ GRPO to further adjust specific preferences on the trade-off between mask quality and token efficiency.\" However, it does not seem to provide a quantitative analysis of the impact of GRPO on mask quality and token efficiency.\n3. In Figure 1, the first three cases presented involve relatively simple segmentation scenarios, where increasing the number of mask tokens does not seem to have a significant effect. The authors should provide more complex scenarios to better demonstrate the effectiveness of the adaptive token approach.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nThe authors state in the checklist that \"The limitations of our work are discussed in the Supplementary Material.\" However, I could not find a corresponding section discussing the limitations in the appendix or supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an adaptive method, ALTo,  for generating adaptive length tokens for the task of image segmentation. The length of the token sequences depends on the difficulty of the segmented objects. In addition, ALTo can be plugged into the Multimodal Large Language Model (MLLM) through SFT and RL, enabling the task of referring segmentation. Through the comprehensive experiment, the paper demonstrates the efficiency of the adaptive length tokens compared with the baselines: Achieving high segmentation results while being fast compared with the fixed-length token mechanism.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**: This paper tries to improve HimTok [1] in terms of the number of tokens for the problem of pixel understanding (segmentation) in the MMLLM model. The approach is to use an adaptive token strategy, which uses a different number of tokens for each segmentation task, depending on the difficulty of the segmentation. This approach is intuitive, and the experiment indicates the efficacy of their approach.\n\n**Weakness**: Most of the parts are borrowed from the HimTok work, and only token adaptation is the main contribution of this work. However, the computation of the token adaptation at some point (straight-through estimator in L153) is not clear. The changes in terms of the number of tokens should not affect the results of the baselines MMLLM in some evaluation metrics besides segmentation, e.g., segmentation reasoning and general image understanding, which is not reported in this paper. Further ablation study is required to indicate the efficacy of each component (training phase) with the overall performance. However, the proposed approach is interesting for reducing the number of tokens used in HimTok.\n\n[1] Wang, Tao, et al. \"Himtok: Learning hierarchical mask tokens for image segmentation with large multimodal model.\" arXiv preprint arXiv:2503.13026 (2025).\n\nQUESTIONS:\n- Restricted only to segmentation: Can this approach be extended to other tasks like Image understanding and region understanding, similar to GLaMM[1]? What is the performance of the adaptive tokenization for the Reasoning Segmentation task? Please refer to Sections 4.4 and 4.7 for metrics which is not Segmentation.\n- Datasets used in the experiments: What is the difference in terms of problem setting between different types of datasets used in this paper? For example, what is the difference between gRefCOCO, RefCOCO, and RefCOCO+? The paper should include the difference between these types of datasets for the reader to understand.\n- Ablation study: This paper should include the ablation study about the importance of different stages to the final results. For example, how important of RL phase compared with the results of the SFT phase only? \n- In L153, what is $\\hat{P}$? It is never defined.\n- In L156, to encourage short token regularization, is $\\mathcal{L}_{length}$ should be $\\lambda L$ instead of $-\\lambda L$.\n- Why is the method used to compare with ALToLLM in Table 2 and Table 3 not consistent? The method list is not similar for both tables, making it harder to understand what this paper is trying to compare.\n- What is the main factor that makes the difference between Fix-length ALTo with Fix-length tokenization compared with HimTok in Table 2 and 4?\n\nReference:\n[1] Rasheed, Hanoona, et al. \"Glamm: Pixel grounding large multimodal model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nLIMITATIONS:\nSee the question section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel adaptive-length tokenizer called ALTo. The contribution of ALTo is its ability to dynamically determine the number of mask tokens needed to represent visual objects based on their complexity, enabling efficient and flexible mask generation for segmentation tasks. At the core of ALTo is a Token Length Predictor (TLP), which predicts the optimal token length using global and local mask features, combined with a differentiable token chunking strategy and length regularization loss. This allows the model to encode segmentation masks into variable-length token sequences—ranging from 2 to 32 tokens—rather than using a fixed length, which often leads to over- or under-representation of content.\n\nBuilt upon this tokenizer, the authors develop ALToLLM, a multimodal large language model trained in four stages: (1) pretraining the mask tokenizer and de-tokenizer, (2) fine-tuning the TLP, (3) supervised fine-tuning of the MLLM with adaptive mask tokens, and (4) Group Relative Policy Optimization (GRPO) to flexibly balance segmentation accuracy (measured by IoU) and token efficiency via a composite reward. Experiments across standard benchmarks demonstrate that ALToLLM achieves state-of-the-art performance while reducing average token usage and inference time.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- **Well-designed and novel Token Length Predictor (TLP):** The training strategy for TLP is innovative and empirically validated across diverse benchmarks, demonstrating its effectiveness.\n- **Careful architectural thoughtfulness:** The attention to detail in TLP’s design (e.g., differentiable chunking, attention-based gating) reflects a high level of engineering insight.\n- **Strong empirical results:** The model consistently achieves state-of-the-art performance while improving token efficiency, showing practical utility of adaptive-length modeling.  \n&nbsp;  \n### Weaknesses\n- **Limited scope of application:** Although the proposed adaptive tokenizer has broader potential (e.g. image tokenization), the paper evaluates it solely on mask tokenization tasks. While this choice may be justified, it would be valuable to discuss why the method is especially suitable for masks and whether it can be generalized beyond this domain.\n- **Lack of baseline with fixed optimal length:** The paper does not compare against a fixed-length model set to the optimal length determined by the adaptive method—this makes it hard to isolate whether the gain comes from adaptivity or just better length calibration.\n\nQUESTIONS:\n- How exactly are the tail tokens dropped during training? It would improve clarity if the paper explained this procedure in more detail.\n- The explanation of the cumulative sum might be clearer if rewritten using the form $P_i = 1 - \\sum_{j < i} p_j$, to emphasize the contribution of earlier tokens to the stopping probability.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper modifies the HimTok approach to image segmentation in several ways, obtaining better IoU scores on multiple referring expression and open vocabulary segmentation datasets. \nHimTok originally uses multimodal LLMs to perform image segmentation by expanding the LLM vocabulary size with discrete mask tokens, training the multimodal LLM to predict mask tokens, and outputting segmentation masks based on the predicted mask tokens.\nThe proposed modifications include (1) adaptively using a variable number of tokens to predict a segmentation mask, (2) adding a pixel encoder to provide additional information to the mask de-tokenizer, (3) using GRPO to tune the multimodal LLM, and (4) freezing and finetuning different parts of the model during different stages. (e.g. freezing everything except the LLM versus only freezing the mask tokenizer during Stage 2)\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe idea of allowing the model to adaptively decide how many tokens to use to represent an image is interesting. \nThe final pipeline brings improvements to both accuracy and efficiency (when compared against the fixed-length pipeline). \nThe length penalty hyperparameters for the loss function in Stage 1.5 and Stage 3  are also reasonably well studied, showing the exact trade-off between IoU and average mask token sequence length in the final models after Stage 3, as well as advantages of using 0.01 as the length penalty in Stage 1.5.\n\n\nWeaknesses\n\nMy main concern is the lack of ablation studies: \nConsidering the high similarity to HimTok, it is crucial to investigate the effects of each modification to the pipeline. However, only the adaptive token length predictor and the pixel encoder are studied, and only to a limited degree. This leaves the reader uncertain where the gains over HimTok come from. For example, does ALToLLM still always outperform HimTok when no pixel encoders are used? What about when no GRPO is used? What if HimTok also only trained the LLM in Stage 2 training? Or vice versa, what if ALToLLM only freezes the mask tokenizer during Stage 2?\n\nQUESTIONS:\nIn addition to the limited ablation studies mentioned above, some details are unclear.\n1. What is the mask token codebook size? 1024 as in HimTok?\n2. The reward function seems to suggest that the composite reward is non-zero for an incorrectly formatted sample. Does this mean that the incorrectly formatted sample is still somehow given to the mask de-tokenizer for generation?\n\nAlso, at the end of the introduction, it is mentioned that for most cases the length is 16, and some examples are given in the first page and the appendix. I'm curious if the authors have a plot of the distribution of mask token sequence length? And are there any oddities where the model uses many tokens for a seemingly simple mask, or few tokens for a seemingly complex mask?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-08"}