WOW-Seg: A Word-free Open World Segmentation Model
Authors: Danyang Li
Tianhao Wu
Bin Lin
Zhenyuan Chen
Yang Zhang
Yuxuan Li
Ming-Ming Cheng
Xiang Li
Venue: ICLR 2026 Poster (2026)
Keywords: vision language model
open world
segmentation
object recognition
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A model with open-world segmentation capabilities. It can receive visual prompts, or word lists, or even input without prompts.

Abstract: Open world image segmentation aims to achieve precise segmentation and semantic understanding of targets within images by addressing the infinitely open set of object categories encountered in the real world.  However, traditional closed-set segmentation approaches struggle to adapt to complex open world scenarios, while foundation segmentation models such as SAM exhibit notable discrepancies between their strong segmentation capabilities and relatively weaker semantic understanding. To bridge discrepancies, we propose WOW-Seg, a Word-free Open World Segmentation model for segmenting and recognizing objects from open-set categories. Specifically, WOW-Seg introduces a novel visual prompt module, Mask2Token, which transforms image masks into visual tokens and ensures their alignment with the VLLM feature space. Moreover, We introduce the Cascade Attention Mask to decouple information across different instances. This approach mitigates inter-instance interference, leading to a significant improvement in model performance. We further construct an open world region recognition test benchmark: the Region Recognition Dataset (RR-7K). With 7,662 classes, it represents the most extensive category-rich region recognition dataset to date. WOW-Seg attains strong results on the LVIS dataset, achieving a semantic similarity of 89.7 and a semantic IoU of 82.4. This performance surpasses the previous SOTA while using only one-eighth the parameter count. These results underscore the strong open world generalization capabilities of WOW-Seg. The code and related resources are available at https://github.com/AAwcAA/WOW-Seg-Meta.

Published as a conference paper at ICLR 2026
WOW-SEG: A WORD-FREE OPEN WORLD SEGMEN- TATION MODEL
Danyang Li Tianhao Wu
Bin Lin
Zhenyuan Chen
Yang Zhang
Yuxuan Li
Ming-Ming Cheng! 123
Xiang Li
'NKIARI, Shenzhen Futian
VCIIP CS, Nankai University AAIS, Nankai University 'Peking University Shenzhen Graduate School
"Sichuan Agricultural University
[danyang. 1i, xiang 1i. implus @nankai. edu. en
category
RR RR-7K
COCO
WILLK carton pot
chee
Box1
Chef vegetables
Large Longuage Model
w/ Cascade Attention Mask
Cityscopes
Garrot apren chef hat person basket
PACO
D Vision Encoder
53
The Word free Open World Segmentation Madel
Tokenizer
R
Open Images
220
Mask2Token
LVIS
visual prompt
mask
ADE-2OK
(a) Application Scenario
(b) WOW-Seg Architecture
(c) RR-7K: new test dataset
Figure 1: (a) Application scenarios for WOW-Seg. WOW-Seg accepts visual prompts of any form as input, outputting corresponding masks and categories.
(b) Core architecture of WOW- Seg: Mask2Token and Cascade Attention Mask. Mask2Token converts input masks into visual tokens aligned with the VILLM feature space. Cascade Attention Mask addresses the issue of "inter- instance interference" by decoupling multi-mask features, enabling efficient parallel training. (c) Comparison the proposed Region Recognition Dataset (RR-7K) with other datasets in terms of categories.
ABSTRACT
Open world image segmentation aims to achieve precise segmentation and se- mantic understanding of targets within images by addressing the infinitely open set object categories encountered the real world. However, traditional closed- set segmentation approaches struggle to adapt to complex open world scenar- ios, while foundation segmentation models such as SAM exhibit notable discrep- ancies between their strong segmentation capabilities and relatively weaker se- mantic understanding. To bridge these discrepancies, we propose WOW-Seg, a Word-free Open World Segmentation model for segmenting and recognizing ob- jects from open-set categories. Specifically, WOW-Seg introduces a novel vi- sual prompt module, Mask2Token, which transforms image masks into visual tokens and ensures their aligmment with the VLLM feature space. Moreover, we introduce the Cascade Attention Mask to decouple information across dif- ferent instances. This approach mitigates inter-instance interference, leading to significant improvement in model performance. We further construct an open world region recognition test benchmark: the Region Recognition Dataset (RR-7K). With 7,662 classes, it represents the most extensive category-rich re- gion recognition dataset to date. WOW-Seg attains strong results on the LVIS dataset, achieving a semantic similarity of 89.7 and a semantic IoU of 82.4.
"qqual Contribution Corresponding Author

---

Published as a conference paper at ICLR 2026
This performance surpasses the previous SOTA while using only one-eighth the parameter count. These results underscore the strong open world generaliza- tion capabillities of WOW-Seg. The code and related resources are available at httpp:://g com/ /A////AAAAAAAAAAA Wtt////W-SSeeee -ee---eg--------
INTRODUCTION
As fundamental technology in computer vision, image segmentation aims to deconstruct images into regional units with semantic value. This technology is widely applied in autonomous driv- ing (Muhammad et al., 2022 Zendel et al. 2022), medical image analysis (Azad et al.] 2024 [Buto ee al. 2023), image editing and synthesis Chen et al. (2019); [Meng et al. (2025). among other fields. Throughout the development of image segmentation research, technological iterations have largely
centred on optimising accuracy and enhancing efficiency (Ronneberger et al.] 2015 Guo et al.)
2022 Xie al. 202T Cheng et al. 2022). However, when contronted with the boundless variety of object categories and complex, ever-changing scene combinations in the real world, these meth- ods commonly suffer from reliance on closed datasets and limitations in model transferability (Li et al. (022). They struggle to satisfy segmentation demands in dynamic scenarios, necessitating breakthrough solutions to propel progress in the field.
Deep learning based image segmentation algorithms can be categorized into three types: closed- set segmentation, open open-vocabulary segmentation, and vision language model based segmentation. Closed-set methods assign each pixel to predefined categories via a fixed segmentation head (Ron-
neberger et al. 2013: Guo et al. 2022; Xie et al. 2021 Cheng et al., 2022). while open-vocabulary
methods match pixel regions with category labels, though their openness is limited by vocabulary
size (Li et al.] 2022 Rao et al.j 2022; Dong et al. 2023). Vision language models drive segmenta-
tion through text instructions, but results heavily depend textual guidance (Lai et al. [0244] Yang jet al. 2023 Wei et al. [025] Ren et al. (024b)). Consequently, these methods are constrained in the categories they can segment and struggle in open-world scenarios.
Recent advances explore open-world segmentation: SAM (Kirillov et al. 2023) and SAM 2 (Ravi al. 2024) segment objects Vasssagnnsticallly, but lack regional semantic understanding. DAM incorporated positional information into the Localised Vision Backbone to enable regional under- standing (Lian et al.] 2025), and PAM combined SAM2 with large language models for diverse regional semantic outputs (Lin et al., 2025). However, both only handle single mask during train- ing and inference. VP-MILMM introduced a visual prompt encoder and multi-mask training, yet. ignored interference between correlated masks (L.in et al. 2024).
Our analysis reveals that the development of current open world segmentation models is constrained by several key factors. Firstly, both existing close set segmentation methods and open vocabulary segmentation approaches are limited their output capabilities by predefined categories. Secondly, segmentation approaches based on visual large language models necessitate the appropriate incorpo- ration of visual prompts. Moreover, prior open world segmentation models can only process a single mask during single forward inference, resulting in speed limitations in multi instance interaction scenarios (Lian et al., 2025) Lin et al.) 2025) Yuan et al., 2024). Additionally, most existing seg- mentation datasets contain only tens to a thousand categories, failing to comprehensively evaluate model's 's open world understanding capabilities. To address the current limitations of segmentation models in open world scenarios, we propose WOW-Seg.
As shown in Fig. aa), WOW-Seg is word-free open world segmentation model. It segments open world scenes through visual prompts and uses the powerful cognitive abilities of VLM to recognize masks The core architecture of WOW-Seg is shown in Fig. b). We have devised a novel vi- sual prompting approach: Mask2Token. extracts features by processing specific masked regions, ensuring the mask tokens (purple squares) align with the VLLM's feature space while pruning ir- relevant background tokens (purple chequered squares). Moreover, our proposed Cascade Attention Mask mitigates interference between object instances by decoupling their respective features dur- ing inlti-instance training and inference. To comprehensively evaluate the performance of open world segmentation models, we propose new test benchmark: the Region Recognition (RR-7K). Its comparison with other datasets is shown in Fig. [c). RR-7K comprises 7,662 categories, over 80k images, and more than 200k mask objects. Compared to previous evaluation methods using
2

---

Published as a conference paper at ICLR 2026
datasets such as LVIS, RR-7K features richer categories, enabling more thorough assessment of models' understanding capabilities in open world scenarios.
Overall, our contributions
follows:
WOW-Seg is a word-free open world segmentation model that encodes multiple masks as visual prompts with Mask2Token and autoregressively identifies each mask via Cascade Attention Mask, enabling detection of any object without text guidance.
We introduce RR-7K, an open-world region recognition dataset with 7,662 categories and more than 200k masks, providing comprehensive benchmark for evaluating model gen- eralization.
WOW-Seg achieves state-of-the-art performance with fewer parameters, surpassing previ- ous SOTA by 4.1 Semantic IoU on LVIS and 4.3 on PACO.
RELATED WORK
Open-set image segmentation. Closed-set image segmentation algorithms have achieved remark- able performance. However, they rely on predefined categories, which limits their ability to handle novel classes adapt to new domains (Bucher al., 2019 Yin et al., (024). Open-vocabulary segmentation extends semantic coverage by integrating text embeddings with visual features (Dong
2023 Li al. 2022 Rao et al. (2022); Chen et al. (022b)) reformulates CLIP's image- ct al.
text matching as a pixel-level task for dense predictions. Despite these advances, open-vocabulary methods still require predefined categories and cannot predict truly unknown concepts. The Segment Anything Model achieves zero-shot segmentation without category annotations (Kirillov et al. 2023] Ravi al. 2024), but cannot assign semantic meaning the segmented regions. To reduce reliance on predefined categories, some studies explore unguided and automatic vocabulary models (Kawano & Aoki 2024 Ulger al., 2025 Rewatbowornwong et al., 2023 Shin et al. 2024). attempting to autonomously discover semantic labels. In contrast, our method encodes multiple masks as visual prompts and employs an autoregressive approach to identify each mask independently, allowing detection of any object without text guidance.
Vision Language Large Model. Vision- languan large models (VLLMs) exhibit strong generaliza-
tion in tasks such as image captioning and visual question answering by jointly learning visual and
textual representations (Liu et al. 2023 Chen et al.] 2024a; Bai et al., 2025 Shabbir et al. 2025
Liu et al. 2025 Li al. 2025 Zhu al. 2025 Zhang al. 2025 2026 Zhao et al. 2025b a). To
transfer this leved cognitive ability to segmentation. Lai et al. (2024) proposed LISA, which
segments user-specified objects based text instructions. However, its performance remains heav- ily dependent on user-proviis provided text. Efforts to incorporate regional understanding into VL.LMs have introduced additional encoders embed both image features and positional information masked
regions (Lin et al.] 2024 Yuan et al.) 2024 Rasheed et al. 2024). Yet, the tokens produced by these encoders lie outside the original
VLLM feature distribution, requiring extensive training for alignment. Moreover, (Lin et al. 2025 Lian et al. 2025 Yuan et al. 2024) rely on single-mask sam- ples during training and inference, substantially prolonging model training. While VP-MLLM (Lin) et al. 2024) supports reasoning over multiple masks in a single sample, it does not account for inter- mask correlations, and direct multi-mask training leads to interference that degrades performance. In contrast, our approach leverages the Cascade Attention Mask module predict each mask inde- pendently, avoiding semantic interference between correlated masks and improving the efficiency of multi-mask parallel inference.
3 METHOD
To achieve the objective segmenting open world scenes using solely visual prompts, we designed WOW-Seg, innovative segmentation and recognition framework based autoregressive mech- anisms. The core concept of WOW-Ses is to transform the object category recognition task from a traditional classification problem into visually text generation problem. The entire frame- work is shown Fig. 2 It consists of four core modules: Mask2Token, Cascade Attention Mask, Mask Generator and VLL.M.
3

---

Published as a conference paper at ICLR 2026
Response: chef, bowl, fchef, vegetables, milk carton, pot, carrot, apron, chef hat, person, basket...
Cascade Attention Mask
Large Language Model
flatten f
Fixed Text: segnent all masks..
Tokenizer
Vision Encoder
image
Dynamic High Resolution
Image
Mask Generator
Mask2Tokee
Multi
Masks
Trainable params
Frozen params
Fixed
Image token
masks token
Per mask token
The overall framework of WOW-Seg. WOW-Seg will perform corresponding tokenisation on fixed text and input images. WOW-Seg employs Mask2Token to convert the received mask into visual tokens. These tokens then collectively fed into a large language model featuring Cascade Attention Mask mechanism. Ulimmaely, WOW-Seg completes the open-words category output.
Figure 2:
3.1 WOW-SEG FRAMEWORK
The proposed WOW-Seg model is built upon encoder-decoder framework and utilizes VLLM as its foundation. Given an image and of corresponding masks as input, the model autoregressively generates the category name for each mask. The visual encoder first processes the input image to extract sequence image tokens, which provide rich contextual information for recognition. During the training phase, use ground truth masks as input. During the inference phase, users may flexibly select from multiple mask generators to produce masks, such as SAM (Kirillov et al.) 2023 The Mask2Token module processes the imput masks by mapping them into tokens within the Vision Language Model's (VLM) emhedding space. These mask tokens, along with tokens from the image and predefined text prompt, then fed into the Large Language Model (LLM) decoder. Within the decoder, we employ our novel Cascade Attention Mask to ensure that the prediction for each mask is generated independently. mitigating interference between instances. By leveraging the standard "next token prediction" mechanism of the LLM, our approach bypasses the category limitations of traditional segmentation heads, enabling the model to recognize an open set of categories.
3.2 MASK2TOKEN
Prior methods have often employed a dedicated module to encode binary masks into feature tokens for downstream Vision Language Models (Lin al.) (2024). While this enables the integration of mask information, the resulting tokens introduce distributional gap, as they exist outside the VLM's pretrained embedding space. Bridging this gap typically requires substantial retraining to align the new feature distribution.
The Mask2Token module, illustrated Fig, 3 designed embed mask-specific visual and spatial features into the VLM. For each input mask, the module first crops a mask region image that includes the mask and its surrounding context (default context scale: 2). The mask region image then resized to the visual encoder's standard input dimension (448 448) and processed by a shared- weight vision encoder to produce grid of 16 16 image tokens. Using a shared-weight encoder crucial as it ensures the resulting features lie in the same embedding space as the global image features. Concurrently, the binary mask downsampled to a 16 x 16 resolution. Finally, this downsampled mask is used Notably, Mask2Token processes multiple masks in parallel, enabling features from several distinct objects be fed into the LLM simultaneously.
guide to select the corresponding tokens from the feature grid.
4

---

Published as a conference paper at ICLR 2026
mask region image 448
region feature
mask feature
mask
extract feature
iagge
vision
encoder
flatten
mask tokens
Figure 3:
Detailed illustration of our Mask2Token. Only the processing procedure of mask is presented here. In fact, Mask2Token receives multiple masks and performs parallel processing.
3.3 CASCADE ATTENTION MASK
While Vision-Language Model can be trained using single mask per sample (SM), this approach is less efficient than training with multiple masks per sam- ple (MM). The MM strategy allows the model to process an image within a single forward pass, which better reflects the nature of open- world scenes that typically contain mul- tiple independent objects. However, this MM strategy introduces critical chal- lenge: inter-instance interference, where the model may incorrectly associate fea- tures from different objects. To explicitly prevent this detrimental information leakage, we designed the Cascade Attention Mask.
imoge
mask3
maxk2
obj2
fixed
object instances
otterts
Traf fic
sigh
(a) Causal attention mask (b) Cascade attention mask
Figure 4: Causal Attention Mask and Cascade Atten- tion Mask.
Under the effect of the original causal attention mask of the large language model, assuming there K objects, the probability of these K object names being predicted is shown in the following formula:
P PO (2,,,2...., OK Image; T; M) = II P PO; Image; T; Oo, OI....,,,,1)) 0,-1). (I)
i=1
Among them, represents object to be predicted, Image represents the image token, repre- sents the text token, and M {m1, m....., ,,. represents the mask tokens. It is easy observe that when predicting the i-th object, the model refers to the mask prompts and output results from the (-thh to the (i 1)-th objects. We hope that the name of the i-th object is guided solely by its corresponding i-th mask. Therefore, we designed the Cascade Attention Mask.
The detailed mechanism of the Cascade Attention Mask is illustrated in Fig. 4T),, which explain using an example of single image containing two masks: a "traffic sign" and a "fire truck". The image token corresponds to the blue region. The token corresponds the purple region, with the input text (start) also corresponding to the purple region. The features corresponding to the first object (traffic sign) are represented by the yellow region. The features corresponding the second object (fire truck) are represented by the red region. (sep) serves as the separator. The attention structured to ensure their category names are decoded independently. Tokens corresponding to the image and the text prompt are universally accessible to all other tokens. However, the core of our approach lies in isolating the instance-specific tokens. The input mask tokens for "traffic sign" and "fire truck" are masked from each other, preventing information leakage. During autoregressive generation, the process is as follows: to generate the output "traffic sign", the model can only attend to the input mask tokens for the traffic sign and its previously generated tokens (e.g.. "traffic"), Crucially, it cannot attend to the mask tokens of the "fire truck". Similarly, the generation "fire truck" is causally conditioned only its mask tokens and preceding outputs. This parallel yet
5

---

Published as a conference paper at ICLR 2026
decoupled decoding process ensures that the prediction for each object is not influenced by others. Overall, the design adheres to the following principles:
1. Masks are independent of each other. Masks appearing later do not carry any information from carlier masks.
2. Objects are independent of each other. The prediction of object is not affected by objects 1 to
i 1-1.
3. Decouple each pair of mask and object. When predicting object i, the information available to the model is limited to image tokens, text prompt tokens, and the i-th mask token.
When predicting the object i, the only information that the model can obtain are image tokens, text tokens, and the i-th mask token. Ultimately, the Cascade Attention Mask enables the model to learn without interference from additional factors. When designing the Cascade Attention Mask, also explored its variants. The specific design of the variant is detailed in Fig. [0 of the appendix. Its primary distinction from the Cascade Attention Mask lies in whether it simultaneously adheres the principles that "Masks are independent of each other" and "Objects are independent of each other". We also conducted ablation experiments comparing the performance of the Cascade Attention Mask and its variants, as shown in Tab.
Under the Cascade Attention Mask, the predictions for different objects are conditionally indepen- dent. Specifically,
P O2........, Image; M) II P P(, Image; T; M), 1=1
where M {m,, m2, ..., mx} denotes the set of all object mask tokens. This factorization reflects the fact that, under the Cascade Attention Mask, the prediction of the i-th object O, is conditionally independent of the mask tokens of all other objects given the image and text tokens. Although the conditioning set M includes the mask tokens of all objects, the attention pattern enforces that O only attend to its own mask tokens m, As result, the probability term satisfies:
P PO; Image; T; M) P (O; Image; T; T;m;), meaning that only the i-th mask contributes to the prediction of the i-th object.
(3)
3.4 RR-7K DATASET
Currently, the performance evaluation of open world models is mainly conducted on common classes. To comprehensively measure the performance of open-world models, we propose the Re- gion Recognition Dataset (RR-7K).
The images and mask annotations of RR-7K are from SA-1B (Kirillov et al. 2023) We designed sophisticated data annotation pipeline to assign categories each mask in RR-7K. As shown in the pipeline consists of three stages.
Fig. 5
Mask patch category inference. There are a large number of meaningless tiny masks in SA- IB, which greatly increases the inference cost but fails to obtain high-quality data. Therefore, we remove masks where the proportion of mask pixels relative to the total pixels of the entire image is excessively low. Subsequently. for each mask, we used tools such as Qwen2.5VL-27 (Bai et al.) 2025) and Grounded SAM (Ren et al. 2024a) to obtain the corresponding category.
Hallucination filtering. After the category inference of mask patches, we obtained a large number of mask-categne categry pairs. These pairs exhibit long tailed distribution, and there is large amount of incorrect data caused by the hallucination phenomenon of large language models. For tail categories, the cognitive ability of large language models is not good. Therefore, we use Intern VL.-78B to perform hallucination filtering only the data of head categories. We will re-ask question about a mask-category pair: "Is the area outlined by the red contour and covered by the red mask in the image (class name)? ? Please answer yes no." By re-asking questions for each mask- pair, a large amount of incorrect data was screened out.
Manual screening. We will manually screen all the tail categories and the head categories after hallucination filtering. Since each mask now has a category name, we only need to delete the samples with inconsistent categories, which incurs low labor costs.

---

Published as a conference paper at ICLR 2026
① Mask patch category inference
QwenVL
window
car
sidewalk
G-SAM
frequent
rare
Hallucination filtering the object covered by the red mask (car)?
Manual screening
Save!
True
lnternVL
car
False
frequent class
Figure 5: Data annotation pipeline. The RR-7K we proposed mainly goes through three stages:
Mask patch category inference, Hallucination filtering and Manual screening.
Number of samples
201
Percent of samples
20a
100
1000
2000
3000
4000
5000
6000
7000
8000
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Sorted category
mask
(a) The number of samples per category.
(b) The distribution of relative mask size
Figure 6: Statistical distribution of RR-7K. The relative mask size denotes the square root of the mask area divided by the image area.
After going through the above data annotation pipeline, we obtained the final RR-7K. contains over 80k images, with more than 200k instances and 7,662 categories. The distribution of sample counts per category and the relative size distribution of masks are shown in Fig. G and Fig. b respectively. Further details are presented in the appendix. The RR-7K contains a large number of categories in the open world, providing a better test benchmark for open world segmentation and understanding.
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We use the pre-trained Intern (Zhu et al. 2025) as the base model for WOW-Seg. The model is trained on 8 NVIDIA H100 GPUs. We use the AdamW optimizer with a learning rate set to le and batch size set to 32. For the final reported model performance, we use LVIS GGupta al. 2019), PACO (Ramanathan et al. 2023), and COCO Stuff (Caesar et al., 2018) for epochs of traiing. By default, the maximum number of masks contained in a single data sample used in all experiments does not exceed 30. This is to ensure load balancing on each GPU during multi-GPU training.
4.2 OPEN WORLD REGION RECOGNITION
This task requires identifying the object category within masked area without given vocabulary or any other text prompts. We follow the settings of Yuan et al. (2024) and and[Lin et al. (2024), and use semantic similarity and semantic intersection over union measure the performance of the model. We evaluate the model on three datasets in total. LVIS (Gupta et al. 2019). a large-scale instance segmentation benchmark, contains over 1,200 long-tailed distributed categories, aiming to test the model's 's generalization ability to rare objects. PACO (Ramanathan et al., (023), a dataset focusing fine-grained understanding. It not only contains complete objects but also provides detailed

---

Published as a conference paper at ICLR 2026
Table 1: Results of open world region level image recognition on LVIS, PACO and RR-7K.
LVIS
PACO
RR-7K
Model
Params
Semantic Similarity
Semantic IoU
Semantic Similarity
Semantic IoU
Semantic Similarity
Semantic
loU
Shikra 2023
7B
49.7
19.8
43,6
11.4
GPT4Rol 2024)
7B
51.3
12.0
48.0
12.1
Osprey (2024)
65.2
38.2
73.1
52.7
61.0
32.5
Ferret 2023)
13B
65.0
37.8
VP-LLAVA
2024)
8B
86.7
61.5
75.7
50.0
VP-SPHINX
2022)
13B
87.1
62.9
76.8
51.3
54.7
17.5
DAM
2025
8B
89.0
77.7
84.2
73.2
PAM
2025
1.5B
87.4
76.5
85.1
73.5
44.5
12.4
PAM
2025
3B
88.6
78.3
87.4
74.9
45.8
13.4
WOW-Seg (Ours)
1B
89.7
82.4
88.5
79.2
69.1
44.8
annotations for their components, posing higher requirements for the model's part perception ability. RR-7K, the open world region recognition test benchmark proposed in this paper, contains 7,662 categories and aims to test the model's understanding of the open world.
As shown in Tab. [ WOW-Seg achieves the best performance LVIS, PACO, and RR-7K. It worth noting that the LVIS dataset, compared with the previous SOTA DAM, the number of parameters of WOW-Seg is reduced by nearly 9x, but it exceeds DAM by 0.7 in the Semantic Similarity. In terms of the Semantic IoU, WOW-Seg is higher than the previous SOTA PAM, and the number of parameters is 3x lower than that of PAM. Moreover, on the RR-7K dataset, WOW- Seg continues to achieve optimal performance. Simultaneously, we observe that previous methods generally perform poorly. This further demonstrates that RR-7K presents a greater challenge than LVIS and PACO.
4.3 OPEN- VOCABULARY SEGMENTATION
The goal of this task to complete the classification task for the cor- responding masked area under the prompt of an open vocabulary. Typ- ically, previous methods would input vocabulary into the model. How- ever, WOW-Seg a model that does not require any text input. In the in- ference stage, we can use Sentence BERT (Reimers & Gurevych] (019)) to calculate the semantic similarity between the region embeddings out- put by WOW-Seg and the text embed- dings of each category the vocab- ulary, and take the category with the highest similarity as the final classifi- cation result.
Table 2: Recognition performance on open-vocabulary panoptic segmentation (PQ) and semantic segmentation (mloU) upon the validation sets Cityscapes and ADE20K. The ground truth box/mask is used for perfor- mance evaluation.
Cityscapes
ADE20K- 150
Method
PQ
mloU
PQ
mloU
CIIP-ConNeeX-LL.
2021
22.53
23.06
36.86
28.74
CLIP-Surgery-ViT: 2023
27.24
21.92
26.55
21.42
Kosmos-2 2023
12.09
13.71
6.53
5,40
Shikra-7BB 2023
17.80
17.77
27.52
18.24
GPT4Ro1- 7B (2024)
34.70
36.73
36.32
25.82
Ferret-7B 22023
35.57
38.40
39.46
31.77
Osprey-7B 20244)
50.64
49.78
41.89
29.63
Table 2 shows the performance of WOW-Seg in open vocabulary
WOW-Seg (Ours)
65.76
66.40
44.46
37.77
panoptic segmentation and semantic segmentation on the validation sets Cityscapes (Cordts al. 2016) and ADE20K (Zhou al. 2017).. The experimental results clearly demonstrate that WOW. Seg significantly outperforms existing methods in all evaluation metrics.
the Cityscapes benchmark. WOW-Seg achieves a panoramic segmentation quality (PQ) of 65,76 and semantic segmentation mloU of 66.40. These results represent new state-of-the-art outperforming the previous leading method, Osprey-7B, by significant margins of +15.12 PQ and +16.62 mloU, respectively. This superior performance is consistent across other benchmarks, with WOW-Seg also surpassing all competing
Our model demonstrates exceptional performance
8

---

Published as a conference paper at ICLR 2026
Table Compare the classification accuracy of ground truuhh--trthhhh masks (maskAcc) using different meth- ods on multiple datasets.
ADE-847
PASCAL-459
ADE-150
COCO Stuff
Cityscapes
Pretrained
OpenAl CLIP 2021
32.0
44.8
51.0
46.2
46.5
EVA02 CLIP
2023
35.2
44.8
52.7
45.0
44.9
Finetuned
CLIPSelf (2023
49.0
56.1
50.6
52.1
CAT-Seg (2024)
33.5
50.7
62.3
63.0
65.3
MaskCLIP++ 2024)
38.4
56.4
67.0
67.8
71.0
WOW-Seg (Ours)
59.2
58.7
68.5
72.0
73.5
Table 4: Performance of different variants of Cascade Attention Mask and the baseline.
Train Setting
Cascade Attention Mask
LVIS
PACO
Region
Output
Sem. Sim.
Sem. ToU
Sem. Sim.
Sem. IoU
SM
85,22
74.70
79.46
66.04
MM
88.75
80.10
86.29
75.49
MM
V
(9.65(+0090)
88.18
87.89 (+-1600
78,38(+2.89
MM
89.56 89.56(+0.81) +-0.81)
81.94(+1.84)
$7.36(+1.07
(+.42(++.93)
MM
89.70 +0.95)
(2.35
88.52 (+2.2(++222))
79.22 (+3.73)
models on the ADE20K-150 dataset. Notably, these gains are achieved with a 1B parameter model, highlighting the advanced efficiency and effectiveness of our architecture compared larger 7B models.
4.4 MASK REGION CLASSIFICATION
As WOW-Seg is vocabulary-free model, benchmark it against leading vocabulary-based VLM methods to provide a fair and comprehensive evaluation of its mask classification performance. We test using ADE20K, PASCAL (Everingham et 2010), COCO Stuff and Cityscapes. The results, presented in Tab. 3 demonstrate that WOW-Seg significantly outperforms all baselines across every reported metric, for both their pretrained and fine-tuned variants.
4.5 ABLATION STUDY
In order to evaluate the effectiveness of the core design in our work, we conducted large number of ablation experiments.
The Effectiveness of Cascade Attention Mask. In the Tab. 4 SM represents training with single- mask data, and MM represents training with multi-mask data. "Region" represents decoupling be- tween mask tokens. "Output" represents decoupling between output object names. For specific details of the variant, please refer Fig. I0 All experiments ensure the same number of training steps. We can find that the performance of the model trained by MM is significantly higher than that of the model trained by SM. This reasonable because the model trained by MM can learn more data in the same number of training steps. We have verified the performance of different variants of Cascade Attention Mask respectively. It can be seen that the performance all variants is higher than that of the MM model without using Cascade Attention Mask. And the highest performance is achieved when decoupling the features the mask area and the object name at the same time.
The Effectiveness of Mask2Token. In the Tab. Scale denotes the multiple of the mask region im- age's ' side length relative to the mask's 's maximum side length in Mask2Token. Although performance similar when scale is set and 1.5, the number of tokens required to encode single mask is

---

Published as a conference paper at ICLR 2026
Table 5: The impact of Region scale on the performance of Mask2Token.
Table 6: Ablation experiments on different mask transformation methods.
LVIS
PACO
Scale
S. Sim.
loU
Sim.
S. loU
LVIS
PACO
Method
1.5
89.91
82.59
88.18
78.76
Sim.
loU
Sim.
S. loU
89.70
82.35
88.52
79.22
Fore2Token
83.67
72.54
76,46
62.58
2.5
89.34
81.61
88.08
78.32
Blur2Token
82.97
71.28
64.35
3
89.13
81.33
88.34
78.76
Mask2Token
85.22
74.70
79.46
66.04
3.5
89.58
81.89
87.79
77.89
approximately 1.8x greater when scale equals 2 compared to when it equals 1.5. Therefore, we have selected scale 2 for reporting the final model. To demonstrate the effectiveness of Mask2Token, we explored several of its variants during the research. Fore2Token only fills the background area with white as the mask patch. Region2Token applies Gaussian blur to the background area as the mask patch. Each mask in both cases is represented by 256 tokens. The Tab, show that Mask2Token has significantly improved performance. Further details are displayed at D.I
4.6 FORWARD EFFICIENCY ANALYSIS
A potential concern regarding the Mask2Token module is that encoding individual mask re- gions may introduce significant computational overhead. We analyse the training forward run- time and computational complexity of WOW- Seg versus PAM, as illustrated in Fig. 7 PAM employs a instance inference paradigm, requiring N forward passes to process N in- stances, resulting in linear increase in compu- tational cost. In contrast, while Mask2Token necessitates cropping and encoding each re- gion, these operations can be efficiently batch- processed on GPUs. Crucially, LLM decoding executes only once simultaneously for all in- stances. Consequently, increasing the number of instances from to 32 results in less than fourfold increase in WOW-Seg's total compu- tational cost. This confirms that Mask2Token effectively distributes the visual encoding bur- den, rendering WOW-Seg highly efficient.
2
PAM Runtime
WcW- Runtimo (ms)
214.
GFLOPs
213
WOW Seg GFLOPs
Runtime (ms)
211
255
244
213
2a
212
27.
211
instance
instances
4 instances
a Instances
instances
Instances
Number of Instances Forward
Figure 7: Comparing forward pass efficiency and GFLOPs on 3090 GPU.
5 CONCLUSION
We propose WOW-Seg, a Word-free Open World Segmentation Model. WOW-Seg utilizes the Mask2Tokee module to represent object masks as tokens. key advantage of this approach that the generated tokens already lie within the VLM's pretrained embedding space, which circumvents the need for extensive retraining for feature alignment. Due to the introduction of the Cascade Attention Mask, WOW-Seg process multiple masks in single forward pass without being affected by correlations between the masks. To better evaluate the model's ability open-world segmentation tasks, introduce the RR-7K open world test dataset. Ultimately, with only IB parameters, WOW-Seg achieves advanced performance on multiple open world tasks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTTS
This research was supported by the Fund of Shenzhen Science and Technology Program BBSMBB202500009900000000 JCYI202506041840270044 JCYJ20240813) 142370448, the Na- tional Natural Science Foundation of China (Grant No. 62576177), Guangdong Basic and Applied Basic Research Foundation (2026A151501 1435). the Fundamental Research Funds for the Central Universities 070-63253222, and the Tianjin Key Laboratory of Visual Computing and Intelligent Perception (VCIP). Computation is supported by the Supercomputing Center of Nankai Univer- sity (NKSC). "Science and Technology Yongjiang 2035" key technology breakthrough plan project (20252053), Chinese government-gun guudeed local science and technology development fund projects (scientific and technological achievement transfer and transformation projects) (25420102G), Na- tional Science Fund of China under Grant No. 62361166677 This work was also supported by the Academy for Advanced Interdisciplinary Studies, Nankai University (AAIS, NKU).
REFERENCES
Reza Azad, Ehsan Khodapanah Aghdam, Amelie Rauland, Yiwei Jia, Atlas Haddadi Avval, Afshin Bozorgpour, Sanaz Karimijafarbigloo, Joseph Paul Cohen, Ehsan Adeli, and Dorit Merhof. Med- ical image segmentation review: The success of u-net. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Shuai Bai, Keqin Chen. Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmenta- tion. Advances in Neural Information Prucessing Systems, 32, 2019.
Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma. Mert Sabuneu, John Guttag, and Adrian V Dalca. Universeg: Universal medical image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21438 21438-21451, 2023.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209- 1218, 2018
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's 's referential dialogue magic. arXiv preprint arXiv:2306. 15795, 2023.
Peng Chen, Qi Xiao, Jian Xu, Xiaoli Dong, and Linjun Sun. Facial attribute editing using semantic segmentation. In 2019 International Conference on High Performance Big Data and Intelligent Systems (HPBD&IS), pp. 97-103. IEEE, 2019.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024a.
Zhenyuan Chen, Lingfeng Yang, Shuo Chen, Zhaowei Chen, Jiajun Liang, and Xiang Li. Revisiting prompt pretraining of vision- vision-language models. arXiv preprint arXiv:2409.06166. 066.4666066666. 2024b.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2290-129,, 2022.
Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arna,, Paul Hongsuck Seo, and Seungryong
Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4113-4123, 2024.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
11

---

Published as a conference paper at ICLR 2026
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icce, 2009.
Xiaoyi Dong, Jianmin Bao, Yinglin Zheng. Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, al. Maskclip: Masked self-distillation advances con- trastive language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10995-11005, 2023.
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voe) challenge. International Journal of Computer Vision, 88 (2):303-338, 2010.
Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. Advances in neu- ral information processing systems, 35:1 140-1156, 2022.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Yasufumi Kawano and Yoshimitsu Aoki. Tag: Guidance-free open-vocabulary semantic segmenta- tion. arXiv preprint arXiv:2403. 11197, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 4015-4026, 2023.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579-9589 2024.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.033: 2022.
Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv e-prints, pp. arXiv-2304, 2023.
Yuxuan Li, Yicheng Zhang, Wenhao Tang, Yimian Dai, Ming-Ming Cheng, Xiang Li, and Jian Yang. Visual instruction pretraining for domain-specific foundation models. arXiv preprint
arXiv:2509.11 17562, 2025.
Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Yu Liu, Trevor Darrell, Adam Yala, et al. Describe anything: Detailed localized image and video caption- ing. arXiv preprint wrXiv:2504.1077 2025.
Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shang- hang Zhang, and Hongsheng Li. arawwand-understandd Leveraging visual prompts enable milms to comprehend what you want. arXiv preprint arXiv:2403.20271, 2024.
Weifeng Lin, Xinyu Wei, Ruichuan An, Tianhe Ren, Tingwei Chen, Renrui Zhang, Ziyu Guo, Wen- tao Zhang, Lei Zhang, and Hongsheng Li. Perceive anything: Recognize, explain, caption, and segment anything in images and videos. arXiv preprint arXiv.2506.05302, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.07788 0755, 2025.
Yuang Meng, Xin Jin, Lina Lei, Chun-Le Guo, and Chongyi Li. Ultraled: Learning to see everything in ultra-high dynamic range scenes, arXiv preprint arXi::2510.0774/ 2025.
12

---

Published as a conference paper at ICLR 2026
Khan Muhammad, Tanveer Hussain, Hayat Ullah, Javier Del Ser, Mahdi Rezaci, Neeraj Kumar, Mohammad Hijji, Paolo Bellavista, and Victor Hugo C De Albuquerque. Vision-based semantic segmentation in scene understanding for autonomous driving: Recent achievements, challenges, and outlooks. JEEE Transactions on Intelligent Transportation Systems, 21(12):2294-227155 2022.
Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 7748-8765 PmLR, 2021.
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez. Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7141-7151, 2023.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guin guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18082- 18091, 2022.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13009-13018, 2024.
Nikhila Ravi, Valentin Gabeur, Yuan- Ling Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint 4xX1v:2408.00714, 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprini arXiv: /908. 10084, 2019.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401. 14159, 2024a.
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 26374-26383, 2024b.
Pitchaporn Rewatbowornwong, Nattanat Chatthee, Ekapol Chuangsuwanich, and Supasorn Suwa- janakorn. grridggue segmentation using zero segment labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1162-11 1172, 2023.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Akashah Shabbir, Mohammed Zumri, Mohammed Bennamoun, Fahad S Khan, and Salman Khan. Geopixel: Pixel grounding large multimodal model in remote sensing. arXiv preprint arXiv:2501.13925. 2025.
Heeseong Shin, Chaehyun Kim, Sunghwan Hong, Seokju Cho, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Towards open-vocabulary semantic segmentation without semantic labels. Advances in Neural Information Processing Systems, 37:9153-9177, 2024.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprini arXiv:2303. 15389, 2023.
13

---

Published as a conference paper at ICLR 2026
Osman Ülger, Maksymilian Kulicki, Yuki Asano, and Martin R Oswald. Auto- aato-vocabblary semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14266-24227 2025.
Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Jie Hu, Dengjie Li, Zheng Zhao, and Yujiu Yang. Hyperseg: Hybrid segmentation assistant with fine -re-grained visual perceiver. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8931-8941, 2025.
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision transformer distills itself for open- open-vocabulary dense prediction. arXiv preprint arXiv:2310.01400; 2023.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077-12090 2021.
former:
Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng. Shu Liu, and Jiaya Jia. Lisa++: An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.172400 2023.
Yufei Yin, Hao Chen, Wengang Zhou, Jiajun Deng. Haiming Xu, and Houqiang Li. Revisiting open- set panoptic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp- 6747-6754, 2024.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere any granularity. arXiv preprint arXiv:2310.0777
2023.
Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 28202-28211, 2024.
Oliver Zendel, Matthias Schörghuber, Bernhard Rainer, Markus Murschitz, and Csaba Beleznai. Unifying panoptic segmentation for autonomous driving. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 21351-21360, 2022.
Quan-Sheng Zeng, Yunheng Li, Daquan Zhou, Guanbin Li, Qibin Hou, and Ming-Ming Cheng- Maskelip++: A mask-based clip fine-tuning framework for open- vocabulaay image segmentation. 2024.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. In European conference on computer vision, pp. 52-70. Springer, 2024.
Xu Zhang. Danyang Li, Xiaohang Dong, Tianhao Wu. Hualong Yu, Jianye Wang, Qicheng Li, and Xiang Li. Unichange: Unifying change detection with multimodal large language model. arXiv preprint arXiv:25/1.0225 022607 2025.
Yang Zhang, Danyang Li. Yuxuan Li, Xin Zhang, Tianyu Xie, Mingming Cheng, and Xiang Li. Crystal: Spontancous emergence of visual latents in milms. arXiv preprint arXiv:2602.20980, 2026.
Penghai Zhao, Jinyu Tian, Qinghua Xing, Xin Zhang, Zheng Li, Jianjum Qian, Ming-Ming Cheng, and Xiang Li. Naipy2: Debiased pairwise learning for efficient paper quality estimation. arXiv preprint wwxiiv2509.25/99 2025a.
Penghai Zhao, Qinghua Xing, Kairan Dou, Jinyu Tian, Ying Tai, Jian Yang, Ming-Ming Cheng, and Xiang Li. From words worth: Newbom article impact prediction with 1lm. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 1 1183-1191, 2025b.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torrala. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2017.
14

---

Published as a conference paper at ICLR 2026
Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXwv:2504.1049, 2025.
15

---

Published as a conference paper at ICLR 2026
A ETHICAL STATEMENT
The RR-7K dataset contributed this work derived from images in the SA-1B dataset, with annotations generated automatically via large model inference. The authors have conducted further manual screening and desensitization processing on the data. Our contribution intended solely for research purposes, with the aim of advancing the study of open-world world segmentation methods. The authors shall not be held responsible for any consequences arising from the practical application of the models the dataset.
B REPRODUCIBILITY STATEMENT
We will provide comprehensive information to facilitate the control of randomness during training and validation, including but not limited to hardware specifications, software environment configura- tions, and model hyperparameters. Furthermore, we will release the complete source code alongside detailed training logs. All relevant details will be exhaustively documented in the supplementary materials.
C DISCLOSURE OF LLM USAGE
In the research and preparation of this manuscript, large language models (L.LMs) were utilized for code implementation and language polishing. All code generated by an LLM underwent rigorous review, debugging, and comprehensive testing by the authors. The core algorithms and experimental design were conceived and executed directly by the authors. The LLM served solely assistive tool, with its outputs consistently under the authors' supervision and control. The authors assume full responsibility for all content presented in this work.
D
METHOD DETAILS
D.1 FURTHER ANALYSIS OF MASK2TOKEN
Dynamic Num Tokens
& Blu
Vision Encoder
Vision Encoder
Mask2Toker
Avg: 27 (LVIS)
OOOO" 256 Takens
OOMO- Dynamic Tokens
256 Takens
(a) Fore2Token
(b) Blur2Token
(c) Mask2 Token
Figure 8: The process of mapping mask tokens in different methods.
When converting the mask into features that can be understood by VLLM, In Fig. ((a), the foreground and background in the area around the mask are classified according to the mask. After filling the background with white, it is combined with the foreground. The synthe- sized image then fed into the Vision Encoder: For an image, the number of tokens encoded by the Vision Encoder is fixed at 256. This became our baseline. In previous research, Gaussian blur visual cues achieved good results in open vocabulary tasks. Therefore, in Fig. Bb), we applied Gaussian blur to the background. The standard deviation of the Gaussian blur was set to 10. Similarly, the mask processed by Blur2Token is still encoded into 256 tokens. In Fig. (c... we use the mask as a prior to extract visual tokens at specific positions. this way, the number mask tokens is lowed to change dynamically. Taking the LVIS dataset an example, each mask is represented by an average of 27 tokens. Compared with the previous two methods, Mask2Token not only greatly reduces the number of tokens but also eliminates interference information. Fig. shows the loss decline trends these three different methods. can be seen that Mask2Token converges faster and has the lowest loss.
tried various methods.
16

---

Published as a conference paper at ICLR 2026
1.0
Blur2Token (Raw)
Blur2Token (Smooth)
0.8
ForeZToken (Raw)
Fore2Token (Smooth)
Loss
0.6
Mask2Token (Raw)
MaskZToken (Smooth)
0.4
0.2
0.0
2500
5000
7500
10000
12500
15000
17500
Train Steps
Figure 9: Comparison of the loss decline trends of Mask2Token, Fore2Token, and Blur2Tooken. For intuitive display, we truncated the y-axis at 1.0. In the figure, the light colored lines represent the actual loss values, and the dark colored lines the smoothed loss decline curves.
D.2 ROBUSTNESS TO VISUAL PROMPT QUALITY
Whilst our primary experiments use GT masks to test recognition performance, real-world ap- plications typically rely masks or heuristic geometries from generators. assess WOW- Seg's robustness to imperfect prompts, con- duct comprehensive stress testing with diverse mask sources, ranging from high-quality gen- erators to coarse geometric approximations. As shown in Table 7 the results demonstrate remarkable robustness. When using masks generated by SAM2-L, WOW-Seg maintains nearly identical performance to the ground truth baseline. More remarkably, even when pro- vided with extremely coarse inputs such as ro- tated bounding boxes, the model retains robust recognition capabilities, achieving semantic similarity of 87.3 and semantic IoU of 78.6 on LVIS. This confirms WOW-Seg's ability to effectively utilise visual prompts for localisation without being overly sensitive to mask precision.
Table 7: Robustness analysis on different visual prompt sources. RBB denotes the use of a rotating bounding box as prompt mask. Ellipse denotes the use of bounding ellipse prompt mask.
LVIS
PACO
Mask Source
S. Sim. S. loU S. Sim. S. loU
Geometric Heuristics
GT Mask
89.7
82.4
88.5
79.2
RBB
87.3
78.6
85.5
74.1
Ellipse
86.9
78.0
84.8
73.4
Mask Generator
SAM (Huge)
88.8
81.0
86.6
75.7
HQ-SAM (Huge)
88.9
81.1
86.2
75.3
SAM2 (Large)
89.0
81.2
86.8
76.3
D.3 CASCADE ATTENTION MASK
In the Region Attention Mask variant in Fig. 10 (a)) the masks are set to be independent of each other. A normal Causal Attention Mask is used during output. In the Output Attention Mask variant in Fig. TO (b), only the object names are set to be independent of each other. A normal Causal Attention Mask used for the multi-mask features during forward propagation.
The Cascade Attention Mask has a wide range of application scenarios. For example, in the vi- sual grounding task, it ensures that the output coordinates of multiple targets do not interfere with each other. We hope that more researchers will continue to explore. Therefore, we provide the pseudocode of the Cascade Attention Mask, as shown in Algorithm
D.4 THE IMPACT OF VISUAL LANGUAGE MODEL FOUNDATIONS
To investigate the influence of pre-trained foundation models on WOW-Seg, we evaluated WOW- Seg across different iterations of the InternVL series (InternVL2, In2. and IntemVV33), As shown in Table g WOW-Seg demonstrates exceptional scalability and architectural superiority. Specifically, when equipped with the earlier Intern VL2-1B (released July 2024), model achieved a semantic similarity of 88.8 and semantic IoU 80.8 on LVIS. These results still significantly out- perform PAM's 87.4 4 and 76.5, confirming that our framework's effectiveness is intrinsic rather than
17

---

Published as a conference paper at ICLR 2026
image
moskt
mask2
text
obji
coep
obj2
fixed text
<starb
traffic
sign
fire
truck
(a) Region Attention mask (b) Output Attention mask
(c) Cascade attention mask
Figure 10: Cascade Attention Mask and its variants.
Algorithm 1: Cascade Attention Mask Construction
Input: Input tokens config
Output: Additive attention mask M E R
Initialize binary causal mask M + tril(1"X*), where |
Parse S into sets: Imusk (mask tokens), Tou (ordered output chunks);
for each mask segment Imusk do Tmack {8) + 0; end for for each outpur chunk Iows do
# Mask inter-segment attention
Mcc, Ukco +0;
# Mask previous output chunks
not C.output.sees.aask then M [c, Imask] +- O; end for
Set M Mii f- 0 for i separator/padding indices; return - M) (-00):
# Convert additive mask
merely a consequence of more advanced backbone. Furthermore, performance steadily improves as the backbone upgraded to InterVL22, and Intern demonstrating that the WOW-Seg ar- chitecture effectively unlocks the potential of increasingly powerful foundational models.
Table 8: The Impact of Different Periods' Base Models WOW-Seg. Owing the unique structure of PAM, it is incapable of performing multi-instance inference during single forward pass.
LVIS
PACO
Method
Base Model
Iulti-Instance
S. Sim.
S. ToU
S. Sim.
S. ToU
PAM-1.5B
SAM2+Qwen2.5
87.4
76.5
85.1
73.5
In2erVLL2
88.8
80.8
85.4
74.0
WOW-Seg
Intern VL2.5
89.5
82.0
87.8
77.8
VL.3
89.7
82.4
88.5
79.2
18

---

Published as a conference paper at ICLR 2026
E RR-7K DATASET
E COARSE CATEGORY LIST
To more intuitively reflect the richness of RR-7K, we have roughly classified the categories of RR- 7K at a coarse grained level and presented the relevant information of the coarse grained categories, as shown in Tab.
Table 9: A preliminary coarse grained division of RR-7K
Coarse category
Fine-grained category
Animals
lion, bull, tiger, buffalo, elephant, cow, kangaroo, sheep, horse, dog, elk, boxer, antelope, cattle, lioness, orangutan, pony, lemur, boar, hippopotamus, yak, polar bear, chimpanzee, gazelle, Ilama, hedgehog, wildebeest,
Mammals
bird, cagle, duck, rooster, swan, parrot, pigeon, vulture, barnacle goose, toucan, pelican, hawk, peacock, egret, ibis, cardinal, ostrich, crow, woodpecker, gull, lorikeet, seagull, cormorant, dove,
Birds
Reptiles
turtle, snake, iguana, tortoise, serpent, crocodile, gecko, komodo dragon, triceratops
Fish
fish, clownfish, sardine, carp. goldfish, catfish, swordfish, guppy, pufferfish
Insects
butterfly, ladyhug, insect, moth, grasshopper, caterpillar, mosquito
jellyfish, octopus, scallop, slug, snail, sea anemone, clam, worm, brain coral, mussel, sea urchin, coral polyp, horseshoe crab, nudibranch, prawn.
Invertebrates
audience, audience member, child, girl, soldier, boy, firefighter, monk, adult, baby, pedestrian, tennis player, fisherman, priest, wrestler, warrior, bride, referce, gymnast, tourist, groom, spectator, teacher, knight..
Humans
Plants
flower, sunflower, tulip, lotus, anthurium, bougainvillea, orchid, poppy. hibiscus, lily, peony, cherry blossom, iris, daffodil, banana flower, carnation, zucchini flower, daisy, petunia, rhododendron, rose, chrysanthemum...
Flowers
Ornamental Plants
flower hed, plant, garden, lavender field, flower seed, laurel, seedlings, hydrangea,
Climbing Plants
grapevinee wisteria, vine
Medicinal Plants
yucca plant, aloe, cannabis plant
pepper, chili, ginger, turmeric, cinnamon sticks, galangal, dried chili peppers, dried tobacco leaves
Spice Plants
Edible Plants
seed pod, agave, seedpod, strawberry leaf, seed, shallots
Nut Plants
almond, chestnut
Herbal Plants
herbal medicine
palm, pine, cherry blossom tree, cypress, joshua tree, banana tree, coconut tree, mangrove,
Trees
Shrubs
bush, cactus, shrub, shrub hedge, hedge, coleus, twigs,
grass, garlic, succulent, sedum, pitcher plant, elephant ear, dandelion, cilantro, basil, cotton, lavender, moss....
Herbs
Aquatic Plants
water lily, aquatic plants, aquatic vegetation, aquatic plant, water lettuce
Bamboo
bamboo, bamboo stick, bamboo lcaf, bamboo pole
Fruit Plants
olive, litchi, coffee cherries, prickly pear, lotus pod
onion, cucumber, strawberry, bottle gourd, cranberry, luffa, arugula, scallion, collard greens, potato
Vegetable Plants
Agricultural Plants
(comn, wheat, buckwheat, rice bundle, ...
19

---

Published as a conference paper at ICLR 2026
Coarse category
Fine-grained category
Man-made Objects
Furniture & Home Decor basket, clock, door, chair, pillow, box, carpet, shelf, vase, bench, curtain, ...
frieze, cube, text, crayon, stickers, canvas, square panel, mercedes henz logo, header, motif, plaid pattern....
Graphics & Design
Tools & Equipment
fan, gauge, handle, padlock, net, nail, control panel, chain, machine, knob, motor,
Vehicles & Transportation
car, truck, wheel, bus, bicycle, train, van, ferry, cruise ship, cable car,
Electronics
light, air conditioner, lamp, camera, phone, robot, wire, display screen, horn,
building, chimney, dome, window, floor, spire, pillar, roof, skyscraper, monument, church, window pane, temple, staircase, shed, ...
Buildings
banner, button, ad board. poster, label, balloon, billboard, advertisement board, menu, neon sign, card, display, mosaic, ...
Advertising & Marketing
face mask, lotion, napkin, cosmetic product, knee pad, face shield, sponge, dispenser. first aid kit, surgical drape, soap, stethoscope, surgical gown.
Health & Hygiene
sign, flag. plaque, traffic light, number, traffic sign, tactile paving, inscription, symbol, barcode, banner sign, icon,
Signs & Symbols
shoe, umbrella, helmet, pants, shirt, chorts,jacket jeans, glove, trousers, bag, scarf, cap. handbag,...
Personal Items
Food & Beverages
Fruits
apple, fruit, pincapple, grape, strawberry, watermelon, banana, cherry, kiwi, orange, ...
nuts, seeds, hazelnuts, kemel, walnuts, coffee beans, pumpkin seeds, lotus seeds, sunflower seeds, pistachios... ...
Nuts & Seeds
chili sauce, condiment bottle, soy sauce, chili sauce hottle, ketchup, mayonnaise, condiment, sauce packet, topping, salsa, thai chili paste.
Condiments
bread, pancake, biscuit, gingerhread house, bun, crepe, gingerbread, panettone, cinnamon roll, quiche,.. ...
Baked Goods
food, tea bag, soup, tempura, doner kebab, dumpling, burrito, stew, food can, taco, natto,
Ready-to-Eat Foods
onions, vegetables, dried tomato, dried mushrooms, bok choy, cherry tomatoes, carrots, okra, spinach, sweet corn, cabbage, squash, legumes, dried vegetables, radicchio,
Vegetables
coconut milk, wine, whiskey, milkshake, cola, beverages, root beer, bubble tea, coca cola,
Drinks
candy, cookie, snack packet, cracker bread, potato chip, chewing gum, gummy bear, pretzels, snack chip, fritter, tortilla chip.
Snacks
tofu, bagel, potato bread, spaghetti, rice bag. patty, flatbread, matzo, muesli, corn kernel, lentils, mashed potatoes,
Staple Foods
cake, ice cream, fried dough, strawberry jam, choco pie, marshmallow, tea cake, waffle cone, heart shaped cookie. sweet, tiramisu.
Desserts
sausage, meat, ham, skewered meat, dried meat, burger, salami, meatball, ribs, raw meat, pork knuckle, pork belly, shredded meat, pepperoni....
Meat
dried seafood, seaweed snack, dried fish, oyster, crab leg, dried seaweed, sashimi, tuna can, unagi, mussels, salmon fillet...
Scafood
Dairy Products
cheese, dairy product, dairy, coffee creamer, milo product, milk powder, yoghurt
20

---

Published as a conference paper at ICLR 2026
Coarse category
Fine-grained category
Arts & Entertainment
mural, coat of arms, fresco, graffiti, landscape painting, religious artwork, portrait, frame painting, altarpiece, religious mural, islamic calligraphy, framed panel, ...
Paintings
playground, christmas market stall, theater, cinema, casino, amusement ride, bullring, backdrop, ball-pit, arcade, ...
Entertainment Venues
ornamental carving. figurine, tarot card, snow sculpture, dreameatcher, religious ornament, crochet, matryoshka doll, maneki neko, chinese knot, batik, souvenir.
Handicrafts
tinsel, maypole, streamers, festival float, papel picado, snowflake omaament chinese new year decoration, offering, festival costume, confetti, christmas
Festivals & Celehrations
statue, carving, sculpture, statue base, mermaid statue, cherub, dog statue, crosier, sphinx statue, buddha statues, clay sculpture, bird statue, relief sculpture,
Sculptures
Movies
film, anime characters, clapper, superhero
trombone, harp, drum, bass guitar, double bass, bagpipes, banjo, wind instrument, lyre, marimba, musical note, brass instrument, ...
Music
book, stamp, word roll, writing, stacked books, book page, mickey mouse, atlas, minnie mouse, book edge.... ...
Books
dancer, marching band member, puppet, figures, juggling club, ballerina, tango, artist, stage.
Performing Arts
Video Games
dragon puppet, character figure, video game cover, game, spongehob squarepants, ...
ellipse, mandala, abstract art, spiral artwork, polygon, figurines, abstract painting, cutout figure, arabesque, crystal ball, sketch, palette, ...
Visual Arts
hieroglyphics, ushnisha, deity, prayer card, ganesha, cartouche, aztec calendar, religious figure, virgin mary, prayer tablet
Culture & History
Environment
Geography & Landmarks hill, path, mountain, ocean, obelisk, stadium, gravestone, bridge,
stupa,
Materials
wood, fabric, cloth, plastic, concrete block, metal, wooden post, wooden surface,
Weather & Climate
sky, snow, smoke, sun, snowbank cloud, crescent, sunburst, shade, cloud,
Reflections & Optical Phenomena
reflection, shadow, water spray, lightning bolt, smoke trail, flash, bubble, laser beam, skyline reflection, water surface, light trails...
Natural Landscapes
water, star, sand, pool. rock formation, ground, ocean water, stalagmite.
E.2 THE SEMANTIC SPACE OF RR-7K
To demonstrate that the 7,662 categories in RR-7K represent a broad semantic space rather than localized lexical inflation, we conduct semantic space comparison between RR-7K and ImageNet- 21K (Deng et al. 2009), as shown in Fig. IT
We use BERT (Reimers & Gurevych 2019 to extract text embeddings for all category names in RR-7K and ImageNet-21K,, then reduce the dimensionality to two-dimensional plane via the t- SNE algorithm for visualisation.
As can be seen from Fig. II RR-7K exhibits semantic breadth comparable to ImageNet-211 21K along- side similar distributional topology. This demonstrates that the 7,000 categories within RR-7K do not constitute mere repetition of a small number of concepts, but rather construct diverse semantic space.
E.3 HIERARCHICAL DISTRIBUTION OF RR-7K
We further present bar charts illustrating the sample distributions for the primary and secondary categories, shown in Fig. 12
21

---

Published as a conference paper at ICLR 2026
100
100
50
50
-50
-50
-100
-100
-100
-50
100
-100
-50
0
50
100
ImageNet-21K
RR-7K
Figure 11: Visualisation of category distributions ImageNet-21K and RR-7K using t-SNE.
Overview of category distribution
Man made Objects
Environment
Animals
Mants
Beverages
Entertainment
10
Number samples
Man made Objects
Environment
Personal tems
Buidings
Furniture
Geography Landmarks
Whiches Yransportation
Electronics
Natural Landscapes
Equioment
Signs Symbols
Weather Giimtee
Mariceting
Health & Hyglene
Graphics Design
Optical Phenomenna
10
101
103
10a
10'
107
10a
165
samples
Hants
Humans
Flowers
Nammals
Shrub5
Ommmmnt Pants
Birds
Vegelable Pants
Spice
Pands
Fruit Pants
Rih
Clibbigg Pants
Insocts
Nut Pantt
Herbal Pants
167
107
107
T0
10'
10
10'
10'
10
Number of samples
Number of samples
Food & Deverages
Entertaimeet
Fruits
Scurptures
Snarks
Paiminng
Vegeraties
Books
Dessert
Performing
Readysto-Eat
Staple
Video Games
Visual Arts
Dairy
Entertainmnnt Venues
Seeds
Festivals Celebrantens
Condimants
Number
Number
Figure 12: Hierarchical distribution visualisation. The top panel displays the distribution of primary categories. The remaining six sub-paned each show the distribution of secondary categories.
Statistical analysis indicates that the dataset exhibits extensive coverage across all major semantic domains while maintaining reasonable internal diversity. This demonstrates that RR-7K possesses exceptional richness in both breadth and depth.
22

---

Published as a conference paper at ICLR 2026
E.4 VISUALIZATION OF RR-7K
In Fig. 13 we present the visualization of part of the RR-7K data.
DOEEOOOIOOO SEOCERRS
Figure 13: Visualization of some rare category objects in RR-7K.
23

---

Published as a conference paper at ICLR 2026
F MORE PERFORMANCE OF WOW-SEG
In Fig. we visualise the regional recognition performance of WOW-Seg on the ADE20K dataset. It is noteworthy that WOW-Seg -Ow-Seg is not trained on the ADE20K dataset.
Frod
JACQUES
Plat
Foal-
CD vellec
Fead
Teat
Rorl pm
CS
LiNE
Figure 14: Visualisation results of WOW-Seg on the ADE-20K dataset.
24

---

Published as a conference paper at ICLR 2026
visualized the performance of WOW-Seg- 1B and the previous SOTA method PAM-3B. It can seen that WOW-Seg has good recognition performance on small targets, super large targets, and rare targets.
In Fig. 13 and Fig. 16
90
Predict: glove
Sen. Sin. 9,808
PAR
Prodict: drawer
Predict: helnet face
Mok-Seg
Sec. Ste. 6.591
Predict: resote control
PAN-98
Sen. Sin. J62
Predict: table:leg
ST: glave
PAM-30
Predict: collutar telephone:back
07c cabinet
cover
8.133
GT:
contooll
MeDEnervare
Wak-Seg
Predite:: toddy bear
Soor SSA. 9.928
PAN-33 Predieed dogchbed Suer. ETC Cay
Prodict: crossbur
Sen, SLN, 1.000
RAM-30
Prodict: pipeinozzle step
Sen, sin, 8.052
ET: crossban
PRODLCTT
Predict:
George
Figure 15: Visualization of the performance of WOW-Seg- 1B and PAM-3B.
25

---

Published as a conference paper at ICLR 2026
MON-Seg
Predict: sunglasses
8.396
PAM-3B
Seg
Predict: helmet:face shetld
Predict: bread
6.628
Sem. Sim. 8.676
GT: goggles
PAM-38
Predict: glass
Sen. Sim. 8.225
STOP
WOM-50B
Predict:
truch
SiR.
NOM-Seg
PAR-9BE
MOM-SSag
Predict: trousers
Predict: conpoct car
Prodict:
Sen. Sin. 8.777
Sem. Sto. 8.797
GT: EPRCK
PAU--38
DicycLe:saddle
Prodict: Body
Sen. Sin 0.291
Seo Sto. 8.379
short
MON-Ség Ség
Predict: wedding cake
Sen. Sin.
PAM--BB diaper
See. Six. 382
WOW NoN-seg
weddång cake
Predict: sursboard
Sew. Sin. 1.888
PAN-38
Predict: shocsoutsole
Son. Sin. 8.199
GT: surfboard
Figure 16: Visualization of the performance of WOW-Seg- IB and PAM-3B.
26


Reviews:
[{"review_id": "amxNftcyc8", "reviewer": "Reviewer_AFoP", "summary": "This article focuses on the problem of open world image segmentation. It introduces a word-free open world segmentation model for segmenting and recognizing objects from open set categories. The core idea is to encode multiple masks as visual prompts and autoregressively identifies each mask. It develops Mask2Token for mask encoding and  Cascade Attention Mask for detection. Further, the article introduces a new dataset, RR-7K, which features with 7000+ categories.", "questions": "- Why authors disregard the inherent semantic relationships among objects when designing the cascade attention mask?\n\n- Could you provide details on the label distribution across the categories in the RR-7K dataset?\n\n- While InternVL-78B is employed to filter hallucinations during data annotation, how can you ensure that the model itself does not introduce hallucinations?\n\n- Osprey exhibits poor performance on established benchmarks like LVIS and PACO, yet achieves relatively good results on the proposed RR-7K dataset. What explains this discrepancy? A thorough investigation into this is essential to validate RR-7K as a fair, representative, and necessary benchmark.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The work contributes a new dataset, RR-7K, which holds notable potential to benefit the research community. \n- The proposed method achieves state-of-the-art performance across multiple benchmarks", "weaknesses": "- The motivation behind the cascade attention mechanism is not sufficiently justified or explained. While the authors acknowledge that “in the natural world, there are inherent correlations between objects,” they ignore this prior knowledge in their design — which seems counterintuitive.\n\n- Furthermore, the proposed cascade attention scheme lacks a formal mathematical description. The current exposition relies heavily on intuition.\n\n- For the dataset, I am unsure whether we indeed need to recognize 7K categories. As observed in Table 7, many categories appear semantically overlapping or synonymous (e.g., \"audience\" vs. \"audience member\", \"ad board\" vs. \"advertisement board\"). This suggests potential redundancy that could inflate category counts without adding meaningful discriminative power. Moreover, the paper provides no statistical analysis of the dataset, such as, label distribution.\n\n- The current visualizations focus primarily on specific objects, however, it is expected to show the overall segmentation results of entire images. \n\n- As the leading information that authors intend to share, the caption of Fig. 1 is minimal and lacks sufficient context. TBH, it is not easy to grasp all details at the first glance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z9xkmg3G7i", "reviewer": "Reviewer_7ZmL", "summary": "This paper introduces WOW-Seg, a Word-free Open World Segmentation model designed for robust segmentation and semantic understanding across open-set object categories. It incorporates Mask2Token, which converts image masks into visual tokens aligned with the VLLM feature space, and Cascade Attention Mask, which reduces inter-instance interference. The authors also propose RR-7K, a large-scale region recognition benchmark with 7,662 categories. WOW-Seg achieves state-of-the-art performance on LVIS while using only one-eighth of the parameters, demonstrating strong open-world generalization and efficiency.", "questions": "- PAM was trained on a large-scale dataset, yet it performs poorly on the RR-7K dataset. In contrast, WOW-Seg, which was trained only on a combination of LVIS, PACO, and COCO-Stuff datasets, achieves significantly higher performance on RR-7K. The authors are encouraged to provide an analysis explaining this discrepancy.\n- In the open-vocabulary segmentation results presented in Table 2, it would be beneficial to include comparisons with more recent state-of-the-art methods, such as CAT-Seg, to provide a stronger benchmark evaluation.\n- The proposed Cascade Attention Mask is designed to ensure that object masks are independent of each other. However, the results indicate that the multiple-masks-per-sample (MM) strategy outperforms the single-mask-per-sample (SM) strategy. The authors should clarify why MM remains effective if the masks are intended to be independent.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality:** The paper introduces two novel and effective components. The first is Mask2Token, a visual prompt module that transforms masks into tokens, which cleverly embeds mask features directly into the VLLM's existing feature space. The second is the Cascade Attention Mask , a custom attention mechanism designed to solve inter-instance interference. This allows for efficient multi-mask processing by explicitly decoupling the features of different masks and their corresponding text outputs during autoregressive generation.\n\n**Significance:** WOW-Seg achieves state-of-the-art results, surpassing prior models on LVIS (e.g., +4.1 Semantic IoU) and PACO , while being substantially more parameter-efficient (1B parameters vs. 3B-8B for SOTA competitors). Furthermore, the introduction of the RR-7K dataset is a major contribution, providing a much-needed benchmark to evaluate true open-world generalization, on which prior SOTA models perform poorly.\n\n**Clarity:** The paper is well-written and easy to follow.", "weaknesses": "- Dependency on Mask Generator Quality: The model's \"word-free\" recognition is fundamentally dependent on the quality of the masks provided by an external \"Mask Generator\" (like SAM) during inference.\n- The Mask2Token module, while effective, appears computationally intensive. It requires cropping each mask region and feeding it through the visual encoder to extract tokens. The computational cost and inference speed are not analyzed.\n- Several closely related works [1,2,3,4] are missing from the discussion. The authors should provide a more in-depth analysis of these studies and incorporate them into the comparative evaluation to ensure a comprehensive and fair comparison.\n- The paper lacks a quality analysis of the newly proposed RR-7K benchmark.\n\n[1] Kawano, Yasufumi, and Yoshimitsu Aoki. \"Tag: Guidance-free open-vocabulary semantic segmentation.\" arXiv preprint arXiv:2403.11197 (2024).\n\n[2] Ülger, Osman, Maksymilian Kulicki, Yuki Asano, and Martin R. Oswald. \"Auto-vocabulary semantic segmentation.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 24266-24275. 2025.\n\n[3] Rewatbowornwong, Pitchaporn, Nattanat Chatthee, Ekapol Chuangsuwanich, and Supasorn Suwajanakorn. \"Zero-guidance segmentation using zero segment labels.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1162-1172. 2023.\n\n[4] Shin, Heeseong, Chaehyun Kim, Sunghwan Hong, Seokju Cho, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. \"Towards open-vocabulary semantic segmentation without semantic labels.\" Advances in Neural Information Processing Systems 37 (2024): 9153-9177.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HoR8dNb8Pb", "reviewer": "Reviewer_GZHt", "summary": "This paper introduces a novel approach to open-world segmentation without predefined label space. The authors present RR-7K, a new benchmark with significantly more evaluable categories than existing ones. Their proposed two-stage segmentation pipeline first uses a universal mask generator to produce instance masks and then employs a multi-modal large language model for mask region classification. This pipeline achieves superior results on both official benchmarks and the new RR-7K dataset.", "questions": "1. A deeper analysis of the proposed benchmark is needed.\n\n2. Will the dataset be released?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Benchmark Contribution: The proposed RR-7K benchmark represents a critical step for open-vocabulary segmentation, featuring a substantially larger category set than previous datasets and thus offering greater real-world relevance.\n\n2. Performance: The novel method achieves strong, state-of-the-art results on both the new RR-7K benchmark and other established datasets.\n\n3. Presentation: The paper is clearly presented with effective and high-quality writing.", "weaknesses": "1. **Insufficient Dataset Analysis and Justification**\nThe paper introduces a new benchmark, RR-7K, but provides insufficient deep analysis and justification for its design choices. While basic class distributions are provided, the authors fail to rigorously validate the dataset's claims, specifically:\n\n    **Real-World Relevance**: The paper lacks compelling evidence that the adopted label system and class distribution are a closer approximation to real-world data compared to existing large-scale benchmarks (e.g., ImageNet-22K).\n\n    **Source of Distribution**: It is unclear whether the resulting class distribution is an intended design choice or merely a bias introduced by the specific mask generation and classification models used in the dataset curation pipeline.\n\n   A more detailed, comparative analysis of RR-7K's categorical structure (including granularity and scale) against established benchmarks is needed to solidify its importance and justify its adoption.\n\n2. **Limited Novelty of the Segmentation Pipeline**\nThe proposed two-stage segmentation pipeline, while effective, appears to have limited algorithmic novelty. The core architecture—using a universal mask generator followed by a classification stage—closely resembles several existing methods in the open-vocabulary segmentation literature (e.g., [1], [2], [3]).  While the integration of a Multi-modal Large Language Model (MLLM) for classification is a modern substitution, the paper does not convincingly demonstrate that this change is a sufficiently novel contribution to anchor the method as the primary technical advance. The main contribution appears to reside more in the RR-7K dataset and its extensive evaluation, which is conflict with the layout of current submission.\n\n[1] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015–4026, 2023. (Mentioned in the text prompt part)\n\n\n[2] Ding, Z., Wang, J. and Tu, Z., 2022. Open-vocabulary universal image segmentation with maskclip. In Proceedings of the 40th International Conference on Machine Learning.\n\n[3] Xu M, Zhang Z, Wei F, Lin Y, Cao Y, Hu H, Bai X. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision 2022 Oct 22 (pp. 736-753).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2zfGh88VCs", "reviewer": "Reviewer_RZJ5", "summary": "This paper introduces WOW-Seg, a novel word-free, open-world segmentation model that leverages a Vision-Language Model (VLM) to recognize objects from visual prompts alone. The core contributions include the Mask2Token module, which converts image masks into visual tokens aligned with the VLM's feature space , and the Cascade Attention Mask, designed to mitigate interference between multiple object instances during processing. The authors also present RR-7K, a new large-scale region recognition dataset with 7,662 categories, to better evaluate open-world models.", "questions": "Please check Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "**SOTA Performance with Exceptional Efficiency.** WOW-Seg achieves outstanding results, setting a new state-of-the-art on challenging benchmarks like LVIS and PACO by a significant margin. This is particularly impressive given its remarkable efficiency; the 1B parameter model consistently outperforms much larger competitors, highlighting the architecture's effectiveness.\n\n\n**New Benchmark for the Community.** The authors contribute the RR-7K dataset, a large-scale and challenging new benchmark for open-world evaluation. With over 7,600 categories, it addresses a critical need for more comprehensive testing and provides a valuable resource to drive future research in model generalization.", "weaknesses": "### **Major**\n\n**Unfair Comparison of Models**\n\nThe paper emphasizes that the 1B-parameter WOW-Seg outperforms larger models like PAM (3B) and DAM (8B). This comparison is potentially misleading as WOW-Seg is built upon InternVL3-1B, a highly advanced (and fictionally dated 2025) backbone. The baseline models may use older, less capable backbones, where a larger parameter count does not guarantee better performance. To ensure a fair comparison, the authors should provide context on the architectures and pre-training of the baseline models or conduct experiments with same base MLLM.\n\n**Limited Motivation**\nThe motivation for the Cascade Attention Mask is underdeveloped and lacks a clear, compelling justification. The paper posits that objects should be treated as independent during segmentation to avoid being \"affected by such correlations\" that exist in the natural world (e.g., a cup and water).\n\nWhile this principle is stated, the paper fails to sufficiently explain why this correlation is detrimental to the model's performance. The authors mention preventing \"inter-instance interference\" and \"detrimental information leakage\", but these concepts are presented without concrete examples or analysis. \n\n\n**Missing Citations**\n\nThe paper relies on several models and datasets without providing formal citations , including Qwen2.5-VL, InternVL-3.0, LVIS, PACO, Sentence BERT, Cityscapes, ADE20K, PASCAL COCO Stuff, etc.\n\n\n### **Minor**\n\nInconsistent Naming: The model is referred to as \"WOW-SEG\" , \"WOW-Seg\" , and \"wow-seg\". Component names also vary, such as \"Mask Generater\" in a figure and \"Mask Generator\" in the text. Consistency should be maintained.\n\nVague Definitions: The term \"mask generators\" is used without specifying which one (e.g., SAM, SAM 2) was employed for the experiments or how it was configured.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)