Benchmarking Open-ended Segmentation
Authors: Cristina González
Santiago Rodriguez
Kevis-kokitsi Maninis
Jordi Pont-Tuset
Pablo Arbelaez
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmarking
Open-ended Segmentation
Evaluation Protocol
Lexical Alignment
Primary area: datasets and benchmarks

Abstract: Open-ended segmentation requires models capable of generating free-form descriptions of previously unseen concepts and regions. Despite advancements in model development, current evaluation protocols for open-ended segmentation tasks fail to capture the true semantic accuracy of the generated descriptions. We empirically demonstrate that embedding‐based similarity score mappings diverge significantly from human judgments. To address this issue, we introduce a novel mapping function that considers multiple lexical relationships between free‐form outputs and test‐vocabulary labels, yielding much closer alignment with human annotations. We integrate this mapping into a robust evaluation framework and re‐benchmark previous state‐of‐the‐art methods. Additionally, we present the first Multi-modal Large‐Language Model trained with a contrastive objective to jointly align visual regions and textual descriptions, achieving new state‐of‐the‐art results in open‐ended panoptic segmentation.

Published as a conference paper ICLR 2026
BENCHMARKING OPEN-ENDED SEGMENTATION
Cristina González 12.3, Santiago Rodríguez', Kevis-Kokitsi Maninis", Jordi cont-Tuset', Pablo Arbeláez'
Universidad de los Andes HHelmholtz Munich "Technical University of Munich "Google DeepMind {ci. gonzalezzo, s rodr i quezr2] | andes edu. Co, (kmnninis, jponttuset) Strettuuuustttooooogooooo @google com, pa. arbe wewallleeeninndden edwaeeesedddeee... co
ABSTRACT
Open-ended segmentation requires models capable of generating free-form descriptions of previously unseen concepts and regions. Despite advancements in model development, cur- rent evaluation protocols for open-ended segmentation tasks fail to capture the true semantic accuracy of the generated descriptions. We empirically demonstrate that embedding- based similarity score mappings diverge significantly from human judgments. To address this issue, we introduce a novel mapping function that considers multiple lexical relationships between free-form outputs and test-vocabulary labels, yielding much closer alignment with human annotations. We integrate this mapping into a robust evaluation framework and re- be-benchmark previous state-of-the-ar methods. Additionally, Multi-modal Large-Language Model trained with a contrastive objective to jointly align visual regions and textual descriptions, achieving new state-of-the-arr results in open-ended panoptic segmentation.
present the first
INTRODUCTION
Recent advances in visual recognition have enabled the successful deployment of artificial intelligence relatively controlled scenarios, under the requirement of high-quality and application-spectiee training
data. However, the real world is dynamic, diverse, and unpredictable. To operate in such unconstrained
environments, research has shifted toward visual recognition in open-ended settings (Radford et al. 2021
Jia al. 2021: Zhang et al. 2202 Chen al. 2023 You al. 2023 Yuan et al. 2024). Unlike traditional visual recognition tasks (Deng et al. 2009 Lin et al. 2014 Kirillov et al. 2019). which rely on predefined and closed sets of semantic categories, open-endee settings require models describe the semantics of visual
entities by generating free-form language instead of choosing one label within finite set of options.
Experimental frameworks for visual recognition incrementally introduce challenges along two axes: finer spatial localization and semantic generalization. In the former, the benchmarks have progressed from coarse labels at the image-level (Deng et al. 2009) to fine-grained, window- (Lin al. 2014) or pixel-level (Lin et al. categories. In terms semantic generalization, early efforts (Deng et al. 2009 He et al. 2017 Cheng et al. 2022 operated in a strictly closed-set regime, training specialist models to recognize the concepts seen during
2014, Zhou et al., 2017) Cordts et al.) 2016 (uupta et 2019) annotations for a predefined set of
2022; Ghiasi et al. 2022 Xu et al. 2022) allow generalizing models to work unseen categories by defining the vocabulary to operate
training for a specific dataset only. open-vocabulary frameworks (Li et al.
input to the model. Despite these advances, these benchmarks are still restricted to classifiers within a limited vocabulary, and evaluation is simplified to measuring accuracy against "correct" answer within it.
Open-ended approaches (Radford et al.) 2021 Jia et al.) 2021 Zhang et al. 2023 Chen et al.) 2023 You
2023
Yuan et al. 2024) enable the segmentation of novel objects test time without assigning them

---

Published as a conference paper ICLR 2026
Description
Senterce- BURT Mapping
Human Selection
Scatenac HERT
"Peppe are shanding"
buiddng
person
sunstad
"Bhrk lights
1_1
0D
Thee anower le yeu"
CD
sky
bockground
answer specturiee"
TV
person
30
1.1
sign"
+ight
Figure 1: Sentence-BERT mapping quality. Examples of Sentence- BERT (Reimers & Gurevych 2019) mappings ADE20K (Zhou et (017) and Cityscapes (Cordis al. (2016) list of categories alongside the -verffee true category (left). conducted a systematic user study to evaluate the quality of Sentence- BERT's mappings on the standard Cityscapes (Cordts et al. 2016) dataset that reveals an overall error with respect to human judgements of 37.7% (right).
a predefined label, but instead generating free-form descriptions. These can differ in wording, granularity, or structure while still conveying the same concept. Multiple valid descriptions (e.g. "yellow dog." "golden retriever," or "a dog's tail") can refer the same entity, and thus rigid label-matching metries fall short in capturing semantic correctness. This one-to-many relationship between ground-truth concepts and acceptable linguistic expressions motivates the need for evaluation methods that go beyond exact string matching.
Prior studies (Guo et al. 2024 Huang et al., 20244 You et al. 2023) Yuan et al. 2024) have repurposed metrics from related vision-language tasks to assess generalization in open-ended visual recognition, where free- form text outputs are typically scored with captioning metrics (Anderson et al. 2016 Banerjee & Lavie 2005 Lin 2004, Papineni et al. 2002 Vedantam et al. 2015). However, these metrics rely on region-description datasets that focus "things" categories with window-level annotations drawn from semantically narrow sources, which limits their ability to measure true generalization towards unseen categories during training. Some works have turned to human user studies to rate output correctness, but these are labor intensive, expensive, and difficult to standardize.
Recently, researchers have established standard evaluation framework for open-vocabulary segmentation, aiming to train on one segmentation dataset while testing on another, which includes entirely novel categories assess semantic generalization directly. To evaluate open-ended segmentation approaches in this setup, prior work (Zhang et al. 2020) applied embedding- -baed similarity scores like Sentence-BEiN BERT (Reimers & Gurevych| [2019) map free-form textual model outputs the target taxonomy of categories in the test dataset. The metric maps a short description to the closest category in the pre-defined vocabulary in embedding space and then evaluates the performance of an open-ended model as if it had produced that category output. Figure (middle) shows examples of Sentence-BBe BERT mappings to target vocabulary alongside the human- verifiee true category. Notably, these mappings are often incorrect due to biases and limitations from the underlying embedding models, which directly affect the results of the evaluation of open-ended models in this setup. We conducted a systematic human verification to evaluate the quality of Sentence- BERT's mappings the standard Cityscapes dataset that reveals an overall divergence of 37.7% from human judgments (Figure (right)). These findings show that existing evaluation protocols fall short in capturing true semantic accuracy and open-ended generalization.
To address this limitation, this paper introduces a novel mapping function for the evaluation of open-ended segmentation that considers multiple lexical relationships among concepts, rather than relying on single
2

---

Published as conference paper ICLR 2026
similarity score. Our proposed mapping function considered lists of valid words for each category in the test dataset, ordered by increasingly flexible lexical relationships (see Figure 2ap. This approach better assigns free-form descriptions to the test vocabulary, resulting in significantly higher alignment with human annotations. We then integrate this lexical mapping into a comprehensive evaluation framework to re-evaluate several state-of-the-art
apen-vocabulary and Multi-modal Large Language Models (MLLMs), revealing systematic weaknesses in previous evaluation frameworks. Finally, we advance the field by introducing, to the best of our knowledge, the first MLLM trained with contrastive learning, which jointly aligns visual and textual representations for open-ended segmentation.
Our main contributions can be summarized as follows:
(1) We empirically demonstrate that current mubbedding-baset similarity score metrics in existing open-ended segmentation benchmarks significantly diverge from human judgments, misestimating models' semantic accuracy.
(2) We establish a robust mapping from free-form outputs to the test vocabulary and an evaluation protocol for open-ended segmentation that considers multiple lexical relationships and aligns more closely with human judgment. We re-benchmark previous state-of-the-arr methods to promote progress in open-ended segmentation.
(3) We introduce OPAL, the first MLLM trained with a contrastive learning objective to align visual re- gions and textual descriptions. This model achieves state-oftthe-an results on open-ended panoptic segmentation.
To ensure the reproducibility of our results and promote further research in open-ended visual segmentation, we make all resources of this paper publicly available | including the implementation of the evaluation protocol with a re-benchmarking suite, as well as the source code for our method and the pretrained model.
Besoription
Fediral Allgeee
Thensina dag rasingunue ze
Yeard
pacc
The muattre with thegraand
Neve
The curtet tallict of
vii/im
erange
Valid Vocabulary
(b) Predicted Mapping
Figure 2: Lexical mapping. (a) Example of the valid vocabulary for the target category "dog". (b) We illustrate how different descriptions are mapped by our proposed lexical mapping function to the "dog" semantic category at various lexical levels.
2 RELATED WORK
2.1 TOWARDS OPEN-ENDED SEGMENTATION
The field of visual recognition has progressed significantly over the years, starting with image recognition (Deng et al.) 2009), where the task was to assign single label to an entire image. The problem evolved into object detection (Lin al. 2014), which added spatial localization by identifying bounding boxes
3

---

Published as a conference paper at ICLR 2026
around objects of interest. Advances then led segmentation tasks such as semantic segmentation (Lin et al. 2014; Zhou et al. 2017 Cordts et al. 2016), which classified pixels into predefined categories, and instance segmentation (Lin et al. 2014 Gupta et al. 2019), which additionally differentiated between individual objects. A more recent development, panoptic segmentation, unifies both approaches to provide complete understanding of scene by distinguishing objects ("things") and amorphous regions ("stuff") (Forsyth eet al. 1996 Adelson, [20T]] |Heitz & Koller [2008] Kirillov et al., (019). However, these advances have operated mainly within a closed-set paradigm. relying on predefined category sets during training and evaluation. While these methods have demonstrated outstanding performance fixed category sets, real- world visual understanding demands capabilities beyond recognizing a limited set of predefined classes. Models trained under the closed- world paradigm tend to be dataset-specific and overly specialized, limiting their generalization ability to novel scenarios.
Open-vocabulary recognition (Li et al. 2022 Ghiasi al. 2022 Xu al. 2022 emerged as an important step toward more flexible visual understanding. The goal of this task is to extend classification beyond the restricted set of base classes by leveraging pretrained language models (Radford et al. 202 Jia et al. 2021) to generalize across broader vocabulary and recognize categories not explicitly seen during training inference time. This task relies on fixed semantic labels during the evaluation, but allows the model to predict unseen categories by aligning visual features with language embeddings. Although open-vocabulary recognition expands the scope of categories a model can handle, it remains constrained by its dependence on predefined labels for evaluation. In contrast, open-ended segmentation (Radford et al. [021] Jia et al.)
2021 Zhang et al. [0233] Chen et al. 2023 You et al. 20233 Yuan et al., 2024, Reichard et al. 2025) requires models to generate natural language descriptions of visual regions without predefined labels, introducing a generative component to the task. This distinction makes evaluation significantly more challenging, as
generated descriptions can vary widely in form while still being semantically accurate.
2.2 EVALUATION METRICS
In open-vocabulary settings, evaluation commonly relies on zero- or few-shot classification protocols that constrain model predictions to a fixed taxonomy. Evaluation strategies are often adapted from vision-language tasks to capture the semantic flcxibility of model outputs. As result, current evaluation approaches for open- vocabulary segmentation typically fall into three broad categories: captioning- based scores, cmbedding-basse similarity measures, and human studies. While each provides valuable insights, all exhibit notable limitations that hinder their ability to assess the performance of open-ended segmentation models fully.
Captioning metries such as BLEU (Papineni et al. 2002). METEOR (Banerjee & Lavie 2005), and CIDEr (Vedantam et al. (2015) quantify n-gram overlap between generated and reference descriptions to evaluate the quality of automatically generated image captions. Additionally, SPICE (Anderson et al., 2016) incorporates scene-graph similarity for better semantic assessment. Still, these metrics were designed initially for con- strained captioning tasks and often correlate poorly with human judgments in more complex or open-ended settings. Embedding- based metrics like Sentence- BERT (Zhang et al., 2020) and CLIPScore (Hessel et al. 202T) leverage pre-trained language models to assess similarity and are more robust to lexical variations by leveraging nigh-dimensional spaces to map from description to category. However, the underlying pre- trained models' biases, limitations, and embedding quality are inherited according to the semantic concepts present in their original training data. Human studies involve collecting evaluations from human annotators who assess model outputs' relevance, accuracy, or correctness through rating scales, preference judgments, or pairwise comparisons. This evaluation methodology serves as the gold standard in complex or ambiguous cases where automated metrics fall short, as they reflect real-world human judgment. However, this evaluation metric remains expensive, subjective, and difficult to scale standardize across benchmarks.
Despite their strengths, existing automated evaluation metrics are largely designed for fixed-world scenarios, making them inadequate to capture the semantic complexity and variability of open-ended segmentation. This
4

---

Published as a conference paper ICLR 2026
limitation underscores the need for new metrics designed for the unique challenges of open-ended settings. To address this gap. we introduce new automated and standardized evaluation protocol designed for open-ended segmentation, significantly more aligned with human judgment than existing automated approaches.
2.3 OPEN-ENDER SEGMENTATION METHODS
Open-vocabulary segmentation methods have significantly advanced vision-language understanding by enabling models recognize and segment arbitrary concepts the pixel level, rather than being limited a fixed set of categories. This progress is built on foundational works like CLIP (Radford et al. (021) and ALIGN (Ji et al. 2021). which used contrastive learning on large-scale image-text datasets achieve strong semantic aligmment and generalization. Recent open-vocabulary segmentation methods (Xu et al. 2023 Dong et al.
2023 Liang al. 2023) have extended these ideas from global image-level tasks to region- and pixel-level understanding, enabling models to align visual regions with diverse textual concepts and perform well on novel categories. Moreover, some state-of-the-art works (Rewatbowornwong et al. 2023 Ülger et al. 2025 Shin et al. 2024) have gone beyond peec-vocabulary segmentation and incorporated language models to automatically generate the input vocabulary solely from visual information, to finally perform mask classification.
More recently, MLLMs have proven highly effective thanks to their ability to process visual inputs alongside natural language instructions and generate free-form text, allowing them predict novel or unseen categories. By incorporating both global visual features and localized region embeddings into prompts, these models can be queried about specific image regions. Early MLLM-based approaches used bounding boxes input regions (Chen et al. 2023 Zhang et al. 2023). However, bounding boxes often include irrelevant background content, introducing noise that impairs recognition performance. To address this, recent works (Yuan et al.
2024: You et al.) 2023) Shi et al. 2025) replaced bounding boxes with pixel-level masks, significantly improving accuracy by providing finer- grained inputs that yield more precise and context-aware predictions. However, while open-vocabulary segmentation has successfully leveraged contrastive learning to align regions with textual information (Xu et al.) 2023 Dong al. 2023 Liang et al. 2023) and achieved strong results, such concepts have been largely underexplored in open-ended segmentation, suggesting promising directions to refine training methodologies. In this work, we introduce, to the best of our knowledge, the first MLLM trained with contrastive learning objective to align visual regions with textual descriptions.
3 LEXICAL ALIGNMENT METRICS
Open-ended segmentation aims at generating accurate semantic descriptions and segmenting visual regions within an image. Formally, given an input image I E H the goal is produce binary masks [M, where M, E {0, 1 W and language descriptions {T; LN the semantic information contained within each visual entity.
Task Formulation:
==1 according to
We follow established open- open- vocabulary evaluation protocols (Li et al. 2022 Ghiasi jet al. 2022; Xu et al.] (022) to assess semantic generalization by training on one dataset and evaluating, without fine-tuning, on a distinct dataset whose vocabulary only partially overlaps with the training set. This protocol ensures that performance reflects true open-ended understanding rather than simply memorizing labels by introducing novel categories and altering label frequencies at test time.
Evaluation Metries
Existing high-quality segmentation benchmarks (Cordts al. 2016 Zhou et al. 2017), provide image regions with a unique ground-truth label within a closed-set vocabulary. However, since open-ended segmentation aims at generating free-form language descriptions, mapping function is required align these outputs with the target taxonomy of the testing benchmark. Previous approaches typically use embedding babeedding-based similarity scores (e.g., Sentence- BERT (Zhang et al. (0200) to match each description to the category with the highest
5

---

Published as a conference paper at ICLR 2026
similarity. Nonetheless, this "forced choice" design requires every output to be assigned to one of the available labels, even when none adequately represent the described concept, leading to systematic misassignments. Moreover, this mapping function inherits the biases and limitations of the underlying embedding models, and its accuracy fluctuates with embedding quality rather than reflecting the model's true visual understanding.
To address the limitations of current approaches, we design a mapping function that relies on lexical relationships among nouns. Our lexical mapping function operates on many-to-many basis, supporting three key characteristics that align with human judgment: (i) semantic accuracy: descriptions associated with each category must be semantically precise, preventing misclassification when no meaningful lexical relationship exists.; (ii) flexibility: each category can be related to multiple descriptions, recognizing that different sentences accurately convey the same concept; and (iii) lexical proximity: single description can have multiple lexical relationships with different categories on varying levels of lexical granularity, giving space hierarchical taxonomies in the target vocabulary.
To meet the characteristics defined above, we start by considering a comprehensive list of nouns mined from large-scale image-text pair dataset (Huang et al. 2023). This list of candidate nouns may contain compound nouns. We then use an LLM to identify which categories are semantically associated with each noun, using the prompt shown in Figure of Appendix A Finally, we compile a list of nouns that are lexically related to each category C; in the target vocabulary {C, Our lexical mapping function considers multiple lexical relationships between the generated descriptions [T:) and potential semantic categories { Thus, we propose organizing them in different lexical levels, including exact matches, synonyms, hyponyms, and meronyms (see Figure 2ap. Exact matches refer to descriptions that are matched directly by comparing the strings. Synonyms involve descriptions that convey the same or similar meaning using different words. Hyponyms refer to descriptions that indicate more specific instance or subtype of the target category, while meronyms describe parts or components of the target category. These lexical levels are cumulative, meaning each level builds upon the relationships defined in the previous levels. This structure allows for nuanced and hierarchical alignment of descriptions and categories.
Given language description Ti and target vocabulary {C:) we define a function T:T - ( maps the output description to a list of all categories with which a lexical relationship exists, based on the specified lexical level. Figure 2 considers a target category "dog" and its associated valid vocabulary (Figure 2ap to illustrate how different descriptions are mapped to this semantic category at various lexical relationship levels (Figure 2b). Particularly, to process predicted free-form description, we use automatic sentence positional tagging methods to extract the sentence's 's subject, identify its singular form, and define a lexical relationship between the description and a category if the extracted subject appears in the list of nouns associated with the category for each lexical level considered. Since standard recognition metrics typically require assigning a single category, the description is mapped to the ground-truth category if a lexical relationship with that category is found. Otherwise, the description is assigned to the category with the strongest available lexical relationship up to the given level. Suppose no lexical relationship can be established with any target category. In that case, the description is assigned to background category, ensuring the output remains meaningful within the task context.
Using this approach, we obtain mapping for a description at each lexical level, which allows us to calculate the desired recognition metric. We compute these metries across all points along the semantic dimension to evaluate the model's overall performance. The results are plotted as a curve, with the x-axis representing the semantic dimension (i.e., lcxical levels evenly spaced) and the y-axis showing the metric value. We name evaluation protocol as the Lexical Alignment Curve (LAC). To provide a comprehensive measure of a model's ' performance, we compute the area under the curve that aggregates all levels of lexical alignment. This evaluation protocol is compatible with any standard recognition metric, making it broadly applicable across diverse open-ended segmentation tasks. Moreover, LAC enables diagnostic insights into how precise, generic, or ambiguous the model's 's descriptions are, based on its performance at each lexical level.
6

---

Published as a conference paper at ICLR 2026
4 OPAL
We introduce the OPen ALignment method (OPAL), which enhances open-ended segmentation by introducing a contrastive training objective alongside the standard generative loss, building upon the Osprey (Yuan et al. 2024 architecture. This complementary learning objective enables the model to better aligned textual and fine- grained visual information in the embedding space which results in more robust generation capabilities. The core components remain unchanged: CLIP-based (Radford al. 2021) vision encoder extracts multi-scale visual features, which are projected into the language embedding space via a visual projector. A mask-aware visual extractor computes embeddings specific to segmented regions, enabling precise representations of visual entities. These embeddings are then processed by LLaMA (Touvron et al. (202) language model, fine- tuned with Low-Rank Adaptation (Hu et al. 2022) (LoRA), which generates detailed semantic descriptions conditioned on the visual context. We provide implementation details in Appendix B.2
As shown in Figure 0 of Appendix [..1 the training process jointly optimizes two complementary loss functions, each requiring different forward pass. The generative loss follows the standard approach, where the LLaMA model prompted using visual and mask embeddings to generate a description. This learning objective encourages fluent and contextually relevant outputs. The contrastive loss aligns textual descriptions to regions in a joint embedding space. For this objective, only the textual description is included in the LLaMA prompt, and contrastive learning is performed between the last language embedding and the mask embedding. This function ensures tighter semantic alignment between the visual and textual modalities. We report the computational complexity overhead of the contrastive loss in Appendix B3
(a) Overall accuracy
(b) Accuracy for diverging mappings
(c) Alignment with human judgment per lexical level
Figure 3: Human verification results. (a) We report the accuracy of Sentence- BERT mapping vs our proposed Lexical mapping with respect to the human- human-selected category. Our lexical mapping function achieves a better alignment with human judgements. (b) Furthermore, we compute the accuracy with respect to the human label for the subset of descriptions where our mapping and Sentence-BBii BERT mapping differs. (c) Finally, we evaluate our lexical mapping alignment with human selected categories when prioritizing ground-truth region labels.
5
EXPERIMENTS AND DISCUSSION
5.1 HUMAN VERIFICATION
To evaluate automated mapping functions, we conducted a two-stage human verification over 2800 region- level descriptions from the Cityscapes (Cordts et al. 2016) validation set, specifically targeting instances where Sentence- -ERTT (Reimers& & Gurevych]] 2019) and our mapping function disagreed at one or more lexical levels. The descriptions were generated by both Osprey-7B (Yuan al. 2024) and OPAL to introduce linguistic variability. In the first stage, annotators assigned one of the Cityscapes semantic categories, or "None" if no valid match was found, to each region description. Disagreement was observed in only 2% of
7

---

Published as conference paper ICLR 2026
cases. These were then re-annotated independently second stage, resulting in a 100% agreement rate across all final annotations used for evaluation. This high inter-annotator agreement underscores the clarity and consistency among humans in performing the task. In Appendix C.T we present additional details on the annotation interface (Figure 7 and show qualitative results (Figure
We compute accuracy for Sentence-BERT mapping and our proposed lexical mapping function against human judgments. Particularly, for our mapping function we consider it true positive if the human-selecter category between the valid vocabulary for any lexical level. As shown Figure Sa our method aligns with human annotations in over 90% of cases and outperforms Sentence-BERR by more than 20% across "things" and "stuff" categories. Sentence- BERT achieves only 60% accuracy overall and performs especially poorly on "stuff" categories, with correct mappings in just 50% of cases.
To gain more insights about disagreement between the two mapping functions, Figure 3b consider only the subset of descriptions where mapping and Sentence- BERRTT differs. In these high nigh-disagreement examples, our method recovers the human-assigned category 84.4% of the time, while Sentence-BERR succeeds in only 4.8%. Moreover, mapping outperforms Centence-BERY at the "Exact" semantic level, indicating that this smmbedding-basee mapping function fails to assign the correct category even when the is a exact string matching between the category and the description's 's subject.
To evaluate whether the heuristic described in Section for selecting a single category when description lexically relates to multiple target categories introduces bias artificially inflates the metric, measured the accuracy of our mapping against human annotations at each lexical level. Additionally, we disaggregated these results into "things" and "stuff" categories. Figure[ccshhore that aligmment between our lexical mapping function and human annotations improves the vocabulary flexibility increases. These results confirm that prioritizing ground-trui -trth labels within the valid vocabulary preserves precision and maintains strong alignment with human understanding across all semantic levels. Notably, our mapping performs consistently well on both "things" and "stuff" categories, achieving 74% accuracy at the meronym level, highlighting robust correspondence with human annotations.
Table I: Open-ended segmentation results for Pan) semantic (mloU), and instance (AP) segmentation tasks evaluated with our proposed evaluation protocol on the validation sets of ADE20K (Zhou et al. 2017) and Cityscapes (Cordts et al. 2016) datasets. Best results are shown in bold.
ADE20K Zhou al.
2017
Cityscapes (Cordts 2016
Method
PQ
mloU
AP
PQ
mloU
OVSeg (Liang et al. 2023
26.2
25.3
19.9
30.7
34.3
18.9
MaskCI.P (Dong el al. 2023
27.1
22.9
19.2
22.4
22.6
11.7
MasQCLIP [uu ct al. 202.5
38.6
34.2
26.7
43.8
45.6
23.4
Baseline
42.9
39.8
53.7
28.8
Shikra-7B (Chen el al. 2023
33.9
25.5
21.8
27.4
29.2
16.3
Ferret-7B You et al 2025
42.3
31.2
31.3
34.3
36.9
26.0
GPT4RoI-7U iZhang el 2023
36.5
33.7
40.6
25.3
Osprey-7B Yulan
2024
46.6
1.1
36.4
50.2 +
55.4
0.7
31.3 0.6
OPAL (Ours)
48.8
0.6
38.9 0.6
38.4
0.7
52.8
1 0.6
56.1 0.4
31.9 1 0.6
5.2
MAIN RESULTS
To advance open-ended segmentation, we re-benchmark statc-of-the-a models using their source code and pretrained weights with our proposed evaluation protocol. Our benchmarking covers models originally devel- oped for open-vocabulary recognition, adapted to open-ended segmentation tasks using a comprehensive noun list input vocabulary approximate the open-ended target taxonomy. As baseline, introduce simple
8

---

Published as conference paper ICLR 2026
9
9
8
8
.7
.7
.6
.6
12
.5
P2
.55
.4
2
<&act
&aat
synonyms hyponyms meronyms
synonyms hyponyms meronyms
(a) ADE20K Zhou et al.
2017
(b) Cityscapes CCrddts et al.
2016
(Ouns)
Yran Osprcy-78
GPT4Rod-7R (hung of
Baseline
Ferret71 7B
You ot al. 2023
Mc0CLLIP (Xa ct 2023
Shikm- 78
Chen al. 2023
MaskCLIP
Dong et
2023
OVSeg
Lanng ct 2043
Figure Lexical Alignment Curve. We re-henchmark previous state-of-the-art methods using our proposed evaluation metric for panoptic segmentation the validation set (a) ADE20K (Zhou et al.) (007)) and (b) Cityscapes (Cordis) ct al. (2066) datasets. OPAL outperforms hoth open-wocahulary and MILLM methods for all lexical levels across the two benchmark datasets.
approach leveraging MILMs: first, an image is passed through an MLLM prompted to generate a compre- hensive list of nouns describing the visual entities present in the scene (Figure Appendix C.2 then, these predicted image tags serve as the input vocabulary for an open-vocabulary recognition model specifically, MasQCLIP (Xu al.) 2023), state-of-the-art MLLMs trained to process images and visual prompts in an open-ended setting. Finally, we compare the performance of our proposed method to validate our contributions.
is the best- performeff open-vocabulary method. Furthermore, evaluate
Figures 4a and present lexical alignment curves on the validation set of ADE20K (Zhou et al. 2017) (research-only, non-commercial) and Cityscapes (Cordts al., 2016) (non-commercial, terms of use) datasets for open-ended panoptic segmentation. As expected, given our cumulative definition of lexical levels, performance improves for all methods as semantic flexibility increases. When grouping methods based on whether they utilize MLLMs, we observe that non-MLLM methods exhibit more pronounced performance jump from synonyms to hyponyms. This suggests that open- apen-vocabulary models tend to select more specific semantic concepts than generation-based models. Moreover, we prove that our method achieves -tate-of-the- art results for all lexical levels. We present additional results in Figures 10 and 12 of Appendix C.2
Table I depicts the overall lexical alignment metric for panoptic, semantic, and instance segmentation on the validation set of ADE20K and Cityscapes datasets. Our method outperforms methods by at least and 0.6 absolute points across all tasks on the ADE20K and Cityscapes datasets, respectively. We conducted inference experiments varying random seeds and generation hyperparameters for Osprey and our method. The box plots of the lexical alignment metric for the open-ended panoptic segmentation task (Figure 11 in Appendix C.2 show that our model not only surpasses the previous state-of-the- but also cuts output variance by nearly half. This demonstrates that incorporating contrastive loss during training enhances model robustness.

---

Published as a conference paper ICLR 2026
Table 2: Lexical mapping impact across different LLMs. We present open-ended segmentation results to compare the effect of using two different LLMs for the lexical mapping process. We include open-ended segmentation results for panoptic (PQ), semantic (mloU), and instance (AP) segmentation tasks, evaluated using our proposed evaluation protocol the validation sets of the ADE20K (Zhou al. 2017) and Cityscapes (Cordts al. 2016) datasets.
ADE20K
Cityscupes
Method
LLM
PQ
mloU
PQ
mloU
AP
GPT-4 (Achiam
2023
45.6
36.4
35.5
49.0
53.7
30.1
Osprey Yuun et al. 2024
Gemini
Comanici al.
2025
45.6
36.2
35.0
46.2
51.7
28.2
GPT-4 (Achiam al
2023
47.9
37.9
37.6
52.4
55.3
32.7
OPAL (Ours)
Gemini
Comanci ot al
2025
48.0
37.7
37.1
48.3
51.7
30.2
5.3 ABLATION STUDY
Impact of Lexical Mapping Across Different LLMs: To further assess robustness to LLM-specific variability, we performed an ablation which the lexical mapping process was repeated using two different LLMs. For all ablation experiments, computational complexity. To ensure fairness, we used identical prompts and set the temperature to 0.1 for all LLMs, minimizing stochasticity. Table 2 presents open-ended segmentation results for panoptic (PQ), semantic (mloU), and instance (AP) segmentation tasks, evaluated under our proposed evaluation protocol on the validation sets of ADE20K and Cityscapes, using GPT-4 and Gemini lexical mappings. The observed performance differences are negligible, and method ranking is preserved, providing additional empirical evidence that our framework is not overly sensitive to any particular LLM's biases.
report the median results for OPAL and Osprey to account for
Impact of Lexical Coverage on Open-ended Segmentation: To address the effects of noun list size on mapping performance, we construct nested noun subsets containing 20, 40, 60, and 80% of the full list by randomly sampling the 20% subset and expanding it so that each larger subset contains the smaller ones. We ensure that all original dataset categories remain present in every subset. We then evaluate Osprey-7B and OPAL on the open-ended panoptic segmentation task using these progressively larger lists. As shown in Figure 13 in Appendix C.2) our method outperforms Osprey-7B across all subset sizes, and LAC increases consistently for both models as the noun list grows, with a 6% gap between the full noun list and the 20% subset. This trend aligns with the nature of our metric: reducing the number of nouns narrows the set of matching candidates, increasing the likelihood that descriptions are mapped the background category. Furthermore, the steady improvements with increasing noun set size highlight the value of comprehensive lexical coverage. These findings confirm that our mapping's 's nouns reliably represent the semantic concepts generated by an LLM in open-ended recognition tasks.
6 CONCLUSIONS
In this work, we introduce a novel evaluation protocol for open-ended segmentation that more accurately captures semantic alignment between generated free-form descriptions and target taxonomies. We empirically demonstrate that sobbedding-bseee similarity score mappings diverge significantly from human judgments through a systematic human verification. To address this issue, we design a mapping function that considers multiple lexical relationships, achieving closer alignment with human annotations. We integrate this mapping into a robust evaluation framework and re-benchmark previous state-of-the-arr methods in open-ended segmentation tasks. Furthermore, propose, to the best of our knowledge, the first MLLM trained with contrastive objective to align visual regions and textual descriptions. OPAL achieves new state-of-the-aa results on two standard benchmark datasets for open-ended panoptic segmentation.
10

---

Published as a conference paper ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results and promote further research in open open-ended segmentation, we provide details of the proposed metric in Section as well as the prompt used to build the Lexical Mapping in Figure Appendix [ The proposed method is described in Section|]ant and additional implementation details can be found in Appendix [ We also include additional results Appendix C code and pretrained models are available at attttps:: : / wttpcccccccccccccc wwwpy--nnnnneeeee andes /open- ended
Finally, all our source
segment at atnen benchmark
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Technical Report. arXiv preprint arXiv:2303.088744 2023.
Edward H Adelson. On Seeing Stuff: the Perception of Materials by Humans and Machines. In Human Vision and Electronic Imaging VI, volume 4299, pp. 1-12. International Society for Optics and Photonics, 2001.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE:
Semantic Propositional Image Caption Evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part V 14, pp. 382-398. Springer, 2016.
Satanjeev Banerjee and Alon Lavie. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, 2005.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic. arXiv preprint 2.XXi:23006.5/95 2023.
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-Attention Mask Transformer for Universal Image Segmentation. 2022.
Gheorghe Comanici, Eric Bieher, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv preprint arXiv:2507.06200 2025.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld. Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Un- derstanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
3213-3223 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A Large-Scale Hierarchical Image Database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. IEEE, 2009.
Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weim- ing Zhang, Lu Yuan, Dong Chen, al. MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11995-11005.. 2023.
David A Forsyth, Jitendra Malik, Margaret M Fleck, Hayit Greenspan, Thomas Leung, Serge Belongie, Chad Carson, and Chris Bregler. Finding Pictures of Objects in Large Collections of Images. In International Workshop Object Representation in Computer Vision, 1996.
11

---

Published as a conference paper ICLR 2026
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Y Yi Lin. Scaling Open-Vocabulary Image Segmentation with Image-Level Labels. In European Conference on Computer Vision, pp. 540-557. Springer, 2022.
Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu. Ping Luo, and Sifei Liu. RegionGPT: Towards Region Understanding Vision Language Model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 37966-3806, 2024.
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A Dataset for Large Vocabulary Instance Segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5356-5364, 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick, Mask R-CNN. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Geremy Heitz and Daphne Koller. Learning Spatial Context: Using Stuff to Find Things. ECCV, 2008
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7514-7528. 2021. URL https: hatp:////oo/laahhooooyyoo
org/ 2021 emnnp-main 5957
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 1(2):3, 2022.
Xiaoke Huang, Jianfeng Wang. Yansong Tang. Zheng Zhang, Han Hu, Jiwen Lu, Lijuan Wang, and Zicheng Liu. Segment and Caption Anything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13405-13417, 2024.
Xinyu Huang, Youcai Zhang, Jinyu Ma. Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2Text: Guiding Vision-Language Model via Image Tagging. arXiv preprint arXiv:2303.056577 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Languee Representation Learning With Noisy Text Supervision. In ICML, 2021.
05918
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár, Panoptic Feature Pyramid Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6399-6408 2019.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-Driven Semantic Segmentation. In International Conference on Learning Representations, 2022.
Feng Liang Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open- Vocabulayy Semantic Segmentation with AdaptAd CLIP. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
Chin- Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pp. 74- 2004.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in COntext, In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
12

---

Published as a conference paper ICLR 2026
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. arXiv preprint arXiv:1711.0510. 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. ICML, 2021.
00020
Klara Reichard, Giulia Rizzoli, Stefano Gasperini, Lukas Hoyer, Pietro Zanuttigh, Nassir Navab, and Federico Tombari. From Open-Vocabulay, to Vocabulary-Free Semantic Segmentation. arXiv preprint 1181 2025.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv preprint arXiv: 10084, 2019.
Pitchaporn Rewatbowornwong. Nattanat Chatthee, Ekapol Chuangsuwanich, and Supasorn Suwajanakorn Zero-guidance Segmentation Using Zero Segment Labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1162 1172, 2023.
Hengcan Shi, Son Duy Dao, and Jianfei Cai. LLMFormer: Large Language Model for Open- Vocabulary Semantic Segmentation. International Journal of Computer Vision, 133(2):742-75) 459, 2025.
Heeseong Shin, Chachyun Kim, Sunghwan Hong, Seokju Cho, Anurag Arna,, Paul Hongsuck Seo, and Seungryong Kim. Towards Open- Vocabulary Semantic Segmentation Without Semantic Labels. Advances in Neural Information Processing Systems, 37:9153-99 9177, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302. /3971, 2023.
Osman Ülger, Maksymilian Kulicki, Yuki Asano, and Martin R Oswald. Auto Sott-Vocabulary Semantic Segmen- tation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 24266-24275 2025.
Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based Image Description Evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566-4575, 2015.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. GroupViT: Semantic Segmentation Emerges from Text Supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18134-18144, 2022.
Xin Xu. Tianyi Xiong, Zheng Ding, and Zhuowen Tu. MasQCLIP for Open-Vocabullee Vocabulary Universal Image Segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 887-898, 2023.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and Ground Anything Anywhere at Any Granularity. arXiv preprint 10.07:23 2023.
13

---

Published as conference paper ICLR 2026
Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel Understanding with Visual Instruction Tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28202-28211 2024.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. GPT4RoI: Instruction Tuning Large Language Model Region-of-Interee arXiv preprint arXiv:2307.03601, 2023. URL wttpaaawww....aaa///aaaaaaaaaaaa 2300/
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. BERTScore: Evaluating Text Generation with BERT. ICLR, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene Parsing through ADE20k Dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 633-641, 2017.
14

---

Published as conference paper ICLR 2026
A LEXICAL ALIGNMENT METRICS
system message-1
("role:: system",
content ": "'Assistant chatbot trained to identify synonyms, hypernyms, holonyms, and semantic
relationships of nouns according
to
predefined list of categories.
Instructions:
Consider following list semantic categories. Every group words between and and separated by comma is category (e.g. "plug, socket' one category): ["road", "sidewalk", "building', "wall", 'fence', "pole', "traffic light', 'traffic sign', 'vegetation', 'terrain', 'sky', 'person', 'rider', "car", 'truck', "bus", 'train', 'motoreycle" bicycle TTee output should exclusively ISON format with selected subsets categories following format: ("synonyms" " ['category 0", "category_ I,.., "category_n n"), "meronyms": ['category 0", "category_I,... "category_n" n"), "hyponyms", l'category 00, 'category. I,...? "category_n"] DO NOT include categories outside the provided list."" list.""")
for noun in nouns:
user message ('role": :" user", "content" of ttppet: "text", "text": P'Please identify synonyms, hyponyms, and meronyms of the noun [noun] from the predefined list of categories. I system message..ppenn(uus... message
Figure 5: Lexical mapping generation prompt. We use this prompt with ChatGPT/GPT-4 to construct our lexical mapping. Starting from a comprehensive list nouns mined from a large-scale image-text dataset, we instruct the language model to identify which categories are semantically related to each noun. For every category in the target vocabulary, the prompt guides the extraction of lexically related terms. Specifically, extract synonyms, hyponyms, and meronyms to support fine-grate graiied semantic alignment between generated descriptions and predefined category set.
B OPAL
B.1
METHOD OVERVIEW
Farge langage
Mald
Lunge Modcl
Laugl
Mactaauke
Generative loss
(b) Contrastive loss
Figure 6: OPAL: OPen ALignment method. Our approach builds upon the Osprey architecture and introduces a dual-loss training strategy: (a) generative loss, defined as next-token prediction for the output description given an image and visual region; and (b) contrastive loss, computed hetween region emheddings and the language embedding of the final token, which enforces tighter semantic alignment between visual and textual modalities.
15

---

Published as a conference paper ICLR 2026
B.2 IMPLEMENTATION DETAILS
Our method is trained on Osprey-742K Yuan et al. (2024) dataset (CC-BY-NC-4.0) to ensure fair comparison. We train our method on four NVIDIA A100 GPUs with 80GB memory. We maintain the same setting used in Osprey for stage 1, which only trains the visual-language connector of the model between the pretrained visual encoder and LLM with image-text pair data. At the second stage, we decrease the learning rate to 5 X 10 with batch size of 128 and train for four epochs. At the final stage, the learning rate is further reduced to 1 X 10 with a batch size of 64 for four epochs. The maximum length of sequence in LLM is set to 2048. We leverage the Deep-Speed framework for efficient large-scale model training. We AdamW Loshchilov & Hutter (2017) optimizer with standard parameters. To ensure training stability, we apply 10% warmup, use cosine annealing learning rate scheduler, and implement gradient clipping with a value of 1.0.
B.3 COMPUTATIONAL COMPLEXITY OVERHEAD
The additional OPAL's computational overhead corresponds to only 1% FLOPs during training, while maintaining the same computational complexity during inference. We consider approach achieves good trade-off between computational complexity and accuracy.
C EXPERIMENTS
C.I HUMAN VERIFICATION
You will see a sentence describing a visual entity (such as an object, animal, body part, etc.). You must select from the dropdown menu the semantic category from the list that best describes the subject of the sentence. The list of possible semantic categories follows:
road, sidewalkk, building, wall, fence, pole, traffii light, traffic sign, vegetation, terrain, river, sand, sea, snow, water, rides, truck, bus, train, motorcycle, bicycle, None of the above
mountain, dirt, rocks, sky,
The subject described may not always match the categories exactly, so you should choose category that most closcly corresponds to it. You may consider synonyms r-g, the subject automobile", select car), hyponyms type
(e-g,, if the subject sedan', select is part bicycle, select bicycle) identtly the underlying concept. the subject does not fit any categories, select "None of the above".
meronyms (e.g., the subject is 'wheel" and
listed
Below examples how categories should assigned different deseriptions:
stopped traffic light"
"Sky with scattered clouds" sky
"Peesttran standing next to a bus" person
"Graffiii wall of old huilding" wall
"Pesson riding a matarcycle
rider
"Corgi playing with a ball" None of the above
Finally, keep the follmwing considerations in mind:
multiple objects mentioned, select
category.
2. If the description does not match any category, select "None of the above."
3. If you unsure, ask yoursell: What object or entity is the subject of the description?
Which semantic category describes object entity?
5. Is any of the listed categories explicitly mentioned?
Figure Human verification instructions. Instructions provided to annotators for evaluating region-leved descriptions in the Cityscapes (Cordts et al.) (2016) validation set. Annotators are asked to assign one of the official Cityscapes semantic categories to each description, or select "None" if the description does not refer to target category. The study focuses on where Metence-BBRTT (Reimers & Gurevych] 2019) and our mapping function disagree, and includes descriptions generated by Osprey-7B (Yuan al. 2024) and our model.
16

---

Published as conference paper ICLR 2026
Sentence-BERY Mapping
Lexical
Description
Mapping
Selection
"The onsier is rod peppers
name
Igg
food
food
"The arowat pkture"
C77
wall
picture picture
picture picture
painiig pleture
"The mnswre taol"
road
box
car
car
CEE
Exact
Hypo Mero
(a) ADE20k Zhou et al. 2017
Deseription
Mapping
Mayping
Selection
"The woman wallingg
tog parson
person
person
The shalow of person the ground"
silmoalk
bg
pesson
tomzin
TThe bealllghh of car"
OD
trafiic light
bg
bg
bg
CaT
Exact
Syn
Hypo
Mero
(b) Cityscapes Cordts et al, 2016
Figure 8: Sentence-BERT vs. our lexical mapping function. Examples of Sentence-BFiN BERT (Reimers & Gurevych 2019 and our Lexical Mapping to (a) ADE20k (Zhou al. (017) and (h) Cityscapes (Cordis al. 2016) list of categories alongside the human-verified true category. The category bg stands for background.
C.2
MAIN RESULTS
('aole: system" [[
"conten"" "Asiistant
trained to images visually by providligg comprehenstve
semantic categories present in them.
Instructions:
Scrutintze the entire including any might appear inconepicuous, identify all things stuff ontegoories
EEclude adjectives, warhs, adverbs.
TTh output should include categories all objects and surfaces, even thuse that seem miner or part of the background context.
The output should be exclustvely ISON format with lowercase and singular categories following this format: (" categories :f 0 eategory_] I b-.." category_n ']}-
Sort them confidence.""
user, message - ('role' user", "contont" : ("type: "text,, "test" "Please provide a comprehensive list semantic categories present in the imager"!)
image in images:
image, content ( type": "image uur", "image url': ['ur': I' data:: image fieg:hsse6s,ffaeeee (aneeeessnneS___aaaa"" "content""user [jsse, message, Image contenti]
system, system_ aessageeappeeddll appprrdd
user", "content"
Figure 9: Prompt for image tag extraction We use this prompt instruct ChatGPT/GPT to generate comprehensive list of nouns describing all visual entities present in the image. The resulting tags used the input vocabulary for the open-vocabulary recognition model MasQCLIP (Xu al. 2023). which we use as our baseline.
When comparing the outputs of model with those of Osprey-7B Yuan al. (2044 in Figure TO observe superior performance from our approach, especially on small objects. This suggests that our model context-aware and better at understanding and extracting information from fine- ine-grained image regions.
17

---

Published as conference paper ICLR 2026
Can you give me shoet description of -region-2 Using short phrase.
"A light on the caillng"
leaves are green
woman?
Beseer"
CEAL
OPLL
"A sigh on the calewoll
"The schee of
Orpno
"The
oNL
sndewals"
Orpave11
Oppary
"A street light on pali"
"A fnnce the sidowall"
OPNL
Figure 10: OPAL's qualitative results. Examples of OPAL output descriptions. We compare our outputs with those from Osprey-7B Yuan al. 2024). the second-best performing open-ended recognition model after ours.
Similar the results presented in the main paper, Figure 12 shows a performance improvement in both instance and semantic segmentation as the level of lexical relation becomes more flexible. Our model consistently achieves sfate-of-the-art performance across all curves, demonstrating strong and reliable generalization across different metrics and datasets.
ATHEOK
Chysoges
Figure 11: Robustness analysis. Box plots illustrating results on our proposed metric. OPAL outperforms Osprey and reduces output variance by nearly 50%, demonstrating enhanced robustness through contrastive learning training.
18

---

Published as conference paper ICLR 2026
ADE20K (Zhou et al. 2017)
Cityscapes (Cordts et al., 2016)
(a) Semantic Segmentation
9
9
.8
.8
.7
.7
-6
.6
mloU
nlou
5
.5
4
.4
3
.
2
2
exact
synonyms hyponyms meronyms
exact
synonyms hyponyms meronyms
(b) Instance Segmentaton
.9
.8
8
.7
77
.6
.6
AP
.5
.5
4
.4
33
.2
.1
exact
synonyms hyponyms meronyms
exact
synonyms hyponyms meronyms
OPAL (Ours)
Osprey-7B Yuan et al. 2024
GPT4RoI-7B Zhang al.
2023
baseline (Xu et al. 2023
Ferret-BB You et al.) (0033
MaskQCLIP (Xu et al. 2023
Shikra-7B (Chen all
MaskCLIP Dong et 2023
OVSeg (Liagg et all 2023
Figure 12: Lexical Alignment Curve. We report state-of-the-art methods results using the lexical alignment for instance and semantic segmentation validation sets of the ADE20K (Zhou et (017) and Cityscapes (Cordis) et al. 2016) datasets.
19

---

Published as conference paper ICLR 2026
ADE20K (Zhou et al. 2017)
Cityscapes Cordts et al. 2016
(a) Panoptic Segmentation
-
-5
.4
2.3
-1-
0%
20%
40%
60%
80%
100%
8%
20%
40%
60%
80%
100%
Noun List Size
Noun List Size
(b) Semantic Segmentation
-5
.4
mloU
mloU
-3
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Noun List Size
Noun List Size
66
(c) Instance Segmentation
2.3
.2
-1
0%
0%
20%
40%
60%
80%
100%
20%
40%
60%
80%
100%
Noun List
OPAL (Ours)
Osprey-7B Yuuan
2024
Figure 13: Lexical coverage impact open-ended segmentation. We report lexical alignment curves varying noun list sizes. We compare the performance of OPAL and Osprey-7B Yuan al. (2024) methods across panoptic, semantic, and instance segmentation in the validation sets of the ADE20K (Zhou et al. (017) and Cityscapes (Cordts) et al. 2016) datasets.
20


Reviews:
[{"review_id": "cIffhysy4w", "reviewer": "Reviewer_zkjh", "summary": "This paper addresses the challenge of evaluating open-ended segmentation, where models generate free-form textual descriptions instead of choosing from a fixed label set. The authors demonstrate that existing embedding-based evaluation methods (e.g., Sentence-BERT) poorly align with human judgment and propose a new Lexical Alignment Curve (LAC) metric that accounts for multiple lexical relationships—such as exact matches, synonyms, hyponyms, and meronyms—to better capture semantic correctness. They further introduce OPAL, the first multimodal large language model trained with a contrastive objective to jointly align visual regions and textual descriptions. Extensive experiments and human verification studies show that the proposed evaluation framework achieves stronger alignment with human perception and that OPAL sets new state-of-the-art results on open-ended segmentation benchmarks.", "questions": "- How does the proposed lexical mapping handle cases where the grammatical subject does not correspond to the true visual referent (e.g., “A man is walking a dog”)?\n- Does the method explicitly account for modifiers such as adjectives or compound nouns (e.g., “toy dog,” “hot dog,” “drawing of a dog”)?\n- Since the lexical mapping depends on large language models for synonym, hyponym, and meronym extraction, how do the authors address potential biases or inconsistencies in these LLM-generated resources?\n- Could the authors discuss how their framework might generalize to fine-grained or domain-specific datasets such as LVIS or COCO-Stuff?\n- Given that the experiments assume pre-defined masks as input, does the proposed setup still align with the notion of “open-ended segmentation”?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality:** The paper tackles a highly relevant and underexplored problem—how to objectively evaluate open-ended segmentation models where outputs are free-form text rather than fixed labels. The proposed Lexical Alignment Curve (LAC) introduces a novel evaluation perspective that bridges semantic granularity levels (exact, synonym, hyponym, meronym). This is a creative and conceptually elegant formulation that advances beyond traditional embedding-based similarity scoring.\n\n**Quality:** The methodology is systematic and carefully designed, combining empirical human studies, metric development, and benchmarking experiments.\n\n**Clarity:** The paper is well-structured and logically presented—the motivation is clearly articulated, the limitations of existing methods are explained with concrete examples, and the proposed framework is illustrated with intuitive figures.\n\n**Significance:** The work addresses a critical bottleneck in open-ended visual understanding: the lack of accurate, scalable evaluation metrics that align with human perception. The introduction of OPAL demonstrates practical impact by achieving measurable improvements on standard datasets, indicating both theoretical and empirical significance.", "weaknesses": "- The proposed lexical mapping relies heavily on automatic subject extraction to identify the key semantic concept in a sentence. However, this assumption breaks down for many common sentence structures where the grammatical subject is not the true visual referent (e.g., “A man is walking a dog,” “This is a picture of a dog,” or “On the grass, there is a dog”). Moreover, the method does not account for contextual modifiers such as adjectives or compound nouns, which can drastically alter meaning (e.g., “toy dog,” “hot dog,” “drawing of a dog”). As a result, the current approach may conflate semantically distinct entities or misidentify the referent.\n- The lexical mapping depends on lexical resources (synonyms, hyponyms, meronyms) extracted using large language models and linguistic heuristics. However, this approach may **also** inherit biases and incompleteness from the LLM itself.\n- While the proposed Lexical Alignment Curve framework effectively addresses the shortcomings of embedding-based mappings, its evaluation scope is relatively narrow. The analysis focuses primarily on Cityscapes and ADE20K, which, although standard, may not fully represent the diversity of open-ended segmentation scenarios (e.g., fine-grained or domain-specific datasets such as LVIS or COCO-Stuff).\n- Although the paper is titled “Benchmarking Open-Ended Segmentation,” the experimental setup assumes that masks are provided as input, meaning that the model does not perform mask generation or spatial localization itself. Consequently, the task formulation aligns more closely with open-ended region or object recognition, rather than full open-ended segmentation.\n- Several closely related works [1,2,3] are missing from the discussion. \n\n[1] Shin, Heeseong, Chaehyun Kim, Sunghwan Hong, Seokju Cho, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. \"Towards open-vocabulary semantic segmentation without semantic labels.\" Advances in Neural Information Processing Systems 37 (2024): 9153-9177.\n\n[2] Ülger, Osman, Maksymilian Kulicki, Yuki Asano, and Martin R. Oswald. \"Auto-vocabulary semantic segmentation.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 24266-24275. 2025.\n\n[3] Rewatbowornwong, Pitchaporn, Nattanat Chatthee, Ekapol Chuangsuwanich, and Supasorn Suwajanakorn. \"Zero-guidance segmentation using zero segment labels.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1162-1172. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WVs1SIhbfN", "reviewer": "Reviewer_BTMF", "summary": "This paper revisits the problem of evaluation of open-ended segmentation models.\nThe authors propose:\n1. a lexical alignment–based evaluation protocol (LAC) that integrates hierarchical word relations—Exact, Synonym, Hyponym, Meronym—for mapping free-form text to target categories.\n2. OPAL, a multi-modal large language model trained with a dual generative–contrastive objective to better align visual regions and textual outputs.", "questions": "- Is this method, maybe be used not only for open-ended segmentation? what about open-ended detection?\n- What about contextual semantics or polysemy? (e.g., “apple” as fruit vs. company). How does the system handle cases where semantically distinct words share lexical proximity (e.g., “truck” → “car”)?\n- How stable are the GPT-4–derived lexical relations across different LLMs, prompts, or randomness?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The proposed lexical mapping and curve alignment method are novel. The paradigm provides a systematic and interpretable way to evaluate free-form segmentation outputs, filling an important methodological gap.\n- The method is aligned with human verification (90% with human judgement).", "weaknesses": "- Dependence on external LLMs (e.g., GPT-4) for lexical relation extraction may reduce reproducibility and introduce hidden biases due to model updates.\n- The metric maybe be hacked by the inclusion of hyponyms and meronyms.\n- The 7B OPAL, as a tool in metric, is not convinent to deploy (specific enviroment). It may cause mismatched aligning and unfair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6LdVttef3r", "reviewer": "Reviewer_5AQi", "summary": "This paper addresses flaws in open-ended segmentation (which needs models to generate free-form descriptions of unseen visual concepts, not select predefined labels) evaluation: it notes existing embedding-based metrics (e.g., Sentence-BERT) conflict with human judgments, and other metrics/studies are misaligned or impractical. It proposes a lexical relationship-based mapping (exact matches, synonyms, hyponyms, meronyms) integrated into the LAC framework, aligning better with human annotations. It also introduces OPAL, the first contrastive-trained MLLM for the task, achieving state-of-the-art results and better robustness.", "questions": "Could you provide evidence of your lexical vocabulary’s coverage of rare/long-tail concepts?\n\n\ncould you add LLM-comparison ablation, release the noun list/LLM outputs, and quantify how noun list size/filtering affects mapping performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It innovatively proposes a lexical relationship-aware mapping function integrated into the Lexical Alignment Curve framework and introduces OPAL, the first contrastive-trained MLLM for open-ended segmentation.\n\nThe LAC framework provides a reliable evaluation benchmark, OPAL sets a new SOTA.\n\nIt ensures rigor by comprehensively re-benchmarking SOTA models on ADE20K and Cityscapes via its proposed framework, and commits to releasing resources upon acceptance to guarantee reproducibility.", "weaknesses": "The paper’s lexical vocabulary lacks verification of its comprehensiveness (e.g., whether it covers low-frequency or niche concepts), which impacts the reliability of the evaluation results.\n\nThe paper has no ablation studies on critical lexical mapping designs (e.g., LLM selection basis, noun list filtering rules), undermining the reproducibility of the mapping.\n\nThe paper only validates the framework on ADE20K and Cityscapes, failing to include widely used datasets like COCO, limiting the generalization verification of its methods.\n\nThe paper does not report OPAL’s efficiency metrics such as inference latency and memory usage, making it hard to assess its practical deployment potential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)