Revisit the Open Nature of Open Vocabulary Semantic Segmentation
Authors: Qiming Huang
Han Hu
Jianbo Jiao
Venue: ICLR 2025 Poster (2025)
Keywords: Open vocabulary segmentation
Evaluation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In Open Vocabulary Semantic Segmentation (OVS), we observe a consistent drop
in model performance as the query vocabulary set expands, especially when it
includes semantically similar and ambiguous vocabularies, such as ‘sofa’ and
‘couch’. The previous OVS evaluation protocol, however, does not account for
such ambiguity, as any mismatch between model-predicted and human-annotated
pairs is simply treated as incorrect on a pixel-wise basis. This contradicts the open
nature of OVS, where ambiguous categories may both be correct from an open-
world perspective. To address this, in this work, we study the open nature of OVS
and propose a mask-wise evaluation protocol that is based on matched and mis-
matched mask pairs between prediction and annotation respectively. Extensive
experimental evaluations show that the proposed mask-wise protocol provides a
more effective and reliable evaluation framework for OVS models compared to the
previous pixel-wise approach on the perspective of open-world. Moreover, analy-
sis of mismatched mask pairs reveals that a large amount of ambiguous categories
exist in commonly used OVS datasets. Interestingly, we find that reducing these
ambiguities during both training and inference enhances capabilities of OVS mod-
els. These findings and the new evaluation protocol encourage further exploration
of the open nature of OVS, as well as broader open-world challenges. Project page: https://qiming-huang.github.io/RevisitOVS/.

Published as a conference paper at ICLR 2025
REVISIT THE OPEN NATURE OF OPEN VOCABULARY SEMANTIC SEGMENTATION
Qiming Huang, Han Hu, Jianbo Jiao The Mix Group, School of Computer Science University of Birmingham {qxh366, hxh864) student bab.menntbbam..c.uuk ac. uk, j.jiaoolhaaat w.jiao@bham.ac.
ABSTRACT
In Open Vocabulary Semantic Segmentation (OVS), we observe a consistent drop in model performance as the query vocabulary set expands, especially when it includes semantically similar and ambiguous vocabularies, such as "sofa" and 'couch'. : The previous OVS evaluation protocol, however, does not account for such ambiguity, as any mismatch between model mreee-predicted and auman-annotated pairs is simply treated as incorrect on pixel-wiiss -iiee basis. This contradicts the open nature OVS, where ambiguous categories may both be correct from an open- world perspective. To address this, in this work, we study the open nature of OVS and propose mask-wise evaluation protocol that is based on matched and mis- matched mask pairs between prediction and annotation respectively. Extensive experimental evaluations show that the proposed mask-wise protocol provides a more effective and reliable evaluation framework for OVS models compared the previous pixel-wise approach on the perspective of open-world. Moreover, analysis of mismatched mask pairs reveals that a large amount of ambiguous cat- egories exist in commonly used OVS datasets. Interestingly, we find that reducing these ambiguities during both training and inference enhances capabilities OVS models. These findings and the new evaluation protocol encourage further explo- ration of the open nature of OVS, as well as broader world challenges. Project page: https://qiming -uanng /ithub.io/ReisiiOOVS.
"The limits of my language the limits of my world."
Ludwig Wirtgenstein
INTRODUCTION
Open-worle learning aims to address the problem of learning with novel and unknown categories data distributions that often encountered in the real world. With the development of large language-vision (LLV) models, such as CLIP (Radford et al., 2021), open vocabulary tasks are proposed utilise the strong language visual alignment capability from LLV identify objects from the data, including new entities not in the training data. Particularly, open vocabulary semantic segmentation (OVS) is a task where models trained on close-set semantic segmentation dataset perform zero-shot inference on an unseen dataset by providing vocabulary set for any object.
In an open- world setting, object category boundaries are often ambiguously defined. For instance, when describing visual objects through language, highly ambiguous vocabulary may be used. While this works closed-set settings, where labels are assumed to be mutually exclusive. However, the case of open vocabulary semantic segmentation (OVS), where any vocabulary, including those with significant semantic ambiguity, can be introduced. For example, as illustrated in Fig. 1, under the current OVS evaluation protocol, predictions such as 'flower' and 'chair' are considered incorrect, despite appearing reasonable to humans from an open-world perspective. address the challenges posed by such ambiguous categories by revisiting the open nature of OVS, trying to answer: whether we should trat the ambiguous categories as incorrect or correct ones, and how encourage OVS be more open?
this paper, we aim to

---

Published as a conference paper at ICLR 2025
trade
base
cabin
rson 0801
Lable
Seat
Image
Human Labels
Prediction
Figure I: Category ambiguity in open vocabulary semantic segmentation. One object can be as- signed multiple possible labels while the human label is only one of them. For example, the area on the left with a yellow star annotated as 'plant' by humans, but predicted to be 'flower' by the OVS model; the bottom part annotated as 'seat' was predicted as 'chair' by OVS model.
Specifically, we first revisit the existing OVS evaluation process from the perspective of open- world learning, and find that it follows a closed-set approach, where predictions are considered incorrect they do not match with the predefined category. We observe that as the number of inference cat- egories increases, the performance of existing OVS models significantly declines, indicating issues caused by category ambiguity. study this, we find that expanding model predictions from pixel- wise argmax category-wise mask-wise predictions effectively mitigates such ambiguity problems. To this end, we propose an open-set prediction approach and a corresponding generalised category- preserving evaluation metric. We find that the proposed new prediction and evaluation framework significantly improves the performance of existing OVS methods. Additionally, based on our pro- posed evaluation framework, construct ambiguous vocabulary graph between model predic- tions and human annotations, revealing clear community structures where vocabularies within the same community correspond visually similar objects.
The main contributions of this study can be summarised as follows: 1) We revisit existing OVS paradigms from open-world perspective, offer insightful observations, and propose feasible lutions to encourage more openness in OVS. 2) Our proposed mask-wise evaluation protocol effec- tively addresses the issue of ambiguous categories in open-world evaluation. 3) Based on the pro- posed evaluation framework, we introduce an approach to conltruct a confusion vocabulary graph for existing OVS datasets, highlighting the significant presence of ambiguous category annotations. 4) Extensive experimental analysis and comparisons validate the effectiveness of the proposed method, which achieves state-of-the-a performance with a new paradigm for OVS the open world.
2 RELATED WORKS
2.1 CLOSE-SET SEGMENTATION
This task aims to segment images into regions with predefined categories. Fully Convolutional Networks (FCNs) (Long al., 2015) marked the beginning of the deep learning era in image seg- mentation. Subsequently, convolution- based (Li et al., 2023) and Transformer- based (Liu et al., 2021) approaches further enhanced the model's performance in semantic segmentation (Zhao al., 2017; Wang et al., 2020; Shen et al., 2022; Chen et al., 2017; Ronneberger et al., 2015; Xie et al., 2021), instance segmentation (He et al., 2017; Lin et al., 2014; Chen et al., 2019a; Liu et al., 2018; Qi et al., 2021), l), and panoptic segmentation (Kirillov et al., 201 2019b;a; Cheng et al., 10 Zhang et al., 2021). Despite the continuous advancements in closed-set segmentation methods, predefined cate- gory sets are inadequate for open-wort world vision applications where the number of object categories is vast and constantly evolving. In this work, we focus on adapting closed-set evaluation metrics to accommodate open-world scenarios, especially for dealing with the case of ambiguous categories.
2

---

Published as a conference paper at ICLR 2025
2.2 OPEN VOCABULARY SEMANTIC SEGMENTATION
Close-set segmentation aims to train the model to segment predetermined categories while OVS aims to segment the objects with arbitrary vocabulary queries (Cho et al., 2023; Xie et al., 2023; Xu et al., 2023) which enables the open ability for model prediction. There has been some re- cent exploratory work in this direction. LSeg (Li et al., 2022) utilised CLIP (Radford et al., 2021) to train a visual encoder that generates pixel-level visual embeddings from an image, which are aligned with the corresponding textual embeddings learned from the training labels within the CLIP embedding space. OpenSeg (Ghiasi et al., 2022) employed a class -ausosggn segmentation module cross-attention to detect local regions in images. Two-stage frameworks, ZegFormer (Ding et al., 2022) and ZSseg (Xu et al., 2022), also extract class-agnostic region pro- posal similar to (Ghiasi et al., 2022) at the first stage, then utilised pretrained vision-language models like CLIP to classify masked regions. Liang et al. (2023) improves CLIP's performance on masked images by finetuning CLIP on image-text pairs. CAT-Seg (Cho et al., 2023) proposed cost aggregation method to optimise the image-text similarity map by fine-tuning the CLIP en- coder and obtaining accurate pixel-level predictions. MaskCLIP (Dong et al., 2023) integrates a novel masked ceff-disiillttion technique into contrastive language-image pretraining, aiming to de- rive pixel-level embeddings from CL.IP for immediate application in segmentation tasks. SED (Xie al,, 2023) introduced an encoder-decoder architecture consisting of hierarchical encoder-based cost map generation and a gradual fusion decoder with category early rejection to obtain pixel-level image-text cost map prediction. Despite the progress achieved by the previous OVS models, the existing training and inference of OVS still adhere to the pipeline of close-set recognition, i.e. a fixed training vocabulary set is used during training, and particular -laaset--pecific vocabulary set is given during inference.
utilising
2.3 THE EVALUATION OF OVS
The current evaluation methods for open vocabulary semantic segmentation primarily rely on the mean Intersection over Union (mloU) metric, which assesses classification accuracy the pixel level. However, this approach depends on strict pixel matching, where predictions are deemed cor- rect only if they match the ground truth class exactly, making it unsuitable for open-world scenarios. Recent studies (Zhou et al., 2023; Liu et al., 2023) attempt to address this by incorporating textual similarity between categories, assigning partial accuracy scores based on the degree of similarity during IoU computation. While this approach introduces flexibility, textual similarity alone cannot reliably capture the visual-semantic relationships between objects, limiting its effectiveness. work, we focus on evaluating models based on visual similarity, directly comparing the overlap be- tween segmentation predictions and human annotations to better distinguish visually similar classes in open-world settings.
this
REVISITING OPEN VOCABULARY SEMANTIC SEGMENTATION
Definition of OVS. Open vocabulary semantic segmentation (OVS) aims to train segmentation models that can leverage textual descriptions to segment arbitrary objects. Given two category sets, Otrain and Ctest, where Ctrain and Ctest are not equal in terms of object categories (Ctrain # Crest), the model is trained on Ctrasn and directly tested Creat- Typically, Ctrain and Creat are described using noun phrases (e.g. sky, ocean, mountains, etc.). During the testing stage, the previous assumption held by OVS is that it is known which dataset the test data originates from, and the whole vocabulary set corresponding to the dataset is provided as the inference vocabulary, CD,, CD....., CD............
Training objective. Consider the Maximum A Posterior (MAP) estimate for training deep learn- ing model with given data X and a certain vocabulary set V, let the prior distribution of θ be g()). We want to find parameter θ that maximises:
+ MAP = arg max P(e V).
(1)
3

---

Published as a conference paper at ICLR 2025
Since the training vocabulary set V considered a fixed set in most previous OVS works (Cho all, 2023; Xie et al., 2023; Xu et al., 2023), we apply Bayes' theorem to get:
MAP & arg max log (x + log s()).
Although OVS intends incorporate vocabulary during training, the objective above fails con- sider the vocabulary during the model optimisation. Here we propose to turn the training vocabulary into random variable represented by P(V), aforementioned, we have our MAP estimate for training an OVS model:
MAP =
arg max P( V)
9
O( P(V()))))()
arg max
P(X
arg max log P(X + log P(V
log P(O).
(3)
θ
likelshood
language likelihood
prior
We notice that compared to the original objective in Eq. 2, the current optimisation incorporates another term P()(@) that relates to the vocabulary distribution (for detailed mathematical deriva- tion, please refer to the supplementary material). Considering vocabulary V as random variable during training, the model parameters θ are dependent on both the observed training data X and the vocabulary V through maximising the P(V(O) and P[X ( V) terms. This implicitly constructs the relationship between model parameters and vocabulary distribution, and could be beneficial to OVS in open world scenarios.
Zero-shot inference capability. Open vocabulary semantic segmentation models can perform zero-shot inference on unseen datasets while providing customisable vocabulary. Given an image I, RB,C.W. and vocabulary candidate set V E RB.D, the OVS model takes I; and V as in- put, and generates class posterior P(y.|X, θ, V). Usually the consequent semantic segmentation predictions are obtained by applying the argmax operation:
Ý = arg max P(y P (). B.EV
(4)
In zero-shot inference, spatial posterior Ý E R.W.H. is generated, where C denotes the number of classes, similar to the vocabulary in open vocabulary semantic segmentation (OVS). Each pixel is as- signed to the class in Viest with the highest posterior probability. Existing OVS methods assume that the test images come from a specific dataset (e.g. ADE20K (Zhou et al., 2019) or PASCAL-Context (Mottaghi al., 2014)), and they the corresponding dataset-specific vocabulary to restrict all predictions to the dataset's 's vocabulary.
an open-world scenario, the open nature of visual semantic objects implies that they may belong to multiple labels (described by different vocabularies or captions), which is overlooked. This open nature in vocabulary segmentation emphasises that object classifi- cation is not rigid or strictly predefined, allowing for greater flexibility in category assignment and interpretation. Specifically, it encompasses two key aspects: 1) Multiple Labels: An object can simultaneously be associated with multiple category labels, reflecting the complexity and richness of real-world concepts. 2) Semantic Similarity: A predicted category may not exactly match the ground truth label, but if is semantically similar, it should not be considered entirely incorrect. For example, predicting "vehicle" instead of "car" still valid, as both vocabularies capture the essence of the object within broader semantic context.
The open nature of OVS.
4 MASK-WISE EVALUATION PROTOCOL
4.1 NOTATIONS AND DEFINITIONS
For the sake of clear understanding, we first define the main symbols used in the proposed evaluation framework and their meanings. For image X, from testing dataset DD we have the class probability
4

---

Published as a conference paper at ICLR 2025
In-matehed pair
Out-rpulched puir
stiroaa
stans
Pred
Community Discovery
Logits
Updag
light
light
>r
wall
wall
Pred
Valid Set
Ensor
Ambiguous Vocabulary Graph
floor
Adjacent
Binary Masks
Prod
Matrix
Figure 2: The proposed mask-wise evaluation protocol. The Valid Set A consists of all masks where A = [M; M, E (M1, M2...., Mx}] ). BB represents the list of masks where the predicted category matches the category annotated in the ground truth (GT). B is the set of masks obtained by performing bipartite matching between (A \ B) and the GT, where the IoU the matched pairs exceeds the threshold TAV. For example, "stairs" belongs to in this figure. C is defined as
= A (B BB. The adjacent matrix helps construct the ambiguous vocabulary graph, enabling community discovery for better understanding model predictions. See Section 4.3 for details.
distribution predicted by the model, referred to as logits, denoted by L, E RC where C represents the total number of classes, and W and H denote the width and height of the image, respectively. Pixel-wise semantic segmentation annotations are represented as Y; E 1, C, 1]W*H, where each value corresponds the class index of the respective pixel. The one-hot encoded form of these annotations is given by M, E {0, 1x x where each pixel is either 0 or M {0, (0,1) Civue xWx H is the mask list with all annotated category mask only, Cirue C. Binary predicted masks, B(r) E {0, 1) CxWxH are obtained by thresholding L, with threshold such that:
1,
If
B(r):
(5)
0, otherwise.
The CM E NCx2x2 a binary confusion matrix for dataset D, where each class c has:
TPc
FP_
CM. =
(6)
FN TN
EM E Cx is a vector used to quantify the error proportion for each class in the model's predic- tions. The error proportion is calculated as the ratio of the number of pixels with value of in the corresponding binary mask to the total number of pixels in the image as follows:
Number of pixels with value of in the binary mask
erric
Total number of pixels
4.2 EVALUATION PROTOCOL
Our mask-wise evaluation protocol provides CM, AV, and EM under different thresholds as detailed Appendix A. Here, CM represents the binary confusion matrix between the predicted categories and the annotated categories. AV captures predictions with high overlap (> TAv) between the pre- dicted and annotated masks but with mismatched categories. Only CM and EM are used to evaluate the model's performance, they are based comparisons with the ground- truth annotations. AV, however, is utilised for subsequent ambiguous vocabulary graph analysis (i.e. analysing ambiguous vocabulary in the model's predictions) rather than directly assessing the model's performance.
5

---

Published as a conference paper at ICLR 2025
Based proposed evaluation protocol, define three metries front, back and err to evaluate OVS model performance across different thresholds, defined as:
[TP. CM, TP. [FP] + [F.
fromt,
(8)
CC| CEC
[TN.
back,,
C
CM, [NN. CM, [P P.j + CM,
DEC
1 C
err,
EM,
(10)
DEC
Here, front,,e and back,,e represent the recognition IoU of the foreground and background classes, respectively, for each category c, where с under threshold т. The foreground class refers to the pixels labelled as relevant the target in category e (i.e. the regions belonging to that category), the background class refers to the pixels labelled as irrelevant to the target category a (i.e. regions outside that category). The err, represents the average proportion of incorrectly predicted pixels across all categories under the threshold т. Inspired by best F1 thresholding (Lipton et al., 2014), the best threshold 7* can be automatically determined by:
Vfrom? + (1 - err,)2
arg max 7E(0.1,0.2....p0 2...
(11)
4.3 AMBIGUOUS GRAPH IN OUT-MATCHER PAIR
Building ambiguous vocabulary graph. Graph is structure used to model pairwise relationships between objects. is commonly described using an adjacency matrix, where each entry in matrix represents the connection or interaction between two nodes (in this case, vocabularies). The confusion graph is constructed based on the model's predictions and the manually annotated classes. The graph used for analysing the model's performance in classification task (Jin et al., 2017).
The adjacency matrix AV for ambiguous vocabulary graph is the ambiguous vocabulary matrix in Appendix A.1. Each element AV.. represents the number of times the model predicts class j. while the ground truth is class i. For each out-matched pair, the adjacency matrix is updated. For example, there is out-matched pair such as "couch"- "soof"" where the ground truth class is "sofa" and the predicted class "couch". " we update the corresponding entry in the adjacency matrix, AV sofa. by incrementing by 1. This represents the frequency with which the model misclassified "sofa" as "couch" with a high IoU overlap (IoU >> f).
Community discovery over ambiguous graph. Given the confusion graph represented by the cor- responding adjacency matrix A E RCXC we can perform community discovery to identify groups of classes that are frequently confused with each other. This process involves partitioning the nodes (classes) into communities such that nodes within the same community have stronger connections, as reflected by higher values in the adjacency matrix, than those across different communities.
One common approach for community discovery is modularity maximisation. The modularity Q of a given partition of the graph defined as:
1 2m
k,kj
Q =
As.
8(c;,
(12)
2m
where Ar, is the weight of the edge between nodes and in the adjacency matrix, k; and ky are the degrees (total edge weights) of nodes and j. respectively. m is the total weight of all edges in the graph, i.e.
Eij Ai,j 8(c,, cy) is the Kronecker delta function, which is I if nodes i and belong to the same community and 0 otherwise. The goal of community discovery is to maximise Q in order to find the optimal partition of nodes into communities. In the ambiguous vocabulary graph, if two categories often confused by the model, they are likely to be in the same community.

---

Published as a conference paper at ICLR 2025
5 EXPERIMENT
5.1 DATASETS AND IMPLEMENTATION DETAILS
Following previous OVS works (Cho et al., 2023; Xie et al., 2023; Xu et al., 2023), we train the mod- els on the COCO-Stuff171 (Caesar et al., 2018) dataset with 171 categories and perform zero-shot evaluation on ADE20K (Zhou et al., 2019) and PASCAL-Context (Mottaghi et al., 2014) datasets. ADE20K has two types of annotations namely ADEI50 with 150 classes, and ADE847 with 847 classes. PASCAL-Context has the most frequent 59 classes annotation version PC59, and fully annotated version PC459 with 459 categories.
In this work, we utilise the following benchmark models for evaluation: SAN (Cho et al., 2023), CAT-Seg (Cho al., 2023), SED (Xie al., 2023), and MAFT+ (Jiao et al., 2024). In addition to the default setting, which uses lataset-speciffe vocabulary during inference for each dataset, we also created joint-dataset inference vocabulary set, denoted by x. This joint-dataset vocabulary set disjoint from PC59, ADE1 PC459, and ADE847, resulting in total of 1, 1,086 vocabularies. The threshold f is set to 0.8.
We follow exactly the same configuration for experiments with the benchmark models. The experi- ments were conducted on a NVIDIA A100 GPU.
5.2 RE-BENCHMARKING
Table I: Quantitative results of our proposed mask-wise evaluation protocol. The symbol indi- cates using a joint-dataset vocabulary set during testing. NULL denotes non-out-matcher mask is predicted. Results of the conventional argmax pixel-wise approach are shown in the first four rows.
PC59
SAN
CVPR:23
$7.70
32.10
15:70
12.40
CAT-Seg
CVPR/24
63.30
37.90
23,80
16,00
SED
CVPR/24
60,90
FCCV134
frast
backf
crr
frcurt
back?
ctr.
Frout?
back?
err.
Front?
back?
en:
SAN
CVBRU22
0.67
92.46
8.41
Seg
93.66
5.20
SED
CVPR2224
66.29
94.21
6.43
44.90
93.50
5.20
31.41
70.72
4.93
26.99
92.61
5.07
MAFT
BCCY'24
64.95
93.57
9.10
46.51
93.10
7.31
31,89
70.82
7.12
28.72
92.15
7.84
CVIR222
10:9
Sag-
CVPR/24
66.35
92.24
2.19
50.04
92.68
2.30
11.56
67.32
2.00
12.83
91. 91.20
2.20
SED
CVPR:24
63.35
91.32
5.31
43.65
91.28
4.52
30.04
68. 40
3.23
27.45
90.05
4.10
MAPT+
ECCV/21
62.05
91.55
8.56
44.30
91.32
6.70
29.04
69.01
4.40
26.01
90.50
6.40
Here we compare the commonly used pixel-wise mloU metric that uses the argmax operation and our proposed mask-wise metric incorporating soft set prediction. The results are shown Ta- ble 1. Our observations are as follows: 1) Simply replacing the pixel-wise argmax-based mloU with the proposed mask-wise metric. front (target), leads to a performance improvement in existing OVS models. The OVS model achieves high accuracy (above 90%) for back (non-target) across all datasets, except for the PC459 dataset. 2) As the inference vocabulary increases (using the joint- dataset vocabulary set during testing), proposed evaluation method maintains relatively stable performance for both front and back, whereas the performance of the previous argmax-based pixel- wise evaluation method drops significantly, shown in Table 2. The performance gap here is caused by the ambiguous prediction while they have a high overlap in IoU with ground truth masks. Fur- thermore, with the increase the number of inference words, the model's 's error rate also increases.
6 DISCUSSION
6.1 ANALYSIS OF MASK-WIS EVALUATION PROTOCOL
To validate the effectiveness of the proposed mask-wise evaluation protocol, we present several key arguments that support its feasibility and practical utility, particularly in the open-world setting.
The effectiveness. Previous methods fail to quantitatively measure model performance in open- world settings because they usually rely category-level matching and evaluate only when the
7

---

Published as a conference paper at ICLR 2025
70
Argmaz Mask-wise
Table 2: The quantitative results of argmax (i.e. pixel-wise) evaluation using joint- dataset vocabulary set during testing.
60
50
Method
PC59
ADEI50
PC459
ADE847
SAN
40.15
22.50
10.50
3.20
20
CAT-Seg
42.90
25.60
12.30
7.00
10
SED
43.70
24.10
11.00
5.20
PC59 ADE150
PC459
ADE347
MAFT+
41.30
23.80
10.80
6.50
Figure 3: The stability of evaluation methods in comparison when evaluated with more vo- cabulary (i.e. in a more open setting).
predictions belong to predefined set of categories. This approach often draws incorrect conclusions when faced with new categories or synonyms (such "sofa" " vs. "long couch" "). The quantitative results of previous evaluation methods are compared with those of our method in Table. 3 where the vanilla represents the argmax- -ased pixel-wise evaluation.
Our proposed evaluation method effectively eliminates the impact of ambiguous category labels via mask matching, focusing on the mask overlap between the predictions and manual annotations. Even in the case of joint datasets, where different datasets use different vocabularies to describe the same or similar categories, our method still able evaluate through accurate mask overlap (e.g. IoU), avoiding evaluation instability caused by differences in category terms. This mask-based evaluation method allows our scheme to seamlessly adapt to the fusion of multiple datasets without unifying or standardising the category labels of each dataset. In open- spendworld settings, despite the limited number of manually annotated categories, the OVS model is still able to predict as many new concepts as possible, further demonstrating the applicability of the evaluation method.
The stability. Our method is eval- uated based mask-wise matching, which means that its main evaluation criterion the spatial overlap (such as IoU) between the predicted mask and the true annotated mask, rather than relying solely on the matching of class labels. When the number of vo- cabulary increases, traditional evalu- ation methods may experience large fluctuations due to the complexity of class label matching, especially when there synonyms inter-class am- biguities between different vocabu- lary sets. However, since our method only relies on matching the geometry and position of the mask, when the vocabulary set expands, the evalua- tion results will not be significantly affected by the expansion or change of class labels.
Table 3: The quantitative comparison with previous pro- posed evaluation methods, SG-IoU (Liu et al., 2024) and Open-IoU (Zhou al., 2023).
ADE150
Model
Venue
vanilla
SG-IoU
Open-IoU
Iront
SAN
CVPR/23
31.88
32.92
39.00
42.89
8.56
CAF-Seg
CVPR'24
35.68
36.75
39.90
45.74
5,53
CVPR'24
35,30
MAPT+
BCCV"24
36.10
37.08
46.51
7.31
PC459
Model
Venue
vanilla
SG-IoU
Open-IoU
front
CVPR/23
20.83
16.72
19:90
6.67
CAF-Seg
CVPR/24
22.25
17.91
20.30
30.95
3,86
SED
CVPR'24
22.10
18.22
31.41
4.93
MAPT+
BCCV"24
21.60
16.45
31.89
7.12
ADE847
Model
Venue
vanilla
SG-IoU
Opca-IoU
front
SAN
CVPR 23
13.07
14.17
19:20
22.84
8.41
CAT-Sog
CVPR
14.53
15.64
18:40
26.39
5.20
SED
CVPR'24
13.70
14.89
26.99
5.07
MAFT+
ECCV"24
15.10
Assume that we have a universal vO-
cabulary set Vopen, given a fixed testing dataset with ground truth categories and correspond- ing Ve vocabularies, suitable vocabulary set Viest for OVS testing on this dataset needs to sat- isfy: V3 C Vieet- If we set the scale of testing vocabulary set to N, there exists C' possible subset vocabulary Voub to satisfy Viece!
good testing vocabulary set for the given dataset, where We then can use the Monte-Carlo method to approximate the up- per bound and lower bound of test accuracy (Chen et al., 2019b) given various point estimations
C'
8

---

Published as a conference paper at ICLR 2025
for both the argmax with pixel-wise mIoU and our mask-wise evalua- tion as presented in Fig. 3. Specifically, the dataset-specific vocabulary set and joint vocabulary set are just two of many point estimations. We observe that the proposed mask-wise evaluation protocol maintains a stable measurement across different datasets compared to the commonly used argmax-based and pixel-wise mloU.
Our method is evaluated based on mask-wise matching, which means that its main evaluation cri- terion is the spatial overlap (such IoU) between the predicted mask and the true annotated mask, rather than relying solely on the matching of class labels. When the vocabulary increases, traditional evaluation methods may experience large fluctuations due the complexity of class label matching, especially when there are synonyms or inter-class ambiguities between different vocabulary sets. However, since our method only relies on matching the geometry and position of the mask, when the vocabulary set expands, the evaluation results will not be significantly affected by the expansion or change of class labels.
Assume that we have universal vocabulary set Vopen, given fixed testing dataset with k ground truth categories and corresponding VE vocabularies, a suitable vocabulary set Viest for OVS testing on this dataset needs to satisfy: Vx C Viest- If we set the scale of testing vocabulary set to N, there exists C possible subset vocabulary Voub to satisfy as a good testing vocabulary set for the given dataset, where:
Veeat!
C
(13)
(N k))(( - (N k))!
We then can use the Monte-Carlo method to approximate the upper bound and lower bound of test accuracy (Chen et al., 2019b) given various point estimations
for both the argmax with pixel-wise mloU and our mask-wise evaluation. Specifically, the dataset-specific vocabulary set and joint vocabulary set are just two of many point estimations. We can observe that the proposed mask-wise evaluation protocol maintains stable measurement across different datasets compared to the commonly used argmax-bases and mixel-wise mloU.
6.2 ANALYSIS OF AMBIGUOUS VOCABULARY GRAPII
Using community discovery methods, we partition the ambiguous graph into communities as shown in Fig. 4a. Each community represents a cluster of classes that are often confused with each other. For example, in object detection or segmentation dataset, might observe that the categories "sofa", "couch", and "armchair" form a tightly connected community, indicating that these classes are frequently misclassified or confused by the model.
This insight suggests that the dataset may contain ambiguous annotations where these objects are not clearly distinguishable where multiple terms used interchangeably in different regions or contexts. By visualising the labels of the same community, as shown in Fig. 4b, we find that
ThE day
bulding
capbbard (agalrue
Dree branch
table
courter
(a)
(b)
Figure 4: (a) A community extracted from the COCO-Stuff17 dataset (showing only 50 classes). (b) Example images from the same community, where images from the same vocabulary community exhibit visually similar semantics (best viewed in colour).

---

Published as a conference paper at ICLR 2025
Table 4: Quantitative results of reducing ambiguous vocabulary through non-target vocabulary re- moval using our proposed evaluation protocol. Specifically, proportion of non-target vocabulary discarded during training, with p defaulting to 0.9. The notations "SED w/ 0.7", "SED w/ 0.5", etc., denote results obtained with different values of p, where 0.7, 0.5, etc.
PCS9
ADEISO
PC459
ADE847
Method
frontt
backt
emvor
front?
back?
error
frontt
backt
error
froatt
back?
error]
MAFT+
-0.87
-0.23
-1.98
-1.56
-0.63
-2.30
-1.12
-0.08
-1.24
-1.81
-1.33
-1.96
SED
+2.03
+0.61
-2.15
+5.28
+0.46
-0.21
+0.50
-0.04
-1.57
+1.51
-0.62
-2.00
SED 0.7
+0.40
-1.80
+3.40
-0.02
-1.50
+1.00
SED 0.5
+1.40
+0.50
-1.70
+3.90
+0.40
-0.60
+0.35
-0.01
-1.40
+1.30
-0.40
-1.70
SED 0.3
+1.70
+0.55
-1.60
+4.20
+0.42
-0.50
+0.40
+0.01
-1.30
+1.40
-0.50
-1.60
SED w.1.1
+2.00
+0.60
-1.50
+4.50
+0.45
-0.40
+0.45
+0.02
-1.20
+1.50
-0.55
-1.50
these labels are extremely similar visually and difficult distinguish through subtle visual differ- ences. From a human perspective, these labels are likely to be classified as the same thing, showing extremely high similarity, which may indicate that they share some core features or attributes.
Additionally, the community discovery process helps reveal systematic biases in the dataset. For example, if certain categories (such as "sedan" "SUV" and "truck") are often clustered in the same community, this may indicate that the dataset lacks diversity in the representation of different vehicle types, or that the model inadequate in distinguishing between these categories. Details of the community discovery results for the datasets used in this paper can be found in Appendix A.2.
Experiments show that reducing ambiguous vocabulary helps the one-stage model focus on mean- ingful semantic distinctions and avoid distractions from subtle differences between similar cate- gories, as shown in Table 4. This non-target vocabulary removal strategy aligns the language like- lihood more closely with real-world distributions (Eq. 3), enhancing training efficiency. Since un- der real-world scenario, vocabulary distributions are inherently uncertain and often include out-of- distribution vocabularies absent from the training dataset. By dynamically perburbating the vocab- ulary set by randomly discarding non-target vocabularies during training, the model is exposed to diverse vocabulary subsets, mitigating reliance on a fixed vocabulary distribution. This perturbation promotes better generalization and adaptability to unseen word distributions, ultimately strengthen- ing the model's robustness in open- wooldw scenarios.
6.3 SUMMARY OF THE OBSERVATIONS
Based on the discussion above, summarise the key findings follows: Through the analysis of our ambiguous vocabulary graph, we identified the presence of numerous ambiguous or synonymous vocabularies in commonly used OVS datasets. 2) After conducting community discovery analysis on the ambiguous vocabulary graph established from the model's predictions, we found that the categories the model tends confuse often belong the same community, and their corresponding images are visually similar. 3) We further proposed to remove such ambiguous vocabularies dur- ing the training stage, by simply randomly discarding non-target vocabularies, and found it led to performance improvements for the OVS model.
CONCLUSION
In conclusion, in this paper, we introduced mask-wise evaluation protocol for vocablooayylary Segmentation (OVS) to address the issue of ambiguous vocabulary in evaluation. These ambiguities often arise under open-won world conditions, where multiple interpretations of labels can be valid. Our experiments validated the effectiveness of the proposed evaluation approach. Moreover, using our evaluation protocol, we can construct an ambiguous vocabulary graph for OVS models, revealing significant presence of confusing annotations in current OVS datasets. The experiments further showed that reducing such ambiguities can enhance the generalisation capability of OVS models, leading to improved performance. In addition, further discussion provided insights for follow-up research. We hope our study could encourage the community to think more about the openness of open-world problems and hopefully inspire research questions.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENTS
This project is partially supported by the Royal Society grants (SIF) 23029000 IES R3) (230550 and an Amazon Research Award. Qiming Huang is supported by the China Scholarship Council (Grant No. 202408060321). The computations in this research were performed using the Baskerville Tier 2 HPC service. Baskerville was funded by the EPSRC and UKRI through the World Class Labs scheme (EP) T022221 (1) and the Digital Research Infrastructure programme (EP) W032244 1) and operated by Advanced Research Computing at the University Birmingham.
REFERENCES
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1209- 1218, 2018.
Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, al. Hybrid task cascade for instance segmentation. In CVPR, 2019a.
Liang- Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):
834-848, 2017.
Wenhu Chen, Yu Su, Yilin Shen, Zhiyu Chen, Xifeng Yan, and William Wang. How large vocab- ulary does text classification need? a variational approach to vocabulary selection. arXiv preprint arXiv:1902.10339, 2019b.
20199.
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. arXiv:2112.01527 2021.
Seokju Cho, Heeseong Shin, Sunghwan Hong, Seungjun An, Seungjun Lee, Anurag Amnb,, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary se- mantic segmentation. arXiv preprint 21X11:2303.11797, 2023.
Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
11583-11 11592, 2022.
Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, al. Maskclip: Masked self-distillation advances trastive language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10995-11005, 2023.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European Conference Computer Vision, pp. 540-557. Springer, 2022.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yunchao Wei, and Humphrey Shi. Collab- orative vision-text representation optimizing for open-vocabulary segmentation. arXiv preprint #:Xiv:2408.00744 2024.
Ruochun Jin, Yong Dou, Yueqing Wang, and Xin Niu. Confusion graph: Detecting confusion communities in large scale image classification. In ICAI, pp. 1980-1986, 2017.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In CVPR, 2019a
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In CVPR, 2019b.
11

---

Published as a conference paper at ICLR 2025
Boyi Li, Kilian Q. Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. -angu semantic segmentation, 2022.
Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Guangliang Cheng, Pang Jiangmiao, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-based visual segmentation: A survey. arXiv pre-print, 2023.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. vocabulcyb semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
Tsung-Yl Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Zachary C Lipton, Charles Elkan, and Balakrishnan Naryanaswamy. Optimal thresholding of clas- sifiers to maximize fl measure. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part II 14, pp. 225-239- Springer, 2014.
Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In CVPR, 2018.
Yong Liu, Sule Bai, Guanbin Li. Yitong Wang, and Yansong Tang. bpen-vocabulary segmentation with semantic-assisted calibration. arXiv preprint arXiv:2312.04009, 2023.
Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang. Open-vocabulary segmentation with semantic-assisted calibration. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 3491-3500 2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic
segmentation. In CVPR, 2015.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whar Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic seg- mentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 891-898 2014.
Lu Qi, Yi Wang, Yukang Chen, Ying-Cong Chen, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Pointins: Point-based instance segmentation. TPAMI, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- image segmentation. In MICCAI, 2015.
Tiancheng Shen, Yuechen Zhang, Lu Qi. Jason Kuen, Xingyu Xie, Jianlong Wu, Zhe Lin, and Jiaya Jia. High quality segmentation for ultra high-resolution images. In CVPR, 2022.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, al. Deep high-resolution representation learning for visual recognition. In TPAMI. IEEE, 2020.
Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder-decoder for open-vocabulary semantic segmentation. arXiv preprint arXiv:231 1..::2331.155... 2023.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
12

---

Published as a conference paper at ICLR 2025
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2955-2966, 2023.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision, pp. 753. Springer. 2022.
Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. NeurIPS, 2021.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. CVPR, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
Hao Zhou, Tiancheng Shen, Xu Yang, Hai Huang, Xiangtai Li, Lu Qi, and Ming-Hsuan Yang. Re- thinking evaluation metries of open-vocabulary segmentaion. arXiv preprint arXiv:2311.0335 2023.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 PSEUDOCODE OF THE MASK- MASK-WISE EVALUATION PROTOCOL
Algorithm Mask- Wise Evaluation Protocol
Require: Annotated masks list {M. J, predicted masks list (B(r),) threshold range
(0.1,0.2..... 0.9), and threshold TAY for ambiguous vocabulary matrix.
Ensure: Results for each threshold CM,, EM,, AV,
1: Initialize results dictionary Results <- 0
2: for 7 (0.1, 0.2. 0.8, 0.9} do
Iterate over all threshold values
3: Initialize confusion matrix CM, <-
4: Initialize ambiguous vocabulary matrix AV,
5: Initialize error matrix EM, f-
6:
for D do
Iterate over all images in the dataset
7:
Obtain binary predicted masks B(T); for с E Caue do
8:
Iterate over true categories
9:
Compute binary confusion matrix for M and B Update confusion matrix CM, with computed values end for
10:
11:
12:
Remove masks not in Ctile and zero masks from predictions:
( rem (B(T),c c Crue B(T),, * 0}
13:
Solve bipartite graph matching between B(T), Jem and M
Matches + {(b, m) b E B(T);" m E M IoU m) > TAV
for each matched pair (b, m) E Matches do
15:
Update AV r,, AV rrmmb + +1
16:
Remove matched mask b from B(T): tem end for
17:
18:
Let B(T), rem be the remaining masks after removal rem for E Categories in B(T); do Compute error err,,e for image i
19:
Iterate over remaining categories
21:
Update error matrix EM, + EM, + erri, end for
23:
end for
24: Store results for current T:
Results[7] + (CM,, AV, }M,)
end for 26: return Results
25:
14

---

Published as a conference paper at ICLR 2025
A.2
THE COMMUNITY DISCOVERY RESULTS ACROSS THE DATASETS.
COCO-Stuf171
(1' elouass, 'oog', 'sky-other', ,, (' bbuilcing-other',
'curtain',
'house',
'skyscraper',
'wall-brick',
'wall-cencrete',
'wall-other',
'wall-panel',
'wall-stone',
'wall-tile',
'wall-wood',
'window-bl ind', 'wincow-other',
'wood'),
'carpet',
'dirt',
'floor-mablee'
*floor-other',
'floor-stcne',
'flcor-tile'
'floor-wood',
'gravel',
ground-other''
pavement',
'platform',
'playingfield',
'road',
trug',
"sand',,
river', 'sea', 'waterrother'' ,
bbrnch',
'bush',
'flower',
'grass',
'hill',
'mountain',
"plant-other',
"pottea plant', 'straw',
'tree' ('cabinet', ' 'cupboard', 'shelf'), h, ('bus', 'car', 'train', 'truck'), ('rock', 'stone'),
'cage' 'fence', 'railing', 'structural-other" ) counter', ddesk-stuf'', 'iinigg table', taalee' h, ''loth', 'textile-other');
(' 'elothes', 'person'), ''urniture-other' 'metal', 'plastic', ssoo sign' h
['backpack', 'handbag'), ), ('cup', 'wine glass'), ('ceilingcother' 'roof'), ) ('hot dog', bandwich']
PC59
(1' bench'
'bildiing'
cabinet'
'ceiling',
'chair'
15

---

Published as a conference paper at ICLR 2025
'eurtain',
diningtable',
door', 'ence', floor',
'flower',
'grass',
"gronn',,
mountain',
platiorm',
potteeplant',
rooad',
rock', shelves',
'sidewalk',
'track',
'trec',
'wall',
'window',
'wood'),
('bus', 'car', 'truck'),
('bag', 'bed', 'bedclothes', 'clcth', dog' sofaa h, ('boat', 'water' I, computer', 'tvmonitor') 1' 'biccycle, 'motorbike') ))
ADE150
(1' l'awning',
'blind',
'booth'
'bilddin',,
"celling'
"handeller',
currain',
door',
"ence',
"hill',
hoouse',
'1amp',
'light',
'mountai',
railing',
rock', sconce' screen door', "skyscraper',
'tower' 'wall', 'windowpane') ),
''anopy', 'dirt track', 'earth',
'field',
'floor'
'flowr'
'grass'
"lnnd',
ppah''
plant',
road',
rug',
runway',
sann''
ssidewalk"
16

---

Published as a conference paper at ICLR 2025
'tre'',, (' eushion'', 'pillow",
('car', 'truck', 'van') ,, Yarmchair', 'chair' 'scat', 'scfa', 'sool'', 'swivll chair' ) 'bookcass',
cabinet', chest of drawers', coffee table', counter', countertop', 'dsk' kitchen island', pool table', 'shelf',
'table',, ('oven', 'stove',,
('lako', 'river', soa', water'', ('ort screen', 'monitor', televisioo receiver' 1,
('ashcan', 'pot', 'vase' ('stairs', 'stairway'), ('poster', 'signboard'))
PC459
((brick''
'biidge',
builling',
cabinet',
cabinetdoor',
cage', ceiling',
closet',
concrete',
counter',
door',
fence',
floor',
"ootbrrdge',
"ground',
handrail',
mat'
'metal',
"patio',
"platform',
"polc',
road',
rug',
sand',
shed',
'shelves',
'sidewalk',
'sign',
'sky',
'table',
'tableware',
'unknown',
'wall'),
('car', 'tcycar', 'truck') ,,
('light', "lightbulb'), ,
''ag',
beaclothes',
'chirr,
'oooth'
relothestree',
cushi on',
17

---

Published as a conference paper at ICLR 2025
fflower'
"raas'', "leaves',
mountain',
pack'', 'pillow', 'plant', "plastic', 'pot', ppotteapaant,,
'sofa',
'stool'
'straw',
'twell'
'tree',
'wood' ,,
(' curtain', 'window', 'windowblinds') ,
''rock', 'stone'),
d'dolphi'', 'water', 'wharf'),
(' ('box', 'paperbox'), l'bicycle', 'tricycle'),
(' l'beer', 'bottle', 'oxygenbottle"), ,
'person', 'player' l'cofee'' 'cup', 'glass'), l'sereen', 'tvmonitor', vvideggameconsole', 'videoplayer' ), ('picture', 'poster'),
('bird', ",uck'),
('rail', 'track'), ('dog', "fo'',, ('book', 'paper' })
ADE847
( baseboard',
central reservation', curb',
floor',
"oooppah',
'att',
'path',
'road',
'rug',
sidewalk',
'skirting board'), ,,
'balcony',
'building'. 'builaing materials',
'cabin', ffirst flcor',
house''
pane',
porch',
shop',
shops',
'skyscraper',
streee number',
'windowpane') },
(' 'over curtain', ' 'curtai' ,, "flower',
'forest',
'plant',
"plant pots', 'pot',
'tree',
'trunk'
18

---

Published as a conference paper at ICLR 2025
'aase' 'wedd'
"bed', 'beds', 'eiderdown'; 1, d'door', door bars', 'doorfrmm',, double door' ), ''buffe'',
cabinet',
chest of drawers', coffee table', 'dekk',
'table',
'table cloth',
'tables',
'television stand'), ),
I'booth',
'brick',
'hill', mountain',
mountain pass',
rock', rocky formation',
shower rocm', 'temple',
'wall'),
apparee', 'dumny', 'person', 'trouser') !'car', 'truck', 'van'), ,
d'cushion' 'pillow'),
('earth', field', 'grass', 'land', 'sand'), ) ('counter', 'countertop', work surface'), ('armchair', 'chair' rrocking chair', 'seat', 'stool', 'swivel chair' ('fireplace', 'fireplace utensils'), ('lake', 'river', 'sea', 'sore', 'water'), , l'awning', 'blind'),
('sofa', 'sofa bed'),
('light', 'light bulb'), ), ('lamp', 'sconce') , ('screen', 'television receiver'), ('ceiling', 'eaves', rrof'', ('cooker', 'stove'), ('clock', 'watch'),
I'games table', 'pool table' ,,
l'ashcan', 'recycling bin'), 'barrier', 'fence', 'railing' ,, ('stairs', 'step' }
19


Reviews:
[{"review_id": "CJ50uLvx1k", "reviewer": "Reviewer_eu9j", "summary": "This study proposes new evaluation metrics for Open-Vocabulary Segmentation (OVS) tasks. A key limitation of evaluating OVS methods on fixed-category datasets is that traditional image segmentation metrics may misclassify visually similar objects as errors, even when they are semantically related but belong to different categories. This issue intensifies with an increasing number of category labels in the test dataset. This issue becomes more pronounced as the number of category labels in the test data increases. Previous research has addressed this challenge, resulting in improved metrics such as Open-mIoU and SG-IOU. The central premise of this work is to focus evaluation on mask similarity rather than textual similarity.", "questions": "Q1: The analysis in Section 3 appears disconnected from subsequent sections.\n\nQ2: In Figure 2, $\\mathbb{A}$ represents a set of predicted binary masks. How are the predicted masks in $\\mathbb{B}$ and $\\mathbb{C}$ derived from $\\mathbb{A}$? If they are matched to GT masks based on IoU using bipartite matching, it seems Figure 2 suggests that the number of predicted masks by the model exceeds that of the ground truth, which is not realistic. Additionally, predicted masks in $\\mathbb{B}$ and $\\mathbb{C}$ should not overlap according to $\\mathbb{C} = \\mathbb{A} \\backslash \\mathbb{B}$.\n\nQ3: The correlation between Algorithm 1 and Section 4 is weak: For example, (1) The CM is not referenced outside the Algorthm 1. (2) The calculations for the core evaluation metrics -- front, back, and errors -- are not represented in Algorithm 1 or any other equations. (3) How is the best threshold $\\tau^*$ used in Algorithm 1? \n\nQ4: What constitutes a good evaluation metric? The last sentence of the introduction (line 83 on page 2) implies that the authors equate higher performance values with better evaluation metrics, which is unreasonable. \nIn Figure 3, the authors seem to suggest that more stable evaluation metrics are preferable; however, this should also be compared with other metrics like Open-mIoU and SG-IoU.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The primary contention of this manuscript is to shift the focus of evaluation from textual to mask similarity in assessing OVS models. The authors have identified a gap in the current assessment metrics, which are deemed inadequate for evaluating OVS models, and have proposed a novel metric to address this issue.", "weaknesses": "The manuscript exhibits a lack of clarity and organization in its writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xGtKgwSHe0", "reviewer": "Reviewer_tdt4", "summary": "The performance of Open Vocabulary Segmentation (OVS) models will decrease as the query vocabulary size increases, especially when semantically similar category names are present, contradicting the original purpose of OVS. To address this, the authors proposed a mask-wise evaluation protocol based on match/mismatch between prediction and annotation mask pairs, avoiding forced category matching. Key innovations include reducing ambiguity and constructing an ambiguous vocabulary graph. Comprehensive experiments and analysis reveal numerous ambiguous categories in current OVS datasets. Utilizing the proposed protocols during the training and testing stages can help to improve the model’s zero-shot inference capability.", "questions": "1. A significant concern is that the proposed evaluation protocol relies on having sufficient data to identify semantically similar categories. In real-world applications, if the training data lacks adequate masks to differentiate similar categories (e.g., \"sofa\" and \"couch\"), the protocol may struggle during testing. To address this, it would be helpful if the authors could analyze the performance of their method with limited training data or provide insights into the minimum data requirements necessary for effective improvement. Additionally, experiments or discussions on the robustness of data scarcity and the impact of potentially misleading information would strengthen the evaluation.\n\n\n2. While the authors' approach to handling ambiguities through the visual modality is quite interesting, it may be more intuitive to identify similar categories based purely on semantic meaning. For instance, using the text modality to assess semantic similarities could potentially provide greater improvements than relying solely on visual information. To explore this, it would be valuable for the authors to compare their visual-based approach with a text-based semantic similarity approach. Or add more discussions about the potential advantages and disadvantages of incorporating textual semantic information into their method.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Good motivation, authors pointed out the current OVS evaluation sets have many semantic similar categories, which may influence the training&testing stages of model, which further influence the inference ability of current OVS methods. Based on this, authors proposed a new evaluation protocols to alleviate this issue.\n\n2. The whole paper is relatively clear and easy to follow.  \n\n3. Very comprehensive experiment results on multiple datasets and multiple OVS methods.", "weaknesses": "Writing Suggestions: \n\n1. In the Abstract, authors claim that OVS models perform better under the new mask-wise protocol needs further clarification. To make fair comparisons between the mask-wise and pixel-wise protocols, the authors should add more details about how they determine \"better\" performance. Providing such details would help readers understand the basis for this improvement claim.\n\n2. In the Abstract, the phrase “enhances zero-shot inference capabilities” likely refers to the capabilities of OVS models. Clarifying this would improve readability. \n\n3. Given the similarity between open-vocabulary segmentation and open-vocabulary semantic segmentation, the authors should add a brief section comparing these two concepts. Highlighting key differences in their applications or objectives would help avoid potential confusion and clarify the unique focus of their work.\n\n4. For Equation (5), the authors should provide more detailed motivation for choosing this to determine the best threshold, rather than simply listing the source. It would be helpful if they could explain why this method was selected over alternative approaches and how it specifically benefits their evaluation protocol.\n\n5. The equation at lines 324 to 327 is missing a number.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5l8sBE0lWh", "reviewer": "Reviewer_TnMo", "summary": "This paper gives a deep observations on open-vocabulary semantic segmentation. To address the ambiguous category issue, the authors propose mask-wise evaluation protocol and a confusion vocabulary graph for open-vocabulary datasets. The experiments validate method defectiveness.", "questions": "Please refer to weakness.  It is important to give more experiments for ambiguous vocabulary graph and more comparsion.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents an interesting analysis on the openness of open-vocabulary semantic segmentation.\n\n2. The mask-wise evaluation protocol sounds reasonable.\n\n3. The experiments are conducted on multiple existing methods.", "weaknesses": "1. The quality of ambiguous vocabulary graph seems important for performance. Currently, the related experiments are not enough. I think it is better to provide more experiments to verify the quality of ambiguous vocabulary graph.\n\n2. The accuracy for front and back is not very clear. I suggest that the authors give an equation to explain it.\n\n3. The comparison of whether reducing ambiguities during training or not is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)