Renovating Names in Open-Vocabulary Segmentation Benchmarks
Authors: Haiwen Huang
Songyou Peng
Dan Zhang
Andreas Geiger
Venue: NeurIPS 2024 poster (2024)
Keywords: vision-language datasets
open-vocabulary segmentation
renaming
Primary area: machine_vision
TL;DR: We address the naming issues in open-vocabulary segmentation benchmarks and demonstrate that renaming improves both model training and evaluation.

Abstract: Names are essential to both human cognition and vision-language models. Open-vocabulary models utilize class names as text prompts to generalize to categories unseen during training. However, the precision of these names is often overlooked in existing datasets. In this paper, we address this underexplored problem by presenting a framework for "renovating" names in open-vocabulary segmentation benchmarks (RENOVATE). Our framework features a renaming model that enhances the quality of names for each visual segment. Through experiments, we demonstrate that our renovated names help train stronger open-vocabulary models with up to 15% relative improvement and significantly enhance training efficiency with improved data quality. We also show that our renovated names improve evaluation by better measuring misclassification and enabling fine-grained model analysis. We provide our code and relabelings for several popular segmentation datasets to the research community on our project page: https://andrehuang.github.io/renovate.

Renovating Names in Open- Veen-Voabbulayy Segmentation Benchmarks
Haiwen Huang 1223 Songyou 3..5 Dan Zhang
Andreas Geiger Bossh loC Lab, University of Tübingen Tubingen Al Center
3 Autonomous Vision Group, University of Tübingen
4 ETH Zurich
MPT for Intelligent Systems, Tübingen
Bosch Center for Artificial Intelligence
Abstract
Names essential to both human cognition and vision-language models. Open- vocabulary models utilize class names as text prompts to generalize to categories unseen during training. However, the precision of these names is often overlooked in existing datasets. In this paper, we address this underexplored problem by presenting framework for "renovating" names in open-vocabulary segmenta- tion benchmarks (RENOVATE). Our framework features a renaming model that enhances the quality of names for each visual segment. Through experiments, we demonstrate that our renovated names help train stronger open-vocabulary models with up 15% relative improvement and significantly enhance train- ing efficiency with improved data quality. We also show that our renovated names improve evaluation by better measuring misclassification and enabling fine-grat fine-grained model analysis. We provide our code and relabelings for several popular segmentation datasets to the research community on our project page: https: /aap:://///deuangg gi thub. io/renovate/.
Introduction
Categorizations which humans make of the concrete world arbitrary but highly determined. taxonomies of concrete objects, there one level abstraction which the most basic category cuts are made.
Eleanor Rosch, Baxic Objects in Natural Categories [1]
We names every day. Imagine wandering the trails of national park do you pause by "lake"
or simply a body of "water"? While driving through urban sprawl, do you see "trees" or "vegetation" along the road? Our instinctual use of terms like "lake" and "trees" illustrates the human propen- sity to categorize the world around us into "basic categories" - rich in information and readily identifiable without unnecessary complexity [1, 3]. Such categorization fundamental human cognition and communication.
In stark contrast, recent advancements in open vocabulayy models [4, 5, 6, 7, 9, 10, 11, 12] strug- gle to replicate this nuanced aspect human categorization. While these models have made strides in generalizing both familiar and novel categories via textual prompts, they often hampered by the imprecise and sometimes wrong names provided existing benchmarks. In fact, most datasets merely as identifiers to distinguish classes within dataset,
labeled with class names that rather than descriptive labels aligning with the "basic categories" that match with the actual visual contents. As shown in Fig. 1, existing names are often inaccurate, context, leading to discrepancies between the model's outputs and the actual visual segments. This misalignment indicates pressing need for reassessment and refinement of name labeling prac-
general, lack enough
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

tices. By adopting a naming scheme that aligns more closely with human categorization, we can pave the way for enhanced open- vecabooayyl generalization and more accurate model evaluation.
In this work, we focus on renovating the names for open-vocabulary segmentation benchmarks, as their dense annotations pose a greater chal- lenge and offer broader applicability than other recognition tasks. The importance of names in open-vocabulary segmentation is often over- looked, with one exception, OpenSeg [5], which manually inspected and modified the class names of several segmentation bench- marks, resulting considerable performance gains of their proposed model. While their work demonstrated the importance of choosing good names, manual approach is subjective and difficult to scale. To date, there exists no systematic study on how rename benchmarks in a scalable and principled way. Our work is the first attempt to tackle this challenge.
Ground-truth Segmentation
Original Name Segmentation
Renovated Name Segmentation
tram
train
monitor
(a) Original names can be inaccurate.
field
crop field
field
field
(b) Original names can he too general.
In this work, we introduce scalable, simple, yet general method for renaming segmentation benchmarks that outperforms manual label ef- forts like OpenSeg [4]. Our approach lever- ages foundation models to automate the renam- ing process, significantly reducing the man- ual labor traditionally required. Towards this goal, our method can be divided into two steps. First, we narrow down the name search space from the whole language space to curated list of candidate names. This list is gener- ated by leveraging the original class names with contextually relevant nouns extracted from vi- sual contents via image captioning model, thereby streamlining the search for names. Next, we employ a specially trained renaming model to identify and select the best-matching candidate name for every ground-truth segmentation mask. In this way. we match an individual name for each instance without any extra human annotations.
fireplace
Indpor
fireplace
fireplace
outdoor
fireplace
(c) Original names can lack contexts.
Figure Problems of names current seg- mentation benchmarks. We demonstrate exam- ples from well-known datasets: MS COCO [13], ADE20K [14], and Cityscapes [15]. Our reno- vated names are visually more aligned and help models to generalize better.
To demonstrate the value of RENOVATE names, explore two practical applications. We first use our renovated names for training open- vocabulary models and show that they substantially enhance generalization performance across various benchmarks by up to 15%, indicating alignment with visual segments. Our new names also significantly increase training efficiency thanks to their enriched textual information. Second, apply renovated names improve evaluation of open- ooca-vocabulary segmentation. Specifically, we reveal that current pre-trained models are making many "benign" misclassifications. i.e.. misclassified names that are semantically close the ground- truth names. RENOVATE names enable more fine-grained model performance analysis, providing valuable information for further model enhancement and dataset curation.
accurate
In summary, our work makes the following contributions:
We point out the naming problem in existing open-vocabulary segmentation benchmarks.
We propose a scalable, simple, and general framework for automatic dataset renaming.
We demonstrate that our RENOVATE names help to train models with stronger open- vocabulayy capabilities and refine evaluation benchmarks, providing valuable insights for future research.
Related Work
Open- Vocabulary Segmentation Methods. Prior work open-vocabulary segmentation [5, 6, 7, 8, 16, 17, 18, 19, 20) focused on adapting vision-language models (VLMs) [21, 22, 23] to the seg-

---

Original name
Context names
Candidate names
lush, field, sky, green.
rural field, roadside field, green field, crop field, sports field, grassland, grassy hillside
field
grass, tree, road,
hillside, grassy, rural
room, stool, chair, floor, market, table, food, lamp, paper
storage box, packing box, file box, box container decorative box, display box, paper box, food box
box
room, infant bed, nursery, dresser. carpet, bedroom, chair, bureau, lamp, armehair
bedroom cradle, cradle, infant bcd, nursery cradle, baby cradle, infant cradie, wooden cradie
cradle
Table 1: Examples of context names and generated candidate names for three selected classes from ADE20K. Context names are key to comprehending general terms such as "field" and "box" and disambiguating polysemous terms like "cradle", which, in this context, refers to baby bed rather than a phone cradle or a mining tool.
mentation task without compromising pre-trained vision-langauge alignment. These efforts have concentrated mainly on the architecture design, exploring different backbones [24, 25], mask pro- posal branch designs [19, 5]. and mask feature extraction techniques [7, 26, 27]. Our work differs from them in that we aim to adapt pre-trained vision-language representations for segment-wiss name matching through our novel designs like candidate name generation techniques and our spe-
cialized renaming model.
Names in Open- Vocabulary Segmentation Benchmarks. Despite its apparent importance, the naming problem in open-vocabulary segmentation benchmarks has not received adequate atten- tion. An exception is the work by [28], which proposes to learn class-specific word embeddings for improved model adaptation to new datasets. However, these learned embeddings are not in the language space and cannot be used by other models. In addition, some works [29, 30] propose to decompose the class names into a set of attributes or prototypes for open-vocabulary segmentation. To the best of our knowledge, OpenSeg [5] is the only work that proposes to directly improve the quality of the names. However, their approach relies on manual inspection of the datasets, which can be subjective and hard to scale. In this paper, we systematically explore the renaming challenge, culminating in scalable, simple, yet general framework to overcome these limitations.
RENOVATE: Renaming Segmentation Benchmarks
Our method RENOVATE aims to rename the original names given dataset with ground-truth mask annotations. As shown in Fig. 2, we first generate pool of candidate names, covering poten- tial variants of the original class names. Then, we train a specially designed renaming model that measure the alignment between segments in pixel space and names in language space. Equipped with the renaming model, we select the top-matching names from the candidate pool for each seg- ment, thereby producing segment-level, refined names that offer enhanced accuracy and details in characterizing the dataset, see Fig. and Fig. 4.
3.1 Generating candidate names
We use GPT-4 [31] for creating pool of candidate names. A naive solution is to prompt GPT-4 with the original name and ask to generate synonyms and hierarchical concepts. However, since the original names are often too general and ambiguous, GPT-4 does not have sufficient knowledge to generate high-quality candidates. Therefore, we propose to exploit the visual contents for generating some "context names" (Table 1) that assist GPT-4 in comprehending the category's meaning prior to generating candidate names.
As shown in Fig. 2, for each category, we use an image captioning method to process all training images that contain that specific category based on ground-truth annotations. From the generated captions, we further extract nouns by text parsing and filtering as done in CaSED [32]. We observe that nouns appearing more frequently tend to offer deeper insights into the typical environments or traits associated with the category. Therefore, we construct context names by selecting the top 10 most recurrent nouns for each category. We then use them additional inputs alongside the original class names to prompt GPT-4.
3

---

Original class names
Candidate Names
bicycle
GPT-4
Generate candidate names
class c city blcycle,
road bleycle, tandem bike, cruiser, pavement bike, recreational bike,
with class
Caption
Context names
tree, pavement, hedge, mountain bike, bicycle rack sidewalk, building. saddlihaag
Aggregate and Sort
building, person, park,
Ca Model
pavement, car, walk, city street, bicycle, street corner
pedestrian bicycle
Generating Candidate Names
Candidate
Ground masks.
dass
city bikee
text quaMies
output queries
Transformer decader
bicydl
Linear
class Ak
Training
road bicycle
CLIP text
CA
SA
FFN
mask
= Cotan + Cowe
Nxt
NNC
encoder
pedestrian
muliinzale feature.
Pixel
vision encodar
perpiiel features HxWXC
Figure 2: Overview of candidate name generation and renaming model training. We generate candidate names based on the context names and train the renaming model to match them with the segments. For illustration clarity, show only segment. In practice, multiple segments jointly trained, pairing with the text queries.
Specifically, we instruct GPT-4 to generate two types candidate names: (1) Synonyms or subcate- gories, which can expand the lexical scope of the overly general class name. For instance, the class name "grass" could yield variations like "lawn" or "turf", For this type, synonyms and subcategories considered together, acknowledging their often interchangeable nature. (2) Providing short, dis- tinct contexts to polysemous names. As an example, the ambiguous name "fan" could be elaborated into "ceiling fan" or "floor fan", offering clearer specificity. As shown in Table the 5-10 can- didate names generated by GPT-4 are more precise than the original names, reflecting the context information encoded by context names. We provide detailed GPT-4 prompts in the supplement.
3.2 Training for candidate name selection
Among the generated candidates, we will only keep those that are better aligned with the visual contents. Particularly, we train a model that assess the alignment between each name and seg- ment, using segmentation as the proxy task. Conceptually, candidate name that well describes the segment in the image should help the model recover its ground-truth mask and make correct classification. Therefore, the model should allow vision-language interaction, and the supervision signal should encourage use textual information for mask prediction and classification.
Our renaming model is illustrated Fig. 2. Its vision part starts from a CLIP-based vision backbone followed by pixel decoder that gradually upscales the backbone features generate per-pixel embeddings Fpix E Mxxxx xc The candidate names the categories present in the image are processed by the CLIP text encoder. The interaction of textual and visual information takes place the transformer decoder. The initial queries X are from the text embeddings, and then updated by the transformer decoder consisting of L blocks using the multi-scale visual features from the pixel decoder. As output of each block, the queries (!) generate intermediate mask and class predictions:
M ()) sigmoid((()) Fpix),
a00) softmax Linear (Linear(x()))
where * refers to pixel-wise multiplication. The final predictions are made by the queries (L), Our pixel decoder, mask prediction and classification follow the design of Mask2Former [33], which handles panoptic, instance, semantic segmentation in a unified manner. The key differences lie in 1) the transformer decoder design, and 2) the training strategy, which detail subsequently.

---

Candidate Narne
Image
person, people,
30%
child
Proportion (9%)
Select with highest 1otspt di)
Renaming Model
baby
Ground-cruth
106
676
persicn
(b) Renovated name distribution within the original "person" class.
(a) Process of obtaining renovated names.
Figure 3: Obtaining renovated names. In (a) we illustrate how we use the renaming model to obtain a renovated name for each segment. In (b) we demonstrate that the renaming results are helpful to dataset analysis with examples from "person" class.
Transformer Decoder. Our transformer decoder uses text embeddings as input queries, unlike most open-vocabulary segmentation models that employ text embeddings solely as weights of the Linear layer in Eq (1) for classification. This key change fosters an earlier integration of text and visual features at varying scales, essential for leveraging text information segmentation tasks. As shown in Fig. 2, the text embedding of each candidate name from the segment's class "bicycle" feeds into the transformer decoder, generating their corresponding mask and class predictions.
Next, within each block of the transformer decoder, the masked cross-attention (CA) layer adopts ground-truth masks as the attention biases, and the self-attention (SA) layer [34] and ferw-fo network (FFN) are the same as in Fask2Former. The queries interact with the visual features at the masked CA layers, which attend within the region defined by the ground-truth mask, i.e., the attention bias. In the example of Fig, 2, the visual features corresponding to the segment "bicycle" can therefore contribute more effectively to refining the query. To encourage the model to rely more on the textual information in prediction, we further randomly replace the ground-truth masks with predicted masks from the preceding block in the intermediate layers. Importantly, while ground- truth mask based attention biases augment the V-L interaction by focusing the precise segment regions, they do not reduce the task to a straightforward use of ground-truthh mask inputs for mask prediction, as the output queries remain a set vectors. The final mask prediction M() is based on the correlation of these output queries (L) with the feature map Fpx generated by the pixel decoder in Eq. (1).
Training Strategy. With the CLIP backbone kept frozen, we train the transformer decoder together with the pixel decoder. To recover the ground-truth mask M, and class c. the transformer decoder makes multiple mask and class predictions with candidate names from the class c₁. The prediction with the highest Intersection over Union (IoU) with M, is selected for loss computation:
C. = Lmakk (M;, Mbest. + Celass (Ci, Goest),
where Lmask and Celass the mask localization and classification loss functions in Mask2Former. To provide extra supervision on the name quality, we append the candidate names with "negative" name randomly selected from different category than the ground-truth class C₁. If a "negative" name scores the highest IoU, we supervise the predictions with an empty mask and the "void" class (Cvota) which one extra class in addition to the training classes:
(3) The "void" class was also used in Mask2Former classification when the prediction mask is not matched any ground-truth mask in the image. Having both the positive and negative supervision, we effectively incentivize the model to favor names of high quality and penalize those of low quality, thereby aiding in the accurate identification of the best-matee mesthmatching names for each segment.
C, Lmusk (0. Mbest) Eeclus(Cvotd Ōbest)-
3.3 Obtaining renovated names
As illustrated in Fig. 3a, to obtain the renovated names, we run the trained renaming model asso- ciate each ground-truth mask with the candidate name. For each segment, renaming model first
5

---

lawn tractor
grass
SnOw ground
grass
car
earth, ground
earth, ground
terrain
engineer
boy
trees
chicken
person
person
food
vegetation
bread
mountain
lake
stream
pool
woman
person
food
water
water
water
Figure 4: Examples of renovated names segments from the validation sets of ADE20K and Cityscapes. For each segment, we show the original name below the image and the renovated name in the text box. See more visual results in the supplements.
ranks the names by the IoU scores between the predicted masks and the ground-trute mask, then selects the top-scoring name as the renovated name. While the IoU ranking scores are not proba- bilities, they are valuable to understand the relative and absolute model confidence in the names, i.e. how the chosen name stands against the other candidates and whether the selection is indeed confident. In the experiments (Section 5.3), make use of this information to speed up human verification. We show some examples of renovated names in Fig. 4 and more the supplements.
Our renovated names are readily usable for analyzing the dataset distribution and biases in a more fine-grained manner, as exemplified in Fig. 3b. Note that our renovated names in each original class are not necessarily mutually exclusive because different visual segments may be matched to names at different hierarchies. For example, in Fig. a person named as "engineer" (second row), likely due to the clothing, while the other person is named as "woman" (third row), possibly from the physical appearance. But "engineer" and "woman" are not mutually exclusive names. An interesting future extension would be to further study the hierarchies of different names in a dataset.
Applications of RENOVATE Names
We further demonstrate the value of the renovated names with two application tasks: training stronger open- vocabulaay segmentation models and improving existing evaluation benchmarks. The improvements in both tasks indicate the quality RENOVATE names.
4.1 Training with RENOVATE names
RENOVATE names offer precise and diverse text information. We exploit this to improve model training and achieve better generalization and data efficiency. Following prior works [24, 25, 35], we replace original ground-truth names with RENOVATE names to compute text embeddings. In addition, we employ the "negative sampling" technique, commonly used in natural language processing to enhance training with large number of classes [36, 37, 38, 39]. For each seg- ment, we first randomly select each. These names then concatenated with the ground-truth RENOVATE name to form the text prompt. This effectively constructs text embeddings of length C' with semantically distinct names, improving training efficiency and generalization. For computing the classification loss, use the cross entropy function on a per-segment basis, i.e., for an image with segments {m, | Letass EN CE(c,, softmax(1;)) (T) where &; is the ground-truth label, E Rd is the predicted
1 negative classes and choose one RENOVATE name from

---

visual embeddings and T; E RCxd is the constructed text embeddings through negative sampling. Throughout our experiments, we use negative sampling in our model training unless specified.
4.2 Improving evaluation with RENOVATE names
RENOVATE names also enable more fine-grained analysis of open-vocabulary models. For instance, for a misclassification from "car" to "van", we may distinguish between "SUV" to "minivan" versus "sedan" to "delivery van". which isn't possible with class-level annotations in current datasets. How- ever, even with RENOVATE names, standard segmentation metrics obscure our desired fine-grain ane-grained performance insights since they penalize all such misclassification cases equally. To this end, adopt "open" evaluation metrics [40] that consider semantic similarity between names. Specifically, for a groundtruth mask pixel 9i with label e; and predicted mask pixel dj with predicted label Cj. we compute soft TP. = Seu.ey. soft Pe = Sa.ey, and soft FNe, = 1 Se..ey where Se,c, is the semantic similarity between C and Cj- These soft metrics are then used to compute panoptic quality (PQ), Average Precision (AP), and mean IoU (mloU) in a standard way. In Sec- tion 5.3, we demonstrate how open metrics effectively reveal that there are indeed many "benign" misclassification scenarios using RENOVATE names.
While renaming process is automated, it is essential to ensure exceptionally high-quality names for evaluation. Therefore, we introduce a rigorous verification process involving five human annota- tors who review the top three name suggestions and their confidence scores from our model to select the best match. If none of the top three names are suitable, verifiers examine the entire candidate list. In case of disagreements, verifiers discuss and decide collectively. Note that human verifica- tion is used only for evaluation sets, while automatically generated names are used for the much larger training sets. Future work could explore using multiple vision-language foundation models to automate this verification process as evaluation sets grow [41, 42, 43).
5
Experiments
5.1 Obtaining renovated names
Setup. We renovate three panoptic segmentation datasets respectively: MS COCO [13],
ADE20K [14], and Cityscapes [15]. To generate context names, our default image captioning model is CaSED [32], which retrieves captions by matching vision embeddings with text embeddings of captions from a large-scale PMD dataset [44]. We train a renaming model for 60k iterations with a
batch size of 16 on the training set, then generate RENOVATE names for the entire dataset.
Results. Table summarizes the statistics of RENOVATE names. Compared to the original class names, each dataset has approximately 5- 6 times more distinct RENOVATE names, indi- cating that they provide more diverse and fine- grained semantic annotations to the visual seg- ments. We show some visual examples in Fig. and more in Appendix E. To further validate the quality of the names, we conduct a human preference study, see Appendix B. We also conduct an ablation analysis on the components of the renaming pipeline, see Appendix B.2.
COCO
ADE
CS
# Original classes
133
150
19
# Segments/Class
11,274
2,009
4,964
# RENOVATE names
741
578
108
# Segments/Name
1.871
521
873
Table 2: Statistics of renovated datasets.
5.2 Training with renovated names
Setup. We train FC-CLIP [25] models on MS COCO with our renovated segment-matched names and compare them with models trained using other name sets on COCO, ADE20K, and Cityscapes. Specifically, "OpenSeg" names [5] were manually curated by previous researchers. "Synonym names" are generated by prompting GPT-4 based on the original class names. "Candidate names" are the outputs of the initial step our renaming process, using image captioning to provide context for prompting GPT-4. A key difference between these names and RENOVATE names is that the former uses a set of names for a class of segments, whereas RENOVATE names provide a name for each individual segment. For model evaluation, we follow prior practices [5, 24, 25] to group
7

---

Source dataset
Target datasets
MS COCO
ADE20K
Cityscapes
Train names
PQ
AP
mloU
PQ
AP
mloU
PQ
AP
mloU
Original names
52.70
42.72
60.91
25.61
15.97
33.06
46,06
23.67
57.99
OpenSeg names
53.60
43.87
60.56
27.14
17.41
35.63
45.34
24.05
58.42
Synonym names
23.38
35.87
47.20
15.56
14.78
29.74
22,94
19.58
45.91
Candidate names
41.32
40.96
56.68
19.80
15.85
33.43
39.20
20.79
51.80
RENOVATE 56.62
45.50
65.48
27.98
17.94
38.50
46.10
27.99
61.25
Table 3: Training with renovated names. During inference, test names are merged from Original, OpenSeg, and RENOVATE names for fair comparison. Our results demonstrate that RENOVATE names can help train stronger open-vocabulary models.
Negalive Song
mile (k)
Origical
Original
45
107
5xx 187 20
S 101 UP
307
5:x UP 50
5% 301 Up
Thitring data doM
Thaining data dos
(a) MS COCO - ADE20K.
(b) MS COCO MS COCO.
RESOWATTE
Figure 5: Data efficiency comparison between RENOVATE and original names.
Figure Effectiveness of negative sampling.
predictions based on the original class divisions. For fair comparison, our test name set merges names from Original, OpenSeg, and RENOVATE names.
Results. Our results in Table 3 demonstrate that using RENOVATE names for training improves seg- mentation on both source dataset and target datasets. For example, compared to the Original names, the model trained with RENOVATE names improves by near 4 PQ on MS COCO and over 5% mloU on ADE20K, showing significantly larger gains offered by OpenSeg names. This underscores the value of name refinement on per-segment level and indicates the high quality of RENOVATE names. We also note that models trained with Synonym names and Candidate names show inferior performance even to the Original names, reflecting the importance of high name quality.
The richer text information offered by RENOVATE names also improves training efficiency. In Fig. 5, we show that models trained with RENOVATE names can achieve comparable perfor- mance with significantly less data compared those trained with the Original names. For instance, models trained with 10t images using RENOVATE names can match or exceed the performance of models trained with 5 10 images using the Original names. This corroborates previous findings that improving data quality can reduce the required quantity of training data [45, 46].
Ablation on the negative sampling. In Fig. 6, study the effectiveness of negative sampling. Interestingly, we found that negative sampling is especially helpful to RENOVATE names while mi- norly impacting models trained with original names. This is because RENOVATE names have much more class names and the names are also more semantically close. Negative sampling effectively controls the number of classes used in cross entropy calculation and keeps the name embeddings relatively distinct, thus significantly improving training effectiveness for RENOVATE names.
5.3 Improving evaluation with renovated names
Setup. We evaluate pre-trained FC-CLIP [25], MasQCLIP [35], and ODISE [24] models on ADE20K and Cityscapes using three different name sets with both standard and open metrics. For open metric computation, we need the similarity between the ground-truth and predicted class names. For Original and OpenSeg names, which use a set of names for a class of segments, we use the averaged pairwise name similarity between classes. For RENOVATE names, we can directly use
8

---

Ii Standard metric
Open
22.6
18.6
OpenSing
RENOVATR
Open8oa
RENOVATE
Original
RENONAEE
(a) FC-CLIP [25]
(b) MasQCLIP [35]
ODISE [24]
Figure 7: Open-vocabulary evaluation on ADE20K with different names, metrics, and models.
1oU (%)
sky
same: nall
35.6
sports car
bathroom sink
80
60
man
Woman
Predd criginal
commr
14.9
0.0
0.0
bollard
railing
churchi Kal
0.0
0.0
0.0
ceiling light tray
orights samer bit biiding
(a) Per-category IoU on the ADE20K with 578 RENOVATE classes.
(b) Miselassification analysis.
Figure 8: RENOVATE names enable more fine-grained analysis models. (a) Per-category IoU with highlighted top/bottom-5 RENOVATE names and selected names from the "person" class in ADE20K. (b) Confusion matrix of frequently misclassified RENOVATE names from "building" to "wall", showing misclassification proportions (numbers) and pairwise name similarity (color).
the pairwise name similarity between renovated names. Fig. 7 shows PQ evaluation of pre-trained models ADE20K. Full evaluation results, including mAP and mloU and results on Cityscapes are provided in Appendix B.3.
Results. In Fig. 7, we first note that the discrepancy between standard and open metrics is signifi- cantly greater when evaluated using RENOVATE names. This indicates that many misclassification cases are indeed "benign", i.e., when evaluated using more fine-graing grained names, the misclassified names turned out to be semantically close to the ground-truth names. For example, when an "area rug" is predicted to be a "floor rug" (RENOVATE names), it will be penalized much less than from "rug" to "floor" (original names); but if "area rug" predicted as "concrete floor", classifi- cation error will be even higher. This shows that current pre-trained models have learned relatively good semantic concepts, while still being sensitive the text prompts inference time.
RENOVATE names also enable a more fine-grained analysis of the models. In Fig. 8a, we present per-category IoU analysis on RENOVATE names within the ADE20K dataset. We can see that large, well-defined objects such as "sky" and "girl" have high IoU scores, while small, deformable, and rare classes like "bollard" and "ceiling light" are more challenging for current models. Note that most of these names do not exist in the original benchmarks. In addition, our analysis reveals model biases, evidenced by the disparity in IoU scores between "man" and "woman" "girl" and "boy", suggesting an imbalance in the training dataset. In Fig. 8b, we further illustrate how RENOVATE names facilitate a more detailed investigation of misclassifications For example, among "building" segments misclassified "wall", 32.7% "churches" from the "building" class are predicted to be "brick wall" from the "wall" class. Additionally, we observe a strong correlation between name similarity and misclassifications. This showcases that our RENOVATE names can help identify problematic sub-categorics within the original class that models still struggle with.
well as
Finally, we note that the open metrics are penalizing more on misclassified concepts with lower semantic similarity, e.g., "car/road" mistakes are penalized more than chrttrck/ mistakes. other use scenarios, different types of mistakes (e.g., outlier patterns) may be preferred to be penalized more and the evaluation metrics should desgined accordingly. However, regardless of the design of the evaluation metric, RENOVATE names make it possible to conduct fine-grained analysis on mistakes with different semantic distances. Without RENOVATE names, we can only see the coarse understanding of the model.
misclassifications without

---

Conclusion and Limitations
In this work, we address the naming issues and show that renaming improves both model training and evaluation of open-vocabulary segmentation. While RENOVATE uncovers model biases, could inadvertently propagate biases from the foundational models into the new names. To mitigate potential negative societal impacts, advocate for verification of names in critical applications, as exemplified by our verification names evaluation sets. As the first attempt to propose a generalized renaming framework, acknowledge that our exploration remains incomplete. In the future, we aim to further refine our method, exploring more design choices such as other model backbones [47], and scale up the publicly available large-scale datasets [48, 49].
Acknowledgement
Haiwen Huang and Dan Zhang were supported by Bosch Industry Campus Lab University of Tubingen. Andreas Geiger was supported by the ERC Starting Grant LEGO-3D (850533) and the DFG EXC number 2064/1 project number 390727645. Haiwen Huang would like thank Tianlin Ye for her help in figure making.
References
[1] Eleanor Rosch, Carolyn B Mervis, Wayne D Gray. David M Johnson, and Penny Boyes-Braem. Basic objects natural categories. Cognitive psychology. 8()):322-339 1976.
[2] Eleanor H Rosch. Natural categories. Cognitive psychology, 4(3):328-350) 1973.
[3] Edward Smith and Douglas L Medin. Categories and concepts. Harvard University Press, 1981.
[4] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In JCML, 2021. 1,2
[5] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open- open- vocabulary image seg- mentation with image-level labels. In ECCV, 2022. 1,2,3,7
[6] Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary panoptic segmentation with maskelip. In ICML, 2023. 1.2
[7] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. In CVPR, 2023. 1, 3,
[8] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapter clip. In CVPR, 2023. 1,2
[9] Xiuye Gu, Lin Tsung-Y, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022. 1,5
[10] Prannay Kaul, Weidi Xie, and Andrew Zisserman. Multi-modaa classifiers for open- object detection. In ICML, 2023. 1
[11] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. simple framework for open-vocabulary segmentation and detection. In ICCV, 2023.
[12] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. Open-vocabulary object detection upon frozen vision and language models. In ICLR, 2023. 1
[13] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014. 2,
10

---

[14] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 7,3
[15] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Ro- drigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 2, 7,3
[16] Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open vocabulary scene parsing. In ICCV, 2017.
[17] Maxime Bucher, Tuan- Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic seg- mentation. NeurIPS, 2019. 2
[18] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In CVPR, 2019.
[19] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-voce ppenvoocabulary semantic segmentation with pre-trate treitriined vision-language model. In ECCV, 2022. 2,3
[20] Seokju Cho, Heeseong Shin, Sunghwan Hong, Seungjun An, Seungjun Lee, Anurag Amab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. arXiv preprint 2rXiv:2303.11797 2023. 2
[21] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.
[22] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. V1-bert: Pre- training of generic visual-linguisic representations. In ICLR, 2020. 2
[23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML., 2021. 2,
[24] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. ODISE: Open- Vocabulary Panoptic Segmentation with Text-to-Imape Diffusion Models. In CVPR, 2023. 3, 6, 7,8, 5,5
[25] Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chen. Convolutions die hard: apen-locabbllay segmentation with single frozen convolutional clip. In NeurIPS, 2023.
3, 6, 8, 9, 5,
[26] Songyou Peng. Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Mare Pollefeys, and Thomas Funkhouser. Openscene: 3d scene understanding with open vocabularies. In CVPR,
2023. 3
[27] Ayça Takmaz, Elisabetta Fedele, Robert W. Sumner, Mare Pollefeys, Federico Tombari, and Francis Engelmann. OpenMask3D: Open- Vocabulary 3D Instance Segmentation. In NeurIPS, 2023. 3
[28] Sarah Parisot, Yongxin Yang, and Steven McDonagh. Learning to name classes for vision and language models. CVPR, 2023.
[29] Chaofan Ma, Yang Yuhuan, Chen Ju, Fei Zhang, Ya Zhang, and Yanfeng Wang. Attrseg: Open-vocabulary semantic segmentation via attribute
caammposiiiion-gggggattii In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 10258 10270. Curran Associates, Inc., 2023. 3
[30] Yuhuan Yang, Chaofan Ma, Chen Ju, Fei Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Multi-modal prototypes for world semantic segmentation. International Journal of Com- puter Vision, pages 1-17, 2024. 3
[31] OpenAl. Gpt-4 technical report. arXiv, 2023. 3,7

---

[32]
Alessandro Conti, Enrico Fini, Massimiliano Mancini, Paolo Rota, Yiming Wang, and Elisa Ricci. Vocabulary-free image classification. arXiv preprint (XXx:2330600011,,, 2023. 3, 1,2
[33] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022. 4,5
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 5
[35] Xin Xu, Tianyi Xiong, Zheng Ding, and Zhuowen Tu. Masqelip for open-vocabular, universal image segmentation. In ICCV, 2023. 6, 8, 5,
[36] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality, In NeurlPS, 2013. 6,
[37] Yoav Goldberg and Omer Levy. word2vec explained: deriving mikolov et al's negative- sampling word-cmbedding method. arXiv preprint 4rXiv:1402.3322.2000 2014. 6
[38] Kamran Kowsari, Kiana Jafari Meimandi, Mojtaba Heidarysafa, Sanjana Mendu, Laura Barnes, and Donald Brown. Text classification algorithms: A survey. Information, 2019.
[39] Zhen Yang, Ming Ding, Tinglin Huang, Yukuo Cen, Junshuai Song, Bin Xu, Yuxiao Dong, and Jie Tang. Does negative sampling matter? review with insights into its theory and applications. IEEE TPAMI, 2024.
[40] Hao Zhou, Tiancheng Shen, Xu Yang, Hai Huang, Xiangtai Li, Lu Qi. and Ming-Hsual Yang. Rethinking evaluation metrics of open-vocabulary segmentaion. arXiv preprint arXiv:2311.03352, 2023. 7,4
[41] Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with verifictiiii.. arXiv preprint M2Xiv:2122.0961.. 2022.7
[42] Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint wxXiv2300.03318. 2023.
[43] Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. Think twice before assure: Confidence estimation for large language models through reflection on multiple answers. arXiv preprint arXiv:2403.0900 09972, 2024. 7
[44] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Mareus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, 2022.
[45] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smymis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. In NeurlPS, 2024. 8
[46] Sachin Goyal, Pratyush Maini, Zachary Chase Lipton, Aditi Raghunathan, and J Zico Kolter. The science of data filtering: Data curation cannot be compute agnostic. In ICLR 2024 Work- shop on Navigating and Addressing Data Problems for Foundation Models. 8
[47] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for lan- guage image pre-training. arXiv preprint 23XX:23303.15343, 2023, 10
[48] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021. 10
[49] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021. 10,
7
12

---

[50] Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming- Yang. Hartwig Adam, Laurent Itti, Balaji Lakshminarayanan, and Jiaping Zhao. Improving zero-shot generalization and robustness of multi-modal models. In CVPR, 2023. 1
[51] George A Miller. Wordnet: lexical database for english. volume 38, pages 39-41. ACM New York, NY, USA, 1995. 1,4
[52] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICIR, 2023. 1
[53] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[54] Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: strong image tagging model. arXiv preprint arXiv:2306.0335 2023. 2
[55] OpenAl chat completions api document. generaaion/chat-ccoplltiissssss
https///llffommoooopaaaccoo/oss/ Accessed: 2023-09-30. 3
[56] Michael Lesk. Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from ice cream cone. In Proceedings of the 5th annual international conference on Systems documentation, pages 24-26, 1986. 4
[57] Zhibiao Wu and Martha Palmer. Verb semantics and lexical selection. In 32nd Annual Meeting of the Association for Computational Linguistics, pages 133-138, Las Cruces, New Mexico, USA, June 1994. Association for Computational Linguistics. 4
[58] Dekang Lin et al. An information-theorrti definition of similarity. In ICML, 1998. 4
[59] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014. 2014. 4
[60] Armand Joulin, Edouard Grave, Piotr Bojanowski, Mathhij Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. arXiv preprint 1rXiv:1612.0365// 2016. 4
[61] Jacob Devlin, Ming-We Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Do- ran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pages 4171-4180 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. 4
[62] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 188):9, 2019.
[63] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020. 4
[64] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021.
[65] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
[66] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR, 2019. 5
13

---

[67] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuu Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint
arXiv:2408.000 2024.7 7
14

---

Renovating Names in Open-Vocabulary Segmentation Benchmarks Supplementary Material
This supplementary material to the main paper "Renovating Names in Vecc-Vocab Segmenta- tion Benchmarks" is structured follows:
In Appendix A, More Literature Review, we provide brief overview of works that ad- dress name problems in open-vocabulary classification.
In Appendix B, More Experiment Results, we provide more experiment results, including a human preference study on the name quality, additional ablation analysis on the renam- ing pipeline, full results on improving evaluation using renovated names and some more examples of using RENOVATE names to conduct fine-grained misclassification analysis.
In Appendix C, More Implementation Details, elaborate the implementation details on the GPT-4 prompts for name generation, human verification process for RENOVATE names evaluation sets, open metrics, and the renaming and pre-trained models.
In Appendix D, More Name Examples, we present more examples of context, candidate, and renovated names. We also show more examples of name selection with the IoU scores.
In Appendix E. More Qualitative Analysis, we provide more qualitative analysis demon- strating that RENOVATE can refine original names, correct wrong annotations, and uncover segments with shared semantic concepts across datasets.
In Appendix F, License for Existing Assets, we list the names of the license for each assets used in this paper.
15

---

select the name that matches more with highlighted mask in the image.
809
Profeesas Raaee (90)
606
earth, ground
806
ground
296
06
RENOVATE
Original
(a) User interface for human preference study.
(b) Human preference results.
Figure B.1: Human preference study. survey of 20 researchers is conducted to compare pref- erences between the original names versus RENOVATE names on the validation sets. RENOVATE names are favored in 82% cases.
A More Literature Review
In this section, we provide brief overview of name problems in open open-vocabulary classification. While OpenSeg [5] is the only prior work for the name problems in open- open-vocabulary segmentation, there are more research works on improving name quality for open- mpen-vocabulary classification, For example, [50] improve class names by including their parents and children from the WordNet hierar- chy [51]. [32] propose to replace the fixed vocabulary list with nouns extracted from best-matching captions. [52] improve the deseriptiveness of the vocabulary by adding class descriptions. How- ever, since the VLMS used by these models are all trained on image-text datasets, it is non-trivial extend them to dense prediction tasks like segmentation. For the renaming strategy, our context name design and LLM-based generation approach ensure the diversity of our names with hierarchi- cal concepts as well as descriptive contexts, without the use of an external word hierarchy [51] or class descriptions as [52]. Our renaming model further improves the name quality based on the visual segments and is more specifically designed for the segmentation tasks.
B More Experiment Results
B.1 Human preference study
As demonstrated in Fig. B.I, we study the name quality of our renovated names with human preference study on ADE20K and Cityscapes datasets, which consists of 30249 and 14670 segments in the validation set, respectively. Specifically, we first randomly select 10% of all the segments in the validation is about hours. Then for each segment, we ask 20 researchers unfamiliar with the project about their preference between its original name and our medel-selected name. Our results demonstrate that our renovated names are preferred in 82% cases, showing a clear advantage over the original names. We provide more details of our human study in the supplements.
each dataset. The average time for human experts to finish the preference study
B.2 Ablation analysis on the renaming pipeline
To conduct ablation study on the renaming pipeline, we leverage pre-trained open- vocabulary model models evaluators of names, automating the name quality evaluation process. Instead of involving humans, we prompt pre-trained open- open-vocabulary segmentation models with different names. Names then considered to be better if they help the model achieve better vece-vocabulary segmentation. Specifically, we use pre-trained FC-CLIP model for the following experiments.
In Table B.I. we first analyze the impact of context names for candidate generation. Our results indicate significant performance improvements when context information provided, as opposed to the "None" approach, which relies only on original names. This highlights the effectiveness of leveraging contextual insights for generating superior candidate names. comparison, CaSED outperforms both BLIP2 and RAM, achieving greater boost in performance through its retrieval-

---

ADE20K
ADE20K
Context
PQ
AP
mloU
Attn bias
Query
Classifier
PQ
AP
mloU
None
25.50
16.53
33.80
mask
leurnable
text
21.09
14.91
26,96
Caption nouns (BLIP2 (53])
25.89
17.36
34.74
gt mask
text
learnable
25.65
16.33
32.67
Image lags (RAM [54]
26.95
17.55
35 .5.54
rand
learnable
26.21
17.50
35.62
Caption nouns (CaSED [32])
27.90
17.88
37.05
rand mask
text w/ neg
learnable
27.90
17.88
37.05
Table B.1: A Ablation on the context name sources.
Table B.2: Ablation on renaming model archi-
tecture.
ADE20K
GPT4 prompt
PQ
AP
mIoU
full prompt
27.9
17.9
37.1
context names
25.5
16.5
33.8
suggestions name types
25.6
16.8
34.0
instructions original names
26.5
17.3
35.2
Table B.3: Ablation GPT-4 prompts.
Prod 07tgmu came aemose
1ibdaato age
66.1
16.5
8.6
conty CE
43.6
23.0
14.7
anoriine-
97.7
1.7
0.6
<45g
622
5.8
2.1
0.8
Ixtermenncio
13.2
5.4
0.0
Leach.
0.0
0.0
0.0
0.0
0.0
0.0
bunce 0.0
0.0
beech
0.0
0.0
0.0
636
GT oiginal nati:: pla11
GTorignu Fami: chail
GTolgg ramo;
Figure B.2: More examples of fine-grained misclassification analysis.
based approach and the utilization of expansive external image-caption datasets, underlining the benefits of contextual enrichment in name generation.
In Table B.2, analyze the design choices of our renaming model. We start by establishing that our approach using text queries superior to a standard text classifier, which only interacts with the transformer decoder's output query embeddings and treats the mask prediction branch in class- agnostic way. In comparison, our text queries interact intimately with spatial details from image features and exploit the mask prediction branch, leading to more precise name-segment matching. We also observe performance benefits from randomly substituting ground-truth masks with pre- dicted ones in attention biases ("rand gt mask"), which suggests training with increased reliance on text queries refines the matching quality. Furthermore, augmenting our model with negative name appending ("text w/ neg") enhances its discriminative capacity, further boosting performance and solidifying the renaming model's 's overall effectiveness.
In Table B.3, ablate the contribution of the different components in our GPT-4 prompts (see Ap- pendix C. Specifically, our GPT prompts have the following components: (1) original name; (2) context names from image captioning; (3) suggestions the two types of names (synonyms and short contexts) to focus, with examples; (4) instructions on how to make use of the original name, with examples. From empirical studies, find that GPT-4 is typically robust to the spe- cific wording of the four components and that removing any key component (esp. (2) and (3)) can significantly affect the results.
B.3 Improving evaluation benchmarks with renovated names
In Table B.4, we show the full evaluation results of the experiments described in Section 5.3. With re- sults on ADE20K and Cityscapes with PQ, AP, and mloU evaluation, we make similar observations as Section 5.3 that the performance gap between standard and open metrics is significantly larger when evaluating using RENOVATE names, indicating that many misclassification cases are indeed "benign"" We also provide more examples of fine-go fine-grained misclassification analysis in Fig. B.2.

---

ADE20K [14]
Cityscapes [15]
Model
Benchmark Names
Metric
PQ
AP
mloU
PQ
AP
mloU
Standard
21.88
13.94
29.16
39.72
27.73
49.60
Original
Open
26.31
14.49
34.39
44.09
27.91
58.20
Standard
21.63
13.95
28.88
43.26
28.45
54.53
ODISE
OpenSeg
Open
28.02
15.32
38,86
46.66
28.88
58.61
Standard
23.69
14.38
31.64
43.61
28.38
57.42
RENOVATE
Open
42.32
50.11
44.65
53.10
34.52
65.05
Standard
23,46
12.80
30.32
33.78
18.11
45.35
Original
Open
26.53
13.01
36.33
37.94
18.35
55.75
Standard
23.70
12.84
31.17
35.05
17.79
46.46
MasQCLIP
OpenSeg
Open
28.04
13.89
39.40
38.83
18.62
52.11
Standard
25.00
12.93
32.30
35.63
18.19
50.07
RENOVATE
Open
47.59
67.01
42.75
49.30
37.25
61.44
Standard
24.30
16.79
32.14
39.42
22.76
53.08
Original
Open
27.04
16.52
36.59
42.57
26.54
62.69
Standard
25.35
17.30
33.06
43.68
26.93
56.15
FC-CLIP
OpenSeg
Open
29.03
17.52
41.03
45.92
27.09
60.21
Standard
27.90
17.88
37.05
45.90
29.79
62.55
RENOVATE
Open
58.25
66.99
48.31
58.63
38.91
70.60
Table B.4: Open-vocabulary segmentation on using both standard and open metries across different pre-trained models using different name sets.
C More Implementation Details
C.1 GPT-4 prompts for name generation
We use GPT-4 ChatCompletion API [55] for candidate name generation. For each class, we provide two messages. The first message, i.e., "system message", describes the role of the system, where describe our overall problem setup and goals. The second message, i.e., "user message", describes the intention of the user, where we provide the original class names and context names. We show our exact prompt messages as follows:
System message: You
helpful assistant aiding in renaming dataset classes. Each class has an inadequate original name and a set of context names derived from related captions (with their frequencies sorted and listed in brackets). These context names provide insights into the category's essence. When renaming, you may: 1. Use synonyms or subcategories of the original class name (e.g., 'grass' can be renamed as 'lawn, turf'). 2. Provide short context address polysemy (e.g., 'fan' can be renamed as 'ceiling fan, floor fan'), Please generate new names for each class in lower Ensure the
listed in a head noun. Avoid arbitrary noun concatenations and nonsensical names. For instance, the class 'sky' should not yield names like 'person under sky'. Ready proceed with naming? Kindly provide the original class names and context names.
names logically connect to the original class, using it as the
Original name: {original class name). }. context names (with frequencies) are [context names). What are the new names? Only provide 5-10 names. And make sure you generate at least 3 reasonable synonyms subcategories.
User message:
For the system message, we start the message with "You are a helpful assistant" following the examples in OpenAl Chat Completions API We then describe our problem setup, input format, and instructions with examples on name generation. (context names) require inputs from each category. For our ablation experiments without context names ("None" Table B. in the main paper), we remove the descriptions about context names in the system message and do not require context names in the user message.
the user message, (original class name) and

---

Please identify a name to match with the highlighted musk. Click "NEXT" continue.
lake
water
river
city
Seccteds
waterfront
pond
others
view
hot water
NEXT
With RENOVATE
Wilhout RENOVATE
(a) User interface for name verification.
(b) Comparison verification speed.
Figure C.I: Human verification for upgrading benchmarks. We ask human annotators to verify the names of segments in the validation sets of ADE20K and Cityscapes. As shown in (a), humans are given the top 3 suggestions from the model and are asked verify the selected name or choose a more matching name from either the other top 3 names or the rest of the candidate names. (b) shows that RENOVATE name suggestions significantly speed up the human verification/annotation process.
C.2 More details on the human verification process
To ensure the high quality of names on the evaluation sets of ADE20K and Cityscapes, verify the names on all the segments. As shown Fig. C.Ia, for each segment, we provide the model-selected name along with the other top 3 names ranked by the model, we also provide the rest of the candidate names in the "others" option. Therefore, the human verifier can quickly continue to the next segment if the selected name itself another name in top 3 names matches the most with the segment. If none of the top 3 names fits well, humans need to choose from the other candidate names in "others". In the latter case, it would be similar the scenario where RENOVATE suggestions and humans have to choose from the whole un-ranked candidate name list. In Fig. C.1b, we compare the name verification speed with and without RENOVATE suggestions 200 segments. As can see, RENOVATE suggestions significantly speed up our verification process. We note that the human verifiers are provided only with candidate names from the same original class, which is typically the original ground-truth class. In the rare cases where our renaming model prefers names from classes other than the original groundtruth class, first verify the class choices, then provide the human verifiers with the name suggestions from the verified class.
provided
We further note that for all the segments verified, 68% of them are matched the default, i.c., top 1, model-selected name, and 92% of them are matched to the top 3 RENOVATE suggestions, reflecting the high quality of RENOVATE names. In Fig. C.2, show typical cases when humans and models disagree, including segments that are too small to recognize and segments whose names cannot be inferred correctly due to insufficient visual cues. We note that such segments only consist minority of all the segments, with inherent difficulty to name precisely.
C.3 More details on the implementation of open metrics
To compute the open metrics [40], we need similarity function on names. Previous research [40] compared three types of similarity measures: WordNet methods [56, 57, 58), text embedding mod- els [36, 59, 60], and pre-trained language models [23, 61, 62, 63]. They demonstrated that both WordNet methods and text embedding models compute reasonable similarity measures while pre- trained language models tend overestimate similarity between names. However, since the names in our work are in free form and may not map to any synonym sets in WordNet [51], we adopt one of the text embedding models, FastText [60], which is shown to compute reasonable similarity measures [40].
C.4 Other implementation details
More details on the renaming model. For the backbone of the renaming model, we use ConvNeXt- Large CLIP [23, 64] from OpenCIII [64] pretrained on LAION-2B dataset [65]. Following

---

baggage tug
road
river
vehicle
street
lake
man
hill
man
person
mountain
person
(a) Segments are too small.
(b) Segments lack sufficient visual cues.
Figure C.2: Typical cases when humans and models disagree during human verification. We show two typical cases when humans may choose different names from our model-selected names. (a) shows when segments
small to recognize, it can be difficult to decide which name is correct. This may indicate a limitation our renaming method on extremely small objects. (b) shows when there is lack of sufficient visual cues to infer the names of the segments, the choices are ambiguous. For example, both "river" and "lake" are reasonable choices for the segment in the top right image without more information of the scene. Note (a) and (b) consist only minority of all the segments.
Fask2Former [33], our transformer decoder consists of nine blocks we use AdamW [66] op- timizer with learning rate of 10-1 and multi-step decay schedule with weight decay of 0.05. Training one renaming model on 4 A-100 GPUs requires approximately 3 days.
More details on training with renovated names.. We follow the training schedule in prior works [25] to train FCCLIP and negative sampling with C = 100 in training all models. Training one model on A-100 GPUs requires approximately 4 days.
More details on the pretrained models. ODISE [24] provides two pretrained models, ODISE (label) and ODISE (caption), trained on category names and caption nouns respectively. We use ODISE (label) since they demonstrated stronger performance than the other model in the original paper. For ODISE and FC-CLIP, we set their geometric mean parameters a = B = 0.5 so we do not differentiate the treatment of test names based on their name overlapping with the training names. In this way, we can compare different test names in fair way. By default, we use FC-CLIP for all of our ablation studies. For the test prompt template, ViLD template [9], which consists of 63 prompt templates such as "There is {article} {category) ) in the scene." These templates are ensembled by averaging the text embeddings.
follow prior work [9, 35, 25] to use the
D More Name Examples
In Table D. we show examples of original names, context names, candidate names, and RENO- VATE names. For RENOVATE, the names for each class are simply the set of unique names for all the instances that class after renaming. We can see that the context names are important for understanding the vague class names such as "earth" and "field" and are helpful for providing more
5

---

Original nome
names
Cundidate names
Renovated names
earth,
bailding, trec, stand,
ground cover,earth lloer cover.
ground. surface, ground,
ground cover, terrestrial terrain, natural ground, snow ground. dirl ground, ground, grass
water, person, floor, sky. road, snow, car
ground
land nutural ground ground.terrain.
earth,dirt ground terrstrial terrain
sky, green,
rural held, roubaade field, green hield.
rural field, roadside field, grassland. field, sports field
field
grass, tree, road.
crop field, sports field, grassland.
hillside, grassy, rural
grassy hillside.
rock,
lush, sky, shereline,
stone, water, tree.
rock, stone standing stone beach pebble.
shorcline rock.national park boulders,
river stone, stones, beach pebble rock, national park boulders.
stone
coust, creek. stand
river stone, stones, lush stome.rocks
shoreline rock
city lountain, building.
garden [ountains, Fountain, urban tountains,
hotel water features, urhan fountains park fountains
fountain
garden, square, park, water,
square fountains, park fountains,
botel. place. design room, arca, lobby, reception, shop, store,
water festure designs.otel water features
chockout counter counterrocciion disk,
information kiosk, reception desk, cushier counter, counter. office counter area, checkout counter
office counter area, www..ooonnttoppooo
lobby counter, store counter.
counter
botel, design, office.
cashier countir.ccunterrree
interior
service desk storefiom counter, information kiosk
stairway.
staincase
ceiling, lead to, pillar, halustrade, stairwell,
buikding, stair, floor, room.
building stairs, stair pullaa-supported stairs, room raaling stairway, floor staircase
halustrade staircase ceiling stairwel stuirway.
pillar supported railing stairway. stairway, floor staircase
bar, restaurant, room, area. hotel, interior, pub, table. house, counter room. bed, bedroom, house, idea, design, palace, garden, home, royal
bar table bar retauratttt bar hotel bar. bur counter interior.bur.pub
bar area, bar table
bar
bar counter
home canopy,came canoopy, cauopy.uuuue house canapy.
bedroom canopy.
canopy
ganden ruyal palace canopy- canopy. bedroom canopy.noom camopy
patio canopy, garden canopy
high-capacity grandstand. park bleadbers, covered gume stand. world class viewing areu. spectator stand, football Beld seating. granddsddnd. sports arena stand, stadium gramdstand. school spectator seats
stadium, game, field, urena,
sports venue grandstand, sports arena stands, park bieachers, spectator stand football field seating
grandstand,
covered stand
world. park, football, school, sport, high
swimming paol, swimming bath. natatorium
villa, resort, beach. indoor, luxury, apartment
pool, swimming, house, hotel,
apartment complex paol.hotel swimming paol,
home swimming pool swiiming pool.
luxury resort pool.swimmingg bath.villa pool,
hotel swimming pool,
beuchfront swimming pool.
villa pool.
beachfront swimming pool,indorr swimming bath
indoor swimming pool
Table D.I: Examples of context names, candidate names, and RENOVATE names for classes from ADE20K.
original name: table
original name: gravel
original name: sereen door / sereen
original name: sund
our name:
our name:
our name:
name:
home office desk
gravel ground
shower door
sundy coast
90.6
93.8
90.6
86.2
computer desk
gravel
bathroom shower door
sandy beach
85.7
86.8
89.3
79.7
white table
ruilway gravel
screen door
resest
84.1
78.7
82.6
77.7
table
gravel
apartment shower door
park sand playground
$7.6
58.5
10
57.7
57.0
Figure D.I: More examples on name selection based on IoU scores.
contexts names with multiple potential meanings like "counter" and "bar". We can also that RENOVATE names provide more comprehensive descriptions of the corresponding classes.
In Fig. D.1, we further show more visual examples of the name selection process (Section 3.3). To- gether with Fig. in the main paper, these examples demonstrate that our renaming model provides mcaningful ranking of the names to match with each segment.

---

E More Qualitative Analysis
E.1 RENOVATE can refine original names
In Fig. E.I and Fig. E.2, we present more examples of RENOVATE names along with the original names of segments from ADE20K and MS COCO. We show that RENOVATE refines the original names by being more precise and closer to human natural language.
E.2 Other potential use cases of RENOVATE
RENOVATE correct annotation errors. From Fig. E.3, discovers wrong annotations and finds more accurate names through RENOVATE. It's important to highlight that these renovated names do not belong to the original ground-truth class; instead, our model ranks them over names from the original class. This demonstrates the renaming model's utility in detecting and correcting inaccuracies in annotations by effectively matching names with visual content.
that renaming model
RENOVATE uncovers segments with shared semantic concepts across datasets. Since RENO- VATE aims to describe visual segments using more human-aligned language, effectively standard- izes the naming conventions for identical concepts in different datasets. In Fig. E.4, we demonstrate this with examples from MS COCO and ADE20K. While renaming each dataset separately, we observe that RENOVATE inadvertently reveals the shared name spaces by identifying more appro- priate names natural language describe the visual contents. This property can be potentially used for analyzing the differences between datasets, merging datasets to construct new benchmarks, and various other data curation processes.
RENOVATE can rename segments in an image when combined with other foundation models. In Fig. E.5, showcase another possible application of our renaming model by further applying it on generated masks from SAM2 model [67] (points per side=12) and generated image tags from the Sooogniz-Anythiin--Nee
(RAM) [49]. Specifically, biases and each mask is paired with the full list of the generated tags as the text queries. The renaming model is used to obtain the best-matching name for each mask. good mask-name matching performance demonstrates that our renaming model can potentially be used on datasets where mask annotations not available.
the generated masks attention
F
License for Existing Assets
CaSED code and model [32]: MIT License.
GPT-4 model [31]: https::/tpppptt : //tp//ai....////////// cowm/pplliccces/tttts-o--------a
FC-CLIP code and model [25]: Apache License 2.0.
ODISE code and model [24]: Custom license (attps://github.ccmm
blob/main/LICENSS
MasQCLIP code and model [35]: Custom license
MS COCO [13]: Creative Commons Attribution 4.0 License.
ADE20K [14]: Creative Commons BSD-3 License Agreement.
Cityscapes [15]:
Custom licence (utp:://ww..tyycopeeddaaatt
license/)

---

bedroom canopy
tray
bar counter
pendant light
canopy
box
bar
garden canopy
snow ground
baggage tug
canopy
car
earth
lamp
park bleachers
crop field
floor lamp
ivy
field
lamp
grandstand
flood
merchandise booth
pendant light
lamp
plant
pedestal, stand
water
lake
lamp post
shower door
pool
lamp
screen, screen door
water
water
cashier counter
file box
counter
reception desk
OLYMPUS
exhibit booth
box
counter
pedestal, stand
Figure E.I: More examples of renovated names segments from the validation set of ADE20K.
For each segment, we show the original name below the image and the renovated name the text box.

---

spectator box
J.P.Morgan
gravel road
tennis court
RLO
building
floor
gravel
pineapple
strawberry
watermelon
pavement
fruit
fruit
fruit
floor
paved path
hill
riverbank
train platform
mountain
pavement
platform
platform
runway
awning
basketball court
tennis court
playingfield
playingfield
road
roof
awning
coffee table
sandy beach
roof
sand
table
office desk
carpet floor
countertop
table
rug
table
river
swimming pool
canal
pond
water
water
water
water
Figure E.2: More examples of renovated names segments from the validation set of MS COCO.
For each segment, we show the original name below the image and the renovated name the text box.

---

UUB
STOP
Original wood wall
Original name
Original name pizza
Original name tent
pavement
Renovated shelf
Revovated rock
Renocated
Renovated
dessert
roadside building
Original name
Original name
Original name
Original nume floor
stone wall Renovated
rug Renovated
rock Ressovated pavement
Renovated
pavement
blanket
kennis court
Figure E.3: RENOVATE can find wrong annotations and suggest corrections.
COCO
ADE20K
RENOVATE
paper
name
paper bag
bag
COCO
ADE20K
RENOVATE
name
bar counter
bar
counter
COCO
ADE2OK
RENOVATE
name
runway
runway
pavement
Figure E.4: RENOVATE uncovers segments with shared semantic concepts across datasets.
10

---

sky
brick
plane
dog
airliner
back yard
yard
airport runway
picnic table
(a) Tags: field, airliner, plane, airport runway. cloudy, floor, land, raceway, sky, tarmac, yellow
backyard, ball, brick, dog, ledge, lemon, picnic table, ramp, stand, yard
(b) Tags:
tree sky
sky
building
hill
dirt field
mountain
stone
stone
stone
cow
cow
stone
COW
giraffe
stone
stone
(c) Tags:
animal, blanket, bull, cattle, trumpet, stone, cow, field, grass, grassy, green, herd, hill, hillside, lush, mountain, stand, white
Tags: area, dirt field, dry, enclosure, fence, field, floor, giraffe, grass, habitat, hay, building, sky, tree, zoo
Figure E.5: RENOVATE can rename segments on generated masks and image tags. We show the full list of the image-level RAM-generated tags captions of each image and plot all SAM2- generated masks with their best-matching tags. Note some objects are not shown the image as they are not segmented out by SAM2.

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The three claims we summarized at the end of Section are matched to Sec- tion 1, Section 3, and Section 5, respectively.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 6, we discussed we didn't t fully explore all the design choices for renaming pipeline, which we aim do in the future. Our renaming procedure could also inadvertently propagate biases from the foundation models to the new names. We suggest future users to verify the names for critical applications.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
12

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide implementation details regarding all our experiments on renam- ing, training with renovated names, and improving evaluation with renovated names in Ap- pendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- reasonable avenue for reproducibility, which may depend
missions to provide on the nature the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibiliiy. In the case of closed-source models, it may be that access to the model is limited in
13

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will release our code upon paper acceptance. The datasets used in this paper are all publicly accessible. For our data processing on MS COCO, ADE20K, and Cityscapes for open-vocabulary segmentation, we follow prior practice [25], whose code is already publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
httll//guiies/ooeeuuuiiii swsssioiicciyoisooooooooooiiiill for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.ao. cc/ wtc/pbblc/gguiiiicc geiic//ggge////iiiiiiioiii CCoee///omdbbb cetbbiission ssionnolll onoolicpyy cyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]]
Justification:
See Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experiments in the paper are costly. Due to resource and time constraints, we didn't 't investigate the statistical significance aspect.
Guidelines:
The answer NA means that the paper does not include experiments.
14

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpp://nents http://///eee../eeeeeeeeee cttps.cc//puaee httpppplllli/tiaaaaiaaiieeeeeeee
Answer:
[Yes]
Justification:
We reviewed Code of Ethics and research conforms with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed broader impacts Section that while RENOVATE names improve the quality of public datasets, it could also propagate the biases from foundation models the renovated names. We therefore suggest future users to verify the names for critical applications.
15

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: In Section discussed how name verification conducted in Sec- tion 4.2 can potentially prevent harmful model biases being propagated into renovated names.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix F,
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
16

---

The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi swwchcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
The model training and data renaming details are in Appendix C.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: See Appendix and Appendix C.2.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: Our human study is at a small scale, we didn't 't obtain IRB approvals or equiv- alent.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
17

---

Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
18


Reviews:
[{"review_id": "TbXlfYzm7e", "reviewer": "Reviewer_fwux", "summary": "The paper considers the problem of open-vocabulary segmentation. The task requires segmentation models to recognize categories outside of the training taxonomy. This usually means learning a joint image-text feature space and classification via similarity matching between class text embeddings and segmentation mask embeddings. The paper aims to improve the suboptimal text descriptions of the dataset classes, which are often too general, ambigous and context dependant. The proposed method describes a strategy for automated renaming of the assigned class descriptions at the segment level. This means that each ground truth segment gets assigned with one or more text descriptions that describe that particular segment more specifically. Training with such enriched text descriptions improves the model performance on both the source and target datasets. The experimental study follows the usual setup where COCO is used as source dataset, and ADE20k and Cityscapes as target datasets.", "questions": "Please consider the questions from the weaknesses section that would clarify training of the renaming model and the evaluation procedure with the renovated names. Also, please discuss the possibility of evaluating the renaming model with another baseline with candidate name selection ability. Please find the suggestion in the weaknesses section, which considers a baseline based on pooling of CLIP features.", "limitations": "The authors discussed the limitations in the section 6. The main concern is regarding the bias transfer from the foundation models.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper considers an important and underexplored problem in open-vocabulary segmentation. To the best of my knowledge, OpenSeg [5] is the only prior work that also considers the problem of suboptimal class descriptions and corresponding renaming solutions. The paper provides a proper comparison and discussion of the differences.\n\nThe proposed renaming method is fully automated, which represents a significant step forward w.r.t. to the related work that requires manual inspection. The described approach is technically sound and properly explained.\n\nThe proposed method seems dataset and model agnostic. This means that it should easily generalize to other datasets and would be beneficial for training of different open-vocabulary segmentation models. The authors promise to share the code and the renamings for popular datasets which would have positive affect on future research in open-vocabulary segmentation. \n\nThe results reveal that training with renovated vocabulary leads to increased segmentation performance across different metrics and datasets. Furthermore, including the renovated vocabulary only in the evaluation phase leads to increased performance of the open-vocabulary models pretrained with regular datasets.", "weaknesses": "Some parts of the paper remain unclear. \n\nFor example, I am confused about the batch collation for training the renaming model. Figure 2 suggests that the transformer decoder in a single forward pass receives candidate names from a single ground truth class. Does this mean that for a single image the number of forward passes through the transformer decoder is equal to the number of ground truth segments? Would you say then that a single training example is consisted of an image, a single ground truth segment and multiple candidate names? Is then a batch size equal to the total number of segments in the sampled images? Does the batch size 16 from line 239 refers to the number of images or number of segments? Does this type of training affects the memory requirements significantly? Are there any redundancy in terms of the tensor representations that have to be kept in the memory in order to enable parallel training on multiple segments? I suggest authors to clarify these questions.\n\nI also dont understand the details of using renovated names during the evaluation. Does this mean that all the candidate names are just appended to other naming variants of that particular class? It seems to me that we do not need to train the renaming model if we would like to use the renovated names just for the evaluation. Is this correct? How many naming variants per class do you get on average when you concatenate the proposals from the original set, the OpenSeg and the renovated names? Does this affect the inference speed significantly? If you also use the usual templates (a picture of *class name*, an image of *class name*, etc.) it seems to me that there might be a lot of different textual representations for a single class. How does this inference really work?\n\nThe renaming model should be compared to some stronger baseline which also has the ability of candidate selection. For example, you could assign each segment with a candidate name that achieves highest similarity with the segment representation based on the CLIP embeddings. The segment representation could be average-pooled feature representation from the CLIP visual encoder. I question the necessity of training the mask prediction module just for renovating the class name associated with some segment, if we already know the corresponding ground truth mask. Could you please comment on this issue and maybe discuss some possible alternatives?\n\nIn Table 3. the results obtained with OpenSeg names should correspond to the original FCClip results. However, in some cases the reported performance is lower than the FCClip performance reported in the official repository. Why is this the case?\n\nAs shown in Table 3., training with renovated names leads to marginal gain in some cases (e.g. PQ on Cityscapes). \n\nI am not sure if the x-axis on figure 5 shows the number of images available in the training dataset, or the number of iterations trained on the full dataset? If these experiments consider subsampled training subsets, you should explain how the training images were selected.\n\nI am a bit surprised about the effectiveness of the negative sampling. What would be the effect of including additional renovated names of the ground truth class as negatives? This reminds of hard example mining, as it would force the model to differentiate between similar semantic concepts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for pointing out that we addressed an “important but underexplored” problem and that our approach is “technically sound”, “properly explained”, and “would have positive impact on future research”. We address the remaining comments below.\n\n**1-(a). Batch collation to train the renaming model**\n\nOur renaming model processes all images in a batch (16 images) in a single forward pass – all the text queries from the gt segments are processed in parallel. Specifically, each input query is to assess the alignment between one gt mask and one of its candidate names — the input query itself is initialized with the text embedding of the candidate name, and it is refined by the cross-attention layers that use the gt mask as attention biases.\n\nSince the model can process multiple queries in a single pass (like Mask2Former), we can process all such mask-name pairs in one batch in a single pass. For images where the number of pairs is less than the number of queries, we simply pad with empty queries for collation. \n\nWe will make the batch collation more clear in the paper.\n\n**1-(b). Memory usage**\n\nOur model **consumes similar memory** to a standard Mask2Former model. For the memory consumption, the key factor is the number of queries. In Mask2Former, this number is 250. In our renaming model, it usually varies between 100 to 350, depending on the number of candidate names and segments in images in a batch. \n\n**2-(a). Name merging for evaluation in Table 3**\n\nFor results in Table 3, when evaluating, we append the **“renovated names”** instead of **“candidate names”** to the other naming variants. After merging the names, the number of names per class is 6.44, 4.98, and 6.89 for COCO, ADE20K, and Cityscapes, respectively. This has barely any influence on inference time (0.3sec/img) since the **inference cost is mostly in processing the image and query features** instead of the final cosine similarity computation. One can even choose to use more test names (e.g. 10 names per class) depending on the use case and will not incur any noticeable extra cost of inference time.\n\n**2-(b). The usage prompt template in inference**\n\nFor each prompt name, we first use ViLD templates to create 63 different sentences and extract their CLIP text embeddings. We then average the text embeddings after normalization and use this **single embedding** as the text embedding for the corresponding name. This is also the common practice in other OVSeg works [1,2].\n\n**2-(c). The need of renaming model for the evaluation**\n\nSince we need “renovated names” for evaluation, we need to train the renaming model to obtain them. Evaluating using candidate names would introduce unnecessary complications in the evaluation as they are quite noisy (as indicated by the inferior performance when training with candidate names in Table 3). \n\n**3-(a). Baselines for the renaming model**\n\nWe thank the reviewer for the suggestion of using average-pooled CLIP features as a baseline. In prior works, this baseline is known to be inferior to tailored OVSeg models[1]. In [1], it’s shown that with gt masks as region proposals, average-pooled CLIP features only reach an mIoU of **20.1%** on ADE20K, being significantly lower than OVSeg models (at least over 30%).\n\nOur baseline for the renaming model is the FC-CLIP with gt attn biases (row 1 in Table B.2), which is much stronger than the average-pooled CLIP features. Our model significantly improves upon this baseline (from **27.0%** mIoU to **37.1%** mIoU) by incorporating our designs of “rand gt masks” and “text queries”.\n\n**3-(b). Necessity of training the mask prediction module.**\n\nRelatedly, since CLIP features are not good enough, we need to improve them. The pixel decoder in our renaming model is responsible for improving the CLIP features to a per-pixel level. The mask prediction module is irreplaceable since it’s the only source of supervision signals for these per-pixel features. \n\nWe also note that even if we use gt masks as attn biases, the mask prediction is still non-trivial. As we write in L160-164, the gt masks are only guiding the cross-attention to focus on the segment regions.\n\n\n**3-(c). Alternatives of the mask prediction module**\n\nAs an alternative to our mask prediction module, one may design different proxy losses to train the renaming model. For example, a region-level contrastive loss that forces matched mask-name embedding pairs to be closer. As our focus of this paper is to demonstrate the importance of names and renaming, we leave such exploration to future work.\n\n**4. Table 3 OpenSeg results are lower than the original FCCLIP results.**\n\nThis is because the performance of OV Seg model depends on the test name set. In Table 3, our test names merge all three name sets. This is for a fair comparison of models trained with different names. We will update the paper accordingly to make this clear.\n\n**5. Figure 5 setups.**\n\nThe x-axis in Figure 5 shows the number of images available in the training set. The training images are randomly selected from the whole training set. \n\n**6. Negative sampling variants**\n\nNegative sampling is highly effective to train with a large vocabulary in NLP. Recently, some OVSeg models also have adopted a similar approach to ours [2].\n\nWhile negative names from other classes are guaranteed “true negatives”, same-class renovated names can be complicated. For example, for one segment, “man” might be the most precise, but “person” is also not wrong. It is thus not ideal to penalize “person” in the same way as names from other classes (e.g., “car”) during training. To remedy this, one may add an extra step to first select “true negatives” among each class. This can indeed be an interesting future extension of our current sampling approach. \n\n[1] Liang, et al. \"Open-vocabulary semantic segmentation with mask-adapted clip.\" CVPR 2023.\n\n[2] Cheng, et al. \"Yolo-world: Real-time open-vocabulary object detection.\" CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RrYz4PTacC", "reviewer": "Reviewer_aQgc", "summary": "The paper explores modifying the class names associated with mask annotations in the segmentation datasets in the context of open-vocabulary segmentation. A method, RENOVATE, is described to perform such renaming in an automated pipeline. RENOVATE leverages an image captioning model to generate contextual words for each class. GPT-4 is prompted by class name and the contextual world to generate a pool of candidates. A segmentation model is trained to select among the candidate names using the mIoU as a proxy metric. The refined class names are then used to train open-vocabulary models, which show improvement in performance.", "questions": "- It would be good if the issue surrounding the training of the renaming model (W1) could be clarified in the rebuttal. \n - Additionally, providing some commentary on the weaknesses (W2) and (W4) would help improve the interpretation of the paper. \n - Furthermore, it would be good to discuss future applicability of the pipeline, or whether the better path is to just collect more precise captions for training data. \n\nOverall, the paper is well-written and presents some insights into the treatment of data for open-vocabulary segmentation. There are some limitations in the evaluation. I currently rate the paper as BA prior to rebuttal.", "limitations": "The paper touches on the potential to propagate existing biases in the foundational models it employs, which is a big issue, and it is important that it is mentioned. \n\nI would recommend conducting an ethics review or obtaining IRB approval for the human study, even if it is not strictly necessary according to the respective institutional guidelines.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper explores an interesting angle of open-vocabulary segmentation: improving the names assigned to each segment. This is a novel study, highlighting the value of precise and varied captions for downstream training.\n2) The writing of the paper is engaging. Particularly, the ample use of examples helps illustrate key points around the proper use of terms.\n3) The paper conducts a very small scale study (n=20) human study showing a preference for new names. \n4) The conducted ablation study verifies some components of the pipeline.", "weaknesses": "1) The section on the training of the ranking model to select among the candidate names could be clearer and more streamlined. It is not particularly clear whether a model is trained for each category or if a single model is trained for all categories. \n   - If it is the former, then there are certainly some computational concerns and limitations that the method imposes. It would be useful to discuss the computation cost wrt to the number of classes in the data.\n   - If it is the latter, then the ability to use or re-use this trained model for the downstream task of OV segmentation could be considered. It seems that the effect of gt-mask attn bias is limited (B.2) as randomisation performs better. Also, reporting the performance of this model on the downstream task is of interest.\n2) In the same section, the design and presentation of a specialised architecture for the renaming could be better motivated. Since a proxy of a segmentation task is employed, is the new architecture even necessary? Could existing model architecture perform this task? Why do existing OV segmentation model pipelines not meet the requirements of the proposal's renaming stage?\n\n3) The evaluation of the study for the improved performance of the segmentation models is limited -- only a single architecture is explored. Other architectures would help rule out particular drawbacks of the FC-CLIP being addressed with renaming and would provide more evidence to the broader conclusion about renaming helping model training (L310).\n\n4) While the new names might help identify particular problems in models (L297), the reasoning behind renovating names in evaluation is rather backwards. While some misclassifications might be due to the similarity in labels, such as those identified as \"benign\" by the authors, they are still misclassifications as defined by the task. This then creates a problem in downstream use. For example, two masses in an X-ray might appear very similar due to both being a collection of cells, but their placement and small characteristic outline patterns will determine the difference between benign and dangerous growths. Thus, one should seek to highlight key errors in the evaluation instead, even if the labels are close. It seems that a different phrasing of the conclusions (L296): the learned semantics are not sufficient to differentiate between semantically adjacent classes.\n\n5) [Minor] Finally, the proposed pipeline itself might be less applicable outside of the academic where established datasets with limited label sets are used. Would gathering a dataset with varied captions from the onset make the RENOVATE method redundant?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the novelty of our paper and our efforts in making our writing “engaging”. We address additional comments as the following:\n\n**1. On the renaming model and downstream segmentation task.**\n\nThanks for the suggestion, we will work on making this part more streamlined and clear. Now to answer the specific questions:\n\nFirst, we train the renaming model for all categories of a dataset. \n\nSecond, it is an interesting idea to use the renaming model for downstream segmentation. To do so, we need to use it without gt-mask attn biases and use names from all the classes as the text queries. Note that the “rand gt mask” in Table B.2 still uses gt masks – it only randomizes in the intermediate layers of the transformer decoder, but always uses gt masks at the initial layer. Using a renaming model without gt masks would compromise the performance significantly and make the model unable to distinguish different instances from the same class, as reflected in its segmentation performance on ADE20K: an mIoU of **33.2%** (semantic segmentation) and AP of only **12.9%** (instance segmentation). Similarly, OVSeg models cannot perform as well as our renaming models in the renaming task as they do not make full use of the available knowledge for renaming (see discussion in **2.**).\n\nWhile the renaming model may not be useful for downstream segmentation, it can be used for other tasks such as **generating mask-name annotations for image datasets** by matching caption nouns or tags with mask proposals, making it potentially useful to improve many large-scale datasets. See the PDF for an example.\n\n**2. The necessity of an architecture for renaming.**\n\nCompared to OVSeg, the renaming task has more knowledge of the inputs (gt masks and gt classes) and only focuses on finding optimal names within candidate names of each class for each visual segment. Thus, when designing renaming model architecture, it is a good idea to make use of these different task requirements to ensure high-quality renaming results. \n\nIn fact, our renaming architecture is nearly identical to Mask2Former except that we use text embeddings as input queries and gt masks (with randomization in intermediate layers) as attention biases for cross-attention computation. As shown in Table B.2, both modifications are essential to improve the renaming performance (e.g., from an mIoU of 27.1% to 37.1%). We further reported the renaming performance when using OVSeg architecture FC-CLIP under the same setup as in Table B.2:\n\n||PQ|AP|mIoU|\n|-|-|-|-|\n|FC-CLIP (for renaming)|19.6|14.3|25.3|\n|Our renaming model |**27.9**|**17.9**|**37.1**|\n\nWe can see that our renaming architecture is indeed much superior at the renaming task.\n\n**3. Other architectures for training with the RENOVATE names.**\n\nWe thank the suggestion for extending the current training experiments to include more architectures.\n\nWe further experiment with a very different architecture, YOLO-World[1], which does not rely on a CLIP visual backbone. To compare the training name quality on COCO, we fine-tune YOLO-World-M on the COCO object detection dataset and evaluate OV object detection on ADE20K. By replacing the original names with RENOVATE names, we improve the AP from **18.1%** to **21.2%**, which is equivalent to 17.1% relative improvement. This further demonstrates the general applicability of RENOVATE names to help model training.\n\n**4. On the evaluation with renovated names.**\n\nThe reviewer raised an interesting misclassification case where the visually close misclassifications are of high interest and suggested that “one should seek to highlight key errors in the evaluation”. We totally agree with this.\n\nWe want to first point out that the “key errors” are highly domain-dependent. For example, in autonomous driving, identifying a “car” as a “truck” is much less dangerous than identifying it as a “road” for an autonomous driving car. In this case, the key error is not “car/truck” but “car/road” misclassification. In this domain, classification mistakes with high semantic differences (e.g., “car/road” mistake) are often thought of as more “wrong” as it is consistent with our intuition of a classification mistake.\n\nIn the X-ray case, we can use mistakes in “placement and small characteristic outline patterns” as the key errors instead of “visual similarity” and design the evaluation metric accordingly.\n\nFinally, regardless of how to design the evaluation metric, renovated names make it possible to conduct fine-grained analysis on mistakes with different semantic distances. Without our renovated names, we can only see the coarse misclassifications without a fine-grained understanding of the model.\n\nWe will add this discussion of the evaluation metric in our revision.\n\n**5. Further applicability of the renaming pipeline with rich captions.**\n\nWe want to first point out the fact that collecting a high-quality multi-caption dataset is very expensive — current caption data collection is mainly from web scraping which is of low quality, often misses many objects in the scene, and doesn’t consist of multiple/rich annotations. Curating these datasets already requires a lot of effort and costs[2,3].\n\nCompared to this, it is very economically efficient to curate a dataset with relatively coarse annotations and use our RENOVATE to **improve the quality of annotations**. From this perspective, RENOVATE is indeed very useful and meaningful to improve current large-scale datasets with coarse web-scraped annotations. In the PDF, we showed an example where **the renaming model is applied to generated masks and tags and produces good mask-name matches**. This shows that RENOVATE has broad future applicability.\n\n\n[1] Cheng, et al. \"Yolo-world: Real-time open-vocabulary object detection.\" CVPR 2024.\n\n[2] Gadre, et al. \"Datacomp: In search of the next generation of multimodal datasets.\" NeurIPS 2024.\n\n[3] Fang, et al. \"Data filtering networks.\" arXiv preprint (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TmtlMmxiLJ", "reviewer": "Reviewer_CT7Q", "summary": "Open-vocabulary models use class names as text prompts for unseen categories’ generalization during training. In this paper, the issue of imprecise or even wrong class names from existing datasets is specially studied. One simple and general framework is proposed for automatic dataset renaming, that is, first narrow down the whole name search space to a curated list of candidate names, by using original class names with contextually relevant nouns from image captioning; next employ a specially trained renaming model to select the best-matching candidate name for every ground-truth segmentation mask. To demonstrate the effectiveness, two types of practical applications are explored, i.e. using renovated names to train open-vocabulary models, and applying renovated names to improve evaluation of open-vocabulary segmentation.", "questions": "[-] In the related-work section, some citations are missing, which may mislead readers/reviewers. [1,2] have explored the text issue of class names in open-vocabulary segmentation, with some overlapping with this paper. The reviewer encourages including them and conducting detailed discussions and comparisons.\n\n[1] AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation.      NeurIPS 2023\n\n[2] Multi-Modal Prototypes for Open-Set Semantic Segmentation. Spring IJCV", "limitations": "As the authors said, RENOVATE could inadvertently propagate biases from the foundational models into the new names. To mitigate potential negative societal impacts, they advocate for verification of names in critical applications.\n\nAlso, the authors acknowledge that, the exploration remains incomplete, and will further make refinement to explore more model backbones and scale up to large-scale datasets.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "[+] Due to the fact that class names are often directly used as text prompts, they have a significant impact on the performance of open-vocabulary tasks. Therefore, focusing on the noisy issue of category names is on the right path. \n\n[+] Leveraging foundation models to automate the renaming process indeed reduces the manual labor.\n\n[+] The paper is easy to follow and understand, having clear logic.\n\n[+] Some experiments are conducted to demonstrate that renovate names help to train models with stronger open-vocabulary capabilities and refine evaluation benchmarks.", "weaknesses": "[-] Noise type & noise rate. The authors partially visualize the noise issue of some class names, such as inaccurate or too general or lack contexts, which is worth encouraging. But, are there only these three types of noise in real-world scenarios? What are the noise rates for different datasets? And is the effectiveness of RENOVATE the same for different types/ratios of noises?\nThorough statistics are needed to clarify the performance bounds.\n\n[-] How many ground-truth masks are needed when sorting candidate names using visual context? How to ensure consistency in visual alignment, as visual masks typically have significant intra-class variances (such as humans having different hairstyles)?\n\n[-] To obtain candidate names, GPT-4 and image captioning are used. For GPT4, it is usually sensitive to prompts, how to ensure the quality of generating candidate names (Quantitative comparisons are needed)? Do you need additional labor to perform a double-check after combining results from GPT-4 and captioning?\n\n[-] Since renovating names for open-vocabulary understanding seems to be one general idea, why only focus on segmentation tasks? Will the conclusion of this paper be exactly the same for classification or detection? For example, when using visual samples to match the best candidate, the intra-class visual variances of classification are definitely greater than those of segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our paper to be “on the right path” and that our presentation to be “easy to follow” and have “clear logic”. We address the rest of the comments below.\n \n**1. Noise types and rates.**\n\nBesides the noise types we visualized in Figure 1, there are indeed many other noise types in the real world, such as “being too specific”, “being too ambiguous”, “being culturally/socially inappropriate”, and many more. However, since no groundtruth names are known, it is infeasible to determine exactly how many types of name mistakes are there. It is thus also impossible to determine the “noise rates” for each noise type. \nHowever, in the future, one can use human annotators or even VQA models to annotate a dataset for such name mistakes. That would certainly be very valuable for our understanding how noise in names affect the models.\n\nNevertheless, we can still use human-verified RENOVATE names on the validation set to estimate the “noise rates” of different datasets. Specifically, we can define noise rate$=1- sim($original name, renovated name$)$ to assess how different the original names are from the RENOVATE names, and use the averaged score to assess the “noise rate” of a dataset. Under this definition, the noise degree of ADE20K and Cityscapes are 0.59 and 0.39, respectively. We can see that ADE20K exhibits a higher noise ratio compared to Cityscapes, potentially because ADE20K has more samples and classes and is likely to have more errors. \n\n**2. On the name selection process.**\n\nAs written in L179-182, we perform name ranking and selection for each segment separately. Therefore, only one groundtruth mask is needed for each segment. Since we do per-segment renaming, there is no intra-class variance to deal with.\n\n**3-(a). Sensitivity to GPT4 prompts.**\n\nOur GPT prompts have the following components: (1) original name; (2) context names from image captioning; (3) suggestions on the two types of names (synonyms and short contexts) to focus, with examples; (4) instructions on how to make use of the original name, with examples. \n\nIn Table B.1, we performed an ablation study on the different choices of the context names and we can see that better context names result in significant gains in the renaming performance. We further ablate components (3) and (4) by removing them from the GPT-4 prompt and report the results in the following table. From our empirical studies, we find that GPT-4 is typically robust to the specific wording of the four components and that removing any key component (esp (2) and (3)) can significantly affect the results.\n\n||PQ|AP|mIoU|\n|-|-|-|-|\n|full prompt |**27.9**|**17.9**|**37.1**|\n|- (2) |25.5|16.5|33.8| \n|- (3)|25.6|16.8|34.0|\n|- (4)|26.5|17.3|35.2|\n\n\n**3-(b). Manual check for GPT-4 results**\n\nOur pipeline does not require additional manual checks at the GPT-4 results as they already incorporate knowledge from our “context names” and the original name. \n\n**4. Why only focus on the segmentation task?**\n\nWe chose the segmentation task as this is one of the most representative tasks for 2D recognition. Please note our segmentation task covers semantic, instance, and panoptic segmentation. Our method is readily applicable to detection (by replacing the mask by the bounding box) and classification (by replacing the mask by the whole image). \n\nSpecifically, we conducted an additional study on an open-vocabulary object detection model YOLO-World [1] and by replacing the original names with RENOVATE names for fine-tuning, we improved the performance on ADE20K from an AP of **18.1%** to **21.2%**, which is equivalent to 17.1% relative improvement. We will add the results to the paper.\n\n\n**5. More related works**\n\nWe thank the reviewer for pointing out more related works. Both works mentioned by the reviewer propose to decompose the class names into a set of attributes or prototypes and are indeed also addressing the problem that class names are typically not descriptive enough for open-vocabulary segmentation tasks. \n\nHowever, note that our work addresses the problem of class names in a different way – our RENOVATE framework is able to deal with various kinds of problems with the original names and directly improve the name quality for the segmentation task. \nWe will discuss these related works in our final camera-ready version.\n\n[1] Cheng, et al. \"Yolo-world: Real-time open-vocabulary object detection.\" CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We’d like to thank all reviewers again for their valuable feedback.

We are pleased to see that reviewers found that our work is **novel** (Reviewer aQgc) and **technically sound** (Reviewer fwux), **considers an important and underexplored problem** (Reviewer fwux, Reviewer CT7Q) and would **positively affect on future research** (Reviewer fwux). We are equally glad that the reviewers found the paper writing **engaging** and **easy to follow and understand** (Reviewer aQgc, Reviewer CT7Q).

We’d like to highlight a few points in our responses to the reviewers:

1. We added an experiment using YOLO-World [1] to train with renovated names and reported performance gains in open-vocabulary object detection on ADE20K from an AP of **18.1%** to **22.2%**, an equivalent of 17.1% relative improvement. This demonstrates the benefit of RENOVATE names in training different model architectures and different tasks (open-vocabulary object detection). See responses to **Reviewer aQgc** and **Reviewer CT7Q**.

2. We added a figure in the PDF that showcased how the renaming model can be used **without gt mask annotations** but **with SAM2-generated masks [2] and RAM-generated image tags [3]**. This demonstrates another possible application of the renaming model and shows that our renaming approach can be further generalized to datasets without mask annotations. See response to **Reviewer aQgc**.

3. We added ablation experiments on different components of the GPT-4 prompts for a better understanding of how GPT-4 prompts influence the renaming performance. See response to **Reviewer CT7Q**.

4. We added an experiment using FC-CLIP (without gt-mask attn bias during training) for renaming. Its significantly inferior performance to our renaming model suggested a need for our designated renaming model. See response to **Reviewer aQgc**.

5. We also added more implementation details on the training of the renaming model, the name selection process of the renaming model, and the evaluation of the models trained with renovated names. We hope our response clarifies the questions from the **Reviewer fwux** and **Reviewer CT7Q**.

We hope our replies to all the reviewers satisfactorily address their questions and comments and we warmly welcome further questions during the discussion phase.

[1] Cheng, et al. "Yolo-world: Real-time open-vocabulary object detection." CVPR 2024.

[2] Ravi, Nikhila, et al. "SAM 2: Segment Anything in Images and Videos." arXiv preprint arXiv:2408.00714 (2024).

[3] Zhang, Youcai, et al. "Recognize anything: A strong image tagging model." CVPR 2024.

Meta-review: In this paper, the authors presented a renaming model for open-vocabulary segmentation. Due to the precision concern of the class names in existing datasets, the authors were motivated to enhance their quality for visual segments. Experiments show that the renovated names help the training of stronger models for open-vocabulary segmentation. The main contribution of this paper is the study of the naming precision of existing datasets for the open-vocabulary segmentation task, which could benefit follow-up research in this field.

The paper received comments from three expert reviewers, and all of them acknowledged the contributions of the important problem that this paper looked into, and the corresponding renaming approach. However, there were some concerns around the unclear design of the approach and motivation, missing comparison to alternative methods/baselines, and experimental evaluations, leading to a mixed recommendation of both positive and negative. After the rebuttal together with a few rounds of discussions among the authors, reviewers and AC, the above major concerns were addressed and the reviewers agreed on the contributions of this paper and its potential impact on the field.

Considering the above-mentioned points, the discussions, and the potential interests of the paper may have on the NeurIPS audience, the AC recommends Accept, while reminding the authors to include the additionally provided clarifications and revisions during the rebuttal to the final version of the paper.
Decision: Accept (poster)