AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation
Authors: Chaofan Ma
Yuhuan Yang
Chen Ju
Fei Zhang
Ya Zhang
Yanfeng Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Open-Vocabulary Semantic Segmentation
Attributes
Decomposition and Aggregation
TL;DR: Exploring an attribute decomposition-aggregation framework to address open-vocabulary semantic segmentation

Abstract: Open-vocabulary semantic segmentation is a challenging task that requires segmenting novel object categories at inference time. 
Recent works explore vision-language pre-training to handle this task, but suffer from unrealistic assumptions in practical scenarios, i.e., low-quality textual category names.
For example, this paradigm assumes that new textual categories will be accurately and completely provided, and exist in lexicons during pre-training.
However, exceptions often happen when meet with ambiguity for brief or incomplete names, new words that are not present in the pre-trained lexicons, and difficult-to-describe categories for users.
To address these issues, this work proposes a novel *attribute decomposition-aggregation* framework, **AttrSeg**, inspired by human cognition in understanding new concepts. 
Specifically, in the *decomposition* stage, we decouple class names into diverse attribute descriptions to complement semantic contexts from multiple perspectives.
Two attribute construction strategies are designed: using large language models for common categories, and involving manually labelling for human-invented categories. 
In the *aggregation* stage, we group diverse attributes into an integrated global description, to form a discriminative classifier that distinguishes the target object from others. 
One hierarchical aggregation architecture is further proposed 
to achieve multi-level aggregation, leveraging the meticulously designed clustering module.
The final result is obtained by computing the similarity between aggregated attributes and images embedding.
To evaluate the effectiveness, we annotate three datasets with attribute descriptions, and conduct extensive experiments and ablation studies. The results show the superior performance of attribute decomposition-aggregation.
We refer readers to the latest arXiv version at https://arxiv.org/abs/2309.00096.

AttrSeg: Open- Semantic Segmentation via Attribute DecompositonnAAgrrgatttoo
Chaofan Ma', Yuhuan Yang', Chen Ju', Fei Zhang',
Ya Zhang Yanfeng Wang 1.
Coop. Medianet Innovation Center, Shanghai Jiao Tong University
2 Shanghai AI Laboratory
{chacfanma, yangyuhuan, ju_chen, ferenas, ya_zhang, wangyanfeng6222 jajj.. edu.
Abstract
Open-vocabulary semantic segmentation is challenging task that requires segment- ing novel object categories at inference time. Recent works explore vision-lange llnguu-language pre- pre-training to handle this task, but suffer from unrealistic assumptions in practical scenarios, i.e., low- www-qulity textual category names. For example, this paradigm assumes that new textual categories will be accurately and completely provided, and exist in lexicons during pre-training. However, exceptions often happen when meet with ambiguity for brief incomplete names, words that are not present in the pre-trained lexicons, and lifficulttto-describe categories for users. To ad- dress these issues, this work proposes novel attribute decomppoiitoo--oggraaal framework, AttrSeg, inspired by human cognition in understanding new concepts. Specifically, in the decomposition stage, we decouple class names into diverse attribute descriptions to complement semantic contexts from multiple perspectives. Two attribute construction strategies are designed: using large language models for common categories, and involving manually labelling for human invented cat- egories. In the aggregation stage. we group diverse attributes into an integrated global description, to form a discriminative classifier that distinguishes the target object from others. One hierarchical aggregation architecture is further proposed to achieve multi-level aggregations, leveraging the meticulously designed cluster- ing module. The final results are obtained by computing the similarity between aggregated attributes and images embeddings. To evaluate the effectiveness, we annotate three types of datasets with attribute descriptions, and conduct extensive experiments and ablation studies. The results show the superior performance of attribute decomposition aggreggttinnnnggooooooooo We refer readers to the latest arXiv version at btpps//aarx...orr/aab///033
cumman-invented
1 Introduction
Semantic segmentation is one of the fundamental tasks in computer vision that involves partitioning image into some semantically meaningful regions. Despite great progress has been made, existing research has mainly focuses on closed-set scenarios, where object categories remain constant during training and inference stages [50] 24). This assumption is oversimplification of real-life and limits its practical application. Another line of research considers a more challenging problem, that requires the vision system handle broader range of categories, including novel (unseen) categories during inference. This problem is referred as open-vocabulary semantic segmentation (OVSS).
To handle OVSS, vision-language pre-training (VLP) paradigm [42
17 49 provides preliminary but popular idea. By leveraging language as an internal representation for visual recognition, segmentation is formulated as similarity between category's textual representation and pixel-level visual representation. Following on this paradigm, recent works focus on minor improvements, e.g.,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Decomposition Stage
reg-tkee green body;
Bowtruckda
leaf-shapeed faok sharp daws
Case
Arant DOl tolerto a biid &racchnee
apc lkk
Narze wings ling
R
Demiguise
large. round eyes: long. silty hair.
fufff oneature:
large eyes;
rounc heae;
Mooncall
palc: grey sian long neck
Mooncalf
Natfer
Word entir pre-tniiin levions @
futty cnout
attribute embecs
Aggregation Stage
apgreggttion
aggreepatizn Jense mm
mask
interactions
Han creen wings
10 ofl
Storp beak.
sual embeds
Figure 1: Left: Open-vocabulary semantic segmentation (OVSS) assumes the given new textual categories are accurate, complete, and exist in pre-trained lexicons. However, in real-life situations, practical uses are limited due textual ambiguity, neologisms, and unnameability. Middle: We propose a novel attribute decomposed into various attribute descriptions (decomposition stage), and then, different attribute representations are aggregated hierarchically into final class representation for further segmentations (aggregation stage). Right: Our framework successfully addresses the aforementioned issues and facilitates more practical applications of OVSS in real-world scenarios.
framework where vanilla class names are first
explore to better align Dision-languee modalities [52 25 32).
Although promising, these researches all maintain one unrealistic assumption real-world scenarios, i.e.. the given new textual categories are accurate and complete, and exist in the pre-trained lexicons. As a result, three main issues persist. (I) Ambiguity: brief or incomplete names bring lexical ambiguity, posing great challenge for semantic discriminability. (2) Neologisms: new words that frequently emerge may not be present in the lexicons during vision-language pre-training, preventing pre-trained language models from interpreting their semantics, let alone aligning them with images. (3) Unnameability. unnamed or -effeerlttt categories, such as specialized terms, rare animal names, or specific objects, can create a labeling problem for users, adding complexity during application. The above three issues result low-quality category comprehension, limiting the empirical segmentation performance.
To address these issues. we turn our attention to cognitive psychology when human understanding new concepts [36] 45). For example, if a child asks how to find flamingo in the zoo, one can explain the process by looking for its pink feathers, long neck, and more. Then by combining these answers, a child can easily recognize a flamingo. Such answers provide detailed descriptions from multiple distinct or complementary perspectives, which we refer to as diverse "attributes". Compared to vanilla categories, attributes have three advantages. (1) For ambiguous categories, attributes can make up for missing context information to achieve completeness. (2) For unseen categories, they can be transformed into known attributes, easily interpreted by pre-trained language models. (3) For unnamed or indescribable categories, attributes can be used to replace in more detailed manner. These attribute descriptions, aggregated at scale, provide a strong basis for visual recognition.
Inspired by this, we propose a novel names are first decomposed into various attribute descriptions, and then different attribute repre- sentations are aggregated into a final class representation for further segmentations. Specifically, for the decomposition stage, our goal to generate various attribute descriptions from coarse category names and build attributes for datasets. We propose two construction strategies: one is to generate using language models, and the other involves manually labelling. The first strategy corresponds to situations where common category names are sometimes brief or incomplete, with semantic ambiguity or insufficient discriminability. this case, we can simply annotate attributes upon existing datasets, such as PASCAL [13 [6] and COCO (28). The second strategy involves
framework where vanilla class

---

newly collected dataset called "Fantastic Beasts", which contains imaginal creatures and their invented names by humans. This dataset is used to simulate situations for new words for pre-trained vision-language models, and cnfficult-to-deserii our aim is combine the separate attribute pieces into an integrated global description, which then serves as classifier to differentiate the target object from others. This stage can also be viewed as the process of combining regions reflected by different attributes into a specific one, which yields the segmentation result. Since attributes describing objects may potentially contain hierarchy, we propose a hierarchical aggregation architecture to leverage this potential. A clustering module is carefully designed to aggregate attributes from different levels, and the final mask is obtained by computing similarities between the grouped attribute embeddings and the image features.
categories for users. In the aggregation stage,
To evaluate the significance of attribute understanding for OVSS, we annotate attribute descriptions
on three types of datasets, namely, PASCAL series [3 16 35), COCO series [28 g and Fantastic
Beasts, Extensive experiments demonstrate the superior performance of our attribute decomposition- aggregation framework over multiple baselines and competitors. Furthermore, we performed thorough ablation studies to dissect each stage and component, both quantitatively and qualitatively.
To sum up, our contributions lie in three folds:
We pioneer the early exploration in leveraging only the attribute descriptions for open- open-vocabulary segmentation, and to achieve this end, we construct detailed attributes descriptions for two types of existing datasets and one newly collected dataset Fantastic Beasts;
We design novel decomppsitiion-aggeennn descriptions, and then aggregate them into a final class representation;
framework that decompose class names into attribute
We conduct thorough experiments and ablations to reveal the significance of attribute decomposition- aggregation, and our model's 's superior performance on all proposed datasets.
2
Related Work
Lingon-L Pre-training (VLP) aims to jointly optimize image-text embeddings with large- scale web data. Recently, some studies have further scaled up the training to form "the foundation models", e.g., CLIP 42). ALIGN (17), Florence 44), and FILIP 48). These foundation models usually contain one visual encoder and one textual encoder, which are trained using simple noise contrastive learning for powerful cross-modal alignment. They have shown promising potential in many tasks: grounding [18] 20], detection [21 [9]. and segmentation [47 [1 [9]. This paper uses CLIP for OVSS, but the same technique should be applicable to other foundation models as well.
Open-Vocabulaa Vocabulary Semantic Segmentation (OVSS) aims to understand images in terms of cat- egories described by textual descriptions. Pioneering works 77 26 [5] use generative models to synthesize visual features from word embeddings of novel categories. SPNet [46) and JoEm [4] employ mapping process that assigns each pixel and semantic word to joint embedding space. Re- cently, researchers have proposed to leverage pre-trained vision-language models (VLMs) for OVSS. OpenSeg [4] aligns region-level visual features with text embedding via region-text grounding. LSeg (25) aligns pixel-level visual embeddings with the category text embedding of CLIP. Subse- quent methods like Fusioner [32) Zegformer [IT. OVseg (27] and CAT-Seg 10 have thoroughly investigated the open- veccbloayyl capability of CLIP. However, these methods heavily rely on category names, ignoring text ambiguity, neologisms, and unnameable are common real-world scenarios. cotoopoositiooaagrgggttii to tackle these issues.
This paper designs novel framework of attribute
Attribute Understanding. Visual attributes are first studied in the traditional zero-shot learning [23 (2) With the emergence of vision-language models, the attribute understanding has developed towards more scalable, open, and practical direction. One line of research is focused detecting and
recognizing an open set of objects, along with an open set of attributes for each object [39 40 5
Another line of research focus on object classification by incorporating attributes as part of text prompts, which aim to evaluate the discriminative ability of VLMs with enriched text 41 37] or to enhance interpretability and explainability of model reasoning [4 [ Different from above, our work investigates attribute understanding from the perspective of open- semantic segmentation, using a decomposition caaampooittionagggggattooo strategy.
3

---

3
Method
This paper considers open-vocabulary semantic segmentation (OVSS). We start by giving the prelimi- nary in Sec. [.I] ].T. then we introduce our attribute decompsttion-agrrgattoo framework in Sec. (.2)
decomposition stage and aggregation stage will be detailed in Sec. 3.3 and Sec. 3.4 respectively.
3.1 Problem Formulation & Preliminary
Problem. Given an image RHx OVSS aims to train one model (()) that can segment the target object according to its text description T, that is, outputting one pixel-Ievel mask M:
M +seg (I, T: +) {0, 13
Under open-vocabulary settings, training classes Crass and testing class Cnovel are disjoint, i.e., Chaso N Gravel Ø. During training, image-mask pairs from the base class are provided, i.e., MM) ~ Obuse J: while during testing, the model is evaluated on the disjoint novel classes, i.e., {I movvel
Vision-Language Paradigm. To enable open- ven-vocabulary capability, recent OVSS studies [22 25
embrace vision-language pre-trainings (VLPs), for their notable ability in cross-modal alignment. Specifically, regarding vanilla class names as textual descriptions, open-vocabulary segmentation can be achieved by measuring the similarity between class-level textual and pixel-level visual embeddings:
F. Fg = PVis(I) E RHxXD
Pixt E RIXD,
(2)
where Pvis and Pixt refer to the visual and textual encoders in VLPs. This paradigm has a fancy dream, but meets poor reality. In practice, the textual names of novel classes may potentially suffer low-quality comprehension in three aspects: (1) Ambiguity. Certain names exhibit lexical ambiguity, while others sometimes may incomplete due excessive simplification. These result in deficiency of semantic discriminability. (2) Neologisms. The pre-training text corpus is inevitably limited in its coverage of vocabulary, and thus may not include certain terms that have emerged neologisms in the real world. (3) Unnameability. Certain categories of entities may lack a known or easily describable name for users, particularly in cases involving specialized terminology, rare or obscure animal names, etc. These issues greatly limit the use and development of open- open-vocabulary segmentation.
3.2 Attribute recompositionnAggregattii Framework
To solve the above issues, we introduce one novel attribute decomppoittion-cggreeaall framework.
Motivation. Such textual semantic issues are caused by the low-quality category comprehension. We consider to decompose class name from multiple perspectives, such as color, shape, parts and material, etc. This forms informative attribute sets for the text stream. Treated as partial representations describing categories, (1) attributes can supplement missing information for incompleteness and ambiguity; (2) for new words, attributes can transformed into known words, which can be easily interpreted by the pre-trained language model; (3) for unnamed or not easily describable categories, attributes can be used to describe them more detailed and accurate way. These attributes, when aggregated to a global description, can provide strong basis for visual recognition.
Framework Overview, As illustrated in Fig. 2 given an image and set of attributes descriptions (Sec. 3.3). we first obtain its visual and attribute embeddings (Sec. B.4.T). Considering the potential hierarchy inside attributes, we suggest hierarchical pipeline to progressively aggregate all given attributes embeddings into one specific token (Sec. 3.4.2) As the final grouped token represents all attributes' information, segmentations can be acquired by computing the similarity between this token and the visual embeddings (Sec. 3.4.3). Formally,
(S.c.3.4.2)
FFF,FFF F, T via (I) E HxWxD F, aggr $ txt $ deop (T) E IIIDD (3)
where T deop (+) denotes the decomposition module that returns the set of n attributes' textual descrip- tions of the target category. Note that, category names (and synonyms) are strictly not contained in this set. & Pagge() refers to the aggregation module that groups attributes into one specific embedding.
3.3 Decompose: Detailed Attribute Descriptions for Class Names
In real-world scenarios, vanilla class names T may be gaarse- ambiguous, or neologisms. Unfortunately, there is lack of existing datasets that provide detailed attribute descriptions to offer

---

Clustering Module
Building Attribute by Large Languagee Models
mask
What are useful visual features for distinguishing faaingo the scone?
Clustering Module
Fusion Module
Mixer
hesthers: long, thin leges large. distinctive wingspar
Slot Attn
X.I. stages
Crass Attn
Building Attribute by Manual Collections
Clustering Module
Crass Attn
Fusion Module
data collection
manual annotattion
0M0'
learnable cluster
(a) Decomposition Stage
(b) Aggregation Stage
Figure 2: Overview of Attribute ceaanosstiionAAgrggatii Framework. (a) Decomposition stage aims to decouple vanilla class names into various attribute descriptions. We design two strategies to build attributes, i.e., using LLMs and manual collections. (b) Aggregation stage aims to merge separated attribute representations into an integrated global description. We propose to hierarchically aggregate attribute tokens to specific token in L stages. Each stage alternates a fusion module and a clustering module. Masks are generated by calculating the similarity.
informative contexts. So, we here propose two strategies to construct diverse attributes A. As shown Fig. 2a one involves utilizing large language models, while the other relies on manual collection.
A = decp (T) = {attr, attr,,.... attr,):
(4)
3.3.1 Attribute Descriptions by Large Language Models
For the cases where vanilla category names are semantically coarse or ambiguous, one promising solution is to describe attributes contexts for better discriminability. To generate such attributes, manual writing can be time-consuming and inefficient, particularly for a large number of classes. Hence, for cost-effeetiiveness, we turn to large language models (LLMs) 66 (8), pre-trained large corpora of data, showing remarkable performance of semantic understanding and text generation.
More specifically, to automatically adapt LLMs to enrich class contexts, carefully designed set question templates for attribute descriptions mining from various perspectives. Taking the category "flamingo" as example, we first pose multiple questions, such as "List all attributes for distinguishing a [flamingo] in a photo" or "What are visual features of a [flamingo) in the image". Then we prompt ChatGPT 38 to obtain answers of attribute descriptions, such as "pink feathers: long neck; thin legs; large wingspan; ...". Finally, we filter and combine answers to form an attribute set for each category. Please refer to the supplementary materials for further details.
3.3.2 Attribute Descriptions by Manual Collection
In addition to the above cases, there are two common cases that require attribute descriptions. One is vanilla category names neologisms that unseen by LLMs and VLPs; the other is when users not familiar with an object, so they may have difficulty naming it, especially when it comes to a rare or obscure category. Given that, existing datasets typically do not include rare obscure vocabulary, so we manually collect a dataset of human-made objects and rare categories for simulation.
The dataset is called "Fantastic Beasts", which consists of 20 categories of magical creatures from the film series Fantastic Beasts 222 3 We first scrap and filter the dataset images from the web, then organize fan volunteers of the film series to carefully annotate the paired masks and the category attributes. Since all these creatures and names are human inventions, they are unlikely to been learned by existing LLMs and VLPs. Some images, alone with their corresponding class names and attributes, are shown in Fig. Please also refer to the supplementary materials for dataset information.
3.3.3 Discussion
(1) Datasets:
developing high-quality attribute datasets is crucial contribution towards advancing the practicality of OVSS. As there are currently no existing benchmarks and evaluations, an essential first step has been made that involves annotating the attributes top of the existing datasets such as
5

---

PASCAL and COCO, as well as manually collecting dataset, Fantastic Beasts. (2) Why LLMs: thanks the capabilities of LLMs, category attributes for existing datasets can be obtained scalable manner. Despite not receiving any visual input during training, LLMMs can successfully imitate visual attributes since they are trained on a large corpus containing descriptions with visual knowledge. (3) Signíficances: we believe that the attribute decomposition strategy, and the produced datasets, will have great impact on the community to further promote practical uses of OVSS.
3.4 Aggregate: Hierarchical Fusion for Vision-Attribute Alignments
Given image I and attribute descriptions A, we aim to aggregate these separate pieces of attribute embeddings hierarchically into one integrated global representation 9, with the help of visual information. Then, it can serve as a discriminative classifier to distinguish the target object.
G Pagge (Pxxt ext (A); (T)) E RRIDD
(5)
3.4.1 Vision Embeddings and Attribute Embeddings
We here adopt vision-language pre-trainings 42 as encoders and mostly consider ViT-based archi- tectures [2]. due to their good performance, and flexibility for encoding different modalities.
Given an image I E H the visual embeddings V E IR NV xd are extracted from the visual encoder, where No is the number of image tokens; and d is the channel dimension. The attribute embeddings A R xd obtained by first feeding each attribute into the text encoder separately, then concatenating all of them, where No is the number of attributes describing one target object.
3.4.2 Hierarchical Aggregation Architecture
Attributes descriptions may potentially contain hierarchy. We propose hierarchically aggregate these attributes in I. stages, hoping to explicitly leveraging this potential, as shown in Fig. 2b
Overview. Fach stage alternates fusion module and clustering module. Specifically, the fusion module facilitates interaction between different modalities. Given these enriched representations, the following clustering module groups attribute tokens to fewer tokens. This procedure utilizes learnable cluster tokens as centers for clustering, and considers both visual and attribute information. Based on the similarity, these clustering centers can gather and merge all attributes tokens into specific groups.
Formally, for the 1-th stage, we denote No visual tokens as V E R.N" xd, Nf attribute tokens describing one target object as A E NF xd, and Nº learnable cluster tokens for aggregation as Gi E NR xd The fusion module fuses and enriches the information globally between V/, At, and G):
Vi. A,, Gi v fuse (Vi,
( Ar Gi).
(6)
To avoid notation abuse, still use the same notation for the output. After fusion, the No attribute tokens At are merged and grouped to fewer N (N < N(J) tokens A through clustering:
A4 = w Viuuste (Ar; V4, Gr) E xd
Note that, the attribute tokens are grouped not only based on itself, but also depending on the visual embeddings. And the number of grouped attributes A (output for this stage) is equal to the number of input learnable cluster tokens Gi for this stage, i.e., Na N?-
Fusion Module. Wluse is flexible and adaptable to multiple network architectures. Here we use multiple transformer encoder layers representatives, in order effectively capture and propagate the long-range information of different modalities, by iteratively attending to each other.
Clustering Module. Learnable cluster tokens are used here to represent the clustering center for each grouping stage. It's unreasonable to aggregate solely based on attribute embeddings. as the visual information also plays part in the segmentation. To better incorporate both modalities, the learnable clustering center first obtain the contextual information through vision and attribute cross attentions:
G Peross-actn (q Gi, V1, Vi), G @crces-atta (q Gi. k Ar, Ar). (8)
where ड RRAP xd is the contextual centers; Vú and Ar are visual and attribute embeddings. By exchanging information for the visual tokens and attribute tokens respectively, & has the knowledge of both modalities, providing good prior for the subsequent processing.

---

Open-Vocabulary Model
Aggregate
Apgreestt
Open-
Model
Vocabulary
Agg
-aca-Vacabyllary
Madel
Text Enc
Aggregate
Model
Agggrega regatee
aattrg
(a) direct aggregation
(b) pre- agrragggrationn
(c) post-aggregation
(d) hierarchy aggregation
Figure 3: Comparison between Various Aggregation Strategies. The orange / blue colors represent visual attribute tokens, respectively. Detailed discussions be found in Sec. 3.4.4
Next, we assign each attribute token to one of the contextual centers, and calculate the representation of newly grouped attributes Au as output. We use slot attention [0], by sceing each cluster as a slot:
A @slut-attn (q G A, Ar), A mixer (Ār Gi).
The final grouped attributes Ar+1 for this stage be obtained after adding the residual G.. and then updating and propagating the information between tokens through @mixer- Here we use MLPMixer 44 with two consecutive group-wise and channel-wise MLPs.
3.4.3 Mask Calculation
The total I. stages aggregation gives the result of one specific token AL+1 E Rixd, which can be thought condensing the collective knowledge of the provided attributes' information. Logits y can be generated by computing the similarity between the final stage visual embedding VL+1 and AL+1:
y Psim (VL+1,
(10)
The final predictions can be obtained by simply reshaping to spatial size and upsampling, then applying sigmoid with a temperature and thresholding.
3.4.4 Discussion
The fundamental concept of aggregation strategies involves identifying one single embedding that represents all attributes. As shown in Fig. 3 we present four optional designs: "Direct", "Pre-". "Post-", and "Hierarchy", which are able to applied to all existing open-vocabulary methods. (1) "Direct" involves listing all attributes in one sentence, which is then directly sent to text encoder. The aggregated embedding obtained from the output [CLS] token, which is subsequently fed into the ppen-vocabblary model for further processing. (2) In contrast to "direct", the other three strategies feed attributes into the text encoder separately. "Pre-" first aggregates all attribute tokens into one token, which is then fed into the model similar to "direct". (3) "Post-" inputs all attribute tokens together into the opea-vocabulary model. The output attribute tokens of this model are further aggregated to one token. (4) "Hierarchy" progressively aggregates multiple tokens into in multiple stages, as detailed in this section. In Sec. [.1 and Sec. 4.2) we conduct comprehensive comparisons and explore the adaptability of these strategies to existing voca-vocyyylary models. This analysis demonstrates the universality and effectiveness our decompositioon-ggreenn motivation.
3.5 Training and Inference
During training. the visual and textual encoders are kept frozen, and we uniformly sample N attributes describing the target category with replacement from the attribute set. The predicted mask is supervised by the ground truth using standard cross-entropy loss. During inference, the user has the flexibility to provide test images along with any number of attributes describing the object of interest. The model can then generate the corresponding segmentation mask based on this input.
4 Experiments
Datasets. We evaluate on PASCAL-5* 16 COCO-20'
following 25 32 and evaluate on
Pascal VOC [13] and Pascal Context 335 following ||] 27
PASCAL-5 contains 20 categories

---

that are divided into 4 folds of 5 classes each, i.e., {5 COCO-20' is more challenging with 80 categories that are also divided into 4 folds, L.e, (20' the four folds in the two datasets, one is used for evaluation, while the other three are used for training. PASCAL VOC is a classical dataset. We evaluate on the 1.5k validation images with 20 categories (PAS-20). PASCAL-Context contains 5k validation images. We evaluate on the most frequent used 59 classes version (PC-59), Besides, we also annotate Fantastic Beasts, which contains 20 categories of human invented magical creatures from the Fantastic Beasts film series [ 2 3 information can be found in the supplementary materials.
3 ==0- with each fold having 20 categories. Of
Detailed datasets'
Evaluation. We report mean intersection-over-miie (mloU), following the recent open-vocabulary semantic segmentation (OVSS) literatures [25] 32 17
Implementation Details. We adopt CLIP VIT-L. and ResNer101 as our backbone, and choose aggregation stages I. 4. Numbers of learnable cluster in each stage are (15, 10, 1). During training, the sampled attributes N 15. Adam W optimizer is used with Cos neinn by first warm up 10 epochs from initial learning rate 4e-6 to 1e-3, and the weight decay is set 0.05.
4.1 Comparison with the State-of-the-art
Table 1: Evaluation on PASCAL-5' and COCO-20', For textual inputs, we compare "cls name" (category names) and "attr" (attributes). "direct", "pre-". "post-" and "Iuchy" are four aggregation strategies in Sec. 3.4.4
PASCAL-5'
COCO-20
Model
Settings
Backbone
5"
5
5
5a
mloU
200
20
202
20
mloU
SPNet 46
cls name
RN101
23.8
17.0
14.1
18.3
18.3
7S3Net
7
name
RNT01
40.8
39.4
39.3
33.6
38.3
18.8
20.1
24.8
20.5
21.1
LSeg 26
RNI01
52.8
44.4
38.5
47.4
22.1
25.1
21.5
23.4
LSeg
261
cls name
VIT-L
61.3
63.6
43.1
41.0
52.3
28.1
27.5
30.0
23.2
27.2
LSeg
25
attr (direct)
RNI01
48.6
51.2
39.7
36.2
44.0
20.9
23.6
20.8
18.4
20.9
LSeg
(pre-)
RN101
49.1
51.4
35.9
44.3
21.7
24.5
22.0
19.6
22.0
LSeg
25
attr (post-)
RN101
50.0
52.2
42.1
36.8
45.3
21.2
24.0
22.5
19.2
21.7
AttrSeg (Ours)
attr (hrchy)
RNI01
52.9
55.3
45.0
43.1
49.1
27.6
28.4
26.1
22.7
26.2
AttrSeg (Ours)
attr (hrchy)
VIT-L
61.5
67.5
46.1
50.5
56.4
34.8
32.6
31.6
24.2
30.8
PASCAL-5' and COCO-20'. We compare our method on PASCAL-5* and COCO-20' with state- of-the-art OVSS models SPNet [46], ZS3Net 7 and LSeg (25). We also compare different textual input settings for LSeg. As shown in Tab. I our method outperform the competitive baselines. Note that, heavily relying on VLP that aligns image modality with its textual class name, it is undeniable that class names hold core and most direct information. Existing open-vocabulary segmentation models are inherently bound by this characteristic, i.e., adding challenges when using only attributes that describe categories from indirect perspectives for segmentation. As representative, LSeeg shows weakness when the textual inputs are attributes descriptions only. In contrast, our method demonstrates good potential for segmenting the target object described by attributes through hicrarchical aggregations, even surpass the SOTA with class name input.
PASCAL-Context and PASCAL VOC. We also
Table 2: Evaluation 59) and PASCAL VOC (PAS-20) with attributes input using RN101 backbone.
(PC-
evalute our method on PASCAL-Context (PC-59)
and PASCAL VOC (PAS-20), to compare with re-
cent sfate-of-the-ar OVSS works Zegformer (II).
Model
Training Data
PC.59
PAS 20
OVSeg 27 and CAT-Seg [10]. They are trained on larger dataset COCO-Stuff [ with more classes and more images. As shown in Tab. 2 our method still demonstrates superiority. Besides, compared with method trained on COCO Stuff and PASCAL VOC-15, the performance increased dramatically.
LSeg 25
PASCAL VOC-15
24.2
44.0
Fusioner 32
PASCAL VOC-15
25.2
44.3
AttrSeg (Ours)
PASCAL VOC-15
29.1
49.1
Zegformer [
COCO-Stuff
39.0
83.7
OVSeg
COCO Stuff
90.3
CAT-Seg [0]
Stuff
53.6
90.9
Attroeg (Ours)
COff-Stuff
56.3
91.6

---

Table Evaluation on Fantastic Beasts (using checkpoints transferred from PASCAL-5' and COCO-20"). For textual inputs, we compare "cls name" (category names) and "attr" (attributes). "direct", "pre-", "post-" and "hrchy" are four aggregation strategies. 5' and 20' refer to the best checkpoints from the 1t fold for evaluation.
PA
COCO-20'
Model
Settings
Backbone
3"
5'
5
5
mloU
20'
20'
20
20'
mloU
LSeg
P5
cls name
RN101
<10
<10
<10
<10
<10
<10
<10
<10
<10
<10
LSeg
25
attr (direct)
RNI01
44.8
46.6
46.4
46.2
46,0
50.1
51.2
51.7
49.3
50.6
LSeg
25
attr (pre-)
RN101
46.4
47.7
48.2
47.3
47.4
51.3
52.7
52,9
50.9
52.0
1.Seg
attr (post-)
RNID1
46.9
48.3
48.8
47.7
47.9
52.8
52.5
52.4
51.2
52.2
AttrSeg (Ours)
attr (hrchy)
RNI01
50.7
53.4
53.6
51.3
52.3
56.4
54.9
55.7
55.3
55.6
AttrSeg (Ours)
attr (hrchy)
VIT-L
54.1
55.8
55,4
54,5
55.0
59.2
58.8
58.3
58.5
58.7
Fantastic Beasts. We take Fantastic Beasts as an outstanding representative of the real- world that may encounter various situations like ambiguity, neologism and unnameability. To simulate real-world scenarios, we directly transfer the checkpoints of previous methods and ours trained from the their corresponding datasets for evaluation. As shown in and Tab. our method establish new solid base- line. The existing SOTA methods are not able to handle these scenarios when taking class name as input. Despite using CLIP 42), which implicitly aligns visual and language features to some extent, these methods performance suffers due to the presence of unfamiliar words that break this alignment. However, when decomposing into attributes and then aggregating them, the performance of these methods remarkably increase, showing significant gains with robust performance. This demonstrates the universality and effectiveness of our motivation.
Table 4: Evaluation on Fantastic Beasts with attributes input using RN101 backbone (using checkpoints trans- ferred from the corresponding training datasets).
Model
Training Data
mloU (cls)
mloU (altr)
LSeg 25
PASCAL VOC-15
<10
46.0
Fusioner 32]
PASCAL VOC:
<10
46.1
Attröeg (Ours)
PASCAL VOC-15
52.3
Zegformer [I]
COCO-Stuff
<20
55.7
OVSeg 27
COCO-Stuff
<20
58.1
Seg 10
COCO Suuf
<20
59.4
Attrreg (Ours)
COCO-Stuff
61.9
4.2 Ablation Study
Various Aggregation Strategies. Here, we present comparison of different aggregation methods, as de- picted in Tab. [5] The results demon- strate that: (1) "dirctt" gives the poor- est results, suggesting that listing all attributes in one sentence cannot pro- vide sufficient interactions between to-
Table 5: Effectiveness of various aggregation strategies on PASCAL-5" with RN101 backbone.
LSeg
Ours
Strat
5"
5'
52
5
mloU
5"
5'
5²
G
mloU
Direct
48.6
51.2
39.7
36.2
44.0
48.8
51.4
39.7
39.2
44.8
Pre-
49.1
51.4
40.9
35.9
44.3
49.8
52.8
41.7
40.3
46.1
Post-
50.0
52.2
42.1
36.8
45.3
50.3
53.0
42.3
41.0
46.7
Hrchy
52.9
55.3
45.0
43.1
49.1
kens. (2) Both the "pre-" and "post-" both exhibit improvements compared with "direct". This may stem from the exchange of information between attribute and visual modalities within the open- vocabulary model. (3) In most cases, "Post-" performs better than "pre-", indicating that attribute aggregation after the model can facilitate greater interactions between different modalities. (4) is not necessarily optimal to have excessive or insufficient number of interactions throughout the entire process. Our proposed hierarchical aggregation method accounts for the regularity that the visual component learns differently at different stages. This necessitates its attribute counterpart to possess varying hierarchy. For instance, lower stages may focus on learning low-level features, requiring more attributes that represent local regions. Conversely, higher stages may concentrate on global information, necessitating fewer high-level attributes that are semantically abstract.
Components in Aggregation Module. We conduct ablation studies to investigate the impor- tance of each component in our clustering mod- ule, as illustrated in Tab. 6 The result show that (1) the full module with all components achieves the best performance. (2) Cross attention on im- ages contribute more compared with on attributes. As the aggregation among attributes finally aims to recognize the object in the image, it's impor- tant for the cluster tokens attending to the visual feature. (3) If no cross attentions given, the performance further degraded. This demonstrates the
Table 6: Ablation of components in clustering mod- ule on PASCAL-5 with ViT-I backbone.
Comp
Cross Atta
Mixer
5"
5
5²
5a
mloU
Img
Attr
Full
d
61.5
67.5
46.1
50.5
56.4
x
59.2
65.6
44.1
47.4
54.1
Cross
Alm
59.6
65.8
44.4
48.6
54.6
x
x
/
58.6
65.0
42.9
46.4
53.2
Mixer Mini
/
x
60.7
67.0
45.6
49.6
55.7
x
x
x
58.4
64.9
42.4
46.0
52.9
9

---

cross attentions on two modalities enable the learnable cluster centers match better with the correct attributes. (4) Mixer also plays a role in the bottom part of the module, as helps to propagate and exchange information between clusters for further aggregation. (5) When neither of these is applied, only the Slot is introduced, which, unsurprisingly, yields the poorest result.
Numbers of Decomposed Attributes.
Table 7: Ablation of #Attributes inputs and #Stages on PASCAL-5' with ViT-L backbone.
Tab. 7 shows the results when given input numbers of attribute (#Attr) changes. As #Attr decreases, the avail- able information diminishes, poten- tially resulting in incomplete object description. In general, such a reduc- tion in attribute quantity can lead to a significant decline in performance. However, our method demonstrates a certain level of robustness even when #Attr decreases from 15 to 10, and even 5. This resilience can be ascribed to the fact that the attributes are randomly sampled (with repetition) from the attribute set during training. Consequently, the attributes describing specific category do not necessarily need to be complete or highly informative for the model to generate accurate outputs.
#Attr
#Stage
#Tokens/Stage
5
5
5²
5
mloU
10,,10, 1)
G1.0
67.3
46.9
50.7
56.5
10,,)
61.5
67.5
46.1
50.5
56.4
15
10,1)
60.1
66.2
44.7
48.9
55.0
(15,
57.8
63.5
41.9
46.4
52.4
(10,5,3,211
59.4
65.8
44.9
49.2
54.8
(10,5,3,11
59.6
66.1
44.7
49,9
55.1
10
(10,
58.0
64.8
43.8
47.1
53.4
(10,1)
55.9
62.8
42.5
45.4
51.7
1,,4,
51.8
62.9
41.9
44.8
50.4
1,
52.4
63.1
43.7
45.2
51.1
(5.3.1)
52.1
62.7
43.0
45.4
50.8
(5, 1)
49.2
61.6
42.1
44.2
49.3
Numbers of Aggregation Stages. As shown in Tab. 7 we reduce the numbers of stage (#Stage) from 5 to 2. In each stage, we attempt to aggregate to half of the tokens until there is only one left. In most cases, less aggregation stages cause deficient results, and more stages result in better performance. However, blindly increasing stages do not necessarily lead to better performance. For #Attr is or 10, 5 stages aggregation lead to a decrease compared with 4 stages. A reason may be that too much aggregation operation excessive for limited attributes, which may disrupt the hierarchy inside, thus negatively affects the result. Considering the trade-off, we choose 4 stages in our method.
Impact of Inaccurate/Incorred Attributes. In real- world scenarios, attribute decomposition may include slight noise. Our aggregation module exhibits certain level of robustness to noise during both training and infer- ence. (1) For inaccurate attributes (attributes that are not visible in a specific image, but are still related to the class, such as "four-legged" to "dog" in dog lying down im- age), during training, we randomly select attributes from the attribute pool for given class, which means that the model trained with potentially noisy and inaccurate at- tributes. However, demonstrated in Tab. g our model can learn to ignore these inaccurate attributes during aggregation, and instead focus on other attributes to produce correct segmentation results. (2) For incorrect attributes (attributes that are completely unrelated to the target class, such as "red" to "dog"), ) a naive approach would be to first filter the input attributes using existing VLMs, and then select the top related attributes for downstream processing. Tab. g assess the impact of incorrect attributes and VLM filtering on inference. The results indicate simple VLM filtering has the effect.
Table Ablation of the impact of inaccu- rate/incorrect attributes during decompo- sition, and the VLM filtering strategy.
Clean Inaccurate Incorrect Attr
VLM Filtering
mloU
Attr
Attr
59.5
59.1
55.4
58.9
Types of Decomposed Attributes. We roughly categorize the decomposed attributes into four types: color, shape, parts, and others, and maintain consistent total number of inputs. As shown in Tab. [ all types of attributes contribute to the overall performance, highlighting the significance of attribute diversity.
Table 9: Ablation of decomposed at- tributes' types on Fantastic Beast with RN101 backbone.
Color
Shape
Parts
Others
mloU
30.8
42.5
5 Conclusion
51.0
52.3
We pioneer the early exploration in utilizing only attribute descriptions for tation, and provide detailed attribute descriptions for two types of existing datasets and one newly collected dataset. Based on this, we propose a novel attribute that first decouples class names into attribute descriptions, and then combines them into final class representations. Extensive experiments demonstrate the effectiveness of our method, showcasing its ability to achieve the state-of-the the performance across various scenarios.
segmen-
caampoostiionagggeggalee
framework
10

---

6 Acknowledgement
This work is supported by the National Key R&D Program of China (No. 2022ZD0160703), STCSM (No. 2251 1 106101, No. 22511 105700, No. 21DZ1 100100,, 111 plan (No. BP0719010) and National Natural Science Foundation of China (No. 62306178).
References
[1] Fantastic beasts and where find them, 2016. Director: David Yates, Distributor: Warnr Bros. Pictures.
[2] Fantastic beasts: The crimes of grindelwald, 2018. Director: David Yates, Distributor: Warner Bros.
Pictures.
[3] Fantastic beasts: The secrets of dumbledore, 2022. Director: David Yates, Distributor: Warner Bros. Pictures.
[4] Donghyeon Back, Youngmin Oh, and Bumsub Ham. Exploiting a joint embedding space for generalized zero-shot semantic segmentation. In Int. Conf. Comput. Vis., pages 9536-9545, 2021.
[5] María Alejandra Bravo, Sudhanshu Mittal, Simon Ging, and Thomas Brox. Open-vocabulary attribute detection. ArXiv, abs/2211.129) 2022.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Adv. Neural Inform. Process. Syst., 33: 33:1877-1901, 2020.
[7] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zera-shot semantic segmentation. Adv.
Neural Inform. Process. Syst. 32, 2019.
[8] H. Caesar, J. Uijlings, and V. Ferrari. Coco-stuff: Thing and stuff classes in context. In IEEE Conf. Comput.
Vis. Puttern Recog. pages 1209-1218, Los Alamitos, CA, USA, jun 2018. IEEE Computer Society.
[9] Keyan Chen, Xiaolong Jiang. Yao Hu, Xu Tang. Yan Gao. Jianqi Chen, and Weidi Xie. Ovarnet: Towards open- vocabulayy object attribute recognition. ArXiv, 02422301.95066 2023.
[10] Seokju Cho, Heeseong Shin, Sunghwan Hong, Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Sco, and Scungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. arXiv preprint arXiv:2333. 1797, 2023.
[11] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. 2022.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlshy. An image is worth 16x words: Transformers image recognition scale. In Int. Conf. Learn. Represent., 2021.
[13] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Int. J. Comput. Vis., 88(2)3003-338 2010.
[14] Golnaz Ghiasi, Xiuye Gu. Yin Cui, and Tsung- Yi Lin. Scaling open vocabulayy image segmentation with image-level labels. In Eur. Conf. Comput. Vis., pages 540-557 Springer, 2022.
[15] Zhangxuan Gu. Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1921-1929, 2020.
[16] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In Int. Conf. Comput. Vis., pages 991 1998. IEEE, 2011.
[17] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yunhsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-languae representation learning with noisy text supervision. In Int. Conf. Mach. Learn., pages 4904-4916, 2021.
[18] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. Eur. Conf. Comput. Vis., 2022.
[19] Chen Ju, Zeqian Li, Peisen 7hao, Ya Zhang, Xiaopeng Zhang, Qi Tian, Yanfeng Wang, and Weidi Xie. Multi-modal prompting for low-shot temporal action localization. arXiv preprint arXiv:2303.11732 2023.
[20] Chen Ju, Haicheng Wang, Jinxiang Liu, Chaofan Ma, Ya Zhang, Peisen Zhao, Jianlong Chang. and Qi Tian. Constraint and union for partiall-supervieed temporal sentence grounding. arXiv preprint arXiv:2302. 09850, 2023.
[21] Chen Ju, Kunhao Zheng, Jinxiang Liu, Peisen Zhao, Ya Zhang, Jianlong Chang, Qi Tian, and Yanfeng Wang. Distilling vision- languaee pre-training to collaborate with weakly-supervised temporal action localization. In JEEE Conf. Comput. Vis. Patter Recog., 2023.
11

---

[22]
Elyor Kodirov, Tao Xiang, and Shaogang Gong, Semantic autoencoder for zero-shot learning. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3174-3183, 2017.
[23]
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 951-958. IEEE, 2009.
[24] Boyang Li, Fei Zhang, Longguang Wang. Yinggian Wang. Ting Liu, Zaiping Lin, Wei An, and Yulan Guo. Ddaug: Differentiable data augmentation for weakly supervised semantic segmentation. IEEE Trans. Multimediaa 199):1-12, 2023.
[25] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl, Language-driven semantic segmentation. In Int. Conf. Learn. Represent., 2022.
[26] Peike Li, Yunchao Wei, and Yi Yang. Consistent structural relation learning for zero-shot segmentation. Adv. Neural Inform. Process. Syst., 33:10317-1000 2020.
[27] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In JEEE Conf. Comput. Vis. Pattern Recog., pages 7061-7070, 2023.
[28] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Eur. Conf. Comput. Vis., pages 740-755. Springer. 2014.
[29] Jinxiang Liu, Yu Wang. Chen Ju. Chaofan Ma. Ya Zhang, and Weidi Xie. Aanotation-free audio-visual segmentation. Winter Conf. App. Comput. Vis., 2024.
[30] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Adv. Neural Inform. Process. Syst. 33:11525 1538, 2020.
[31] Chaofan Ma, Yuhuan Yang. Chen Ju, Fei Zhang. Jinxiang Liu, Yu Wang. Ya Zhang, and Yanfeng Wang. Diffusionseg: Adapting diffusion towards unsupervised object discovery. arXiv preprint arXiv:2303. 2023.
[32] Chaofan Ma, Yuhuan Yang. Yanfeng Wang, Ya Zhang, and Weidi Xie. Open-vocabulary semantic segmentation with frozen vision-language models. In Brit. Mach. Vis. Conf,, 2022.
[33] Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang, Xin Wang, and Carl Vondrick. Doubly right object recognition: A why prompt for visual rationales. arXiv preprint arXiv: Arxiv-2212.0623 2022.
[34] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In Int. Conf. Learn. Represent., 2023.
[35] Roozbeh Mottaghi. Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho. Seong- Whan Lee, Sanja Fidler. Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In IEEE Conf. Comput. Vis. Pattern Recog., pages 891-898, 2014.
[36] Gregory L. Murphy and Douglas L. Medin. The role of theories in conceptual coherence. Psychological review, 92(3):289-311 316, 1985.
[37] Zachary Novack, Saurabh Garg, Julian McAuley, and Zachary C Lipton. Chils: Zero-shot image classifica- tion with hierarchical label sets. arXiv preprint 202551 2023.
[38] OpenAL. Chatgpt: Optimizing language models for dialogue, 2022.
[39] Khoi Pham, Kushal Kafle, Zhe L. Lin, Zhi Ding, Scott D. Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes the wild. IEEE Conf. Comput. Vis. Pattern Recog.. pages 13013-
13023, 2021.
[40] Khoi Pham, Kushal Kafle, Zhe L. Lin. Zhihong Ding. Scott Cohen, Quan Tran, and Abhinav Shrivastava. Improving closed and open- vocabulary attribute prediction using transformers. In Eur. Conf. Comput. Vis.. 2022.
[41] Sarah Pratt, Rosanne Liu, and Ali Farhadi. What does platypus look like? generating customized prompts for zero-shot image classification. arXiv preprint arXiv:2209.0 2332000000 2022.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn., pages 8748-8763. PMLR, 2021.
[43] Bernardino Romera- Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learning. In Int. Conf. Mach. Learn., pages 2152 2161. PMLR, 2015.
[44] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner. Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mixemixer: An all-mlp architecture for vision. Adv Neural Inform. Process. Syst., 34:24261-22277777 2021.
12

---

[45] Edward J. Wisniewski and Bradley C. Love. Relations versus properties in conceptual combination. Journal Memory and Language, 38(2):177-202, 1998.
[46] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-ant zero-and few-label semantic segmentation. In JEEE Conf. Comput. Vis. Pattern Recog., pages 8256 8265, 2019.
[47] Yuhuan Yang. Chaofan Ma. Chen Ju. Ya Zhang, and Yanfeng Wang. Multi-modal prototypes for open-set semantic segmentation. arXiv preprint arXiv:2307.020000 2023.
[48] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: aineggrained interactive language-image pre- pre-training. In Int. Conf. Learn. Represent., 2022.
[49] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: new foundation model for computer vision, arXiv preprint arXiv:21 I. 1432, 2021.
[50] Fei Zhang, Chaochen Gu, Chenyue Zhang, and Yuchao Dai. Complementary patch for weakly supervised semantic segmentation. Int. Conf. Comput. Vis., pages 7242-7251, 2021.
[51] Fei Zhang, Tianfei Zhou, Boyang Li, Hao He, Chaofan Ma, Tianjiao Zhang. Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Uncovering prototypical knowledge for weakly open- veen-vocabulary semantic segmentation. Adv. Neural Inform. Process. Syst., 2023.
[52] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In Eur. Conf. Comput. Vis., 2022.
13


Reviews:
[{"review_id": "sGgCcdKDCu", "reviewer": "Reviewer_cgtT", "summary": "This paper proposes a framework for decomposing class names into attributes for open vocabulary semantic segmentation. The authors obtain attributes from LLMs or hand-curate for unheard-of classes, then aggregates these attributes for obtaining a general descriptor for each classes. Moreover, the authors collect a dataset to demonstrate their effectiveness on addressing neologisms and unnameability for OVSS.", "questions": "1. The collected dataset, \"Fantastic Beasts\" is collected from a movie series, which may raise concerns regarding copyright issues. Can the  authors include discussions, or clarify how the copyright may not be of an issue?", "limitations": "Limitations regarding the collection of the dataset should be further discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation for the paper, addressing the problems existing in pre-trained VLM-based paradigm for OVSS seems solid.\n2. Evaluating the proposed method on a newly collected dataset, consisted of imaginary creatures, is an interesting method for evaluating how models can handle \"neologism\" and \"unnameability\" addressed in the paper.", "weaknesses": "1. Insufficient evidence for supporting the work\n- The authors evaluate with COCO-20i[19] and PASCAL-5i[10, 13], which have brief, but well-known classes. These classes barely seem to suffer problems such as neologism or unnameability, and does not illustrate cases where ambiguity might be an issue.\n- Moreover, the collected attributes for COCO and PASCAL are not present in the paper or the supplementary materials, making it harder to validate how these attributes may contribute to the quality of the output segments.\n\n2. Weaknesses in quantitative comparison and ablations\n- The current comparison with LSeg[17] does not seem to sufficiently provide evidence for the core contribution of the work of attribute decomposition-aggregation, as LSeg[17] does not improve with provided attributes. \nThe improvements may seem to come form the fusion layers, which seems to beg similarity with Fusioner[21]. Comparison with Fusioner[21] would better illustrate the significance of this work, as it would be able to solely ablate the proposed attribute aggregation.\n\n3. Flaws for addressing the presented problem\n- If the class names are ambiguous by their own, how are LLMs capable of distinguishing between this ambiguity? For example, how would LLMs be able to distinguish between the bird \"crane\" and the machine \"crane\" with the following prompt \"Describe what a {crane} looks like in the image.\", without further elaborating the context?\n- The claim that \"segmentation through attributes is a more challenging task\"(L.281) seems to be a bold claim, as the authors obtain the attributes as additional information from the class names, with the help from LLMs or human annotation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging our novelty and motivation, we address your kind concerns below.\n\n> **1.1. COCO/PASCAL have well-known classes without neologism etc**\n\nThere may be some misunderstandings. We admit COCO/PASCAL have well-known classes. However, the aim of using them are **_not_** intended for neologism or unnameability. Instead, as we pioneer the segmentation exploration via attribute, there's no existing benchmark to meet our requirements. We hence annotate attributes on widespread COCO/PASCAL.\n1. For training, these datasets give *representative simulation environments*, i.e., image-mask-attributes pairs with well-defined 4 folds for cross validation. Models training on this dataset can successfully adapt to real-world scenarios, such as Fantastic Beasts, with neologism or unnameability. This proves the rationality of this environment.\n2. For evaluation, COCO/PASCAL are *standard benchmarks* to show our effectiveness(admit by R-fEW5&mZMs). Besides, they also allow us to easier assess the performance of other established baselines, facilitating understanding.\n\nFurthermore, as suggested, we also included datasets used in SOTA of OVSS for additional comparison (see Table 1,2).\n\n> **1.2. Attributes for COCO/PASCAL are not shown**\n\nWe apologize for the inconvenience. Limited by page, we only provide some attributes of PASCAL. Whole list attached in supplementary of the revised version.\n```\nbird: two wings,two legs,a beak,feathers,a tail,bright colors,a curved neck,small eyes,a pointed head,a small body,a pointed beak,webbed feet\n```\n\n> **2.1. The comparison with LSeg does not seem to be sufficient, as LSeg does not improve with attributes**\n\nWe believe there are some misunderstandings.\n1. For common classes, our intention is not to rely solely on attributes inputs to surpass the SOTA that uses class name inputs. As clarified in the general response, attribute-based segmentation presents greater challenges.\nWithout considering the crucial information of class names and comparing it to class name-based methods would be unfair.\n2. A fair comparison is to use class name with attributes as input for the usual categories (see Table 6 of R-mZMs), to show attributes input can supplement incomplete class information.\n3. When faced with real-world scenarios that have possibility of neologisms or unnameability, attribute-based methods are significantly better than that with class name inputs (see Table 2 of the general response). This directly demonstrates the effectiveness of our idea.\n4. To our knowledge, we are the first to work on attribute-based segmentation. Our initial successful exploration aims to lead the community's attention towards this practical scenario, thus promoting further development.\n\n> **2.2. Improvements come from the fusion layers? Comparison with Fusioner**\n\nWe should like to clarify that improvements do not mainly come from the fusion layers.\nTable 3 below gives comparisons with Fusioner. Results show the gain mainly comes from the clustering module. The fusion layer only gives 1.2% mIoU gains on average, compared with a greater 4.3% of clustering module. Besides, for more comprehensive comparisons, please refer to Table 1,2 in the general response.\n\n*Table 3: Ablation of fusion layer and aggregation module on PASCAL-5$^i$.*\nModel (Components)|Settings|5$^0$|5$^1$|5$^2$|5$^3$|mIoU|$\\Delta$w/\"full\"\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nFusioner|cls name|46.8|56.0|42.2|40.7|46.4|-\n|\nOurs (full)|attr|52.9|55.3|45.0|43.1|49.1|-\nOurs (w/o fusion layer)|attr|52.0|54.6|43.1|41.9|47.9|$\\downarrow$ 1.2\nOurs (w/o clustering module)|attr|48.8|51.4|39.7|39.2|44.8|$\\downarrow$ 4.3\n\nAnd although both work use fusion-like layers, ours is very different from Fusioner:\n1. Fusioner belongs to class-name-based segmentation method like LSeg, while we are attribute-based without class name.\n2. Fusioner designes fusion layers to merge image and text features, while we aim to information propagation, i.e., exchanging visual information with grouped attribute features and learnable cluster tokens for each stage.\n\n> **3.1. LLMs cannot distinguish ambiguous names**\n\nWe agree, LLMs cannot distinguish ambiguity without additional inputs, and human is required at this point. However, the intention of this work is *not* to eliminate ambiguity *for LLM*s. As a pioneer for attribute segmentation, our main challenge lies in building standard datasets with minimal cost. Using LLMs to assist proves to be an efficient approach, particularly when dealing with many classes. Human annotators only need to manually filter and verify generated attributes, which significantly reduces costs.\n\n> **3.2. \"segmentation via attributes is more challenging\" seems to be bold**\n\nWe believe that there are some misunderstandings. Please see the general response for detailed clarification.\nAttribute segmentation is more challenging can also be observed from Table 1 in the original paper (LSeg (attr) vs LSeg (cls name)), especially for common classes. Nevertheless, for common classes, this issue can be solved by combining class names and attributes as input. Further details are shown in Table 6 (R-mZMs).\n\n> **4. computational concerns**\n\nComputational is not a concern, and it's not the primary focus.\nWe transform OVSS into multiple binary segmentation tasks, rather than treating it as a multi-class segmentation with inputting multi-class attributes simultaneously. \n1. The computation time may demonstrate a linear relationship with class number. This is acceptable for our pioneer exploration.\n2. The GPU memory is significantly reduced, since the input sequence is much shorter, only 20 tokens for all classes compared to 20*N tokens for N classes.\n\nOverall, our key goal is to show the value of attribute decomposition-aggregation. Futures work may explore ways to reduce computational time.\n\n> **5. Copyright concern**\nWe believe it is \"fair use\" and is not an infringement of copyright. We will contact the movie company to validate."}, {"review_id": "7qOHRfdupV", "reviewer": "Reviewer_fEW5", "summary": "Previous work does not address the issue of ambiguity or potentially incomplete textual categories and directly incorporates them into vision-language models without any post-processing. This approach hinders the text encoder's ability to fully comprehend the category. To overcome this limitation and provide comprehensive categorical information, the author suggests decomposing the category into more descriptive texts, such as attributes, using large language models (or human-designed prompts). Subsequently, the decomposed texts are aggregated using the proposed method.", "questions": "The authors chose to aggregate visual and attribute information together in one model. However, considering the pre-training nature of vision-language models, the most naive way to achieve the same goal would be to concatenate all the attributes into a sequence and input them into a text encoder. The resultant text embedding can then be used as a classifier, which can be applied to existing methods such as Zegformer or OVSeg. I'm curious about the performance of this naive method, which is the most easiest way to confirm the validity of the motivation of decomposition. ", "limitations": "The motivation seems promising; however, I think this motivation can be applied to all of the existing methods in open-vocabulary literature. In other words, the method the author suggests does not seem to emphasize the advantage of the motivation but rather has high similarity to previous work (Fusioner [1]). This paper would be much stronger if the author provided an analysis regarding the motivation instead of focusing solely on the technical design.\n\n[1] Ma et. al., \"Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models\", BMVC'22\n[2] Ding et. al., \"Decoupling Zero-Shot Semantic Segmentation\", CVPR'22\n[3] Liang et. al., \"Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP\", CVPR'23\n[4] Cho et. al, \"CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation\", arXiv'23", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Sound motivation.\nThe core idea of distinguishing categories by decomposing them into attributes and aggregating them sounds promising.\n\n2. Suggesting a new dataset to support the motivation.\n", "weaknesses": "1. Missing citations.\nThe authors should include citations to recent open-vocabulary semantic segmentation literature and make a comparison with them [2, 3, 4]. \n\n2. Although the motivation is good, the experiment is insufficient.\nThe benchmark conducted by the authors is not sufficient to provide a complete comparison with recent open-vocabulary segmentation works. The authors conducted experiments on benchmarks widely used in few-shot segmentation literature, rather than in open-vocabulary semantic segmentation. To the best of my knowledge, only two papers have utilized this benchmark in an open-vocabulary setup (LSeg, Fusioner [1]). This choice prevents a comparison with recent state-of-the-art open-vocabulary methods such as Zegformer [2], OVSeg [3], or CAT-Seg [4].\n\n3. Lack of analysis of the motivation for **decomposition**.\nThe provided analyses primarily focus on demonstrating the effectiveness of the aggregation modules, rather than specifically proving the effectiveness of the decomposition approach. It would be beneficial for the author to include ablation studies and analyses of the **decomposition** within the main paper.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n## Reviewer fEW5\n\nThank you for your appreciations on the novelty, motivation, and results of the idea, as well as the detailed and constructive review. \nWe address individual questions and concerns below.\n\n> **1. Missing citations**\n\nThanks for pointing out.\nWe will add these literatures in related works, and the comparisons such as Table 1, 2 in general response will also be added into the revised paper.\n\n\n> **2. Comparison with recent state-of-the-art open-vocabulary methods such as Zegformer, OVSeg, or CAT-Seg**\n\nThanks for acknowledging our motivation and the constructive suggestions.\nAs reviewer suggested, we have added comparisons with recent SOTA methods in Table 1, 2 in the general response. \nNote that, the mentioned papers are evaluated on ADE20K, PASCAL VOC, and PASCAL-Context datasets, and they are trained on a larger dataset COCO-Stuff with more classes and more images.\nConsidering the burden of annotating the attributes for all datasets during rebuttal, we here annotate attributes of some representative dataset: COCO-Stuff for training, and PASCAL-Context (PC-59) and PASCAL VOC (PAS-20) for evaluation.\n\n\n> **3. Ablation studies of the decomposition**\n\nGood suggestions!\nWe conducted two types of ablations on Fanatic Beasts: \n\n(1) Number: varying the number of attributes used for class decomposition (Table 4). \nIncreasing the number of attributes resulted in improved performance.\n\n(2) Diversity: exploring different types of decomposed attributes (Table 5). We roughly categorize the decomposed attributes into four types: color, shape, parts, and others, and maintain a consistent total number of inputs.\nAll types of attributes contribute to the overall performance, highlighting the significance of attribute diversity.\n\n\n*Table 4: Ablation on the number of the attributes for the class decomposed into.*\n| Number of decomposed attr |  5   |  10  |  15  |\n| :-----------------------: | :--: | :--: | :--: |\n|           mIoU            | 47.8 | 50.9 | 52.3 |\n\n\n\n*Table 5: Ablation on different types of decomposed attributes.*\n| Color | Shape | Parts | Others | mIoU |\n| :---: | :---: | :---: | :----: | :--: |\n|   √   |       |       |        | 30.8 |\n|   √   |   √   |       |        | 42.5 |\n|   √   |   √   |   √   |        | 51.0 |\n|   √   |   √   |   √   |   √    | 52.3 |\n\n\n\n\n> **4. The performance of a naive method which can confirm the validity of the motivation of decomposition**\n\nWe totally agree with the reviewer that \"the most naive way to achieve the same goal would be to concatenate all the attributes into a sequence and input them into a text encoder\".\nIn our work, we have referred to this approach as \"direct aggregation,\" as indicated in Figure 3.(a) of the original paper.\n\nTo compare various aggregation strategies, we had conducted an ablation in Sec.4.2, *i.e.*, Table 3, in the original paper. This ablation mainly focuses on our method and LSeg on PASCAL.\nAdditionally, as suggested by the reviewer, \nwe extended experiments on \"direct aggregation\" to existing SOTA methods such as Zegformer or OVSeg, on multiple benchmarks.\nThe results of these experiments are presented in Table 1 and 2 of the general response.\n\nOverall, all the results demonstrate that, \"direct aggregation\" strategy is generally effective, especially in real-world scenarios where neologisms and unnameable objects may occur. \nOur proposed hierarchical aggregation achieved superior results.\nThis definitely confirm the validity of the motivation.\n\n\n\n> **5. The motivation can be applied to all of the existing methods in open-vocabulary literature**\n\n\nWe agree with the reviewer that the motivation can be applied to all of the existing methods in open-vocabulary literature.\n \nThe motivation of our work is to first decompose class names into attributes, \nwhich can solve the issue of ambiguity, neologisms and unnameability, then aggregate them for segmentations.\nThis can be equipped on all recent SOTA methods on-the-shelf.\nAs shown in Table 2 in the general response, the proposed decomposition-aggregation ideology successfully validate its effectiveness in challenging real world scenarios that may encounter various situations like ambiguity, neologism and unnameability.\n\n\n"}, {"review_id": "mZLzr1D8BF", "reviewer": "Reviewer_mZMs", "summary": "[Task] This paper presents a novel decomposition-aggregation framework for open-vocabulary semantic segmentation, which addresses the challenges of segmenting novel object categories during inference. The existing vision-language pre-training approaches make unrealistic assumptions about low-quality textual category names, such as the availability of accurate and complete category descriptions in pre-trained lexicons. This work aims to overcome these limitations and proposes a framework inspired by human cognition for understanding new concepts.\n\n[Method] The framework consists of two stages: decomposition and aggregation. In the decomposition stage, class names are decoupled into diverse attribute descriptions to enrich semantic contexts. Two attribute construction strategies are employed: leveraging large language models for common categories and manual labeling for human-invented categories. In the aggregation stage, diverse attributes are grouped into an integrated global description, forming a discriminative classifier that distinguishes the target object from others. A hierarchical aggregation technique is introduced to achieve multi-level alignment and deep fusion between vision and text. The final segmentation result is obtained by computing the embedding similarity between aggregated attributes and images.\n\n[Experiments] To evaluate the effectiveness of the proposed approach, three datasets with attribute descriptions are annotated, and extensive experiments and ablation studies are conducted. The results demonstrate the superior performance of the attribute decomposition-aggregation framework. The paper concludes by mentioning the release of datasets and codes upon publication, which will enable further exploration and validation of the proposed method.", "questions": "My main questions are listed in the weakness sections. I have a few more questions on the ablation study and model evaluation as follows:\n\nIn addition to evaluating with exact class names, it would be beneficial to know if the proposed approach can handle hybrid inputs. For instance, can it work with rough class names such as \"dog\" without specifying the breed (e.g., \"corgi\"), combined with additional textual descriptions? Supporting hybrid inputs would enhance the flexibility and practicality of the proposed framework, enabling more realistic scenarios where users might provide partial or general class names along with descriptions. Addressing these concerns would provide a better understanding of the comparative performance of the proposed framework and its compatibility with evaluating segmentation results using hybrid inputs.", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[Novel Framework]: The paper introduces a novel decomposition-aggregation framework that addresses the challenges of open-vocabulary semantic segmentation. By drawing inspiration from cognitive psychology, the framework leverages diverse attribute descriptions to enhance category comprehension and improve segmentation performance.\n\n[Practical Relevance]: The paper highlights the limitations of existing vision-language pre-training approaches in real-world scenarios and addresses issues such as ambiguity, neologisms, and unnameability, which are critical for achieving accurate and comprehensive open-vocabulary semantic segmentation results.\n\n[Construction Strategies]: The paper proposes two construction strategies for generating attribute descriptions, utilizing language models and manual labeling. This provides flexibility and accommodates different scenarios, including both common categories and situations involving novel or difficult-to-describe categories. The inclusion of a hierarchical aggregation architecture enables the exploitation of potential attribute hierarchy, which can enhance the segmentation process and improve the quality of the final segmentation result.\n\n[Experimental Evaluation]: The paper evaluates the proposed framework on multiple datasets, demonstrating its superior performance over various baselines and competitors. The inclusion of thorough ablation studies further enhances the understanding of each component's contribution.", "weaknesses": "[Weak baselines] The paper primarily compares the proposed work to LSeg, which may be considered a weak baseline as it is not the latest work in the field of open-vocabulary semantic segmentation. The achieved performance gains with the proposed framework, using RN101 as a backbone, are relatively modest at around 3% (3% is modest since the baseline is LSeg rather than the current SOTA work in open-vocab semantic segmentation). Given the complexity of the framework and training pipeline, it raises questions about the overall effectiveness of the proposed approach. Can you compare your work to some latest works as well? \n\n[Performance evaluation with class names alone] The experimental results presented in Tables 2 and 3 demonstrate that the performance on the custom datasets Pascal-5 and COCO-20 is only comparable to LSeg (LSeg is evaluated based on class names). While the capability of evaluating the model's performance without relying on class names is intriguing, it is more common and practically useful to support evaluation with class names. Therefore, could you please clarify whether the proposed approach's performance would be influenced when class names are provided as part of the inputs? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer mZMs\n\nThank you for your appreciations on the novelty, contribution, and results of the idea, as well as the detailed and constructive review. \nWe address individual questions and concerns below.\n\n\n> **1. compare with some latest works** \n\n\nThanks for the suggestions, we have added the comparison with the latest works including Fusioner, Zegformer, OVSeg, and CAT-Seg as also suggested by R-cgtT and R-fEW5. This is shown in Table 1, 2 in the general response.\n\nWe would like to classify the reason of our method seems relatively modest in improvement.\nMost of latest works are trained on a larger dataset COCO-Stuff with more classes and more images (118k densely annotated training images with 171 categories), while ours are strictly following LSeg trained on PASCAL-5$^i$ with only 15 categories and fewer images.\nCompared with our method trained on COCO-Stuff and PASCAL VOC-15 in Table 1 in the general response, the performance increased dramatically.\n\n\n\n> **2. the proposed approach's performance when class names are provided as part of the inputs**\n\nGood suggestions!\nWithout making modifications of our model, \nwe design a simple strategy for class name alone as input (cls) and class name as part of the inputs (cls+attr). \\\n(1) For \"cls\", we simply repeat the class name tokens as the same number of attribute tokens, and send into our model. \\\n(2) For \"cls+attr\", we append class name token as one of the attribute token, then send into our model.\n\nAs shown in Table 6 below, \nwhen we send class name only, our model is comparable with the SOTA, even if it is not intentionally designed for inputting class names.\nAppending class name with attributes has further greatly facilitated the advancement of performance, and our model fully demonstrated its capability.\nThis emphasizes that attributes can effectively supplement missing information and address incompleteness of the class name input. \nWith this potential of practical functionality., we firmly believe that our model can make significant contributions to the research community.\n\n*Table 6: Performance of combining class names and attributes as inputs on PAS-20. All other methods is under \"direct aggregation\" strategy when possible.*\n|   Model   | Backbone | Training data | mIoU (cls) | mIoU (cls+attr) | $\\Delta$mIoU |\n| :-------: | :------: | :-----------: | :--------: | :-------------: | :----------: |\n|   LSeg    |  RN101   | PASCAL VOC-15 |    47.4    |      48.3       |     0.9      |\n| Fusioner  |  RN101   | PASCAL VOC-15 |    46.4    |      47.4       |     1.0      |\n| Zegformer |  RN101   |  COCO-Stuff   |    86.2    |      87.1       |     0.9      |\n|   OVSeg   |  RN101   |  COCO-Stuff   |    92.6    |      93.2       |     0.6      |\n|  CAT-Seg  |  RN101   |  COCO-Stuff   |    93.7    |      94.1       |     0.4      |\n| **Ours**  |  RN101   | PASCAL VOC-15 |    50.0    |      51.5       |   **1.5**    |\n| **Ours**  |  RN101   |  COCO-Stuff   |    93.3    |      94.6       |   **1.3**    |\n\n\n> **3. it would be beneficial to know if the proposed approach can handle hybrid inputs**\n\nWe agree with reviewer that provide partial or general class names along with descriptions may enabling more realistic scenarios.\nAnd we are happy to say that our model are able to handle hybrid inputs, such as rough class names (e.g., \"dog\") combined with additional textual descriptions (e.g., \"furry or smooth coat\", \"four-legged\", \"wet nose\", etc.).\nThis can be demonstrate in Table 6 above, with settings denote as \"cls+attr\".\n\nIt is worth noting that although breed (e.g., \"corgi\") can be seen as one of the attributes, our model can handle more general descriptions, for example, colors, shapes, object parts, material, etc., which largely increased the flexibility and practicality.\n\n"}, {"review_id": "SeTA8QmXV6", "reviewer": "Reviewer_PB8W", "summary": "The paper proposes a decomposition-aggregation framework to address the problem of open-vocabulary semantic segmentation. The framework consists of a decomposition stage, where pre-trained language models (LLMs) such as ChatGPT are used to extract textual attributes for class names, and an aggregation stage, where a hierarchical structure is employed to progressively aggregate attribute features. The experimental results demonstrate the effectiveness of the proposed method.", "questions": "When decomposing a class name into a set of attributes using a language model like ChatGPT, is it correct to say that the attributes generated by the language model are based on common sense rather than visual inspection, considering that the language model does not have access to the images?", "limitations": "Refer to Weaknesses", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea behind the framework is straightforward and easy to understand.\n2. The experimental results provide evidence that the proposed method is effective.", "weaknesses": "1. The performance of the framework heavily relies on the quality and accuracy of the selected attributes. Incorrect or inaccurate attributes may degrade the segmentation performance. It would be beneficial to explore ways to regulate the risk introduced during the decomposition stage, both during training and inference.\n2. While leveraging additional attributes from external knowledge improves the performance of open-vocabulary semantic segmentation (OVSS), it is important to consider whether explicitly decomposing class names into attributes align with the long-term goals of OVSS. Implicitly learning the \"decomposition and composition\" process might be a more desirable solution in the future.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n## Reviewer PB8W\n\nThank you for your valuable comments and suggestions, as well as acknowledging the novelty and results of the idea.\nWe address individual questions and concerns below.\n\n\n\n> **1. It would be beneficial to explore ways to regulate the risk introduced during the decomposition stage, both during training and inference.**\n\n\nWe agree that in real-world scenarios, attribute decomposition may include slight noise. \nWe considered this when designed, so our aggregation module exhibits a certain level of robustness to noise during both training and inference.\n\n(1) For inaccurate attributes (attributes that are not visible in a specific image, but are still related to the class, such as \"four-legged\" to \"dog\" in a dog lying down image):\nDuring training, we randomly select attributes from the attribute pool for a given class, which means that the model is trained with potentially noisy and inaccurate attributes. \nHowever, as demonstrated in Table 7 below, our model can learn to ignore these inaccurate attributes during aggregation, and instead focus on other attributes to produce correct segmentation results.\n\n(2) For incorrect attributes (attributes that are completely unrelated to the target class, such as \"red\" to \"dog\"):\nA naive approach would be to first filter the input attributes using existing VLP models such as CLIP, and then select the top related attributes for downstream processing. \nWe conducted an ablation study to assess the impact of incorrect attributes and VLM filtering on inference. \nAs shown in the table 7 below, the results indicate a simple VLM filtering has the effect.\n\nMore in-depth investigations belong to the area of \"learning with noisy labels\", which is beyond the scope of this paper. \nWe leave it as future work.\n\n\n*Table 7: Ablation of the impact of inaccurate/incorrect attributes and VLM filtering during decomposition.*\n| Clean attr | Inaccurate attr | Incorrect attr | VLM filtering | mIoU |\n| :--------: | :-------------: | :------------: | :-----------: | :--: |\n|     √      |                 |                |               | 59.5 |\n|     √      |        √        |                |               | 59.1 |\n|     √      |        √        |       √        |               | 55.4 |\n|     √      |        √        |       √        |       √       | 58.9 |\n\n\n\n\n\n> **2. Implicitly learning the \"decomposition and composition\"**\n\n\nWe appreciate the reviewer's suggestion regarding the potential future solution of implicitly learning decomposition and composition. \n\nSince the specific method for implicitly learning the \"decomposition and composition\" has not been explicitly mentioned by the reviewer, \nwe propose one possible approach for implicit modeling. \nIn this approach, given a category name as input, the model outputs N category representations, with each representation representing an attribute. \nThese attributes are then constrained by a loss function to ensure orthogonality among them. \n\nHowever, we believe that learning implicit attributes from raw data is not a trivial task and faces several limitations. \nIt incurs high computational costs, lacks interpretability, lacks direct constraints, introduces potential noise, and is hard to evaluating the quality and relevance of implicit attributes.\n\nThese challenges can be addressed more easily by explicitly decomposing attributes. \nOur work, as a pioneer in this area, mainly focuses on demonstrating the feasibility and effectiveness of attribute decomposition-aggregation. \nWe believe that our method can serve as a solid baseline approach in this field. \nBesides, we also provide a novel perspective for the community to address the existing limitations, thus making a significant step towards the ultimate goal of OVSS.\n\nFuture research could explore how to move beyond explicit attributes and instead learn representations that implicitly capture the compositional structure of novel object categories.\n\n\n\n\n> **3. is it correct to say that the attributes generated by the language model are based on common sense rather than visual inspection?**\n\nYes, this is correct.\n\nLLMs can successfully imitate visual attributes since they are trained on a large corpus containing descriptions with visual knowledge.\nBesides in LLM, a high-dimensional knowledge graph potentially exists, enabling the decomposition of text into attributes that do not necessarily depend on visual elements.\n\nWhen multi-modal foundation models, such as multi-modal GPT-4, become more advanced, the attribute generation may be able to base on common sense together with image content.\n\n\n"}]

Author rebuttal:
## General Response and Clarifications

We kindly thank the reviewers for their detailed reading, and considerate feedback. We are also grateful of the appreciations by the reviewers in these aspects:

(1) **Novel and well-motivated idea**: "motivation seems solid" (R-cgtT); "sound motivation and promising" (R-fEW5); "novel framework, limitations of existing works are critical" (R-mZMs); "idea is straightforward and easy to understand" (R-PB8W).

(2) **Significance of the newly collected dataset**: "interesting for evaluation" (R-cgtT); "support the motivation" (R-fEW5).

(3) **Superior performance and thorough experiments**: "flexibility and accommodates different scenarios" (R-mZMs); "superior performance" (R-mZMs); "thorough ablation"(R-mZMs); "the proposed method is effective" (R-PB8W).

Based on these acknowledgements, reviewers suggest that further empirical results will make this paper stronger, serving to highlight its strengths and clarify motivations. We agree, and have made every effort to execute the majority of the suggested experiments within the limited time of the rebuttal phase.

We would like to offer these general concerns and clarifications below.

### 1. Additional Experiments

#### 1.1 Comparison on recent SOTA methods on more benchmarks

Recent SOTA methods like Zegformer, OVSeg and CAT-Seg are trained on a larger dataset COCO-Stuff with more classes and more images.
Following them, we select some representative datasets PASCAL-Context (PC-59) and PASCAL VOC (PAS-20) and annotating the attributes them for evaluation.

The results are in Table 1 below. Our method still demonstrates superiority.

*Table 1: Comparison on recent SOTA methods on more benchmarks. All other methods is under "direct aggregation" strategy.*
|Model|Settings|Backbone|Training data|PC-59|PAS-20
|:-:|:-:|:-:|:-:|:-:|:-:
|LSeg|attr|RN101|PASCAL VOC-15|24.2|44.0
|Fusioner|attr|RN101|PASCAL VOC-15|25.2|44.3
|Zegformer|attr|RN101|COCO-Stuff|39.0|83.7
|OVSeg|attr|RN101|COCO-Stuff|51.2|90.3
|CAT-Seg|attr|RN101|COCO-Stuff|53.6|90.9
|**Ours**|attr|RN101|PASCAL VOC-15|29.1|49.1
|**Ours**|attr|RN101|COCO-Stuff|**56.3**|**91.6**

#### 1.2 Apply decomposition-aggregation for existing SOTA methods in real-world scenarios

We demonstrate the advantage of the motivation by showing that the proposed decomposition-aggregation ideology can be applied to existing methods in open-vocabulary literature.
We take Fantastic Beasts as an outstanding representative of the real-world that may encounter various situations like ambiguity, neologism and unnameability.

As shown in Table 2 below, the existing SOTA methods are not able to handle the real-world scenarios when taking class name as input (mIoU (cls)).
However, when decomposing into attributes and then aggregating them,
the performance of these methods remarkably increase, showing significant gains with robust performance (mIoU (attr)).
This demonstrates the universality and effectiveness of our motivation.

*Table 2: Effectiveness of our motivation in real-world scenarios. All other methods is under "direct aggregation" strategy when possible.*
Model|Backbone|Training data|mIoU(cls)|mIoU(attr)
:-:|:-:|:-:|:-:|:-:
LSeg|RN101|PASCAL VOC-15|<10|46.0
Fusioner|RN101|PASCAL VOC-15|<10|46.1
Zegformer|RN101|COCO-Stuff|<20|55.7
OVSeg|RN101|COCO-Stuff|<20|58.1
CAT-Seg|RN101|COCO-Stuff|<20|59.4
**Ours**|RN101|PASCAL VOC-15|-|52.3
**Ours**|RN101|COCO-Stuff|-|**61.9**

### 2. Is segmentation by only attributes more challenging than by class names under the current VLP diagram?

Yes, segmentation by only attributes is *indeed* more difficult than by class names as it poses additional challenges.

(1) Defining attributes: The definition of attributes is crucial and requires careful consideration.

(2) Noise in attributes: Developing effective methods to resist noise in attribute is essential for accurate results.

(3) How to aggregating complementary information: The critical aspect lies in effectively aggregating the complementary information provided by different attributes.

(4) Attributes describe the categories from *indirect* perspectives:
As the fundamental aim of VLP is to align image modality with its textual *class name*,
it is undeniable that class names hold core information.
Recent OVSS are all based on VLP, for example, they use CLIP as classifiers, which are inherently bound by this characteristic.
In contrast to class names, which provide the most direct information, attributes describe categories from indirect perspectives, adding to the challenge when using only attributes for segmentation.

### 3. Why do we opt for attribute inputs instead of utilizing class names, since segmentation by attributes are more challenging?

Because of the inherent flaws of VLP, there may be situations where it is not possible to input the category name.
For example, ambiguity, unknown, or unnameable category names.
Input attributes is capable of effectively addressing these issues, by making up for missing context information, transforming into known attributes or describing in a more detailed and accurate way.
This is also the motivation of this work, and Table 2 in the original paper and Table 2 above demonstrate this point.

For common classes such as in PASCAL and COCO, instead of input attributes only, as suggested by R-mZMs, we give a simple strategy for combining class names and attributes as input.
Despite not specifically designed for mixed input, our method can still handle this situation.
This further enhanced its practicality in real life, please refer to the response for R-mZMs for details.

### 4. Conclusion

To summarize, we thank the reviewers for their careful feedback and additional suggestions for evaluation, which will make our paper significantly stronger.
We look forward to further discussion, and are happy to answer any questions that might arise.


Meta-review: This paper has been reviewed by four experts in the field. After rebuttal and discussion, all reviewers acknowledged contributions of this paper, including technical novelty and thorough experiments. The ACs agreed with the reviewers.
Decision: Accept (poster)