SlowGuess/NeurIPS_2023_OCR huggingface dataset:
AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation
Chaofan Ma $^{1}$ , Yuhuan Yang $^{1}$ , Chen Ju $^{1}$ , Fei Zhang $^{1}$ , Ya Zhang $^{1,2}$ , Yanfeng Wang $^{1,2\text{图}}$
<sup>1</sup> Coop. Medianet Innovation Center, Shanghai Jiao Tong University
2 Shanghai AI Laboratory
{chaofanma, yangyuhuan, ju_chen, ferenas, ya_zhang, wangyanfeng622}@sjtu.edu.cn
Open-vocabulary semantic segmentation is a challenging task that requires segmenting novel object categories at inference time. Recent works explore vision-language pre-training to handle this task, but suffer from unrealistic assumptions in practical scenarios, i.e., low-quality textual category names. For example, this paradigm assumes that new textual categories will be accurately and completely provided, and exist in lexicons during pre-training. However, exceptions often happen when meet with ambiguity for brief or incomplete names, new words that are not present in the pre-trained lexicons, and difficult-to-describe categories for users. To address these issues, this work proposes a novel attribute decomposition-aggregation framework, AttrSeg, inspired by human cognition in understanding new concepts. Specifically, in the decomposition stage, we decouple class names into diverse attribute descriptions to complement semantic contexts from multiple perspectives. Two attribute construction strategies are designed: using large language models for common categories, and involving manually labelling for human-invented categories. In the aggregation stage, we group diverse attributes into an integrated global description, to form a discriminative classifier that distinguishes the target object from others. One hierarchical aggregation architecture is further proposed to achieve multi-level aggregations, leveraging the meticulously designed clustering module. The final results are obtained by computing the similarity between aggregated attributes and images embeddings. To evaluate the effectiveness, we annotate three types of datasets with attribute descriptions, and conduct extensive experiments and ablation studies. The results show the superior performance of attribute decomposition-aggregation. We refer readers to the latest arXiv version at https://arxiv.org/abs/2309.00096.
Semantic segmentation is one of the fundamental tasks in computer vision that involves partitioning an image into some semantically meaningful regions. Despite great progress has been made, existing research has mainly focuses on closed-set scenarios, where object categories remain constant during training and inference stages [50, 24]. This assumption is an oversimplification of real-life and limits its practical application. Another line of research considers a more challenging problem, that requires the vision system to handle a broader range of categories, including novel (unseen) categories during inference. This problem is referred as open-vocabulary semantic segmentation (OVSS).
To handle OVSS, vision-language pre-training (VLP) paradigm [42, 17, 49] provides a preliminary but popular idea. By leveraging language as an internal representation for visual recognition, segmentation is formulated as a similarity between a category's textual representation and pixel-level visual representation. Following on this paradigm, recent works focus on minor improvements, e.g.,
Figure 1: Left: Open-vocabulary semantic segmentation (OVSS) assumes the given new textual categories are accurate, complete, and exist in pre-trained lexicons. However, in real-life situations, practical uses are limited due to textual ambiguity, neologisms, and unnameability. Middle: We propose a novel attribute decomposition-aggregation framework where vanilla class names are first decomposed into various attribute descriptions (decomposition stage), and then, different attribute representations are aggregated hierarchically into a final class representation for further segmentations (aggregation stage). Right: Our framework successfully addresses the aforementioned issues and facilitates more practical applications of OVSS in real-world scenarios.
explore to better align vision-language modalities [52, 25, 32]. Although promising, these researches all maintain one unrealistic assumption in real-world scenarios, i.e., the given new textual categories are accurate and complete, and exist in the pre-trained lexicons. As a result, three main issues persist. (1) Ambiguity: brief or incomplete names bring lexical ambiguity, posing a great challenge for semantic discriminability. (2) Neologisms: new words that frequently emerge may not be present in the lexicons during vision-language pre-training, preventing pre-trained language models from interpreting their semantics, let alone aligning them with images. (3) Unnameability: unnamed or difficult-to-describe categories, such as specialized terms, rare animal names, or specific objects, can create a labeling problem for users, adding complexity during application. The above three issues result in low-quality category comprehension, limiting the empirical segmentation performance.
To address these issues, we turn our attention to cognitive psychology when human understanding new concepts [36, 45]. For example, if a child asks how to find a flamingo in the zoo, one can explain the process by looking for its pink feathers, long neck, and more. Then by combining these answers, a child can easily recognize a flamingo. Such answers provide detailed descriptions from multiple distinct or complementary perspectives, which we refer to as diverse "attributes". Compared to vanilla categories, attributes have three advantages. (1) For ambiguous categories, attributes can make up for missing context information to achieve completeness. (2) For unseen categories, they can be transformed into known attributes, easily interpreted by pre-trained language models. (3) For unnamed or indescribable categories, attributes can be used to replace in a more detailed manner. These attribute descriptions, aggregated at scale, provide a strong basis for visual recognition.
Inspired by this, we propose a novel decomposition-aggregation framework where vanilla class names are first decomposed into various attribute descriptions, and then different attribute representations are aggregated into a final class representation for further segmentations. Specifically, for the decomposition stage, our goal is to generate various attribute descriptions from coarse category names and build attributes for datasets. We propose two construction strategies: one is to generate using language models, and the other involves manually labelling. The first strategy corresponds to situations where common category names are sometimes brief or incomplete, with semantic ambiguity or insufficient discriminability. In this case, we can simply annotate attributes upon existing datasets, such as PASCAL [13, 16] and COCO [28]. The second strategy involves
a newly collected dataset called "Fantastic Beasts", which contains imaginal creatures and their invented names by humans. This dataset is used to simulate situations for new words for pre-trained vision-language models, and difficult-to-describe categories for users. In the aggregation stage, our aim is to combine the separate attribute pieces into an integrated global description, which then serves as a classifier to differentiate the target object from others. This stage can also be viewed as the process of combining regions reflected by different attributes into a specific one, which yields the segmentation result. Since attributes describing objects may potentially contain hierarchy, we propose a hierarchical aggregation architecture to leverage this potential. A clustering module is carefully designed to aggregate attributes from different levels, and the final mask is obtained by computing similarities between the grouped attribute embeddings and the image features.
To evaluate the significance of attribute understanding for OVSS, we annotate attribute descriptions on three types of datasets, namely, PASCAL series [13, 16, 35], COCO series [28, 8], and Fantastic Beasts. Extensive experiments demonstrate the superior performance of our attribute decomposition-aggregation framework over multiple baselines and competitors. Furthermore, we performed thorough ablation studies to dissect each stage and component, both quantitatively and qualitatively.
To sum up, our contributions lie in three folds:
- We pioneer the early exploration in leveraging only the attribute descriptions for open-vocabulary segmentation, and to achieve this end, we construct detailed attributes descriptions for two types of existing datasets and one newly collected dataset Fantastic Beasts;
- We design a novel decomposition-aggregation framework that decompose class names into attribute descriptions, and then aggregate them into a final class representation;
- We conduct thorough experiments and ablations to reveal the significance of attribute decomposition-aggregation, and our model's superior performance on all proposed datasets.
Vision-Language Pre-training (VLP) aims to jointly optimize image-text embeddings with large-scale web data. Recently, some studies have further scaled up the training to form "the foundation models", e.g., CLIP [42], ALIGN [17], Florence [49], and FILIP [48]. These foundation models usually contain one visual encoder and one textual encoder, which are trained using simple noise contrastive learning for powerful cross-modal alignment. They have shown promising potential in many tasks: grounding [18, 20], detection [21, 19], and segmentation [47, 31, 51, 29]. This paper uses CLIP for OVSS, but the same technique should be applicable to other foundation models as well.
Open-Vocabulary Semantic Segmentation (OVSS) aims to understand images in terms of categories described by textual descriptions. Pioneering works [7, 26, 15] use generative models to synthesize visual features from word embeddings of novel categories. SPNet [46] and JoEm [4] employ a mapping process that assigns each pixel and semantic word to a joint embedding space. Recently, researchers have proposed to leverage pre-trained vision-language models (VLMs) for OVSS. OpenSeg [14] aligns region-level visual features with text embedding via region-text grounding. LSeg [25] aligns pixel-level visual embeddings with the category text embedding of CLIP. Subsequent methods like Fusioner [32], Zegformer [11], OVseg [27] and CAT-Seg [10] have thoroughly investigated the open-vocabulary capability of CLIP. However, these methods heavily rely on category names, ignoring text ambiguity, neologisms, and unnameable are common in real-world scenarios. This paper designs novel framework of attribute decomposition-aggregation to tackle these issues.
Attribute Understanding. Visual attributes are first studied in the traditional zero-shot learning [23, 43, 22]. With the emergence of vision-language models, the attribute understanding has developed towards a more scalable, open, and practical direction. One line of research is focused on detecting and recognizing an open set of objects, along with an open set of attributes for each object [39, 40, 5, 9]. Another line of research focuses on object classification by incorporating attributes as part of text prompts, which aim to evaluate the discriminative ability of VLMs with enriched text [41, 37], or to enhance interpretability and explainability of model reasoning [34, 33]. Different from above, our work investigates attribute understanding from the perspective of open-vocabulary semantic segmentation, using a decomposition-aggregation strategy.
This paper considers open-vocabulary semantic segmentation (OVSS). We start by giving the preliminary in Sec. 3.1; then we introduce our attribute decomposition-aggregation framework in Sec. 3.2; decomposition stage and aggregation stage will be detailed in Sec. 3.3 and Sec. 3.4, respectively.
3.1 Problem Formulation & Preliminary
Problem. Given an image  $\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$ , OVSS aims to train one model  $\Phi(\Theta)$  that can segment the target object according to its text description  $\mathcal{T}$ , that is, outputting one pixel-level mask  $\mathcal{M}$ :
$$
\mathcal {M} = \Phi_ {\text {s e g}} (\mathcal {I}, \mathcal {T}; \Theta) \in \{0, 1 \} ^ {H \times W \times 1}. \tag {1}
$$
Under open-vocabulary settings, training classes  $\mathcal{C}_{\mathrm{base}}$  and testing class  $\mathcal{C}_{\mathrm{novel}}$  are disjoint, i.e.,  $\mathcal{C}_{\mathrm{base}} \cap \mathcal{C}_{\mathrm{novel}} = \emptyset$ . During training, image-mask pairs from the base class are provided, i.e.,  $\{(\mathcal{I}, \mathcal{M}) \sim \mathcal{C}_{\mathrm{base}}\}$ ; while during testing, the model is evaluated on the disjoint novel classes, i.e.,  $\{\mathcal{I} \sim \mathcal{C}_{\mathrm{novel}}\}$ .
Vision-Language Paradigm. To enable open-vocabulary capability, recent OVSS studies [52, 25, 32] embrace vision-language pre-trainings (VLPs), for their notable ability in cross-modal alignment. Specifically, regarding vanilla class names as textual descriptions, open-vocabulary segmentation can be achieved by measuring the similarity between class-level textual and pixel-level visual embeddings:
$$
\mathcal {M} = \mathcal {F} _ {v} * \mathcal {F} _ {t}, \quad \mathcal {F} _ {v} = \Phi_ {\mathrm {v i s}} (\mathcal {I}) \in \mathbb {R} ^ {H \times W \times D}, \quad \mathcal {F} _ {t} = \Phi_ {\mathrm {t x t}} (\mathcal {T}) \in \mathbb {R} ^ {1 \times D}, \tag {2}
$$
where  $\Phi_{\mathrm{vis}}$  and  $\Phi_{\mathrm{txt}}$  refer to the visual and textual encoders in VLPs. This paradigm has a fancy dream, but meets poor reality. In practice, the textual names of novel classes may potentially suffer low-quality comprehension in three aspects: (1) Ambiguity. Certain names exhibit lexical ambiguity, while others sometimes may incomplete due to excessive simplification. These result in a deficiency of semantic discriminability. (2) Neologisms. The pre-training text corpus is inevitably limited in its coverage of vocabulary, and thus may not include certain terms that have emerged as neologisms in the real world. (3) Unnameability. Certain categories of entities may lack a known or easily describable name for users, particularly in cases involving specialized terminology, rare or obscure animal names, etc. These issues greatly limit the use and development of open-vocabulary segmentation.
3.2 Attribute Decomposition-Aggregation Framework
To solve the above issues, we introduce one novel attribute decomposition-aggregation framework.
Motivation. Such textual semantic issues are caused by the low-quality category comprehension. We consider to decompose class name from multiple perspectives, such as color, shape, parts and material, etc. This forms informative attribute sets for the text stream. Treated as partial representations describing categories, (1) attributes can supplement missing information for incompleteness and ambiguity; (2) for new words, attributes can be transformed into known words, which can be easily interpreted by the pre-trained language model; (3) for unnamed or not easily describable categories, attributes can be used to describe them in a more detailed and accurate way. These attributes, when aggregated to a global description, can provide a strong basis for visual recognition.
Framework Overview. As illustrated in Fig. 2, given an image and a set of attributes descriptions (Sec. 3.3), we first obtain its visual and attribute embeddings (Sec. 3.4.1). Considering the potential hierarchy inside attributes, we suggest a hierarchical pipeline to progressively aggregate all given attributes embeddings into one specific token (Sec. 3.4.2). As the final grouped token represents all attributes' information, segmentations can be acquired by computing the similarity between this token and the visual embeddings (Sec. 3.4.3). Formally,
$$
\mathcal {M} = \mathcal {F} _ {v} * \mathcal {F} _ {t}, \quad \mathcal {F} _ {v} = \Phi_ {\text {v i s}} (\mathcal {I}) \in \mathbb {R} ^ {H \times W \times D}, \quad \mathcal {F} _ {t} = \Phi_ {\text {a g g r}} \circ \Phi_ {\text {t x t}} \circ \Phi_ {\text {d e c p}} (\mathcal {T}) \in \mathbb {R} ^ {1 \times D}, \tag {3}
$$
where  $\Phi_{\mathrm{deep}}(\cdot)$  denotes the decomposition module that returns the set of  $n$  attributes' textual descriptions of the target category. Note that, category names (and synonyms) are strictly not contained in this set.  $\Phi_{\mathrm{aggr}}(\cdot)$  refers to the aggregation module that groups attributes into one specific embedding.
3.3 Decompose: Detailed Attribute Descriptions for Class Names
In real-world scenarios, vanilla class names  $\mathcal{T}$  may be coarse-grained, ambiguous, or neologisms. Unfortunately, there is a lack of existing datasets that provide detailed attribute descriptions to offer
Figure 2: Overview of Attribute Decomposition-Aggregation Framework. (a) Decomposition stage aims to decouple vanilla class names into various attribute descriptions. We design two strategies to build attributes, i.e., using LLMs and manual collections. (b) Aggregation stage aims to merge separated attribute representations into an integrated global description. We propose to hierarchically aggregate attribute tokens to one specific token in  $L$  stages. Each stage alternates a fusion module and a clustering module. Masks are generated by calculating the similarity.
informative contexts. So, we here propose two strategies to construct diverse attributes  $\mathcal{A}$ . As shown in Fig. 2a, one involves utilizing large language models, while the other relies on manual collection.
$$
\mathcal {A} = \Phi_ {\mathrm {d e c p}} (\mathcal {T}) = \left\{\operatorname {a t t r} _ {1}, \operatorname {a t t r} _ {2}, \dots , \operatorname {a t t r} _ {n} \right\}. \tag {4}
$$
3.3.1 Attribute Descriptions by Large Language Models
For the cases where vanilla category names are semantically coarse or ambiguous, one promising solution is to describe attributes or contexts for better discriminability. To generate such attributes, manual writing can be time-consuming and inefficient, particularly for a large number of classes. Hence, for cost-effectiveness, we turn to large language models (LLMs) [6, 38], pre-trained on large corpora of data, showing remarkable performance of semantic understanding and text generation.
More specifically, to automatically adapt LLMs to enrich class contexts, we carefully designed a set of question templates for attribute descriptions mining from various perspectives. Taking the category "flamingo" as an example, we first pose multiple questions, such as "List all attributes for distinguishing a {flamingo} in a photo" or "What are visual features of a {flamingo} in the image". Then we prompt ChatGPT [38] to obtain answers of attribute descriptions, such as "pink feathers; long neck; thin legs; large wingspan; ..." Finally, we filter and combine answers to form an attribute set for each category. Please refer to the supplementary materials for further details.
3.3.2 Attribute Descriptions by Manual Collection
In addition to the above cases, there are two common cases that require attribute descriptions. One is vanilla category names are neologisms that are unseen by LLMs and VLPs; the other is when users are not familiar with an object, so they may have difficulty naming it, especially when it comes to a rare or obscure category. Given that, existing datasets typically do not include rare or obscure vocabulary, so we manually collect a dataset of human-made objects and rare categories for simulation.
The dataset is called "Fantastic Beasts", which consists of 20 categories of magical creatures from the film series of Fantastic Beasts [1, 2, 3]. We first scrap and filter the dataset images from the web, then organize fan volunteers of the film series to carefully annotate the paired masks and the category attributes. Since all these creatures and names are human inventions, they are unlikely to be learned by existing LLMs and VLPs. Some images, alone with their corresponding class names and attributes, are shown in Fig. 1. Please also refer to the supplementary materials for dataset information.
3.3.3 Discussion
(1) Datasets: developing high-quality attribute datasets is a crucial contribution towards advancing the practicality of OVSS. As there are currently no existing benchmarks and evaluations, an essential first step has been made that involves annotating the attributes on top of the existing datasets such as
PASCAL and COCO, as well as manually collecting a dataset, Fantastic Beasts. (2) Why LLMs: thanks to the capabilities of LLMs, category attributes for existing datasets can be obtained in a scalable manner. Despite not receiving any visual input during training, LLMs can successfully imitate visual attributes since they are trained on a large corpus containing descriptions with visual knowledge. (3) Significances: we believe that the attribute decomposition strategy, and the produced datasets, will have a great impact on the community to further promote practical uses of OVSS.
3.4 Aggregate: Hierarchical Fusion for Vision-Attribute Alignments
Given image  $\mathcal{I}$  and attribute descriptions  $\mathcal{A}$ , we aim to aggregate these separate pieces of attribute embeddings hierarchically into one integrated global representation  $\mathcal{G}$ , with the help of visual information. Then, it can serve as a discriminative classifier to distinguish the target object.
$$
\mathcal {G} = \Phi_ {\text {a g g r}} \left(\Phi_ {\text {t x t}} (\mathcal {A}); \Phi_ {\text {v i s}} (\mathcal {I})\right) \in \mathbb {R} ^ {1 \times D}. \tag {5}
$$
3.4.1 Vision Embeddings and Attribute Embeddings
We here adopt vision-language pre-trainings [42] as encoders and mostly consider ViT-based architectures [12], due to their good performance, and flexibility for encoding different modalities.
Given an image  $\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$ , the visual embeddings  $\mathcal{V} \in \mathbb{R}^{N^v \times d}$  are extracted from the visual encoder, where  $N^v$  is the number of image tokens; and  $d$  is the channel dimension. The attribute embeddings  $\mathcal{A} \in \mathbb{R}^{N^a \times d}$  are obtained by first feeding each attribute into the text encoder separately, then concatenating all of them, where  $N^a$  is the number of attributes describing one target object.
3.4.2 Hierarchical Aggregation Architecture
Attributes descriptions may potentially contain hierarchy. We propose to hierarchically aggregate these attributes in  $L$  stages, hoping to explicitly leveraging this potential, as shown in Fig. 2b.
Overview. Each stage alternates a fusion module and a clustering module. Specifically, the fusion module facilitates interaction between different modalities. Given these enriched representations, the following clustering module groups attribute tokens to fewer tokens. This procedure utilizes learnable cluster tokens as centers for clustering, and considers both visual and attribute information. Based on the similarity, these clustering centers can gather and merge all attributes tokens into specific groups.
Formally, for the  $l$ -th stage, we denote  $N^v$  visual tokens as  $\mathcal{V}_l \in \mathbb{R}^{N^v \times d}$ ;  $N_l^a$  attribute tokens describing one target object as  $\mathcal{A}_l \in \mathbb{R}^{N_l^a \times d}$ ; and  $N_l^g$  learnable cluster tokens for aggregation as  $\mathcal{G}_l \in \mathbb{R}^{N_l^g \times d}$ . The fusion module fuses and enriches the information globally between  $\mathcal{V}_l$ ,  $\mathcal{A}_l$ , and  $\mathcal{G}_l$ :
$$
\mathcal {V} _ {l}, \mathcal {A} _ {l}, \mathcal {G} _ {l} = \Psi_ {\text {f u s e}} ^ {l} \left(\mathcal {V} _ {l}, \mathcal {A} _ {l}, \mathcal {G} _ {l}\right). \tag {6}
$$
To avoid notation abuse, we still use the same notation for the output. After fusion, the  $N_{l}^{a}$  attribute tokens  $\mathcal{A}_l$  are merged and grouped to fewer  $N_{l + 1}^{a}$  ( $N_{l + 1}^{a} < N_{l}^{a}$ ) tokens  $\mathcal{A}_{l + 1}$  through clustering: