SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {A} _ {l + 1} = \Psi_ {\text {c l u s t e r}} ^ {l} \left(\mathcal {A} _ {l}; \mathcal {V} _ {l}, \mathcal {G} _ {l}\right) \in \mathbb {R} ^ {N _ {l + 1} ^ {a} \times d}. \tag {7}
$$
Note that, the attribute tokens are grouped not only based on itself, but also depending on the visual embeddings. And the number of grouped attributes  $\mathcal{A}_{l + 1}$  (output for this stage) is equal to the number of input learnable cluster tokens  $\mathcal{G}_l$  for this stage, i.e.,  $N_{l + 1}^{a} = N_{l}^{g}$ .
Fusion Module.  $\Psi_{\mathrm{fuse}}^{l}$  is flexible and adaptable to multiple network architectures. Here we use multiple transformer encoder layers as representatives, in order to effectively capture and propagate the long-range information of different modalities, by iteratively attending to each other.
Clustering Module. Learnable cluster tokens are used here to represent the clustering center for each grouping stage. It's unreasonable to aggregate solely based on attribute embeddings, as the visual information also plays a part in the segmentation. To better incorporate both modalities, the learnable clustering center first obtain the contextual information through vision and attribute cross attentions:
$$
\mathcal {G} _ {l} = \phi_ {\text {c r o s s - a t t n}} (q = \mathcal {G} _ {l}, k = \mathcal {V} _ {l}, v = \mathcal {V} _ {l}), \quad \tilde {\mathcal {G}} _ {l} = \phi_ {\text {c r o s s - a t t n}} (q = \mathcal {G} _ {l}, k = \mathcal {A} _ {l}, v = \mathcal {A} _ {l}), \tag {8}
$$
where  $\tilde{\mathcal{G}}_l\in \mathbb{R}^{N_l^g\times d}$  is the contextual centers;  $\nu_{l}$  and  $\mathcal{A}_l$  are visual and attribute embeddings. By exchanging information for the visual tokens and attribute tokens respectively,  $\tilde{\mathcal{G}}_l$  has the knowledge of both modalities, providing a good prior for the subsequent processing.
(a) direct aggregation
(b) pre-aggregation
(c) post-aggregation
Figure 3: Comparison between Various Aggregation Strategies. The orange / blue colors represent visual / attribute tokens, respectively. Detailed discussions can be found in Sec. 3.4.4.
(d) hierarchy aggregation
Next, we assign each attribute token to one of the contextual centers, and calculate the representation of newly grouped attributes  $\tilde{\mathcal{A}}_l$  as output. We use slot attention [30], by seeing each cluster as a slot:
$$
\tilde {\mathcal {A}} _ {l} = \phi_ {\text {s l o t - a t t n}} (q = \tilde {\mathcal {G}} _ {l}, k = \mathcal {A} _ {l}, v = \mathcal {A} _ {l}), \quad \mathcal {A} _ {l + 1} = \phi_ {\text {m i x e r}} (\tilde {\mathcal {A}} _ {l} + \tilde {\mathcal {G}} _ {l}). \tag {9}
$$
The final grouped attributes  $\mathcal{A}_{l + 1}$  for this stage can be obtained after adding the residual to  $\tilde{\mathcal{G}}_l$  and then updating and propagating the information between tokens through  $\phi_{\mathrm{mixer}}$ . Here we use MLPlexer [44] with two consecutive group-wise and channel-wise MLPs.
3.4.3 Mask Calculation
The total  $L$  stages aggregation gives the result of one specific token  $\mathcal{A}_{L + 1} \in \mathbb{R}^{1 \times d}$ , which can be thought as condensing the collective knowledge of the provided attributes' information. Logits  $\mathcal{V}$  can be generated by computing the similarity between the final stage visual embedding  $\mathcal{V}_{L + 1}$  and  $\mathcal{A}_{L + 1}$ :
$$
\mathcal {Y} = \phi_ {\text {s i m}} \left(\mathcal {V} _ {L + 1}, \mathcal {A} _ {L + 1}\right). \tag {10}
$$
The final predictions can be obtained by simply reshaping to spatial size and upsampling, then applying sigmoid with a temperature  $\tau$  and thresholding.
3.4.4 Discussion
The fundamental concept of aggregation strategies involves identifying one single embedding that represents all attributes. As shown in Fig. 3, we present four optional designs: "Direct", "Pre-", "Post-", and "Hierarchy", which are able to be applied to all existing open-vocabulary methods. (1) "Direct" involves listing all attributes in one sentence, which is then directly sent to a text encoder. The aggregated embedding is obtained from the output [CLS] token, which is subsequently fed into the open-vocabulary model for further processing. (2) In contrast to "direct", the other three strategies feed attributes into the text encoder separately. "Pre-" first aggregates all attribute tokens into one token, which is then fed into the model similar to "direct". (3) "Post-" inputs all attribute tokens together into the open-vocabulary model. The output attribute tokens of this model are further aggregated to one token. (4) "Hierarchy" progressively aggregates multiple tokens into one in multiple stages, as detailed in this section. In Sec. 4.1 and Sec. 4.2, we conduct comprehensive comparisons and explore the adaptability of these strategies to existing open-vocabulary models. This analysis demonstrates the universality and effectiveness of our decomposition-aggregation motivation.
3.5 Training and Inference
During training, the visual and textual encoders are kept frozen, and we uniformly sample  $N$  attributes describing the target category with replacement from the attribute set. The predicted mask is supervised by the ground truth using standard cross-entropy loss. During inference, the user has the flexibility to provide test images along with any number of attributes describing the object of interest. The model can then generate the corresponding segmentation mask based on this input.
Datasets. We evaluate on PASCAL-5 $^i$  [13, 16] COCO-20 $^i$  [28] following [25, 32], and evaluate on Pascal VOC [13] and Pascal Context [35] following [11, 27, 10]. PASCAL-5 $^i$  contains 20 categories
that are divided into 4 folds of 5 classes each, i.e.,  $\{5^i\}_{j=0}^3$ . COCO-20 $^i$  is more challenging with 80 categories that are also divided into 4 folds, i.e.,  $\{20^i\}_{i=0}^3$ , with each fold having 20 categories. Of the four folds in the two datasets, one is used for evaluation, while the other three are used for training. PASCAL VOC is a classical dataset. We evaluate on the 1.5k validation images with 20 categories (PAS-20). PASCAL-Context contains 5k validation images. We evaluate on the most frequent used 59 classes version (PC-59). Besides, we also annotate Fantastic Beasts, which contains 20 categories of human invented magical creatures from the Fantastic Beasts film series [1, 2, 3]. Detailed datasets' information can be found in the supplementary materials.
Evaluation. We report mean intersection-over-union (mIoU), following the recent open-vocabulary semantic segmentation (OVSS) literatures [25, 32, 11, 27, 10].
Implementation Details. We adopt CLIP ViT-L and ResNet101 as our backbone, and choose aggregation stages  $L = 4$ . Numbers of learnable cluster in each stage are (15, 10, 5, 1). During training, the sampled attributes  $N = 15$ . AdamW optimizer is used with CosineLRScheduler by first warm up 10 epochs from initial learning rate 4e-6 to 1e-3, and the weight decay is set to 0.05.
4.1 Comparison with the State-of-the-art
Table 1: Evaluation on PASCAL-5 $^i$  and COCO-20 $^i$ . For textual inputs, we compare "cls name" (category names) and "attr" (attributes). "direct", "pre-", "post-," and "hrchy" are four aggregation strategies in Sec. 3.4.4.
Model	Settings	Backbone	PASCAL-5i	COCO-20i
50	51	52	53	mIoU	200	201	202	203	mIoU
SPNet [46]	cls name	RN101	23.8	17.0	14.1	18.3	18.3	-	-	-	-	-
ZS3Net [7]	cls name	RN101	40.8	39.4	39.3	33.6	38.3	18.8	20.1	24.8	20.5	21.1
LSeg [25]	cls name	RN101	52.8	53.8	44.4	38.5	47.4	22.1	25.1	24.9	21.5	23.4
LSeg [25]	cls name	ViT-L	61.3	63.6	43.1	41.0	52.3	28.1	27.5	30.0	23.2	27.2
LSeg [25]	attr (direct)	RN101	48.6	51.2	39.7	36.2	44.0	20.9	23.6	20.8	18.4	20.9
LSeg [25]	attr (pre-)	RN101	49.1	51.4	40.9	35.9	44.3	21.7	24.5	22.0	19.6	22.0
LSeg [25]	attr (post-)	RN101	50.0	52.2	42.1	36.8	45.3	21.2	24.0	22.5	19.2	21.7
AttrSeg (Ours)	attr (hrchy)	RN101	52.9	55.3	45.0	43.1	49.1	27.6	28.4	26.1	22.7	26.2
AttrSeg (Ours)	attr (hrchy)	ViT-L	61.5	67.5	46.1	50.5	56.4	34.8	32.6	31.6	24.2	30.8
PASCAL-5 $^i$  and COCO-20 $^i$ . We compare our method on PASCAL-5 $^i$  and COCO-20 $^i$  with state-of-the-art OVSS models SPNet [46], ZS3Net [7] and LSeg [25]. We also compare different textual input settings for LSeg. As shown in Tab. 1, our method outperforms the competitive baselines. Note that, heavily relying on VLP that aligns image modality with its textual class name, it is undeniable that class names hold core and most direct information. Existing open-vocabulary segmentation models are inherently bound by this characteristic, i.e., adding challenges when using only attributes that describe categories from indirect perspectives for segmentation. As a representative, LSeg shows weakness when the textual inputs are attributes descriptions only. In contrast, our method demonstrates good potential for segmenting the target object described by attributes through hierarchical aggregations, even surpass the SOTA with class name input.
PASCAL-Context and PASCAL VOC. We also evaluate our method on PASCAL-Context (PC-59) and PASCAL VOC (PAS-20), to compare with recent state-of-the-art OVSS works Zegformer [11], OVSeg [27] and CAT-Seg [10]. They are trained on a larger dataset COCO-Stuff [8] with more classes and more images. As shown in Tab. 2, our method still demonstrates superiority. Besides, compared with our method trained on COCO-Stuff and PASCAL VOC-15, the performance increased dramatically.
Table 2: Evaluation on PASCAL-Context (PC59) and PASCAL VOC (PAS-20) with attributes input using RN101 backbone.
Model	Training Data	PC-59	PAS-20
LSeg [25]	PASCAL VOC-15	24.2	44.0
Fusioner [32]	PASCAL VOC-15	25.2	44.3
AttrSeg (Ours)	PASCAL VOC-15	29.1	49.1
Zegformer [11]	COCO-Stuff	39.0	83.7
OVSeg [27]	COCO-Stuff	51.2	90.3
CAT-Seg [10]	COCO-Stuff	53.6	90.9
AttrSeg (Ours)	COCO-Stuff	56.3	91.6
Table 3: Evaluation on Fantastic Beasts (using checkpoints transferred from PASCAL-5 $^i$  and COCO-20 $^i$ ). For textual inputs, we compare "cls name" (category names) and "attr" (attributes). "direct", "pre-", "post- and "hrchy" are four aggregation strategies.  $5^i$  and  $20^i$  refer to the best checkpoints from the  $i^{th}$  fold for evaluation.
Model	Settings	Backbone	PASCAL-5i	COCO-20i
50	51	52	53	mIoU	200	201	202	203	mIoU
LSeg [25]	cls name	RN101	<10	<10	<10	<10	<10	<10	<10	<10	<10	<10
LSeg [25]	attr (direct)	RN101	44.8	46.6	46.4	46.2	46.0	50.1	51.2	51.7	49.3	50.6
LSeg [25]	attr (pre-)	RN101	46.4	47.7	48.2	47.3	47.4	51.3	52.7	52.9	50.9	52.0
LSeg [25]	attr (post-)	RN101	46.9	48.3	48.8	47.7	47.9	52.8	52.5	52.4	51.2	52.2
AttrSeg (Ours)	attr (hrchy)	RN101	50.7	53.4	53.6	51.3	52.3	56.4	54.9	55.7	55.3	55.6
AttrSeg (Ours)	attr (hrchy)	ViT-L	54.1	55.8	55.4	54.5	55.0	59.2	58.8	58.3	58.5	58.7
Fantastic Beasts. We take Fantastic Beasts as an outstanding representative of the realworld that may encounter various situations like ambiguity, neologism and unnameability. To simulate real-world scenarios, we directly transfer the checkpoints of previous methods and ours trained from the their corresponding datasets for evaluation. As shown in Tab. 3 and Tab. 4, our method establish a new solid baseline. The existing SOTA methods are not able to handle these scenarios when taking class
name as input. Despite using CLIP [42], which implicitly aligns visual and language features to some extent, these methods performance suffers due to the presence of unfamiliar words that break this alignment. However, when decomposing into attributes and then aggregating them, the performance of these methods remarkably increase, showing significant gains with robust performance. This demonstrates the universality and effectiveness of our motivation.
Table 4: Evaluation on Fantastic Beasts with attributes input using RN101 backbone (using checkpoints transferred from the corresponding training datasets).
Model	Training Data	mIoU (cls)	mIoU (attr)
LSeg [25]	PASCAL VOC-15	<10	46.0
Fusioner [32]	PASCAL VOC-15	<10	46.1
AttrSeg (Ours)	PASCAL VOC-15	-	52.3
Zegformer [11]	COCO-Stuff	<20	55.7
OVSeg [27]	COCO-Stuff	<20	58.1
CAT-Seg [10]	COCO-Stuff	<20	59.4
AttrSeg (Ours)	COCO-Stuff	-	61.9
4.2 Ablation Study
Various Aggregation Strategies. Here, we present a comparison of different aggregation methods, as depicted in Tab. 5. The results demonstrate that: (1) "direct" gives the poorest results, suggesting that listing all attributes in one sentence cannot provide sufficient interactions between to-
Table 5: Effectiveness of various aggregation strategies on PASCAL-  $5^{i}$  with RN101 backbone.
Strat	LSeg	Ours
50	51	52	53	mIoU	50	51	52	53	mIoU
Direct	48.6	51.2	39.7	36.2	44.0	48.8	51.4	39.7	39.2	44.8
Pre-	49.1	51.4	40.9	35.9	44.3	49.8	52.8	41.7	40.3	46.1
Post-	50.0	52.2	42.1	36.8	45.3	50.3	53.0	42.3	41.0	46.7
Hrchy	-	-	-	-	-	52.9	55.3	45.0	43.1	49.1
kens. (2) Both the "pre- and post- both exhibit improvements compared with "direct". This may stem from the exchange of information between attribute and visual modalities within the open-vocabulary model. (3) In most cases, "Post- performs better than "pre-", indicating that attribute aggregation after the model can facilitate greater interactions between different modalities. (4) It is not necessarily optimal to have an excessive or insufficient number of interactions throughout the entire process. Our proposed hierarchical aggregation method accounts for the regularity that the visual component learns differently at different stages. This necessitates its attribute counterpart to possess varying hierarchy. For instance, lower stages may focus on learning low-level features, requiring more attributes that represent local regions. Conversely, higher stages may concentrate on global information, necessitating fewer high-level attributes that are semantically abstract.
Components in Aggregation Module. We conduct ablation studies to investigate the importance of each component in our clustering module, as illustrated in Tab. 6. The result show that (1) the full module with all components achieves the best performance. (2) Cross attention on images contribute more compared with on attributes. As the aggregation among attributes finally aims to recognize the object in the image, it's important for the cluster tokens attending to the visual
Table 6: Ablation of components in clustering module on PASCAL-5  ${}^{i}$  with ViT-L backbone.
Comp	Cross Attn	Mixer	50	51	52	53	mIoU
Img	Attr
Full	✓	✓	✓	61.5	67.5	46.1	50.5	56.4
Cross Attn	X	✓	✓	59.2	65.6	44.1	47.4	54.1
✓	X	✓	59.6	65.8	44.4	48.6	54.6
X	X	✓	58.6	65.0	42.9	46.4	53.2
Mixer	✓	✓	X	60.7	67.0	45.6	49.6	55.7
Mini	X	X	X	58.4	64.9	42.4	46.0	52.9
feature. (3) If no cross attentions given, the performance further degraded. This demonstrates the
cross attentions on two modalities enable the learnable cluster centers match better with the correct attributes. (4) Mixer also plays a role in the bottom part of the module, as it helps to propagate and exchange information between clusters for further aggregation. (5) When neither of these is applied, only the SlotAttn is introduced, which, unsurprisingly, yields the poorest result.
Numbers of Decomposed Attributes. Tab. 7 shows the results when given input numbers of attribute (#Attr) changes. As #Attr decreases, the available information diminishes, potentially resulting in an incomplete object description. In general, such a reduction in attribute quantity can lead to a significant decline in performance. However, our method demonstrates a certain level of robustness even when #Attr decreases from 15 to 10, and even 5. This resilience can be ascribed to the
Table 7: Ablation of #Attributes inputs and #Stages on PASCAL-5<sup>i</sup> with ViT-L backbone.
#Attr	#Stage	#Tokens/Stage	50	51	52	53	mIoU
15	5	(15,10,5,3,1)	61.0	67.3	46.9	50.7	56.5
4	(15,10,5,1)	61.5	67.5	46.1	50.5	56.4
3	(15,10,1)	60.1	66.2	44.7	48.9	55.0
2	(15,1)	57.8	63.5	41.9	46.4	52.4
10	5	(10,5,3,2,1)	59.4	65.8	44.9	49.2	54.8
4	(10,5,3,1)	59.6	66.1	44.7	49.9	55.1
3	(10,5,1)	58.0	64.8	43.8	47.1	53.4
2	(10,1)	55.9	62.8	42.5	45.4	51.7
5	5	(5,4,3,2,1)	51.8	62.9	41.9	44.8	50.4
4	(5,3,2,1)	52.4	63.1	43.7	45.2	51.1
3	(5,3,1)	52.1	62.7	43.0	45.4	50.8
2	(5,1)	49.2	61.6	42.1	44.2	49.3
fact that the attributes are randomly sampled (with repetition) from the attribute set during training. Consequently, the attributes describing a specific category do not necessarily need to be complete or highly informative for the model to generate accurate outputs.
Numbers of Aggregation Stages. As shown in Tab. 7, we reduce the numbers of stage (#Stage) from 5 to 2. In each stage, we attempt to aggregate to half of the tokens until there is only one left. In most cases, less aggregation stages cause deficient results, and more stages result in better performance. However, blindly increasing stages do not necessarily lead to better performance. For #Attr is 5 or 10, 5 stages aggregation lead to a decrease compared with 4 stages. A reason may be that too much aggregation operation is excessive for limited attributes, which may disrupt the hierarchy inside, thus negatively affects the result. Considering the trade-off, we choose 4 stages in our method.
Impact of Inaccurate/Incorrect Attributes. In real-world scenarios, attribute decomposition may include slight noise. Our aggregation module exhibits a certain level of robustness to noise during both training and inference. (1) For inaccurate attributes (attributes that are not visible in a specific image, but are still related to the class, such as "four-legged" to "dog" in a dog lying down image), during training, we randomly select attributes from the attribute pool for a given class, which means that the model is trained with potentially noisy and inaccurate at-
tributes. However, as demonstrated in Tab. 8, our model can learn to ignore these inaccurate attributes during aggregation, and instead focus on other attributes to produce correct segmentation results. (2) For incorrect attributes (attributes that are completely unrelated to the target class, such as "red" to "dog"), a naive approach would be to first filter the input attributes using existing VLMs, and then select the top related attributes for downstream processing. Tab. 8 assess the impact of incorrect attributes and VLM filtering on inference. The results indicate a simple VLM filtering has the effect.
Table 8: Ablation of the impact of inaccurate/incorrect attributes during decomposition, and the VLM filtering strategy.
Clean Attr	Inaccurate Attr	Incorrect Attr	VLM Filtering	mIoU
✓	59.5
✓	✓	59.1
✓	✓	✓	55.4
✓	✓	✓	✓	58.9
Types of Decomposed Attributes. We roughly categorize the decomposed attributes into four types: color, shape, parts, and others, and maintain a consistent total number of inputs. As shown in Tab. 9, all types of attributes contribute to the overall performance, highlighting the significance of attribute diversity.
Table 9: Ablation of decomposed attributes' types on Fantastic Beast with RN101 backbone.
Color	Shape	Parts	Others	mIoU
✓	30.8
✓	✓	42.5
✓	✓	✓	51.0
✓	✓	✓	✓	52.3
We pioneer the early exploration in utilizing only attribute descriptions for open-vocabulary segmentation, and provide detailed attribute descriptions for two types of existing datasets and one newly collected dataset. Based on this, we propose a novel attribute decomposition-aggregation framework that first decouples class names into attribute descriptions, and then combines them into final class representations. Extensive experiments demonstrate the effectiveness of our method, showcasing its ability to achieve the state-of-the-art performance across various scenarios.
This work is supported by the National Key R&D Program of China (No. 2022ZD0160703), STCSM (No. 22511106101, No. 22511105700, No. 21DZ1100100), 111 plan (No. BP0719010) and National Natural Science Foundation of China (No. 62306178).
[45] Edward J. Wisniewski and Bradley C. Love. Relations versus properties in conceptual combination. Journal of Memory and Language, 38(2):177-202, 1998.
[46] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recogn., pages 8256-8265, 2019.
[47] Yuhuan Yang, Chaofan Ma, Chen Ju, Ya Zhang, and Yanfeng Wang. Multi-modal prototypes for open-set semantic segmentation. arXiv preprint arXiv:2307.02003, 2023.
[48] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. In Int. Conf. Learn. Represent., 2022.
[49] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
[50] Fei Zhang, Chaochen Gu, Chenyue Zhang, and Yuchao Dai. Complementary patch for weakly supervised semantic segmentation. In Int. Conf. Comput. Vis., pages 7242-7251, 2021.
[51] Fei Zhang, Tianfei Zhou, Boyang Li, Hao He, Chaofan Ma, Tianjiao Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Uncovering prototypical knowledge for weakly open-vocabulary semantic segmentation. Adv. Neural Inform. Process. Syst., 2023.
[52] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In Eur. Conf. Comput. Vis., 2022.