SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Main Architecture. As shown in Figure 3, the overall framework of PGSeg is mainly comprised of a text encoder and an image encoder. The transformer-based text encoder is used to output the text embedding, denoted as  $\mathbf{Z}^{\mathrm{t}} \in \mathbb{R}^{n \times c}$ , where  $n$  denotes the sample batch size and  $c$  denotes the feature dimension. For the image encoder, we adopt a ViT-based architecture as the image encoder. To instill the image encoder with the segmenting ability, we propose PG Unit, which is a plug-and-play module to perform grouping based on SGM. Intuitively, a multitude of PG Units, along with several transformer layers, can be sequentially connected to perform hierarchical grouping during the forward learning process. The image embedding  $Z^{\mathrm{i}} \in \mathbb{R}^{n \times c}$ , as the output of the image encoder, is generated by average-pooling and mapping the output tokens in the final PG Unit. Based on the architecture of PGSeg, we assume that the image encoder could be split into  $L$  levels if  $L$  PG Units are inserted. Formally, we denote the input tokens in the  $l$ -th level as  $\mathbf{S}_l \in \mathbb{R}^{n \times m_l \times d}$ , and  $q_l$  learnable group tokens as  $\mathbf{G}_l \in \mathbb{R}^{n \times q_l \times d}$ , where  $m_l(q_l)$  denotes the number of input patch (group) tokens in level  $l$ . Likewise, denote the output tokens in the  $l$ -th level as  $\hat{\mathbf{S}}_l \in \mathbb{R}^{n \times q_l \times d}$ . Intuitively, we hold that  $\mathbf{S}_{l+1} = \hat{\mathbf{S}}_l, l \in \{1, ..., L\}$  due to the sequential connection among the PG Units, namely the output tokens in the previous level serve as the input patch tokens in the next level.
PG Unit. To instantiate NPR, we propose PG Unit that mines out the multi-modal prototypical source from image-text representations. In this way, we propose two NPR-based strategies based on the type of prototypical pattern, namely image-level NPR (I-NPR) and text-level NPR (T-NPR). For I-NPR, we adopt the input tokens placed before the transformer layers as the image-level prototypical pattern, which are denoted as  $\mathbf{V}_l \in \mathbb{R}^{n \times m_l \times d}$ . Based on extra non-learnable prototypes  $\mathbf{P}_l \in \mathbb{R}^{n \times q_l \times d}$ , we follow the Algorithm 1 and regularize  $\mathbf{G}_l$  with  $\mathbf{V}_l$  and  $\mathbf{P}_l$ . Besides, we further reform the input tokens by  $\mathbf{S}_l = \mathbf{S}_l + \mathbf{V}_l'$ , where  $\mathbf{V}_l' \in \mathbb{R}^{n \times m_l \times d}$  are the reconstructed image-level source, as the input tokens of SGM to enhance the robustness of the model learning [10].
For T-NPR, we turn to the text embedding  $Z^{\mathrm{t}} \in \mathbb{R}^{n \times c \times 1}$  as the text-level prototypical pattern to improve the group tokens  $\mathbf{G}_l$  in capturing semantic information. Specifically, we introduce additional text prototypes  $\mathbf{T}_l \in \mathbb{R}^{n \times q_l \times 1}$  and update them with  $Z^t$ . Subsequently, we regularize  $\mathrm{AVG}(\mathbf{G}_l)$  with the updated  $\mathbf{T}_l$ , where  $\mathrm{AVG} : \mathbb{R}^{n \times q_l \times d} \to \mathbb{R}^{n \times q_l \times 1}$  averages the  $\mathbf{G}_l$  along dimension  $d$ . Essentially, T-NPR aligns the score of each group token with the center value clustered from dimension  $d$  in  $\mathbf{Z}^t$ .
Training Loss. Based on the proposed I-NPR and T-NPR in PG Unit, the overall training loss is
$$
\mathcal {L} _ {\mathrm {A L L}} = \mathcal {L} _ {\mathrm {I T}} \left(\boldsymbol {Z} ^ {\mathrm {t}}, \boldsymbol {Z} ^ {\mathrm {i}}\right) + \sum_ {l = 1} ^ {L} \sum_ {j = 1} ^ {n} \lambda \mathcal {L} _ {\mathrm {P G}} \left(\mathbf {G} _ {l} ^ {j}, \mathbf {P} _ {l} ^ {j}\right) + \beta \mathcal {L} _ {\mathrm {P G}} \left(\mathbf {A V G} \left(\mathbf {G} _ {l} ^ {j}\right), \mathbf {T} _ {l} ^ {j}\right), \tag {4}
$$
where  $\mathcal{L}_{\mathrm{IT}}$  is the symmetric image-text contrastive loss in [39],  $\lambda$  and  $\beta$  are the hyper-parameters to balance the loss. Here we empirically set  $\lambda = 0.1$  and  $\beta = 0.01$ .
Momentum update of Prototype. It is critical to set appropriate initialization of  $\mathbf{P}$  to guarantee robust convergence [42]. To this end, we leverage the exponential moving average (EMA) strategy to globally update the initial prototypes after each training round:  $\mathbf{P}_{\mathrm{(new)}} = \gamma \mathbf{P}_{\mathrm{(old)}} + (1 - \gamma)\sum_{i = 1}^{n}\mathbf{P}_{(T)}^{i}$ , where  $\mathbf{P}_{(T)}^{i}$  denotes the updated prototypes at the final  $T$ -th iteration in NPR, and  $\mathbf{P}_{\mathrm{(new)}} / \mathbf{P}_{\mathrm{(old)}}$  is the initial prototypes for the next/current training round. Empirically we set  $\gamma = 0.9$ .
Model Architecture. Following conventions in [50, 41, 35], we adopt ViT-S [47] as the visual backbone, which consists of 12 transformer layers with a hidden dimension of 384. The input image size is  $224 \times 224$  and the patch size is  $16 \times 16$ . Based on the experimental performance in [50, 41], we set  $L = 2$  for PGSeg, which means two individual PG Units are incorporated into the ViT module at different places. The number of learnable group tokens is 64 in level 1 and 8 in level 2, namely  $q_{1} = 64$ ,  $q_{2} = 8$ . Two PG Units are inserted after the  $6^{th}$  and  $9^{th}$  transformer layers, respectively. The transformer-based text encoder is the same as [39], with the hidden dimension of 256.
Datasets and Evaluation Metric. Following [50, 41, 35, 51], we use CC12M [7] and RedCaps [12] as the training sets, and each of them contains 12 million image-text pairs. For the downstream evaluation datasets, we select 5 benchmarks: PASCAL VOC12 (20 foreground classes) [16], PASCAL Context (59 foreground classes) [38], COCO Object (80 foreground classes) [32], ImageNet-S (919 foreground classes) [17] and LVIS [20] (1203 foreground classes). All of them contain 1 extra background class. We use the mean Intersection-over-Union (mIoU) as the evaluation metric.
Training and Inference Settings. During the training stage, we use Adam [25] optimizer with a weight decay of 0.05. We set the batch size as 4096, and use the cosine learning strategy with an initial learning rate of  $1.6\mathrm{e}^{-3}$ . We train the PGSeg for 40 epochs with 5 epochs of linear warm-up. As the generated features are unreliable in early epochs, we set  $\lambda = \beta = 0$  at the first 30 epochs. For the selecting threshold  $\phi$  of HRS in NPR, we experimentally set it to 0.1. The whole training process is implemented on 4 A100 GPUs, each with 80 GB of memory. During the inference, where no additional training is involved, we obtain the patch-group assignment with the cross-attention maps in two PG Units:  $A_{\mathrm{final}} = A_1A_2 \in \mathbb{R}^{m \times q_2}$ . We then generate the foreground semantic mask, and set a fixed threshold as the background score to evaluate the segmentation mask. Note that the templates of the text prompt could significantly impact the segmentation results [40]. To ensure fair comparisons, we follow the same prompt template [50, 39, 41] (a photo of...) to generate the text embedding with the foreground class name for the evaluated benchmark datasets.
5.2 Zero-shot Segmentation Performance
Comparison with Zero-shot Baselines. Similar to [41, 50], we compare our methods with seven baselines. Four of them train a plain ViT and a text encoder simply by  $\mathcal{L}_{\mathrm{IT}}$  [39], and generate the segmentation mask by using different pixel-grouping methods on the image features.
[52] find that increased [CLS] tokens in ViT serve as meaningful centers for perceptual grouping. Inspired by this, we additionally build a baseline ViT-8S to serve as a more potent parametric baseline. It trains a plain ViT-S but with increasing the [CLS] token amount into 8, and then aligns the averaged embedding of the final 8 [CLS] tokens with the text embedding in a contrastive way. ViT-8S generates the assignment by summing the cross-attention maps ([CLS] to patch) in each transformer layer. Table 1 shows
Table 1: Comparison with zero-shot baselines.
Architecture	Pixel-Grouping Methods	mIoU (%)
ViT-S	pixel-wise	20.1
ViT-S	K-means	25.0
ViT-S	Mean-shift	20.7
ViT-S	Spectral clustering	19.7
ViT-8S	X	38.1
GroupViT [50]	X	43.2
ViewCo [41]	X	45.7
PGSeg	X	49.0
the performance of these methods on the validation set of PASCAL VOC12, note that all methods here are trained simply with CC12M. It is intuitive that PGSeg outperforms all the compared baselines. Notably, PGSeg adopts the same backbone as GroupViT [50] and ViewCo [41], while achieving a  $5.8\%$  and  $3.3\%$  performance improvement over them.
Table 3: Comparison with SOTA in terms of mIoU(\%). All the image encoders here are built on ViT-S [47].  $ST$  means that [55] uses the self-training strategy on the evaluated datasets to refine the mask. The best results are highlighted in bold (underline marks the cases under the same volume).
Methods	Training Data (volume)	Pre-trained Models	VOC12	Context	COCO
RECO [45]	CC400M [39] + ImageNet1M (401M)	CLIP [39] + MOCO [21]	25.1	19.9	15.7
MaskCLIP [55]	CC400M [39] (400M)	CLIP [39]	29.3	21.1	15.5
ViL-Seg [33]	CC12M [7] (12M)	X	34.4	16.3	16.4
MaskCLIP [55]	CC400M [39] + ST (400M)	CLIP [39]	38.8	23.6	20.6
GroupViT [50]	CC12M [7] (12M)	X	41.1	18.2	18.4
OVSegmentor [51]	CC12M [7] + ImageNet1M [11] (13M)	BERT [13] + DINO [5]	44.5	18.3	19.0
PGSeg	CC12M [7] (12M)	X	49.0	20.6	22.9
GroupViT [50]	CC12M [7] + RedCaps12M [12] (24M)	X	50.8	23.6	27.5
SegCLIP [35]	CC403M [39, 7] + COCO400k [32] (403.4M)	CLIP [39]	52.6	24.7	26.5
GroupViT [50]	CC12M [7] + YFCC14M [46] (26M)	X	52.3	22.4	20.9
ViewCO [41]	CC12M [7] + YFCC14M [46] (26M)	X	52.4	23.0	23.5
PGSeg	CC12M [7] + RedCaps12M [12] (24M)	X	53.2	23.8	28.7
Group in GroupViT
Group in PGSeg
Figure 4: Qualitative results on PASCAL VOC12. Compared with GroupViT, the group tokens in PGSeg could capture the object (marked with a white circle) in a more complete and delicate way.
Semantic Output
Comparison with SOTA. Table 3 lists the mIoU of recent state-of-the-art (SOTA) methods on the validation splits of PASCAL VOC12, PASCAL Context, and COCO datasets. Since there is a huge gap among these methods in terms of the datasets and pre-trained model, we particularly report the total data volume and the specific pre-trained model that each method used during the whole training process, with the expectation for a clear and sound fair comparison. As shown in Table 3, our PGSeg achieves the SOTA performance among all the methods with the same data volume. Particularly, our method, with comparably small data volume, even achieves a more stunning performance than the methods with huge foundation models, validating the superiority and effectiveness of our PGSeg.
Challenging Segmentation. Here we introduce two challenging benchmarks to explore the potential of PGSeg in real-world segmentation. ImageNet-S, distilled from the ImageNet [11], contains 919 classes of object-centric images with human-annotated mask labels.
LVIS contains 1203 classes that are incorporated with various real-world objects. As shown in Table 2, both PGSeg and GroupViT (CC12M + RedCaps12M) are weak in segmenting these two datasets. Such frustrating performance might be due to inadequate image-text alignment of new
Table 2: Challenging segmentation results. TLG refers to the True Label Guidance strategy.
Methods	ImageNet-S (- / +TLG)	LVIS (- / +TLG)
GroupViT	19.6 / 32.5	3.2 / 6.9
PGSeg	19.7 / 33.8	3.4 / 7.2
vocabies. It is observed that both their segmentation performance could be boosted through the True Label Guidance, considering only the true labels of the evaluating samples. It is also found that PGSeg performs better than GroupViT with the TLG strategy. Therefore, we believe that the segmenting performance of PGSeg could be further enhanced by including more image-text pairs.
In this section, we use the version of PGSeg with CC12M+RedCaps12M to implement all ablation studies on PASCAL VOC12 in detail, which contain the effectiveness of the modules in the PG Unit, some analysis of the prototypes, and the computational performance of PGSeg.
Effectiveness of PG Unit. Table 4 shows the effectiveness of each designed module in the PG Unit. Recall that in Section 4.2 we propose two different NPR-based strategies in the PG Unit, namely I-NPR and T-NPR. As shown in Table 4, both these strategies are effective in enhancing the baseline, delivering  $1.75\%$  and  $0.58\%$  improvements, respectively. We also propose the HRS to further improve the performance of NPR by filtering the group-prototype pairs with a fixed selecting threshold  $\phi$ . Consequently, it is observed that proper threshold could lead to the boosting of PGSeg  $(+0.41\%)$ , which finally achieves  $53.24\%$  performance together with T-NPR.
The number of Prototypes. Recall that in Section 4.1 we mention that the number of prototypes could be different from the group tokens. Table 5 reports the performance of PGSeg with different numbers of prototypes. Note that at the  $1^{st}$  ( $2^{nd}$ ) level, the number of group tokens remains constant at 64 (8). Here we exclude the HRS to ensure a comprehensive consideration of all prototypes. We only consider the matched group-prototype pairs as the positive samples, and all other extra groups/prototypes would be considered to form the negative samples. In other words, if the number of group tokens is smaller (larger) than the number of prototypes, the symmetric  $\mathcal{L}_{\mathrm{PG}}$  would simply calculate the left (right) part accordingly. As depicted in Table 5, it has been observed that the optimal performance is attained when the number of prototypes is equal to the number of group tokens. Moreover, any increase or decrease in the number of prototypes beyond this optimal value tends to negatively impact the segmentation performance to some extent. This reveals that the number of negative samples or positive samples is vital to the performance of prototypical alignment. Clearly, the number of positive sample pairs would decrease if the number of prototypes is less than 64/8, otherwise, the number of negative sample pairs would increase. Therefore, our experimental results on the number of prototypes reach a consistent conclusion with [48, 53, 21].
Table 4: Ablation studies on the PG Unit.
Baseline	I-NPR	T-NPR	HRS-0.5	HRS-0.3	HRS-0.1	mIoU (%)
✓	50.76
✓	✓	52.51
✓	✓	51.34
✓	✓	✓	51.62
✓	✓	✓	52.57
✓	✓	✓	52.92
✓	✓	✓	✓	53.24
Table 5: Ablation studies on the number of prototypes (w.o. HRS). The group token amount is 64/8.
2ndlevel1stlevel	4	8	16
32	52.12	52.49	51.94
64	52.71	52.83	52.23
128	52.21	52.62	52.14
Figure 5: Compactness analysis. The results come from 5 clustered patch tokens based on the group tokens.
Figure 6: Dimension distributions of each group token.
Two benefits of NPR. Here we aim to investigate the nature of the two benefits, i.e., compactness and richness (mentioned in Section 4.1), to understand the effectiveness of NPR better. To validate the first one, we use tSNE to visualize the clustered 5 input patch tokens of SGM among 50 samples. As shown in Figure 5, it is intuitively found that the patch tokens in PGSeg are more tightly clustered than GroupViT, where most input tokens are comparably scattered. With the help of the compact basis in NPR, the input patch tokens become less noisy in the feature space, which is also supported by the visualized results in Figure 4 and Figure 1. The second benefit aims to enrich the feature representation to capture more accurate semantic patterns by relieving the dimensional collapse [8, 18]. As shown in Figure 6, we calculate the mean and variance of 384 dimensions for each  $1^{st}$  level group token (64 in total) based on 300 samples. It is evident that although the means are nearly identical, the dimensional variance of PGSeg is significantly larger than that of GroupViT, indicating a better dimensional representation for each group token with prototypical regularization. Overall, these two explicit benefits are validated to contribute to the enhanced segmenting ability of the group tokens.
Computational Performance. Here we present the floating-point operations (FLOPs) and model parameters for three methods in Table 6 as well. The FLOPs are calculated based on an image size of  $448 \times 448$ . It is observed that PGSeg introduces a 3.2G increase in FLOPs compared to GroupViT, but still maintains a lower FLOP count (-3.1G) than ViT-S. Therefore, the computational
Table 6: Computational performance.
Methods	FLOPs	Params
ViT-S	16.7G	21.6M
GroupViT	10.4G	28.7M
PGSeg	13.6G	35.1M
complexity of PGSeg remains manageable, indicating a reasonable level of computational efficiency.
6 Discussion with SAM
Recently, the Segment Anything Model (SAM) [26], an impressive model for interactive segmentation, has demonstrated significant progress in image segmentation tasks. SAM supports segmenting everything in an arbitrary image, which is a powerful foundation model to address OVSS. SAM is trained on a massive dataset comprising 11 billion images. In contrast, PGSeg is trained on a smaller dataset consisting of only 24 million image-text pairs. Intuitively, the data volume of SAM is approximately 460 times that of PGSeg. Despite their vast data amount, SAM also incorporates over 1 billion accurately annotated masks. Therefore, SAM is clearly better than PGSeg. Though a huge performance gap between our PGSeg and SAM, we would like to present some comparisons between these two models to present the research value of WOVSS. As illustrated in Figure 7, the segmenting groups in SAM provide comprehensive coverage of objects in an extremely fine-grained manner. In comparison, our learnable groups effectively capture entire objects without requiring instance-level recognition. For example, in the  $1^{st}$  column of the image, our yellow group can represent the overall forest background, while SAM can differentiate between individual trees within the forest background. However, it is important to note that our PGSeg model achieves comparable segmentation capabilities for certain intuitive objects, such as umbrellas, ships, babies, etc., with significantly fewer image-text pairs compared to SAM. Although vast attention has been paid to investigating the huge foundation model with vast data collection, given this impressive performance, we believe that WOVSS is a fascinating research topic that merits future investigation.
Figure 7: Class-agnostic segmentation comparison between SAM and PGSeg on LVIS.
The majority of efforts in weakly open-vocabulary semantic segmentation (WOVSS) have focused on implementing explicit grouping recognition while overlooking the customized supervision needed for the inherent clustering of group tokens. This oversight has led to a granularity inconsistency between the training and inference stages. To address this issue, our paper proposed to leverage prototypical knowledge to mine out explicit supervision, which encourages the generation of compact and rich feature representations for the group tokens. Additionally, we introduced multi-modal prototypes derived from image-text information to instantiate this knowledge, resulting in diverse patterns that enhance the segmenting ability of the group tokens. Through quantitative and qualitative experiments, we have demonstrated the effectiveness and superiority of this straightforward concept. In all, our bottom-up concept further validates the potential of prototypes, which exhibit compactness and richness, as promising elements for the top-down segmentation methods. Therefore, we believe that it is worth further exploring the full potential of prototypes in more weakly supervised tasks.
[54] Fei Zhang, Chaochen Gu, Chenyue Zhang, and Yuchao Dai. Complementary patch for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7242-7251, 2021.
[55] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII, pages 696-712. Springer, 2022.
[56] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmentation: A prototype view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2582-2593, 2022.
[57] Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pages 27367-27377. PMLR, 2022.
[58] Benjin Zhu, Zhe Wang, Shaoshuai Shi, Hang Xu, Lanqing Hong, and Hongsheng Li. Conquer: Query contrast voxel-detr for 3d object detection. arXiv preprint arXiv:2212.07289, 2022.