Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation
Authors: Fei Zhang
Tianfei Zhou
Boyang Li
Hao He
Chaofan Ma
Tianjiao Zhang
Jiangchao Yao
Ya Zhang
Yanfeng Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Weakly (Text-based) Open-Vocabulary Semantic Segmentation
Vision-Language Pretraining
Prototypical Knowledge
TL;DR: Exploring the prototypical knowledge to address Weakly Open-Vocabulary Semantic Segmentation

Abstract: This paper studies the problem of weakly open-vocabulary semantic segmentation (WOVSS), which learns to segment objects of arbitrary classes using mere image-text pairs. Existing works turn to enhance the vanilla vision transformer by introducing explicit grouping recognition, i.e., employing several group tokens/centroids to cluster the image tokens and perform the group-text alignment. Nevertheless, these methods suffer from a granularity inconsistency regarding the usage of group tokens, which are aligned in the all-to-one v.s. one-to-one manners during the training and inference phases, respectively. We argue that this discrepancy arises from the lack of elaborate supervision for each group token. To bridge this granularity gap, this paper explores explicit supervision for the group tokens from the prototypical knowledge. To this end, this paper proposes the non-learnable prototypical regularization (NPR) where non-learnable prototypes are estimated from source features to serve as supervision and enable contrastive matching of the group tokens. This regularization encourages the group tokens to segment objects with less redundancy and capture more comprehensive semantic regions, leading to increased compactness and richness. Based on NPR, we propose the prototypical guidance segmentation network (PGSeg) that incorporates multi-modal regularization by leveraging prototypical sources from both images and texts at different levels, progressively enhancing the segmentation capability with diverse prototypical patterns. Experimental results show that our proposed method achieves state-of-the-art performance on several benchmark datasets.

Uncovering Prototypical Knowledge for Weakly Open-Vocabulayy Semantic Segmentation
Fei Zhang!
Tianfei Zhou
Boyang Li
Hao He
Chaofan Ma
Tianjiao Zhang'
Jiangchao Yao 1.2
Ya Zhang 1.
Yanfeng Wang"
1CMIC, Shanghai Jiao Tong University
Shanghaii Al Laboratory
Beijing Institute of Technology *National University of Defense Technology
CUHK
ferenasūsjtu. edu. ztfei. sefeii.debu@gmail.com {chaofanma, xiaoeyuztj, Sunarker, ya ya_zhang, vangyanfengg6222)eejjhaa edu.
Abstract
This paper studies the problem of weakly open-vocabulary semantic segmenta- tion (WOVSS), which learns to segment objects of arbitrary classes using mere image-text pairs. Existing works turn to enhance the vanilla vision transformer by introducing explicit grouping recognition, i.e., employing several group to- kens/centroids to cluster the image tokens and perform the group-text alignment. Nevertheless, these methods suffer from a granularity inconsistency regarding the usage of group tokens, which are aligned in the all-to-one one-to-one manners during the training and inference phases, respectively. We argue that this discrepancy arises from the lack of elaborate supervision for each group token. To bridge this granularity gap, this paper explores explicit supervision for the group tokens from the prototypical knowledge. To this end, this paper proposes the non-learnable prototypical regularization (NPR) where non-learnable prototypes estimated from source features to as supervision and enable contrastive matching of the group tokens. This regularization encourages the group tokens to segment objects with less redundancy and capture more comprehensive semantic regions, leading to increased compactness and richness. Based on NPR, we pro- pose the prototypical guidance segmentation network (PGSeg) that incorporates multi-modal regularization by leveraging prototypical sources from both images and texts different levels, progressively enhancing the segmentation capability with diverse prototypical patterns. Experimental results show that our proposed method achieves state-oftthe-a performance on several benchmark datasets. The source code is available at https: / thub. homm/Feeena//PGGC
Introduction
Recently, the remarkable success of vision-language pre-training (VL.P) methods [39, 29, 2] has invigorated the field of semantic segmentation, one of the prominent computer vision tasks. This advancement has led to the emergence of an intriguing task known as open-vocabulary semantic segmentation (OVSS), which aims to segment object pixels belonging arbitrary classes beyond pre-defined categories. To address this challenge, most works [22, 19, 34, 37] have turned to large quantity of image-text pairs equipped with precisely-annotate masks. To liberate OVSS from exhaustive pixel-level ground truth, we in this paper excavate weakly open- vpccbblcayylrry semantic segmentation (WOVSS), a more arduous setting that achieves OVSS with mere image-text pairs.
denotes the corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Under//Overseegmentng Region
Group Token
(Baseline Method)
Informative Expansion/Uniinformatie Reduction
Prototypical Knowledge
Richness
Compactness
Group Token
(Our Method)
Figure I: Illustration of our motivation. Our method explores the prototypical knowledge to provide explicit supervision for the group tokens, which improves the segmentation results with the increased richness and compactess.. The former improves the feature representation of group tokens, resulting in enlarged semantic regions, and the latter reduces cluster redundancy and noise.
To learn from vast image-text data, vision transformer (ViT) [14] has exhibited impressive advances in acquiring powerful visual representations from the text [39, 29J. However, the vanilla ViT lacks an explicit grouping component, rendering it inadequate for achieving comparable fine-grained segmentation with only text supervision. To imbue ViT with the potential for segmenting ability, most WOVSS approaches [33, 50, 41, 51, 35] proposed to cluster the patch-level visual features into several learnable group tokens/centroids, and process the group-text alignment generate the corresponding categories. Though effective, these methods inevitably are plagued with granularity inconsistency concerning the group tokens. During the training stage, these learnable group tokens are averaged to facilitate all-to-one group-text alignment, while a one-to-one alignment strategy is employed during zero-shot inference (please refer Figure for more details). This inconsistency arises due to the weak supervision inherent in WOVSS, otherwise, they could be regularized with, e.g. pixel-level ground truth, to perform promising segmentation results as normal OVSS methods do [9, 28]. To break this ill ill-reguariized learning pattern, this paper aims to bridge the granularity gap by exploring explicit supervision for the group tokens, remedying the flawed supervision in WOVSS.
Before delving into the proper guidance for group tokens, let us revisit an interesting question: What constitutes good cluster? Such question drives to put forward two properties that a reliable group centroid should possess. 1) Compactness ensures that a cluster centroid and its clustered items are closely located in the feature space, forming a compact cluster with minimal noise and redundancy [31, 24, 30). 2) Richness refers to the capability of a centroid to capture diverse and accurate patterns, thereby enhancing zero-shot generalization capability [3, 56]. These two properties motivate us to find the supervision by exploiting the prototypical knowledge [4, 31, 30] from an expectation- maximization (EM) estimated data density. To this end, we propose the non- learnable prototypical regularization (NPR) that adopts the Gaussian mixture models (GMM) [42], one of the soft clustering models, to generate the supervision from the source features for each group token. Specifically, we treat the learned Gaussian distributions from the source features prototypes, which are then used align with the group tokens in a contrastive manner. Notably, each non- learnable prototype (uninvolved in gradient backpropagation) is able to regularize the corresponding group token, enabling to segment compactly and richly. As shown in Figure 1, the group tokens could benefit from this prototypical supervision to segment the objects with less redundancy and more accurate semantic patterns, effectively alleviating the under- and over-segmentation problems.
To instantiate the prototypical patterns in NPR, this paper introduces novel investigation into using multi-modal information the source features. The low-level image features, with detailed texture information, could be an intuitive choice to refine the segmentation results [44, 49, 1]. Beyond such simple uni-modality, we further mine the prototypes from the text regularize the group token with textual information. Hence we propose two strategies, namely image-level NPR (I-NPR) and text- level NPR (T-NPR), to provide multi-m miltiity regularization to the group tokens. Based on this, we propose the protorypical guidance segmentation network (PGSeg), hierarchical segmentation model that incorporates I-NPR and T-NPR into the group tokens different levels, progressively improving the segmenting ability of the group tokens. Overall, we make the following contributions:

---

We propose NPR that explores and exploits the prototypical knowledge to serve as a valid supervi- sion of the group tokens in segmenting objects. This explicit regularization is encouraged bring compact and rich feature representation to the group tokens.
We propose PGSeg, simple yet effective segmentation architecture that extracts the prototyp- ical knowledge from both the image and text to regularize the group tokens at different levels, progressively guiding the group tokens segment in an explicit way.
Extensive results on several benchmarks demonstrate the superiority and effectiveness of our method. Particularly, method yields new state-of-the-ar performance by reaching 53.2% mloU and 28.7% mloU on PASCAL VOC12 [16] and COCO [32], respectively. It is worth highlighting that our PGSeg model, trained solely on the CC12M dataset [7], surpasses some sfate-of-the- art methods utilizing large foundation models such as CLIP [39] and BERT [13] by up to 14.5 % and 5.2% in terms of mloU on PASCAL VOC12 and COCO, respectively.
2 Related Work
Weakly Open- Veee-Vocabllrry Semantic Segmentation. Most existing works addressing WOVSS can be categorized into two groups based on whether CLIP [39] or Diffusion Models [43] is employed as the fundamental model. The first category focuses on extracting coarse localization features from CLIP or Stable Diffusion Models and subsequently refining them to achieve fine- ane-grained segmentation results [55, 6, 36]. The second category of approaches, distinct from those focused on CLIP, centers around enhancing the plain VIT by incorporating grouping recognition, resulting in foundational segmentation model [33, 50, 41, 51J. In these methods, several learnable group tokens/centroids are introduced to extract visual concepts from the image features. [50] proposed Group ViT that assigns these tokens to input patch tokens, enabling a learnable clustering process during training. [35] also presented grouping-basee approach, and introduced a reconstruction loss and a superpixel-base regularization loss to improve the inner clustering results. Our work aligns with the second category of approaches. Note that the setting of WOVSS is extremely similar to weakly supervised semantic segmentation (WSSS), where segmentation model is obtained with simply image- level labels. Most works addressing WSSS aim to use the low-level of the image information to iteratively refine the segmentation mask [44, 1, 54], requiring massive additional training inference stages the target dataset. Therefore, this paper explores an end-to-end mechanism that efficiently incorporates low-level information on the segmentation mask.
Prototypes for Deep Representation Learning. The prototypes typically refer to the centroids from conventional clustering methods [15]. Based on the expectation-maxxiizztiie prototypes are learned by estimating the data features through a mixture of prior distributions. As a result, prototypes are often regarded as "non-learnable" since they deviate from the usual gradient- learning in deep neural networks [31, 56). The inclusion of prototypical patterns has been extensively explored to enhance feature representation learning in contrastive learning (CL) [3, 4, 5, 31, 57], which aims to match the feature embeddings of a pair of aligned samples. The success of these approaches highlights two important benefits that prototypes bring to feature approaches alignment. The first goes to the compactness [31, 30, 24], where they find that the prototypes could reformulate features into more compact representation, reducing redundancy and noise in feature alignment. This leads to more reliable feature representations. Another benefit is to enhance the richness of the feature representation by capturing more learning patterns. CL often suffers from the dimensional collapse, where embedding vectors occupy a lower-dimensional subspace than their original dimension, resulting limited diversity in feature representation. To address it, line of work has leveraged the prototypes to serve as a constraint on the feature alignment, which is validated effectively enrich the feature representation [3, 4, 5, 53]. This work explores the use of prototypical knowledge with the expectation of providing the aforementioned advantages to segmentation clusters.
(EM) algorithm,
3 Rethinking the Semantic Grouping Mechanism in WOVSS
To effectively tackle WOVSS, recent works [50, 41, 35, 51] have placed significant emphasis on incorporating explicit grouping recognition into the plain model. To this end, these meth- ods developed a semantic grouping mechanism (SGM) based on ViT [14]. Formally, given
m input patch tokens S = [sL, 82, .., 8m] pm and extra g learnable group tokens G
3

---

U1.92...... E Roxd where d is the dimension of data and q m. SGM clusters S and out- puts clustered tokens $ Rexd by S Gumbel-Softmax: (()
G, where
Q Rexd + Rexd XC (V) IR= + m represent the Query, Key (Value) mapping function. Figure 2 clearly demonstrates this cross-attentionnbatt ken is assigned to corresponding group token by the Straight- Thrrohhh fumbel-Softane making this process end-to-end trainable. We formulate this patch-grout assignment as A = ((G) E R2xx.. By plugging the SGM into ViT, the vanilla image encoder could be empowered with potential segmenting ability. However, it could be observed that this mecha- nism presents a granularity inconsistency for the group tokens between the training and inference stages (as shown in Figure 2). More specifically, during the training phase, all group tokens are globally averaged to match the corresponding text embedding for the final group-text alignment, while during the inference phase, each group token ne- cessitates a separate com- parison with each class em- bedding in order to acquire the semantic label, which is then used to annotate the corresponding image re- gions based on the patch- group assignment. Con- sequently, the group to- kens used in the one-to-one group-text alignment do not receive explicit supervision they subjected all-to-one text-based regularization during the training stage. This discrep- ancy in supervision may contribute to the performance gap observed between OVSS and WOVSS. In OVSS, each learnable group token can be treated as query embedding, generating dense embedding for object mask prediction. This dense embedding can be further regularized by the patterns extracted from pixel-level ground truth annotations [9, 28, 58]. Therefore, such granularity discrepancy essentially is derived from the weak supervision of WOVSS. Despite the challenges posed, we motivated explore customized regularization techniques for each group token, aiming to compensate for the absence of pixel-level annotations. By explicitly addressing the granularity gap, we aim to enhance the segmentation performance WOVSS.
clustering process. Here each patch to-
[23],
SGM
All-to-one (Training)
df
Embedding man standing
Avg-Paol
Gumbel- Softmax
Output Token (6)
Granularity Inconsstency
One- to one (Inference)
Class Embedding
Class1 (Person)
Mappiing
mxd
mx
Class2 (Bicycle)
Classă (Background)
Patch Token (5)
Group Token
Figure 2: The granularity inconsistency of SGM.
Methods
4.1 Exploring Prototypical Knowledge for Explicit Supervision
To find explicit and reliable supervision, turn to the prototypical knowledge to form regularized basis that can bring certain benefits to the group token in segmentation. Gaussian mixture models (GMM) [42] has been experimentally validated to form basis that could reduce the redundancy of the features [31, 24]. Inspired by this, we propose the non nea-laereable prototypical regularization (NPR) that uses GMM to extract the prototypes from the prototypical source (like way of data mining), and then aligns such prototypes with the group centroids in a contrastive way.
Prototype Generation. The first stage of NPR is to generate the supervision by GMM. GMM is based on a mixture Gaussian distributions, where each component of the mixture corresponds to cluster in the data. Formally, given the prototypical source features V [V1, ..., vm.] mg and extra q randomly-initializze prototypes P [Pt,... Pgl Raxd where m and d represents the number and the dimension the prototypical source. this way, the distribution of V could be expressed as P(V) = Σi= TWN(VV (V pp, Σ,), where E R' P, E Rd and the Σ; E Rdxd are the weight, mean and covariance the i-th Gaussian component. Here we take the means as the prototypes. To work out (P, Σ, π). the log-likelihood of p(V) is maximized through the syprecation-muxiiizutiii (EM) algorithm, which optimizes the model, until convergency reached, by alternating between an expectation (E) step and maximization (M) step. In the E step, the probability of j-th source feature

---

Algorithm Non-learnable Prototypical Regularization (NPR)
Require: Group tokens G E Rxxd prototypes P E Rix4 prototypical source features V E R
iterations T (T' 10 in our setting), selecting threshold
1: >> Prototype Generation
2: for iteration f
do
3: (E-step) Calculate the probability of V belonging to P in Eq. (1)
4: (M-step) Update the prototypes P by using Eq. (2)
5: end for
6: D- Prototype Supervision
7: Generate the matched prototypes pi by using the Hungarian matching between P and G
8: Select the matched pairs (Pli G) whose similarity scores are beyond Φ
9: Regularize the selected G' with the matched ph by using LpG in Eq. (3)
2 belonging to the i-th Gaussian prototype could be expressed as
Z(u;(p:)
exp(p.u,)) Σ exp(p.u))" i
Vij E1 (U,)pp))
(0.......
(1)
where 2: R - R denotes kernel function. For simplicity, set Σ the identity matrix I and leave out æ. We experimentally observe that the different choices of 2 negligible effects on the final results, so simplify the Gaussian Kernel exp (-I1a: yll2/ /2)²) to the exponential inner dot exp (xyy) Based on the estimated Yıj. the prototypes in the M step could be updated
p:
(2)
Mij
After alternatively repeating the E step and step, bunch of compact prototypes representing the prototypical information could be obtained to supervise the group tokens. Note that we could reconstruct the prototypical source by V
P mmd Y Warlox.
Prototype Supervision. The second stage is to regularize G with the updated P. Based on the cosine similarity between P and G, we first perform Hungarian matching [27] ensure that each centroid g has a corresponding target prototype p. Denote the matched prototypes as P =
... E Raxa Then, combine the matched pair of (g, p") as the positive samples, and propose the Prototypical Guidance (PG) loss LPG to regularize the group centroids in a contrastive way:
log exp(S(p).g))) Lpa(G, - - (log Ej (xp( //T Σ exp(S(pp) g))//
exp(S(gr, p )/T))
(3)
where 0.1 is the temperature hyper-parameter, and S(a, b) similarity between two vectors. Based on the PG loss, we introduce a simple hard rejecting strategy (HRS) that only considers the positive pairs whose similarity is beyond fixed threshold ø. We claim that one group centroid could be wrongly guided by the matched prototype once they have significant difference, which will be discussed in Section 5.3. Besides, here assume that the number of prototypes and group centroids is the same, and we will also show the case where the number of them is different in Section 5.3 (only the matched pairs are considered to calculate LPG).
ab |||l||||| calculates the cosine
Compactness Richness. The essence of NPR is to regulate each group centroid with a normalized prototype from prior distribution, yielding two essential benefits as discussed 2. The first one is the compactness, which helps refine the clustered results by reducing noise and redundancy [31, 24]. The second one goes to the richness that empowers the group tokens with rich feature representation by relieving the dimensional collapse through the application of normalized regularization [4, 18), capturing more accurate patterns possible. In all, believe that NPR could augment the segmenting ability of the group tokens with these two benefits, which will be validated in Section 5.
Complexity Analysis. The EM algorithm is the key part in NRP (Prototype Generation in Algo- rithm 1). crucial to carefully consider its time complexity regarding iterative learning. However, through simplified implementation, we demonstrate that the time complexity of prototype gener- ation in NPR is Olq x m x d) for single sample. This complexity is reasonably acceptable for implementation purposes. The actual computational performance will be demonstrated in Section 5.3.
5

---

A dog on
the floor Text leput
Token
Text Encoder
Text Embedding
Contrastive
Loss
Image Embedding (z))
MLP
AVG Pool
Level Unit
Output Token
PG
SGM
AVG
Updated Prototype
Updated Prototype
Patch
Transformer
(
Layer
Reconstructed Patch Taken
Text Embocdding
Image Irpect
Patch Taken (6)
Group Tokken
Prototype
C Prototype (R)
1|00 Patch Token (S)
I Group Token (G)
I Prototype (1)
Level
Forward Learning
T-NPR Flow
(Non- Learnalara Figure 3: The overall framework of our proposed PGSeg. PGSeg consists of an image encoder and a text encoder. Through sequential connections facilitated by the PG Unit and several transformer layers, the image encoder is organized into multiple hierarchical levels (highlighted orange), enabling the progressive grouping of the input patch tokens. Best viewed in color.
(Learnable)
(Non-Learnable)
4.2 Exploiting Prototypical Guidance for WOVSS
In this section, we introduce our proposed prototypical guidance segmentation network (PGSeg) in detail, which incorporates the proposed NPR into SGM to address WOVSS.
Main Architecture. As shown in Figure 3, the overall framework of PGSeg is mainly comprised of a text encoder and an image encoder. The transformer-basee text encoder is used to output the text embedding. denoted as Z' TRaxe, where n denotes the sample batch size and e denotes the feature dimension. For the image encoder, we adopt a ViT-based architecture as the image encoder. To instill the image encoder with the segmenting ability, we propose PG Unit, which a plug-and-play module to perform grouping based on SGM. Intuitively, multitude of PG Units, along with several transformer layers, can sequentially connected to perform hierarchical grouping during the forward learning process. The image embedding Z E gnxe, the output of the image encoder, is generated by scorage-pooling and mapping the output tokens in the final PG Unit. Based on the architecture of PGSeg, assume that the image encoder could be split into L levels if L PG Units are inserted. Formally, we denote the input tokens in the 1-th level as S E Rnx xd and qi learnable group tokens as G Raxqiix xd where my (qr) denotes the number of input patch (group) tokens in level 1. Likewise, denote the output tokens in the 1-th level as $ E | xd Intuitively, we hold that S = S, 1 {l.... L} due the sequential connection among the PG Units, namely the output tokens in the previous level serve as the input patch tokens in the next level.
PG Unit. To instantiate NPR, propose PG Unit that mines the multi-moda, prototypical source from image-text representations. In this way, we propose two NPR-based strategies based on the type of prototypical pattern, namely image-level NPR (I-NPR) and text-level NPR (T-NPR). For I-NPR, we adopt the input tokens placed before the transformer layers as the image-level prototypical pattern, which are denoted Vi Raxmixd Based on extra non-learnable prototypes P1 Ruxqixd we follow the Algorithm and regularize G2 with V, and P,. Besides, we further reform the input the reconstructed image-level source, as the
tokens by
S + V,, where V Rnxm:xe input tokens of SGM to enhance the robustness of the model learning [10].
For T-NPR, turn to the text embedding Z 1xxx the text-level prototypical pattern to improve the group tokens G, in capturing semantic information. Specifically, we introduce additional text prototypes T2 Rax x1 and update them with Z'. Subsequently, regularize AVG G.) with the updated T,, where AVG gnx + x averages the G along dimension d. Essentially, T-NPR aligns the score each group token with the center value clustered from dimension Z'.
Training Loss. Based on the proposed I-NPR and T-NPR in PG Unit, the overall training loss is
LALI = Lir(Z' (Z', z²) Σ Σ ACpG (G., ) + BCrG (AVG T(),
(4)
where LIT is the symmetric image-text contrastive loss in [39], and are the hyper-parameters to balance the loss. Here empirically set À 0.1 and B 0.01.

---

Momentum update of Prototype. It is critical to set appropriate initialization of P to guarantee robust convergence [42]. To this end, we leverage the exponential moving averaage (EMA) strategy to globally update the initial prototypes after each training round: P (nnew) yP (old) + (1
7) Di P (ry where Pir)) denotes the updated prototypes at the final 7- iteration in NPR, and
P (neww/ (oldd is the initial prototypes for the next/current training round. Empirically we set y 0.9.
5 Experiments
5.1 Implementation Details
Following conventions in [50, 41, 35], we adopt VIT-S [47] as the visual backbone, which consists of 12 transformer layers with hidden dimension of 384. The input image size is 224x224 and the patch size is 16x 16. Based on the experimental performance in [50, 41J, we set L 2 for PGSeg, which means two individual PG Units are incorporated into the VIT module at different places. The number of learnable group tokens is 64 in level and in level 2, namely q1 64, 92 8. Two PG Units are inserted after the 6th and 9th transformer layers, respectively. The transformer-base text encoder is the same as [39], with the hidden dimension of 256.
Model Architecture.
Datasets and Evaluation Metric. Following [50, 35,35,51 we use CC12M [7] and RedCaps [12] as the training sets, and each of them contains 12 million image-text pairs. For the downstream evaluation datasets, select 5 benchmarks: PASCAL VOCI2 (20 foreground classes) [16], PASCAL Context (59 foreground classes) [38]. COCO Object (80 foreground classes) [32], ImageNet-S (919 foreground classes) [17] and LVIS [20] (1203 foreground classes). All of them contain I extra (mloU) as the evaluation metric.
background class. We use the mean Antereection-over-mmee
Training and Inference Settings, During the training stage, we use Adam [25] optimizer with a weight decay of 0.05. We set the batch size as 4096, and use the cosine learning strategy with an initial learning rate of 1.6e-3 We train the PGSeg for 40 epochs with 5 epochs of linear warm-up. As the generated features are unreliable in early epochs, we set X = For the selecting threshold & of HRS in NPR, we experimentally set it to 0.1. The whole training process is implemented on 4 A100 GPUs, each with 80 GB of memory. During the inference, where no additional training is involved, we obtain the patch-group assignment with the cross-attention maps in two PG Units: Afinll = A A E mmx We then generate the foreground semantic mask, and set a fixed threshold as the background score evaluate the segmentation mask. Note that the templates of the text prompt could significantly impact the segmentation results [40]. To ensure fair comparisons, we follow the same prompt template [50, 39, 41] (a photo of...) to generate the text embedding with the foreground class name for the evaluated benchmark datasets.
0 at the first 30 epochs.
5.2 Zero-shot Segmentation Performance
Comparison with Zero-shot Baselines. Similar to [41, 50], we compare our methods with seven baselines. Four of them train a plain ViT and a text encoder simply by frr [39], and generate the segmentation mask by using different pixel-grouping methods the image features. [52] find that increased [CLS] tokens in VIT
Table 1: Comparison with zero-shot baselines.
serve meaningful centers for perceptual grouping. Inspired by this, we additionally build baseline VIT-8S to serve metric baseline. It trains plain ViT-S but with increasing the [CLS] token amount into 8, and then aligns the averaged embedding of the final 8 [CLS] tokens with the text embedding in a con- trastive way. ViT-8S generates the assignment by summing the cross-attention maps (CCLS) to patch) in each transformer layer. Table shows the performance of these methods on the validation set of PASCAL VOC12, note that all methods here are trained simply with CC12M. It is intuitive that PGSeg outperforms all the compared baselines. Notably, PGSeg adopts the same backbone as Group [50] and ViewCo [41], while achieving a 5.8% and 3.3 % performance improvement over them.
Architecture Pixel-Grouping Methods mloU (%)
more potent para-
ViT-S
pixel-wise
20.
VIT-S
K-means
25.0
VIT-S
Mcan-shift
20.7
VIT-S
Spectral clustering
19.7
VIT-8S
x
38.1
Group ViT [50]
x
43.2
ViewCo [41]
x
45.7
PGSeg
X
49.0
7

---

Table 3: Comparison with SOTA in terms of ml). All the image encoders here are built on ViT-S [47]. ST means that [55] uses the self-training strategy on the evaluated datasets to refine the mask. The best results are highlighted bold (underline marks the cases under the same volume).
Methods
Training Data (volume)
Pre-trained Models
VOC Context
COCO
RECO [45]
CC400M [39] + ImageNetIM (401M)
CLIP [39] + MOCO [21]
25.1
19.9
15.7
MaskCLIP [55]
CC400M [39] (400M)
CLIP [39]
29.3
21.1
15.5
ViL-Seg [33]
CC12M [7] (12M)
X
34.4
16.3
MaskCLIP [5S]
CC400M [39] ST (400M)
CLIP [39]
38.8
23.6
20.6
GrT [50]
CC12M [7] (12M)
X
41.1
18.2
18.4
OV OVSegmentor [51]
CC12M [7] ImageNetIM (13M)
BERT [13) DINO [5]
44.5
18.3
19.0
PGSeg
CC12M [7] (12M)
X
49.0
20.6
22.9
Group [50]
CC12M [7] RedCaps 2M [12] (24M)
X
50.8
23.6
27.5
SegCLIP [35]
CC403M [39, 7] + COCO400k [32] (403.4M)
CLIP [39]
52.6
24.7
26.5
Group VTT [50]
CC12M [7] + YFCC14M [46] (26M)
X
52.3
22.4
20.9
ViewCO [41]
CC12M [7] YFCC14M [46] (26M)
X
52.4
23.0
23.5
PGSeg
CC12M [7] RedCaps12M [12] (24M)
X
53.2
23.8
28.7
Inpul
Group GrouNViT'
Group POSeg
Cuigut
Figure 4: Qualitative results on PASCAL VOC12. Compared with Group VIT, the group tokens in PGSeg could capture the object (marked with white circle) in more complete and delicate way.
Comparison with SOTA. Table 3 lists the mloU of recent state-of-the-ar (SOTA) methods on the validation splits of PASCAL VOCI2, PASCAL Context, and COCO datasets. Since there is a huge gap among these methods in terms of the datasets and pre-trained model, we particularly report the total data volume and the specific pre-trained model that each method used during the whole training process, with the expectation for clear and sound fair comparison. As shown in Table our PGSeg achieves the SOTA performance among all the methods with the same data volume. Particularly, our method, with comparably small data volume, even achieves more stunning performance than the methods with huge foundation models, validating the superiority and effectiveness of our PGSeg.
Challenging Segmentation. Here we introduce the potential of PGSeg in real-world segmentation. ImageNet-S, distilled from the Ima- geNet [11], contains 919 classes of object-centric images with human-annotated mask labels. LVIS contains 1203 classes that are incorporated with various real-world objects. As shown in Table 2, both PGSeg and Group VIT (CC12M+ + RedCaps12M) are weak in segmenting these two datasets. Such frustrating performance might be due to inadequate image-text alignment of new vocabularies. It is observed that both their segmentation performance could be boosted through the also found
challenging benchmarks explore
Table 2: Challenging segmentation results. TLG refers to the True Label Guidance strategy.
Methods ImageNet-S (-/ +TLG) LVIS (-/ +TLG)
GroupViT PGSeg
19.6/ 32.5
3.2
19.7 / 33.8
3.4/7.2
True Label Guidance, considering only the true labels of the evaluating samples. that PGSeg performs better than Group ViT with the TLG strategy. Therefore, we believe that the segmenting performance PGSeg could be further enhanced by including more image-text pairs.
5.3 Ablation Studies
In this section, we use the version of PGSeg with CC12M-RedCaps 12M to implement all ablation studies on PASCAL VOC12 detail, which contain the effectiveness of the modules the PG Unit some analysis of the prototypes, and the computational performance of PGSeg.
8

---

Effectiveness of PG Unit. Table 4 shows the effectiveness of each designed module in the PG Unit. Recall that in Section 4.2 we propose two different NPR-based strategies in the PG Unit, namely I-NPR and T-NPR. As shown in Table 4, both these strategies are effective in enhancing the baseline, delivering 1.75% and 0.58% improvements, respectively. We also propose the HRS to further improve the performance of NPR by filtering the prouppprototype pairs with fixed selecting threshold ø. Consequently, it is observed that proper threshold could lead to the boosting of PGSeg (+0.41%), which finally achieves 53.24% performance together with T-NPR.
The number of Prototypes. Recall that Section 4.1 we mention that the number of prototypes could be different from the group tokens. Table 5 reports the performance of PGSeg with different numbers prototypes. Note that the 1s (2nd) level, the number of group tokens remains constant at 64 (8). Here we exclude the HRS to ensure a comprehensive consideration of all prototypes. We only consider the matched group-preee poouppppttttpee pairs as the positive samples, and other extra groups/prototypes would be considered to form the negative samples. In other words, if the number of group tokens smaller (larger) than the number of prototypes, the symmetric LrG would simply calculate the left (right) part accordingly. As depicted in Table 5, it has been observed that the optimal performance is attained when the number of prototypes is equal to the number group tokens. Moreover, any increase or decrease in the number of prototypes beyond this optimal value tends to negatively impact the segmentation performance to some extent. This reveals that the number of negative samples positive samples is vital to the performance of prototypical alignment. Clearly, the number of positive sample pairs would decrease if the number of prototypes is less than 64/8, otherwise, the number of negative sample pairs would increase. Therefore, experimental results on the number of prototypes reach a consistent conclusion with [48, 53, 21J.
Table 4: Ablation studies on the PG Unit.
Table 5: Ablation studies the number of prototypes (w.o. HRS). The group token amount is 64/8.
Baseline I-NPR T-NPR HRS-0.5 HRS-0.3 HRS-0.1 mloU (%)
50.76
52.51
51.34
2nd level
8
16
51.62
1 level
52.57
52.12
52.49
51.94
52.92
64
52.71
52.83
52.23
53.24
128
52.21
52.62
52.14
The Mean Variance of ALl Dinvessios
GIT
PGSag
GroupVit VIT
PGSeg
24
40
Group Token ID
Figure 5: Compactness analysis. The results come from 5 clus- tered patch tokens based on the group tokens.
Figure 6: Dimension distribu- tions of each group token.
Two benefits of NPR. Here we aim to investigate the nature of the two benefits, i.e., compactness and richness (mentioned in Section 4. 1), to understand the effectiveness of NPR better. To validate the first one, we use tSNE to visualize the clustered 5 input patch tokens of SGM among 50 samples. As shown in Figure 5, it is intuitively found that the patch tokens in PGSeg are more tightly clustered than Group ViT, where most input tokens are comparably scattered. With the help of the compact basis in NPR, the imput patch tokens become less noisy in the feature space, which also supported by the visualized results in Figure and Figure 1. The second benefit aims enrich the feature representation to capture more accurate semantic patterns by relieving the dimensional collapse [8, 18]. As shown in Figure 6, calculate the mean and variance of 384 dimensions for each 1st level group token (64 in total) based on 300 samples. It is evident that although the means are nearly identical, the dimensional variance of PGSeg significantly larger than that of GroupViT, indicating a better dimensional representation for each group token with prototypical regularization. Overall, these two explicit benefits are validated to contribute the enhanced segmenting ability of the group tokens.

---

Computational Performance. Here we present the floating- point operations (FLOPs) and model parameters for three methods in Table 6 as well. The FL.PPs are calculated based on an image size of 448 x 448. is ob- served that PGSeg introduces a 3.2G increase in FLOPs compared to Group VIT. but still maintains lower FLOP count (-3.1G) IG) than ViT-S, Therefore, the computational complexity of PGSeg remains manageable, indicating a reasonable level of computational efficiency.
Table 6: Computational performance.
Methods
FLOPs
Params
ViT-S
16.7G
21.6M
Group ViT
10.4G
28.7M
PGSeg
13.6G
35.1M
Discussion with SAM
Recently, the Segment Anything Model (SAM) [26], an impressive model for interactive segmentation, has demonstrated significant progress in image segmentation tasks. SAM supports segmenting everything in an arbitrary image, which is a powerful foundation model to address OVSS. SAM trained on a massive dataset comprising 11 billion images. In contrast, PGSeg trained on smaller dataset consisting of only 24 million image-text pairs. Intuitively, the data volume of SAM approximately 460 times that of PGSeg. Despite their vast data amount, SAM also incorporates over billion accurately annotated masks. Therefore, SAM is clearly better than PGSeg. Though huge performance gap between our PGSeg and SAM, we would like to present some comparisons between these two models to present the research value of WOVSS. As illustrated in Figure 7, the segmenting groups in SAM provide comprehensive coverage of objects in an extremely fine-grained manner. In comparison, our learnable groups effectively capture entire objects without requiring instance-level recognition. For example, in the 1a column of the image, our yellow group can represent the overall forest background, while SAM can differentiate between individual trees within the forest background. However, is important to note that our PGSeg model achieves comparable segmentation capabilities for certain intuitive objects, such umbrellas, ships, babies, etc., with significantly fewer image-text pairs compared to SAM. Although vast attention has been paid to investigating the huge foundation model with vast data collection, given this impressive performance, we believe that WOVSS a fascinating research topic that merits future investigation.
PGSe
SAM
Figure 7: Class-agnostic segmentation comparison between SAM and PGSeg LVIS.
7 Conclusion
The majority of efforts in weakly open-vocabulary semantic segmentation (WOVSS) have focused on implementing explicit grouping recognition while overlooking the customized supervision needed for the inherent clustering of group tokens. This oversight has led granularity inconsistency between the training and inference stages. To address this issue, our paper proposed to leverage prototypical knowledge mine out explicit supervision, which encourages the generation of compact and rich feature representations for the group tokens. Additionally, we introduced multi-mode prototypes derived from image-text information to instantiate this knowledge, resulting in diverse patterns that enhance the segmenting ability of the group tokens. Through quantitative and qualitative experiments, we have demonstrated the effectiveness and superiority of this straightforward concept. In all, our bottom-up concept further validates the potential of prototypes, which exhibit compactness and richness, as promising elements for the top-down segmentation methods. Therefore, we believe that it is worth further exploring the full potential of prototypes in more weakly supervised tasks.
10

---

8 Acknowledgement
This work is supported by the National Key R&D Program of China (No. 20222D0160703), STCSM (No. 2251 1 106101, No. 22511 105700, No. 21DZ1 100100), 111 plan (No. BP0719010) and National Natural Science Foundation China (No. 62306178).
References
[1] Jiwaon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4981-4990 2018.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 11--3736,,,,,,, 2022.
[3] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning visual features by contrasting cluster assignments. Advances in neural information processing systems, 1119912299244 2020.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 1650-9660, 2021.
[6] Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning generate text-grounded mask for open- world semantic segmentation from only image- text pairs. arXiv preprint arXiv:2212.00785, 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Sorieut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-356 3558-3568, 2021.
[8] Ting Chen, Simon Korbliihh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-16007. PMLR, 2020.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:17864-17875. 2021.
[10] Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, Diane Larlus. Prohabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8415-8424, 2021.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[12] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Weh-curated image-text data created by the people, for the people. arXiv preprint arXiv:2111.11433 2021.
[13] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprini 1rXivv1810.04805. 2018
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 2020.
[15] Richard Duda. Peter Hart, al. Puttern classification and scene analyxis, volume 3. Wiley New York. 1973.
[16] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111:98-111 136, 2015.

---

[17] Shanghua Gao, Zhong- Yu Li, Ming-Hsuan Yang. Ming-Ming Cheng, Junwei Han, and Philip Torr Large-scale unsupervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[18] Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun, On the duality between contrastive and non-contrastive self-supervised learning. arXiv preprint arXiv:220600577, 2022.
[19] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open- open-vocabulary image segmentation with image-level labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 540-557. Springer, 2022.
[20] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356-5364, 2019.
[21] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 9729-9736 38, 2020.
[22] Dat Huynh, Jason Kuen, Zhe Lin, Jiuxiang Gu, and Ehsan Elhamifar. Open-vocabulary instance segmenta- tion via robust cross-modal pscudo-labeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7020-7031, 2022.
[23] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 6rX1::/61..0114, 2016.
[24] Peng Jin, Jinfa Huang. Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David Clifton, and Jie Chen. popectation-maximizatn
contrastive learning for compact video-and- representations. Advances in Neural Information Processing Systems, 35:30291 -3000,,, 2022.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
[26] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan Lo, et al. Segment anything. arXiv preprint arXiv:2304.0266 026443000..3.... 2023.
[27] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly. 2(1-2):83-97, 1955.
[28] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.03566. 2022.
[29] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- pre-training for unified iisionllanuss understanding and generation. In International Conference Machine Learning, pages 12888- 12900. PMLR, 2022.
[30] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations.
[31] Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu. Expectation-maximizze attention networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9167-9176. 2019.
[32] Tsung- Yi Lin, Michael Maire, Serge Belongic, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[33] Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Open- semantic segmentation via contrasting and clustering lanion-language embedding. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XX. pages 275-292. Springer, 2022.
[34] Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7086-7096. 2022.
[35] Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segelip: Patch aggregation with learnable centers for voca-vocy semantic segmentation. arXiv preprint 2rXiv:2211.148/1 2022.
12

---

[36] Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Jinxiang Liu, Yu Wang. Ya Zhang, and Yanfeng Wang. Diffusionseg: Adapting diffusion towards unsupervised object discovery. arXiv preprint arXiv:2303.08813 2023.
[37] Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Ya 7hang, and Yanfeng Wang. Open-vocabulary semantic segmentation via attribute decompsstionn-ggreaaiinn arXiv preprint arXi::2309.00066, 2023.
[38] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Wii Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 891-898, 2014.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-878 8763, PMLR, 2021.
[40] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34: 1e::1216611110 111112228, 2021.
[41] Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang. Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency. arXiv preprint ar0iv 10307, 2023.
[42] Sylvia Richardson and Peter Green. On bayesian analysis of mixtures with an unknown number of components (with discussion). Journal of the Royal Statistical Society: series B (statistical methodology). 59(4):731-792, 1997.
[43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[44] Lixiang Ru, Yibing Zhan, Baosheng Yu. and Bo Du. Learning afsinity from attention: end-to-end weakly- supervised semantic segmentation with transformers. In InPPro Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16846-16855, 2022.
[45] Gyungin Shin, Weidi Xie, and Samuel Alhanie. Reco: Retrieve and co-segment for zero-shot transfer. arXiv preprint arXiv:2206. 000455 2022.
[46] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications the ACM, 59(2):64 73, 2016.
[47] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolics, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. International conference on machine learning. pages 00347-10351 PMLR, 2021.
[48] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939. PMLR, 2020.
[49] Xiang Wang, Shaodi You, Li, and Huimin Ma. soally-supervieed semantic segmentation by iteratively mining common ohject features. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1354-1362, 2018.
[50] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18 18144, 2022.
[51] Jilan Xu, Junlin Hou, Yuejic Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Leaminng open- vocabulary semantic segmentation models from natural language supervision. arXiv preprint ArXiv:2301. 00111, 2023.
[52] Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, and Dan Xu. Multi-class token transformer for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 4310-4319, 2022.
[53] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self Sulf-super learning via redundancy reduction. In International Conference on Machine Learning pages 123200.. PMLR, 2021.
13

---

[54] Fei Zhang, Chaochen Gu, Chenyue Zhang, and Yuchao Dai. Complementary patch for weakly supervised semantic segmentation. In Proceedings the IEEE/CVF international conference computer vision, pages 7242-7251, 2021.
[55] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII. pages 696-712. Springer, 2022.
[56] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, prototype view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2582 2593, 2022.
Van Gool. Rethinking semantic segmentation:
[57] Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang. Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pages 27367-27377 PMLR, 2022.
[58] Benjin Zhu, Zhe Wang. Shaoshuai Shi, Hang Xu, Lanqing Hong. and Hongsheng Li. Conquer: Query contrast voxel-detr for 3d object detection. arXiv preprint W2XV:2212..07289, 2022.
14


Reviews:
[{"review_id": "jGnThRFgSj", "reviewer": "Reviewer_ae73", "summary": "The paper addresses the problem of weakly open-vocabulary semantic segmentation, which is to segment images of novel classes without any segmentation annotation.\nTo segment unseen classes, a model is trained to align between image features and text caption to learn to associate between image patches and class names.\nThe proposed work argues that there is an discrepancy between visual-textual alignment during training and pixel-wise grouping during inference.\nThus, the paper proposes a non-learnable prototypical regularization (NPR), which leverages Gaussian Mixture Model to provide pixel-wise grouping supervision to better align the model for segmentation task.\nExperiments are conducted on PASCAL, COCO, ImageNet and LVIS datasets.", "questions": "Please refer to the weakness section", "limitations": "Sufficiently addressed", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The idea of direction of weakly open-vocabulary semantic segmentation is impactful since it can significantly reduce the annotation effort especially for the costly segmentation tasks.\n+ The proposed direction of using Gaussian Mixture Model to group image patches into groups for additional training supervision is sensible and interesting especially this doesn't require any supervision.\n+ The paper is self-contained and provides enough details for result reproduction.", "weaknesses": "+ The reviewer finds that the emphasis on the non-learnable aspect of the algorithm is misleading. Specifically, the proposed method does use expectation-maximization to compute the prototype which is a form of learning algorithm, which contradicts with the claim about non-learnable prototype regularization. Moreover, the reviewer is not convinced on why \"non-learnable\" is a desirable feature for segmentation given given that learnable/optimizable components would make the model fully differentiable and can be easily trained with gradient based optimization. Can the paper clarify this?\n+ The experimental results appears to be inconclusive. Although the paper shows that the proposed method can achieves strong improvement with limited number of training data (using only CC12M table 3), but when scaling to large dataset, the improvement diminishes to less than 1%. Can the paper provide justification on this?\n+ It is quite surprising that the model seems to be insensitive to the number of groups used during training (as shown in Table 5) where the difference is less than 1%? The number of groups should strongly influence performances as too many groups could introduce training noise and too few would not capture the full information in image. Can the paper justify this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1\n>The reviewer finds that the emphasis on the non-learnable aspect of the algorithm is misleading. Specifically, the proposed method does use expectation-maximization to compute the prototype which is a form of learning algorithm, which contradicts with the claim about non-learnable prototype regularization. Moreover, the reviewer is not convinced on why \"non-learnable\" is a desirable feature for segmentation given given that learnable/optimizable components would make the model fully differentiable and can be easily trained with gradient based optimization. Can the paper clarify this?\n\n**(1)** Sorry for the confusion. Here we call prototypes ``non-learnable`` because they are **detached from the gradient updating** of the network and online updated in a momentum manner (please refer to L235-239, page 6). The concept and definition are recently used in [ref1] and we adhere to it. We will include this in the updated version for clarity.\n\n**(2)** For Reviewer's proposition on using ``learnable prototypes`` for regularization, while the method appears logical on the surface, it would **not perform as well as anticipated.** Note that one major motivation of our method is to regularize group tokens using prototypical knowledge of image/text. However, once learning prototypes and group tokens together, the **model can be easily degraded** (e.g., simply making prototypes and group tokens identical and loss will be zero), and consequently **no proper regularization will be imposed.** To verify this, we further design a new baseline in which learnable prototypes are introduced by **replacing the I-NPR and T-NPR with another two SGMs.** In this way, the groups in the newly-added SGM serve as the prototypes, which are then used to regularize the original group tokens. Results are shown in the following table, and demonstrate that the baseline, with this learnable prototypical regularization, **detracts significantly from the segmentation performance.**\n|Baseline (GroupViT)|  PGSeg w/ learnable prototypes | PGSeg |\n|:----------:|:--------------:|:--------------:\n| 41.1 (CC12M) | 35.5 (\t**$\\downarrow$ 5.6**)|**49.0** (**$\\uparrow$ 7.9**)\n| 50.8 (CC12M+RedCaps12M) | 46.3 (**$\\downarrow$ 6.5**) | **53.2** (**$\\uparrow$ 2.4**)\n\n***\n\n\n\n\n\n\n\n### Q2\n>The experimental results appears to be inconclusive. Although the paper shows that the proposed method can achieves strong improvement with limited number of training data (using only CC12M table 3), but when scaling to large dataset, the improvement diminishes to less than 1%. Can the paper provide justification on this?\n\nThanks for the comment. First of all, we'd like to highlight that **PGSeg in general scales well, i.e., on VOC12 and COCO, under the same data volume.** Here we show the performance of PGSeg with YFCC14M for a clear illustration.\n\n|Method | VOC12   |   Context     |  COCO  |\n|:----------:|:------:|:------:|:------:|\n|GroupViT (YFCC14M + CC12M)| 52.3 | 22.4 | 20.9 |\n|ViewCO (YFCC14M + CC12M)| 52.4 | 23.0 | 23.5|\n|PGSeg (YFCC14M + CC12M)| **53.9** | **23.5** | **24.7** |\n\nTherefore, the reduced improvement gap (less than 1%) pertains only to the Context dataset. Compared to VOC and COCO, where a majority of their foreground classes represent distinct entities (like dog, cat, etc.), Context additionally includes some foreground classes, such as \"building\" and \"wall\", which can **easily be misconstrued as the \"background\".** (Please refer to the **Supplementary Material** for some samples and their annotations of Context). Such distinct essence requires superior recognition among these (near-)\"background\" classes to achieve better performance on Context. Thus, this semantic uniqueness of Context may **trouble the prototypes to perform expected segmentation on these \"background\" classes**. To demonstrate that, we present the performance of PGSeg on some \"background\"-like classes in Context. As shown in the following table,  PGSeg could not distinctively segment these \"background\" classes, even with scaling up to RedCaps12M. This leaves room for potential improvement of our approach. We will add these analyses!\n|\"background\"-like classes in Context|  Wall | Building |ceiling |ground |Average\n|:----:|:-------:|:--------:|:----------:|:----------:|:----------:\n| PGSeg (CC12M, 20.6)| 0.4 | 5.1 | 4.4| 7.3|*4.3*\n| PGSeg (Redcaps12M+CC12M, 23.8)| 0.8 |4.7 |8.8|5.7|*5*\n\n***\n\n\n\n\n### Q3\n>It is quite surprising that the model seems to be insensitive to the number of groups used during training (as shown in Table 5) where the difference is less than 1%? The number of groups should strongly influence performances as too many groups could introduce training noise and too few would not capture the full information in image. Can the paper justify this?\n\n**(1)** Sorry for the misleading. We clarify that Table 5 is intended to validate the impact of **prototype number rather than group number**. As mentioned in the caption, **the numbers of group tokens remain fixed at 64 (8) for the 1st (2nd) PG Unit**. The conclusion from Table 5 is just that PGSeg is insensitive to **prototype number**. We will revise Table 5 to avoid any potential misunderstandings. \n\n**(2)** We totally agree with the Reviewer's insight that the number of group tokens is important to segmentation performance. As such, we have conducted an ablation study on the number of group tokens in PGSeg using CC12M. Note that we maintain the number of prototypes equal to the number of group tokens. It is evident in the following table that **too many or too few group tokens have detrimental effects on the segmentation performance.** We will add this analysis.\n\n|Group Token Number (at 1st----2nd PG Unit)|  Performance on VOC12 (%)|\n|:----------:|:--------------:|\n| 16----4 | 29.3 |\n| 16----8 | 39.5 |\n| 64----8 | **49.0** |\n| 128----8 | 43.3 |\n| 128----4 | 37.3 |\n\n\n***\n\n### Reference\n\n\n>[ref1] Rethinking Semantic Segmentation: A Prototype View, CVPR2022.\n"}, {"review_id": "GJkIIEmxcW", "reviewer": "Reviewer_nyAz", "summary": "This paper studies the problem of weakly open-vocabulary semantic segmentation, which aims to segment objects of arbitrary classes using only image-text pairs. The paper proposes a prototypical guidance segmentation network, which uses non-learnable prototypes estimated from image and text features to regularize the group tokens that cluster the image patches. The paper shows that the proposed method improves the segmentation results with increased compactness and richness of the group tokens, and achieves state-of-the-art performance on several benchmark datasets.", "questions": "1. The comparison is not fair enough in Table 3 since the training data are different. Adding PGSeg(CC12M+YFCC14M) is fair to compare the proposed method and ViewCO and GroupViT.\n\n2. Why the proposed method performs much better on  VOC and COCO but much lower on Context? What is the uniqueness of this dataset?\n\n3. Some previous methods that tackles the semantic segmentation task using GMM should be discussed. e.g., \"GMMSeg: Gaussian Mixture Models for Deep Generative Semantic Segmentation\"", "limitations": "The authors did not addressed the limitations. I suggest to add more experiments and ablation studies.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel concept of non-learnable prototypical regularization. The idea of the paper is reasonable and novel.\n\n\n2. The motivation and the methods are illustrated clearly in Introduction.\n\n\n3. The performances improvement on VOC and COCO are significant.\n", "weaknesses": "1. Ablation studies are not enough. The number of prototypes should be more for ablation study. The affect of the  position and number of PG-Unit should be explored in ablation studies.\n\n2. In Table 3, PGSeg shows the superiority on two datasets  VOC and COCO, but on Context, the performance of PGSeg is lower than pervious methods.\n\n\n3. Fig.3 is not clear enough to illustrate the proposed method.  The text in the picture is too small to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1\n>Ablation studies are not enough. The number of prototypes should be more for ablation study. The affect of the position and number of PG-Unit should be explored in ablation studies.\n\nThanks for this suggestion! Here we first present the improved Table 5 by expanding the range of the number of prototypes. The settings in the following table are identical to Table 5 (please refer to L318-L321, page 8 for more details). Note that here the **group token number remains fixed** (64/8 for the 1st/2nd PG Unit). The result shows that **the optimal prototype number shall be set equally to the group token number.**\n|Prototype Number (1st\\2nd Level)| 4 | **8** |16|32|64|\n|:----------:|:--------------:|:----------:|:--------------:|:----------:|:----------:|\n| 16 | 51.78 | 52.1 | 51.42 | - | -\n| 32 | 52.12 | 52.49 | 51.94 | 51.33 | -\n| **64** | 52.71 | **52.83** | 52.23 | 51.87 | 51.56|\n| 128 | 52.21 | 52.62 | 52.14 | 51.64 | 51.33|\n| 256 | 52.04 | 52.33 | 51.97 | 51.37 | 51.14|\n\nSecondly, we report the effect of the position of PG Unit on PGSeg with CC12M. Based on the following results, it is found that the **optimal position for the 1st (2nd) PG Unit in ViT-S is the 6th (9th) layer**, which is also consistent with the finding from GroupViT and ViewCO.\n|PG Unit Position (The layer position of the 1st/2nd PG Unit at ViT-S)|  Performance on VOC12 (%)|\n|:----------:|:--------------:|\n| (5/8) | 47.3 |\n| (6/9) | **49.0** |\n| (7/10) | 47.9 |\n| (4/11) | 48.3 |\n\nFinally, we present the ablation studies on the number of PG Unit in ViT as follows (Here the number of group tokens in the 11th layer is 4). As shown in the following table, the presence of **more or fewer PG Units would lead to some performance loss**, indicating the importance of the PG Unit amount in PGSeg.\n|PG Unit Number--(PG Unit position)| Performance on VOC12 (%)|\n|:----------:|:--------------:|\n| 1--(6) | \t45.2 |\n| 2--(6, 9) | **49.0** |\n| 3--(6, 9, 11) | 47.2 |\n\n\nWe will add all these analyses in the updated version.\n\n***\n\n### Q2\n>In Table 3, PGSeg shows the superiority on two datasets VOC and COCO, but on Context, the performance of PGSeg is lower than pervious methods. \n\nThanks for the comment, but there seems to be a misunderstanding. While it is true that PGSeg exhibits lower performance against MaskCLIP and SegCLIP on Context, it is crucial to consider the **substantial difference in training pipelines:** MaskCLIP and SegCLIP are both built on a powerful foundation model, *i.e.*, **CLIP**, which contains rich knowledge pattern from **~400M** image-text pairs. In contrast, our proposed PGSeg is **trained from scratch with simply 12M/24M** image-text pairs. Despite the significant data volume gap, PGSeg achieves **comparable performance on Context** and even **outperforms them on VOC12/COCO.** This actually underscores our strength of **data-efficient learning** rather than the weakness of our approach. Notably, when evaluating the methods under the same level of training data volume, PGSeg is the best across all three datasets. \n\n***\n\n### Q3\n>Why the proposed method performs much better on VOC and COCO but much lower on Context? What is the uniqueness of this dataset?\n\nCompared to VOC and COCO, where a majority of their foreground classes represent distinct entities (like dog, cat, etc.), Context additionally includes some foreground classes, such as \"building\" and \"wall\", which can easily be misconstrued as the \"background\" (Please refer to the **Supplementary Material** for some samples and their annotations of Context). Such distinct essence requires **superior recognition among these (near-)\"background\" classes** to achieve better performance on Context. The following table validates that PGSeg is unable to well segment these \"background\"-like classes, correspondingly leaving the room for our proposed PGSeg to segment on complex scenes. \n|\"background\"-like classes in Context |  Wall | Building |ceiling |ground |Average\n|:----:|:-------:|:--------:|:----------:|:----------:|:----------:\n| PGSeg (CC12M, 20.8)| 0.4 |5.1 |4.4|7.3|*4.3*\n\n***\n\n### Q4\n>The comparison is not fair enough in Table 3 since the training data are different. Adding PGSeg(CC12M+YFCC14M) is fair to compare the proposed method and ViewCO and GroupViT.\n\nThanks for the comment. The requested results are reported in the following table. It is evident that PGSeg consistently surpasses GroupViT and ViewCO under the (CC12M+YFCC14M) setting. *Based on the results, we can more confidently conclude the superior performance of PGSeg.*\n\n|Method | VOC12   |   Context     |  COCO  |\n|:----------:|:------:|:------:|:------:|\n|GroupViT (YFCC14M + CC12M)| 52.3 | 22.4 | 20.9 |\n|ViewCO (YFCC14M + CC12M)| 52.4 | 23.0 | 23.5|\n|PGSeg (YFCC14M + CC12M)| **53.9** | **23.5** | **24.7** |\n\n\n***\n\n### Q4 & Q5\n>Fig.3 is not clear enough to illustrate the proposed method. The text in the picture is too small to read. \n\n>Some previous methods that tackles the semantic segmentation task using GMM should be discussed. e.g., \"GMMSeg: Gaussian Mixture Models for Deep Generative Semantic Segmentation\".\n\n\nThanks for pointing these out. We will improve the figure and add the missing citations in the updated version.\n\n***\n### Reference\n\n>[ref1] Unsupervised domain adaptation for semantic segmentation via class-balanced self-training, CVPR2018.\n\n>[ref2] A Closer Look at Self-training for Zero-Label Semantic Segmentation, CVPR2021.\n\n>[ref3] Learning from future: A novel self-training framework for semantic segmentation, Neurips2022."}, {"review_id": "CKoo6JU7iw", "reviewer": "Reviewer_YVCC", "summary": "\nThis paper presents a method for addressing the challenge of granularity inconsistency in weakly open-vocabulary semantic segmentation (WOVSS). To tackle this issue, the authors propose a technique called non-learnable prototypical regularization (NPR). NPR leverages explicit supervision from prototypical knowledge to guide group tokens, leading to improved object segmentation characterized by enhanced compactness and richness. Extensive experiments have been conducted to validate the effectiveness of this method on several benchmark datasets.\n", "questions": "Please see the weakness above.", "limitations": "No, the authors did not discuss the limitations and potential negative social impact in the main text.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors' use of prototypical knowledge to tackle the performance bottleneck caused by granularity inconsistency is a well-motivated and intriguing idea. \n\n- The effectiveness of the proposed method is demonstrated through comprehensive experiments on multiple benchmark datasets. The results surpass those of previous works, establishing a new state-of-the-art performance. ", "weaknesses": "- The term \"granularity inconsistency\" is initially unclear in the paper, specifically in lines 37-40 and the caption of Fig.2. Although Sec.3 provides clarification, it would be preferable to have a clear explanation in the abstract or introduction, as it serves as the main motivation for the proposed method. Alternatively, improving the caption of Figure 2 would help better understanding.\n\n- In Sec.5, there is a lack of explanations for the experimental results. For instance, in the \"Comparison with Zero-shot Baselines\" section, there are insufficient explanations regarding why specific methods (e.g., ViT-8S) were chosen as baselines and the implications of the results beyond superior performance. To provide a more comprehensive analysis, the experiments section should include additional explanations elucidating the intended message conveyed by the results.\n\n- Conducting ablation studies to evaluate the benefits of each NPR strategy (T-NPR only or I-NPR only) in terms of compactness and richness would be valuable. This would offer a clearer understanding of the effectiveness of each module by confirming the respective impacts of the NPR strategies on compactness and richness.\n\n- (minor) In Fig.~4 (bottom), the airplane and contrail exhibit similar colors (purple and red), making it difficult to ascertain whether PGSeg is superior to GroupViT. To demonstrate the superiority of the proposed method, it would be helpful to differentiate the airplane and contrail more distinctly by modifying their colors or exploring alternative visual representation approaches.\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1\n>The term \"granularity inconsistency\" is initially unclear in the paper, specifically in lines 37-40 and the caption of Fig.2. Although Sec.3 provides clarification, it would be preferable to have a clear explanation in the abstract or introduction, as it serves as the main motivation for the proposed method. Alternatively, improving the caption of Figure 2 would help better understanding.\n\nThank you for this helpful suggestion! We will improve the caption of Fig. 2 as *\"Illustration of granularity inconsistency in SGM. During the training phase, group tokens follow an all-to-one manner to be averaged to align with text embedding. However, during the inference stage, each group token is directly used to match the class embedding, i.e., the one-to-one alignment.\"* Meanwhile, the Introduction part will be improved as well to further clarify this aspect. \n\n***\n\n### Q2\n>In Sec.5, there is a lack of explanations for the experimental results. For instance, in the \"Comparison with Zero-shot Baselines\" section, there are insufficient explanations regarding why specific methods (e.g., ViT-8S) were chosen as baselines and the implications of the results beyond superior performance. To provide a more comprehensive analysis, the experiments section should include additional explanations elucidating the intended message conveyed by the results.\n\nSorry for the confusion. Here we provide a detailed analysis of Table 1, which shall be incorporated in the updated version.\n\n1. **Motivations for baseline design**. First, for **ViT-S based baselines** (rows 1-4 in Table 1), they are widely adopted in recent works, *e.g.*, [ref2, ref3], based on a simple idea: train a ViT and a text encoder jointly with an image-text contrastive loss (as CLIP [ref1]), and subsequently perform non-parametric grouping of final-layer pixel features using, *e.g.*, k-means, to obtain segmentation. \n\n    Second, for **ViT-8S** (row 5), it is inspired by findings in [ref4]: increased [CLS] tokens in ViT serve as meaningful centers for weakly supervised segmentation. We then increase the [CLS] token number of ViT-S to 8 to create a more potent parametric baseline ViT-8S (than ViT-S) to examine our approach. \n\n    \n\n2. **More result analysis.** Despite effectiveness, ViT-8S is limited in lacking explicit grouping recognition/regularization. SGM-based solutions relieve this issue and show consistently better performance against ViT-8S, which confirms the importance of integrating a grouping component into ViT for WOVSS. By further addressing granularity inconsistency in SGM, our PGSeg leads to substantial performance improvements against existing methods, *i.e.*, **5.8\\%** and **3.3\\%** over GroupViT and ViewCo.\n\n\n***\n\n\n### Q3\n>Conducting ablation studies to evaluate the benefits of each NPR strategy (T-NPR only or I-NPR only) in terms of compactness and richness would be valuable. This would offer a clearer understanding of the effectiveness of each module by confirming the respective impacts of the NPR strategies on compactness and richness.\n\nThanks for the suggestion! We have added additional analysis to examine the influence of I-NPR and T-NPR on compactness and richness. Please refer to the enclosed **rebuttal_figure.pdf** for visualized plots. \n\nWe find that both components contribute to compactness and richness, but:\n1. As shown in **Fig. I**, **I-NPR exerts a more pronounced impact on amplifying compactness.** This observation aligns with the intuitive understanding that the image information embedded within I-NPR is inherently structured, thereby leading to more cohesive clustering. \n\n2. As shown in **Fig. II**, **T-NPR has a more significant effect on improving richness**, since it leads to a larger dimensional variance compared to I-NPR. We conjecture that the prototypes, which are sourced from text embeddings, could impose stronger semantic regularization on the group tokens.\n\nThe results also underscore the complementary roles of I-NPR and T-NPR, further substantiating their importance in PGSeg. Relevant discussions will be added to the paper.\n\n***\n\n### Q4\n>(minor) In Fig.~4 (bottom), the airplane and contrail exhibit similar colors (purple and red), making it difficult to ascertain whether PGSeg is superior to GroupViT. To demonstrate the superiority of the proposed method, it would be helpful to differentiate the airplane and contrail more distinctly by modifying their colors or exploring alternative visual representation approaches.\n\nThanks! We agree and will improve it in the updated version.  \n\n***\n\n### Reference\n>[ref1] Learning Transferable Visual Models From Natural Language Supervision, ICML2021.\n\n>[ref2] Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency, CVPR2022.\n\n>[ref3] GroupViT: Semantic Segmentation Emerges from Text Supervision, CVPR2022.\n\n>[ref4] Multi-class Token Transformer for Weakly Supervised Semantic Segmentation, CVPR2022."}, {"review_id": "mHltO9Gv0Y", "reviewer": "Reviewer_hVxT", "summary": "This paper proposes that the traditional WOVSS approach lacks effective supervision of group tokens. In this paper, visual patch tokens are modeled through the GMM model to obtain prototype features that can be used to supervise each group token. Experimental results show that the proposed PGSeg achieves state-of-the-art performance on several benchmark datasets.", "questions": "See weaknesses.", "limitations": "No limitations are discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Appling explicit supervision for each group token by constructing prototypes is novel.\n2. The proposed PGSeg can reach good performance without large vision-language foundation models.\n", "weaknesses": "Major weakness:\n1. As shown in Table 3, The performance of the baseline (GroupViT) drops a lot when using CC12M+RedCaps compared to CC12M+YFCC14M(Setting in the original GroupViT). To demonstrate the effectiveness of PGSeg, the performance of PGSeg trained with CC12M+YFCC14M should be reported.\n2. Prototype P is an important parameter of this article. If prototype P is not updated during training, how much impact will it have on PGSeg performance?\n3. The proposed PGSeg heavily relies on the EM algorithm to estimate the parameters of GMM, which may makes the model hard to train. What is the training time cost of PGSeg compared to the baseline(GroupViT).\n\nMinor weakness:\n1. Text in Figure 2 and Figure 3 are too small, which makes the workflow of PGSeg hard to read.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 \n>As shown in Table 3, The performance of the baseline (GroupViT) drops a lot when using CC12M+RedCaps compared to CC12M+YFCC14M(Setting in the original GroupViT). To demonstrate the effectiveness of PGSeg, the performance of PGSeg trained with CC12M+YFCC14M should be reported.\n\nThanks for this comment! The requested results are reported in the following table:\n|Method | VOC12   |   Context     |  COCO  |\n|:----------:|:------:|:------:|:------:|\n|GroupViT (YFCC14M + CC12M)| 52.3 | 22.4 | 20.9 |\n|ViewCO (YFCC14M + CC12M)| 52.4 | 23.0 | 23.5|\n|PGSeg (YFCC14M + CC12M)| **53.9** | **23.5** | **24.7** |\n\n**(1)** As can be seen,  with the training data of (YFCC14M + CC12M), PGSeg consistently outperforms GroupViT and ViewCO across all three datasets. *It enables us to make the conclusion about the superiority of PGSeg more rigorously.*\n\n**(2)** Regarding the performance drop of GroupViT on COCO (27.5% -> 20.9%) when replacing YFCC14M with RedCaps, we observe a similar performance drop of PGSeg (28.7% -> 24.7%). We posit this might be attributed to the semantic descriptors of entities in RedCaps12M having a **more aligned resonance** with COCO labels compared to those in YFCC14M (please refer to the examples presented of **Fig. III** in the uploaded **rebuttal_figure.pdf**).\n\n\n***\n\n### Q2 \n>Prototype P is an important parameter of this article. If prototype P is not updated during training, how much impact will it have on PGSeg performance?\n\nThanks for highlighting this point. To address your concern, we conduct the experiments to verify PGSeg w/ or w/o updating prototypes and summarize the performance in the following table. According to the results, w/o updating, the model suffers from **a decrease of approximately 1.5% mIoU**, which confirms the pivotal role of the prototype updating mechanism. \n|Method | w/ update   |   w/o update |\n|:------------------:|:------:|:------:|\n|PGSeg (CC12M)| 53.2 | 51.8 (\t$\\downarrow$  **1.4**) |\n|PGSeg (RedCaps12M + CC12M)| 49.0 | 47.5 (\t$\\downarrow$ **1.5**)  |\n\n***\n\n### Q3\n>The proposed PGSeg heavily relies on the EM algorithm to estimate the parameters of GMM, which may makes the model hard to train. What is the training time cost of PGSeg compared to the baseline(GroupViT).\n\nThanks for the comment. As discussed in **Section 4.1 (Complexity Analysis) and Table 6**, our EM algorithm is **not expensive.** With RedCaps12M+CC12M, the training time cost of GroupViT is **1.9 hours per epoch** and that of PGSeg is **2.5 hours per epoch**, which is at a reasonable level. \n\n***\n\n### Q4 \n> (Minor weakness) Text in Figure 2 and Figure 3 are too small, which makes the workflow of PGSeg hard to read.\n\nThanks for pointing this out. We will improve it."}]

Author rebuttal:
## Summary



We thank reviewers for their valuable feedback, and appreciate the great efforts made by all reviewers, ACs, SACs and PCs.

We are invigorated by the **positive evaluation** from all reviewers. Specifically, they find the method **novel and reasonable** (nyAz, ae73), **well-motivated and intriguing** (YVCC, hVxT), the experimental results being **state-of-the-art** (YVCC) and showing  **significant improvements** (nYAz), the writing **self-contained** (ae73) and **effectively illustrated** (nYAz).


In response to the comments and suggestions, we have provided a detailed respective rebuttal for each reviewer, and here we summarize major points for convenience. 

- We have extended ablation studies to investigate the effect of **group token number**(ae73),  **prototype number and PG-Unit position/number** (nyAz), **PGSeg with YFCC14M** (nyAz, hVxT), **prototype updating** (hVxT), as well as **the respective effect of T-NPR and I-NPR** (YVCC).
- We have conducted a more in-depth and comprehensive analysis of experimental design and results, including **indistinctive performance on Context** (ae73, nyAz), **training costs** (hVxT), and **baseline design** (YVCC). 


All these will be merged into the article.  

>Below is the uploaded **"rebuttal_figure.pdf"**

Meta-review: All reviews are positive (1 weak accept, 3 borderline accept). The most confident reviewer is most positive. Reviewers praise the problem setting, key idea in the method, results, and presentation. The reviewers request clarifications and more comparisons, and two reviewers respond to explicitly state they maintain their positive rating after reading the rebuttal. For the two that did not respond, their concerns seem well addressed.
Decision: Accept (poster)