Push and Pull:  Competing Feature-Prototype Interactions  Improve Semi-supervised Semantic Segmentation
Authors: Yuhang Ding
Yifan Sun
Yi Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Semi-supervised
Segmentation
Competing Interactions
Classifier Prototype

Abstract: This paper challenges semi-supervised segmentation with a rethink on the feature-prototype interaction in the classification head. Specifically, we view each weight vector in the classification head as the prototype of a semantic category. The basic practice in the softmax classifier is to pull a feature towards its positive prototype (i.e., the prototype of its class), as well as to push it away from its negative prototypes. In this paper, we focus on the interaction between the feature and its negative prototypes, which is always “pushing” to make them dissimilar. While the pushing-away interaction is necessary, this paper reveals a new mechanism that the contrary interaction of pulling close negative prototypes is also beneficial. We have two insights for this counter-intuitive interaction: 1) some pseudo negative prototypes might actually be positive so that the pulling interaction can help resisting the pseudo-label noises, and 2) some true negative prototypes might contain contextual information that is beneficial. Therefore, we integrate these two competing interactions into a Push-and-Pull Learning (PPL) method. On the one hand, PPL introduces the novel pulling-close interaction between features and negative prototypes with a feature-to-prototype attention. On the other hand, PPL reinforces the original pushing-away interaction with a multi-prototype contrastive learning. While PPL is very simple, experiments show that it substantially improves semi-supervised segmentation and sets a new state of the art.

Under review as a conference paper at ICLR 2023
PUSH AND PULL: COMPETING FEATURE-PROTOTYPE INTERACTIONS IMPROVE SEMI-SUPERVISED SEMAN- TIC SEGMENTATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
This paper challenges sumissupervieed segmentation with a rethink on the feature- prototype interaction in the classification head. Specifically, we view each weight vector in the classification head as the prototype of a semantic category. The basic practice the softmax classifier to pull a feature towards its positive proto- well as to push it away from its negative
type (i.e., the prototype of its class), prototypes. In this paper, we focus on the interaction between the feature and its negative prototypes, which is always "pushing" to make them dissimilar. While the pushing-away interaction is necessary, this paper reveals a new mechanism that the contrary interaction of pulling close negative prototypes is also benefi- cial. We have two insights for this counter-intuitive interaction: 1) some pseudo negative prototypes might actually be positive that the pulling interaction can help resisting the pseudo-label noises, and 2) some true negative prototypes might contain contextual information that beneficial. Therefore, we integrate these two competing interactions into Push-and- Learning (PPL) method. On the one hand, PPI. introduces the novel pulling-close interaction between features and negative prototypes with a feature-to-prototy attention. On the other hand, PPL reinforces the original pushing-aw interaction with multi-prototype con- trastive learning. While PPL is very simple, experiments show that it substantially improves -uperviiee segmentation and sets a new state of the art.
INTRODUCTION
This paper considers the semi-supervised semantic segmentation task. We focus on an essential component in the segmentation model, i.e. the classification head, which consists of set of learn- able weight vectors. These weight vectors are usually viewed as a set of prototypes representing the corresponding semantic categories. The essential training process is to pull each deep feature towards its positive prototype (i.e., the prototype of its class), as well as to push it away from its negative prototypes. The "pushing-away" interaction between feature and its negative prototypes makes them dissimilar to each other and is critical for discriminating different classes. In all the fol- lowing parts of this paper, our discussion focuses on the interaction between features and their negative prototypes and neglects the positive prototypes (unless explicitly pointed out).
While this "pushing away" interaction is necessary, this paper reveals new mechanism that the contrary interaction of pulling close features and their negative prototypes is also beneficial. This "pulling close" interaction may seem counter-intuitive at the first glance but actually reasonable from two insights as below:
1) It brings a task-specific benefit for semi-supervised segmentation by resisting the pseudo-label noises. Specifically, the popular pscudo-label-basee pipeline is inevitably confronted with the pseudo-label noises: some pseudo negative prototypes might be actually positive. Under this con- dition, the "pulling close" interaction gives the feature chance to approach its actual-positive pro- totype. Experiments confirm that the pulling interaction effectively reduces the pseudo label noises (Section 4.4) and this task task-specific benefit is the primary reason for our improvement (Section 4.4).
2) It brings a general benefit for both the semi-supervised and supey-sue segmentation because some negative prototypes contain contextual information. Specifically, since our prototypes (i.e.,

---

Under review as a conference paper at ICLR 2023
pull
close
feature
push
proxy
feature space
away
feature space
feature space
(a) Standard pushing interaction
(b) Proposed push-and-pull learning
Figure Comparison between the standard Peature-prototyn interaction and our push-and learning. (a) In the standard classification head, the interaction with the negative prototypes is always "pushing-away". (b) This paper reveals that in addition the aushing-away interaction, competing pulling-close close interaction between features and negative prototypes is also beneficial. Therefore, the proposed push-and-pull learning (PPL) combines the pushing-away and pulling-close interactions for negative prototypes. Moreover, we use multiple prototypes to represent each individual class to better reflect the intra-class diversity, which is also beneficial. Circles and squares represent prototypes and features, respectively. Different classes are in different colors.
the weight vectors) are learned from the whole training set through pack-propagation, they naturally provide clues for deriving the contextual information. Some prior works (Yuan al., 2020; Jin al., 2021: Zhou et al., 2022) already show that exploring the contextual information from mean features improves the segmentation accuracy. In contrast, we find that directly using the weight vectors for contextual information can bring the similar benefit and is relatively simple. Empirically, we show this general benefit is secondary reason for our improvement (as detailed in Section 4.4).
These two insights motivate us to propose Push-and-Puil Learning (PPL) for semi-supervised seg- mentation, as shown Fig. 1. On the one hand, PPI. reinforces the basic "pushing negative pro- totypes" interaction with multi-prototyyn contrastive learning, Le., using multiple (instead of one) prototypes to better represent each category. the other hand, PPI. enforces the "pulling negative prototypes" interaction through a feature-to-protove information from all the prototypes using a standard cross-attention layer. Although these two in- teractions share same effect of pulling close positive prototypes, we name the resulting method as Push-and- -ull Learning to highlight their competing effect on the negative prototypes.
attention. Specifically, each feature absorbs
Another important advantage of the proposed PPL is that it can be cascaded to enlarge its benefit. It because, after the pulling-close interaction, the features are refined with the improved resistance against pseudo-label noises. Therefore, re-evaluate their pseudo labels increase label accuracy. Based on the refined deep features, we may append another round of PPL for further improvement. Extensive experiments
popular benchmarks validate that cascading multiple PPL accumulates multiple rounds of improvement (Appendix B.2) and sets a new state of the art (e.g., 77.03% and 74.22% mloU on PASCAL VOC 2012 using 1/8 and 1/16 labels). Moreover, ablation study shows that within the two benefits of PPL, the task- mask-specific benefit is the major reason for the superiority of PPL: while the general benefit improves the fully-supervised segmentation (the upper-bound of semi-supervised segmentation) by small margin, the benefit of resisting pseudo-label noises largely reduces the gap between the few-label and full-label regimes.
To sum up. this paper makes the following contributions:
We investigate the interactions between features and negative prototypes for semi-supervised seg- mentation and reveal novel mechanism, i.e., while the standard pushing-away interaction is neces- sary, the contrary interaction of pulling close negative prototypes is also beneficial.
We correspondingly propose pull-and-push learning (PPL) method. PPL combines two com- peting interactions and brings two benefits, i.e., resisting the pseudo-label noises and leveraging the contextual information. Moreover, cascading multiple PPL stages accumulate its benefits and thus enlarge the improvement.
We empirically show that PPL substantially improves semi-supervised segmentation by reducing the accuracy gap between full supervision and semi supervision. The achieved results set new state of the art popular benchmarks.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
2.1 SEMI-SUPERVISED SEMANTIC SEGMENTATION
Recently, consistency regularization and pseudo-labeling have been extensively explored in semi- supervised segmentation (French et al., 2019; Ouali et al., 2020; Ke et al., 2020; Chen et al., 2021; He et al., 2021; Hu et al., 2021: Liu et al., 2022), especially after the success of Mean Teacher (Tarvainen & Valpola, 2017) and FixMatch (Sohn et al., 2020) in semi-supervised classification. For example, French et al. (2019) design consistency supervision based on CutMix augmentation to exploit unlabeled data while Ouali al. (2020) and Ke et al. (2020) consider feature and network perturbations, respectively, when designing consistency regularization for unlabeled data. He et al. (2021) propose improve model training by producing unbiased pseudo labels which match the true class distribution in labeled data. Hu et al. (2021) design a confidence bank to balance the training of well and badly performed categories. Liu et al. (2022) extend the mean-teacher model for semi-supervised segmentation by including a new auxiliary teacher and developing stricter confidence- weeighe cross-entropy loss. In this work, we also use consistency regularization and pseudo-labeling to exploit unlabeled data and propose pushing and pulling feature- prototype inter- actions to improve the network training for semi-supervised segmentation.
2.2 CONTRASTIVE LEARNING
Contrastive learning aims to pull positive pairs close while pushing negative pairs far away from each other in the feature space. Recently, contrastive learning has shown its great potential to exploit un- labeled data in sulf--upervised learning (Hjelm al.. 2018: Wu al., 2018; He al., 2020; Chen et al., 2020). This inspires the exploration of contrastive learning in semi- supervised segmentation (Zhou et al., 2021; Alonso al., 2021; Zhong et al., 2021: Lai al., 2021; Wang et al., 2022). For example, Lai et al. (2021) propose to establish consistency between the pixels at the same location but with different contexts and design a directional contrastive loss with the help of pseudo labels. Alonso et al. (2021) maintain memory bank to save high-quality feature vectors from labeled data and perform contrastive learning with this memory bank to improve the network training. Wang et al. (2022) propose to effectively state-of-the methods usually consider pixel-to-pixx contrastive learning. In contrast, the con- trastive learning part in our method compares features against prototypes. We represent each class with multiple prototypes so as to reflect intra-class diversity. Therefore, it better accommodates the mixture of labeled and unlabeled data and is one important reason for the superiority of our method.
unreliable pseudo labels with contrastive learning. These
2.3 CONTEXT LEARNING
Context learning is popular way to improve semantic segmentation by exploring contextual infor- mation from pixels or regions. Yuan et al. (2020) propose OCRNet to augment features with the object-contcxtual representations from local contexts. Jin et al. (2021) leverage object-contextual representations from the whole datasets to improve features. Zhou et al. (2022) apply context learn- ing to weakly supervised semantic segmentation. In this work, our PPL provides alternative approach for context learning, i.e., directly using the weight vectors. However, we note that context learning is not our novelty and is merely the secondary reason for our improvement.
METHODOLOGY
The overview of Push-and-Pull Learning (PPL) is illustrated in Fig. PPL cascades multiple stages for successive improvement (Fig. 2(a)). Each PPL stage consists of 1) a standard classifier for segmentation (with the cross-entropy loss), 2) a contrast-based pushing interaction (Fig. 2(b)) and 3) an attention-based pulling interaction (Fig. 2(c)). Both the contrast-based pushing interaction and the attention-based pulling interaction require storing the historical prototypes from the classifier into classifier memory bank CM, so that each class has multiple prototypes. After the attention- based pulling interaction, the features are refined to some extent and are fed into the classifier of the sub-sequential PPL stage as its input. We note that although our method cascades multiple PPL stages to refine the pseudo labels, only introduces slight computational overhead (as detailed in Appendix C).
3

---

Under review as a conference paper at ICLR 2023
p
Classifier
C.F. Losss
update
f1
Dr/uu
Encoder
CM
Contrast-bare pushing
f2 PPI.
PPI.
stage
PPL sttae n
Classifier
Feature flow
Attention- pulling
Prototype flow
PPL stage 1
Xz/
Label flow
(a) Our Push-and-pus learning
CM
softmax
Negative prosatope
close
NTXIIW
128KHW
Assign label
Pash
120xiXX KTx128
Positive
CM
Scntiure space
256xX
128xXT
(b) Contrast-bused pushing
(c) Attention-basee pulling
Figure 2: Overview of the push-and-pull learning. We perform the pushing interaction by multi- prototype contrastive learning, as shown in (b), and the pulling interaction via a feature-to-prototype attention, as shown in (c). The classifier memory bank CM is designed as queue store class prototypes and is updated by the corresponding classifier weights. Multiple pulling and pushing interactions leveraged sequentially enlarge their benefit. "g" and "Ø" in (c) denote matrix multiplication and element -iiee sum, respectively. P. V,, and are four convolutional modules.
Section 3.1 and Section 3.2 elaborate on the contrast-based pushing and the iitention-baseee pulling, respectively. Section 3.3 illustrates how to integrate the proposed PPL into popular teacher-student framework for semi-supervised segmentation.
3.1 CONTRAST-BASER PUSHING INTERACTION
The contrast-based pushing interaction uses multiple prototypes represent each single semantic class and complements the standard classifier (single prototype for each class). The motivation: us- ing a single prototype lacks consideration for the intra-class diversity and is prone over-fitting the noisy pseudo-labeled samples in the current minibatch. Therefore, we collect additional prototypes from historical classifiers (i.e., the classifiers memory bank CM, as illustrated in Fig. 2(a). Specifically, CM is updated in first in last out manner and maintains constant size K x T x C (K the total number of classes, T the number of prototypes per class and C' is the dimension of each prototype). CM improves the robustness and effectiveness of pushing-away away interaction through a multi multi-prototype contrastive loss function:
previous iterations) and store them in a classifier
H
W
T
Leone(f. CM, M)
Σ
M log
+
exp()(-6(ph cM (D))
[|
h=1
==1
1=1
K
T
Σ
exp()(o(fh.u (ttpo(ffh....kkk.y. + m)),
Jimn 1m11h gh.w Tm
where H and W denote the height and width of the feature map, respectively. M {0, R is mask for selecting high-confident pseudo-labelled pixels for training, ó(a, b) x denotes the cosine similarity between a feature vector and a prototype. fhh and hhw denote the feature vector and the corresponding pseudo label the location (h, w). respectively. cy denotes the prototype of the k-th class in the t-th previous iteration. 7 and m are the scale factor and the margin, respectively. In Eq. (1), the first component aims to pull one pixel feature fh, close its positive prototypes "M (belonging to the same class (i" as the feature) while the second component designed to push the feature far away from its negative prototypes.
a-b
4

---

Under review as a conference paper at ICLR 2023
Aug
fuz
in
Encoder
PPL Stage
PPL Stage
Final classiffe
CE Loss
Xy
EMA
Student network
update
Aug'
8g
PPL Stage
argmax
Aug
Encoder
Stage
Teacher network
9u
Figure 3: Integrating the proposed PPL into popular teacher-student framework for semi- supervised segmentation. The student and teacher networks share the same architecture and both contain multiple (n) PPL stages. The detailed structure of each PPL stage is illustrated in Fig. 2(a). Please note that while each PPL has its own classifier, there is one more classifier ("Final classifier") after the last PPL stage. This final classifier uses the refined features of the last PPL as its input features. The student network uses the ground truth on the labeled data and the pseudo labels on the unlabeled data for supervision. The teacher network applies weak augmentation (Aug) to the unlabeled data, while the student network mixes both weak and strong augmentation (Aug').
3.2 ATTENTION-BANE BASED PULLING INTERACTION
Given the prototypes in CM and the features, the proposed PPL incorporates a pulling-close in- teraction to compete with the contrast-based pushing interaction. This pulling-close interaction is implemented with a feature-prototype attention as shown in Fig, 2(c). Specifically, we first compute the affinity between each feature f and each prototype CM which is defined as,
exp(p(ph.m) (wb(k()))
Whowkt
ΣK_ Σ exp(p(fh.") (") (
(2)
where Whw.th is the feature-prototype affimity score, P and vb are two convolutional blocks. Each block consists of two "I convolutional layers batch normalization ReLU layer".
According to the affinity scores, each feature absorbs some information from all the prototypes into itself for the pulling-close interaction, which formulated as:
fh. (g( k=
Σ
Whow Fx (CMrk.t) + //2
where g and 8 are two convolution blocks consisting of 1 convolutional layer, sub-sequential batch normalization and a ReI.U layer. fi. denotes the refined feature vector at the location (h, w) after absorbing information from the prototypes.
Discussion. The pulling interaction between features and prototypes can be also viewed an in- teraction among all the labeled and unlabeled data across the whole training set. This is because the class prototypes are learned from both the labeled and pseudo-labeled samples, simultane- ously. Therefore, when a feature absorbs information from these prototypes (Eq. (3)), constructs pulling-close interaction towards all its maherently-similae samples in the training set. Under the case where an unlabeled sample is assigned with an incorrect pscudo label, this interaction is partic- ularly beneficial because allows the feature approach its ground-truth class prototype, well as its true- true-postie positive samples. Consequently, the pulling interaction facilitates resistance against noisy pseudo labels and helps refine the features. Using the oracle view, find that using the refined fea- tures to assign the pseudo labels improves the pseudo label quality, as detailed in the supplementary.
3.3 INTEGRATING PPL INTO SEMI-SUPERVISED TEACHER-STUDENT FRAMEWORK
We show how integrate the proposed PPL into popular teacher- framework for semi- supervised segmentation, as illustrated in Fig. 3. We first present the reacher-student framework as follows. The student and teacher models share the same network architecture and parameterized
5

---

Under review as a conference paper at ICLR 2023
with 9 and 8', respectively. The teacher model 0' is updated as the exponential moving average (EMA (Tarvainen & Valpola, 2017)) of the student model 8, which is formulated as:
02 = a 01-1 + (1 ai Or,
(4)
where a is a coefficient controlling the update rate.
In the teacher-student framework, the labeled samples and unlabeled samples are jointly used for training the student. Specifically, the labeled samples 2; (after data augmentation) are directly fed into the student network for training. In contrast, the unlabeled samples Ja are fed into both student and teacher networks in parallel, because we require the teacher prediction for generating the pseudo labels to supervise the student. We note that the student and teacher networks have different data augmentations for the unlabeled data: the teacher network prefers relatively weak data augmentation and transfers Za a.. while the student mixes both weak and strong augmentation and transfers Z. to x This data augmentation strategy is consistent with FixMatch (Sohn et al., 2020).
We integrate the proposed PPL into this training framework by inserting between the encoder (feature extractor) and the classifier, as shown in Fig. 3, for both the student and teacher networks. We note that each PPL stage has its own classifier (as shown in Fig. 2 (a)), and there is one more classifier ("Final classifier" in Fig. 3) after the last PPL stage. For the stu- dent network, all these classifiers need be learned on the combination of labeled and unlabeled data. Specifically, for the labeled data, we employ the cross-entrope loss for supervising the student predictions, which is formulated as:
cascaded PPL stages
n+1 H
W
L H W ΣΣΣ 31
h.
log(Pt N.35
(5)
where 3/-.0 denotes the ground-truth of one labeled image at the location (h, w) and Pi, denote the corresponding prediction. i = 1,..., n indicates the prediction is predicted in the i-th PPI. stage indicates the prediction is from the final classifier. is the number of PPI. stages.
h,w
while i
As for the unlabeled data, given the pseudo labels from the teacher, we apply both cross-entropy loss and the proposed contrastive loss (Eq, (1)) for supervising the student predictions, which is formulated as:
#+1 H W
Lu [MIll
1
ΣΣΣ
M (log(p + Σ Leont(fure CM2, Bu. M).
(6)
where fu, and Pu,i denote the feature before the classifier and the corresponding prediction one weakly or strongly augmented unlabeled image from the i-th PPL stage. is one-hot pscudo label map generated according to the prediction Pu. from the teacher network. M = 1(max(pww) > r)
aims select reliable pseudo labels for training with threshold 0.8.
The overall loss consists of losses for the labeled and unlabeled data, i.e., L, and Lu, written as,
L = Li AL,,
where À is coefficient which controls the contributions different losses. The teacher network updated as the momentum of the student network and thus requires supervision.
EXPERIMENTS
4.1 DATASETS AND EVALUATION
Datasets: We conduct our experiments two standard segmentation datasets, i.e., PASCAL VOC 2012 (Everingham al., 2015) and Cityscapes (Cordts et al., 2016). PASCAL VOC 2012 contains 21 semantic classes to segment (including the background class). In PASCAL VOC 2012 (the augmented set (Hariharan et al., 2011)), 10, 582, 1, 449 and 1, 456 images are used for training, validation and test, respectively. Cityscapes focuses on urban scene understanding and contains 19 classes to segment. Cityscapes, 2, 975, 500 and 1, 525 images are used for training, validation and test, respectively. In our semi-supervised segmentation experiments, randomly sample 1/2,

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison with state-of-the-an methods on PASCAI VOC 2012. Two backbones, i.e. ResNet-50 and ResNet-101 101 and four data partition protocols, ie., 1/16, 1/8 1/4 and 1/2 are consid- ered. The results of U2PLLWang et al., 2022) are from the authors' GitHub issues.
ResNet-50 (%)
ResNet-101 (%)
Methods
1/16
1/8
1/4
1/2
1/16
1/8
1/4
1/2
MT (Tarvainen & Valpola, 2017)
66.77
70.78
73.22
75.41
70.59
73.20
76.62
77.61
CCT (Ouali et al., 2020)
65.22
70.87
73.43
74.75
67.94
73.00
76.17
77.56
CutMix-Seg (French al., 2019)
68.90
70.70
72.46
74.49
72.56
72.69
74.25
75.89
GCT (Ke et al., 2020)
64.05
70.47
73.45
75.20
69.77
73.30
75.25
77.14
CPS (Chen et al., 2021)
71.98
73.67
74.90
76.15
74.48
76.44
77.68
78.64
U PLLWagg et al., 2022)
74.43
77.60
78.70
79.94
PS-MT (Liu et al., 2022)
72.83
75.70
76.43
77.88
75.50
78.20
78.72
79.76
PPL (Ours)
74.22
77.03
77.33
78.03
77.10
78.57
79.50
80.20
1/4, 1/8 and 1/16 of the whole training set as labeled images and treat the rest as unlabeled images. We use the same split lists as (Chen et al., 2021) for both two datasets.
Evaluation: We leverage mean Intersection-overs Union (mIoU) metric measure the segmentation performance. Following (Chen et al., 2021), we report results on 1,456 PASCAL VOC 2012 val set (or 500 Cityscapes val set) on one single scale testing.
4.2 IMPLEMENTATION DETAILS
In this work, we leverage DeepLabv3+ (backboned on ImageNet-pretrainn ResNet-50 and ResNet- 101 (He et al., 2016)) our baseline network. The encoder in Fig. 2(a) denotes all components of DeepLabv3+ except the classifier, including the backbone, ASPP and decoder module.
During training, we adopt random scaling, cropping and flipping for both labeled and unlabeled data. For the unlabeled data, we conduct strong augmentation by additionally applying GrayScale, GaussianBlurring, and ColorJitter. Stochastic gradient descent (SGD) with momentum set to 0.9 is leveraged to optimize the network, and the weight decay is set to le-1. Besides, we adopt the "ploy" learning rate policy where the initial learning rate multiplied by (1 max iter }2 with p 0.9. The learning rate for the pretrained backbone is additionally multiplied by 0.1. The length of the classifier memory bank is set to 10 and the number n of our PPL stages is set to 2. y and m in Eq. (1) is set to 32 and 0.25, respectively. The updating rate a for the teacher network in Eq. (4)
changes during training, written as a = 0.99 + 0.01 (1 cos(T π imax_ite
Her + 1)/2).
On PASCAL VOC 2012, randomly crop images into 512 512 images and train network for 80 epochs with the base learning rate set to 0.01. In the first 10 epochs, we warm up our network by only using labeled data. We set the batch size 8 and the loss coefficient À in Eq. (7) is set to 0.1. The iteration number per epoch is measured by the whole training set size and the batch size. On Cityscapes, we randomly crop images into 800 800 images and train networks for 240 epochs with the base learning rate set to 0.1. We use 20 epochs to warm up our network. We set the batch size to 8 and the loss coefficient À to 0.5. Besides, OHEM loss is used for labeled images and CutMix augmentation is used for unlabeled images on Cityscapes.
4.3 COMPARISON WITH STATE- STATE-OF-THE-ATEE METHODS
Table and Table 2 compare the proposed PPL with seven state-of-the-art methods on PASCAL VOC 2012 and Cityscapes, respectively. From these comparisons, we draw two observations as below:
First, PPL achieves accuracy par with the state of the art. For example, Pascal VOC 2012, PPL based on ResNet-50 surpasses the strongest competitor PS-MT (Liu et al., 2022) by 1.39%, 1.33%, 0.90% and 0.15% under 1/16, 1/8, 1/4 and 1/2 data partition protocols, respectively.
Second, focusing on the accuracy decrease from "1/2" to "1/16" data partition, we observe that PPL presents slower decreases, compared with the competing methods. For example, Pascal VOC 2012, the competing methods suffer decreases of 4.17% (CPS) and 5.05% (PS-MT) when the labeled data is reduced from "1/2" "1/16". In contrast, given the same condition, the accuracy decrease
7

---

Under review as a conference paper at ICLR 2023
Table Comparison with state-of-the-ar methods on Cityscapes, Two backbones, i.e., ResNet-50 and ResNet- 101 and four data partition protocols, i.e, 1/16, 1/8, 1/4 and 1/2 are considered.
ResNet-50 (%)
ResNet-101 (%)
Methods
1/16
1/8
1/4
1/2
1/16
1/8
1/4
1/2
MT (Tarvainen & Valpola, 2017)
66. 14
72.03
74.47
77.43
68.08
73.71
76.53
78.59
CCT (Ouali al., 2020)
66.35
72.46
75.68
76.78
69.64
74.48
76.35
78.29
GCT (Ke et al., 2020)
65.81
71.33
75.30
77.09
66.90
72.96
76.45
78.58
CPS (Chen et al., 2021)
74.47
76.61
77.83
78.77
74.72
77.62
79.21
80.21
PS-MT (Liu et al., 2022)
77.12
78.38
79.22
PPL. (Ours)
77.02
78.54
79.72
80.29
77.54
78.84
80.21
80.79
Table 3: Effectiveness of the attention-based pulling interaction and the contrast- pushing interaction. "Pulling" and "Pushing" denote the developed pulling and pushing interactions, respec- tively. "Semi" and "Fully" denotes networks are trained in semi-supervised and fully-supervised manners, respectively, while "Gap" denotes the accuracy gap between "Semi" and "Fully". Results with semi-supervised learning are evaluated on PASCAL VOC 2012 under 1/8 partition protocol.
ResNet-50 (%)
ResNet- 101 (%)
Method
Pulling
Pushing
Semi
Fully
Gap
Semi
Fully
Gap
Baseline
72.15
75.71
3.56
75.35
78.68
3.33
74.40
77.00
2.60
77.35
79.63
2.28
PPL
74.15
76.84
2.69
76.33
79.21
2.88
77.03
78.05
1.02
78.57
80.64
2.07
of the proposed PPL is relatively small, ie., 3.81%. Therefore, PPL reduces the gap between abundant labeled data and very few labeled data. We recall that PPL has two benefits for semi- supervised segmentation, i.e., general one (i.e., contextual information for general segmentation) and a task-specific one (resistance against pseudo-label noises). This observation shows that the task-specific benefit is the main reason for the superiority PPL.
4.4 EFFECTIVENESS OF THE PUSHING AND PULLING INTERACTIONS
We investigate the pushing and pulling interactions through ablation study under the semi-supervised (1/8 labeled data) and the fully-supervised scenario. The results are summarized Table 3, from which we draw three observations: First, both the pulling and pushing interactions bring independent improvement to the baseline. For example, based on ResNet-50, the pushing and pulling interaction improve the baseline by 2.00% and 2.25%, respectively.
Second, combing the pushing and pulling interactions achieves further improvement. For example, based ResNet-50, PPL achieves 4.89% mloU overall improvement. It indicates that these two interactions are complementary to each other.
Third, comparing "Semi" against "Fully", supervision and full supervision. For example, the baseline undergoes a 3.56% gap, while PPI. reduces this gap 1.02%. It is consistent with our second observation in Section 4.3, further con- firming that the superiority of PPL mainly comes from its task-specific benefit for semi-supervision.
notice PPL reduces the "Gap" between semi-
4.5 PPL IMPROVES THE PSEUDO LABEL QUALITY
PPI. enables to improve the pseudo label quality. We evaluate the generated pseudo labels dur- ing training and present the quantitative and qualitative results in Fig. 4 and Fig. 5, respectively. In particular, three methods are considered, as follows: 1) "baseline" (the popular teacher-student framework), 2) "PPL one stage" (adding PPL stage into the baseline) and 3) "PPL two stage" (Ours). In Fig. 4, the used pseudo labels are collected and evaluated per epoch, and mloU and averaged recall used measure the quality of the pseudo labels.
We draw two observations from Fig. 4. First, we observe that PPL is able to improve both mloU and recall of pseudo-labeled pixels. It indicates that our PPL is able incorporate more pixels with
8

---

Under review as a conference paper at ICLR 2023
10
uiou
Resalli
baadine
besoline
soge
50
17L D0LLow stage
30
(eo stage
12
31
51
11
21
31
71
Ppoch
Epoch
Figure 4: Quantitative comparison of generated pseudo labels from 1) the popular teacher-student "baseline", 2) "PPL one stage" (using one PPL stage) and 3) "PPL two stage" (Ours).
Input
Ground-truth
Baseline
PPL stage
PPL stage
Figure 5: Qualitative comparison of the pseudo label quality. Some pixels are in white color because the predicted pseudo labels are filtered out due to low prediction confidence.
more accurate pseudo labels into training. Second, in Fig. 4, we also notice adding one more PPI. stage improves the recall sightly (while maintaining the mloU). In particular, we average the mloU and recall from all epochs for "PPL stage" and "PPL two stage". "PPL one stage" achieves 73.16% the averaged mloU and 70.26% in the averaged recall, while "PPL two stage" achieves 73.27% and 70.93% in the averaged mloU and recall, respectively.
Combining the above two observations, we infer that PPL refined the features and thus improves the pseudo label quality and this refining effect can be enlarged by cascading multiple PPI. stages.
Fig. visualizes the pseudo labels generated from the baseline, one-stage PPL. and two-stage PPL, respectively. Different colors denote different classes while white indicates the pseudo labels filtered out by the threshold t and are not considered during training. The visualization is consistent with the quantitative results.
CONCLUSION
This paper proposes a novel push-and-pul learning (PPL) for semi-supervised semantic segmenta- tion. In contrast to the common practice of pushing features away from their negative prototypes, PPL considers pulling features close to some negative prototypes sometimes beneficial, as well. Therefore, PPL integrates these two competing feature-prototyy interactions for cooperation. On the one hand, PPL reinforces the fundamental pushing interaction with a multi-prototype contrastive learning. On the other hand, PPL uses naturee-o-proootypp attention to make the feature absorb information from the negative prototypes, yielding the pulling effect. These two interactions both improve the semi-supervised baseline, and bring further improvement when PPL combines them together. Extensive experiments show that PPL effectively reduces the accuracy gap between semi- supervision and full-supervision, and the achieved results are par with the state of the art.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Inigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano, and Ana C Murillo. Semi-supervised semantic segmentation with pixel-level contrastive learning from class-wise memory bank. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8219-8228,
2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference on Machine Learning, pp. 1597-1607, 2020.
Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmen- tation with cross pseudo supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2613-2622, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 3213-3223, 2016.
Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: retrospective. International Journal of Computer Vision, 11 111(1):98-11) 136, 2015.
Geoff French, Timo Aila, Samuli Laine, Michal Mackiewicz, and Graham Finlayson. Semi- supervised semantic segmentation needs strong, high-dimensional perturbations. 2019.
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Seman- tic contours from inverse detectors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 991-998, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Ruifei He, Jihan Yang, and Xiaojuan Qi. Re-distributing biased pseudo labels for semi-supervised semantic segmentation: A baseline investigation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1930-6940, 2021.
R Devon Hjelm, Alex Fedorov, Samuel Davoie-Marchildon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 06670, 2018.
Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised seman- tic segmentation via adaptive equalization learning. Advances in Neural Information Processing Systems, 34, 2021.
Zhenchao Jin, Tao Gong, Dongdong Yu, Qi Chu, Jian Wang, Changhu Wang, and Jie Shao. Min- ing contextual information beyond image for semantic segmentation. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7231-7241, 2021.
Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson WH Lau. Guided collaborative training for pixel-wise semi-supervised learning. In European Conference Computer Vision, pp. 429- 445. Springer, 2020.
Xin Lai, Zhuotao Tian, Li Jiang, Shu Liu, Hengshuang Zhao, Liwei Wang, and Jiaya Jia. Semi- supervised semantic segmentation with directional context-aware consistency. In Proceedings of the IEEF/CV Conference Computer Vision and Pattern Recognition, pp. 1205-1214, 2021.
10

---

Under review as a conference paper at ICLR 2023
Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasileios Belagiannis. and Gustavo Cameiro. Perturbed and strict mean teachers for supervipe semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross-consistency training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 12674-12684, 2020.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li, Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33:596-608, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: sistency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems, 30, 2017.
Weight-averaged con-
Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic seg- mentation. In European Conference on Computer Vision, pp. 173-190, 2020.
Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. Pixel contrastive-consistent semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7273-7282. 2021.
Tianfei Zhou, Meijie Zhang, Fang Zhao, and Jianwu Li. Regional semantic contrast and aggregation for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4299 2299-4309, 2022.
Yanning Zhou, Hang Xu, Wei Zhang, Bin Gao, and Pheng- Ann Heng. C3-semiseg: Contrastive semi-supervised segmentation via cross-set learning and dynamic class-balancing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7036-7045, 2021.
A PPL IMPROVES THE ROBUSTNESS AGAINST NOISY PSEUDO LABELS
Table Comparison between our PPL method and baseline with noisy pseudo labels. "mIoU Gap" denotes the mloU gap between situations with ground-truth and with noisy pseudo labels.
PPL(
baseline (%)
Noise ratio
mIoU
mloU Gap
mIoU mloU Gap
0%
78.37
74.69
10%
77.74
-0.63
73.90
-0.79
20%
77.67
-0.70
72.94
-1.75
40%
76.35
-2.02
71.17
-3.75
To explicitly show that our PPL can resist pseudo-label noises, we manually add noises into the training process. Specifically, during training, we simulate experiments under the 1/8 partition pro- tocol, i.e., sampling 1/8 data the whole training set as the labeled samples and using the remaining 7/8 data as the unlabeled samples. For the unlabeled data, use oracle view (ground truth) to set up the conditions with clean labels, i.e. there 0% pseudo label noise. We set up three noisy condi- tions, i.e., there are 10% / 20% 40% pseudo label noises, by replacing ground-truth with inaccurate pseudo labels at the corresponding ratios. To be specific, we randomly select the index correspond- ing to the 2nd the 3rd largest softmax prediction of the teacher network replace the ground
11

---

Under review as a conference paper at ICLR 2023
truth. Since the 2nd and the 3rd largest indexes correspond the "casy-to-confuse" classes, using them as the incorrect pseudo labels are reasonable.
The results summarized in Table from which clearly observe that when the pseudo labels become noisy, PPL has less accuracy drop than the baseline. For example, when the ratio of noisy pseudo labels is 10%, 20% and 40%, the accuracy drop of the baseline -0.79%, -1.75% and 3.75%, respectively. In contrast, the accuracy drop of the proposed PPL is smaller (-0.63%, -0.70% and -2.02%, respectively). Specifically, the superiority of PPL more significant when the noise ratio is larger. This observation confirms that PPL improves the robustness against noisy pseudo labels.
In Table 4, "0% noise" achieves different accuracy from "Fully" in Table (main text) because they have different experiments setup. "Fully" experiments. all data instead partial data are used for both supervised training and consistency based unsupervised training. Besides, in "Fully" experiments, pseudo labels rather than ground-truth are used for the consistency- -based unsupervised training.
B IMPACT OF HYPER-PARAMETER PARAMETERS
B. IMPACT OF THE LENGTH OF CLASSIFIER MEMORY BANKS (T)
We recall that PPL collects multiple (T) historical prototypes into the memory bank, therefore better reflecting the intra-class diversity. Table 5(a) investigates the impact of T with ResNet-50-basee network under the 1/8 partition protocol on PASCAL VOC 2012, from which we observe that the accuracy first gradually increases (when T increases from 1 to 10) and then decreases. It is rea- sonable because when 7 is too large, some prototypes are severely out-of-date and actually become distractors. Therefore, we recommend using T 10 as an optimum.
B.2 IMPACT OF THE NUMBER OF PPL STAGES (n)
In Table 5(b), we conduct experiments to anal- yse the impact of the number of PPL stages (n). As shown in Table 5(b), PPL achieves the best accuracy when n is set to We infer is because using fewer stages cannot take the full advantage of PPL while using too many stages may distort the output feature which in- volves too much class-wise prototype informa- tion. Therefore, we set 2 in all our experi- ments.
Table 5: Analysis of the impact of classifier mem- ory bank length and PPL stage number n.
T
mloU (%)
n
mloU (%)
0
72.15
75,86
76. 16
5
76.95
2
77.03
10
77.03
3
76.85
20
76.20
4
76.18
(a)
(b)
B.3 IMPACT OF THE SCALE FACTOR 7 AND THE MARGIN I7
We recall that the scale factor and the margin adopted in the developed contrastive loss (in Eq. (1)) for better similarity separation. Table investigates the impact of y and m with a ResNet-
50-based network under the 1/8 partition protocol PASCAL VOC 2012. As shown in Table 6, better accuracy is achieved when and m are set to 32 and 0.25, respectively. Thus, we set yy to 32 and to 0.25 in our experiments.
C COMPUTATIONAL COST OF PPL.
We recall that PPL cascades multiple (2) stages which bring additional computational during inference. Thus, we report the number of parameters and GFLOPs of PPL stages in Table 7, and find that the computational cost of PPL stages are affordable.
12

---

Under review as a conference paper at ICLR 2023
Table 6: Analysis of the impact of the scale factor y and the margin
Y
mIoU (%)
771
mIoU (%)
16
76. 14
0.15
75.80
32
77.03
0.25
77.03
64
75.19
0.35
76.63
(a)
(b)
Table 7: Computational cost PPL during inference.
Method
#params (M)
FLOPs (G)
wo/ PPL
40.42
76.24
PPL stage
40.58
77.60
Two PPL stage
40.75
78.96
13


Reviews:
[{"review_id": "vAgk3um-iwx", "reviewer": "Reviewer_ewoj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a Push-and-Pull Learning (PPL) semantic segmentation, which focuses on the interaction between features and prototypes (similar to classifier weights). The proposed PPL consists of a contrast-based pushing module to pull close/push away features according to the contrastive loss between features and CM (classifier memory). The proposed PPL also contains an attention-based pulling module to absorb contextual information from the classifier memory through attention. This paper extends the proposed PPL into a multi-stage cascade structure and incorporates it into a teacher-student framework for semi-supervised segmentation. The experiments on PASCAL VOC and Cityscapes under semi-supervised settings show some good results.", "summary_of_the_review": "This paper presents a semi-supervised semantic segmentation with the proposed Push-and-Pull Learning. The proposed method achieves promising results under semi-supervised settings. However, the technical novelty of this paper is somewhat limited. Moreover, the main claim of this paper about the negative prototypes has not been well investigated and current experiments cannot demonstrate it and it’s unclear what exactly works in this Push-and-Pull Learning. In all, I’d like to upgrade my score if the authors can better support this claim and clearly state the difference with previous methods using similar methods.", "strength_and_weaknesses": "* Strength\n1. This paper explores the impact of the interaction between the pixel features and negative prototypes in semantic segmentation and presents a Push-and-Pull Learning mechanism.\n2. The proposed PPL performs well on both semi-supervised and fully-supervised semantic segmentation and the experimental results are promising.\n3. The proposed method achieves the state-of-the-art results of semi-supervised semantic segmentation on both Cityscapes and PASCAL datasets.\n\n* Weakness\n1. This paper is motivated to explore the interactions between the pixels and negative prototypes for the following two reasons, i.e., contexts and robustness. However, the proposed two techniques, i.e., contrastive-based module and attention-based module, will simultaneously perform the interaction between pixels and positive/negative prototypes. There is no significant clue that the interaction with negative prototypes could benefit from the two above reasons. The multiple positive prototypes, concretely, the temporal positive prototypes, also bring more contextual information and robustness to the noise. For me, current experimental results cannot support the (main) claim well. \n2. Although the authors discuss the difference between the proposed PPL with [1,2], the authors also claim that pulling interaction is approximately performed across the dataset, the difference of the attention-based module between this paper and [1,2] is minor. And the contrastive learning between pixels and prototypes has also been explored in [3]. The authors should provide more detailed comparisons with previous similar works, which might limit the novelty of this paper.\n\n[1] Jin et.al. Mining Contextual Information Beyond Image for Semantic Segmentation. ICCV 2021.\n[2] Zhou et.al. Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentation. CVPR 2022.\n[3] Zhou et.al. Rethinking Semantic Segmentation: A Prototype View. CVPR 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "tEQ6LYAroj9", "reviewer": "Reviewer_xCik", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for semi-supervised semantic segmentation, via a push-and-pull strategy between features and prototypes. Specifically, given a feature, a contrastive loss is used to perform the pull effect for positive prototypes, while the push effect is used for negative prototypes. Moreover, the authors design another module to refine the feature vector by aggregating it with attention-weighted prototype features. This module interact features with prototypes from each category, so the context information can be extracted. To further utilize the proposed scheme into the semi-supervised framework, the same process can be iterated for many times (2 times in the paper) to refine features and pseudo-labels of unlabeled data. Experiments are conducted in the PASCAL VOC and Cityscapes datasets.", "summary_of_the_review": "Overall, the authors provide an interesting idea of exploring relationships between features and all prototypes of every class. However, the claims are not consistent with the proposed method in Section 3.2. Moreover, it's not clear about the main contribution based on provided experimental results. The authors should carefully address the above-mentioned comments.", "strength_and_weaknesses": "**Strength**\n* The paper is easy to follow\n* The idea of considering the relationship between the feature and prototypes of every category is interesting for semi-supervised learning\n* Results show improvement by including the proposed modules for push and pull effect\n* Final performance is competitive with existing methods\n\n**Weakness**\n\nTechnical novelty and motivation\n* One of the main concerns of this paper is the technical novelty, where the contrastive loss has been commonly used for many tasks, e.g., works mentioned in the related work section. It's not clear how the proposed method in Section 3.1 is different from these methods.\n* For Section 3.2, the authors have mentioned many times that the proposed method would try to pull the features closer to negative prototypes. However, it appears that in Eq(3), the process is to aggregate features with the prototypes, weighted by the feature-prototype attention scores. This is actually very different from the statements claimed in the paper, and thus some of the motivations in the proposed method is not valid.\n\nTechnical clarity and choice\n* For the contrastive loss in Eq(1), have the authors considered the infoNCE based formulation to further improve it? \n* f_1 and f_2 on Fig 2 are not clearly defined\n* The paper mentioned that the memory module CM is updated FILO. However, it's usually designed in a FIFO fashion to keep up-to-date samples. The authors may explain this and provide additional results.\n\nExperimental results\n* Experimental results only show limited performance gains over existing methods, e.g., many of the entries are less than 0.5% gain in Table 1.\n* For Cityscapes, the authors use CutMix augmentation and OHEM loss. It is not clear whether these boost the performance a lot.\n* In the ablation study, Table 3, it's not clear about the effect of using the refined feature in Eq(3). The authors should ablate the push and pull effect in Eq(1), and then also ablate whether the feature is refined in Eq(3).\n* In Fig 4, it's not clear to see the advantage of using the 2nd stage of PPL. With more than 1 stage, which proposed component provides more performance gains?\n* It would be interesting to know the overhead of training time when using more than one stage PPL.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Zmsyv7Bj2qL", "reviewer": "Reviewer_gCrj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the interaction between the feature vectors and classifier vectors (i.e., prototype) to improve the performance of semi-supervised semantic segmentation. Specifically, the paper proposes Push-and-Pull Learning (PPL) which consists of a multi-prototype contrastive loss and an attention-based pulling block. The former pushes the current feature vectors to be far from the negative prototypes in the memory bank. The latter adjusts the current feature vector by fusing it with the negative prototypes via a sequential of attention and convolutions. The experiments are conducted using semi-supervised learning settings on two semantic segmentation datasets, PASCAL VOC and Cityscapes.", "summary_of_the_review": "The paper provides a new perspective on leveraging historical classifiers via contrasting features with historical classifier embeddings. But I still have several concerns listed above. I would rate it as borderline at this stage.", "strength_and_weaknesses": "## Strengths\n- The motivation of the paper is clear. The paper contributes a new perspective, i.e., the feature-prototype interaction, and proposes to leverage the historical classifier vectors to enhance the model.\n- The proposed methods are simple. It is easy to follow and reproduce the results.\n- Clearly better semi-supervised semantic segmentation results are achieved over the baseline and previous methods on PASCAL VOC and Cityscapes.\n\n## Weaknesses\n- Figure 5 with only one example is not sufficient to demonstrate the effectiveness quantitatively. More examples should be provided.\n- It's not clear whether the methods are still beneficial when transferring to more challenging datasets, e.g., ADE20K and COCO, and stronger baselines, e.g., Swin Transformer. \n- More discussion and analysis are needed to explain why the proposed method works. In addition, the limitation of the method is not discussed.\n- Why only the negative classifiers are used? How could we also leverage the positive classifiers?\n- Why the most of the improvements still hold when the classifier memory bank length T is set to 1? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E90jr6LkS5", "reviewer": "Reviewer_CAt6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a semi-supervised learning method for semantic segmentation, which learns a multi-prototype representation for each semantic class and a feature refinement step based on those prototypes. To achieve this, it develops a multi-prototype contrastive loss function and an attention-based prototype-to-feature fusion module. The proposed feature learning strategy is then integrated into a teacher-student framework, in which the student network is augmented with the feature refinement modules and trained with a hybrid loss consisting of cross-entropy losses on labeled data and pseudo-label of unlabeled data, and the contrastive loss on unlabeled data. The authors evaluate the method on Pascal VOC and Cityscapes benchmark with comparisons to the prior art.    ", "summary_of_the_review": "The proposed feature refinement strategy seems interesting and effective for the SS task. However, there are several concerns on the lack of clarity in motivation and presentation, and some missing ablation studies. ", "strength_and_weaknesses": "Strengths:\n- The proposed push-and-pull feature refinement strategy seems interesting and novel for the task of semi-supervised semantic segmentation.  \n- The empirical results show that the method outperforms the previous SOTA methods. \n\nWeaknesses:\n- The motivation of this work seems unclear, especially for the task of semantic segmentation. While the proposed idea is interesting, it is rather orthogonal to the segmentation problem. It is unclear why the proposed method, which is essentially a general SSL method, was only applied to semantic segmentation, and what segmentation property has been exploited here. \n- The multi-prototype and attention-based feature refinement has been explored in the few-shot semi-supervised semantic segmentation, which is related. \nLing et al,  Semi-Supervised Few-shot Learning via Multi-Factor Clustering, CVPR 2022.\n- Some part of the paper is less clear and lacks details: \n1) Attention-based pulling: It is unclear what specific type of \"contextual information\" is captured since the current description is slightly vague. \n2) The PPL classifiers used in Equations 5 and 6 (first term) are less clear. \n3) The settings for the ablation study are unclear. What is the baseline? How the PPL is trained in the fully-sup setting? \n- There are a few concerns on the experiments:\n1) The method adds multiple stages of PPL, which essentially leads to deeper networks and deep supervision for multiple stages. What is the model complexity of this design compared to the previous methods and the baseline? It would be interesting to see a comparison of network design with other types of feature refinement, such as the non-local neural network, and/or with deep supervision. \n2) It is unclear how the conclusion (Sec 4.3 & 4.4) of \" the task-specific benefit is the main reason\" was reached based on the results as both modules make important contributions to the final improvement. \n3) The difference between one-stage and two-stage PPL seems rather small according to fluctuation in the performance curves. How significant is the improvement from the two-stage PPL?  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject