Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning
Authors: Hanzhe Hu
Fangyun Wei
Han Hu
Qiwei Ye
Jinshi Cui
Liwei Wang
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: semantic segmentation
semi-supervised learning
adaptive equalization learning

Abstract: Due to the limited and even imbalanced data, semi-supervised semantic segmentation tends to have poor performance on some certain categories, e.g., tailed categories in Cityscapes dataset which exhibits a long-tailed label distribution. Existing approaches almost all neglect this problem, and treat categories equally. Some popular approaches such as consistency regularization or pseudo-labeling may even harm the learning of under-performing categories, that the predictions or pseudo labels of these categories could be too inaccurate to guide the learning on the unlabeled data. In this paper, we look into this problem, and propose a novel framework for semi-supervised semantic segmentation, named adaptive equalization learning (AEL). AEL adaptively balances the training of well and badly performed categories, with a confidence bank to dynamically track category-wise performance during training. The confidence bank is leveraged as an indicator to tilt training towards under-performing categories, instantiated in three strategies: 1) adaptive Copy-Paste and CutMix data augmentation approaches which give more chance for under-performing categories to be copied or cut; 2) an adaptive data sampling approach to encourage pixels from under-performing category to be sampled; 3) a simple yet effective re-weighting method to alleviate the training noise raised by pseudo-labeling. Experimentally, AEL outperforms the state-of-the-art methods by a large margin on the Cityscapes and Pascal VOC benchmarks under various data partition protocols. Code is available at https://github.com/hzhupku/SemiSeg-AEL.

Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning
Hanzhe Hu Fangyun Wei²t Han Hu² Qiwei Ye²
Jinshi Cui! Liwei Wang! L.3
Key Laboratory of Machine Perception (MOE), School of EECS, Peking University
Microsoft Research Asia
Institute for Artificial Intelligence, Peking University *Zhejiang Lab
hubzupku. edu. en {fawe, hanhu, qiwye}-@mi crosoft. {cjs, wanglu)0cis. pku. en
Abstract
Due the limited and even imbalanced data, semi-supervised semantic segmen- tation tends to have poor performance on some certain categories, e.g., tailed categories in Cityscapes dataset which exhibits a long-tailed label distribution. Existing approaches almost all neglect this problem, and treat categories equally. Some popular approaches such as consistency regularization pseudo-labeling may even harm the learning of under-performing categories, that the predictions or pseudo labels of these categories could too inaccurate to guide the learning on the unlabeled data. In this paper, we look into this problem, and propose novel frame- work for suni-supervieed semantic segmentation, named adaptive equalization learning (AEL). AEL adaptively balances the training of well and badly performed categories, with confidence bank to dynamically track category-wise performance during training. The confidence bank is leveraged as an indicator to tilt training towards under- sadder-prffoming categories, instantiated in three strategies: I) adaptive Copy-Paste and CutMix data augmentation approaches which give more chance for under-performing categories to be copied or cut; 2) an adaptive data sampling approach to encourage pixels from mnder-performing category to be sampled; 3) simple yet effective re- reiweigtting method to alleviate the training noise raised by pseudo-labeling. Experimentally, AEL outperforms the state-of of-the-art methods by a large margin on the Cityscapes and Pascal VOC benchmarks under various data partition protocols. Code is available https://github.
Introduction
Supervised semantic segmentation requires pixel-level labeling, which is expensive and time- consuming. This paper is interested in semi-supervised semantic segmentation, which can greatly reduce the efforts of pixel-level annotation, yet may maintain reasonably high accuracy. One problem of common semantic segmentation datasets is that the pixel categories tend to be imbalanced, e.g., the pixel amount of head classes be hundreds of times larger than that of tailed classes in the widely used Cityscapes dataset [1]. The situation is more serious in the semi-supervised setting where tailed classes may have extremely few samples. We note that recent approaches are mainly dedicated the design of consistency regularization [2, 3, 4, 5, 6, 7] and pseudo-labeling [8], almost all of which neglect the imbalance problem and treat each category equally, leading to a biased training. These approaches may even harm the learning of tailed classes, as inaccurate predictions or pseudo labels of under- perforrin categories could falsely guide the learning on unlabeled data.
This work is done when Hanzhe was an intern in MSRA. Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

1.4
1.4
266
1.2
1.2
Pixel Number
1.0
Pixel Number
1.0
0.8
Baseline
Baseline
0.6
AEL
0.6
AEL
0.4
0.4
0.2
0.2
0.0
3
5
11 Class Index
9
15 17 19
15
Class Index
(a) 1/16 data partition protocol.
(b) 1/32 data partition protocol.
Figure 1: We count the the training samples of each category on Cityscapes train set under 1/16 and 1/32 data partition protocols, and compare the proposed AEL with a strong semi-supervised learning baseline described in Section 3.2 which treats each category equally. Our method strives to tilt training towards tailed categories which usually tend be under-performing.
This paper aims to alleviate this biased training problem. We propose novel Adaptive Equalization Learning (AEL) framework, which adaptively balance the training of different categories as shown in Figure 1. Our design follows two main principles: 1) increasing the proportion of training samples from the under-performing categories; 2) tilting training towards under-performing categories. Concretely, we maintain a confidence bank to dynamically record the category-wise performance at each training step. which indicates the current performance of each category. Following principle 1), we propose two data augmentation approaches named adaptive Copy-Paste and adaptive CutMix, which give more chance for inder-performing categories to be copied cut. Following principle 2), we present an adaptive equalization sampling strategy to encourage pixels from under-performing categories to be sufficiently trained. In addition, also introduce simple yet effective re-weighting strategy which takes the model predictions into account to alleviate the issue that semi-supervised learning usually suffers from the training noise.
Experimentally, by using the DeepLabv3+ with ResNet-101 backbone, the proposed AEL outperforms Mate-offthheart methods by a large margin the Cityscapes and PASCAL VOC 2012 benchmarks under various data partition protocols. Specifically, it achieves 74.28%, 75.83% and 77.90% on Cityscapes dataset under 1/32, 1/16 and 1/8 protocols, which is +16.3994 +12.87% and +8.09% better than the supervised baseline. When evaluated on PASCAL VOC 2012 benchmark, it achieves 76.97% 77, 77.20% and 77.57% under 1/32, 1/16 and 1/8 protocols, which is +6.83% +6.60% and +4%45% better than the supervised baseline. Moreover, the proposed approach also proves to improve the segmentation model trained the full Cityscapes train set by +1.03% by leveraging 5, 000 images from the Cityscapes coarse set as unlabeled data, achieving 81.95%.
2 Related Work
Semi-Supervised Learning. Recent years have witnessed a significant progress in the SSL field. Most of them can be categorized into consistency regularization, entropy minimization [9] and pseudo-labeling. Consistency regularization [10, 11, 12] enforces consistency in predictions between different views of unlabeled data. Pscudo-labeling [13, 14] trains the model on the unlabeled data with pseudo labels generated from the model's own predictions. Furthermore, [10, 15, 16, 17, 18] use low softmax temperature to sharpen the predictions of unlabeled set. Our method refers to Mean Teacher [1]] and FixMatch [14] when designing our basic framework.
Semi-Supervised Semantic Segmentation. Existing semi-superee supervisd semantic segmentation meth- ods mainly focus on the design of consistency regularization and pseudo-labeling, Cutmix-Seg [2] applies CutMix augmentation on the unlabeled data. CCT [4] introduces feature-level perturbation and enforces consistency among the predictions of different decoders. GCT [19] performs network perturbation by using two differently initialized segmentation models and encourages consistency between the predictions from the two models. PseudoSeg [8] focuses improving the quality of

---

Labeled Data
Addatiiee Copy-Paste
Student
EMA Update
Sepmentation
Category Wise
Model
Confidence Bank
Prob Map
S
AES
Dynamic
Unlabeled Data
Lu
Re-weighting
EMA Update
Map
Segmentation
Prob Map
Model
Pseudo Labels
Teacher
Prob Map
Adaptive CutMix
Figure 2: Overview of AEL. We adopt the teacher-student architecture as our basic framework. The teacher model is updated by the exponential moving average (EMA) of the student model. Confidence bank is uesd to dynamically record the category-wise performance during training. Adaptive CutMix and adaptive Copy-Paste are applied on the unlabeled and labeled data respectively to provide sufficient training samples from the under-performing categories. Adaptive equalization sampling (AES) encourages the training to involve more samples from the under-performing categories make the training unbiased. Dynamic re-weighting strategy aims to alleviate the noise of pscudo-labeling.
pseudo labels. Though achieving satisfactory improvements the supervised baseline, none of the aforementioned methods explore the biased learning issue in semi-supervised semantic segmentation.
Class Imbalance in Semi-Supervised Learning. Although SSL has been extensively studied, class imbalance problem SSL is relatively under-explored, especially for semantic segmentation. Yang et al. [20] demonstrate that leveraging unlabeled data alleviate imbalance issue. Hyun al. [21] propose a suppressed consistency loss for class muass-imbllanced image classification problems. CReST [22] introduces self- self-training framework for imbalanced SSL. Our method, though not explicitly targeting at the class imbalance problem, focuses on improving the performance under- performing categories which mostly tailed classes. Moreover, refer the ideas of re- sampling [23, 24] and re-weighting [25, 26]. which are designed for class imbalance problem.
3 Method
Given labeled set D {(x), bt)} and an unlabeled D {x the objective of semi- supervised semantic segmentation is to learn a segmentation model by efficiently leveraging both labeled and unlabeled data. In this section, Section 3.1. Then we describe our basic framework for semi-supervised semantic segmentation in Section 3.2. Finally, the details of AEL introduced in Section 3.3.
first present overview of the proposed AEL in
3.1 Overview
Figure 2 displays an overview of AEL, which is data-efficient framework for semi-supervised semantic segmentation. It composed of two parts: 1) a basic framework which contains teacher model for pscudo-labeling and a student model for online learning; 2) dedicated modules which encourages the ander-performing categories to be sufficiently trained by effectively leveraging both labeled and unlabeled data. We use the proposed confidence bank dynamically record the category- wise performance during training, and thus we can easily identify which categories are not sufficiently trained. For those unsatisfactory categories, we present two data augmentation methods increase their frequency of occurrence in training batch, namely adaptive CutMix which is applied on unlabeled data, and adaptive Copy-Paste which is applied on the labeled data. make model towards the unbiased learning, we propose the adaptive equalization sampling and dynamic re- weighttting strategies to involve enough samples from the under-performing categories into the training, and alleviate the noise raised by pseudo-labeling simultancously.
3

---

3.2 Basic Framework
We first set up basic framework for sami-supereieed semantic segmentation. The framework consists of student model and teacher model. The teacher model has the same architecture as the student model, but uses a different set of weights which are updated by exponential moving average (EMA) of the student model [11]. Following FixMatch [14], we use the teacher model to generate set of pseudo labels y = } on the weakly augmented unlabeled data D". Subsequently, the student model is trained on both labeled data D' (of weak augmentation) with the ground-truth and unlabeled data D (of strong augmentation) with the generated pseudo labels y We use standard random resize and random horizontal flip as the weak augmentation. Strong augmentation includes CutMix [27] and all data augmentation strategies used in the weak augmentation.
The overall loss consists of the supervised loss Ls and the unsupervised loss fu:
N₂
WH
C.
1
Σ
tce ((yse.PP))
N
WH
N
WH
C. 1 N
1
Lce
(2)
WH
where Pij is the prediction of the j-th pixel in the i-th labeled (or unlabeled) image, N₁ and N₂ denote the number of labeled images and unlabeled images in a training batch, W and H represent the width and height of the input image, and toe denotes the standard pixel-wise cross-entropy loss. We define the overall loss function as:
L = Las + OhLur
where a controls the contribution of the unsupervised loss.
3.3 Adaptive Equalization Learning
The baseline framework, though achieving competitive results compared with previous related works, neglects the key issues in semi-supervised semantic segmentation. Due to the limited labeled data, semi-supervised learning tends have poor performance on some certain categories, e.g., tailed categories in Cityscapes dataset which exhibits a long-tailed label distribution. Insufficient training on these categories introduces more noise of pseudo labels which can disrupt the learning process. The proposed AEL framework aims to alleviate the degradation of under-perie seddrr-prffrrmng categories during the semi-supervised training. Concretely, we maintain a confidence bank record the performance of each category during training. The confidence bank enables us to identify the under-performn performing categories. To improve the performance of these categories and further make the training unbiased, we propose a series of technologies to efficiently leverage both labeled and unlabeled data, namely adaptive CutMix, adaptive Copy-Paste adaptive equalization sampling and dynamic re-weighting.
Confidence Bank. To tackle the biased training, previous methods [25, 26, 22, 28] always rely on the prior knowledge such as the number of training samples of each category design the ad hoc sampling and weighting strategies. However, the performance of each category is not always strictly proportional to the number of training samples, because some categories tend have discriminative features and thus fewer samples are required for training. Inspired by the recent progress [29] which applies active learning on semantic segmentation, we propose to maintain confidence bank to record the category-wise performance during training. An indicator is needed to assess the performance of each category.
We consider several indicators, namely Confidence, Margin and Entropy. Formally, we define Confidence indicator
N Σ
Conf
Σ Vig CE (1....,
where C is the category number, No denotes the number of pixels belonging to category according to its ground-truth yı pi, denotes the e-th channel prediction of the j-th pixel in the i-th image.

---

Define Margin indicator as:
Ni
N Pij
Margin
max2 e'e(.....
ply), Ce (L....,c}
(5)
Ni ==1
where max2(-) denotes the second largest value operator. At last, we define Entropy indicator as:
Mi
N
1 N
Ent
Piy log pig. CE (}.....,O..
(6)
For all of the indicators, we only take into account predictions from labeled data. Experimentally, the confidence indicator serves best in our AEL and thus we adopt it by default (see Section 4.3 for the comparison). We use EMA update the category-wise confidence each training step:
Conf& f -Conf + (1 - C)nfff c C {l....,C).
(7)
where denotes the k-th iteration, r E [o. 1) is the momentum coefficient which is set to 0.999 experimentally. Through the confidence bank, can easily identify the ander-performin categories for the current model.
Adaptive CutMix. Here we introduce the proposed adaptive CutMix (see Figure 2 for illustration) which applied on the unlabeled data. aims to increase the frequency of occurrence of the under-performing samples from the unlabeled data. We first formulate the original CutMix [27] as:
CutMix(Cropp(4) I2),
(8)
where / and 12 denote randomly selected unlabeled images, î is the augmented image, and Crop(-) represents the random crop operation.
Different from the original CutMix where unlabeled images are randmoly selected, the proposed adaptive CutMix gives under-perte peder-performing categories a higher sampling probability. Specifically, we first convert the category-wise confidence stored in the confidence bank the normalized sampling probability r E RC, which can be formulated as:
Softmax(1 Conf).
According to the sampling probability, randomly select unlabeled image containing the sampled category as I,, and another unlabeled image from the training batch is randomly selected as Iz. The Crop(-) operation is performed on the region containing the chosen category. After that, we can generate the augmented image by Eq 8. Since the adaptive CutMix is performed on the unlabeled data without any annotations, we use predictions approximate ground-truth, which works well in practice.
Adaptive Copy-Paste. Copy-Paste [30] is an effective data augmentation strategy for instance segmentation. It yields significant gains on the challenging LVIS benchmark [31], especially for rare object categories. The key idea behind the Copy-Paste augmentation is to paste objects from the source image to the target image. Inspired by this, we further propose the adaptive Copy-Paste (see Figure 2 for illustration) for semi-supervised semantic segmentation. Different from adaptive CutMix, adaptive Copy-Paste augmentation strives for efficiently leveraging the labeled data. Similarly, we involve confidence bank to assess category- performance and Eq 9 to compute sampling probability. The unreo-ipgo categories have higher probability to be selected for Copy-Paste. Experimentally, the proposed adaptive Copy-Paste Paste augmentation yields slightly better performance in the category level than the instance level. Thus we copy all pixels belonging to the sampled category in the source image and paste them on the target image. Following [30], the augmented image is composed of two randomly selected images from the labeled data and a large scale jittering is applied.
Adaptive Equalization Sampling, As described in Section 1, due the limited and unbalanced labeled data, the training tends to be biased. To alleviate the training bias, we propose a novel adaptive sparse set of under- perfornng samples
equalization sampling strategy which focuses training and prevents the vast number of well-trained samples from overwhelming the model during training. Coneretely, we define the sampling rate se for category e as:
- Conf
8
cE [L,...,C],
(10)
max max..e1.....om (I....)))))) Conf*)
5

---

where B denotes tunable parameter. Instead of using all pixels to compute the unsupervised loss, for category e with the sampling rate s", we randomly sample subset of pixels according to their predictions. Then the unsupervised loss in Eq 2 can be reformulated as:
N L 1=1
WH Σ Li 1=1
Lu =
(ec (lij
(11)
Nu
where 1iy = indicates that the j-th pixel in the i-th image is sampled according to the sampling rate, otherwise 1,y is set to 0, the other terms are the same in Eq 2.
Dynamic Re-Weighting. The performance of the model depends on the quality of pseudo labels. Existing methods [2, 4, 19] usually adopt higher threshold on classification score to filter out most of the pixels with low-confidence. Though this strategy could alleviate the noise raised by pseudo- labeling, the strict criteria leads to lower recall for the pixels from under-performing categories, which hinders the training. Another option is to discard the threshold and involve all pixels into the training. However, much more noise is introduced simuitaneously. To alleviate this issue, we propose a dynamic re-weighting strategy which adds a modulating factor to the unsupervised loss in the way of semi-supervised learning. On the basis of Eq 11, we formulate our final unsupervised loss as:
Nu
WH
1
Σ
Wky wxylee (iij.
(12)
N
WH j=1
E
Wij
==1
Wij
max eE(1.....c
(p 1ay.
(13)
where 7 is the tunable parameter. Different from the Focal Loss [32] where the modulating factor is used for reducing the loss contribution from easy samples, contributions for the convincing samples. The combination of adaptive equalization sampling and
formulation aims allocate more
dynamic weighting not only involves more samples from the under-performing categories into the training, but also alleviate the noise raised by pseudo-labeling
Experiments
4.1 Setup
Cityscapes [1] dataset is designed for urban scene understanding. It contains 30 classes and only 19 classes of them used for scene parsing evaluation. The dataset contains 5, 000 finely annotated images and 20, 000 coarsely annotated images. The finely annotated 5, 000 images are split into 2, 975, 500 and 1, 525 images for training, validation and testing respectively.
Datasets.
PASCAL VOC 2012 [33] dataset is a standard object-centric semantic segmentation dataset. contains 20 foreground object classes and a background class. The strand training, validation and testing sets consist of 1, 464, 1, 449 and 1, 556 images, respectively. Following common practice, we use the augmented set [34] which contains 10, 582 images the training set.
ADE20K dataset [35] a large scale scene parsing benchmark which contains dense labels of 150 stufffobject categories. The dataset includes 20K/2K/3K images for training, validation and testing.
For both Cityscapes and PASCAL VOC 2012 datasets, 1/2, 1/4, 1/8, 1/16 and 1/32 training images are randomly sampled as the labeled training data, and the remaining images are used as the unlabeled data. For each protocol, AEL provides 5 different data folds and the final performance is the average of 5 folds. In addition, we also evaluate our method on the setting where the full Cityscapes train set is used as the labeled data and 1, 000, 000 and 5, 000 images and randomly selected from the Cityscapes coarse set as the unlabeled data.
We use single scale testing and adopt mean of Intersection over Union (mloU) as the metric to evaluate the performance. We report the results on the Cityscapes val set and PASCAL VOC 2012 val set in comparisons with state-of-the-ee methods. All ablation studies are conducted on the Cityscapes val set under 1/16 and 1/32 partition protocols.
Evaluation.
Implementation Details. We use ResNet- 101 pretrained on ImageNet [36] our backbone, remove the last two down-sampling operations and employ dilated convolutions in the subsequent convolution layers, making the output stride equal to 8. We use DeepLaby3+ [37] the segmentation head. For

---

Table I: Comparison with hate-of-theear methods the Cityscapes val set under different partition protocols. All the methods are based on DaepLabv3+ with ResNet- 101 backbone.
Method
1/32 (93)
1/16 (186)
1/8 (372)
1/4 (744)
1/2 (1488)
Supervised
57.89
62.96
69.81
74.23
77.46
MT [11]
64.07
68.05
73.56
76.66
78.39
CCT [4]
66.35
69.32
74.12
75.99
78.10
Cutmix-Seg [2]
69.11
72.13
75.83
77.24
78.95
GCT [19]
63.21
66.75
72.66
76.11
78.34
AEL (Ours)
74.28
75.83
77.90
79.01
80.28
Table 2:
Comparison with state-of-the-art methods on the PASCAL VOC 2012 val set under different partition protocols. All the methods are based on DeepLabv3+ with ResNet- 101 backbone.
Method
1/32 (331)
1/16
(662)
1/8 (1323)
1/4 (2646)
1/2 (5291)
Supervised
70.14
70.60
73.12
76.35
77.21
MT [11]
70.56
71.29
73.33
76.61
78.08
CCT [4]
71.22
71.86
73.68
76.51
77.40
Cutmix-Seg [2]
73.39
73.56
73.96
77.58
78.12
GCT [19]
70.32
70.90
73.29
76.66
77.98
AEL (Ours)
76.97
77.20
77.57
78.06
80.29
Cityscapes dataset, weight decay 0.0005 and momentum 0.9. Moreover, we adopt the 'poly' learning rate policy, where the initial learning rate is multiplied by
stochastic gradient descent (SGD) optimizer with initial learning rate 0.01,
0..9 We adopt the crop size as 769 769, batch set the initial learning
max ier
size 16 and training iterations as 18k. For PASCAL VOC 2012 dataset, rate as 0.001, weight decay as 0.0001, crop size as 513 x 513, batch size as 16 and training iterations 30k. We random horizontal flip and random resize the default data augmentation if not specified. All the supervised baselines are trained on the labeled data.
4.2 Comparison with State-of-the-Art Methods
We compare method with recent semi-supervised semantic segmentation methods, including Mean Teacher (MT) [11]. cross-Consistency Training (CCT) [4]. Guided Collaborative Training (GCT) [19] and Cutmix-See [2]. For a fair comparison, re-implement all above methods and adopt the same network architecture (DeepLabv3+ with ResNet-101 backbone).
Results on Cityscapes Dataset. Table compares AEL with state-of-the-an methods on the Cityscapes val set. Without leveraging any unlabeled data, the performance of the supervised baseline is unsatisfactory under various data partition protocols, especially for the fewer data settings, e.g., 1/32 and 1/16 protocols. Our method consistently promotes the baseline, achieving the improve- ments of + +1..4% +12 +12.9% +8.1 18, +4.8% and +2.8% under 1/32, 1/16, 1/8, 1/4 and 1/2 partition protocols respectively. Our method also significantly outperforms the existing state-of-the-an meth- ods by large margin under all data partition protocols. In particular, AEL outperforms the existing best method Cutmix-Seg by +5.2% under extremely few data setting (1/32 protocol), and surpasses Cutmix-Seg by +1.3% under the 1/2 protocol.
Results on PASCAL VOC 2012 Dataset. Table 2 shows comparison with state-of-the art methods on the PASCAL VOC 2012 val dataset. AEL achieves consistent performance gains over the supervised baseline, obtaining an improvements +6.8%, +7.0%, +4.1 %, +1.7% and +3.1 1% under 1/32, 1/16, 1/8, 1/4 and 1/2 partition protocols respectively. We can see that over all protocols, ABL outperforms the sfate-of-the-ar methods. For example, our method outperforms the previous best method by +3.6% and +2.2% under the 1/32 and 1/2 partition protocols.

---

Table 3: Ablation study the effectiveness of different components: Dynamic Re- Re-weighting (DR),
Adaptive Equalization Sampling(AES), Adaptive CutMix (ACM), Adaptive Copy-Paste (ACP).
DR
AES
ACM
ACP
1/32 (93)
1/16 (186)
69.11
72.13
70.27
73.85
71.65
74.12
70.49
73.89
69.69
72.64
72.51
74.39
73.43
75.12
74.28
75.83
4.3
Ablation Study
To further understand the advantages of AEL, we conduct series of ablation studies that examine the effectiveness of different components and different hyper-parameters. All experiments are conducted on the validation set Cityscapes dataset.
The Effectiveness of Different Components. We ablate each component of AEL step by step. Table reports the studies. We use the basic framework described in Scction 3.2 our baseline, which achieves 69.1 11% and 72. 13% under 1/32 and 1/16 protocols respectively. We first evaluate the effectiveness of each single component. As shown in the table, Dynamic Re- weightitt (DR) improves the baseline by +1.1% % and +1.7% under 1/32 and 1/16 partition protocols. Adaptive Equalization Sampling (AES) alleviates the biased training issue, achieving the improvements of +2.5% and +2.0% over the baseline. Adaptive CutMix (ACM) and Adaptive Copy-Paste (ACP) data augmentation approaches give more chance for under-performing categories to be sampled, and bring the improvements of +1.3% +1.7% and +0.5% / +0.5% respectively. Furthermore, we present the performance gains in progressive manner. On top of the DR, by leveraging AES strategy on the unsupervised loss, our method obtains improvements of +2.3% and +0.5% under 1/32 and 1/16 protocols. The two proposed data augmentation approaches further boost the performance to 74.28% and 75.83%, demonstrating the effectiveness of our adaptive learning.
Ablation Study on Hyper-Parameters. Table ablates the tunable parameter ? dynamic re- weighting (in Eq 13), where 2 yields slightly better performance. Dynamic weighting is found to be insensitive to
Table 6 ablates the influence of different indicators, including Confidence (in Eq 4), Margin (in Eq 5), and Entropy (in Eq 6). We use the Confidence the default indicator assess the category-wise performance during training due to its best performance.
Adaptive CutMix requires criteria to identify whether an unlabeled image contains certain class. We use the ratio between pseudo labels of a certain category and total pixels the input image as the criteria. Table 7 ablates different ratios.
Table 8 studies the number sampled categories K in the Adaptive Copy-Paste. find that K achieves the best performance. One potential reason is that smaller K provides less training samples from the under-performing categories while larger K may increase the difficulty for training.
Table 9 ablates the loss weight a which is used to balance the supervised loss and unsupervised loss as shown in Eq 3. As illustrated in the table, a 1 achieves the best performance. We use a in our approach for all the experiments.
4.4 Per-class Results
Since the class imbalance problem severe the Cityscapes dataset, we provide per-class results under 1/32 data partition protocol in Table 4. We choose 9 classes with the least training samples the Cityscapes dataset as tail classes, i.e. wall, traffic light, traffic sign, rider, truck, bus, train, motorcycle and bicycle. As shown in the table, method not only achieves the best overall mloU,

---

Table 4: Per-class results Cityscapes val under 1/32 data partition protocol. All the methods are based on DeepLabv3+ with ResNet-101 backbone.
Head Classes
Tail Classes
mlol Joi
vegetution
tniliic light
sign
motveeyele
mllel
r0od
sidewalls
building
Fence
pole
pers00
wall
nieer
bus
Inan
bicycle
Methods
Supervised
57.9
39.8
94.6
72.5
42.4
51.6
89.4
21.7
33.7
43.3
37.2
11.4
42.1
62.2
GCT[
63.2
48.1
75.8
57.5
53.5
23.6
434
45.7
49.2
21.4
MT [1]]
64.1
50.4
96.7
75.6
89.5
40.0
57.3
91.0
53.2
92.80
77.9
91.3
26.2
61.1
72.3
45.8
28.0
48.1
51.6
47.1
73.8
CCT [-]
66.4
54.2
95.7
77.2
88.6
46.5
58.5
90.1
55.5
91.5
77.9
91.8
27.9
60.5
71.8
48.0
44.5
61.4
50.7
52.0
70.5
Cutmix Seg [2]
69.1
58.7
97.2
78.6
90.1
48.1
60.1
91.5
57.2
93.0
79.6
93.3
32.4
64.8
76.5
52.3
49.4
66.0
54.8
56.7
75.1
AEL (Ours)
74.3
67.9
97.1
78.7 90.3 52.3 62.0
91.7
59.2
93.8
81.6
94.0
37.3
67.9 77.6
60.5
65.6
83.8
74.3
66.9
77.0
Table 5: Study on y of dy- namic re-weighting
Table 6: Study on different in- dicators for AES.
Table 7: Study on different ra- tios in ACM.
1/32
1/16
Indicator
1/32
1/16
Ratio
1/32
1/16
0
69.11
72.13
None
70.27
73.85
0.001
73.27
74.28
0.5
69.74
73.28
Ent
71.38
73.21
0.003
73.29
74.36
1
69.35
73.67
Conf
72.51
74.39
0.005
73.43
75.12
2 70.27
73.85
Margin
70.86
73.05
0.01
72.78
73.66
3
70.26
73.40
but also obtains sigmificant improvements on tail classes. In particular, Our method outperforms the existing best method Cutmix-Seg by +5.2% in overall mloU and 49. 2% mloU for tail classes under 1/32 partition protocol.
4.5 Performance on the Full Labeled Set
We conduct experiments where the full Cityscapes train set is used as the labeled dataset and the Cityscapes coarse set is used as the unlabeled dataset. We do leverage any annotations from the coarse set though it provides coarsely annotated ground-trume and 5,000 images from the coarse set to verify the proposed method. As shown Table 10, the proposed AEI. can still improve the supervised baselines by leveraging the unlabeled data though large amount labeled data is provided.
We randomly sample 1, 1,000, 3,000
4.6 Results on ADE20K Dataset
We further provide results on the ADE20K dataset [35]. Since no previous methods in cemi-supervised segmentation conducted experiments ADE20K dataset, we compare our method with supervised baseline and existing best method Cutmix-Sep on the dataset. As shown in Table 11, our method consistently promotes the supervised baseline by 6.36%, 5.70%, 5.67%, 3.18% and 1.45%, and outperforms the Cutmix-Seg by 2.25%, 3.38%, 2.48%, 1.31% and 1.26% under 1/32, 1/16, 1/8, 1/4 and 1/2 partition protocols, respectively.
4.7 Qualitative Results
Figure 3 shows the visualization results of different methods evaluated on the Cityscapes val set. We compare the proposed AEL with ground-truth, supervised baseline and our basic framework described in Section 3.2. Benefiting from series of technologies designed for the balanced training, AEL achieves great performance on not only head categories (e.g. Road), but also tailed categories (c.g. Rider and Bicycle).

---

Table 8: Study number of sampled categories K in ACP.
Table 9: Study loss weight
Table 10: Performance the full Cityscapes train set.
a.
K
1/32
1/16
Q
1/32
1/16
Number
Baseline
AEL
1
72.18
74.85
0.5
71.85
74.61
0
80.16
2
72.84
74.95
1.0
74.28
75.83
1000
80.22
80.28
3
74.28
75.83
1.5
74.10
73.44
3000
80.55
81.36
4 73.43
74.10
2.0
73.79
72.86
5000
80.92
81.95
Table 11: Comparison with supervised baseline and Cutmix-Seg the ADE20K val under different partition protocols. All the methods are based on DeepLabv3+ with ResNet- 101 backbone.
Method
1/32 (631)
1/16 (1263)
1/8 (2526)
1/4 (5052)
1/2 (10105)
Supervised
22.04
27.52
32.36
36.39
41.97
Cutmix-Seg [2]
26.15
29.84
35.55
38.26
42.16
AEL (Ours)
28.40
33.22
38.03
39.57
43.42
Input
Ground Truth
Supervised
Basic Framework
ARL.
Figure 3: Qualitative results the Cityscapes val set. From left right: input image, ground-trute truth, predictions of the supervised baseline, predictions of our basic framework and predictions of the proposed AEL. Orange rectangles highlight the unsatisfactory segmentation results.
5 Conclusion
In this paper, propose a novel Adaptive Equalization Learning (AEL) framework for semi- supervised semantic segmentation. Different from the existing methods dedicating to the design of consistency regularization pseudo-labeling, AEL aims adaptively balance the training based the fact that pixel categories in common semantic segmentation datasets tend to be imbalanced. We introduce confidence bank to dynamically record the category-wise performance each training step, which enables us to identify the under-performing categories and adaptively tilt training towards these categories. Several technologies are proposed to make the training unbiased, namely adaptive Copy-Paste and CutMix, adaptive equalization sampling and dynamic re- weighting. Through the adaptive design, AEL outperforms the state-of of-the-art methods by a large margin on the Cityscapes and Pascal VOC benchmarks under various data partition protocols.
Acknowledgment
This work was supported by the National Key R&D Program China under grant 2017YFB 100280084 and National Natural Science Foundation of China (No. 3177 1230).
10

---

References
[1] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In 2016 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 3213-3223. IEEE Computer Society, 2016.
[2] Geoffrey French, Samuli Laine, Timo Aila, Michal Mackiewicz, and Graham D. Finlayson. Semi-supervised semantic segmentation needs strong, varied perturbations. In 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7-10, 2020. BMVA Press, 2020.
[3] Jongmok Kim, Jooyoung Jang, and Hyunwoo Park. Structured consistency loss for semi- supervised semantic segmentation. CoRR, bbs/2001.04644/ 2020.
[4] Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross-consistency training. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 12671-12681 IEEE, 2020.
[5] Zhanghan Ke, Daoye Wang, Qiong Yan, Jimmy S. J. Ren, and Rynson W. H. Lau. Dual student: Breaking the limits of the teacher in sumi-super learning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 -Novmmer 2. 2019, pages 6727-6735. IEEE, 2019.
[6] Jianlong Yuan, Yifan Liu, Chunhua Shen, Zhibin Wang, and Hao Li. A simple baseline for semi-supervised semantic segmentation with strong data augmentation. arXiv preprint arXiv:2104.07256. 2021.
[7] Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2613-2622, 2021.
[8] Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. Pseudoseg: Designing pseudo labels for semantic segmentation. arXiv preprint arXiv:2010.09713, 2020.
[9] Yves Grandvalet, Yoshua Bengio, et al. Semi-supervised learning by entropy minimization. In CAP, pages 281-296, 2005.
[10] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Sth International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[11] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1195-1204, 2017.
[12] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii Virtual adversarial training:
A regularization method for supervised and semi-supervised learning. IEEE Trans. Puttern Anal. Mach. Intell., 41 (8):119799119 2019.
[13] Dong- Lee et al. Pseudo-label: The simple and efficient semi-super learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, 2013.
[14] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.

---

[15] David Berthelot, Nicholas Carlini, lan J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 5050-5060, 2019.
[16] Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoc Le. Unsupervised data aug- mentation for consistency training. Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December
6-12, 2020, virtual, 2020.
[17] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[18] Qizhe Xie, Minh-Thang Thang Luong, Eduard H. Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seartle, WA, USA, June 13-19, 2020, pages 10684-10695. IEEE, 2020.
[19] Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson W. H. Lau. Guided collaborative training for pixel-wise semi-supervised learming. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XIII, volume 12358 of Lecture Notes in Computer Science, pages 229-445. Springer, 2020.
[20] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeuriPS 2020, December 6-12, 2020, virtual, 2020.
[21] Minsung Hyun, Jisoo Jeong, and Nojun Kwak. Class-imbalanced semi-supervised learning. CoRR, abs/2002.068155 2020.
[22] Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan L. Yuille, and Fan Yang. Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning. CoRR, abs/2102.09559, 2021.
[23] Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249- 2018.
[24] Jonathon Byrd and Zachary Chase Lipton. What is the effect of importance weighting in deep learning? In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 872-881. PMLR, 2019.
[25] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. Learning imbalanced datasets with ahbelddistrbbution-wware margin loss. In Advances in Neural Information Process- ing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1565-1576, 2019.
[26] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge J. Belongie. Class-balanced loss based on effective number samples. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 9268-9277. Computer Vision Foundation IEEE, 2019.
[27] Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 IEFF/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- - November 2, 2019, pages 6022-6031. IEEE, 2019.
12

---

[28] Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang, Jintao Li, and Jiashi Feng. Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR),
June 2020.
[29] Gyungin Shin, Weidi Xie, and Samuel Albanie. All you need are few pixels: semantic segmentation with pixelpick. CoRR, abs/2104.06394, 2021.
[30] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D. Cubuk, Quoc V. Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. CoRR, abs/2012.07177, 2020.
[31] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3356-5364, 2019.
[32] Tsung- Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2999-3007. IEEE Computer Society, 2017.
[33] Mark Everingham, Luc Van Gool, Christopher K. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis., 88(2):303-
338, 2010.
[34] Bharath Hariharan, Pablo Arbelaez, Lubomir D. Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, November 6-13, 2011, pages 991-998. IEEE Computer Society, 2011.
[35] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 633-641, 2017.
[36] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097- 1105, 2012.
[37] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018. 2018.
13


Reviews:
[{"review_id": "X_Oo7HeuF2a", "reviewer": "Reviewer_4n1x", "summary": "The paper addressed the inevitable data (and performance) imbalance across different categories in semi-supervised semantic segmentation framework by adaptively balancing the over- and under- performed classes. The proposed methods empirically showed good performance compared to other SOTA works on multiple benchmark datasets. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The originality of the work is moderate yet solid. Imbalanced performance across classes can plague the performance of semi-supervised learning models and this work proposed a practical way to alleviate this issue.  In my opinion it deserves to be seen by the community. \n\nThere are still a few questions that could be addressed in more details:\n\n1. The predictions from the underperforming class by definition are inaccurate. By biasing to the low performance class, how to balance the negative impact of the inaccurate prediction with the adaptive equalization?\n\n2. With normalized sampling probability from confidence banks, what is the exact way to sample images of a given class in adaptive CutMix and Copy-Paste. The introduction in the relevant sessions lacks details. \n\nThe clarity of the presentation though clearly has room to improve. For example, Eqn (4) - (6), not sure why the same letter N is used in one equation to represent different concepts. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No negative societal impact as far I know.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hrs"}, "extra_text": {}}, {"review_id": "VoA-st9_o5U", "reviewer": "Reviewer_XHRW", "summary": "(1) The paper proposes a novel framework for semi-supervised semantic segmentation, namely adaptive equalization learning;\n\n(2) The paper proposes two data augmentation strategies including adaptive Copy-Paste and CutMix to give more chance for under-performing categories to be copied or cut;\n\n(3) The paper presents an adaptive equalization sampling strategy to encourage pixels from under-performing categories to be sufficiently trained;\n\n(4) A simple yet effective re-weighting strategy is proposed to alleviate the noise problem in semi-supervised learning;\n\n(5) Experimental results illustrate the effectiveness of the paper.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors have presented the adaptive equalization learning for semi-supervised semantic segmentation, which aims at improving under-performing categories during training. Overall, I am satisfied with the paper and tend to accept the paper due to its novelty and strong experimental results.\n\n**Strengths:**\n\n(1) The paper is well-written and easy to follow; \n\n(2) The proposed method is novel and well-motivated;\n\n(3) The proposed augmentation strategies utilize limited labeled data in a more efficient way;\n\n(4) The proposed equalization sampling and re-weighting strategies are effective and can be applied directly to other semi-supervised semantic segmentation algorithms;\n\n(5) Experimental results on Cityscapes and PASCAL VOC 2012 are promising and ablation studies are also helpful to understand the contributions of each component of the proposed method.\n\n**Weaknesses:**\n\n(1) In adaptive CutMix, how to determine whether an image contains a certain category?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact in the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1vKckNQ8_HL", "reviewer": "Reviewer_VExi", "summary": "This paper tackles the challenging semi-supervised semantic segmentation task, with a focus on dealing with the imbalanced/long-tailed distribution. The authors propose two adaptive data augmentation strategies, one adaptive sampling schedule, and one re-weighting method to handle the long-tailed issue. The proposed method works well on two public standard benchmark datasets.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Strength\n- The proposed framework is intuitive, simple yet effective, achieving state-of-the-art performance on two benchmark datasets\n- Extensive ablation analysis to study every single proposed component\n- The proposed method not only works under low-data regimes, but can consistently improve the model performance in the high-data regime where the whole training set is available\n\n2. Weakness\n- Since this paper claims that previous paper do not take care of the imbalanced distribution issue in semi-supervised semantic segmentation while this one handles it, I think it will be better to break down the mean IoU to per-class IoU to see how the proposed method improve those tailed classes.\n- Since VOC and Cityscapes only have ~20 classes, I wonder if the proposed equalization training framework will work better in other datasets with more classes (and thus the long-tail may be more obvious), such as COCO or ADE20k. It will be great if the authors can conduct some experiments, but not mandatory.\n- I wonder if the pseudo labels (in L102) are one-hot or soft labels.\n- From my understanding, the adaptive data augmentation is an image/path-level operator to balance the distribution, and the equalization sampling is at the pixel-level. While the data augmentation applies for both the labeled and unlabeled data, why is the equalization sampling only applied on the unlabeled data but not labeled data?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper offers an adaptive equalization strategy for semantic segmentation to improve performance of underperforming or underrepresented categories.  All reviewers found positive strengths in the paper, and recommended acceptance. The reviewers rebuttal provided additional information that were compelling and confirmed that this paper is acceptable, preferably as a spotlight.
Decision: Accept (Spotlight)