Expansion and Shrinkage of Localization for Weakly-Supervised Semantic Segmentation
Authors: Jinlong Li
ZEQUN JIE
Xu Wang
xiaolin wei
Lin Ma
Venue: NeurIPS 2022 Accept (2022)
Keywords: deformable
expansion
shrinkage
weakly-supervised semantic segmentation

Abstract: Generating precise class-aware pseudo ground-truths, a.k.a, class activation maps (CAMs), is essential for Weakly-Supervised Semantic Segmentation. The original CAM method usually produces incomplete and inaccurate localization maps. To tackle with this issue, this paper proposes an Expansion and Shrinkage scheme based on the offset learning in the deformable convolution, to sequentially improve the recall and precision of the located object in the two respective stages. In the Expansion stage, an offset learning branch in a deformable convolution layer, referred to as ``expansion sampler'', seeks to sample increasingly less discriminative object regions, driven by an inverse supervision signal that maximizes image-level classification loss. The located more complete object region in the Expansion stage is then gradually narrowed down to the final object region during the Shrinkage stage. In the Shrinkage stage, the offset learning branch of another deformable convolution layer referred to as the ``shrinkage sampler'', is introduced to exclude the false positive background regions attended in the Expansion stage to improve the precision of the localization maps. We conduct various experiments on PASCAL VOC 2012 and MS COCO 2014 to well demonstrate the superiority of our method over other state-of-the-art methods for Weakly-Supervised Semantic Segmentation. The code is available at https://github.com/TyroneLi/ESOL_WSSS.

Expansion and Shrinkage of Localization for Weakly-Supervised Semantic Segmentation
Jinlong Li
Zequn Jie Xu Wang lit
Xiaolin Wei? Lin Ma 2.
1 College of Computer Science and Software Engineering, Shenzhen University, China Meituan Inc.
jinlong.szuu szu@gmail com, zequn nus@gmail com, wangxu szu edu. cn, meixiaolin02ema uan. com, forest Linma@gmail com
Abstract
Generating precise class-aware pseudo ground-truths, a.k.a, class activation maps (CAMs), essential for Seakll-SSuperiied Semantic Segmentation. The original CAM method usually produces incomplete and inaccurate localization maps. To tackle with this issue, this paper proposes an Expansion and Shrinkage scheme based on the offset learning in the deformable convolution, to sequentially improve the recall and precision of the located object in the two respective stages. In the Expansion stage, an offset learning branch in deformable convolution layer, referred to as "expansion sampler", seeks to sample increasingly less discriminative object regions, driven by an inverse supervision signal that maximizes image-level classification loss. The located more complete object region the Expansion stage is then gradually narrowed down to the final object region during the Shrinkage stage. In the Shrinkage stage, the offset learning branch of another deformable convolution layer, referred to "shrinkage sampler", introduced to exclude the false positive background regions attended in the Expansion stage to improve the precision of the localization maps. We conduct various experiments PASCAL VOC 2012 and MS COCO 2014 to well demonstrate the superiority of our method over other state-of-the-ar methods for The code is available at attttps: http::/:/ggtttubb.o//////o////os WSSS
Semantic Segmentation.
Introduction
Image Semantic Segmentation is the task of pixel-level semantic label allocation for recognizing objects in image. The development of Deep Neural Networks (DNNs) has promoted the rapid development of the semantic segmentation task 7 20 64 in recent years. However, training such Semantic Segmentation model requires large numbers of pixel-wise annotations. Preparing such a segmentation dataset needs considerable human- and resources. Recently, researchers have studied Yeakky-Supervisee Semantic Segmentation (WSSS) methods alleviate the issue of high dependence on accurate pixel-level human annotations for training semantic segmentation models under cheap supervision. Weak supervision takes the forms of image-level [2]
21 52 54 151. point-level [3), scribbles [36, (50] or bounding box [ 28 32 47 In this paper, we focus on WSSS method based on image-level labels only, because the cheapest and most popular option of weak supervision annotation which only provides information on the existence of the target object categories.
"quual contributions, 'Corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Most WSSS methods utilize class labels 2 21 52 54 55 to generate pseudo ground-truths for training a segmentation model obtained from a trained classification network with CAM [65] or Grad- CAM 44 method. However, image-level labels can not provide specific object position and boundary information for supervising the network training, resulting in that these localization maps identify only local regions of a target object that are the most discriminative ones for the classification prediction. Therefore, with the incomplete and inaccurate pseudo ground-truths, training a Fully-Supervised Semantic Segmentation network to reach a decent segmentation performance is challenging. Existing WSSS methods usually attempt to gradually seek out more less discriminative object regions starting from the very small and local discriminative regions [I 2. 61. Differ from the existing works, in this paper, we attack the partial localization issue of the CAM method with a novel deformable transformation operation. We empirically observe that the classification models can re-discover more discriminative regions when we fix a trained classifier and equip the network with more "sampling" freedom attend to other less discriminative features. This inspires to explore a proper way to improve the quality of the initial localization maps via a new training pipeline following a Mildee-and-onneeee manner, Expansion and Shrinkage, shown in Figure
The Expansion stage aims to recover the entire ob- ject much as possible, by sampling the exterior object regions beyond the most discriminative ones, to improve the recall of the located object regions. We introduce a deformable convolution layer af- ter the image-level classification backbone, whose offset learning branch serves as a sampler seeking for sampling increasingly less discriminative object regions, driven by an inverse image-level supervision signal. We call this newly embedded deformable con- volution layer "expansion sampler" (ES). During the inverse optimization process, the backbone network is frozen to provide fixed pixel-wise features obtained in the image-level classification training to be sampled by the offset learning branch in the ES. In this way, the inverse supervision target solely enforces the offset learning in the ES branch to optimize its sampling strategy to gradually attend to the less discriminative regions, given that the pixel-wise features cannot be changed. Hence, the image-level loss maximization allows the network to pay more attention to the less discriminative regions via deformation transformation achieved by ES in the inverse optimization, which are easily ignored during the normal image-level classification task.
Class Labels
PC
Reflacast
X
FC Lager
Pooado Madio
Images
PC Laver
Steinkage
Expuradon
Seed A
Roflassen
Panab-Mala
The pipeline comparison for training a WSSS model. Our contribution is to improve the quality of initial localization maps generated by the classification network with new "Expanding first then Shrinking" scheme.
Figure 1:
Having obtained the high- necall rejicn after the Expansion stage, then propose a Shrinkage stage to exclude the false positive regions and thus further enhance the precision of the located object regions. The Shrinkage stage remains the same network architecture as in the Expansion stage except that an extra deformable convolution layer, referred as "shrinkage sampler" (SS), is introduced to narrow down the object region from the high-recal one. However, we observe feature activation bias issue, i.e., the initial most discriminative parts are more highlighted in the feature map after the ES in the Expansion stage while the newly attended regions have much weaker feature activation. Such activation bias serves as prior knowledge which encourages the later shrinkage to converge to the same discriminative parts as the initial highlighted regions in the original CAM. To alleviate such issue, we introduce a feature clipping strategy after the ES in the Expansion stage of training to normalize pixel-wise feature values, allowing each target pixel to be activated evenly and have a relatively fair chance to be selected by the SS in the Shrinkage stage. Similarly, in the training of Shrinkage stage, all the layers before the SS are fixed to provide stable pixel-wise features and only the offset learning branch in the SS is updated to sample the true positive pixels, optimized by the standard image-level classification supervision.
The main contributions of this paper are summarized as follows. First, this paper proposes an Expansion and Shrinkage scheme to sequentially improve the recall and precision of the located object in the two respective stages, leading to high-quality CAM which can be used as strong pseudo ground-truth masks for WSSS. Second, both the Expansion and Shrinkage stages are realized by carefully applying deformable convolution combined with two contrary training signals. To avoid the repeated convergence to the initial discriminative parts, a feature clipping method applied to alleviate the activation bias of these regions. Third, our approach significantly improves the quality of

---

the initial localization maps, exhibiting a superior performance on the PASCAL VOC 2012 and MS COCO 2014 datasets for WSSS.
Related Work
2.1
Weakly-Supervised Semantic Segmentation
Weakly-Supervised Semantic Segmentation pipeline 21 27 with image-level labels only mostly
consists of two steps: pseudo trouhh generation and segmentation model training [T] 2
23 26
33
34
53 55], Erasure methods [19 46 [54] applied various iterative erasing strategies to
prevent the classification network from focusing only the most discriminative parts of objects by feeding the erased image or feature maps to the model. MDC [55]. layerCAM (24) and FickleNet (30 aggregated different contexts of target object by considering multiple attribution maps from different dilated convolutions or the different layers of the DCNNs. Some works utilized diverse image contexts explore cross-image semantic similarities and differences
[]6 49
CONTA
662 analyzed the
causalities among images, contexts and class labels and used intervention to remove the confounding bias in the classification network. Recently, Anti-Adv [3] utilized anti-adversarial manipulation method to expand the most discriminative regions in the initial CAMs to other non- aon--d-iimmintive regions. RIB [29] used the information bottleneck principle to interpret the partial localization issue of a trained classifier and remove the last double-sided saturation activation layer to alleviate this phenomenon, However, the generated location maps obtained by the classifier cannot reveal the entire object areas with accurate boundaries, the initial CAM seeds obtained using the methods above were further refined by a subsequent refinement network 2 g In this paper, we also follow this pipeline and our contribution is to propose new training pipeline to generate high-quality localization maps. Different from MDC (55) using multi-dilated convolutions to combine multiple contexts for better feature mining, our method utilizes deformation transformation for less discriminative feature discovery.
2.2 Deformation Modeling
We refer to deformation modeling as learning geometric transformations in 2D image space without regarding to 3D. One popular way to attack deformation modeling is to craft certain geometric invariances into networks. However, to achieve this usually needs specific designs for certain kinds of
deformation, such as offset shifts, rotation, reflection and scaling [4] TO 13 45 56) Another line of this work on deformation modeling task learns to recompose data by either semi parameterized or completely free-form sampling in image space. STN [22) learnt 2D affine transformations to construct feature alignment. Deformable Convolutions [12 66) applied learnable offset shifts for better feature learning in free-from transformations. WSSS community, applying deformation modeling is still less explored. In this paper, utilize deformation transformation to act as a feature "sampling" to re-discover other non-discriminative regions, instead of better feature representation learning.
3 Proposed Method
Weakly-Supervee Semantic Segmentation methods use given class labels to produce pixel-level
We first give Section 3.] Then, we present the whole framework of our method, Expansion and Shrinkage with Offset Learning
localization maps from classification model using CAM [65] or Grad-CAM 44
some fundamental introduction to localization maps generation with CAM 65
(ESOL), to obtain high-quality localization maps covering more complete and accurate target object regions in Section 3.2 and Section 3 respectively. We then explain how we train the final semantic segmentation model with the generated localization maps Section 3.4
3.1 Prerequisties
We first present the way to generate localization maps via the CAM (65 A class activation map of a target object focuses on the regions of an image by trained classification network for a specific category prediction. The CAM is based on DCNN with global average pooling (GAP) before its final classification layer, which trained by a sigmoid cross-entropy loss function, formulated as follows:
3

---

Defertablc Consuiution Layer
CAM
DCN
offect
Felip
Fllp Up
Posture Clipping
Operation
DCN
Cily Op
Clessiitonion Loss
CAM
Class Artivationn Maps
f(.08))
Feruard Computation
Compatation
Expansion
Detsch
carv
offer
aftirt
CAM
oflsct licld
DCN Op
Clip Op
2N
DCN Up
Chediicatton Loss
TG.0,)
Arn Lane
W2
Shrinkage
Figure 2: Our proposed Expansion and Shrinkage training pipeline. The Expansion scheme consists of feature extractor f(;, ff), deformable convolution layer, hand-craft feature clipping operation and classifier layer. A loss maximization training is implemented to enable the offset learning in the ES to attend to other res--dscriminntive regions. For Shrinkage scheme, extra deformable convolution layer is introduced to exclude the false positive regions with loss minimization optimization, including classification loss and an area loss function, respectively. The Expansion and Shrinkage are optimized via contrary training signals to sequentially improve the recall and precision the initial localization maps.
C(0.y) = - _ vlil log((1 + ----11)--)) + (1 vlil) log(
(1)
1 1+ [[1
where is the total number of training classes, represents the 7th training sample the mini-batch, y|i] is the ground-truth label of 7th class with the value of either 0 or I while ūit] is the model prediction. The localization maps is realized by considering the class-specific contribution of each channel of the last feature map, before the GAP layer, to the final classification prediction. Given trained classifier network parameterized by f = {0y, w) where f(: fy) is the feature extractor, and denotes the weight of the final classification layer. For some class the localization map is then computed from an input image x as follows:
W fx; CAM(x; w) = max w (x:
where max(-) is the maximization over the spatial locations for normalization. The above method can only locate the most discriminative regions and fails in locating other less- duss-discrmmintive regions that are semantically meaningful as well. In the following work, we elaborate our method by presenting a new training pipeline to capture high-quality object localization maps.
3.2 Expansion
As mentioned above, the localization maps generated by a commonly trained classifier usually struggle with the partial localization issue of the target objects, since the image-level labels cannot provide detailed position or boundary information. To alleviate this issue, we devise a new training improve
pipeline, Expansion, firstly recover the entire object regions much as possible the recall of the located object regions. Then, we further introduce another training scheme, Shrinkage, exclude false positive regions, e.g., background regions, to enhance the precision of the located regions.
A commonly trained classifier usually considers only the local regions that make the most contribu- tions to final classification prediction, resulting in incomplete and inaccurate pseudo labels. Differ from other works, we first enforce the network seek out the entire target object regions via proposed Expansion scheme with an offset learning branch a deformable convolution layer, as shown the top of Figure
trained classifier utilized to prepare our Expansion scheme firstly providing the regular convolu- tional weights in a deformable convolution layer, that can capture the most discriminative feature

---

(b)
(d)
Figure 3: Feature visualizations for well demonstration of the propesed Expansion scheme. Examples of input images. (b) Examples of feature visualization of from trained classifier. (c) Examples of feature visualization of Fas from our Expansion stage. (d) Plot Recall values of pre-traine pre-traieed classifier v.s. our Expansion stage to demonstrate the generated high-recall results, in terms of the foreground object Recall. Red boxes point out the difference hetween the trained classifier and Expansion while yellow boxes point out the negative positive regions (e.g., background).
activation, e.g., conv or conv 2. Given an input image x, the classification network 0 (0,, w tends to locate the coarse object regions. For a specific convolution layer, cone_ __ with trainable weights W1, the output feature maps F1 is computed for each location Po as follows:
(Po)
w..) x(Po + Pa),
(3)
P.ER
where defines a specific kernel size with a specific dilation size (e.g., 1) and Pa enumerates the locations in R. For Expansion scheme, embedded deformable convolution layer is introduced after the feature extractor (:) The new feature maps Fi is then computed as:
F(()) = Σ wi(Pn) x(Po + Pn Apın),
where the regular grid R is augmented with learnable offset fields 1,, NJ, N denotes the number of offset points, e.g., x 1 or 3 3. The learned Apin obtained from the offset learning branch in the deformable convolution layer further acts as "cxpansion sampler" to sample the exterior object regions gradually beyond the most discriminative ones. This training scheme utilizes image- level loss maximization to enforce the network seek for increasingly less-discimiinative object regions, given unchanged pixel-wise features that is implemented by detaching the loss back- sockprropagation to the backbone. This can be validated by our experimental results shown in Figure Thus, the training loss function Lexpansion utilized in Expansion scheme becomes:
Lexpansion -xL(),))
(5)
where a (denoting positive numbers) controls the loss weight for updating and C(j, y) the corre- sponding multi-label classification loss function mentioned in Eq T
3.3 Shrinkage
After the Expansion stage, though most possible target object regions are sampled, the noisy back- grounds are also included. This would inevitably cause imprecise localization maps generation, which results in poor quality pseudo ground-truths and hampers the final segmentation performance. To enhance the precision of such high-recall regions, Shrinkage scheme proposed to exclude the false positive regions of the localization maps. The Shrinkage stage remains the same network architecture as in Expansion stage, except that an extra deformable convolution layer is implemented narrow down the high-recall regions. Specifically. the model weights obtained from the Expansion are used to initialize the Shrinkage model and a loss minimization is adopted to train the network, including multi-label classification loss and an area loss. Area regularization is adopted to constraint the size of the localization maps to ensure that the irrelevant backgrounds are excluded in the localization map Pk. which is also studied in [58 59
5

---

1 C
Lahrinkage = x) + pharea:
Larea
Σ Se:
(6)
where S² = HW
HW hh=1 Ew= Pa (h, w), C' the total class category numbers of the dataset, H and W denotes the height and width of the localization maps, respectively.
We empirically observe that the feature activation bias issue obvious, the initial most discriminative parts got more highlighted after ES in the Expansion stage while the newly attended ones have weaker feature activation. This activation bias would encourage the later Shrinkage to seek for the same discriminative parts the initial activation parts, even more local. To address such issue, feature clipping strategy is then proposed after the ES in the Expansion stage to normalize pixel-wise features, providing relatively fair chances for the pixels to be selected by SS in the Shrinkage stage. The feature clipping strategy is formulated as follows:
{ [mmx((x); 2 fmax(x)
Ferst (xx)
-x,,
others
where x; is the input feature maps over the spatial dimension, max(-) is applied obtain the maximal values along the spatial dimension as well.
We present examples of PASCAL VOC 2012 14) training images for better demonstration shown in Figure [ During the Expansion stage, the image-level loss maximization enforces the offset learning branch attend other ress-discriminatii regions, indicating that the ES indeed serves sampler locating the entire object regions, thanks to the deformation transformation the deformable convolution layer. For example, the body legs of the activated by our Expansion scheme, and the overall recall of foregrounds is significantly improved among diverse hard-threshold settings. Then, the included noisy background regions excluded by the SS sampler in the Shrinkage stage, gradually sampling the true positive foreground regions and finally providing high-quality pseudo labels.
cow shown in Figure 3 is successfully
3.4 Pseudo Ground-truth Generation
The final localization map M is generated by our final Shrinkage trained model via CAM 665 method. Since a CAM is computed from high- level small size intermediate feature maps pro- duced by a classification network, it need to be up-sampled match the size of the original im- age. Thus, it tends to localize the most discrim- inative target object regions coarsely and cannot cover the entire regions with exact boundary.
Many WSSS methods 30 51 53 62) pro- duce pseudo ground ground-truths by extending their initial CAM seeds using another seed refine- ment methods [1 2 3) Similarly, we obtain using IRN [
Figure 4: Examples of the final segmentation predictions on PASCAL VOC 2012 validation. The top row, middle and bottom row denotes the input images, corre- sponding ground truth and our final segmentation model predictions, respectively.
our final pseudo ground-trutie state-of-fe the-art art refinement method, to refine the coarse map M for generating better segmenta- tion model supervision. Then semantic segmentation model is trained using the generated pseudo labels following Fully-SSpervised manner and Figure (illusttrttes the final segmentation results on VOC2012 val set.
4 Experiments
4.1 Experimental Setup
Dataset and Evaluation metric: Experiments are conducted on two publicly available datasets, PASCAL VOC 2012 [14 and MS COCO 2014 (37]. The Pascal VOC 2012 dataset contains 20 foreground categories and the background. It has three sets, the training, validation, and test set, each

---

Refinement Method
PASCAL VOC
MS COCO
Method
Seed
CRF
Mask
Seed
Mask
PSA CVPR '18
48.0
61.0
Mixup-CAM
5
50.1
61.9
BMVC 20
CDA ICCV '21
48
48.9
57.5
63.3
PSA
SC-CAM CVPR '20
50.9
55.3
63.4
ESOL (Ours)
53.6
61.4
66.4
IRN CVPR '19
48.8
53.7
66.3
33.5F
42.9
MBMNet ACMMM 20
39
50.2
66.8
BES ECCV '20
8
50.4
67.2
IRN I
CONTA NeurIPS 20
62
48.8
67.9
28.7
35.2
CDA ICCV '21
48
50.8
58.4
67.7
ESOL (Ours)
53.6
61.4
68.7
35.7
44.6
Table I: Comparison of the initial localization maps (Seed), the seed with CRF (CRF), and the pscudo ground- truths mask (Mask) on PASCAL VOC 2012 and MS COCO 2014 training images, terms mloU (%).
denotes the results reported by CONTA 622 and denotes results obtained by us.
(a) Image
(b) CAM
(c) DT
(d) Expanding
(e) Shrinkage
(f) GT
Figure 5: Examples of localization maps on PASCAL VOC 2012 training images. (a) Input Images. (b) Original baseline CAMs. (c) DT denotes Expansion training without feature clipping strategy. (d) Expansion results. (e)
Shrinkage results. (I) Ground Truth.
containing 1464, 1449 and 1456 images, respectively, Following most previous works 30
51
62). we also adopt the augmented training set [17] to yield totally 10582 training images. The MS COCO 2014 dataset has 80 foreground categories, including approximately 82K training images and 4K validation images. We evaluate our method on 1449 validation images and 1456 test images from the PASCAL VOC 2012 dataset and on 40504 validation images from the MS COCO 2014 datasets. The mean intersection--ver-une
(mIoU) [4 is used the evaluation metric.
Implementation Details: We implement CAM 65 by following the procedure from Ahn al. M implemented with the PyTorch framework
43 on 12G Nvidia XP Graphic Cards. We adopt the ResNet-50 [18 as backbone for the classification model. To prepare the regular convolutional weights for the deformable convolution layers, opt to follow the common baseline settings to train classifier for our method, except that we add two additional convolution layers would be used to perform the deformable transformation in our proposed Expansion and Shrinkage, respectively. For Expansion stage, we train the network 6610 iterations for PASCAL VOC 2012 and 51730 iterations for MS COCO 2014. To carefully train the model with a loss maximization, set relatively small learning rate, 0.01 and 0.001 for PASCAL VOC 2012 and MS COCO 2014, while the controlling parameter is set to 0.001. Noting that the loss gradients not allowed to
X that
7

---

back proo-aatee to the backbone, as we want to provide pixel-wise unchanged feature maps for ES and SS. For Shrinkage stage, the network is initialized from the Expansion model weights. The learning rate is set to be 0.1 and 0.02, and the training iteration is set to be 6610 and 51730 for PASCAL. VOC 2012 and MS COCO 2014, respectively. The threshold value of the hand-craft feature clipping strategy is 0.15. The "y and j are both set to be 1.0. To generate reliable initial localization maps, the scale ratio of multi-scale CAM is {0.5, 1.0, 1.5, 2.0}. During testing, DenceCRF [28) is used as post-processigg to refine the generated localization maps. For the final semantic segmentation, we use the Py Torch implementation of DeepLab-v2 1aaee----eeee-eO-- provided by 42
4.1.1 Quality of the initial localization maps and pseudo ground-truths
PASCAL VOC 2012 dataset: In Table the mloU values are used to estimate the quality of initial localization maps (denoted seed) and pseudo ground-truths masks produced by our method, and other recent WSSS techniques. Following common practice [1 2 6 531, we perform range of hard-threshold values to distinguish the positive and negative regions in localization maps M to determine the best initial seeds result. Our initial seeds improves 5.2% mloU over the original CAM seeds (48.4% to 53.6%). a baseline for comparison, and outperform those simultaneously from the other methods. Noting that our initial seeds are superior to those of CDA [48] or SC-CAM [6]. which applied complicated context decoupling augmentation on the network training or adopted sub-category exploration to enhance the feature representation via an iterative clustering method.
4.2
Weakly-Supervised Semantic Segmentation
After obtaining the initial seeds based on CAM 655. we then apply the Conditional Random Field (CRF) (28) for pixel-level refinement on the re- sults from the method proposed by SC-
Method
Backbone
mIoU (%)
ADL TPAMI '20
VGG16
30.8
CONTA NeunPS '20
62
ResNet50
33.4
SEAM CVPR 20
53
Wider-ResNet338
32.8
CAM CDCA 488 IRN (1, and our
IRN CVPR '19
ResNet101
41.4
method in Table 1 We can observe that applying CRF improves all the initial seeds over 5% mloU. When the seeds generated by our method is then post- processed with CRF, it obtains 7.8% mloU better than the original CAM (53.6% to 61.4%), and consequently surpasses all the recent competitive methods. A seed refinement is applied to generate the pseudo ground- truth masks with other methods. For a fair comparison,
ESOL (Ours)
ResNet101
42.6
Table 2: Comparison of semantic segmentation on MS COCO validation images.
compute our pscudo ground-truth. masks using both seed refinements, PSA [2 or IRN [T. Table illustraaee the masks results from method yield 66.4% mIoU with PSA 2 and 68.7% mIoU with IRN [1. respectively.
Figure 5 visualizes initial localization maps adopted different components or training phases for the PASCAL VOC 2012 dataset. And the visualizations demonstrate the effectiveness of the proposed Expansion and Shrinkage approach, sequentially balancing the recall and precision of the initial localization maps. More examples are shown in the Appendix.
4.2.1 Performance of the Weakly-Supervised Semantic Segmentation
PASCAL VOC 2012 dataset: Table 3 shows the segmentation performance (mloU) on PASCAL VOC 2012 validation and test set. We illustrate the results predicted by our method and other recently proposed WSSS methods, which use either bounding box image-level labels. All the segmentation results in Table were implemented by a ResNet-based backbone [8]. Our proposed method achieves 69.9% and 69.3% mloU values for the validation and test set on the PASCAL VOC 2012 dataset, outperforming all the WSSS methods that utilize image-level class labels only.
In particular, our method surpasses CONTA [62] state-of-the-art WSSS competitors recently, obtaining 66. 1% mloU. CONTA adopted SEAM [53] which is applied with WiderResNet-based [57] backbone that is known to be more powerful than IRN M based on ResNet-based. When it was implemented with IRN I for a fair comparison with our method, its segmentation performance got only 63.5%, which is inferior to ours by 6.4% mloU.
Recently, saliency map cues are introduced to supervise the network training for better localization performance, since the offline saliency maps provide detail foreground boundary priors. In Table we also compare our method with other methods using extra saliency maps. We combine our final pseudo ground-truth masks with saliency maps produced by Li et al. (35) or Liu et al. (38). We can see that
8

---

Method
val
test
Supervision: Bounding box labels
Method
Sup.
val
test
BoxSup 1CCV '15
[1
62.0
64.6
SeeNet NeurPs
19
63.1
62.8
Song CVPR
70.2
FickleNet CVPR 19
30
64,9
65.3
BBAM CVPR
32
73.7
73.7
CIAN AAAI 22
[6
64.3
65.3
Zhang et al. ECCV 20
66.6
66.7
Supervision: Image class labels
Fan al. BCCV
15
S
67.2
66.7
IRN CVPR '19
63.5
64.8
Sun et al. ECCV '20
66.2
66.9
SEAM CVPR
53
64.5
65.7
LIID TFAMI 20
40
66.5
67.5
BES BCCV 20
65.7
66.6
Chang el al.
CVPR
66.1
65.9
Yao el al. CVPR '21 60
Li et al. AAAI 21
R5
S 68.2
68.3
68.5
68.5
RRM AAAI 61
66.3
66.5
ESOL (Ours)
S
71.1
70.4
CONTA NeudPP '20
62
66.1
66.7
ESOL (Ours)
69.9
69.3
Table 4: Comparison of semantic segmentation perfor- PASCAL. VOC 2012 validation and test set using saliency maps, marked as S.
Table 3: Comparison of semantic segmentation performance on PASCAL VOC 2012 validation and test set.
Method
DT.
FL
DT,
Carea
seed mloU (%)
Baseline
48.4
Expansion w/o (DT, + FL) Shrinkage
48.5
Expansion w/o DT, + Shrinkage
v 49.1
Expansion w/o FL + Shrinkage
d
V
51.8
Expansion + Shrinkage w/o (DT, + Carea)
50.1
Expansion Shrinkage w/o DT,
/
/ /
/
50.9
Expansion + Shrinkage w/o Larea
52.8
Expansion + Shrinkage
53.6
Table 6: Ablation studies with some components removed in approach. DT, and DT, denote the offset learning branch the deformable convolution layers in Expansion and Shrinkage, respectively. FL is the feature clipping operation
Larea denotes area loss regularization. The DT,, and FT. enforce the network pay attention to suspicious object regions as much as possible in Expansion, while the DT, and Lavea enforce the network to boost the precision of the results in Shrinkage to generate high-quality localization maps.
our method achieves 71.1% and 70.4% mIoU values for PASCAL VOC 2012 validation and set, respectively, consistently outperforming all other methods under the salient object supervision.
MS COCO 2014 dataset: Table illustrates the segmentation performance on MS COCO 2014 compared with other methods. Our method achieves 42.6% in terms of the mloU values on validation set, surpassing 1.2% over the IRN [. also regarded as our baseline and out- performing the other recent competitive meth- ods 19 53 62 by large margin. We further compare the relative improvements for comparison: CONTA reaches a 0.8% mloU improvement compared with IRN (32.6% to 33,4%), while our method achieves 1.2% mloU improvement (41.4% to 42.6%).
Method
Benchmark GPUs Time (h)
AdvCAM 31
VOC2012
1
106
RIB 29
VOC2012
1
27
ESOL (Ours)
VOC2012
10
Table Comparison of overall pseudo-lahel generation time.
4.3 Ablation Studies
In this section, we conduct various ablation experiments on PASCAL VOC 2012 dataset to validate the effectiveness of each component or training scheme of our method.
4.3.1 Expansion Training Analysis
Loss Maximization Controller a: To analyze the influence of the loss maximization controller on
Expanding training sensitivity, we conduct range from 0.001 to 0. for a as shown Figure (a). We found that too small or large values degrade to attend on backgrounds extremely, showing lower foreground recall ratio. We choose appropriate value: a 0.01, which balances the recall and precision of the localization maps. Visualizations shown in the Appendix.

---

CAM Geakk (h
CAN Soo.k
Forgurad
dec Jes
Yee-ane Ine
Eoss
Pypergaaze Senslivity
Figure 6: Ablation Studies. (a) a sensitivity analysis. (b) 3 sensitivity analysis. (c) losses combinations. (d) and y balance analysis.
In Expansion stage, we tend to provide high-recall foregrounds with relatively fair chance to be sampled by Shrinkage. The feature clipping strategy is introduced after the ES and we study the impact of hand-craft threshold settings in (ig
Feature Clipping Strategy Effect::
4.3.2 Shrinkage Training Analysis
Although Expansion training brings high-recall foreground regions, backgrounds cannot be ignored. A classification loss and an area loss are used to optimize the network select true foreground pixels via a loss minimization. We provide the ablative study Figure (c) to demonstrate the impact of each one and find that both of them are useful for the network training, constraint the size of the localization maps to ensure that the irrelevant backgrounds are excluded in the localization maps Pk.
Impact of the Loss Functions:
Loss Minimization Controller y and μ:: The sensitivity of these two shown in Figure 6 (d). It is observed that the performances of our approach are stable with the variation of (from 0.1 5) and p (from 0.01 to 1.5), i.e., our method sensitive such two hyper-parameters. In our experiments, the default values of y and are 1.0 and 1.0 simply.
are performed
4.3.3 Training and Inference Time Comparison
In this part, we make comparisons with recent SOTAs, AdvCAM and RIB 29 that perform single step to obtain the localization maps. From the Table we can see that the overall generated pseudo-label time of the AdyCAM [T and RIB (29 are much longer than ours which demonstrates that our method is more applicable in practice.
We further validate the effectiveness of each component of methods by removing some compo- nents to perform ablative studies in T We can see that each proposed component, including deformable transformation, feature clipping operation and, training losses, enables the overall training pipeline to generate more complete and accurate initial localization maps.
Conclusion
In this paper, we explore a deformation transformation operation address the major challenge in weakly-supervised semantic segmentation with image-level class labels. A novel training pipeline for the WSSS task, Expansion and Shrinkage, proposed first recover the entire target object regions as much as possible while the network is driven by an inverse image-level supervision. Then, we apply feature Clipping operation to provide even high-recall regions for Shrinkage to sample high-precision regions, while the network is optimized by two loss functions, classification loss and area loss. Our method significantly improves the quality of the initial localization maps, exhibiting a competitive performance on the PASCAL VOC 2012 and MS COCO 2014 datasets.
Societal implications: This work may have the following societal impacts. Semantic Segmentation without the reliance on pixel-level annotation will save resources for research and commercial development. is particularly useful in online network with large-scale image-tags. However, the final segmentation performance is still inferior to the fully-supervised ones and the annotations- support companies need consider another business pattern.
Acknowledgments
This work supported in part by the National Natural Science Foundation of China (Grant 61871270). in part by the Shenzhen Natural Science Foundation under Grants JCYJ202001091 (0410133 and 202008121 10035000..
10

---

References
[1] Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2209-2218. 2019.
[2] Jiwoon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pages 4991-4990, 2018.
[3] Amy Bearman, Olga Russakovsky. Vittorio Ferrari, and Li Fei-Fei. What's the point: Semantic segmen- tation with point supervision. In European conference on computer vision, pages Springer. 2016.
[4] Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 35(8):1872-1886, 2013.
[5] Yu- Tinging Chang, Qiaosong Wang, Wei-Chih Hung. Robinson Piramuthu, Yi- Tsai, and Ming- Husunn Yang. Mixup-cam: superv-s semantic segmentation uncertainty regularization. In The British Machine Vision Conference, 2020.
[6] Yu-Ting Chang, Qiaosong Wang. Wei-Chih Hung, Robinson Piramuthu, Yi-Hsuan Tsai, and Ming- -Hsuan Yang. -uueev--ueee semantic segmentation sub-category exploration. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8991-9000, 2020.
[7] Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplah: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
[8] Liyi Chen, Weiwei Wu, Chenchen Fu, Xiao Han, and Yuntao Zhang. Weakly supervised semantic segmentation with boundary exploration. ECCV, 2020.
[9] Junsuk Choe, Scungho Lee, and Hyunjung Shim. Attention-based dropout layer for weakly supervised single object localization and semantic segmentation. IEEE transactions on pattern analysis and machine intelligence, 43(12):2256 4271, 2020.
[10] Taco Cohen and Max Welling, Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999. PMLR, 2016.
[11] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In Proceedings of the IEEF international conference on computer vision, pages 1635-1643, 2015.
[12] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 764-773, 2017.
[13] Carlos Esteves, Christine Allen-Blanchette, Xiaowei Zhou, and Kostas Daniilidis. Polar transformer networks. arXiv preprint arXiv:1709.01889 2017.
[14] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338 2010.
[15] Junsong Fan, Zhaoxiang Zhang, and Tieniu Tan. Employing multi-estimations for weakly-superrised semantic segmentation. In European Conference on Computer Vision 2020.
[16] Junsong Fan, Zhaoxiang Zhang, Tieniu Tan, Chunfeng Song, and Jun Xiao. Cian: Cross-image affinity net for weakly supervised semantic segmentation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 10762 10769, 2020.
[17] Bharath Hariharan, Pablo Arbeláez. Lubomir Bourdev, Subhransu Maji. and Jitendra Malik. Semantic contours from inverse detectors. In 2011 International Conference on Computer Vision, pages 991-998. IEEE, 2011.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778. 2016.
[19] Qibin Hou, Tao Jiang, Yunchao Wei, and Ming-Ming Cheng. Self-erasing network for integral object attention. In Advances in Neural Information Processing Systems, pages 549-559 2018.
11

---

[20] Zilong Huang, Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu, Conet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vixion, pages 603-612, 2019.
[21] Zilong Huang, Xinggang Wang, Jiasi Wang, Wenyu Liu, and Jingdong Wang. Weakly- semantic segmentation network with deep seeded region growing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7014-7023, 2018.
[22] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
[23] Peng-Tao Jiang, Qibin Hou, Yang Can, Ming-Ming Cheng, Yunchao Wei, and Hong-Kai Xiong. Integral object mining via online attention accumulation. In Proceedings of the IEFF International Conference on Computer Vision, pages 2070-2079, 2019.
[24] Peng- Tao Jiang, Chang Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. Layercam: Exploring hierarchical class activation maps. IEEE Transactions on Image Processing, 2021.
[25] Angjoo Kanazawa, Abhishek Sharma, and David Jacohs. Locally scalc-invariant convolutional neural networks. arXiv preprint arXiv:1412.5104, 2014.
[26] Beomyoung Kim, Sangeun Han, and Junmo Kim. Discriminative region suppression for weakly-supervine semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1754-1761, 2021.
[27] Alexander Kolesnikov and Christoph H Lampert. Seed, expand and constrain: Three principles for weakly- supervised image segmentation. In European conference on computer vision, pages 695-711. Springer, 2016.
[28] Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In Advances in Neural Information Processing Systems, pages 109-117, 2011.
[29] Jungheom Lee, Jooyoung Choi, Jisoo Mok, and Sungroh Yoon. Reducing information bottleneck for weakly supervised semantic segmentation. Advances Neural Information Processing Systems, 34, 2021.
[30] Jungheom Lee, Funji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon. Ficklenet: Weakly and semi- supervised semantic image segmentation using stochastic inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5267-5276, 2019.
[31] Jungbeom Lee, Eunji Kim, and Sungroh Yoon. Anti-adversrrially manipulated attributions for weakly and semi-superviee supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4071 108000 2021.
[32] Jungheom Lee, Jihun Yi, Chachun Shin, and Sungroh Yoon. Bbam: Bounding box attribution map for weakly supervised semantic and instance segmentation. In Proceedings the IEEE/CVF conference on computer vision and pattem recognition, pages 2643-2652, 2021.
[33] Seungho Lee, Minhyun Lee, Jongwuk Lee, and Hyunjung Shim. Railroad is not a train: Saliency as pseudo-pixel supervision for weakly supervised semantic segmentation. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 5495-5505, 2021.
[34] Jinlong Li, Zequn Jie, Xu Wang, Yu Zhou, Xiaolin Wei, and Lin Ma. Weakly supervised semantic segmentation via progressive patch learning, IEEE Transactions Multimedia, 2022.
[35] Xucyi Li, Tianfei Zhou, Jianwu Li, Yi Zhou, and Zhaoxiang Zhang. Group-wise semantic mining for weakly supervised semantic segmentation. In AAAI, 2021.
[36] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3159- 3167, 2016.
[37] Tsung-Yr Yi Lin, Michacl Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[38] Jiang-Jian Jiang Liu, Qibin Hou, Ming-Minn Cheng, Jiashi Feng, and Jianmin Jiang. A simple pooling- based design for real- time salient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3917-3926, 2019.
12

---

[39] Weide Liu, Chi Zhang, Guosheng Lin, Tzu-Yi Hung, and Chunyan Miao. Weakly supervised segmentation with maximum bipartite graph matching. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2085-2094, 2020.
[40] Yun Liu, Yu-Huan Wu, Pei-Song Wen, Yu-Jun Shi. Yu Qiu, and Ming-Minn -Ming Cheng. Leveraging instance. image-and datasct-level information for weakly supervised instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[41] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[42] Kazuto Nakashima. Deeplab with pytorch.
[43] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Zeming Lin, Alhan Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[44] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[45] Laurent Sifre and Stéphane Mallat. Rotation, scaling and deformation invariant scattering for texture discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1233-1240, 2013.
[46] Krishna Kumar Singh and Yong Jac Lec. Hide-and-scek: Forcing a network be meticulous for weakly- supervised object and action localization. In 2017 IEEE international conference on computer vision (ICCV), pages 3544-3553, IEEE, 2017.
[47] Chunfeng Song, Yan Huang. Wanli Ouyang, and Liang Wang. Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3 3136-3145, 2019.
[48] Yukun Su, Ruizhou Sun, Guosheng Lin, and Qingyao Wu. Context decoupling augmentation for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[49] Guolei Sun, Wenguan Wang, Jifeng Dai, and Lue Van Gool. Mining cross-image semantics for weakly supervised semantic segmentation. In Luropean Conference on Computer Vision, 2020.
[50] Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, and Christopher Schroers. Normalized cut loss for weakly-supervised can segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1818-1827, 2018
[51] L. Subelj and M. Bajec Ubiquitousness of link-density and link-pattem communities in real-world networks. Eur. Phys. B., 85:1 11, 2012.
[52] Xiang Wang, Shaodi You, Xi Li, and Huimin Ma. Weakly- semantic segmentation by iteratively mining common object fcatures. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1354-1362, 2018
[53] Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. In Proceddings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 12275-12284, 2020.
[54] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Minn Cheng, Yao Zhao, and Shuicheng Yan. Object region mining with adversarial crasing: A simple classification to semantic segmentation approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1568 -176,, 2017.
[55] Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jic, Jiashi Feng, and Thomas Huang. Revisiting dilated convolution: A simple approach for wcd semi- supi-vuuee semantic segmentation. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 7268-7271 2018
[56] Daniel E Worrall, Stephan Garbin, Daniyar Turmukhambetov, and Gabriel Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5028-5037, 2017.
[57] Zifeng Wu, Chunhua Shen, and Anton Van Den Hengel. Wider or deeper: Revisiting the resnet model for visual recognition. Pattern Recognition, 90:119 1331 2019.
13

---

[58] Jinheng Xie. Xianxu Hou. Kai Ye, and Linlin Shen. Clims: Cross language image matching for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 4483-4492, 2022.
[59] Jinheng Xie, Cheng Luo, Xiangping Zhu, Ziqi Jin, Weizeng Lu, and Linlin Shen. Online refinement of low-level feature based activation map for weakly supervised object localization. Proceedings of IEEE/CVF International Conference on Computer Vision, pages 121,141,222 2021.
[60] Yazhou Yao, Tao Chen, Guosen Xie, Chuanyi Zhang, Fumin Shen, Qi Wu, Zhenmin Tang, and Jian Zhang. Non-salient region object mining for weakly supervised semantic segmentation. In CVPR, 2021.
[61] Bingfeng Zhang, Jimin Xiao, Yunchao Wei, Mingjie Sun, and Kaizhu Huang. Reliability does matter: end-to-end weakly supervised semantic segmentation approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1 12772, 2020.
[62] Dong Zhang, Hanwang Zhang, Jinhui Tang. Xiansheng Hua, and Qianru Sun. Causal intervention for weakly-supervisee semantic segmentation. In Advances in Neural Information Processing Systems, 2020.
[63] Tianyi Zhang, Guosheng Lin, Weide Liu, Jianfei Cai, and Alex Kot. Splitting vs. merging: Mining object regions with discrepancy and intersection loss for weakly supervised semantic segmentation. In European Conference Computer Vision, 2020.
[64] Hengshuang Zhao, Jianping Shi. Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.
[65] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929 2016.
[66] Xizhou Zhu Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9308-9316, 2019.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] We believe our work will benefit society by providing another perspective to attack the partial localization issue on Weakly-Supervised Semantic Segmentation task with image-level labels only.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [N/A] We plan to release the code of our method upon acceptance.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] Please refer Section 4.1
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer to Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] We have followed terms for using the PASCAL VOC 2012 and COCO 2014 Dataset for pure academic research purposes.
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "sfhAbJwiEFx", "reviewer": "Reviewer_k5T6", "summary": "This paper proposes a method for weakly-supervised segmentation.\n\nThe authors refine localization maps from CAM with expansion and shrinkage steps.\n\nThe authors expand discriminative regions utilizing the deformable convolution layers in the expansion step.\n\nThe authors shrink expanded regions by minimizing the classification loss and the norm of localization maps via additional deformable convolution layers in the shrinkage step.\n\nThe authors demonstrate that refined localization maps boost the performance of weakly supervised segmentation.", "questions": "- I am not sure about the advantage of the proposed two-phase approach comparing one phase approach from this paper. Do the authors explain the advantage of this approach?\n\n- DRS[3] is also a method for refinement of localization maps, and this method reports better scores than the proposed method. What is the advantage of the proposed method comparing DRS?\n\n- The format of Table 1 in this paper seems to be similar to the format of Table 1 in the paper[4]. Why do the authors not compare the proposed method with AdvCAM[5] and RIB[4] in Table1?\n\n- Why do the authors not list the results of RIB[4] in the Tables?\n\n[3] Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation, AAAI 2021\n\n[4] Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation, NIPS 2021\n\n[5] Anti-adversarially manipulated attributions for weakly and semi-supervised semantic segmentation, CVPR 2021", "limitations": "I do not have concerns about limitations and social impacts for this paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros)\n- The final semantic segmentation performance is comparable with state-of-the-art methods.\n\n- The two-phase refinement of localization maps is a novel attempt, and the authors demonstrate that the proposed two-phase steps can achieve high performance.\n\nCons)\n- The motivation of the proposed two-phase refinement approach for localization maps is unclear.\n\n- Important references[1][2] are missing. They report better scores on the benchmarks than the scores in this paper.\n\n- The effectiveness of deformable convolution layers is unclear.\n\n[1] Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation, AAAI 2021\n\n[2] Railroad is not a train: Saliency as pseudo-pixel supervision for weakly supervised semantic segmentation, CVPR 2021\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cQYGEeD7GN", "reviewer": "Reviewer_ezBz", "summary": "This manuscript proposes a expansion and shrinkage method for weakly-supervised semantic segmentation. As the initial CAM map only focuses on discriminative regions, the authors first present the expansion module for enlarging the localization, then the shrinkage method is introduced for removing false positives. A feature clipping is also used for better shrinkage.", "questions": "Some symbols are not defined, thus is confusing, for example, Equ(5)\n\nFigure 6, it is better for the authors present results with some components removed in a new table to better demonstrate their effects.", "limitations": "Not applicable", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :\n\nThis paper is well written and easy to follow.\n\nAs the initial CAM map only focuses on discriminative regions, the authors first present the expansion module for enlarging the localization with a deformable convolution layer. In the second stage, another deformable convolution layer is used to shrink the expanded regions. To avoid only removing the expanded regions, feature clipping is present for being shrunk with relatively fair chances.\n\nThe experimental results are good compared to state-of-the-art methods.\n\nAblation studies show the effectiveness of each component.\n\nWeaknesses:\n\nSome symbols are not defined, thus is confusing, for example, Equ(5)\n\nFigure 6, it is better for the authors present results with some components removed in a new table to better demonstrate their effects.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NilvGM-1Ab4", "reviewer": "Reviewer_texX", "summary": "This paper presents a new method for computing better class activation maps (CAMs), which have been used as crucial evidence for weakly-supervised semantic segmentation. A well-known drawback of CAM and its variants lies in that they often highlight only the small discriminative regions of an object. To alleviate such an issue, an “Expansion and Shrinkage” method is proposed, which first seeks out as much as the more positive foreground of the target object in Expansion, and then excludes the wrongly included negative regions in Shrinkage, e.g., backgrounds. In short, the proposed method embraces the offset learning of deformable convolution to introduce an expansion stage to sample out more less discriminative regions with an inverse classification loss and a shrinkage stage to filter out background regions with two losses, respectively. As a result, more object features are able to be activated so as to alleviate the partial localization issue of the CAM. The method enhances the quality of the original CAM noticeably and consequently improves the performance of a weakly-supervised semantic segmentation method relying on CAMs.\n\nThe main idea of this work is novel and interesting, and indeed useful to alleviate the well-known limitation of the CAM and its variants. Moreover, the efficacy of the proposed method has been demonstrated by extensive experiments and various ablation studies are conducted to validate the effectiveness of the proposed expansion and shrinkage stage training. This paper sheds new light on addressing the limitation of the CAM for future works with an interesting perspective.\n", "questions": "(1)Although the authors conduct various ablation studies to investigate the effectiveness of their proposed method, this reviewer still wonders how to choose the training iteration or corresponding hyper-parameters to train the classification network in the first stage, the expansion stage.\n\n(2)The authors lack consideration if there were any possible fusion strategies to combine the generated location maps between the proposed expansion and shrinkage stage to make this paper more convincing.\n\n(3)More insightful investigations between the proposed two stages are recommended to demonstrate, especially the changing of precision and recall during such two training stages.\n\n", "limitations": "Limitations of this work are not provided in the paper, this reviewer could not find any potential negative social impact of this work. I recommend offering a discussion on the limitations in the paper, which need to be addressed in the revision.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1)The main idea sounds novel and interesting in that there is no previous work alleviating the limitation of the CAM and its variants through an inverse loss supervision training and feature mining via a deformation transformation.\n\n(2)This paper is well-organized and easy to follow in most parts. And experimental details necessary to reproduce the proposed method are presented in the paper. \n\n(3)The efficacy of the proposed method has been well demonstrated by extensive experiments on PASCAL VOC and MS-COCO, where weakly-supervised semantic segmentation based on the proposed method achieves the state-of-the-art performance. Particularly, this reviewer appreciates the CAM results of the expansion stage which seems surprising that most of the object regions are located by the proposed method with high activation response.\n(4)The efficacy of hyper-parameters is investigated by extensive ablation studies. Besides the experiments in “+saliency” setups are also well-executed.\n\nWeakness:\n(1)The proposed method demands a larger number of hyper-parameters, compared to the original CAM. And also it needs more training stage to obtain the initial CAM seeds compared to other related works that also focus on improving the quality of the very initial CAM seeds. \n\n(2)It is not explained why the final localization maps are computed only from the shrinkage stage, instead of the fusion of the proposed two training stages.\n\n(3)Figure 4 is omitted to be referred to in the paper which causes confusion. And the final experimental setting of the kernel size of deformable convolution seems missing in the current manuscript. \n\n(4)The authors utilize inverse loss supervision to sample out as many positive foreground regions as possible, however, it may, unfortunately, cause wrongly located target object regions in the expansion stage when there were multiple object categories present in the image. Especially the co-occurring objects appear simultaneously, e.g., dog and person. Thus, the proposed method could damage precision for the following shrinkage training, since the wrong activated features could cause confusion for the network to make the correct classification. In the same context, this reviewer would recommend analyzing, evaluating, and comparing the proposed method with other related works through precision-recall curves.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept