Exploit CAM by itself: Complementary Learning System for Weakly Supervised Semantic Segmentation
Authors: Wankou Yang
Jiren Mai
Fei Zhang
Tongliang Liu
Bo Han
Venue: Accepted by TMLR ()

Abstract: Weakly Supervised Semantic Segmentation (WSSS) with image-level labels has long been suffering from fragmentary object regions led by Class Activation Map (CAM), which is incapable of generating fine-grained masks for semantic segmentation. To guide CAM to find more non-discriminating object patterns, this paper turns to an interesting working mechanism in agent learning named Complementary Learning System (CLS). CLS holds that the neocortex builds a sensation of general knowledge, while the hippocampus specially learns specific details, completing the learned patterns. Motivated by this simple but effective learning pattern, we propose a General-Specific Learning Mechanism (GSLM) to explicitly drive a coarse-grained CAM to a fine-grained pseudo mask. Specifically, GSLM develops a General Learning Module (GLM) and a Specific Learning Module (SLM). The GLM is trained with image-level supervision to extract coarse and general localization representations from CAM. Based on the general knowledge in the GLM, the SLM progressively exploits the specific spatial knowledge from the localization representations, expanding the CAM in an explicit way. To this end, we propose the Seed Reactivation to help SLM reactivate non-discriminating regions by setting a boundary for activation values, which successively identifies more regions of CAM. Without extra refinement processes, our method is able to achieve improvements for CAM of over 20.0% mIoU on PASCAL VOC 2012 and 10.0% mIoU on MS COCO 2014 datasets, representing a new state-of-the-art among existing WSSS methods. The code is publicly available at: https://github.com/tmlr-group/GSLM.

Published in Transactions on Machine Learning Research (02/2024)
Exploit CAM by itself: Complementary Learning System for Weakly Supervised Semantic Segmentation
Wankou Yang
wkyong@seu.edu..or
School of Automation Southeast University
Jiren Mai
majjiren@lsu..dee edu.cc
School of Automation Southeast University
Fei Zhang
Jerenas@sjtu.edu..om
Department of Electronic Engineering Shanghas Jiao Tong University
Tongliang Liu
tongliang heggllannggiu@@yeee BöuSsydney.... edu.on
Sydney AI Centre The University of Sydney
Bo Han
bhanmm Mhammpm@coop.. hkbu. edua.
TMLR Group
Hong Kong Boptist University
Reviewed OpenReview: https:// opennrvve/eerrvviee. net/ forum? id= Yas
Abstract
Weakly Supervised Semantic Segmentation (WSSS) with image-level labels has long been suffering from fragmentary object regions led by Class Activation Map (CAM), which is incapable generating fine-grained masks for semantic segmentation. To guide CAM to find more non-discriminating object palterns, this paper turns to an interesting working mechanism in agent learning named Complementary Learning System (CLS). CLS holds that the neocortex builds sensation of general knowledge, while the hippocampus specially learns specific details, completing the learned patterns. Motivated by this simple but effective learning pattern, we propose General-Specific Learning Mechanism (GSLM) to explicitly drive coarse-grained CAM to a fine-grained pseudo mask. Specifically, GSLM develops a General Learning Module (GLM) and a Specific Learning Module (SLM). The GLM is trained with lmaee-level supervision lo extract coarse and general localization representations from CAM. Based on the general knowledge the GLM, the SLM progressively exploits the specific spatial knowledge from the localization representations, expanding the CAM in an explicit way. To this end, we propose the Seed Reactivation to help SLM reactivate non-discriminating regions by setting boundary for activation values, which successively identifies more regions of CAM. Without extra refinement processes, our method is able to achieve improvements for CAM of over 20.0% mloU on PASCAL VOC 2012 and 10.0% mloU on MS COCO 2014 datasets, representing a new state-of-the-art among existing WSSS methods. The code is publicly available at: https://github.com http://aw...oom/mmmgooo/////////
1 Introduction
Semantic segmentation (SS) plays an important role in computer vision, which aims classify each pixel in an image. Due to the success of deep learning and CNNs, SS has witnessed great progress in recent years, giving

---

Published in Transactions on Machine Learning Research (02/2024)
1
Salient Foreground
GLM
Confusion
SLM
fPg
Salient Background
Coarse-gralned CAM
Fine-grained CAM Step3. Confusion Reassign
1tep CAM Sense
Step2. Confidence Prominence
Figure 1: Overview of our method. The General Learning Module (GLM) senses coarsc grasnegraaned CAM, which is divided into three parts by thresholds of Ong. fzg, and the corresponding local ground-truth shown below (brown for foreground and black for background). With the guidance of the marked CAM, the Specific Learning Module (SLM) supports confusion reassigning to build a fine-grained CAM.
birth numerous remarkable works (Chen et al. 2017a b Huang al. 2019 Strudel et al. 2021 Cheng
et al. 2021b. However, these methods heavily rely accurate pixel-wise annotations for fully supervised training, which is time-consuming and labor-intensive to collect, making them less economical to apply. Weakly-supervisel semantic segmentation (WSSS) developed to liberate humans from these exhaustive annotation efforts, using weaker and cheaper annotations to achieve semantic segmentation. Image-level labels (Ahn al.] 2019 Zhang et al. 2021 Jiang et al. 2022 Lee et al.) 2022), scribbles (Lin et al.) 2016), bounding boxes (Dai al.) 2015 Khoreva al. 2017). and points (Bearman al.) 2016) are some annotation types commonly used for WSSS. Especially, this paper focuses WSSS based solely image-level labels.
A key problem in WSSS is how to derive localization cues with only the supervision of image-level labels. By exploiting the contribution of local regions to classification confidences, Class Activation Maps (CAM) (Zhou
et al. 2016) provides a key idea and has been commonly used in current WSSS methods (Kolesnikov &
Lampert 2016 Qin et al.) 2022 Lee et al.) 2021a; Zhang et al. 2021 2020 Wang et al. 2020 However,
CAM is incapable of serving as fine-grained mask, as it simply captures
the small salient regions with
Kim et al. 2021 Zhang et al, 2020 2021) have tried to drive the CAM to cover more of the target regions. These methods have achieved some success, but the artificially designed modules are complex and it is difficult to integrate their advantages. In this paper, we turn to an interesting concept in agent learning theories, improving CAM based on an experiential learning process.
discriminating features. To address this issue, many works (Wang al. 2020
The Complementary Learning Systems (CLS) (McClelland al. 1995 Kumaran el al. 2016 hypothesis suggests that two learning systems play complementary roles between the neocortex and the hippocampus human learning process. The neocortex misses some details but generally builds perception of patterns in target knowledge, while the hippocampus repeatedly consolidates learned patterns and completes details. We contend that the problem of the non-discriminating regions overlooked by CAM is similar to that of the missing details in the neocortex. As shown by the confidence distribution of the CAM in Figure ! most pixels in the CAM with neutral confidence are difficult to assign as foreground or background. Indeed, the confusion regions represented by these pixels are mixed with true foreground (brown pixels in the figure) and true background (black pixels). As the salient foreground and background are what CAM has learned, we treat the confusion regions the missing details of CAM. Therefore, we propose a General-Specific Learning Mechanism (GSLM) to help CAM reactivate the missing details and reassign these confusion regions, similar to the collaborative work of the neocortex and hippocampus in CLS. In GSLM, General Learning Module (GLM) learns patterns of imago-level labels and provides localization representations. A Specific Learning Module (SLM) inherits GLM and adjusts the network weights with these representations, supported by the proposed activation loss (Eq. 5 The representations are termed Confidence CAM and are calculated from the original CAM by our proposed Coarse Generation (Eq. 3 which marks salient and confusion regions with boundary constraints. To achieve the reassignment of the confusion regions, Seed Reactivation (Eq, 4 is integrated into SLM, which uses bounded functions to suppress salient regions and activates non-discriminatin regions.
2

---

Published in Transactions on Machine Learning Research (02/2024)
By applying GSLM, fine-grained CAM can be directly obtained by the conventional CAM network. In addition, we show that GSLM is compat ible with other CAM generators (Wang et al. 2020 Lee et al. 2021a Zhang et al. 2021 Ahn et al. 2019) and can be applied to advanced methods for further improvements.
We set up performance experiments on the PASCAL VOC 2012 (Everingham et al.) and the MS COCO 2014 (Lin et al.] (2044) datasets to verify the effectiveness of our method. For accuracy of CAM, our method improves the baseline (48.6% mloU) by 22.1% mloU on the PASCAL VOC 2012 train set and the baseline (32.5% mloU) by 10.8$ mIoU on the MS COCO 2014 train set, ahead of existing methods over 11.6% mloU. For the accuracy of pseudo-masks, our method achieves 75.1% mloU on the PASCAL VOC 2012 train set and 43.7% mloU on the MS COCO 2014 train sct, which is far beyond existing methods. For segmentation results, our method achieves a new state-of-the-ard performance of 70.6% mloU both on val and test set of PASCAL VOC 2012 and 40.9% mloU on the MS COCO 2014 val set.
Our main contributions are summarizod as follows:
We proposc a simple yet efficient training process, Gencral-Specific Learning Mechanism (GSLM), to drive CAM to reassign the confusion regions, producing fine-grained CAM.
We apply GSLM to different advanced methods and achieve further improvements the accuracy of the CAMs.
Experimental results on both PASCAL VOC 2012 and MS COCO 2014 show that our method outperforms the previous state-of-the-art. In particular, method boosts baseline CAM by 22.1% mloU the PASCAL VOC 2012 train set, ahead of the best existing methods over 11.6% mIoU.
2 Related Work
Semantic Segmentation. Semantic Segmentation aims at assigning a predefined category to every pixel on a given image. After introducing the fully convolutional network (FCN) (Long et al. 2015) into this task, researchers have designed various efficient models to improve performance, inchuding dilated convolutions (Chen et al. 2017a). encoder-decoder architecture (Chen et al., 2017b; Ronneberger et al. 2015), and feature pyramid (Zhao et al. 2017 Chen al. 2017b). To further promote the model's ss ability of context aggregation (Zhang et al., 2019 Yuan et al. 2021 self-attention (Huang et al., 2019 Liet al. 2019 paradigms are utilized to learn Long-range dependency. Based on this, vision transformers (Strudel et al. 2021 Cheng et al. 2021) are adapted to segmentation tasks, which occupy the statc-of-the-art model in different benchmarks. This paper focuses mainly on semantic segmentation in the weakly supervised scenario.
Weakly Supervised Semantic Sogmentation. Weakly Supervised Semantic Sogmentation (WSSS) aims to generate pixel-level annolation for segmentation tasks. For Image-level based WSSS, most works (Kleennkovv & Lampert 2016 Qin et al., 2022: Lee et al. 2021aa) Zhang et al., 2021 2020: Wang et al. (2020) have followed prevailing pipcline to address WSSS, which could be described 1) training pscudo-mask generator with image-level labels, and 2) training a fully-supervised semantic segmentation with the generated pseudo-mask labels. Clearly, generating an initial seed map to serve as the pseudo-mask (step 1) is vital for WSSS. To achieve this goal, nearly all works have turned to CAM, which mark out the localization information of an image by merely using a classification network. However, CAM merely focuses on incomplete and fragmentary object regions, driving a gap toward the fine-grained target segmentation maps. To address this issue, there are two mainstream mothods to help CAM extract more potential scods.
The first category mainly focuses on generating better seed regions during the generation of CAM Wang
et al. 2020 Zhang et al.) 2020 2021 Lee et al.
2021a Chen et al. 2022. Kim et al. 2021 Jiang et al. 2019
2021a] focused on crasing the discriminating arcas or features Lee et al. 2019), forcing CAM to pay attention to discriminate proposed to deliberately erase specific regions recognized from CAM, and iteratively re-train them for completing object region. Another idca is to artificially add prior knowledge to lead the CAM network to pay attention to potential features. Cross-image mining methods (Fan)
Some methods (Wei et al. 2017 Kweon et al.
in the classification network (Hou et al. 2018
other potential object areas. Wei et al. 2017
et al.
2020b Sun et al. 2020: added cross-image modules that collect semantics a higher level, from the
3

---

Published in Transactions on Machine Learning Research (02/2024)
Neocrotex General
(a)
Romegronn
Hackground
head
Bird
Confidence
beak
sky
wing
Cours-grained CAM
Hippocampus Specific
(b)
Confidence
head
Bird
wing
beak
Fine-gralned
Pixel Location (Flatten)
Figure 2: Learning stages of CLS and GSLM. For target patterns of birds, ① ncocortex / General Learning Module (GLM) builds general sense of the target, and @ hippocampus Specific Learning Module (SLM) further learns details for completing patterns. The distribution of confidence of (a) Coarse-grained CAM and (b) Fino-grained CAM produced from two stages are shown in right.
relation of multiple images with the same classes. Besides, global memory methods [Meng et al. 2021 Jiang ee al.
2019 turned to constantly record the category features during the training period, expanding the CAM on the knowledge learned in previous stages. Furthermore, some works (Wang et al., 2020 Zhang et al.
based
2021)
investigated specific data rogularizations to lead the CAM expansion in an explicit way. SEAM (Wang
el al.
2020) proposed to leverage the scale invariance of the spatial characteristics while CPN (Zhang el. al.
2021
turned to help CAM by using complementary pair of image inputs.
The second category investigates post-processing to refine the well-trained CAM. Conditional Random Fields (CRF) (Krähenüühl & Koltun 2011) is one of the most frequently used non-training refiner methods in WSSS. To take full advantage of the pixel-level semantic relationship, PSA (Ahn & Kwak 2018) and IRN (Ahn ct al. 2019) aimod learn the similarity between pixels obtained from CRF, and apply random walk further refine the seed areas.
3 Preliminaries & Motivation
Class Activation Map. CAM (Zhou et al. 2016) identifies the contribution distribution towards classifica- tion predictions, extracted from the classification network, and extensively utilized as the initial localization cues for WSSS methods. Specifically, the classification network for CAM consists of a feature extract backbone, Global Average Pooling (GAP) and classification layer. Given the input image of size 3 x H x W, with a height H, width W, and C' potential categories, we denote the combination of the feature extract backbone and the classification layer as f R3 + HC The classification prediction score á E RC can be calculated as (G)A(((())), where a denotes the sigmoid activation function. To train the network, the classification loss is defined as the binary gross-entrops between prediction y and ground-truth 9, i.e.
1 Σ In + - ve) 1n (c)).
Lob(
(1)
After training f. the CAM of an image x related to the c-th class, denoted as Me E RC ,ce (,2,,..C) can be directly calculated as,
M. = ReLU(f(z)))
(2)
max fx)
4

---

Published in Transactions on Machine Learning Research (02/2024)
Shortage of CAM. In principle. CAM roughly indicates the contribution of local image regions to classification confidence (Ahn et al. 2019). The discriminating features always play a dominant role in the classification, as can be seen from Figure 2(a), where the confidence of the bird head regions (highlighted by red boxes) is much ahead of other regions. The phonomenon results in CAM mainly confined to the small salient. regions. Most methods (Wang et al.) 2020 Kim et al. 2021 Zhang el al. 2020; 2021 modify the structure of the CAM generating network or introduce well-designed modules, which alter the distribution of image classification confidence to expand the CAM. However, Figure 2 suggests that foreground points (in red) generally lie above background points (in blue), indicating that CAM has the potential to distinguish non-diseriminating features from the background. Here, pose the question Is CAM fully exploited!
Fully exploit CAM. We hold that CAM is not fully exploited by simply training a classification net- work. As shown by the shaded domain Figure aa), most pixels with neutral confidence in CAM are difficult accurately assign to either foreground or background, representing "confusion regions". For a classification network, such "confusion regions" can neither symbolize the target nor do not belong to the target. Intuitively, CAM could be enlarged if the pixels of "confusing regions" could be reassigned to the right gpreround//acckgromn foreground/ background areas, as shown by Figure b(.). To drive CAM from coarse-grailn wrrse-grrined to fine-grained we turn to the complementary learning system, one of the famous frameworks agent learning theory, to help CAM reassign these confusing pixcls.
Complementary Learning System. We consider the perspective of how our brain learns to address WSSS. Figure 2 shows this process according to CLS (McClelland et al. 1995 Kumaran et al. 2016) theory. Since bird heads appear in most bird images, our neocortex associates the shape of their heads with birds and pays special attention to heads in images. The association is not entirely exact but effective. After several practical trials, the brain expertly picks out the birds head and excludes the sky. In this process, the hippocampus completes more details of the head, causing attention to spread around. In this way, we learn distinguish birds from collection of bird images without pixel-level labels. By analogy with CLS, CAM can be considered to be inadequately trained in the stage of Figure 2p. Our goal is to build learning mechanism to adequately train CAM. Therefore, we propose the General-Specific Learning Mechanism (GSLM). We develop the General Learning Module (GLM) to act as the ncocortex, which generates the coarse-grained CAM and further processes it to serve as general localization representation of the target. The Specific Learning Module (SLM) is constructed to complete the pattern complement process similar to the hippocampus, which receives guidance from the representations and reactivates non-discriminatingg features. In recent years, the inspiration of CLS theory on Deep Neural Networks benefits many fields of Artificial Intelligence (e.g. Deep Q-Network (Blakeman & Mareschal 2020 ) It makes lot of sense to introduce CLS into WSSS.
4 Methodology
This section introduces the proposed General-Specific Learning Mechanism (GSLM). In Section 4.1 we illustrate the overall framework of GSLM. Section 1.2 introduces the General Learning Module (GLM), which extracts Confidence CAM with Coarse Generation. Section 4.3 introduces the Specific Learning Module (SLM), into which Seed Reactivation and activation loss are integrated to support pixel-level training and confusion reassignment.
4.1 Overall framework of GSLM
GSLM represents novel training process for a CAM generating network, which consists of General Learning stage and multiple Specific Learning stages. At each stage, the CAM generating network is wrapped as GLM or SLM, which does not change the topology of the network but allects gradient propagation. Figure 3 shows the GSLM process and the structure GLM and SLM. Firstly, the network at Stage 1 is wrapped as GLM, which is trained with image-level labels for producing coarso-grained CAM. The coarso-grained will later be refined into Confidence CAM. Secondly, the network at stage 2 is wrapped as SLM, with weights shared from GLM and readjusted under the extra pixel-level supervision of Confidence CAM. The Specific Learning stage
5

---

Published in Transactions on Machine Learning Research (02/2024)
General Leaming
Specific Learing
Stage
Stage
Stage
Stage
GLM
SLM
SLM
SLM
CAM
CPM
CAM
CAM
CAM
CodMionenCM
Coddinee
General Learinng Module (GLM)
Spedific Meatiig
CAM
CAM Badibone
Chrssfier
Cls Loss
Chessiier
Cls Loss
Backbone
Sextl Revetvation
Aot
ReLU
O9
Generation
Globel Average Pudlin
CAM
CAM
Confederon CAM
Figure 3: The GSLM framework. In General Learning, the GLM is trained using image-level labels to generate Confidence CAM. Moving on Specific Learning, the SLM is trained using both the image-level labels and the Confidence CAM generated in the previous process. The CAM is refined through GLM multiple stages of SLM, resulting in an improved CAM.
is repeated several times, and specifically, the provider of Confidence CAM and shared weights will be the previous SLM instead of GLM.
As shown by Figure the CAM generation network consists of two branches: classification loss caleulation and CAM generation. GLM retains the structure of the original CAM generating network and introduces the Coarse Generation for Confidence CAM generation. SLM uses Seed Reactivation to take over the CAM generation branch of the network and supports activation loss calculation.
4.2 General Learning Module (GLM)
As the first stage, GLM aims to generate localization knowledge representation provision of pixel-level supervision for subsequent stages. Despite carrying localization information, CAM is not suitable for direct use as supervision due to its incomplete object information (Kim et al., 2021 Zhang et al 2021). We focus on the trusted local CAM and propose the Coarse Generation algorithm to generate Confidence CAM, an improved localization map that reinforces salient regions and remarks confusion regions, as suporvision.
Confidence CAM. A Confidence CAM indicates three areas with three values, where salient foreground is set -1 the flag to be ignored. The
set 1, salient background is set. to and rest. confusion visualization of sample Confidence CAM is shown input of SLM in Figure 3
Coarse Generation. For generating Confidence CAM, we introduce thresholds 0ʃg and 0bg to divide CAM into salient. foreground (> 0gg), salient background (< Obg), and confusion area. In addition, conditional random fields (CRF) Krähenbühl & Koltun 2011), a widely used non-learnable CAM refinement method, is applied to further refine the salient regions, introducing the boundary constraint information for such knowledge representations. We define a confidence mapping as g and CRF as R. RCxHxx xx For the e-th class, Confidence CAM N, be generated from given CAM Ma as follows.
1
r 2 0
N. = R(g(MM)),
g(x)
0
I < 0bg
(3)
else

---

Published in Transactions on Machine Learning Research (02/2024)
(a) Conventional CAM
(b) CAM DRS
(c) Cf SR
1.0
10
10
Confidence
Conficeacce
Relative Bound
Confidence
k
SF N-SF B
01
SF N-SF
SF N-SF B
SF N-SF B
SF N-SF
Figure 4: Comparison between DRS Kim et al. 2021 and Seed Reactivation (SR) of GSLM on refinement
principles of CAM. SF: Salient Foreground, N-SF: Non-Salient Foreground, Background.
4.3 Specific Learning Module (SLM)
SLM aims to readjust the connection weights of CAM generating network with Confidence CAM for Specific Learning. Specifically, Specific Learning refers to consolidating the object patterns of salient regions learned and discovering new patterns to complement the attention of CAM. Therefore, we introduce activation loss, which provides pixel-level supervision with the guiding Confidence CAM. And we propose Seed Reactivation to help the network activate more regions, free from the crushing effects of salient regions. Besides, the classification loss of GLM is relained for consolidating the learned patterns.
Seed Reactivation On the one hand, Seed Reactivation is designed to address the issue that the conventional CAM generation is not compatible with pixel-level supervision based on Confidence CAM. Specifically, the ional CAM is generated by global maximum normalization (Fq. 2 where all pixels are divided by the maximum pixel value. However, there are cases where the maximum pixel is located in the confusion regions of Confidence CAM and thus ignored during training, failing model convergence:
SLM may use different weights (and even different network structure) than GLM, and thus the maximum pixel can occur anywhere in CAM.
Even though SLM and GLM are identical at the beginning of training, and the maximum pixel is plotted in the salient foreground regions of Confidence CAM. However, the training progresses, SLM will activate some confusion regions as the foreground as expected, and the maximum pixel may be located here.
To address the issue, we introduce a bounded ReLU-k function v using constant denominator for normalization, i.c.,
x 2
M (f(())c) (xxx)
x clsc
(4)
K
On the other hand, Seed Reactivation (Eq. replaces the conventional CAM generation method (Eq. 2 thereby affecting the resulting CAM. As shown in Figure (a)(c), Seed Reactivation a constant upper bound k for regions in the feature map. With pixel-level training, each pixel will be activated (approaching k) suppressed (approaching 0) independently of the other pixels. Similarly, DRS (Kim et al.) 2021 applics relative upper bound to network feature maps to suppress the salient foreground with high confidence. However, the truncation of the high-confidence pixels increases the relative confidence of all remaining pixels, thus incorrectly activating the background region, as shown in Figure 4 (b). For example, a bound of 0.5 relatively increases the confidence of all regions by a factor 2. The background with high confidence will be misclassified as foreground, resulting over-activation. By introducing extra supervision saliency maps background masks, DRS sidesteps the issue of aver-activation, but with Seed Reactivation, GSLM can achieve this effect without saliency maps.

---

Published in Transactions on Machine Learning Research (02/2024)
Activation Loss The activation loss narrows the gap between the improved CAM M' produced by SLM and Confidence CAM N produced by GLM (or SLM in the previous stage). Specifically, we ignore the non-discimminating regions (-1 in Confidence CAM). By measuring the gap with the smooth L1 norm, the activation loss is defined as,
Lact = ||M - Nihh, iE [x|N2 2 )
(5)
Activation loss improves the CAM prediction M' with Confidence CAM N as the target. The foreground confidence will approach 1, while the background confidence will approach to O, and the confusion area shrinks gradually. The total loss function of SLM is then defined as follows,
L = Lols + alLact:
(6)
where a denotes the balance factor between classification loss and activation loss.
5 Experiments
5.1
Experimental Setting
Dataset and Evaluation Metrics. We performed experiments on the PASCAL VOC 2012 dataset (Ever ingham et al.),, visual object class challenge with 20 categorics built for real scenes. It contains a total of 10,582 training images, of which 1,464 images have pixel-level labels and the rest are bounding box labels, 1,449 validation images with pixel-level labels, and 1,456 test images. We also verify our method using the MS COCO 2014 dataset (Lin et al.) 2014) which contains 82,783 train and 40,504 validation images with 81 categories. For each dataset, we train our network with the training images and image-level classification labels only, and evaluate the pseudo-masks with pixel-level labels. All image-level labels for training are extracted from pixel-level labels or bounding box labels. The fully supervised semantic segmentation network is trained with the pseudo-masks and evaluated on PASCAL VOC 2012 validation and test set, and MS COCO validation set, respectively. Performance is evaluated by the mean intersection-over-unig (mloU). The evaluation PASCAL VOC 2012 test, without any annotation, is obtained from the official PASCAL evaluation server.
Implementation details. We use an ImageNet (Dnng et al. 2009) pretrained ResNet50 (He et al. 2016 as the CAM backbone in GSLM and a 1 x convolution as the classification layer. For hyper-paraneeters k in Eq. is 6, a in Eq. X is 0.5, and the thresholds 05g and 0og in Coarse Generation are 0.30 and 0.05, respectively. The number of itcrations of SLM is set to 3 for GSLM. For training setting, the batch size is 16, the training epochs is 5, and the weight decay is 0.001. The remaining optimization settings are the same as in IRN (Ahn et al. 2019). Moreover, we build a stronger baseline GSLM+ by replacing GLM in GSLM with IRN and keeping SLM the same but with a single iteration. All other settings are the same as GSLM.
5.2 State-of-the-arts Comparison
Improvements on CAM. Table I reports the performance comparison of CAM seed and pseudo-masks produced by ours and existing WSSS methods. By convention, pseudo-masks are refined from CAM seeds using PSA (Ahn & Kwak 2018) or IRN dAhn et al.) 2019). Without refinement, our GSLM reports an excellent result of 67.5% mloU on the PASCAL VOC 2012 train set, outperforming other methods. With the refinement of CRF and IRN (shown as "Mask" in Table D) GSLM also leads the way and generates pseudo-masks with mloU of 69.2% and 72.4%, respectively. Figure 5 shows by GSLM. Moreover, we set up G by replacing GLM to IRN (reporting 66.3% mIoU in Table I achieving greater performance of 70.7% mloU (for CAM seed), 73.4% mloU (for refinement with CRF) and 75.1% mloU (for pseudo-masks), respectively. Compared with the baseline method IRN, GSLM+ provides an improvement of 8.8% mloU and is thus the first method to produce CAM scods with an accuracy exceoding 70% mloU. Furthermore, GSLM also performs well on the challenging MS COCO 2014 train set, reporting 43.3% mIoU for CAM seed and 45.0% mIoU for pseudo-masks.
samples of CAM produced
8

---

Published in Transactions on Machine Learning Research (02/2024)
Table 1: Comparison of pseudo-masks on PASCAL VOC 2012 train and MS COCO 2014 train set in
mIoU (%). Seed: accuracy of CAM. CRF: accuracy of CAM with CRF. Mask: accuracy of CAM with PSA/IRN. fdenotes the re-implemented results.
VOC2012
COCO
Method
Seed
CRF
Mask
Seed
Mask
PSA Ahhn & Kwak
2018
48.0
61.0
IRN
Ahn al.
2019
48.8
54.3
66.3
32.5
38.4
CONTA
Zhang el al. 2020
48.8
67.9
28.7
35.2
SEAM (Wang et al. 20201
55.4
56.8
63.6
25.1
31.5
CPN Zhang et al. 2021
57.4
67.8
AdvCAM Lee et al. 2021a
55.6
62.1
69.9
AMR QQin et al.
2022
56.8
69.7
CLIMS Xie
2022
56.6
70.5
SIPE Chen et al. 2022
58.6
64.7
W-OoD Lee et al.
2022
59.1
65.5
72.1
PPC Du et al.
2022
61.5
64.0
70.1
GSLM (Ours)
67.5
69.2
72.4
43.3
45.0
GSLM (Ours)
70.7
73.4
75.1
41.4
43.7
@
(b)
(c)
Figure 5: Visualizations of CAM produced by GSLM at different stages on the PASCAL VOC 2012 train. (a) Original image. (b) CAM in Stage (equivalent to baseline). (c) CAM after Stage (d) CAM after Stage
(e) CAM after Stage 4. With the progress of iterations, GSLM activates broader and accurate regions.
Improvements on segmentation results. To apply GSLM in semantic segmentation, we train DeepLabV2-ResNee (Chen et al. 2017a) with the pasus generated by GSLM and GSLM+, respectively. Table 2 shows the comparison of the performance of state-of-the-art WSSS methods on the PASCAL VOC 2012.
Our GSLM achieves 69.4% mloU on val and 69.7% mloU on test. With the seeds provided by IRN, G achieves 70.6% on val and 70.6% on test, an improvement of 7.1% and 5.8% over IRN. GSLM outperforms existing methods with only image-level labels and most methods with extra supervision, representing new state-of-the-art performance
the PASCAL VOC 2012. GSLM+ is only slightly behind several methods with extra saliency maps (L2G (Jiang et al. 2022 by 1.5% and PPC (Du et al. 2022) 2.0%), validating its excellent performance in constraining over over-activation of initial seeds without saliency maps. Figure b shows some qualitative samples of semantic segmenlation results of our GSLM+. Table 3 shows that our GSLM is also efficient for challenging MS COCO 2014 and achieves 40.9% mloU, ahead of existing methods.
Besides, OCR (Cheng al., (023) also reports bascline with MCTformer (Xu et al. 2022) which achieves 72.7% and 72.0% mloU on PASCAL VOC 2012 and 42.5% mIoU on MS COCO 2014, slightly ahead of
GSLM. However, by using DeiT-S (Touvron et al. 2021 the classification network, COR-MCTforman

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: Comparison of state-of-the-art WSSS methods on PASCAL VOC 2012 in mloU (%). Marks of supervision (Sup.) denote image-level labels (Z), saliency maps (S) and texts (T). All results are based on ResNet backbone.
Method
Pub.
Sup.
Val
Test
FickleNet. (Lee
al. 2019)
CVPRI9
I+8
64.9
65.3
ICD
Fan et al.
2020a
CVPR20
I+S
67.8
68.0
AuxSeg Net [Lee et al.
2021b
ICCV21
I+
69.0
68.6
NSROM (Yao et al.] 2021
CVPR21
I+
70.4
70.2
CLIMS (Xie el al. 2022
CVPR22
I+T
70.4
70.0
L2G Jiang et al.
2022)
CVPR22
I+
72.1
71.7
PPC et al. 2022)
CVPR22
I+
72.6
73.6
IRN Ahn et al. 2019
CVPR19
I
63.5
64.8
SEAM Wang et al. 2020
CVPR20
I
64.5
65.7
CONTA Zhang et al. 2020
NIPS20
66.1
66.7
AdvCAM Lee
al. 2021a
CVPR21
68.1
68.0
CPN
Zhang et al.
2021
ICCV21
67.8
68.5
PPC
Du et al. 2022
CVPR22
I
67.7
67.4
SIPE CChnn et al. 2022
CVPR22
I 68.8
69.7
W-OoD (Lee et al. 2022
CVPR22
I
69.8
69.9
Kho et al.
[Koo et a.. 2022
PR22
I
69.5
70.5
OCR (Cheng ct al. 2023
CVPR23
67.8
68.4
GSLM (Ours)
I
69.4
69.7
GSLM+ (Ours)
I
70.6
70.6
(a)
(b)
(c)
Figure 6: Qualitative results PASCAL VOC 2012 val set. (a) Input Images. (b) Ground-truths. (c) results of GSLM+ (w/ CRF).
cannot be compared fairly with the methods in Table and Table that use ResNet, so it is not included. BECO (Rong et al. 2023) works on robust learning for pseudo-label on WSSS and performs well on PASCAL VOC 2012 (72.1% and 71.8% mloU) and MS COCO 2012 (45.1%), but it is based on a modified (co-training) DecpLabV3+ (Chen et al. 2018) as a fully supervised segmentation network and is therefore not listed. BECO has made lot of efforts in training and inference of fully supervised segmentation networks, which can be applied to boost the listed methods, but cannot be fairly compared with them using common DeepLab.
5.3 Ablation Study
Effectiveness on different baselines. Our GSLM generates CAM seeds and then refines them from 48.6% mIoU 67.5% mIoU, showing excellent performance. Furthermore, GSLM could also be applied various WSSS frameworks. As shown in Table g GSLM greatly improves the accuracy of the CAM seeds for various WSSS methods. Specifically, it boosts SEAM to 67.6% mloU (+12.2% mloU), CPN to 66.9% (+9.6%
10

---

Published in Transactions on Machine Learning Research (02/2024)
Table Comparison of state-of-the-art WSSS methods on MS COCO 2014 mloU (%). Marks of supervision
(Sup.) denote image-lovel labels (I) and saliency maps (S). Methods marked by use VGG backbonc and
marked by ResNet backbone.
Method
Pub.
Sup.
Val
ADL
(Choe al.
2020
PAM120
I + S
30.8
AuxSegNex (Lee al.
2021c
ICCV21
I+ S
33.9
TPS
Lee et al. 2021c
CVPR21
I + S
35.7
T.2G
Jiang et al. 2022
CVPR22
I S
42.7
SEC
Kolesnikov & Lampert 2016)
ECCV16 I
22.4
IRN
Ahn et al.
2019)
CVPR19 I
32.6
CSE
Kweon et al.
2021b)
TCCV21
I
36.4
RCA
Zhou et al. 2022)
CVPR22
I
36.8
1OC
Cheng et a..
2023
CVPR23
I
33.2
GGSLM (Ours)
I
40.9
Table 4: The refining effectiveness of GSLM with initial seed produced by different baseline in mloU (%) PASCAL VOC 2012 train set.
Method
+GSMM Seed
CAM Zhouu
al. 2016
48.6
67.5
SEAM [aan et al.
2020
55.4
67. 67.6412.2
CPN (Zhang et al. 2021
57.3
66.949.6
Adv CAM Lee et al. 2021a)
55.5
68.0
IRN [Ahn et al. 2019
66.5
70.7 70.7
mloU), and AdvCAM to 68.0% (+12.5% mloU), all significantly ahead existing seate-of-the-ars methods in Table In particular, GSLM improves the performance of IRN up to 70% mloU, providing fairly high quality CAM seeds for WSSS.
Effect of main modules. To evaluate the effect of each part of GSLM, remove one of Coarse Generation (CR), Seed Reactivation (SR), and classification loss (Tass in Eq ① respectively, and calculate the mloU of GSLM, as shown in Table 5 When CR or SR is removed, GSLM loses its usefulness and the performance drops to 49.3% mloU and 49.3 49.5% mloU. When Less is removed, performance decreased by 1.5% compared to the complete GSLM. In other words, CR and SR are critical modules, and Late improves the performance of GSLM. Moreover, by applying SLM iteratively, the result further achieves a 7.4% improvement.
Effect of boundary constraint in coarse generation. In Section in Coarse Generation introduces boundary constraint information to Confidence CAM for advantage of reactivation in SLM. To validate this view, Figure 7 shows the visual comparison of GSLM with and without
contend that CRF used
Table The ablation study for each part of GSLM on PASCAL VOC 2012 train set. CR: Coarse Generation.
SR: Seed Reactivation. Lets: classification loss in Eq (1] Iter: Iteration of SLM.
baseline CR SR Lets Iter mloU (%)
48.6
49.3
49.5
58.6
60.1
67.5
11

---

Published in Transactions on Machine Learning Research (02/2024)
Stage 1
Stage 2
Stage 3
Stage 4
Figure Comparison of CAM changes different stages of GSLM without (a) and with (b) boundary constraint, (BC).
80
Daseline
0.8
baseline
0.8
GSLM w/o BC
CSLM w/o BC
GSLM w/o GSLM
GSLM
70
mloU/ 7%
0.6
0.6
Mynder
Mover
60
0.4
0.4
50
40
0.2
0.2
2
3
Stage (a)
Stage (b)
Stage
(c)
Figure Comparison of GSLM with and without boundary constraint (BC) on (a) CAM accuracy, (b) under-activation, and (c) over-activation in different stages. For (b), the lower the degree of under-activation, the more regions of objects activated by the network. For (c), the lower the degree of over-activation, the more accurate the network activation area is.
boundary constraint (by adopting CRF in Coarse Generation or not). Figure 7(a) demonstrates that, with the progress of iteration, the activated foreground of GSLM w/o BC is excessively diffused the background region. This diffusion impedes the enhancement of accuracy, which can be prevented by BC, as shown in Figure b). For further validation, Figure reports the comparison of GSLM w/ BC and GSLM w/o BC in metrics of CAM accuracy, under-activation, and over-activation, respectively. Specifically, we evaluate the degree of under-activation and over-activation by following metrics proposed in (Wang et al. 2020
FN
Munder C-1
(7)
TP.
C=1
C-1
FP.
Mover 0-1
(8)
TP.
where C' denotes the number of categories with background, TP. denotes the pixel number of the true positive prediction of class FPc and FN, denote false positive and false negative, respectively.
In Figure S a), GSLM with BC demonstrates better performance than GSLM without BC in all rounds. c) report the changes of the degree of under-activation and over-activation of the settings of GSLM the itcration process, respectively. Both of them show similar performance in the optimization of anteraactivation. However, in the absence of boundary constraints, GSLM fails control the over-activation degree, shown in Figure B(c), the over-activation degree of GSLM without BC is close
Figure b)) and Figure S
12

---

Published in Transactions on Machine Learning Research (02/2024)
65
@
0gg ra
75.0
67.5
67.1
basdine
miou (%)
55.0
67.1
005-0.30
35.0
17.7
moU (%)
53
0.10-0.20
15.0
1
6
36
216
00F--0200
50
(065-.10
70.0
43
0,10-030
nioU (%6)
66.0
66.3
67.9
0.20-030
62.0
40
58.0
0.05
0.15
0.25
0.35
0.45
0.55
0.65
7
Sample threshold
With only once Iteration
Number of iterations
Figure 9: The CAM performance of GSLM with different (a) 0ng. 5gg (b) l. and (c) number of iterations PASCAL VOC 2012 train set.
to the baseline. conclusion, the under-activation problem of CAM is reduced due to the reactivation of nin-disrrmnnnating regions by the SLM of GSLM. Moreover, the over-activation problem is reduced by introducing boundary constraints in Coarse Generation.
Effect of dyper-parameters Figure reports the effects of some of GSLM, respectively.
on the CAM accuracy
Ong. Og: Figure ((a) shows the CAM performance curve of GSLM for difference Ong: fg.- The sampling threshold is 0.15, with which baseline achieves 48.6% mloU, When (Oog. 0rg) is (0.05, 0.3). the best threshold of GSLM coincides with the sampling threshold. However, decrcasing 0,g (dashed line item) or incrcasing Ong (dotted line item) makes the best threshold deviate from the sample threshold, which is not conducive more iteration of GSLM. Narrowing the interval of 0bg. fig (blue item) will slightly reduce performance.
bb shows that GSLM is insensitive to h when k < 45, and GSLM diverges when k > 45. This is due to the excessively large value of k, resulting in an excessively high output of the loss function Eq. 0
k: Figure g
number of iterations: Figure c) shows that the accuracy of GSLM is stable after 3 iterations.
label compatibility of CRF: The label compatibility (using Potts model) of CRF (Krähenüühl & Koltun 2011 indicates a penalty for nearby similar pixels that are classified as different class. higher allows CRF to focus on refining class boundaries rather than correcting misclassificatiions, which complements the fine-grained CAM of GSLM after iterations. Therefore, we increase the μ from 3 10 in the last iteration of GSLM and GSLM+, resulting in 0.6% mIoU improvement of masks for GSLM and 0.8% mloU for GSLM+, respectively.
Table 6: The comparison of the foreground recall rate (Ryg), background recall rate (Reg) and mloU in % of CAM produced by DRS, SR (GSLM without iteralion, i.e., line 5 of Table 5) and GSLM with (w/) and without (w/o) saliency maps (S) on PASCAL VOC 2012 train set. All results of DRS are re-implemented.
Method
Rgg
Rbg
mIoU
DRS w/o S
70.1
78.3
42.2
SR w/o S
76.4
92.5
60.1
GSLM w/o S
84.9
92.8
67.5
DRS w/
72.5
96.0
66.3
SR w/ S
72.3
98.8
66.8
GSLM w/ S
81.1
97.8
72.4
13

---

Published in Transactions on Machine Learning Research (02/2024)
Comparison of Seed Reactivation and DRS. In Section 4.3 we have introduced Seed Reactivation and contended it addresses the drawback of over-activation in DRS. In support of the statement, Table G reports the comparison of the CAM recall rates (R = TP+FN where TP and FN denotes true positive and false negative) of DRS, SR and GSLM. For fair comparison, we evaluate all methods with (w/) and without (w/o) saliency maps The lower the background recall rale (Rig), the more background regions mistakenly get high activation values, i.c. the more serious the over-activation. As shown Table 0 without the background correction by saliency maps, the DRS only achieves a 78.33% on Rog, resulting in a low mloU of 42.2%. Saliency maps increase the Rog of DRS to 96.0%, yielding mloU of 66.33% (this value is 62.9% reported by Kim et al. (202Tp). SR obtaines high Rag of 92.5% without saliency maps, which proves the accurate activation of SR on the background. In addition, iteration can improve Rog of GSLM (from 76.4% to 84.9%) while maintaining the high Rig of SR, contributing to CAM precision.
TP
6 Conclusion
In this paper, we aim to address WSSS by exploring and exploiting the Complementary Learning System (CLS) for CAM, which is mostly confined small discriminating object areas. Specifically, we proposed General-Specific Learning Mochanism (GSLM) to help drive CAM advance a fine- ine-grained way. GSLM consists of General Learning Module (GLM) and Specific Learning Module (SLM). Specifically, GLM extracts localization information with boundary constraints from images and stores it Confidence CAM. SLM organizes region reassigning guided by Confidence CAM, which reinforces salient regions and remarks confusion regions. Experimental results validate that our method achieves new stale-of-arl performance.
Acknowledgement
This work was supported by the National Natural Science Foundation of China under Nos. 62276061. BH was supported by the NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A1515016622, HKBU Faculty Niche Research Arcas No. RC-FNRA-IG/22-22 22//S//// SCC//4, and HKBU CSD Departmental Incentive Scheme. TLL was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, L.P22020994,, and IC190100031.
References
Jiwoon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4981-4990, 2018.
Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. In Proceedings of the IEEE/CVE conference on computer vision und pallern recognilion, pp. 2209-2215 2019.
Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. What's the point: Semantic segmentation with point supervision. Tn European conference computer vision, pp. 549-565. Springer, 2016.
Sam Blakeman and Denis Mareschal. A complementary learning systems approach to temporal difference learning. Newral Networks, 122:218 230, 2020.
Liang-Chise Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEFE transactions on pattern analysis and machine intelligence, 40(4):834-845 2017a.
Stang-Chiehe Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolutionn for semantic image segmentation. arXiv preprint arXiv:1706.05553 2017b.
Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decodor with atrous separable convolution for semantic image segmentation. In Proceedings of the Europeon conference computer vision (ECCV), pp. 801-818, 2018.
14

---

Published in Transactions on Machine Learning Research (02/2024)
Qi Chen, Lingxiao Yang, Jian-Huang Lai, and Xiaohua Xie. Self-supervised image-specific prototype exploration for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4288 4298, 2022.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34, 2021.
Zesen Cheng, Pengchong Qiao, Kehan Li, Siheng Li, Pengxu Wei, Xiangyang Ji, Li Yuan, Chang Liu, and Jie Chen. Catddd--a rectification for weakly supervised semantic segmentation. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23673 23673-23684, 2023.
Junsuk Choe, Seungho Lee, and Hyunjung Shim. Attention-based dropout layer for weakly supervised single object localization and semantic segmentation. IEEE transactions on pottern analysis and machine intelligence, 4222) 4271, 2020.
Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In Proceedings of the IEEE international conference on computer vision, pp. 1635-1643, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pottern recognition, pp. 248 255, leee, 2009.
Ye Du, Zehua Fu, Qingjie Liu, and Yunhong Wang. Weakly supervised semantic segmentation by pixel- to-prototype contrast. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4320 4329,,43,..0 2022.
M. Everingham, T. Van Gool, K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. WOC/VOC2000 wwlele/gee//////oooo////////ree/ www.//oo//ideeeese.se
http://www.pasch/
Junsong Fan, Zhaoxiang Zhang, Chunfeng Song, and Tieniu Tan. Learning integral objects with intra-class discriminator for supervie semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1283-4292. 2020a.
Junsong Fan, Zhaoxiang Zhang, Tieniu Tan, Chunfeng Song, and Jun Xiao. Cian: Cross-image affinity net for weakly supervisod semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10762 10769, 2020b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Qibin Hou, PengTao Jiang, Yunchao Wei, and Ming-Ming Cheng. Self-erasing network for integral object. attention. Advances in Neural Information Processing Systems, 31, 2018.
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss- cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Peng-Tas Tao Jiang, Qibin Hou, Yang Cao, Ming- Ming Cheng, Yunchao Wei, and Hong-Kai Xiong, Integral object mining via online attention accumulation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2070-201 2079.07999 2019.
Peng-Tao Jiang, Yuqi Yang, Qibin Hou, and Yunchao Wei. L2g. A simple local-to-global knowledge transfer framework for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16886-16896, 2022.
Sungpil Kho, Pilhyeon Lee, Wonyoung Lee, Minsong Ki, and Hyeran Byun. Exploiting shape cues for weakly supervised semantic segmentation. Pattern Recognition, 132:108953, 2022.
15

---

Published in Transactions on Machine Learning Research (02/2024)
Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In Proceedings of the TEEE conference compuler vision and pottern recognition, pp. 876 885, 2017.
Beomyoung Kim, Sangeun Han, and Junmo Kim. Discriminative region suppression for weakly-supervised semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2021.
Alexander Kolesnikov and Christoph H Lampert. Scod, expand and constrain: Three principles for weakly- supervised image segmentation. In European conference on computer vision, pp. 695-711. Springer, 2016.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems, 24, 2011.
Dharshan Kumaran, Demis Hassabis, and James L McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in cognitive sciences, 20(7):512 534, 2016.
Hycokjum Kwcon, Sung-Hoon Yoon, Hyconscong Kim, Dachce Park, and Kuk-Jin Yoon. Unlocking the potential of ordinary classifier: Class-specifs adversarial erasing framework for weakly supervised semantic segmentation. In Proceedings of the IEFE/CVP International Conference on Computer Vision, pp- 6994- 7003, 2021a.
Hyeokjun Kweon, Sung-Hoon Yoon, Hyconseong Kim, Dachee Park, and Kuk-Jin Yoon. Unlocking the potential of ordinary classifier: Class-speciff adversarial crasing framework for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 6994- 7003, 2021b.
Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon. Ficklenet: Weakly and semi- supervised semantic image segmentation using stochastic inference. In Proceedings of the IEEE/C Conference on Compuler Vision and Pollern Recognition, pp. 5267 5276, 2019.
Jungboom Loe, Eunji Kim, and Sungroh Yoon. Anti-adversarially manipulated attributions for weakly and semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4071-4084 2021a.
Jungbeom Lee, Eunji Kim, and Sungroh Yoon. Anti anversalersarillly manipulated attributions for weakly and semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4071-4080, 2021b.
Ju Jungbeom Lee, Seong Joon Oh, Sangdoo Yun, Junsuk Choe, Eunji Kim, and Sungroh Yoon. Weakly supervised semantic segmentation using out-of-disttibattio data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16897-16906, 2022.
Seungho Lee, Minhyun Lee, Jongwuk Lee, and Hyunjung Shim. Railroad is not train: Saliency as pseudo- pixel supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 5495-5505, 2021c
Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu. Expectation-maxinizttin allention networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-sppervised convolutional networks for semantic segmentation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 3159 3167, 2016.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Europeon conference on computer nision, pp. 740-755. Springer, 2014.
16

---

Published in Transactions on Machine Learning Research (02/2024)
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
James I, McClelland, Bruce L. McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Meng Meng, Tianzhu Zhang, Qi Tian, Yongdong Zhang, and Feng Wu. Foreground activation maps for weakly supervised object localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3385-3395, 2021.
Jie Qin, Jie Wu, Xuefeng Xiao, Lujun Li, and Xingang Wang. Activation modulation and recalibration scheme for weakly supervised semantic segmentation. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 36, pp. 2117-2125, 2022.
Shenghai Rong, Bohai Tu, Zilci Wang, and Junjic Li. Boundary-enhanced co-training for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pallern Recognition, pp. 19574-195844 19584, 2023.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image compuling and compuler-ussisted intervention, pp. 1941241. Springer, 2015.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7262- 7272, 2021.
Guolei Sun, Wenguan Wang, Jifeng Dai, and Lue Van Gool. Mining cross-image semantics for weakly supervised semantic segmentation. In European conference computer vision, pp. 347-365. Springer, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347 10357. PMLR, 2021.
Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. Self-supervised equivariant atlention mechanism for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12275-12284, 2020.
Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. Object region mining with adversarial crasing: A simple classification to semantic segmentation approach. In Proceedings of the IEFE conference computer vision and pattern recognition, pp. 1568-1576, 1576, 2017.
Jinheng Xie, Xianxu Hou, Kai Ye, and Linlin Shen. Clims: Cross language image matching for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 1483-4492 2022.
Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, and Dan Xu. Multi-class token transformer for weakly supervised semantic segmentation. In Proceedings of the IEEE/ /VV/ Conference on Computer Vision and Pottern Recognition, pp. 4310-4319, 2022.
Yazhon Yao, Tao Chen, Guo-Sen Xie, Chuanyi Zhang, Fumin Shen, Qi Wu, Zhenmin Tang, and Jian Zhang. Non-salient region object mining for weakly supervised semantic segmentation. In Proceedings of the IEFE/CC CCFF Conference on Computer Vision and Pattern Recogrition, pp. 8623-2632 2021.
Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. Ocnet: Object context for semantic segmentation. International Journal of Computer Vision, pp. 1 1-24, 2021.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, and Qianru Sun. Causal intervention for weakly-supervisedd semantic segmentation. Advances in Neural Information Processing Systems, 33:655-666 666, 2020.
Fei Zhang, Chaochen Gu, Chenyue Zhang, and Yuchao Dai. Complementary patch for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7242 7251, 2021.
Hang Zhang, Han Zhang, Chenguang Wang, and Junyuan Xie. Co-occurrent features in semantic segmentation. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pollern Recognition, pp. 548 548-557, 2019.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 2881-2890, 2017.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 2921-2929, 2016.
Tianfei Zhou, Meijie Zhang, Fang Zhao, and Jianwu Li. Regional semantic contrast and aggregation for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 4299 4309, 2022.
18


Reviews:
[{"review_id": "o1hMssMEYS", "reviewer": "Reviewer_hhvb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:  \n1. The method is simple and performs good on standard benchmarks.  \n2. The work has extensive results and ablations.  \n\nWeaknesses:  \n1. **Motivation of seed reactivation**  \nThe author argue that the maximum pixel value maybe located in confusion regions of Confidence CAM (page 6 Section 4.3), which might result in failure in model convergence. I find it difficult to understand, how is this possible ? For conventional CAMs from equation 2, the highest value pixel for a given class map (c) , has to be in salient foreground as it is the highest pixel value. With max normalisation (per channel) as in eq 2, the activation value will be 1. How can this go in confusion region ? The motivation of seed reactivation feels confusing to me.   \n\n2.  **Similarity with DRS**  \nThe seed reactivation (equation 4) which seems to be the key of the proposed approach is extremely similar to DRS. The authors try to delineate DRS and this work with the line - “Similarly, DRS (Kim et al., 2021) has been applied a relative upper bound to network feature maps to suppress the salient foreground with high confidence. However, the background may be incorrectly activated because the regions are not independent, as shown in Figure 4 (b).” The authors argue that backgrounds maybe incorrectly activated for DRS, which in my opinion should be explained more, as how it happens. Maybe a qualitative example should help (from real samples). \n\n3. **Comparison with DRS**\nThe proposed approach comparison with DRS is missing. One baseline should also be comparison with DRS with training in different stages/iterations to show the impact of proposed seed reactivation as authors claim it to be superior to DRS.  \n\n4. **Comparison with prior works missing**   \na. Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast, CVPR 22. \nb. Out-of-Candidate Rectification for Weakly Supervised Semantic Segmentation, CVPR 23. \nc. Boundary-enhanced Co-training for Weakly Supervised Semantic Segmentation, CVPR 2023.   \n\n5. **Understanding Activation loss**   \nThe activation loss in equation 5, aims to bridge the GAP between improved CAM M’ and confidence CAM N. In this what is fixed and what has gradients (what is learnt ?) (M’ vs N). From what I understand, N is fixed which is given as supervision to improve M’ (which seems like to be the case from the figure). If that is true, then from the equation 5, non significant foreground regions in N with value 0, will pull the values in improved CAM M’ non significant foreground region towards 0, which seems to be conflicting towards the whole idea and also from Figure 4c. Can you provide more details on this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work aims to solve weakly supervised semantic segmentation(WSSS) with image-level labels by improving the generated Class Activation Maps(CAM). In particular, the authors propose two modules, General Learning Module, which produces the coarse CAMs and Specific Learning Module, which enhances the coarse CAMs with finer details. Overall, the method is competitive with prior state of the art approaches for WSSS.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Changes in writing**  \n1.  Improve the motivation of seed reactivation  \n2. Proper distinction with DRS should be made. Current distinction is not clear.    \n3. Improvement in activation loss description. Current one seems a bit confusing to me regarding its working.  \n\n**Experimental changes**  \n1.  Comparison with DRS as a baseline with multi stage training  \n2. Comparison with prior works in the main table", "broader_impact_concerns": "I do not see any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yYeSIFeaIK", "reviewer": "Reviewer_oBpv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Simple method for refining the CAM towards segmentation masks which seem to yield good improvements with respect to CAM and competing methods.\n* The method can be combined with other approaches and still lead to improvements (Table 4).\n* The paper is well written and easy to follow for the most part.\n* The use of many hyperparameters ($\\theta_{fg}, \\theta_{bg}, k, \\alpha$) seem relatively tricky to adjust; however, their effect is well studied in the ablation study and answered most of my questions.\n\nWeaknesses:\nThe paper is not exhaustive about the state-of-the art, and it seems multiple competing approaches using the same supervision reach significantly higher numbers than reported. For instance, \"Boundary-enhanced Co-training for Weakly Supervised Semantic Segmentation\" (Rong et al., CVPR 2023) reaches 73.7% mIoU on VOC-val, and multiple other examples can be found [1].\nUnless I am mistaken about this, I invite the authors to be clearer about the WSSS state-of-the art while reporting their results.\n\n[1] https://paperswithcode.com/sota/weakly-supervised-semantic-segmentation-on", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method for weakly supervised sementic segmentation called GLSM. The approach relies on successive trainings of the classifier using a combination of image-level labels and pseudo-labelled segmentation masks.\nThe initial pseudo-labels are given by CAM, which is then used as ground truth in the second stage after some post-processing consisting of thresholding between very confident positives and negative pixels, and applying a CRF to infer the segmentation masks for uncertain pixels. The trainings are repeated until the CAM converges to the desired segmentation masks.\nThe authors evaluate their approach on Pascal VOC 2012 and MS-COCO 2014, and perform multiple ablation studies to show the impact of each of the components of their method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe this method is worthy of acceptance in TMLR as it seems both technically correct and will interest the audience. I particularly appreciated the ablation study. I invite the authors to list the state of the art more clearly, and relativize their claims for surpassing baselines in light of this.\n\nHere are some additional questions that would benefit being answered in the paper:\n* What are the hyperparameters used for the CRF and IRN?\n* Figure 4. c) is confusing, as it represents the graph both before and after transformation, in different scales. I suggest to make the graph clearer by e.g. making two graphs for a clearer before / after visualization.\n* Experimental setings: how are the image-level labels selected in VOC and COCO?\n\nMinor:\npage 2 missing parentheses \"Seed Reactivation Eq. 4 is integrated\" -> \"Seed Reactivation (Eq. 4) is integrated\"\npage 5 \"Artificial Intelligent\" -> Artificial Intelligence", "broader_impact_concerns": "No broader impact concern, beyond the existing literature.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xV2pIAq7WC", "reviewer": "Reviewer_tf98", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is generally well-written, clearly structured, and quite easy to follow. \n2. The target issues of the paper are meaningful and worth exploring. The motivation is clear.\n3. This submission gives a valuable implementation of such an idea.\n\nWeaknesses:\n1. The results in Table 2 cannot demonstrate the advantage of this method. For example, GSLM+ achieves 70.6/70.6 in Val/Test, while L2G achieves 72.1/71.7.\n2.  Using colorful images in Figure 8, and Figure 9 will be better.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of generating fine-grained masks for semantic segmentation. It uses a Complementary Learning System and proposes a General-Specific Learning Mechanism. General-Specific Learning Mechanism contains a General Learning Module and a Specific Learning Module.  General Learning Module extracts localization representations from CAM.  The SLM expandsthe CAM in an explicit way.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers acknowledge that the method is simple and effective, and that the paper is clearly written. Nevertheless, the paper should be updated based on the authors' feedback to the initial reviews. The AE recommends the authors to:
- clarify the motivation for seed reactivation;
- incorporate the comparison to DRS;
- discuss and compare with the missing related works mentioned by reviewers oBpv and hhvb;
- clarify the activation loss in Eq. 5;
- discuss the hyper-parameter settings and sensitivity;
- clarify Figure 4;
- clarify the image-level experimental setting;
- fix the minor typos mentioned by reviewer oBpv;
- clarify the comparison with L2G;
- provide color versions of Figs. 8 and 9.
Decision: Accept (poster)