Disentangled Feature Swapping Augmentation for Weakly Supervised Semantic Segmentation
Authors: Junehyoung Kwon
Eunju Lee
YunSung Cho
YoungBin Kim
Venue: Submitted to ICLR 2023 (2023)
Keywords: Weakly Supervised Semantic Segmentation
Data Augmentation
Feature Disentanglement

Abstract: Weakly supervised semantic segmentation utilizes a localization map obtained from a classifier to generate a pseudo-mask. However, classifiers utilize background cues to predict class labels because of a biased dataset consisting of images, in which specific objects frequently co-occur with certain backgrounds. Consequently, the classifier confuses the background with the target objects, resulting in inaccurate localization maps. To this end, we propose DisEntangled FeaTure swapping augmentation(DEFT) to prevent the classifier from being biased by a misleading correlation. Our method first disentangles the foreground and background features. Then, we randomly swap the disentangled features within mini-batches via a two-way process. These features contain various contexts that do not appear in the biased dataset, but the class relevant representation is preserved. In addition, we introduce training schemes to obtain further performance gains. Experimental results showed that when our augmentation was used in various weakly supervised semantic segmentation methods trained on the Pascal VOC 2012 dataset, the performance of the localization maps and pseudo-mask as well as the segmentation results improved. 

Under review as a conference paper at ICLR 2023
DISENTANGLED FEATURE SWAPPING AUGMENTATION FOR WEAKLY SUPERVISED SEMANTIC SEGMENTA- TION
Anonymous authors Paper under double-bling brind review
ABSTRACT
Weakly supervised semantic segmentation utilizes localization map obtained from a classifier to generate a pseudo-mask. However, classifiers utilize back- ground cues to predict class labels because of biased dataset consisting of im- ages, in which specific objects frequently co-oceur with certain backgrounds. Consequently, the classifier confuses the background with the target objects, re- sulting in inaccurate localization maps. To this end, we proposed a disentangled feature swapping augmentation method to make the classifier focus more in- ternal objects other than on the background. Our method first disentangles the foreground and background features. Then, we randomly swap the disentangled features within mini-batches via a two-way process. These features contain vari- ous contexts that do not appear in the biased dataset, but the class relevant repre- sentation is preserved. In addition, we introduce training schemes to obtain fur- ther performance gains. Experimental results showed that when our augmentation method was used in various weakly supervised semantic segmentation methods trained on the Pascal VOC 2012 dataset, the performance the localization maps and pseudo-mask well the segmentation results improved.
INTRODUCTION
Semantic segmentation is task that classifies objects in an image in pixels. A large number of pixel- level labels are required train semantic segmentation network. However, acquiring such labels are costly and time consuming. To alleviate this problem, research on weakly supervised semantic segmentation (WSSS) is being actively conducted. WSSS uses a weak label that contains less in- formation about the location of an object than pixel-level label but has cheaper annotation cost. Examples such weaker forms of labels are image-level class labels et al. (202 by (022b)), bounding boxes(Khoreva et al. (2017); Lee et al. (2021c); (Song et al. (2019)), points( Bearman al. (2016); Kim et al. (2022)), and scribbles (Tang et al. (2018); Lin et al. (2016)). Among these weak labels, herein, we focus on the image-level class label that is the most accessible and has the lowest annotation cost.
Most of the research WSSS, utilizing image-level class labels, generate greudo-maske based on localization maps using class activation map (CAM). Therefore, the performance of the segmenta- tion network that uses the pseudo-mask as training data is dependent on the quality of the CAM. However, the classifier trained with a class label confuses the target object with the background, which in turn generates a blurry CAM(Lin al. 2016 This is due the biased dataset composed of images in which the target object frequently co-occurs with a certain background context(Geirhos al. 2020). For instance, an object corresponding to the "sheep" class always appears in "grass landscape" and the visual layout is similar in many images. Among the PASCAL VOC 2012 training datasets, more than 20% of the images with "sheep" class contain "grass landscape" as the context. The same goes for cow-grass landscape, boat-water, and aeroplane-sky pairs{Lee et al.) 2022a). Therefore, a classifier trained with a biased dataset depends not only on the target object but also on the bias, such as the background contexts, Contexts.Because Because of such misleading correlations, the classifier often assigns higher scores for background regions that are adjacent to objects or fails to activate the target object region, where such objects appear, outside of typical scenes. To mitigate this short-

---

Under review as a conference paper at ICLR 2023
coming, data augmentation method is required prevent the classifier from being overfitted to misleading correlations.
In this paper, we propose DisEntangled FeaTure swapping augmentation(herraffer we refer to DEFT) method, to alleviate the problem classifier biased with misleading correlations between the target object and the background. First, we disentangle the feature representation between the target object and the background as these features are highly entangled with each other, causing the classifier to become confused between these cues. To this end, we aggregate information about the target object and background, and use this information with explicitly defined labels to train the classifiers. Then, based on the disentangled representation, in each mini-batch, we randomly swap the background representation, while the foreground representation is fixed and vice versa. The swapped representation augmented with diverse contextual information. The classifier can fo- cus more internal objects because the dependency between the object and a specific background broken. The classifier trained using this augmented representation can effectively suppress the scores on background region and in turn yield a high-quality CAM.
The main contribution of this work can be summarized as follows:
We proposed DEFT as method to alleviate the problem that the classifier trained with the classical augmentation method suffers from false correlations between target objects and backgrounds.
Our proposed DEFT method operates in the feature space, does not require any heuristic decisions or re-training of the network, and can be easily added to other WSSS methods.
When DEFT was applied to other WSSS methods, the quality of the localization map and pseudo-mask generated through the classifier increased, and the performance of the seg- mentation network on Pascal VOC 2012 also improved.
RELATED WORK
Weakly Supervised Semantic Segmentation WSSS methods that use image-level class labels gen- erate localization map based on the initial seed CAM, and then produces a pseudo-mask through an additional refinement process. Because the initial seed identifies only the discriminative regions in the image, numerous studies have been conducted to expand such regions. AdvCAM al.) 2021b) identified more regions of objects by manipulating the attribute map through adversarial climbing of the class scores. DRS(Kim et al.) 2021a) suppresses the most discriminative region, thereby enabling the classifier to capture even the non-discriminative regions. SEAM (Wang et al.) 2020) regularizes the classifier so that the differently transformed localization maps are equivalent. AMN (Lee et al. 2022b) leverages a less discriminative part through per-pixel classification.
Further, several studies have been conducted to develop feasible methods to prevent the classifier from learning misleading correlations between the target object and the background. SIPE ct al. 2022) captured the object more accurately through a prototype modeling for the background. ICD et al. 2020a) includes intra-class discriminator that discriminates the foreground and background within the same class. W-OoD(Lee et al.) 2022a) utilizes out-of-distribution data as extra supervision to train the classifier suppress spurious cues. In addition, various studies have employed a saliency map as an additional supervision or post-processing (Lee et al. (2021e); Fan al. (2020b); Lee et al. (2019); Wei al. (2017 2018 Yao Gong (2220). Our proposed method disentangles the background information in the feature space, and thus, no additional supervision is
required.
Data Augmentation Data augmentation aims to improve the generalization ability of a classifier for unseen data by improving the diversity the training data. The image erasing method removes one or more sub-regions in an image and replaces them with zero or random values. Cutouu Vries ries & Taylor 2017) randomly masks a specific part of the image, and HiideaandSeee(Sinnn allows the classifier to seek the class relevant features after randomly hiding the patch in the image. In contrast, the image mix-based method mixes two or more images. Mixup(Zhang et al.) 2017) interpolates two images and labels, and CutMix et al. 2019) replaces certain region of an image with a patch of another image. However, because the method that uses this regional patch randomly occludes the sub-regions, including both the object and background areas, the classifier
et al. 2018)
2

---

Under review as a conference paper at ICLR 2023
trained with this method cannot distinguish the foreground from the context. Additionally, there is an augmentation using rich saliency information for combining patches Kim et al. (2021b) 2020);
Dabouei et al. (20TT). However, since these methods only the salient regions, difficult for the classifier to learn representation for non-diseriminative regions or background cues. In addition, there methods augment the representation in the feature space al. (202T): Verma al.
(019
In context decoupling augmentationn(DAA)) method is introduced to a WSSS task. CDA decouples the object and context by pasting the pseudo- mask obtained in the first stage to another image. However, this method uses single class image to obtain an accurate object instance and restricts the scale of the mask. Consequently, the diversity of the augmented representation limited. Our proposed method synthesizes features irrespective of the number or size of the object mask, and thus, it can provide a more diverse representation to the classifier.
al. 2021), the copy-and-paste augmentation
METHOD
3.1 MOTIVATION
Although the WSSS method generates localization maps using class labels, it is inappropriate to use augmentation in the classification task. Even if augmentation is used, the classifier might not distinguish the foreground and background cues owing the biased dataset. We designed toy experiment to analyze this phenomenon. First, we collected images, including the "aeroplane" class and "sky" images, which are the most frequent object-backgnnee background pairs the Pascal VOC 2012(Ev- eringham et al., 2010) dataset. The Pascal CONTEXT((Mttag)) et al.) (0144) dataset was used to determine whether "sky" was included. total of 437 images of the seroplane-sky pair were used for the toy experiment.
0.36
0.58
0.30
Image
Ground Truth
Pscudo-mask
(a)
(b)
Figure 1: (a) Plot of AFA and ABA values for each iterations (b) Examples of generated pseudo- mask when using CutMix
ResNet-50((H et al. 2015) pretrained with imagenet(Deng al. 2009) was trained the Pascal VOC 2012 dataset after applying the CutMix augmentation( et al. 2019), which is mainly used in image classification tasks. Then, extract the feature map before the last batchnorm layer in the network for each iteration. This is to prevent the activation value of the feature map from being altered due the affine parameters of batchnorm. Next, after dividing all pixels in the feature map into foreground and background regions using the pixel-level labels provided by the Pascal VOC 2012 dataset, we calculate the average activation value of each region, which are the average foreground activation (AFA) and average background activation (ABA), respectively. Since images contain objects of various sizes in our experiments, the AFA and ABA values are invariant to the object's scale. By analyzing trends in the AFA and ABA values, it is possible to determine whether foreground and background regions are spuriously correlated. As the number of training iterations increases, the classifier learns to discriminate between the foreground and background regions; that is, in the case of the classifier biased by the spurious cues, the ABA does not decrease even when AFA increases. Figure [ (a) shows the AFA and ABA values for each iteration for the toy experiment. Evidently, when the AFA value increases, the ABA value shows similar trend for increasing and decreasing.
well-trained classifier, when AFA increases, ABA decreases. However, in the case
3

---

Under review as a conference paper at ICLR 2023
This trend implies that the values of the target object and background regions are highly correlated. Figure (b) is an example of a pseudo-mask generated by a classifier trained with CutMix, and can be seen that the sky, which is the background, is erroneously identified confirm that the object and background features are spuriously correlated. The toy experiment shows that when pseudo-mask is generated using augmentation for the classification task, the classifier is biased with misleading correlations.
aeroplane. We can
3.2 DISENTANGLED FEATURE SWAPPING AUGMENTATION
Disentangling foreground and background representation. In Section 3.1, we confirmed that classical augmentation is not suitable for generating high-level localization maps. Further, the CDA(Su et al. (021)) available in the WSSS task entails complicated procedure for selecting the optimal mask to paste. Therefore, we propose DEFT, which is heuristic-free augmentation method suitable for generating localization maps. Our proposed method disentangles the foreground and background-related features and then swaps them between different training samples. First, we in- put the image z into the backbone network to obtain feature vector 2. In general, global average pooling (GAP) is used when aggregating features. However, if GAP is used, then a coarse localiza- tion map can be generated by summarizing even the bias attributes. For this reason, several existing studies have proposed a new pooling methodi[Lee et al.] (2021a); [Araslanov & Roth (2020): (Zhu al. (021I). Inspired by previous research, we aggregate foreground and background by different aggregators Mfg Mág respectively. We compute the foreground and background-red -related attention maps from the output of the backbone network, and then use them to ageregate information respec- tively. First, for the attention map calculation, we utilize output 2 E IR XXXXXI HW from the backbone network. In this case, N, C, HW each denotes the batch size, the output channel and the spatial dimension. For output we implement different 1x1 convolution functions 0(-) and ((-) to reduce the input dimension. We E IIIIIMxC, Wo E RIXIXMxC is the learnable kernel of each func- tion, which encodes information about foreground and background, respectively. Afterwards, the softmax function (-) is calculated on the spatial dimension to capture the region where the output of each function attended for the foreground and background.
f/g (o(): Ang (b(2; W)))
Asa E IR-NXXXHW Ang E NXMXXH are attention maps that activate spatial importance for and foreground and background. To aggregate different features for each pixel each attention map, M channels are introduced. Then based attention maps, aggregate features corresponding to foreground and background. The final output of the aggregator computes & and the attention map as follows:
2fg - mm 8 2), 2hg = mm(A0g 8 2)
where @ is matrix multiplication, and m(.) is a function that averages the features over the Af di- mension. Zfg Nxxx 2bg E NCxi is aggregated feature for the foreground and background respectively.
Existing studies added linear layer used another backbone to effectively separate and learn different attributes (Zhu et al.] (202T): Lee et al. (2021d)). In aligning with previous studies, we train the network by adding classifier Jog that models the background in addition to the liner layer J1g modeling the object. We obtain four classification scores by feeding two disentangled features, 2fg and Zbg. into different classifiers fig. fug. We effectively separate the two features by supervising them with different labels for each classification score. In the case of the classification score obtained by feeding (zg() to ffg- the ground truth y is used as the label. (bg(21g) is the score obtained by inputting 2ng to the classifier fuy. Therefore, the predicted score acts negative sample for the target label, but as a positive sample for class other than the target label, such that the inverse of y, i.e., 1 y, is the label. Considering the background as the inverse of the target object might be a naive approach, so it may not be appropriate to use - y as a label to guide 2bg- However, the classifier fog causes Bby to be far away from the foreground representation containing the target and non-target. Since 2hg becomes a negative sample for all the classes when the background information is aggregated, the score f (Zby) is supervised with zero-vector label
4

---

Under review as a conference paper at ICLR 2023
Output
M/p
Plg
May
swap
Figure 2: This is an overview of DEFT when only background feature swapping is implemented. The linear classifiers fig and Sug learn class relevant features and class irrelevant features, respec- tively. The disentangled representations 2/g and 2bg obtained through this procedure are swapped in the mini-bate mani-bbtch.. denotes prediction by fswap. + denotes channel-wise concatenation operation and P denotes localization maps.
yo (0, ...y 0). For the classification score (ig() Thg is a positive sample for the background, and thus, vector consisting of only is used as the label.
Edisen = =BCE(f19(31)).8)) + ABCE (26g), + (BBCE(fog(1g))..0 ABCE fbg (2bg).
During training. binary cross entropy (BCE) is used, and the loss Laisen used for feature disentangle is the same as Eq. 3 À is scalar to balance between to loss terms, which were set to 0.3. Laisen guarantees disentanglement between foreground and background features, as ffg Fbg are trained to predicted opposite labels by two classifiers fig fog- As training progresses, Efg Zbg move away from each other in representation space due to the inverse labels.
Swapping Disentangled Representation. Even if the foreground and background features are ef- fectively disentangled using the structure discussed in section 3.2, the classifier is still not free from misleading correlations because it is still trained with the biased data. Therefore, propose method that allows the classifier to learn representations out of the general context by swapping the disentangled features between the training sets.
We disentangle the features to be exclusive to each other with ffg and Ebg. 2fg is directly related to predicting the class label, whereas Zhg is correlated with the object, but it is not necessary predict the class label. In other words, when the image x is given, even if the background feature ing is changed another feature Zbg we assumed that there should be no influence on the predictions. Therefore, the optimal classifier f should output consistent predictions without being affected by bias.
f" (1z5g #bgl) = f ([2]g) 2bg-1]
(4)
That is, to achieve such invariance prediction, implement feature swapping by randomly permut- ing the foreground and background features on a mini-batch in a two-way manner. The first way is combine the class relevant attributes with the bias that do not often appear, and the second way is to combine the class irrelevant bias with the target object related attribute that does not frequently co-occur. In the first method, 2bg obtained by randomly permuting the disentangled background feature, and then concatenating it with the foreground feature to Zywapa, = [ 2bgl. In the sec- ond method, the foreground features are randomly permuted to obtain 2/g. and then the background feature is concatenated to Eswap = [z]g: Shg] The classifier is set to predict the target label by using Zswapsy and Eswapj,, as the inputs to the linear layer sswap- In the case of zswapyy since the target object is swapped on the mini-batch, the target label y is also relocated to ÿ. according to the permuted index.
Two-way swapping enables the class relevant attribute in the feature space to be combined with the class irrelevant attribute that does not appear frequently with the corresponding class. Consequently, the classifier learns the representation that does not frequently appear in the biased dataset and is thus not misled by spurious correlations. For example, when images of an aeroplane with sky and cows with grass landscape are used as the input images, the classifier can learn representations related to
5

---

Under review as a conference paper at ICLR 2023
the aeroplane with grass landscape and appearing in sky in the feature space through two-way swapping. Unlike the Su et al. 2021), our proposed method augments the representation at the feature-level, and thus, do not manually decide which mask paste to decouple the object from the context. In addition, our method has the advantage of learning more diverse representations, because features in the mini-batch Lewap for the swapped features as in Eq. 5
combined randomly at every iteration. We design the loss
Lawap = BCE (Ssaap/ (2saaaps) + BCE(Sssaphh (ssoop((ssppp))...
(5)
L = Ldisen + L Lamap
(6)
The classifier was trained with the swapped features by adding loss term in Eq. 3 with Craap Eq. denotes total loss function used for training the classifier. The overview of DEFT are shown in Figure
Training schemes. We use several training schemes to improve the performance of the classifier. First, proposed augmentation method is applied after certain iteration. We assume that the two features are disentangled, and perform swapping. However, the swapping is performed when the features are still entangled, then the classifier will be trained with wrong signals. The classi- fier preferentially learns the bias-aligned sample (e.g.. boat in water), in which there is strong correlation between the bias and labels, the beginning of the learning process, and it learns the bias- bias-conflicting sample (e.g.. boat on a railroad), in which there is a low correlation between the bias and labels, later (Nam et al. 2020). In other words, it is difficult for the classifier to distin- guish the target object related features from the bias in the early stages of learning. Therefore, the augmentation is implemented after specific iteration laug: where the two features are disentangled.
In addition, we do not update the concatenated background features based on the swap loss Lswap-
The swapped feature is supervised by the target labels and ÿ. and if the background feature is affected by this supervision, then the feature disentanglement is not done properly. Therefore, to prevent this unintended representation learning, the corresponding feature is detached, so that the swap loss does not backpropagate to the background features 2bg and 2ewapag
3.3 GENERATING SSUUDO-MASE
CAM identifies the class-relevant regions captured by the classifier within the image(Zhou et al, 2016). In general, class labels are predicted through the results of global average pooling on CAM. In this study, we fully utilize disentangled features to generate CAM. w'y and Why are the weights of the linear layers ffg. fog. respectively, and is the output of the backbone network. The localization map P is as follows:
P = max (wa 2 1-wi,
where w is the activation map for target object and w is the activation map for the background of target object. The localization map obtained by combining both activation maps. max(.) :) is the maximum value over the channels.
We resize the original image various scales to obtain high resolution localization map. Since the boundary is coarse CAM, an accurate boundary is obtained by applying the refinement method(Ahr & Kwak (2018); Ahn al. (2019)) similar other studies(Lee al. (2022b (021b Ahn et al. (2019)). We apply the IRN(Ahn al. (2019) to the localization map to produce a pseudo- mask, which than used for training segmentation network.
EXPERIMENT
4.1 EXPERIMENTAL SETUP
Dataset and evaluation metric. We used the Pascal VOC 2012 (Everingham et al. 2010) dataset with total of 21 classes (20 object categories and 1 background) and 10,582 image-level class label

---

Under review as a conference paper at ICLR 2023
images augmented by (Hariharan al. 2011) I) for the training. For the validation and testing, 1,449 and 1,456 pixel-level labels were used. We analyzed the performance of the generated localization map, pseudo-mask, and segmentation results through mloU (i.e., mean intersection over union) metric, which is generally used for evaluating segmentation results.
Implementation details. For the classifier, adopted ResNet-50(He al. 2013) pretrained on Dmng et al. 2009), and the stochastic gradient descent was used as the optimizer. The learning rate was set 0.1 the beginning of the training, and then decreased for every iteration by using a polynomial decay function. The classifier was trained for 10 epochs, and the point at which augmentation was applied was
through extensive experiments.
4.2 EXPERIMENTAL RESULT
Quantitative results of localization maps.
Table I: Comparison of localization map performance when DEFT is applied to WSSS method Pascal VOC 2012 train set.
Comparison of localization map performance.
(b) Comparison with other augmentation methods.
Method
Seed
Method
Seed
PSA Ahn & Kwak
2018
48.0
w/o augmentation
48.3
(OuFT(Ours)
51.6
Mixup Zhang al., 2017
49.0
IRN(Ahn al.
2019)
48.3
Manifold mixup Verma et al. 2019
48.7
DEFT (Ours)
52.3
Cutout (D DatooutDeVritt & Taylor 2017
48.9
AdvCAM et al.
2021b
55.6
CutMix Yun et al. 2019
49.2
+ DEFT (Ours)
57.0
CDA Su et 202T
50.8
[Lee al.
2022b
62.1
(EFT(Our))
52.3
DEFT (Ours)
64.3
Table compares the local- ization map obtained by WSSS baseline and DEFT. We evalu- ated the value mloU(%) of the initial seed (Seed) by apply- ing the proposed augmentation method to various WSSS meth- ods. This localization map can be easily applied to other meth- ods by adding the feature dis- entangling and swapping steps in the initial seed generation step. We applied our method to PSA(Ahn & Kwak 2018),
Image
Ground
Trub
IRN + DEFT CDA
Figure 3: Examples of pseudo-maske from CDA Su et al. 2021 and DEFT for PASCAL VOC 2012 images.
IRN(Ahn et al.) 2019 Adv-
CAMgLee et 2021b). and
AMN et al.) 20226). Table
shows quantitative evaluation results the localization map. The evaluation results confirmed that when the proposed method was applied, the performance improved for all the methods.
Table Th compares the performance of the localization maps obtained by applying different aug- mentations. The experimental results show that compared with the other augmentations, including the ones used in the classification and WSSS tasks, our proposed method showed the highest mloU used, the classifier becomes confused between
value. Evidently, when classical augmentation the foreground and background cue. addition, the result was 1.4%p higher than that obtained us- ing the CSA al. 2021). Based these results, can be concluded that proposed method enables the combination of object instances and contexts in the feature space.
Quantitative results pseudo-mask and segmentation network

---

Under review as a conference paper at ICLR 2023
Table Comparison of pseudo-mask and segmentation results with other WSSS methods on Pascal VOC 2012 dataset.
(a) Comparison pseudo-mask performance on the PASCAL VOC 2012 train set.
(b) Comparison of segmentation results with other
Method
Seed
Mask
WSSS methods on Pascal VOC 2012 dataset.
PSA(Ahn & Kwak
2018
48.0
61.0
IRN et al.
2019
48.3
66.3
Method
val
test
CDA
202T
50.8
67.7
PSA(Ahn & Kwak 2018
61.7
63.7
AdvCAM Lee et al.
2021b
55.6
69.9
+DEFT (Ours)
65.9
66.8
AMN(Lee et al.
2022b
62.1
72.2
IRN IRn et 2019
63.5
64.8
PSA+ DEFT (FrTT"OOrs (Ours)
51.6
64.2
DEFT DEFT(OUrs)
69.1
68.7
IRN+ DEFT (Ours)
52.3
68.6
AMN+ DEFT (Ours)
64.3
72.8
Table 2a compares the initial seed(Seed) and the pseudo-mask(Mask, obtained by the WSSS base- line and our proposed methods. When our method was applied to different baselines(Ahn Kwak (2118; Ahn et al. (2019): Lee et al.] (20226)), their performances showed considerable improve- ments in all the cases. In particular, when we applied our method on AMNeLee et al. 2022b), an mloU value of 72.8% was achieved for the aseudo-mask, and this value exceeded those obtained using the previous methods. In the case of IRN(Ahn et al. 2019), the result was 2.3 %pp%p ahead of the existing one. Figure B shows examples of the pseudo masks generated through the CDA(Su et al. 2021], IRN, and the proposed method. These examples also show that compared to the CDA, our method captures the target object accurately and does not mis-assign the background to the foreground.
Table 26 shows the results of training the segmentation network with the pseudo- mark obtained by applying the proposed augmentation method to PSA(Ahn & Kwak 2018) and IRN(Ahn et al., 2019), and the evaluated mloU value for the Pascal VOC 2012 val and test sets. The experimental results obtained using the val and test sets were 4.2%p and 3.1%p, respectively, were higher than those obtained using the PSA, and 5.6%p and 3.9%p higher than those obtained using the IRN. These observations confirmed that our proposed method can be effectively applied to the previous WSSS methods and can notably boost their performance.
4.3 ANALYSIS AND DISCUSSION
Analysis for spurious correlations
Table 3: Comparison of the mIoU values for the images with a high co-occurrence ratio in the Pascal
VOC 2012 training set.
aeroplane (w/ sky)
sheep (w/ grass)
cow (w/ grass)
boat
train (w/ railroad)
class
(w/ water)
co-occurence ratio
0.23
0.22
0.2
0.18
0.11
IRN(Ahn al. 2019)
83.72
85,95
86.24
75.05
68.83
DEFT (Ours)
87.34
86.01
87.79
76.36
74.69
The mloU values presented in Table 3 were obtained for images corresponding to frequently ap- pearing object classes and background pairs. First, identify the pairs that frequently appear in the training dataset, the PASCAL CONTEXT dataset was used. This dataset includes labels for vari- ous contexts, including 20 class labels of Pascal VOC 2012. We sorted the ratio of context labels that appear together by class label of Pascal VOC 2012 in descending order, and selected the most frequent pair of combinations. The target objert-background combinations were aeroplane-sky, grass, cow-grass, boat-water, and trainarailroad. The co-occurrence ratio denotes the ratio in which the corresponding background appears among the images corresponding to a specific class label. For example, in the entire aeroplane image, the sky coincides by more than 23%. The removal of harmful correlations was confirmed by evaluating the performance of the images of the above- mentioned combinations. All the combinations showed results than were higher than those obtained
8

---

Under review as a conference paper at ICLR 2023
using the IRN et al. 2019). Thus, it can be confirmed that our proposed method can effectively alleviate the spurious correlation problem caused by a biased dataset.
Ablations for loss terms localization maps
We performed an ablation study for each loss term as shown in Table 4 baseline denotes the baseline loss without the proposed augmentation method. Stmap and wa are the first and second terms of Lymaps indicating the losses for the background swapping and foreground swapping, respectively. (a) is the baseline result. The performance improvement by 1.5%pp for
Table 4: Effectiveness of each loss on the localization map.
Loss
(a)
(b)
Seed
baseline
48.0
disen
/
49.8
C
/
/
50.9
(a) -G (b) implies that, benefiting from feature disen- tanglement, the classifier focuses more on the target object region. (b) -ú (c) is adding L indicating that randomly swapping the background features is ef-
awap
C
/
52.3
S1oop
fective. The improvement along (c) -t (d) reflects the importance of two-way swapping.
Manifold visualization
foroground backkround
Figure 4: Visualization of the features corresponding to the target class and background by using the T-SNE(Van der Maaten & Hinton 2008) dimension reduction method.
We performed manifold visualization by using T-SNE(Van der Maaten & Hinton 2008 which is dimensionality reduction method, to assess the utility of the disentangled foreground and back- ground features. We used the intermediate features for the features 2/ and 2hg obtained using the aggregators Mfg and Mbg Figure reveals that the features related to the foreground and back- ground are semantically different.
5 CONCLUSION
We confirmed that the WSSS performance degrades because the classifier trained with a biased dataset that contains an image which a specific target object frequently appears the background utilizes background cues to identify objects. In addition, we revealed that the existing augmentation methods cannot effectively resolve this bias. To alleviate this problem, we proposed an augmenta- tion method that disentangles the target object and background-related features in the feature space and swaps them a mini-batch. We confirmed that the performance of the localization map and pseudo-mask, obtained through the classifier, was improved in the Pascal VOC 2012 dataset upon the application of our proposed method. Furthermore, the misleading correlation was effectively removed through the improved results obtained from the experiment performed on the class with high co-occurrence ratio. In the future, we plan to adopt metric learning for a more effective feature disentanglement.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jiwoon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4981 1-990, 2018.
Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 2209-2218, 2019.
Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4253- 4262, 2020.
Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. What's the point: Semantic segmentation with point supervision. In European conference on computer vision, pp. 549-565. Springer, 2016.
Qi Chen, Lingxiao Yang. Jian-Huang Lai, and Xiaohua Xie. Self-supervised image-specific proto- type exploration for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Ali Dabouci, Sobhan Soleymani, Fariborz Taherkhani, and Nasser M Nasrabadi. Supermix: Super- vising the mixing data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13794-13803 13803, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: /708. 2017.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Junsong Fan, Zhaoxiang Zhang, Chunfeng Song, and Tieniu Tan. Learning integral objects with intra-class discriminator for weakly-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4283-4292, 2020a.
Junsong Fan, Zhaoxiang Zhang, Tieniu Tan, Chunfeng Song, and Jun Xiao. Cian: Cross-image affinity net for weakly supervised semantic segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10762-10769, 2020b.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11 ((11):665667) 2020.
Bharath Hariharan, Pablo Arbeléez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Se- mantic contours from inverse detectors. In 20/1 international conference computer vision, pp- 991-998. IEEF, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learningfor image recog- nition. ComputerScience, 2015.
Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 876-885, 2017.
Beomyoung Kim, Sangeun Han, and Jummo Kim. Discriminative region suppression for weakly- supervised semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pp. 1761, 2021a.
10

---

Under review as a conference paper at ICLR 2023
Beomyoung Kim, Youngjoon Yoo, Chae Eun Rhee, and Junmo Kim. Beyond semantic to instance segmentation: Weakly-supervised instance segmentation via semantic knowledge transfer and self-refinement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2022.
Lang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pp. 5275-5285. PMLR, 2020.
Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. arXiv preprini urXiv:2102.0300 2021b.
Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon. Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5267-5276, 2019.
Jungbeom Lee, Jooyoung Choi, Jisoo Mok, and Sungroh Yoon. Reducing information bottleneck for weakly supervised semantic segmentation. Advances in Neural Information Processing Systems, 14227408-2742,, 2021a.
Jungbeom Lee, Eunji Kim, and Sungroh Yoon. Anti-adversarially manipulated attributions for weakly and supersupe semantic segmentation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 4071-4088 2021b.
Jungbeom Lee, Jihun Yi, Chaehun Shin, and Sungroh Yoon. Bbam: Bounding box attribution map for weakly supervised semantic and instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2643-2652, 2021c
Jungbeom Lee, Seong Joon Oh, Sangdoo Yun, Junsuk Choe, Eunji Kim, and Sungroh Yoon. Weakly supervised semantic segmentation using out-of-distribution data. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16897-16906, 2022a.
the
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 34:25123-22 233-23333,,,, 2021d.
Minhyun Lee, Dongseob Kim, and Hyunjung Shim. Threshold matters in wsss: Manipulating the activation for the robust and accurate segmentation model against thresholds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4330-4339, 2022b.
Seungho Lee, Minhyun Lee, Jongwuk Lec, and Hyunjung Shim. Railroad is not train: Saliency as pseudo-pixel supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5495-5505, 2021e.
Soon Hoe Lim, N Benjamin Erichson, Francisco Utrera, Winnie Xu, and Michael W Mahoney. Noisy feature mixup. arXiv preprint arXiv:2110.021800 2021.
Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervised convolu- tional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3159-3167. 201.-316...0111 2016.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic seg- mentation in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 891-898, 2014.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De- biasing classifier from biased classifier. Advances Neural Information Processing Systems, 33:20673-20633 2020.
Krishna Kumar Singh, Hao Yu, Aron Sarmasi, Gautam Pradeep, and Yong Jae Lee. dide-and-sed A data augmentation technique for supery-- localization and beyond. arXiv preprint arXiv: 181 1.05455, 2018.
11

---

Under review as a conference paper at ICLR 2023
Chunfeng Song, Yan Huang, Wanli Ouyang, and Liang Wang. Box-driven class-wise region mask- ing and filling rate guided loss for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3136-3164 2019.
Yukun Su, Ruizhou Sun, Guosheng Lin, and Qingyao Wu. Context decoupling augmentation for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF international con- ference on computer vision, pp. 7004-7014. 2021.
Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, and Christopher Schroers. Nor- malized cut loss for superl-- cnn segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1818-1827, 2018.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp. 6438-6447. PMLR, 2019.
Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. Self-supervised equivari- ant attention mechanism for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12275-12284, 2020.
Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. Ob- ject region mining with adversarial erasing: A simple classification to semantic segmentation approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1568-1576, 2017.
Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jie, Jiashi Feng, and Thomas S Huang. Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7268- 7277, 2018.
Qi Yao and Xiaojin Gong. Saliency guided self-attention network for weakly and semi-supervised semantic segmentation. IEEE Access, 8: 8:14413-14422 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. Proceed- ings of the IEFF/CVF international conference computer vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09411, 2017.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Lei Zhu, Qi She, Qian Chen, Xiangxi Meng, Mufeng Geng, Lujia Jin, Zhe Jiang, Bin Qiu, Yunfei You, Yibao Zhang, et al. Background classification activation map for weakly supervised object localization. arXiv preprint arXiv:2112. 14379, 2021.
12


Reviews:
[{"review_id": "uv5OprkVHcu", "reviewer": "Reviewer_3SEq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To alleviate the problem that the classifier confuses the background with the target objects, this paper proposes to disentangle the feature representation between the target object and the background,  and swaps the background representation while the foreground representation is fixed and vice versa. The classifier trained using this augmented representation can generate more accurate localization maps. ", "summary_of_the_review": "The paper is well-motivated and clear, and the experimental results verify the effectiveness of the proposed augmentation method. However, the authors should clarify the differences from the works.\nAfter a discussion with other reviewers, I decide to change my score to 5 since the idea of this paper is similar to Lee et al.(2021) and saliency-based argumentation methods.", "strength_and_weaknesses": "Strength:\n1. The paper is clear and well-written.\n2. The paper is well-motivated.\n3. Extensive experimental results showed that when the proposed augmentation method was used in various weakly supervised semantic segmentation methods, the performance of the localization maps and pseudo-mask and the segmentation results improved.\n\nWeaknesses:\n1. The description of the dataset is not clear. For example, it is better to clarify whether the results in Table 2 (a) is evaluated on the test set or validation set.\n2. The novelty is somewhat weak. There are some works[1,2,3] that use saliency-based mixup data augmentation. Please discuss the differences from these works.\n\n[1] Kim J H, Choo W, Song H O. Puzzle mix: Exploiting saliency and local statistics for optimal mixup[C]//International Conference on Machine Learning. PMLR, 2020: 5275-5285.\n[2] Kim J H, Choo W, Jeong H, et al. Co-mixup: Saliency guided joint mixup with supermodular diversity[J]. arXiv preprint arXiv:2102.03065, 2021.\n[3] Dabouei A, Soleymani S, Taherkhani F, et al. Supermix: Supervising the mixing data augmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 13794-13803.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mv3FRwYrYGu", "reviewer": "Reviewer_vkH3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a feature augmentation method to improve the performance of the WSSS task. The problem with the existing methods is that the pseudo groundtruth for the specific foreground (fg) class also includes information of a highly correlated background (bg). In order to completely separate this background information, the authors divide the features and classifiers for fg and bg and make a loss for each combination. As an augmentation method, these separated fg/bg features are swapped between images in the mini-batch. The proposed method was applied to existing WSSS methods to improve performance.", "summary_of_the_review": "There is weak evidence on the motivation and method claimed by the authors. Also, the performance comparison was not made sufficiently. As a result, my initial rating is weak reject.", "strength_and_weaknesses": "### Strengths\n1. The paper is clearly written and easy to follow with detailed equations and figures.\n2. Many analyzes were performed to explain motivation and results.\n\\\n&nbsp;\n\n### Weaknesses\n\n**1. Motivation** \\\nIt is hard to understand the motivation (in Section 3.1) by Figure 1. The authors claim that backgrounds highly correlated with objects are included in pseudo groundtruth. However, it seems that the pseudo groundtruth is not just fine-grained and contains the surrounding area of the object. When we see the CAM results, the values are not high in the background that is far away from the object. For Figure 1a, I guess the reason ABA and AFA have similar values is that the surrounding pixels are included in proportion to the object's size. Also, for the qualitative results of the baseline method in Figure 3, it is difficult to find a case where the background far away from the object is recognized as a foreground class.\n\n**2. Disentanglement loss (Eq. 3)** \\\nBasically, it is hard to understand why this loss can disentangle the fg and bg features well. As with existing methods, even if a highly correlated background (for a specific class) is included in the fg feature, this loss seems to work properly. Of course, I agree that the proposed architecture has the potential to operate in the ideal direction (as the authors claimed). However, the training method seems insufficient to make this possible.\n\nAdditionally, the following questions exist.\n- Why is (1-y) the target for the result of f_fg(z_bg)? I think there should be no information for the foreground objects in the background feature, so it is hard to understand why f_fg should predict as 1 for classes that are not in the image.\n- Also, why do y_zero or y_one have to be vectors with the same shape as y? f_bg is a classifier that simply predicts whether the input feature is background or not, so it seems the scalar value is sufficient as an output.\n\n**3. Performance comparison** \\\nPerformance comparison with the recent methods is lacking. The mIoU values for the actual WSSS task are summarized in Table 2b, but only two outdated methods are listed. (Also, only IRN (2019) is compared in Table 4.) The performance of the proposed method (68.7) is far behind the SOTA methods below.\n- RCA + EPS: 72.8 [A]\n- Puzzle- CAM: 72.2 [B]\n- SPML: 71.6 [C]\n\n[A] Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentation, CVPR 2022. \\\n[B] Puzzle-CAM: Improved localization via matching partial and full features, ICIP 2021. \\\n[C] Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning, ICLR 2021.\n\\\n&nbsp;\n\n### Other Comments\n“For the evaluation and testing, 1,449 and 1,456 pixel-level labels were used.”: evaluation &rarr; validation", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "UEcM8IyzD6w", "reviewer": "Reviewer_sGvK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a data augmentation technique for weakly supervised semantic segmentation. It is well known that previous weakly supervised semantic segmentation works have an issue of spurious correlation between foreground and background. This paper proposes the augmentation technique by swapping the disentangling features. The proposed techniques improve several previous techniques.", "summary_of_the_review": "The paper is easy to understand and straightforward. However, my major concerns are 1) whether the disentangling can be achieved as we intended and 2) the marginal contribution over previous work. ", "strength_and_weaknesses": "Strength: \n\n+ The paper is overall well-written and easy to understand. \n\n+ The authors address the important problem in weakly supervised semantic segmentation.\n\n+ The proposed technique improves several previous works. \n\nWeakness:\n\n- Motivation in Section 3.1 is not convincing for me. \n\n1) The correlation between AFA and ABA seems trivial, because both values are determined by the scale of network output. The affine parameters of batchnorm determine the scale of activation scale, so AFA and ABA may be determined by those affine parameters. For example, if we split the image into two regions (left-half and right-half) and plot the average activation score of each region, the two scores will be also correlated.\n\n2) In my opinion, the problem in Figure 1(b) is not spurious correlation, but the smoothness of CAM. \n\n3) If the authors would like to check if the augmentation for the classification task brings misleading correlation, the baseline without any augmentation should be compared.\n\n- The overall structure is too similar to Zhu et al. (2021). In addition, it is not clear the disentangling can be achieved as we intended. For example, if the network thinks the grass is a part of cow, z_fg can be {cow, grass} and z_bg  can be {other patterns}. \n\n- Following recent works, experiments on the MS COCO dataset should be conducted.\n\n- Minor points\n\n1) If I understand correctly, in Eq (3), f_fg(z_bg) and f_bg(z_fg) should be changed.\n\n2) In the last part of Section 3.2, the sentence \"an unintended representation can be learned\" is ambiguous. Can you present more details?\n\n3) In Table 4, I recommend the authors to include improvements for all classes. In particular, I want to see the performance change for 'train' class, because 'train' has a significant correlation with 'rail'. \n\n4) Table 3 seems strange. What is the meaning of (a, b, c, d)? \n\n5) In Section 3.2, why M channels should be introduced? Can you present the effectiveness of values of M?\n\n6) I want to see the results of AMN+DEFT in Table 2(a).\n\n7) Please consider to include the T-SNE of IRN in Figure 4. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AMNgXogFDrC", "reviewer": "Reviewer_LuUC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the phenomenal that WSSS performance will degrade with dataset biases whee a specific target object frequently appears in the background. To resolve it, the paper proposed an augmentation method that disentangles the target object and background-related features, such that the localization map will more focus on the real foreground. Experimental results show that the proposed module can largely improve the mIoUs on top of various existing approaches.", "summary_of_the_review": "The paper proposed an augmentation method that disentangles the target object and background-related features, such that the localization map will more focus on the real foreground. The work demonstrates an interesting idea and consistent improvements on existing WSSS approaches. Still, many questions/concerns exists. I am currently in the borderline position, and would like to render a final decision upon the author's response. \n\nPost rebuttal/Discussion:\nThe authors' rebuttal/revision has partially address several of my points, including the AFA/ABA descriptions, the formulations of equations. However, there are serval concerns remaining: 1. For the idea, CAM is based for classification which explores both fg and bg. Thus, I am not sure whether the CAM itself can be well-trained if they are randomized with this augmentation.  2. As pointed by other reviewers, the presentation is not perfect and still has potential to be significantly improved. 3. Performance concern, as pointed by sGvK. Thus, I'd lower my rating to 5.", "strength_and_weaknesses": "Strength:\n+ The author provide a detailed explanation to the unresolved issue for data augmentation of current WSSS approaches (based on CAM).\n+ The proposed idea is simple and reasonable. Improvements are obvious and consistent.\n+ The two-way swapping idea is also interesting and reasonably improved the performance.\n\nWeaknesses:\n- The overall idea is similar to Lee et al. (2021d) despite with different tasks.\n- I am not 100% convinced by the paper being motivated as \"biased data leading to degraded WSSS performance\". My understanding is that the CAM will attend to the background since classification also utilize the background context, so this paper proposed an approach to avoid such \"leaking\" and to have more accurate foreground seeds. The goal here is very different as Lee et al. (2021d) that de-bias to increase the diversity and improve the generalization. I am not sure if describing it as de-bias is a good choice here.\n- There are many unclear presentation that need to be improved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject