Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation
Authors: Jungbeom Lee
Jooyoung Choi
Jisoo Mok
Sungroh Yoon
Venue: NeurIPS 2021 Poster (2021)
Keywords: semantic segmentation
weakly supervised learning
computer vision

Abstract: Weakly supervised semantic segmentation produces pixel-level localization from class labels; however, a classifier trained on such labels is likely to focus on a small discriminative region of the target object. We interpret this phenomenon using the information bottleneck principle: the final layer of a deep neural network, activated by the sigmoid or softmax activation functions, causes an information bottleneck, and as a result, only a subset of the task-relevant information is passed on to the output. We first support this argument through a simulated toy experiment and then propose a method to reduce the information bottleneck by removing the last activation function. In addition, we introduce a new pooling method that further encourages the transmission of information from non-discriminative regions to the classification. Our experimental evaluations demonstrate that this simple modification significantly improves the quality of localization maps on both the PASCAL VOC 2012 and MS COCO 2014 datasets, exhibiting a new state-of-the-art performance for weakly supervised semantic segmentation.

Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation
Jungheom Lee!
Jooyoung Choi
Jisoo Mok' Sungroh Yoon 2
Department of Electrical and Computer Engineering, Seoul National University 2 Interdisciplinary Program in Artificial Intelligence, Seoul National University {jbeom. 1ee93, jy_choi, magicshop1118, sryoon) Sanu ac.kr
Abstract
Weakly supervised semantic segmentation produces pixel-level localization from class labels; however, a classifier trained on such labels is likely to focus on a small discriminative region of the target object. We interpret this phenomenon using the information bottleneck principle: the final layer of a deep neural network, activated by the sigmoid or softmax activation functions, causes an information bottleneck, and result, only subset of the task-relevant information is passed on to the output. We first support this argument through a simulated toy experiment and then propose a method to reduce the information bottleneck by removing the last activation function. In addition, we introduce a new pooling method that further encourages the transmission of information from dindiicrimiiaeive regions to the classification. Our experimental evaluations demonstrate that this simple modification significantly improves the quality of localization maps both the PASCAL VOC 2012 and MS COCO 2014 datasets, exhibiting a new state-of- the-art performance for weakly supervised semantic segmentation. The code is available at: https:/ /// //t/t/gitttt gigthu....... con/ www.bmeeee////eeeeme
1 Introduction
Semantic segmentation is the task of recognizing objects in an image using pixel-level allocation of a semantic label. The development of deep neural networks (DNNs) has led to significant advances in semantic segmentation [9, 24]. Training a DNN for semantic segmentation requires dataset containing a large number of images annotated with pixel-level labels. However, preparing such dataset requires considerable effort; for example, producing a pixel-level annotation for a single image in the Cityscapes dataset [12] takes more than 90 minutes. This high dependence on pixel-level labels can be alleviated by weakly supervised learning [62, 31, 34, 2].
The objective of weakly supervised semantic segmentation is to train segmentation network with weak annotations, which provide less information about the location of a target object than pixel- level labels, but cheaper obtain. Weak supervision takes the form of scribbles [52], bounding boxes [50, 29, 34], or image-level class labels [31, 3, 2]. In this study, we focus on image-level class labels, because they are the cheapest and most popular option of weak supervision. Most methods that use class labels [31, 32, 55, 2, 7] generate pseudo ground truths for training a segmentation network using localization (attribution) maps obtained from trained classifier, such CAM [66] or a Grad-CAM [48]. However, these maps identify only small regions of target object that are discriminative for the classification [31, 2, 5] and do not identify the entire region occupied by the object, making the attribution maps unsuitable for training a semantic segmentation network. We interpret this phenomenon using the information bottleneck principle [54, 49, 46, 53].
"Correspondence to: Sungroh Yoon <sryoon@snu.ac.kr an
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

The information bottleneck theory analyzes the information flow through sequential DNN layers: information regarding the imput is compressed as much as possible as passes through the layers of a DNN, while preserving as much of the task-relevant information as possible. This is advantageous for obtaining optimal representations for classification [15, 1] but is disadvantageous when apply- ing the attribution maps from the resulting classifier to weakly supervised semantic segmentation. The information bottleneck prevents the non-diseriminative information of the target object from being considered in the classification logit, and thus, the attribution maps focus on only the small discriminative regions of the target object.
We argue that the information bottleneck becomes prominent in the final layer of the DNN due to the use of the double-sided saturating activation function therein (e.g. sigmoid, softmax). We propose a method to reduce this information bottleneck in the final layer of the DNN by retraining the DNN without the last activation function. Additionally, we introduce a new pooling method that allows more information embedded in non-discriminative features, rather than discriminative features, to be processed in the last layer of a DNN. As a result, the attribution maps of the classifier obtained by our method contain more information on the target object.
The main contributions of this study are summarized as follows. First, we highlight that the infor- mation bottleneck occurs mostly in the final layer of the DNN, which causes the attribution maps obtained from a trained classifier to restrict their focus to small discriminative regions of the target object. Second, we propose a method to reduce this information bottleneck by simply modifying the existing training scheme. Third, our method significantly improves the quality of the localization maps obtained from a trained classifier, exhibiting a new sfte-oofthhe-art performance on the PASCAL VOC 2012 and MS COCO 2014 datasets for weakly supervised semantic segmentation.
2 Preliminaries
2.1
Information Bottleneck
Given two random variables X and Y. the mutual information I(X:Y) Y)) quantifies the mutual dependence between the two variables. Data processing inequality (DPI) [13] infers that any three variables X. Y. and that form a Markov Chain X Y + Z satisfy Z(X; 2 I(X; Z). Each layer in DNN processes the input only from the previous layer, which means that the DNN layers form a Markov chain. Therefore, the information flow through these layers can be represented using DPI. More specifically, when an L L-layered DNN generates an output Y from a given input X through intermediate features T/ (1 < 1 & L), it forms a Markov Chain X - Ti + .... + Tr. + Y, and the corresponding DPI chain can be expressed follows:
I(X: 2 T2); 2 ... 2 I(X;TT-1) 2 I(X;TL) 2 ((x:P).
This implies that the information regarding the input X is compressed as it passes through the layers of the DNN.
Training a classification network can be interpreted as extracting maximally compressed features of the input that preserve as much information as possible for classification; such features are commonly referred to as minimum sufficient features (i.e., discriminative information). The minimum sufficient features (optimal representations T*) can be obtained by the information bottleneck trade-off between the mutual information of X and T (compression), and that of T and Y (classification) [54, 15]. In other words, T argming I(X;T)- - (I; Y), where 3 2 0 a Lagrange multiplier.
Shwartz-Ziv et al. [49] observe a compression phase in the process of finding the optimal represen- tation I*: when observing I(X, Ti) for fixed I(X, Ti) steadily increases during the first few epochs, but decreases in the later epochs. Saxe et al. [46] argue that the compression phase is mainly observed in DNNs equipped with double-sided saturating non-lincasities (e.g., tanh and sigmoid), and is not observed in those equipped with single-sided saturating non-linearities (e.g., ReLU). This implies that DNNs with single-sided saturating non- -ineeiieassties experience less information bottleneck than those with double-sided saturating non-linearisies. This can also be understood in terms of gradient saturation in the double-sided saturating con-linearities: the gradient of those non-linearities with respect to an input above a certain value saturates close to zero [8]. Therefore, features above a certain value will have near-zero gradients during the back-propagation process and be restricted from additionally contributing to the classification.
2

---

2.2 Class Activation Mapping
class activation map (CAM) [66] identifies regions of an image focused by a classifier. The CAM is based on a convolutional neural network with global average pooling (GAP) before its final classification layer. This is realized by considering the class-specific contribution of each channel of the last feature map to the classification score. Given a classifier parameterized by 0 = (0,, w) where f+) is the feature extractor prior to GAP, and w is the weight of the final classification layer, a CAM of the class c is obtained from an image as follows:
w f (x; 0y) CAM 0) = max w! f (x; 0y)
(2)
where max(-) the maximum value over the spatial locations for normalization.
2.3 Related Work
Weakly Supervised Semantic Segmentation: Weakly supervised semantic segmentation methods with image-level class labels first construct an initial seed by obtaining high-quality localization map from a trained classifier. Erasure methods [56, 37, 22] prevent a classifier from only focusing on the discriminative parts of objects by feeding the image from which the discriminative regions have been erased to the classifier. Several contexts of a target object can be considered by combining multiple attribution maps obtained from differently dilated convolutions [31, 57] or from the different layers of DNN [35]. Diverse images of a target class can be utilized by considering cross-image semantic similarities and differences [17, 51J. Zhang el al. [62] analyze the causalities among images, contexts, and class labels and propose CONTA to remove the confounding bias in the classification. Because the localization maps obtained by a classification network cannot accurately represent the boundary of a target object, the initial seed obtained by the above methods is refined using subsequent boundary refinement techniques such as PSA [3] and IRN [2].
Information Bottleneck: Tishby et al. [54] and Shwartz et al. [49] use the information bottleneck theory to analyze the inner workings of DNN. The concept of the information bottleneck has been employed in many research fields. Dubois et al. [15] and Achille et al. [1] utilize the informa- tion bottleneck to obtain optimal representations from DNNs. DICE [45] is proposed for model ensemble with the information bottleneck principle: it aims to reduce not only the unnecessary mutual information between features and inputs but also the redundant information shared between features produced by separately trained DNNs. Jeon et al. [26] study the disentangled representa- tion learning of generative model [19] using the information bottleneck principle. Yin et al. [60] design a regularization objective based on information theory to deal with the memorization problem in meta-learning. The information bottleneck principle can also be adopted generate the visual saliency map of classifier. Zhmoginov et al. [65] find important regions for the classifier with the information bottleneck trade-off, and Schulz et al. [47] restrict the information flow by adding noise to intermediate feature maps and quantify the amount of information contained in the image region.
3 Proposed Method
Weakly supervised semantic segmentation methods using class labels produce a pixel-level localiza- tion map from classifier using CAM [66] or Grad-CAM [48]; however, such map identifies only small discriminative regions of the target object. We analyze this phenomenon with the information bottleneck theory in Section 3.1 and propose RIB, method to address this problem, in Section 3.2. We then explain how we train a segmentation network with localization maps improved through RIB in Section 3.3.
3.1 Motivation
As mentioned in Section 2.1, the DNN layers with double-sided saturating con-linearities have a larger information bottleneck than those with single-sided saturating non- conelineas.sis. The intermediate layers of popular DNN architectures (e.g., ResNet [21] and DenseNet [23]) are coupled with the ReI.U activation function, which is single-sided saturating non-linearity. However, the final layer of these networks is activated by a double-sided saturating non-linearity such as sigmoid or softmax, and the class probability p is computed with the final feature map TŁ and the final classification
3

---

2
HGR
8
8
8
8
8
8
Disormi--aiive
Non Discrimina
Irelecan:
% 10 %
Image
G1
G2
G3
G4
G5
G6
(a) Dataset
(b) Gradient Examples
(c) HGR values
Figure I: (a) Examples of toy images. (b) Examples of gradient maps Gk- (C) Plot of HGR values of
RD. RND, and RBG for each layer, averaged over 100 images.
layer w. i.c., = sigmoid(w:Taar G)P(TT.)) Therefore, the final layer parameterized by w has significant bottleneck, and the amount information transmitted from the last feature Tz to the actual classification prediction will be limited.
These arguments are analogous to the observations in existing methods. The information plane provided by Saxe et al. [46] shows that the compression of information is more noticeable in the final layer than in the other layers. Bae et al. [5] observe that although the final feature map of the classifier contains rich information on the target object, the final classification layer filters out most of it; thus, the CAM cannot identify the entire area the target object. This observation empirically supports the occurrence of the information bottleneck in the final layer of DNN.
To take closer look at this phenomenon, we design toy experiment. We collect images containing the digits '2' or "8' from the MNIST dataset [30]. For only small subset (10%) of these images, we add circle and a square to the images containing the digits '2' and '8', respectively. at random location (see Figure (a)). When classifying images into the digits '2' or '8', pixels corresponding to the digit are discriminative regions (Ro), those corresponding to the added circle or square are non-discriminative but class-relevant regions (RND), and those corresponding to the background are itrse-irreeevant regions (Rac).
We train a neural network with five convolutional layers followed by a final fully connected layer. We obtain the gradient map G, of each feature Ti with respect to an input image æ: G = V== E.,e Ti(u, v), where u and v are the spatial and channel indices of the feature Ti, and for the final classification layer (1 = 6), G. = Vayf Because this gradient map indicates the extent to which each pixel of the image affects each feature, it can be used examine how much information is passed from the input image to the feature maps of successive convolution layers.
We present examples of G in Figure 1(b). As input image passes through the convolution layers, the overall amount of gradient with respect to the input decreases, indicating the occurrence of the information bottleneck. Specifically, the gradient of RgG decreases early on (G1 + G2), which implies that the insk-irrelevant information is rapidly compressed. From G, to G5. the gradient in RD or RND gradually decreases. However, the decrease in the amount of gradient is prominent in the final layer (G5- - Go), and in particular, the gradients in RND (red boxes) almost disappear. This supports our argument that there is significant information bottleneck in the final layer of DNN, while also highlighting that the non- non-discrmiinative information in RND is particularly compressed.
We analyze this quantitatively. We define the high gradient ratio (HGR) of region R as the ratio of pixels that have a gradient above 0.3 to the total pixels in region R. HGR quantifies the amount of transmitted information from region R of input image to each feature. The trend in the HGR values of each region for each layer is shown in Figure Icc). The observed trend is analogous to the above empirical observation, once again supporting that significant information bottleneck for RND occurs in the final layer (the red box).
We argue that the information bottleneck causes the localization map obtained from trained classifier to focus on small regions of the target object. According to Eq. 2, the CAM only includes information that is processed by the final classification weight w.. However, because only a subset of the information in the feature is passed through the final layer we due to the information bottleneck. leaving out most of the non-discriminative information, CAM cannot identify the non-discriminative regions of the target object. It is undesirable to use such CAMs to train a semantic segmentation network, for which the entire region of the target object should be identified. Therefore, aim to bridge the gap between classification and localization by reducing the information bottleneck.

---

3.2 Reducing Information Bottleneck
In Section 3.1, we observed that the information contained in an input image is compressed particularly in the final layer of the DNN, due to the use of the double-sided saturating activation function therein. Therefore, we propose method to reduce the information bottleneck of the final layer by simply removing the sigmoid or softmax activation function used in the final layer of the DNN. We focus on multi-class multi-label classifier, which is the default setting for weakly supervised semantic segmentation. Suppose we are given an input image I and the corresponding one-hot class label t t= [41, ............ tc), where le {0, 1) I f C) is an indicator of class c, and C is the set of all classes While existing methods use the sigmoid binary cross-entropy (BCE) loss (CBCE) to train a multi-label classifier, our method replaces it with another loss function LRu that does not rely on the final sigmoid activation function:
LBCE Σ te log sigmoid(y") (1 te) log(1 sigmoid(y)), CRBB
Σ te min(m, y)),
where m is a margin, and y is the classification logit of image x.
However, training classifier with CRB from scratch causes instability in the training because the gradient cannot saturate (please see the Appendix). Therefore, we first train an initial classifier with LaCF whose trained weights are denoted by Oo. and for given image we adapt the weights toward a bottleneck-fres model of x. Specifically, we fine-tune the initial model using LRI computed from and obtain model parameterized by 0g (0 < l S K), where 0₂ 0x-1 A
AV Vor_: LRIB. and K and À are respectively the total number of iterations and the learning rate for fine-tuning. We name this fine-tuning process RIB. Employing RIB reduces the information bottleneck for and we can obtain CAMs that identify more regions of the target object, including non-discriminative regions. We repeat the RIB process for all the training images to obtain the CAMs.
However, the model that adapted further stabilize the RIB process, we construct a batch of size B for RIB by sampling random B - samples other than x at each RIB iteration. Note that for each iteration, B- samples are randomly selected, while a is fixed.
given image r can be easily over-fitted to x. Therefore, to
We demonstrate the effectiveness of RIB by applying it to the same classifier as that used for the toy experiments described in Section 3.1. Figure 2 presents (a) examples of G2 and (b) the HGR values for RD. RND. and RaG of Gg. which showed the most sig- nificant information bottleneck, at each RIB iteration. The HGR values are aver- aged over 100 images. The HGR values of RBG remain fairly constant during the RIB process, while the HGR values of RD and RND increase significantly. This indicates that the RIB process can indeed reduce the information bottleneck, thereby ensuring that more information corresponding to both Rd and RND is processed by the final classification layer.
Effectiveness of RIB:
(a)
(b)
Dicriminstive
Non. Dicrminaan Inceveett
HGR
Image
kg
k=
k=9
RIB Iteration (k)
Figure 2: Analysis of Gg for Rp, RND. and RBG at cach RIB iteration.
Limiting the transmission of information from discriminative regions: Zhang et al. [64] showed the relationship between a classification logit y and a CAM, i.e., y = GAP(CAM). This implies that increasing y" with RIB also increases the pixel values in the CAM. For CAM to identify wider area of the target object, it is important to increase the pixel scores of the non-discriminative regions, rather than the discriminative regions. Therefore, we introduce a new pooling method to the RIB process, so that the features that were previously delivering small amount of information to the classification logit contribute more to the classification.
We propose global non-discriminative region pooling (GNDRP). Contrary GAP which aggregates all the values of the spatial location in the feature map Ti, our GNDRP selectively aggregates the values of spatial locations whose CAM scores are below threshold T, as follows:
GAP(T))
1
Σ Ti(w), GNDRP(T))
1 [,
Σ Ti(u), U = {u u CAM(u) S ST),
WEW
where U is set of all spatial location indices in Tj.
5

---

Refinement
PASCAL VOC
MS COCO
Method
Method
Seed
CRF
Mask
Seed
Mask
PSA CVPR 188 [3]
48.0
61.0
Mixup-CAM BMVC 20 [6]
50.1
61.9
Chang el al. CVPR 220 [7]
50.9
55.3
63.4
PSA [3]
SEAM CVPR 22 [55]
55,4
56.8
63.6
25.1*
31.51
AdvCAM CVPR [33]
55.6
62.1
68.0
RIB (Ours)
56.5
62.9
68.6
IRN cvpr '19 [2]
48.8
54.3
66.3
33.5
42.9
MBMNet ACMMM 220 [40]
50.2
66.8
BES ECCV -20 [10]
50.4
67.2
IRN [2]
CONTA NeurIPS 20 [62]
48.8
67.9
28.7
35.2
AdvCAM CVPR [33]
55.6
62.1
69.9
RIB (Ours)
56.5
62.9
70.6
36.5
45.6
Table I: Comparison of the initial seed (Seed), the seed with CRF (CRF), and the pseudo ground truth mask (Mask) on PASCAL VOC and MS COCO training images, in tems of mloU (%). denotes the results reported by Zhang al. [62], and denotes the results obtained by us.
Other methods of weakly supervised semantic segmentation also considered new pooling methods other than GAP obtain better localization maps [4, 27, 44J. The pooling methods introduced in previous works make the classifier focus more on discriminative parts. In contrast, GNDRP excludes highly activated regions, encouraging non -ondiccriminative regions to be further activated.
We obtain the final localization map M by aggregating all the CAMs obtained from the classifier at each RIB iteration k: .M Eucask (a;(2;0e)
Obtaining a final localization map:
3.3 Weakly Supervised Semantic Segmentation
Because CAM [66] is obtained from down-sampled intermediate features produced by classifier, it should be up-sampled to the size of the original image. Therefore, it tends to localize the target object coarsely and cannot represent its exact boundary. Many weakly supervised semantic segmentation methods [7, 6, 55, 62, 40, 31] produce pseudo ground truths by modifying their initial seeds using established seed refinement methods [25, 3, 2, 27, 10]. Similarly, obtain pseudo ground truths by applying IRN [2], a state-of-the-art seed refinement method, to the coarse map M.
In addition, because image-level class label is void of any prior regarding the shape of the target object, salient object mask supervision is popularly used in existing methods [59, 31, 22, 38]. Salient object mask supervision can also be applied to our method to refine the pseudo ground truths: when foreground pixel in a pseudo label is identified as background on this map, or background pixel is identified as foreground, ignore such pixels in the training of the segmentation network.
4 Experiments
4.1
Experimental Setup
Dataset and evaluation metric: We evaluated our method quantitatively and qualitatively by conducting experiments on the PASCAL VOC 2012 [16] and the MS COCO 2014 [39] datasets. Following the common practice in weakly supervised semantic segmentation [3, 2, 31, 62), we used the PASCAL VOC 2012 datasct, which augmented by Hariharan et al. [20], containing 10,582 training images with objects from 20 classes. The MS COCO 2014 dataset contains approximately 82K training images containing objects of 80 classes. We evaluated our method on 1,449 validation images and 1,456 test images from the PASCAL VOC 2012 dataset and on 40,504 validation images from the MS COCO 2014 dataset, by calculating the mean Intersection-overr -vviuuuuon union (mloU) values.
Reproducibility. We implemented CAM [66] by following the procedure from Ahn et al. [2], which implemented with the PyTorch framework [43]. We used the ResNet-50 [21] backbone for the classification. We fine-tuned our classifier for K = 10 iterations with learning rate of 8 10-6 and a batch size of 20. We set the margin m to 600. For the GNDRP, we set 7 to 0.4. For the final semantic segmentation, we used the Py Porch implementation of DeepLab-v2 Berp-aab-2-BBe 22-ReNNet101 offered

---

Image
CAM
RIB process
(b)
Image
CAM
RIB process
Figure 3: Examples localization maps obtained during the RIB process for (a) PASCAL VOC 2012 training images and (b) MS COCO 2014 training images.
by [42]. used an initial model pre-trained on ImageNet dataset [14]. For the MS COCO 2014 dataset, cropped the training images with the crop size of 481 x481 rather than 321x321 used for the PASCAL VOC 2012 dataset, considering the size of the images in this dataset.
4.2 Weakly Supervised Semantic Segmentation
4.2.1 Quality of the initial seed and pseudo ground truth
PASCAL VOC 2012 dataset: In Table 1, we report the mloU values of the initial seed and pseudo ground truth masks generated from our method and from other recent techniques. Fol- lowing SEAM [55], we evaluate range of thresholds to distinguish between the foreground and the background in the map M and then determine the best initial seeds. Our initial seeds exhibit 7.7%p improvement from the original CAMs, a baseline for comparison, and simultaneously outperform those from the other methods. Note that our initial seeds are better than those of SEAM, which further refines the initial CAM on a pixel-level by considering the relationship between pixels through an auxiliary self-attention module.
We applied post-processing method based on conditional random field (CRF) [28] for pixel-level refinement of the initial seeds obtained from the method proposed by Chang et al. [7], SEAM [55], IRN [2]. and our method. On average, applying CRF improved all the seeds by more than 5%p, with the exception of SEAM. CRF improved SEAM by only 1.4%p, and it is reasonable to believe that this unusually small improvement occurred because the self-attention module had already refined seed from CAM. When the seed produced by our method is refined with CRF, it is 6. &% better than that from SEAM and consequently outperforms all the recent competitive methods by large margin.
Additionally, we compare the pseudo ground truth masks obtained after seed refinement with those obtained using other methods. Most of the compared methods use PSA [3] or IRN [2] refine their initial seeds. For a fair comparison, we generate pseudo ground truth masks using both seed refinement techniques. Table shows that the masks from our method yield an mloU of 68.6 with PSA [3] and 70.6 with IRN [2]. thereby outperforming other methods by large margin.
MS COCO 2014 dataset: Table 1 presents the mloU values of the initial seed and pseudo ground truth masks obtained by our method and by other recent methods for the MS COCO 2014 dataset. We obtained the results of IRN [2] using the official code to set the baseline performance. Our method improved the initial seed and pseudo ground truth masks our baseline IRN [2), by mIoU margins of 3.0%p and 2.7%p. respectively.
Figure illustrates localization maps gradually refined by the RIB process for the PASCAL VOC 2012 and the MS COCO 2014 datasets. More samples are shown in the Appendix.
4.2.2 Performance of weakly supervised semantic segmentation
PASCAL VOC 2012 dataset: Table 2 presents the mloU values of the segmentation maps on
PASCAL VOC 2012 validation and test images, predicted by our method and other recently introduced weakly supervised semantic segmentation methods, which use bounding box labels or image-level class labels. All the results in Table were obtained using a ResNet-based backbone [21]. Our method achieves mloU values of 68.3 and 68.6 for the validation and test images, respectively,
7

---

a
Image
Ground Truth
RIB
RIB w/. Sal
(b)
Image
Ground Truth
RIB
N
Figure 4: Examples of predicted segmentation masks from IRN [2] and method for (a) PASCAL
VOC 2012 validation images and (b) MS COCO 2014 validation images.
Method
val
test
Method
Sup.
val
Test
Supervision: Bounding box labels
SeeNet Neurles '18
[22]
63.1
62.8
Song al. CVPR 19 [50]
70.2
FickleNet CVPR 19 [31]
64.9
65.3
BBAM CVPR [34]
73.7
73.7
CIAN AAAI 20 [17]
64.3
65.3
Supervision: Image class labels
Zhang et al. ECCV 220
[63]
S
66.6
66.7
IRN CVPR '19 [2]
63.5
64.8
al. ECCV 220
[18]
67.2
66.7
SEAM cvPR [55]
64.5
65.7
Sun et al. BCCV 220
[51]
66.2
66.9
BES ECCV 22 [10]
65.7
66.6
LIID TRAMI '20 [41]
S
66.5
67.5
Chang et al. CVPR 20
77]
66.1
65.9
et al. AAAU [38]
68.2
68.5
RRM AAAI '20 [6]]
66.3
66.5
Yao et al. RIB (Ours)
[59]
S
68.3
68.5
CONTA NeutPS '20 [62]
66.1
66.7
70.2
70.0
RIB (Ours)
68.3
68.6
Table 3: Comparison of semantic segmentation
Table 2: Comparison of semantic segmenta- tion performance on PASCAL VOC 2012 val- idation and test images.
performance on PASCAL VOC 2012 validation and test images using explicit localization cues. S: salient object, Sy: salient instance.
the PASCAL VOC 2012 semantic segmentation benchmark, outperforming all the methods that use image-level class labels weak supervision. In particular, method outperforms CONTA [62], the best-performing method among our competitors, achieving an mloU value of 1. However, CONTA depends SEAM [55], which is known outperform IRN [2]. When CONTA was implemented with IRN for a fairer comparison with our method, its mloU value decreased to 65.3, which our method surpasses by 3.0%p.
Table 3 compares our method with other recent methods using additional salient object supervision. We utilized salient object supervision used by Li et al. [38] and Yao al. [59]. Our method achieves mloU values of 70,2 and 70.0 for the validation and test images, respectively, outperforming all the recently introduced methods under the same level of supervision.
Figure 4(a) shows examples of predicted segmentation maps by our method with and without saliency supervision. The boundary information provided by saliency supervision allows our method to produce more precise boundary (yellow boxes). However, the non-salient objects in an image are often ignored when using saliency supervision, while RIB successfully identifies them (e.g., a 'sofa' the first column and 'person' in red boxes Figure 4(a)). This empirical finding inspires a potential future work that can simultaneously identify a precise boundary and non-salient objects.
MS COCO 2014 dataset: Table compares our method with other recent methods on MS COCO 2014 validation images. Our method achieves an improve- ment of 2.4%p in terms of the mloU score compared with our baseline IRN [2], and outperforms the other re- cent competitive methods 111,6 62, 58] by a large margin. the comparison with CONTA [62]. the result of IRN reported in CONTA [62] differs from the one we ob- tained. Therefore, we compare relative improvements: CONTA achieves 0.8%p improvement compared with
Method
Backbone mloU
ADL TEAMI '20 [11]
VGG16
30.8
CONTA NeurIPS '20 [62]
ResNet50
33.4
Yao et al. Access '20 [58]
VGG16
33.6
IRN CVPR '19 [2]
ResNet101
41.4
RIB (Ours)
ResNet101
43.8
Table 4: Comparison of semantic segmenta- tion on MS COCO validation images.

---

Fine-tuning
Seed
1/t
Seed
Method
Seed
Init.
48.8
300
8 x 10
54.0
CAM
48,8
BCE w/. Tanh
49.7
600
5 x 10-6
54.9
RIB-GAP
54.8
BCE w/. Sigmoid
50.5
600
8 x 10-6
56.5
RIB-GNDRP 0..3)
55.8
BCE w/. Softsign
50.9
600
1 10-5
56.0
RIB- GEDDP (r=0.4)
56.5
CRIB
56.5
1000
8 x 10
55.9
RIB-GNDRP (T=0.5)
56.0
(a)
(b)
(c)
Table Comparison of mIoU scores of the initial seed (a) with different activation functions for the final layer, (b) with different values of m and X, and (c) with different values of T.
IRN (32.6 + 33.4), whereas our method achieves 2,4%p (41.4 1-44--00 43.8). Figure 4(b) presents examples of predicted segmentation maps by our method for the MS COCO 2014 validation images.
4.3 Ablative Studies
In this section, we analyze our method through various ablation studies conducted the PASCAL VOC 2012 dataset to provide more information about the effectiveness of each component of our method.
Influence of the total number of RIB iterations K:
We analyze the influence of the iteration number K on the effectiveness the RIB process. Figure 5 shows the mloU score of the initial seed obtained by our bascline CAM, and that of each iteration of the RIB process with GAP GNDRP. As the RIB process progresses, the localization map is significantly improved, regard- less of the pooling method. However, the increase in the performance of RIB with GAP is limited, and even slightly decreases in later iterations (K > 5). This is because GAP allows features that have already de- the classification to
mlou (%)
52
Initial CAM
RIB with GAP
50-
RIB with GNDRP
48
RIB iteration (K) Figure 5: Analysis of RIB with GAP or GNDRP in terms of mloU of the initial seed.
livered sufficient information become even more involved in the classification. Be- cause our proposed GNDRP limits the increase in the contribution of these discriminative regions to the classification, RIB with GNDRP can effectively allow non-d dissriiiiiiaiive information to be more involved in the classification, resulting in better localization map in later iterations. We observe that changing the value of K to be larger than 10 (even 20) produces less than 0.8%p drop in mIoU, suggesting that not difficult to select a good value of K.
Fine-tuning with CRIB: To verify the effectiveness of CRIB. we fine-tune model using the BCE loss with various double-sided saturating activation functions. Table 5 (a) shows the mloU scores of the initial seeds, obtained from a model fine-tuned by the BCE loss with sigmoid, tanh, and softsign activations, and our LRIB- We adjusted the output of tanh and softsign to have value between zero and one through the affine transform. Fine-tuning using the BCE loss with double-sided saturating activations improves the initial seed to some extent, which demonstrates the effectiveness of per-sample adaptation; however, their performance improvement is limited due the remaining information bottleneck. Note that the softsign activation function provides better localization maps than tanh and sigmoid. We believe this because the gradients from softsign reach zero at a higher value compared with the others (please see the Appendix), and consequently, softsign has less information bottleneck. Our CRIB effectively addresses the information bottleneck and achieves the best performance.
Analysis of the sensitivity to initial seed to the hyper- parameters involved in the RIB process. Table (b) presents the mloU values of the initial seed obtained using different combinations of values for the margin m and the learning rate A. Overall, a slightly lower performance is observed when the strength of the RIB process is weakened by small values of m and For sufficiently large m and X, the performance of the RIB process is competitive. Table 5 (c) analyzes the influence of the threshold 7 involved in the GNDRP. Increasing 7 from 0.3 to 0.5 results in less than I %p change in the mIoU, and thus, we conclude that the RIB process is robust against the changes in T.
We analyze the sensitivity of the mloU of the

---

Method
Prec.
Recall
Fl-score
Method
Boat
Train
IRN CVPR '19 [2]
66.0
66.4
66.2
IRN CVPR 11 [2]
35.3
51.3
Chang et al. cvPR '20 [7]
61.0
77.2
68.1
Chang et al. CVPR 20 [7]
34.1
53.1
SEAM cvrr 22 [55]
66.8
76.8
71.5
SEAM cveR 20 [55]
31.0
54.2
RIB (Ours)
67.3
78.9
72.6
RIB (Ours)
38.8
55.1
Table 6: Comparison of precision (Prec.), recall, and Fl-score on PASCAL VOC 2012 train images.
Table 7: mloU (%) of the initial seed for 'boat' and "train' classes.
4.4 Analysis of Spurious Correlation
In natural image, the target object and the background can be spuriously correlated when objects of a certain class primarily occur together in a specific context [36, 62] (e.g., boat on the sea and a train on the rail). Since image-level class labels do not provide an explicit localization cue of the target object, the classifier trained with these labels is vulnerable to spurious correlation. The localization map obtained from the classifier may also highlight the spuriously correlated background, which reduces precision. This is long-standing problem that is commonly found in weakly supervised semantic segmentation and object localization.
RIB may also activate some portion of the spurious background. However, we find that the amount of discovered areas correctly belonging to the foreground is noticeably more significant by comparing the precision, recall, and F1-score of our method with those of other recent methods in Table 6. Chang er al. [7] achieve high recall but experience large decrease in precision. SEAM [55] avoids this loss of precision with the help of pixel-level refinement implemented with an additional module mentioned in Section 4.2.1. Our method improves precision as well as recall of our baseline IRN [2] without an external module.
To further analyze the spuriously correlated background, we present class-wise seed improvement by our method and other recent methods. We select two representative classes, 'boat' and 'train', which are known to have the background that is spuriously correlated with the foreground (a boat on the sea and train on the rail). Table 7 shows that RIB can improve localization quality (mloU) even for classes known to have spurious foreground- foreground-background backgroud correlation.
5 Conclusions
In this study, we addressed the major challenge in weakly supervised semantic segmentation with image-level class labels. Through the information bottleneck principle, we first analyzed why the localization map obtained from classifier identifies only a small region of the target object. Our analysis highlighted that the amount of information delivered from an input image to the output classification is largely determined by the final layer of the DNN. We then developed method to reduce the information bottleneck through two simple modifications to the existing training scheme: the removal of the final non-linear activation function in the DNN and the introduction of new pooling method. Our method significantly improved the localization maps obtained from a classifier, exhibiting new state-of-the-art performance the PASCAL VOC 2012 and MS COCO 2014 datasets.
Societal implications: This work may have the following socictal impacts. Object segmentation without the need for pixel-level annotation will save resources for research and commercial develop- ment. It is particularly useful in fields such medicine, where expert annotation is costly. However, there are companies that provide annotations for images as a part of their services. If the dependence of DNNs labels is reduced by weakly supervised learning, these companies may need to change their business models.
Acknowledgements
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea goVer)men((MSIT) [NO.2021-0-015 Artificial Intel- ligence Graduate School Program (Seoul National University)), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) [201 8R1 I A A2B30006228 (28)). AIRS Company in Hyundai Motor and Kia through HMC/KIA-SNU AT Consortium Fund, and the Brain Korea 21 Plus Project in 2021.
10

---

References
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. TPAMI, 2018.
[2] Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. In CVPR, 2019.
[3] Jiwaon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In CVPR, 2018.
[4] Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In CVPR, 2020.
[5] Wonho Bae, Junhyug Noh, and Gunhee Kim. Rethinking class activation mapping for weakly supervised object localization. In European Conference Computer Vision, pages 618-634. Springer, 2020.
[6] Yu-Ting Chang, Qiaosong Wang, Wei-Chih Hung, Robinson Piramuthu, Yi-Hsuan Tsai, and Ming- Hsuan Yang. Mixup-cam: Weakly-supervised semantic segmentation via uncertainty regularization. In BMVC, 2020.
[7] Yu-Ting Chang, Qiaosong Wang, Wei-Chih Hung, Robinson Piramuthu, Yi-Hsuan Tsai, and Ming-Hsuan Yang. Weakly-supervised semantic segmentation via sub-category exploration. In CVPR, 2020.
[8] Binghui Chen, Weihong Deng, and Junping Du. Noisy softmax: Improving the generalization ability of denn via postponing the early softmax saturation. In CVPR, 2017.
[9] Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplah: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE TPAMI, 2017.
[10] Liyi Chen, Weiwei Wu, Chenchen Fu, Xiao Han, and Yuntao Zhang. Weakly supervised semantic segmentation with boundary exploration. In ECCV, 2020.
[11] Junsuk Choe, Seungho Lee, and Hyunjung Shim. Attention-based dropout layer for weakly supervised single object localization and semantic segmentation. IEEE TPAMI, 2020.
[12] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[13] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009.
[15] Yann Dubois, Douwe Kiela, David J Schwab, and Ramakrishna Vedantam. Learning optimal representations with the decodable information bottleneck. In NeurlPS, 2020.
[16] Mark Everingham, Luc Van Gool, Christopher KT Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJC,, 2010.
[17] Junsong Fan, Zhaoxiang Zhang, and Tieniu Tan. Cian: Cross-image aftinity net for weakly supervised semantic segmentation. AAAI, 2020.
[18] Junsong Fan, Zhaoxiang Zhang, and Tieniu Tan. Employing multi-estimations for weakly-supervised semantic segmentation. In FCCV, 2020.
[19] lan Goodfellow, Jcan Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[20] Bharath Hariharan, Pablo Arbeláez. Lubomir Bourdev, Subhransu Maji. and Jitendra Malik. Semantic contours from inverse detectors. In ICCV, 2011.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[22] Qibin Hou, Pan Jiang. Yunchao Wei, and Ming- Ming Cheng. Self-erasing network for integral object attention. In NeurIPS. 2018

---

[23] Gao Huang, Zhuang Liu. Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[24] Zilong Huang, Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu, Cenet: Criss-cross attention for semantic segmentation. In ICCV, 2019.
[25] Zilong Huang, Xinggang Wang, Jiasi Wang. Wenyu Liu, and Jingdong Wang. Suakly-superrised semantic segmentation network with deep seeded region growing. In CVPR, 2018
[26] Insu Jeon, Wonkwang Lec, Mycongjang Pyeon, and Gunhee Kim. Ib-gan: Disengangled representation learning with information bottleneck generative adversarial networks. In AAAI, 2021.
[27] Alexander Kolesnikov and Christoph H Lampert. Seed, expand and constrain: Three principles for seakly-supervised image segmentation. In ECCV, 2016.
[28] Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In NeurIPS, 2011.
[29] Viveka Kulharia, Siddhartha Chandra, Amit Agrawal, Philip Torr, and Ambrish Tyagi. Box2seg: Attention weighted loss and discriminative feature learning for weakly supervised segmentation. In ECCV, 2020.
[30] Yann LeCun. The mnist database handwritten digits. hrtp://yann. lecun, com/exdhhnwistt/ 1998.
[31] Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon. Ficklenet: Weakly and semi-
supervised semantic image segmentation using stochastic inference. In CVPR, 2019.
[32] Jungbeom Lee, Eunji Kim, Sungmin Lec, Jangho Lee, and Sungroh Yoon. Frame-to coame-to-framse frame aggregation of active regions in web videos for weakly supervised semantic segmentation. In ICCV, 2019.
[33] Jungbeom Lee. Eunji Kim, Sungroh Yoon. Anti-adversarially manipulated attributions for weakly and semi-supervised semantic segmentation. In CVPR, 2021.
[34] Jungbcom Lec, Jihun Yi, Chachun Shin, and Sungroh Yoon. Bbam: Bounding box attribution map for weakly supervised semantic and instance segmentation. In CVPR, 2021.
[35] Sungmin Lee, Jangho Lee, Jungbeom Lee, Chul-Kee Park, and Sungroh Yoon. Robust tumor localization with pyramid grad-cam. arXiv preprint arXiv:1805. 11393, 2018.
[36] Kumpeng Li, Ziyan Wu, Kuan- Shan-Chuan Peng, Jan Emstt and Yun Fu. Tell me where to look: Guided attention inference network. In CVPR, 2018.
[37] Kunpeng Li, Ziyan Wu, Chan-Chhan Peng, Jan Ernst, and Yun Fu. Guided attention inference network. IEEE TPAMI, 2019.
[38] Xueyi Li, Tianfei Zhou, Jianwu Li, Yi Zhou, and Zhaoxiang Zhang. Group-wise semantic mining for weakly supervised semantic segmentation. AAAI, 2021.
[39] Tsung-Y Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zimick. Microsoft coco: Common objects in context. In ECCV, 2014.
[40] Weide Liu, Chi Zhang, Guosheng Lin, Tzu-Yi H and Chunyan Miao. Weakly supervised segmentation with maximum bipartite graph matching. In ACMMM, 2020.
[41] Yun Liu, Yu-Huan Wu, Pei-Song Wen, Yu-Jun Shi, Yu Qiu, and Ming-Ming Cheng. Leveraging instance- image-and dataset-level information for weakly supervised instance segmentation. TPAMI, 2020.
[42]
Kazuto Nakashima. DeepLab with PyTorch. https: : httpaa/..ooomaaatttot///////eppe http:///t////eppaaappaprooa
[43] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Ve Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[44] Pedro O Pinheiro and Ronan Collobent. From image-level to pixel-level labeling with convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1713-1721, 2015.
[45] Alexandre Rame and Matthieu Cord. Dice: Diversity in deep ensembles via conditional redundancy adversarial estimation. In ICLR, 2021.
12

---

[46] Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In International Conference on Learning Representations. 2018.
[47] Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. ICLR, 2020.
[48] Ramprasaath R Selvaraju, Michael Cogswell, Ahhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
[49] Ravid Shwartz-Zii Ziv and Naftali Tishby, Opening the black box of deep neural networks via information. arXiv preprint arXiv:/703 100.00000,,, 2017.
[50] Chunfeng Song, Yan Huang, Wanli Ouyang, and Liang Wang. Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation. In CVPR, 2019.
[51] Guolei Sun, Wenguan Wang, Jifeng Dai, and Lue Van Gool. Mining cross-image semantics for weakly supervised semantic segmentation. In ECCV, 2020.
[52] Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, and Christopher Schroers. Normalized cut loss for weakly-supervine supervised enn segmentation. In CVPR, 2018.
[53] Naftali Tishby, Fernando C. Pereira, and William Bialck. The information bottleneck method. arXiv preprint physics/0004557, 2000.
[54] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1-5. IEEE, 2015.
[55] Yude Wang. Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. Self-supee equivariant attention mechanism for weakly supervised semantic segmentation. CVPR, 2020.
[56] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. Object region mining with adversarial crasing: A simple classification to semantic segmentation approach. In CVPR, 2017.
[57] Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jie, Jiashi Feng, and Thomas Huang. Revisiting dilated convolution: A simple approach for weakly- and supervipe semantic segmentation. In CVPR, 2018.
[58] Qi Yao and Xiaojin Gong. Saliency guided self-attention network for weakly and semi-supervised semantic segmentation. IEEF Access, 2020.
[59] Yazhou Yao, Tao Chen, Guosen Xie, Chuanyi Zhang, Fumin Shen, Qi Wu, Zhenmin Tang, and Jian Zhang, Non-salient region object mining for weakly supervised semantic segmentation. In CVPR, 2021.
[60] Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn, Meta-learning without memorization. In ICLR, 2019.
[61] Bingfeng Zhang, Jimin Xiao, Yunchao Wei, Mingjie Sun, and Kaizhu Huang. Reliability does matter: An end-to-end weakly supervised semantic segmentation approach. AAAI, 2020.
[62] Dong Zhang, Hanwang Zhang, Jinhui Tang. Xiansheng Hua, and Qianru Sun. Causal intervention for weakly-supervised semantic segmentation. 2020.
[63] Tianyi Zhang, Guosheng Lin, Weide Liu, Jianfei Cai, and Alex Kot. Splitting vs. merging: Mining object regions with discrepancy and intersection loss for weakly supervised semantic segmentation. In ECCV, 2020.
[64] Xiaolin Zhang, Yunchao Wei, Jiashi Feng, Yi Yang, and Thomas Huang. Adversarial complementary learning for weakly supervised object localization. CVPR, 2018
[65] Andrey Zhmoginov, Ian Fischer, and Mark Sandler. information--ootllenee approach to salient region discovery. In ICML Workshop on SulerSiue Learning 2019.
[66] Bolei Zhou, Aditya Khosla, Agata Lapedriza. Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Our theoretical assumption is already widely accepted by previous literature.
(b) Did you include complete proofs of all theoretical results? [Yes] Our theoretical proofs are already widely done by previous literature.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes] Please see the Appendix
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see Section
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] Please see Appendix
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Please the Appendix
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Please see Section
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "8M_8w5nEKqv", "reviewer": "Reviewer_NvrK", "summary": "This work addresses a limitation of CAMs (Class Activation Mappings) in the context of weakly supervised semantic segmentation (W3S) taking the information bottleneck perspective. The reason for the CAMs being highly selective, it argues, lies in the use of double-sided non-linearities (e.g. sigmoid) in the final layer of the classification network. The work removes this non-linearity and develops a strategy of obtaining high-recall attention maps through a one-shot adaptation process. Used as initial masks in a SotA W3S pipeline, these masks impressively increase the final segmentation accuracy achieving new state-of-the-art on established benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality.*\nThe approach proposed in this work has a number of novel and interesting elements. However, the problem with CAMs and the classification providing only “discriminative” regions is well-known and some attempts to address were not discussed or compared, such as LSE [A] and nGWP [B]. In particular, the motivation in [B] is not so dissimilar from the one in this work, as it also encourages the increased mask size through a loss term. Adding a discussion and a comparison to this alternative approach would make this work stronger.\n\n[A] Pinheiro and Collobert. From image-level to pixel-level labeling with Convolutional Networks. CVPR 2015.\n[B] Araslanov and Roth. Single-Stage Semantic Segmentation From Image Labels. CVPR 2020.\n\n*Quality.*\nThe work is well-executed. I particularly appreciate the experiments both in the “labels-only” and “+saliency” setups on VOC-12, as well as experiments on MS-COCO. Model sensitivity to hyperparameters $m$, $\\tau$, $K$ and $B$ are well-studied.\n\nOne remark however:\nI do not see the necessity of introducing the circles and squares in the motivating experiment in Sec. 3.1. $G_6$ in Fig. 1 shows perfectly well that only a fraction of the digit is relevant for the final layer, whereas in Fig. 2 we now observe the tendency towards uncovering the full extent of the digit. So, why do we need the circles and squares? In natural images those may represent spurious correlations (e.g. an umbrella, which often co-occurs with the class “person”), hence the method will only pick up on those, as this toy experiment suggests.\n\n*Clarity.*\nThe manuscript is well-written and follows a clear structure. The analysis and qualitative examples are insightful.\n\nMinor issues are:\n- ll. 134-135 “significant bottleneck”; “amount of information … will be limited”: these points is hard to agree with, since no measure is introduced. I’d suggest to make it relative to one-sided non-linearities, for example, to connect it with the discussion in Sec. 2.1.\n- I’d consider renaming HGR, since it does not reflect the custom threshold of 0.3 (l. 164). As a result, Fig. 1 (c) and 2 (b) are unclear without a reference to the text.\n- Please, add numbers to the equations on p. 5.\n\n*Significance.*\nThe paper offers an interesting perspective on addressing the limitations of CAMs on which virtually all weakly supervised semantic segmentations  rely. The RIB process is relatively straightforward, the experiments demonstrate clear benefits. Future works on W3S and, perhaps, attribution methods, may find this technique useful. \n\n**Post-rebuttal update**\n\nI thank the authors for their response, and I keep my initial recommendation to accept this work. Nevertheless, I hope the design of the toy experiment can be still re-considered in the camera-ready version in view of my remark on spurious correlations.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations seems to be scattered throughout Sec. 4, which does not give a clear impression that could be useful for follow-up work. I recommend offering a discussion on the limitations in a more prominent way. The work does not discuss the potential societal impact which needs to be addressed in the revision. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "57vzVSOf-tX", "reviewer": "Reviewer_PSWB", "summary": "This paper presents a new method for computing class activation maps (CAMs), which has been used as crucial evidence for object detection and segmentation in a weakly supervised learning setting. A well known drawback of CAM and its variants is that they often highlight only small discriminative regions of an object. This paper interprets such a phenomenon through “information bottleneck” and proposes a way to alleviate it accordingly. In short, the proposed method finetune a classification network for generating the original CAM with a different classification loss and a different global pooling layer that allow more features to contribute to classification. The method enhances the quality of the original CAM noticeably and consequently improves performance of a weakly supervised semantic segmentation method relying on CAMs.\n\nThe main idea of this work is novel, and useful to alleviate the well-known limitation of the CAM and its variants. Moreover, the efficacy of the proposed method has been demonstrated by extensive experiments. However, the overall pipeline for semantic segmentation depends heavily on an existing method (IRN [2]), and it demands a much larger number of hyper-parameters compared to the original CAM. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n- The paper is overall well-written; it is easy to follow in most parts and well-organized. Also, details necessary to reproduce the proposed method are presented in the paper.\n- The main idea is novel in that there is no previous work explaining and alleviating the well-known limitation of the CAM and its variants through information bottleneck.\n- The efficacy of the proposed method has been demonstrated by extensive experiments on PASCAL VOC and MS-COCO, where weakly supervised semantic segmentation based on the proposed method achieves the state of the art. This reviewer in particular appreciates its strong performance on the challenging MS-COCO dataset. \n- Also, the effect of hyper-parameters is investigated by extensive ablation studies.\n\n[Weaknesses / questions]\n- The proposed method is applied only on weakly supervised semantic segmentation and its overall pipeline for semantic segmentation relies heavily on IRN [2]; considering this work as a weakly supervised semantic segmentation paper, its contribution looks quite narrow. One way to resolve this issue might be applying the proposed method to other weakly supervised visual recognition tasks, e.g., object localization and instance segmentation.\n- The method demands a larger number of hyper-parameters, compared to the original CAM. Although their effects are investigated by ablation studies, this reviewer still wonders how to set the number of RIB iterations, i.e., K, and what happens when K > 10; these analyses are missing in the current manuscript.\n- It is not explained why the final localization map is computed as the sum of CAMs over RIB iterations, instead of just choosing the CAM of the last RIB iteration.\n- The design of experiments in Section 3.1 is not convincing. In particular, it is not explained at all why the circles and squares should be considered as class-relevant yet non-discriminative regions. \n- The authors seem to assume that class-relevant yet non-discriminative regions are always parts of an object, which is not true unfortunately since such regions may include class-specific background stuff (e.g., rails near trains). Thus the proposed RIB finetuning could damage precision while improving recall. In the same context, this reviewer would recommend analyzing, evaluating, and comparing the proposed method with others through precision-recall curves. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of this work are not summarized in a separate section. This reviewer could not find any potential negative societal impact of this work. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issue has been found.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "1Gts3rQ__KT", "reviewer": "Reviewer_cUpz", "summary": "This paper focuses on the problem of weakly-supervised semantic segmentation, where the goal is to conduct segmentation by mainly using image-level annotation as such weak form of annotations can be collected more efficiently. \n\nSpecifically, authors analyze the issue of Information Bottleneck and propose a simple method RIB to reduce it to obtain more complete CAM localization maps. A e a global non-discriminative region pooling (GNDRP) is further applied to enhance the effects of RIB. \n\nExperiments on PASCAL VOC  and MS COCO datasets show the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ This paper begins with toy experiments to illuminate the problem of information bottleneck, which is intuitive and clear.\n\n+ The ideas of RIB and GNDRP are reasonable and effective. \n\n+ Extensive experiments have been conducted to show the contributions of the proposed method. Parameter analysis has also been conducted to help analyze the behavior of the model. \n\n+ The proposed method shows strong performance when compared with recent state-of-the-art methods.  \n\n\n\nConcerns and Questions:\n\n- The descriptions and motivations of RIB training method are not clear (line 189-201). For each RIB iteration, is the model updated by the gradient of L_RIB for a fixed image x as well as L_RIB for other N-1 randomly sampled images? Why is it necessary to set several RIB iterations for each specific image x? Why not just conduct RIB loss for images in a batch and fine-tuning the model for several epochs? \n\n- For the details of using GNDRP, is it only applied at the training stage (RIB iteration)? Then at the inference stage of CAM maps, I believe the original GAP layer is still used rather than GNDRP. It that correct? The idea of GNDRP is quite similar to hide-and-seek method operated on feature maps, which is not new. \n\n- For results in Figure 3, are they obtained by only applying RIB iterations (RIB-GAP)? Or they are actually obtained by RIB-GNDRP. If they are from RIB-GAP then it is fine. Otherwise, showing results of RIB-GAP would be more helpful. \n\n- Can the proposed method generalize to Grad-CAM instead of only on CAM? Analysis, discussions and even experimental results regarding this point would help to show the generalization of the method. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable. Authors did not discuss limitations and potential negative societal impacts of the work. See check list 1 (c). ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "1C13IdbeYX", "reviewer": "Reviewer_Az2H", "summary": "This paper is about semantic image segmentation training with image-level labels. The method is using an auxiliary loss to help to find a larger localization area for pixel-wise training. Experiments are done on Pascal VOC2021 and MSCOCO.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.A flowchart and an Algorithm to describe the whole method is absent.\n\n2.This paper claims the information bottleneck or Data Processing Inequality is important for recognition and remove the last sigmoid. For a L-layer network with same resolution, ResNet-v2 style network has absolutely less information loss than ResNet-v1. However, we often find ResNet-v1 behaves better than ResNet-v2. It seems the information loss is not that important? Or what kind of suggestion could this concept provide to design a better network structure? \n\n3.In page 5 Line 200, “Note that for each iteration, B−1 samples are randomly selected, while x is fixed.” This sentence is confusing.\n\n4.In Table 4 it seems the backbone of proposed method is only ResNet-101. It is necessary to include more backbone, such as ResNet-50, Efficient, Transformer, etc., to prove the robustness of the method.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes an improved method for computing class activation maps, which are then used in weakly (image level label) supervised segmentation.  The main problem with class activation maps are that they often only highlight a small portion of an object.  Addressing this issue is the main focus of the paper, and does so with a different classification loss and a different global pooling layer.  The reviewers were unanimous in their opinion that the paper is above the bar for acceptance at NeurIPS, and appreciated the extensive experiments and utility of the setting.
Decision: Accept (Poster)