Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation

1 Introduction . Semantic segmentation is the task of recognizing objects in an image using pixel-level allocation of a semantic label . The development of deep neural networks ( DNNs ) has led to significant advances in semantic segmentation [ 9 , 24 ] . Training a DNN for semantic segmentation requires a dataset containing a large number of images annotated with pixel-level labels . However , preparing such a dataset requires considerable effort ; for example , producing a pixel-level annotation for a single image in the Cityscapes dataset [ 12 ] takes more than 90 minutes . This high dependence on pixel-level labels can be alleviated by weakly supervised learning [ 62 , 31 , 34 , 2 ] . The objective of weakly supervised semantic segmentation is to train a segmentation network with weak annotations , which provide less information about the location of a target object than pixellevel labels , but are cheaper to obtain . Weak supervision takes the form of scribbles [ 52 ] , bounding boxes [ 50 , 29 , 34 ] , or image-level class labels [ 31 , 3 , 2 ] . In this study , we focus on image-level class labels , because they are the cheapest and most popular option of weak supervision . Most methods that use class labels [ 31 , 32 , 55 , 2 , 7 ] generate pseudo ground truths for training a segmentation network using localization ( attribution ) maps obtained from a trained classifier , such as a CAM [ 66 ] or a Grad-CAM [ 48 ] . However , these maps identify only small regions of a target object that are discriminative for the classification [ 31 , 2 , 5 ] and do not identify the entire region occupied by the object , making the attribution maps unsuitable for training a semantic segmentation network . We interpret this phenomenon using the information bottleneck principle [ 54 , 49 , 46 , 53 ] . ∗Correspondence to : Sungroh Yoon < sryoon @ snu.ac.kr > . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The information bottleneck theory analyzes the information flow through sequential DNN layers : information regarding the input is compressed as much as possible as it passes through the layers of a DNN , while preserving as much of the task-relevant information as possible . This is advantageous for obtaining optimal representations for classification [ 15 , 1 ] but is disadvantageous when applying the attribution maps from the resulting classifier to weakly supervised semantic segmentation . The information bottleneck prevents the non-discriminative information of the target object from being considered in the classification logit , and thus , the attribution maps focus on only the small discriminative regions of the target object . We argue that the information bottleneck becomes prominent in the final layer of the DNN due to the use of the double-sided saturating activation function therein ( e.g. , sigmoid , softmax ) . We propose a method to reduce this information bottleneck in the final layer of the DNN by retraining the DNN without the last activation function . Additionally , we introduce a new pooling method that allows more information embedded in non-discriminative features , rather than discriminative features , to be processed in the last layer of a DNN . As a result , the attribution maps of the classifier obtained by our method contain more information on the target object . The main contributions of this study are summarized as follows . First , we highlight that the information bottleneck occurs mostly in the final layer of the DNN , which causes the attribution maps obtained from a trained classifier to restrict their focus to small discriminative regions of the target object . Second , we propose a method to reduce this information bottleneck by simply modifying the existing training scheme . Third , our method significantly improves the quality of the localization maps obtained from a trained classifier , exhibiting a new state-of-the-art performance on the PASCAL VOC 2012 and MS COCO 2014 datasets for weakly supervised semantic segmentation . 2 Preliminaries . 2.1 Information Bottleneck . Given two random variables X and Y , the mutual information I ( X ; Y ) quantifies the mutual dependence between the two variables . Data processing inequality ( DPI ) [ 13 ] infers that any three variables X , Y , and Z that form a Markov Chain X → Y → Z satisfy I ( X ; Y ) ≥ I ( X ; Z ) . Each layer in a DNN processes the input only from the previous layer , which means that the DNN layers form a Markov chain . Therefore , the information flow through these layers can be represented using DPI . More specifically , when an L−layered DNN generates an output Ŷ from a given input X through intermediate features Tl ( 1 ≤ l ≤ L ) , it forms a Markov Chain X → T1 → · · · → TL → Ŷ , and the corresponding DPI chain can be expressed as follows : I ( X ; T1 ) ≥ I ( X ; T2 ) ≥ · · · ≥ I ( X ; TL−1 ) ≥ I ( X ; TL ) ≥ I ( X ; Ŷ ) . ( 1 ) This implies that the information regarding the input X is compressed as it passes through the layers of the DNN . Training a classification network can be interpreted as extracting maximally compressed features of the input that preserve as much information as possible for classification ; such features are commonly referred to as minimum sufficient features ( i.e. , discriminative information ) . The minimum sufficient features ( optimal representations T ∗ ) can be obtained by the information bottleneck trade-off between the mutual information of X and T ( compression ) , and that of T and Y ( classification ) [ 54 , 15 ] . In other words , T ∗ = argminT I ( X ; T ) − βI ( T ; Y ) , where β ≥ 0 is a Lagrange multiplier . Shwartz-Ziv et al . [ 49 ] observe a compression phase in the process of finding the optimal representation T ∗ : when observing I ( X , Tl ) for a fixed l , I ( X , Tl ) steadily increases during the first few epochs , but decreases in the later epochs . Saxe et al . [ 46 ] argue that the compression phase is mainly observed in DNNs equipped with double-sided saturating non-linearities ( e.g. , tanh and sigmoid ) , and is not observed in those equipped with single-sided saturating non-linearities ( e.g. , ReLU ) . This implies that DNNs with single-sided saturating non-linearities experience less information bottleneck than those with double-sided saturating non-linearities . This can also be understood in terms of gradient saturation in the double-sided saturating non-linearities : the gradient of those non-linearities with respect to an input above a certain value saturates close to zero [ 8 ] . Therefore , features above a certain value will have near-zero gradients during the back-propagation process and be restricted from additionally contributing to the classification . 2.2 Class Activation Mapping . A class activation map ( CAM ) [ 66 ] identifies regions of an image focused by a classifier . The CAM is based on a convolutional neural network with global average pooling ( GAP ) before its final classification layer . This is realized by considering the class-specific contribution of each channel of the last feature map to the classification score . Given a classifier parameterized by θ = { θf , w } where f ( · ; θf ) is the feature extractor prior to GAP , and w is the weight of the final classification layer , a CAM of the class c is obtained from an image x as follows : CAM ( x ; θ ) = wᵀc f ( x ; θf ) maxwᵀc f ( x ; θf ) , ( 2 ) where max ( · ) is the maximum value over the spatial locations for normalization . 2.3 Related Work . Weakly Supervised Semantic Segmentation : Weakly supervised semantic segmentation methods with image-level class labels first construct an initial seed by obtaining a high-quality localization map from a trained classifier . Erasure methods [ 56 , 37 , 22 ] prevent a classifier from only focusing on the discriminative parts of objects by feeding the image from which the discriminative regions have been erased to the classifier . Several contexts of a target object can be considered by combining multiple attribution maps obtained from differently dilated convolutions [ 31 , 57 ] or from the different layers of a DNN [ 35 ] . Diverse images of a target class can be utilized by considering cross-image semantic similarities and differences [ 17 , 51 ] . Zhang et al . [ 62 ] analyze the causalities among images , contexts , and class labels and propose CONTA to remove the confounding bias in the classification . Because the localization maps obtained by a classification network can not accurately represent the boundary of a target object , the initial seed obtained by the above methods is refined using subsequent boundary refinement techniques such as PSA [ 3 ] and IRN [ 2 ] . Information Bottleneck : Tishby et al . [ 54 ] and Shwartz et al . [ 49 ] use the information bottleneck theory to analyze the inner workings of a DNN . The concept of the information bottleneck has been employed in many research fields . Dubois et al . [ 15 ] and Achille et al . [ 1 ] utilize the information bottleneck to obtain optimal representations from DNNs . DICE [ 45 ] is proposed for a model ensemble with the information bottleneck principle : it aims to reduce not only the unnecessary mutual information between features and inputs but also the redundant information shared between features produced by separately trained DNNs . Jeon et al . [ 26 ] study the disentangled representation learning of a generative model [ 19 ] using the information bottleneck principle . Yin et al . [ 60 ] design a regularization objective based on information theory to deal with the memorization problem in meta-learning . The information bottleneck principle can also be adopted to generate the visual saliency map of a classifier . Zhmoginov et al . [ 65 ] find important regions for the classifier with the information bottleneck trade-off , and Schulz et al . [ 47 ] restrict the information flow by adding noise to intermediate feature maps and quantify the amount of information contained in the image region . 3 Proposed Method . Weakly supervised semantic segmentation methods using class labels produce a pixel-level localization map from a classifier using CAM [ 66 ] or Grad-CAM [ 48 ] ; however , such a map identifies only small discriminative regions of the target object . We analyze this phenomenon with the information bottleneck theory in Section 3.1 and propose RIB , a method to address this problem , in Section 3.2 . We then explain how we train a segmentation network with localization maps improved through RIB in Section 3.3 . 3.1 Motivation . As mentioned in Section 2.1 , the DNN layers with double-sided saturating non-linearities have a larger information bottleneck than those with single-sided saturating non-linearities . The intermediate layers of popular DNN architectures ( e.g. , ResNet [ 21 ] and DenseNet [ 23 ] ) are coupled with the ReLU activation function , which is a single-sided saturating non-linearity . However , the final layer of these networks is activated by a double-sided saturating non-linearity such as sigmoid or softmax , and the class probability p is computed with the final feature map TL and the final classification layer w , i.e. , p = sigmoid ( wᵀGAP ( TL ) ) . Therefore , the final layer parameterized by w has a significant bottleneck , and the amount of information transmitted from the last feature TL to the actual classification prediction will be limited . These arguments are analogous to the observations in existing methods . The information plane provided by Saxe et al . [ 46 ] shows that the compression of information is more noticeable in the final layer than in the other layers . Bae et al . [ 5 ] observe that although the final feature map of the classifier contains rich information on the target object , the final classification layer filters out most of it ; thus , the CAM can not identify the entire area of the target object . This observation empirically supports the occurrence of the information bottleneck in the final layer of a DNN . To take a closer look at this phenomenon , we design a toy experiment . We collect images containing the digits ‘ 2 ’ or ‘ 8 ’ from the MNIST dataset [ 30 ] . For only a small subset ( 10 % ) of these images , we add a circle ( ) and a square ( ) to the images containing the digits ‘ 2 ’ and ‘ 8 ’ , respectively , at a random location ( see Figure 1 ( a ) ) . When classifying images into the digits ‘ 2 ’ or ‘ 8 ’ , pixels corresponding to the digit are discriminative regions ( RD ) , those corresponding to the added circle or square are non-discriminative but class-relevant regions ( RND ) , and those corresponding to the background are class-irrelevant regions ( RBG ) . We train a neural network with five convolutional layers followed by a final fully connected layer . We obtain the gradient map Gl of each feature Tl with respect to an input image x : Gl = ∇x ∑ u , v Tl ( u , v ) , where u and v are the spatial and channel indices of the feature Tl , and for the final classification layer ( l = 6 ) , G6 = ∇xyc . Because this gradient map indicates the extent to which each pixel of the image affects each feature , it can be used to examine how much information is passed from the input image to the feature maps of successive convolution layers . We present examples of Gl in Figure 1 ( b ) . As an input image passes through the convolution layers , the overall amount of gradient with respect to the input decreases , indicating the occurrence of the information bottleneck . Specifically , the gradient of RBG decreases early on ( G1 → G2 ) , which implies that the task-irrelevant information is rapidly compressed . From G1 to G5 , the gradient in RD orRND gradually decreases . However , the decrease in the amount of gradient is prominent in the final layer ( G5 → G6 ) , and in particular , the gradients in RND ( red boxes ) almost disappear . This supports our argument that there is significant information bottleneck in the final layer of a DNN , while also highlighting that the non-discriminative information inRND is particularly compressed . We analyze this quantitatively . We define the high gradient ratio ( HGR ) of regionR as the ratio of pixels that have a gradient above 0.3 to the total pixels in regionR . HGR quantifies the amount of transmitted information from region R of an input image to each feature . The trend in the HGR values of each region for each layer is shown in Figure 1 ( c ) . The observed trend is analogous to the above empirical observation , once again supporting that significant information bottleneck forRND occurs in the final layer ( the red box ) . We argue that the information bottleneck causes the localization map obtained from a trained classifier to focus on small regions of the target object . According to Eq . 2 , the CAM only includes information that is processed by the final classification weight wc . However , because only a subset of the information in the feature is passed through the final layer wc due to the information bottleneck , leaving out most of the non-discriminative information , CAM can not identify the non-discriminative regions of the target object . It is undesirable to use such CAMs to train a semantic segmentation network , for which the entire region of the target object should be identified . Therefore , we aim to bridge the gap between classification and localization by reducing the information bottleneck .

Summary: This paper is about semantic image segmentation training with image-level labels. The method is using an auxiliary loss to help to find a larger localization area for pixel-wise training. Experiments are done on Pascal VOC2021 and MSCOCO.