Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation

1 Introduction . Semantic segmentation is the task of recognizing objects in an image using pixel-level allocation of a semantic label . The development of deep neural networks ( DNNs ) has led to significant advances in semantic segmentation [ 9 , 24 ] . Training a DNN for semantic segmentation requires a dataset containing a large number of images annotated with pixel-level labels . However , preparing such a dataset requires considerable effort ; for example , producing a pixel-level annotation for a single image in the Cityscapes dataset [ 12 ] takes more than 90 minutes . This high dependence on pixel-level labels can be alleviated by weakly supervised learning [ 62 , 31 , 34 , 2 ] . The objective of weakly supervised semantic segmentation is to train a segmentation network with weak annotations , which provide less information about the location of a target object than pixellevel labels , but are cheaper to obtain . Weak supervision takes the form of scribbles [ 52 ] , bounding boxes [ 50 , 29 , 34 ] , or image-level class labels [ 31 , 3 , 2 ] . In this study , we focus on image-level class labels , because they are the cheapest and most popular option of weak supervision . Most methods that use class labels [ 31 , 32 , 55 , 2 , 7 ] generate pseudo ground truths for training a segmentation network using localization ( attribution ) maps obtained from a trained classifier , such as a CAM [ 66 ] or a Grad-CAM [ 48 ] . However , these maps identify only small regions of a target object that are discriminative for the classification [ 31 , 2 , 5 ] and do not identify the entire region occupied by the object , making the attribution maps unsuitable for training a semantic segmentation network . We interpret this phenomenon using the information bottleneck principle [ 54 , 49 , 46 , 53 ] . ∗Correspondence to : Sungroh Yoon < sryoon @ snu.ac.kr > . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The information bottleneck theory analyzes the information flow through sequential DNN layers : information regarding the input is compressed as much as possible as it passes through the layers of a DNN , while preserving as much of the task-relevant information as possible . This is advantageous for obtaining optimal representations for classification [ 15 , 1 ] but is disadvantageous when applying the attribution maps from the resulting classifier to weakly supervised semantic segmentation . The information bottleneck prevents the non-discriminative information of the target object from being considered in the classification logit , and thus , the attribution maps focus on only the small discriminative regions of the target object . We argue that the information bottleneck becomes prominent in the final layer of the DNN due to the use of the double-sided saturating activation function therein ( e.g. , sigmoid , softmax ) . We propose a method to reduce this information bottleneck in the final layer of the DNN by retraining the DNN without the last activation function . Additionally , we introduce a new pooling method that allows more information embedded in non-discriminative features , rather than discriminative features , to be processed in the last layer of a DNN . As a result , the attribution maps of the classifier obtained by our method contain more information on the target object . The main contributions of this study are summarized as follows . First , we highlight that the information bottleneck occurs mostly in the final layer of the DNN , which causes the attribution maps obtained from a trained classifier to restrict their focus to small discriminative regions of the target object . Second , we propose a method to reduce this information bottleneck by simply modifying the existing training scheme . Third , our method significantly improves the quality of the localization maps obtained from a trained classifier , exhibiting a new state-of-the-art performance on the PASCAL VOC 2012 and MS COCO 2014 datasets for weakly supervised semantic segmentation . 2 Preliminaries . 2.1 Information Bottleneck . Given two random variables X and Y , the mutual information I ( X ; Y ) quantifies the mutual dependence between the two variables . Data processing inequality ( DPI ) [ 13 ] infers that any three variables X , Y , and Z that form a Markov Chain X → Y → Z satisfy I ( X ; Y ) ≥ I ( X ; Z ) . Each layer in a DNN processes the input only from the previous layer , which means that the DNN layers form a Markov chain . Therefore , the information flow through these layers can be represented using DPI . More specifically , when an L−layered DNN generates an output Ŷ from a given input X through intermediate features Tl ( 1 ≤ l ≤ L ) , it forms a Markov Chain X → T1 → · · · → TL → Ŷ , and the corresponding DPI chain can be expressed as follows : I ( X ; T1 ) ≥ I ( X ; T2 ) ≥ · · · ≥ I ( X ; TL−1 ) ≥ I ( X ; TL ) ≥ I ( X ; Ŷ ) . ( 1 ) This implies that the information regarding the input X is compressed as it passes through the layers of the DNN . Training a classification network can be interpreted as extracting maximally compressed features of the input that preserve as much information as possible for classification ; such features are commonly referred to as minimum sufficient features ( i.e. , discriminative information ) . The minimum sufficient features ( optimal representations T ∗ ) can be obtained by the information bottleneck trade-off between the mutual information of X and T ( compression ) , and that of T and Y ( classification ) [ 54 , 15 ] . In other words , T ∗ = argminT I ( X ; T ) − βI ( T ; Y ) , where β ≥ 0 is a Lagrange multiplier . Shwartz-Ziv et al . [ 49 ] observe a compression phase in the process of finding the optimal representation T ∗ : when observing I ( X , Tl ) for a fixed l , I ( X , Tl ) steadily increases during the first few epochs , but decreases in the later epochs . Saxe et al . [ 46 ] argue that the compression phase is mainly observed in DNNs equipped with double-sided saturating non-linearities ( e.g. , tanh and sigmoid ) , and is not observed in those equipped with single-sided saturating non-linearities ( e.g. , ReLU ) . This implies that DNNs with single-sided saturating non-linearities experience less information bottleneck than those with double-sided saturating non-linearities . This can also be understood in terms of gradient saturation in the double-sided saturating non-linearities : the gradient of those non-linearities with respect to an input above a certain value saturates close to zero [ 8 ] . Therefore , features above a certain value will have near-zero gradients during the back-propagation process and be restricted from additionally contributing to the classification . 2.2 Class Activation Mapping . A class activation map ( CAM ) [ 66 ] identifies regions of an image focused by a classifier . The CAM is based on a convolutional neural network with global average pooling ( GAP ) before its final classification layer . This is realized by considering the class-specific contribution of each channel of the last feature map to the classification score . Given a classifier parameterized by θ = { θf , w } where f ( · ; θf ) is the feature extractor prior to GAP , and w is the weight of the final classification layer , a CAM of the class c is obtained from an image x as follows : CAM ( x ; θ ) = wᵀc f ( x ; θf ) maxwᵀc f ( x ; θf ) , ( 2 ) where max ( · ) is the maximum value over the spatial locations for normalization . 2.3 Related Work . Weakly Supervised Semantic Segmentation : Weakly supervised semantic segmentation methods with image-level class labels first construct an initial seed by obtaining a high-quality localization map from a trained classifier . Erasure methods [ 56 , 37 , 22 ] prevent a classifier from only focusing on the discriminative parts of objects by feeding the image from which the discriminative regions have been erased to the classifier . Several contexts of a target object can be considered by combining multiple attribution maps obtained from differently dilated convolutions [ 31 , 57 ] or from the different layers of a DNN [ 35 ] . Diverse images of a target class can be utilized by considering cross-image semantic similarities and differences [ 17 , 51 ] . Zhang et al . [ 62 ] analyze the causalities among images , contexts , and class labels and propose CONTA to remove the confounding bias in the classification . Because the localization maps obtained by a classification network can not accurately represent the boundary of a target object , the initial seed obtained by the above methods is refined using subsequent boundary refinement techniques such as PSA [ 3 ] and IRN [ 2 ] . Information Bottleneck : Tishby et al . [ 54 ] and Shwartz et al . [ 49 ] use the information bottleneck theory to analyze the inner workings of a DNN . The concept of the information bottleneck has been employed in many research fields . Dubois et al . [ 15 ] and Achille et al . [ 1 ] utilize the information bottleneck to obtain optimal representations from DNNs . DICE [ 45 ] is proposed for a model ensemble with the information bottleneck principle : it aims to reduce not only the unnecessary mutual information between features and inputs but also the redundant information shared between features produced by separately trained DNNs . Jeon et al . [ 26 ] study the disentangled representation learning of a generative model [ 19 ] using the information bottleneck principle . Yin et al . [ 60 ] design a regularization objective based on information theory to deal with the memorization problem in meta-learning . The information bottleneck principle can also be adopted to generate the visual saliency map of a classifier . Zhmoginov et al . [ 65 ] find important regions for the classifier with the information bottleneck trade-off , and Schulz et al . [ 47 ] restrict the information flow by adding noise to intermediate feature maps and quantify the amount of information contained in the image region . 3 Proposed Method . Weakly supervised semantic segmentation methods using class labels produce a pixel-level localization map from a classifier using CAM [ 66 ] or Grad-CAM [ 48 ] ; however , such a map identifies only small discriminative regions of the target object . We analyze this phenomenon with the information bottleneck theory in Section 3.1 and propose RIB , a method to address this problem , in Section 3.2 . We then explain how we train a segmentation network with localization maps improved through RIB in Section 3.3 . 3.1 Motivation . As mentioned in Section 2.1 , the DNN layers with double-sided saturating non-linearities have a larger information bottleneck than those with single-sided saturating non-linearities . The intermediate layers of popular DNN architectures ( e.g. , ResNet [ 21 ] and DenseNet [ 23 ] ) are coupled with the ReLU activation function , which is a single-sided saturating non-linearity . However , the final layer of these networks is activated by a double-sided saturating non-linearity such as sigmoid or softmax , and the class probability p is computed with the final feature map TL and the final classification layer w , i.e. , p = sigmoid ( wᵀGAP ( TL ) ) . Therefore , the final layer parameterized by w has a significant bottleneck , and the amount of information transmitted from the last feature TL to the actual classification prediction will be limited . These arguments are analogous to the observations in existing methods . The information plane provided by Saxe et al . [ 46 ] shows that the compression of information is more noticeable in the final layer than in the other layers . Bae et al . [ 5 ] observe that although the final feature map of the classifier contains rich information on the target object , the final classification layer filters out most of it ; thus , the CAM can not identify the entire area of the target object . This observation empirically supports the occurrence of the information bottleneck in the final layer of a DNN . To take a closer look at this phenomenon , we design a toy experiment . We collect images containing the digits ‘ 2 ’ or ‘ 8 ’ from the MNIST dataset [ 30 ] . For only a small subset ( 10 % ) of these images , we add a circle ( ) and a square ( ) to the images containing the digits ‘ 2 ’ and ‘ 8 ’ , respectively , at a random location ( see Figure 1 ( a ) ) . When classifying images into the digits ‘ 2 ’ or ‘ 8 ’ , pixels corresponding to the digit are discriminative regions ( RD ) , those corresponding to the added circle or square are non-discriminative but class-relevant regions ( RND ) , and those corresponding to the background are class-irrelevant regions ( RBG ) . We train a neural network with five convolutional layers followed by a final fully connected layer . We obtain the gradient map Gl of each feature Tl with respect to an input image x : Gl = ∇x ∑ u , v Tl ( u , v ) , where u and v are the spatial and channel indices of the feature Tl , and for the final classification layer ( l = 6 ) , G6 = ∇xyc . Because this gradient map indicates the extent to which each pixel of the image affects each feature , it can be used to examine how much information is passed from the input image to the feature maps of successive convolution layers . We present examples of Gl in Figure 1 ( b ) . As an input image passes through the convolution layers , the overall amount of gradient with respect to the input decreases , indicating the occurrence of the information bottleneck . Specifically , the gradient of RBG decreases early on ( G1 → G2 ) , which implies that the task-irrelevant information is rapidly compressed . From G1 to G5 , the gradient in RD orRND gradually decreases . However , the decrease in the amount of gradient is prominent in the final layer ( G5 → G6 ) , and in particular , the gradients in RND ( red boxes ) almost disappear . This supports our argument that there is significant information bottleneck in the final layer of a DNN , while also highlighting that the non-discriminative information inRND is particularly compressed . We analyze this quantitatively . We define the high gradient ratio ( HGR ) of regionR as the ratio of pixels that have a gradient above 0.3 to the total pixels in regionR . HGR quantifies the amount of transmitted information from region R of an input image to each feature . The trend in the HGR values of each region for each layer is shown in Figure 1 ( c ) . The observed trend is analogous to the above empirical observation , once again supporting that significant information bottleneck forRND occurs in the final layer ( the red box ) . We argue that the information bottleneck causes the localization map obtained from a trained classifier to focus on small regions of the target object . According to Eq . 2 , the CAM only includes information that is processed by the final classification weight wc . However , because only a subset of the information in the feature is passed through the final layer wc due to the information bottleneck , leaving out most of the non-discriminative information , CAM can not identify the non-discriminative regions of the target object . It is undesirable to use such CAMs to train a semantic segmentation network , for which the entire region of the target object should be identified . Therefore , we aim to bridge the gap between classification and localization by reducing the information bottleneck .

Summary: This work addresses a limitation of CAMs (Class Activation Mappings) in the context of weakly supervised semantic segmentation (W3S) taking the information bottleneck perspective. The reason for the CAMs being highly selective, it argues, lies in the use of double-sided non-linearities (e.g. sigmoid) in the final layer of the classification network. The work removes this non-linearity and develops a strategy of obtaining high-recall attention maps through a one-shot adaptation process. Used as initial masks in a SotA W3S pipeline, these masks impressively increase the final segmentation accuracy achieving new state-of-the-art on established benchmarks.