Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper is about semantic image segmentation training with image-level labels. The method is using an auxiliary loss to help to find a larger localization area for pixel-wise training. Experiments are done on Pascal VOC2021 and MSCOCO.
main_review: 1.A flowchart and an Algorithm to describe the whole method is absent.

2.This paper claims the information bottleneck or Data Processing Inequality is important for recognition and remove the last sigmoid. For a L-layer network with same resolution, ResNet-v2 style network has absolutely less information loss than ResNet-v1. However, we often find ResNet-v1 behaves better than ResNet-v2. It seems the information loss is not that important? Or what kind of suggestion could this concept provide to design a better network structure? 

3.In page 5 Line 200, “Note that for each iteration, B−1 samples are randomly selected, while x is fixed.” This sentence is confusing.

4.In Table 4 it seems the backbone of proposed method is only ResNet-101. It is necessary to include more backbone, such as ResNet-50, Efficient, Transformer, etc., to prove the robustness of the method.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work addresses a limitation of CAMs (Class Activation Mappings) in the context of weakly supervised semantic segmentation (W3S) taking the information bottleneck perspective. The reason for the CAMs being highly selective, it argues, lies in the use of double-sided non-linearities (e.g. sigmoid) in the final layer of the classification network. The work removes this non-linearity and develops a strategy of obtaining high-recall attention maps through a one-shot adaptation process. Used as initial masks in a SotA W3S pipeline, these masks impressively increase the final segmentation accuracy achieving new state-of-the-art on established benchmarks.
main_review: *Originality.*
The approach proposed in this work has a number of novel and interesting elements. However, the problem with CAMs and the classification providing only “discriminative” regions is well-known and some attempts to address were not discussed or compared, such as LSE [A] and nGWP [B]. In particular, the motivation in [B] is not so dissimilar from the one in this work, as it also encourages the increased mask size through a loss term. Adding a discussion and a comparison to this alternative approach would make this work stronger.

[A] Pinheiro and Collobert. From image-level to pixel-level labeling with Convolutional Networks. CVPR 2015.
[B] Araslanov and Roth. Single-Stage Semantic Segmentation From Image Labels. CVPR 2020.

*Quality.*
The work is well-executed. I particularly appreciate the experiments both in the “labels-only” and “+saliency” setups on VOC-12, as well as experiments on MS-COCO. Model sensitivity to hyperparameters $m$, $\tau$, $K$ and $B$ are well-studied.

One remark however:
I do not see the necessity of introducing the circles and squares in the motivating experiment in Sec. 3.1. $G_6$ in Fig. 1 shows perfectly well that only a fraction of the digit is relevant for the final layer, whereas in Fig. 2 we now observe the tendency towards uncovering the full extent of the digit. So, why do we need the circles and squares? In natural images those may represent spurious correlations (e.g. an umbrella, which often co-occurs with the class “person”), hence the method will only pick up on those, as this toy experiment suggests.

*Clarity.*
The manuscript is well-written and follows a clear structure. The analysis and qualitative examples are insightful.

Minor issues are:
- ll. 134-135 “significant bottleneck”; “amount of information … will be limited”: these points is hard to agree with, since no measure is introduced. I’d suggest to make it relative to one-sided non-linearities, for example, to connect it with the discussion in Sec. 2.1.
- I’d consider renaming HGR, since it does not reflect the custom threshold of 0.3 (l. 164). As a result, Fig. 1 (c) and 2 (b) are unclear without a reference to the text.
- Please, add numbers to the equations on p. 5.

*Significance.*
The paper offers an interesting perspective on addressing the limitations of CAMs on which virtually all weakly supervised semantic segmentations  rely. The RIB process is relatively straightforward, the experiments demonstrate clear benefits. Future works on W3S and, perhaps, attribution methods, may find this technique useful. 

**Post-rebuttal update**

I thank the authors for their response, and I keep my initial recommendation to accept this work. Nevertheless, I hope the design of the toy experiment can be still re-considered in the camera-ready version in view of my remark on spurious correlations.
limitations_and_societal_impact: Limitations seems to be scattered throughout Sec. 4, which does not give a clear impression that could be useful for follow-up work. I recommend offering a discussion on the limitations in a more prominent way. The work does not discuss the potential societal impact which needs to be addressed in the revision.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on the problem of weakly-supervised semantic segmentation, where the goal is to conduct segmentation by mainly using image-level annotation as such weak form of annotations can be collected more efficiently. 

Specifically, authors analyze the issue of Information Bottleneck and propose a simple method RIB to reduce it to obtain more complete CAM localization maps. A e a global non-discriminative region pooling (GNDRP) is further applied to enhance the effects of RIB. 

Experiments on PASCAL VOC  and MS COCO datasets show the effectiveness of the proposed method.
main_review: Strength:
+ This paper begins with toy experiments to illuminate the problem of information bottleneck, which is intuitive and clear.

+ The ideas of RIB and GNDRP are reasonable and effective. 

+ Extensive experiments have been conducted to show the contributions of the proposed method. Parameter analysis has also been conducted to help analyze the behavior of the model. 

+ The proposed method shows strong performance when compared with recent state-of-the-art methods.  



Concerns and Questions:

- The descriptions and motivations of RIB training method are not clear (line 189-201). For each RIB iteration, is the model updated by the gradient of L_RIB for a fixed image x as well as L_RIB for other N-1 randomly sampled images? Why is it necessary to set several RIB iterations for each specific image x? Why not just conduct RIB loss for images in a batch and fine-tuning the model for several epochs? 

- For the details of using GNDRP, is it only applied at the training stage (RIB iteration)? Then at the inference stage of CAM maps, I believe the original GAP layer is still used rather than GNDRP. It that correct? The idea of GNDRP is quite similar to hide-and-seek method operated on feature maps, which is not new. 

- For results in Figure 3, are they obtained by only applying RIB iterations (RIB-GAP)? Or they are actually obtained by RIB-GNDRP. If they are from RIB-GAP then it is fine. Otherwise, showing results of RIB-GAP would be more helpful. 

- Can the proposed method generalize to Grad-CAM instead of only on CAM? Analysis, discussions and even experimental results regarding this point would help to show the generalization of the method. 
limitations_and_societal_impact: Not applicable. Authors did not discuss limitations and potential negative societal impacts of the work. See check list 1 (c).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new method for computing class activation maps (CAMs), which has been used as crucial evidence for object detection and segmentation in a weakly supervised learning setting. A well known drawback of CAM and its variants is that they often highlight only small discriminative regions of an object. This paper interprets such a phenomenon through “information bottleneck” and proposes a way to alleviate it accordingly. In short, the proposed method finetune a classification network for generating the original CAM with a different classification loss and a different global pooling layer that allow more features to contribute to classification. The method enhances the quality of the original CAM noticeably and consequently improves performance of a weakly supervised semantic segmentation method relying on CAMs.

The main idea of this work is novel, and useful to alleviate the well-known limitation of the CAM and its variants. Moreover, the efficacy of the proposed method has been demonstrated by extensive experiments. However, the overall pipeline for semantic segmentation depends heavily on an existing method (IRN [2]), and it demands a much larger number of hyper-parameters compared to the original CAM. 

main_review: [Strengths]
- The paper is overall well-written; it is easy to follow in most parts and well-organized. Also, details necessary to reproduce the proposed method are presented in the paper.
- The main idea is novel in that there is no previous work explaining and alleviating the well-known limitation of the CAM and its variants through information bottleneck.
- The efficacy of the proposed method has been demonstrated by extensive experiments on PASCAL VOC and MS-COCO, where weakly supervised semantic segmentation based on the proposed method achieves the state of the art. This reviewer in particular appreciates its strong performance on the challenging MS-COCO dataset. 
- Also, the effect of hyper-parameters is investigated by extensive ablation studies.

[Weaknesses / questions]
- The proposed method is applied only on weakly supervised semantic segmentation and its overall pipeline for semantic segmentation relies heavily on IRN [2]; considering this work as a weakly supervised semantic segmentation paper, its contribution looks quite narrow. One way to resolve this issue might be applying the proposed method to other weakly supervised visual recognition tasks, e.g., object localization and instance segmentation.
- The method demands a larger number of hyper-parameters, compared to the original CAM. Although their effects are investigated by ablation studies, this reviewer still wonders how to set the number of RIB iterations, i.e., K, and what happens when K > 10; these analyses are missing in the current manuscript.
- It is not explained why the final localization map is computed as the sum of CAMs over RIB iterations, instead of just choosing the CAM of the last RIB iteration.
- The design of experiments in Section 3.1 is not convincing. In particular, it is not explained at all why the circles and squares should be considered as class-relevant yet non-discriminative regions. 
- The authors seem to assume that class-relevant yet non-discriminative regions are always parts of an object, which is not true unfortunately since such regions may include class-specific background stuff (e.g., rails near trains). Thus the proposed RIB finetuning could damage precision while improving recall. In the same context, this reviewer would recommend analyzing, evaluating, and comparing the proposed method with others through precision-recall curves. 

limitations_and_societal_impact: Limitations of this work are not summarized in a separate section. This reviewer could not find any potential negative societal impact of this work.
review_title: null

Meta-review: The submission proposes an improved method for computing class activation maps, which are then used in weakly (image level label) supervised segmentation.  The main problem with class activation maps are that they often only highlight a small portion of an object.  Addressing this issue is the main focus of the paper, and does so with a different classification loss and a different global pooling layer.  The reviewers were unanimous in their opinion that the paper is above the bar for acceptance at NeurIPS, and appreciated the extensive experiments and utility of the setting.
Decision: Accept (Poster)