Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Recent semantic segmentation methods often learn biased representation from the dataset with highly co-located objects (e.g., motorcycle and road) as well as the class imbalance. This paper proposes a training scheme of dropping the class-specific representations in the highly-entangled features, mitigating the bias induced by the dropped class. The proposed DropClass is also model-agnostic and demonstrates the improved segmentation results in Cityscapes and Pascal VOC datasets.
main_review: Strength
1)	The paper is well organized and easy to follow. 
2)	The main problem setting of the dataset bias in the semantic segmentation task is novel. 
3)	The improved IoU scores, particularly for the under-represented classes, are significant.

Weakness
I think the overall experimental settings and results only show the improved performance on under-represented classes without evaluating the debiased performance between highly correlated classes, which is the main topic addressed in this paper.
-	Tables 1 and 2 do not contain the information on which classes suffer from the bias. I am concerned that the low class frequency does not necessarily cause the classes to be co-located with other classes. In other words, frequently appearing classes may also suffer from the performance degradation due to the highly correlated object classes.
-	Similarly, are Train-Truck, Truck-Bus, and Bicycle-Motorcycle highly co-located objects in the Cityscapes dataset? Does Vanilla fail because they have the correlation between the classes or because the frequency of such classes is low?

I would like the authors to 1) specify the classes vulnerable to the bias due to the unwanted correlation with other classes and 2) provide the quantitative comparison between classes with bias and without bias. (e.g., class A is often highly co-located with class B, and the segmentation results of A are xxx worse compared to the class C which does not have the unwanted correlation with other classes.) Additionally, the authors need to demonstrate the improved results of these classes by the proposed DropClass.
limitations_and_societal_impact: Yes. The authors included the limitations and negative societal impact in Section 5.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents DropClass, a new training scheme for reducing feature dependencies among classes to learn debiased and disentangled feature representations. Toward this, the paper 
consider Grad-CAM for obtaining the class-specific information, and then randomly discard such information to remove any bias towards a certain class. 

A set of experiments on benchmark datasets demonstrate the model’s effectiveness, primarily on under-represented classes. 
main_review: Strength:
* The proposed training scheme is novel, model-agnostic, and straightforward. 
* The experiment covers two crucial datasets in the segmentation literature. Results demonstrate that the inclusion of the proposed training improves over baseline (two well-established frameworks), notably in the under-represented classes. 
 
Weakness:
* The paper aims to remove the issue of feature entanglement and bias vulnerability. These two topics have been well studied in the current ML/DL literature. In my opinion, the presentation of these topics is a bit hand waving. While reading the paper, one gets the impression that these two are interchangeable. The primary goal seems to make sure the model generalizes well across test sets. If so, is this work trying to use disentanglement to achieve debiased representations and then generalization? I think the paper should be clear enough to answer them.  
* Lack of comparison against other debiasing techniques. Paper acknowledges and reviews some of them, but no technical comparisons have been made on how such methods would perform in the presented dataset and network architecture. The same applies to disentanglement techniques. 
*  The motivation behind L_sup (Eqn 8) is not clear. The impact of Eqn 8 also looks pretty minimal (Table 3). 
* The methodological contribution is limited. 
* The provided results are not enough to clearly understand the benefit of the proposed training scheme. It is not clear how biased or entangled the baseline framework is before using the proposed training scheme. Only figure 2 is provided to discuss this critical issue. 
 
Questions:
* Authors are encouraged to answer/comment about the points raised in the weakness section. 
* Some recent works (Frosst et al. 2019) in disentanglement have pointed out that "maximizing the entanglement of representations of different classes in the hidden layers is beneficial for discrimination in the final layer, possibly because it encourages representations to identify class-independent similarity structures". Can authors discuss them in relation to the proposed training approach? 
Frosst, Nicholas, Nicolas Papernot, and Geoffrey Hinton. "Analyzing and improving representations with the soft nearest neighbor loss." International Conference on Machine Learning. PMLR, 2019.
* Isn't this a training procedure? Why does one need to analyze "Test time behavior"? 
 
Grammatical errors:
* Line 140: "encouraged learn debiased" —> "encouraged to learn debiased" 

********************* Update **********************
The score has been updated (5 to 6) after the authors' rebuttal. 
 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a DropClass training scheme where class-specific feature maps are used to obtain debased and disentangled representations. This is achieved by randomly dropping out class-specific representation from the feature maps, implemented in an annealing manner, reinforced with a regularization term to further remove class-specific information from the remaining information. Experiments are performed on three segmentation networks on two distinct data sets. The presented method seems to achieves an improvement of performance in most classes, with the most noticeable margin of improvement in minority classes.
main_review: The presented concept is interesting and original. The methodology is well motivated and clearly described. Experiment results are promising. Some technical questions can be clarified. 

First, to make sure that the gradient estimated is reasonable, the authors mentioned that the loss function slowly incorporates DropClass "by linearly increasing the vlaue of lambda from 0 to 1 across the duration of training". How is this linear rate determined exactly, assuming that the training is terminated based on some validation loss and thus the number of epochs needed is not known in advance. Or is the linear ratio pre-determined and irrelevant to the number of training epochs? Furthermore, since it is important to wait until the gradient estimation is reasonably accurate, would some exponential or other types of increase more effective than a linear increase rate?

To avoid heavy computation associated with gradient calculation for each class, the authors focused on convolution layers and implemented the final classification layer using 1x1 convolution as well. Does this mean the presented method will be restricted to convolutional networks? 

In experimental results in Pscal, it seems that nearly half to the classes were negatively impacted. What is th implication? Does it suggest that DropClass should be targeting certain classes only rather than all classes, or targeting different classes with different weights? If so, how?

Overall this is a very well written paper that presents an original idea in a simple fashion. It has high discussion potential. 
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: This work describes a new training scheme for semantic segmentation models (“DropClass”) that is intended to encourage the learning of feature representations that avoid entangling features from objects (e.g. car & road) that tend to co-occur/co-locate. 

Reviewers were generally positive about the work, and found it to be interesting and novel. Reviewers and authors had something of a philosophical debate about the nature of dataset bias and entanglement, and it will be helpful for the authors to clarify these terms in the resulting paper, and to ensure that limitations of the method (e.g. tendency for low frequency classes to become entangled with semantically similar class) are appropriately highlighted for readers. All in all, this is strong work that should be of interest to the NeurIPS community.

Decision: Accept (Poster)