Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: - The paper analyzes ensemble-based debiasing (EBD) methods for training models that don’t rely on dataset biases and thus has good out-of-distribution performance
- The theoretical analysis shows that the debiased model performance decreases monotonically as calibration error increases, and properly calibrating the model can improve not just out-of-distribution performance, but also in-distribution performance under certain conditions
- Based on the theoretical analysis, the authors propose a modification to EBD methods, where the bias-only model is calibrated. Empirically, this improves the performance of debiased models.

main_review: - The paper highlights the importance of calibrating bias-only models for EBD methods. Although this is not so surprising given that EBD methods rely on probabilistic estimates outputted by bias-only models, the findings (both theoretical and empirical) are novel to my knowledge.
- The theoretical analysis clearly illustrates the benefits of calibrating the bias-only model both for out-of-distribution and in-distribution model, and this directly motivates the proposed modification to the EBD methods. The modification (adding a calibration step) is simple, but well-motivated, practical, and novel to my knowledge.
- The paper includes thorough, systematic experiments showing that the proposed modification improves the debiased model performance, in line with the theoretical analysis. The empirical improvement in performance is small, especially relative to the empirical gains yielded by the standard EBD methods, and this could make the key takeaway of the paper limited in its impact. However, the empirical gains are consistent across multiple datasets, calibration methods, and debiasing methods. They also have additional empirical analysis, which are also consistent with the theoretical results.
- The paper is clearly written, and I enjoyed reading it! 

Minor comments
- For Theorem 1, what kind of values should I expect for the threshold? I understand it depends on latent constants, but it’d be nice to get a rough range.
- It took me some time to parse some of the theoretical notation and statements. For example, I found the notation for debiasing performance in Theorem 1 unclear.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper reveals that uncertainty calibration of the biased model is important for debiasing performance in EBD methods. By calibrating the biased model, the proposed method improves the performance of existing methods. 
main_review: Originality: to the best of the reviewer's knowledge, this paper is the first to discuss calibration in EBD methods even though the topic of calibration has been studied extensively.  To some extent, the paper lacks novelty because the paper uses existing techniques without modification to improve existing EBD models. 

Clarity: the paper is well organized and easy to follow. The method starts with the formalization of the EBD (Sec.3) method and theoretical (Sec.4.1) and empirical analysis (Sec.4.2) of the bias-only model. Then it introduces calibration techniques and the complete pipeline (Sec.5). However, there are some technical details that require further clarification. Please see the questions. 

Quality: the paper conducts theoretical analysis and verifies assumptions empirically. 

Significance: the paper points out an overlooked problem in EBD methods. The work can inspire further investigation of calibration problems in ensemble-based debiasing methods. 

Questions:

1, Is the predicted label $Y(s)$ from the intrinsic model (line 136) different from the label $\tilde{Y}(x)$ predicted by the ideal predictor (line 166)? Should they be the same? 

2, The global certainty level $\alpha:=min_{X^s} max_{i\in\{0,1\}} P_D(Y=i|X^s)$ (line 149) is the least confidence prediction from the true principle.  Can the author justify its role as a "global" certainty level? It's not clear why this quantity conveys a "global" view of certainty. 

2, Regarding Figure 3: the plot shows monotonically increasing performance with temperature. As the temperature increases,  the biased models will become more and more uncertain. At some point, this must hurt the debiasing performance because when the distribution of the biased model is almost flat, it does not provide any information to the unbiased model.  Can the authors comment on this thought? 

**--------------------post-rebuttal review--------------------**

I appreciate the authors' response and discussion on my questions. I think the paper uncovers an overlooked problem in EBD methods and can inspire further research. 
limitations_and_societal_impact: Yes, the authors have addressed the limitations of the paper especially the inconsistent results on image data.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper provides theoretical and empirical proof that the bias-only model used for two-step debiasing methods does not provide adequate and accurate uncertainty estimation. Via both mathematical proofs and empirical experimentation on two NLP tasks, the authors show that bias-only models have been over-estimated and overlooked, and that further work is needed to improve them. 
main_review: This paper provides a clear and interesting critique of the bias-only model used in two-step debiasing methods. 

Some comments that I had while reading the paper: 

Please note that  some recent works [22; 9] propose to jointly learn the bias-only model and the debiased main model  in an end-to-end manner. Since it is difficult to quantify the impact of the bias-only model in this scheme, we mainly focus on the two-stage methods  == How can your methodology be applied to these kinds of methods?

For Figure 1, wouldn't it be worth discussing the difference between the two plots (and the two tasks?)

Line 228-229: "In other words, it only changes the uncertainty estimation229 and *maintains* the model’s accuracy? 
== not sure what 'remain' means

Why were fact verification and NLI the tasks chosen?

Is using a hand-crafted classifier for the NLI task comparable to a data-driven one for the fact verification one? Are they equivalent and comparable?

Any hypotheses why the method doesn't work as well on Learnd-Mixin on HANS? Is there a difference between that approach and others?

Line 359: "poorly calibrated"

"Further experimental results on image classification show inconsistent improvements" == maybe it depends on the image task? would it be worth trying tasks that are more complex and multi-modal, as opposed to simple classification? e.g. captioning or VQA?

What are some ways in which the bias-only model can be improved? I feel like there should be a lot more discussion of this, as well as the advantage of the joint bias and debiased model approach that was not studied at all in this paper.
limitations_and_societal_impact: The fact that the paper focusses purely on NLP tasks and shows inconsistent improvement on image tasks is worrisome for me. What if the phenomena that were observed during the emprical experimentation were intrinsic to the nature of the two NLP tasks chosen (which is actually very few)? The fact that this method doesn't apply in image domains is worth exploring and explaining.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper shows an increasing connection between the uncertainty estimation of the biased only models and proposed MoCaD, a three-stage EBD approach considering model calibrating.
main_review: Originality: High. The combination of uncertainty calibration and EBD is a very novel idea and the impressive results shown in this paper validates the effectiveness of such combination.

Quality: The submission is technically sound, with the major claims in Section 4 being backed up by both theoretical analysis and experimental results in Section 5. however, I do feel that the experiments section can be improved to reveal more insight into why MoCaD works well. 

Clarity: This paper is well organized, although I struggled to remember the complicated notation system used in the paper. The authors should spend some time on improving the nomenclature even at the cost of abusing some notations, consider that various kinds of probabilities are needed in both main body and appendix. The paper does provide enough information to reproduce its results.

Significance: High. Debiasing is an increasingly important area in ML, and this work has the potential to serve as a foundational piece for future development. 

Some areas where improvement can be done:

Experiments. From Theorem 1-2, the authors nicely connected the uncertainty calibration of bias-only models to the final performance of the debiased models, and motivated the use of calibration methods. In particular, in Section 6.2.1, the Dirichlet scaling method seems to work much better than the temperature scaling method. This is understandable, as Dirichlet scaling is more expressive, but it can also introduce some accuracy change that might hurt the debiasing model. Therefore, I would like to see  if more expressive calibration methods are indeed helpful, for example, the more expressive temperature scaling (http://proceedings.mlr.press/v119/zhang20k.html) and the nonparametric calibration <http://proceedings.mlr.press/v108/wenger20a.html>. Besides the class-wise ECE, the authors might also want to look into the confidence-based ECE (http://proceedings.mlr.press/v70/guo17a.html), which is related with theorem 2, as well as the accuracy of the calibrated bias-only model (to check if the calibration method introduces non-negligible hurt on that). Section 6.2.2 is indeed very informative and completely backs up Theorem 2, good job!

The title might be confusing. It might be interpreted to how to calibrate the uncertainty of final debiased model, while the uncertainty is in fact only calibrated for the bias-only model.

Some typos, e.g., Figure 3 'Syntatic Bias'.

------------After Rebuttal------------
I have read the authors rebuttal and appreciate the new experiments comparing accuracies. This moves the scores upward. 
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper studies the uncertainty calibration properties of the bias-only model in ensemble-based de-biasing methods. The authors argue that the accurate estimation of uncertainty in the bias-only model has been overlooked in prior research, but can have an important influence on the overall de-biasing process. The method proposed in the paper applies calibration methods to the output of the bias-only model. Thus, from a novelty perspective, the method itself is a novel combination of existing approaches, while at a conceptual level the paper is addressing a previously overlooked issue and thus has stronger novelty at this level. The reviewers judged the approach and the theoretical development to be technically correct. The experiments were judged to provide adequate support for claims. The writing was judged to be clear. The reviewers indicated that all of their questions had been adequately addressed following the author response. As a result, the paper is recommended for acceptance. The authors should be sure to incorporate the discussed clarifications and updates in the final manuscript.
Decision: Accept (Poster)