Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a theoretical analysis of the methods of subspace removal using probing classifiers. The main argument is that even the features are perfectly separable, the probing classifier will learn something about the task-related features, in addition to the concept related features. Then, based on this argument, this paper claims that the existing probing-classifier-based methods are not sufficient to remove concept-related feature/information. 
strengths_and_weaknesses: **Strengths**

Overall, I think this paper is solid in both theoretical analysis and empirical study. 

First of all, with some mild assumptions, this work presents a theoretical analysis of why probing-classifier-based method is not reliable in removing concept-related information. To some extent, the assumption is too ideal; for example, it assumes disentangled representation, which I am not sure it is true for most cases. However, this ideal case demonstrates the concept. 

In addition to the theoretical analysis, it also presents some empirical studies, which further strengthens this work. Compared to the existing works that only demonstrate the algorithm with empirical results, I like the combination of theory and empirical results. 
questions: No further question
limitations: No limitation
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper analyzes the potential issues with previously proposed methods for sensitive concept removal from text representations. The authors mainly study two branches of methods, null-space removal and adversarial removal, which both use probing classifiers operated on frozen representations. The authors present theoretical analyses to demonstrate that both types of methods cannot remove the sensitive attributes completely and may distort main-task features that hurt the target task performance. The empirical studies are conducted on a set of synthetic and real datasets to support the claims.
strengths_and_weaknesses: Pros:
* The problem studied is interesting and important -- how to effectively remove sensitive features from text representations is of great significance in mitigating ethical concerns about machine learning models.
* The paper presents both theoretical and empirical analyses to support the claims.

Cons:
* The major contribution of the paper does not appear to be that significant to me. While it's important to know the potential issues with previous concept removal methods, the main arguments in this paper (that both null-space removal and adversarial removal methods cannot entirely remove the sensitive features) are more or less already mentioned in the original papers proposing those methods. In this regard, the paper is reiterating some previous observations by backing them up with more theoretical analyses (however, these theoretical analyses appear to be based on very strong assumptions, see below).
* Some assumptions made in the paper are too strong to be meaningful in practice. The authors assume the sensitive features need to be removed entirely from the representations (i.e., a perfect separation between main-task features and spurious ones). However, the concept removal methods are usually not expected to be producing a perfectly clean representation in the first place; they are meant to mitigate the influence of the spurious features. Those methods should be considered useful as long as they can (to some extent) remove the sensitive concepts. Therefore, the "perfect removal" assumption appears too strong--I doubt whether there even exists any method that could satisfy such an assumption. If the assumptions are made in a way that the requirements are essentially impossible to satisfy, the subsequent theoretical analyses that show how previous methods may not work will appear much less meaningful.
* Although the authors claim the spuriousness score as one of the contributions, there are no further evaluations on how accurately this metric corresponds to its expected purpose. 

**Post-Rebuttal**: The authors have addressed my concerns raised above, so I am increasing my rating.
questions: I would like to see some insights/ideas derived from the analyses regarding how the previous methods may be improved. Usually, good theoretical analyses not only reveal the challenges/limitations, but also shed lights on the potential directions to overcome them.
limitations: I hope the authors would discuss how the assumptions made could be relaxed (e.g., the spurious features may not be assumed to be removed entirely, but to some extent).
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper discusses the problem of spurious correlations in natural language processing (NLP). The authors discuss the well-known issue that machine learning-based NLP models tend to learn representations that encode spurious correlations between input feature attributes (the authors refer to these as “concepts”) and class labels, thereby causing models to fail to generalize to out-of-distribution data. Specifically, this paper investigates the limitations of two classes of methods proposed to remove spurious correlations, namely null space projection and adversarial training. The authors provide theoretical and empirical evidence that the aforementioned methods fail to alleviate the problem of spurious correlations in NLP by showing that they cannot successfully remove undesirable concepts. For the latter, empirical experiments are conducted on a synthetic as well as three real-world datasets: MultiNLI, Twitter-PAN16 and Twitter-AAE.
strengths_and_weaknesses: Strengths:
This paper addresses the highly relevant problem of spurious correlations in NLP and shows that existing methods to tackle that problem are limited at doing so. The paper is well-written, the theoretical analysis and the experimental efforts for empirical validation are well-documented and easy to follow, and the obtained results for the latter are convincing. Furthermore, the paper’s main objectives and findings are well differentiated from the existing literature (as stated in Section 5). 

Weaknesses:
My main concern with this work is a lack of discussion on potential ways forward. The paper successfully shows the limitations of existing methods for the removal of spurious correlations. However, the authors do not extensively discuss the implications of their findings and show little effort in discussing potential ways forward and/or alternative approaches to improve existing methods. The manuscript contains various spelling mistakes and missing words (e.g., lines 182, 190, 201, 280, 342), which I encourage the authors to correct for better readability. Finally, some crucial information (e.g., around the predictive correlation; also see my question below) are moved to the Appendix. This is understandable due to limited space but can disrupt the flow for the reader. 
questions: * For the “Ablations” paragraph in Section 4: would it be possible to briefly elaborate on the results with other classifiers and model design choices in the manuscript (not only in the Appendix)? I appreciate that the experiments have been conducted under various model types and configurations, yet it would be beneficial to the reader if the main findings for this could be mentioned in the manuscript.
limitations: The authors do not explicitly mention the limitations of their work in the paper (a discussion beyond the results is missing) and the paper would benefit from a few sentences on that topic as well as future work based on the main findings.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose that probing-based concept removal and detection methods are unreliable given that the representation space may be corrupted, or the resulting model may still reply to those seemingly removed concepts. The authors try to ground their hypothesis on theoretical based on toy examples in 2D space, and then provide pieces of evidence using a set of naturalistic and synthetic benchmarks where the correlations between the concept and the sequence label are controllable. Although the topic is interesting and the research question is worth pursuing further, I found a set of opening questions that remain unresolved in the paper.
strengths_and_weaknesses: **Strengths**:
- This paper aims to address a pressing issue in building debiasing methods of neural networks.
- The toy examples provided in the paper are intriguing.

**Weaknesses**:
- **Is the failure mode of INLP really a valid failure mode?**

The main argument for disqualifying INLP from being a sound method is that INLP will corrupt the representation even though we know there are no concept-level relations with respect to the final label in the task. In Sec. 4.2, the authors train the model using a special subset of training data where all examples are ``with the same value of the property`` (L305). I assume the authors only allow the model to see training examples with the same concept label y_{p}. With this, the authors claim that INLP should have a null effect in removing information in the representation space. I find this argument unconvincing, if not wrong. First of all, INLP in this case will first train a probe classifier with simply 100% accuracy, since it only needs to predict the same label for all examples. For the iterative project process, it needs to remove information to decrease the accuracy from 100% to 0% (if possible). It is obvious that INLP will definitely remove sequence-related information from the embedding no matter what, which eventually will just randomly corrupt the representations. As a result, the results in Fig. 2 are indeed expected. These results are not because INLP is effective, the experiment setups naturally lead to these results.

- **Is the failure mode of the adversarial removal methods really a valid failure mode?** 

All experiments conducted in Sec. 4.3 relies on the predictive correlation (κ). Let us say we have κ=1.0. As we are training a probing classifier to have an adversarial loss to remove concept-related information, there is no way to guarantee this will happen, especially in this case where the main-task label is 100% correlated with the concept label. The probing classifier may just be another main-task classifier that is trained to lower the main-task accuracy. In this case, it is not actually a failure mode about the method again, it is a tricky design of the dataset that fools the method. In fact, when the main-task label is 100% correlated with the concept label, there is no way we can ensure we are removing concept-related information if only with classification-driven losses.

I found the topic interesting, but my main concern is those two real-world study cases are not really exposing weaknesses of those concept removal methods, and are rather just some corrupted datasets that will trick any method.

[Update]: Thanks for the clarification on the INLP, as the description was a little confusing to me. Maybe considering updating the text. In the case study of the adversarial removal methods, I am still not convinced yet that building spurious correlations within the dataset is a good way to say the failure mode of these methods. I am raising my score to 5 as I am leaning towards acceptance.

questions: - L146, says ``w.r.t. task label y_p (Assumption 3.3)``, should it be the concept label instead of the task label?

- Fig 1. (a) is confusing, if not wrong. For instance, for the Gender Probing Classifier (annotated as the blue dashed line), with accuracy = 100%, it is clearly mislabeling multiple examples.

- I would suggest moving at least the essence of the *Proof Sketch* to the main text as it is essential to see formal proofs to back up the authors' main hypotheses. Otherwise, it feels ungrounded.

- In Sec. 4.2, the authors train the model using a special subset of training data where all examples are ``with the same value of the property`` (L305), could you clarify a bit what is the property? and how to have the same value?

- Can authors define ERM before using it as a term in L338?
limitations: The authors adequately addressed the limitations and potential negative societal impact
review_title: null

Meta-review: metareview: This papers analyzes failure modes of methods that aim to remove spurious features from the representation. The key finding is that since the spurious features are correlated with the core features, such methods will inevitably also remove core features during the process, thus hurting performance. Both the theoretical results and the empirical findings are important for understanding concept-removal methods which are widely used in domain adaptation and robust learning. All reviewers agree that the contribution is significant. The authors may strengthen the paper by discussion ways forward. 
confidence: Certain
Decision: Accept