Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a new benchmark called CEBaB for evaluating concept-based explanation methods. This benchmark dataset consists of 15,089 restaurant reviews obtained by editing 2,299 original reviews from Opentable. Crowdworkers are asked to modify an original in order to alter a specific aspect (food, ambiance, service, or noise) in the review while holding all other aspects constant. This process creates pairs of examples where only one aspect of them is different, which allows estimating the causal effect of changing the particular aspect. The paper evaluates 6 different explanation methods by measuring the distance of the concept-based explanations w.r.t. the estimated causal effect.

strengths_and_weaknesses: Strengths:

The paper collects a natural language dataset with pairs of human-written counterfactuals, which allows building.

The experiments cover an array of popular explanation techniques with varying requirements of access to the models.

The writing is clear and easy to follow.

Weakness:

While the collected benchmark and the experiments of evaluating different concept-based explanation methods are a central contribution, the paper does not provide some important details .

The paper does not provide thorough discussion of results.

The paper does not clearly discuss the relationship with other work on evaluating concept-based explanation methods.

questions: *Benchmarks:*

Have the authors verified whether edits affect other aspects? The paper verifies that the edits successfully achieve the aspect-level goal (altering a particular aspect towards a specific direction), but there are no descriptions on whether other aspects are constant, which can be done with the help of crowdworkers on a small number of examples in the benchmark

Are the final aggregated review and model score sensitive to different edits of a particular aspect-level goal? For each original review and a specific aspect-level goal, the paper collects one edited text. Ideally, it’s better to have multiple samples (as stated in Eq (2)) for estimating the ICaCE and ATE. It might be worthwhile to collect multiple edits for a relatively small set of examples and investigate the sensitivity.

*Experiments:*

The paper benchmarks the performance of different explanation techniques and presents some interesting findings. E.g. TACV and ConceptShap performs worse (or on par) with the random baseline, even on the straightforward food aspect. It might be worth elaborating or providing qualitative analysis of the failures of these methods.

*Related work*

While the central contribution of this paper is benchmarking. The paper mainly discusses different explanation techniques. It might be good if there could be more discussion on the evaluation side, especially metrics and experimental findings. E.g., whether other work [11,56] uses the same or mostly similar evaluation metrics but evaluates on synthetic tasks.


limitations: See questions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces CEBaB, a new benchmark dataset for assessing concept-based explanation methods in the domain of Natural Language Processing (NLP). 
CEBaB consists of short restaurant reviews with human-generated counterfactual reviews in which an aspect has been modified. 
CEBaB is leveraged to compare various concept-based explanation methods.
The purpose of the paper is to establish natural metrics for comparative assessments of these methods.
More precisely, five leading concept-based explanation methods: CONEXP, TCAV, ConceptSHAP, INLP, and CausaLM.
The dataset of reviews is composed of four aspects (food, service, ambiance, noise), associated with binary evaluation and an overall evaluation estimated from 1 to 5 stars.
The dataset is crowdsourced from an initial set of OpenTable reviews.
As a first step, the crowdsourcers have to expand the reviews to focus on one specific aspect.
In a second step, the crowdsourcers have labeled each aspect of the edited reviews.
The notion of the concept associated with the reviews is not clear, I presume it will be words.
The authors have used it as a classifier to explain a fine-tuned pretrained language model to predict the overall binary sentiment of all restaurant reviews
strengths_and_weaknesses: Strengths.
 * The paper addressed a pertinent question of the explainability of large neural models
 * The paper is clear and properly federate in a unified description method a relatively large spectrum of explanation methods
 * The dataset is large and addressed the problem of explanation in NLP which is particularly known as tedious
 * The dataset has been crowdsourced in two separated steps which is a reasonable indicator of the limited noise in the data.
 * The paper already compares a large set of explanation methods with the resulting dataset

Weakness
 * Unfortunately, the dataset is only in English, as most of the pretrained models are now multi-lingual it could have been interesting to consider this point.
 * The notion of concept is not clear in the context of the reviews, I assume it is the tokens.

questions: Can you please detail the definition of the concept in the dataset? This point is central to all evaluated explanation methods.
Did you consider augmenting the dataset through paraphrasing, do you think it would cause an additional challenge?
limitations: I do not see a strong limitation in regard to the purpose of the paper and the associated contribution and results.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduced CEBaB, a new  benchmark dataset for assessing concept-based explanation methods in Natural Language Processing (NLP). CEBaB consists of short restaurant reviews with human-generated counterfactual reviews in which an aspect (food, noise, ambiance, service) of the dining experience was modified. 
strengths_and_weaknesses: Strengths:
Evaluating explanation methodology for causal effect is important for the research community. The dataset would be useful for that purpose. 

Weakness:
The dataset evaluates the model explanation methods based on aspects (food, noise, ambiance, service). This dataset can be used for sentiment analysis. I am not sure if this dataset can help in other classification problems for evaluating causal effects. 

questions: How this paper help developing evaluation techniques for other problems apart from sentiment analysis?
limitations: They have described the limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces CEBaB, a new benchmark dataset to assess model explanations for NLP models. The key idea here is to edit short restaurant reviews with human generated counterfactual reviews in which one aspect of the review is modified. With aspect level and review level sentiment annotations on CEBaB, authors are able to approximate how a particular model might weigh some abstract concepts more than others when performing sentiment classification. The authors evaluate six (existing and proposed) explanation methods on this benchmark and find that there are significant differences between them. I like that this is a simple and intuitive idea that appears to work well in practice. However, there is a disconnect between the general claims the paper makes and the benchmark. The claims are about NLP model behavior in general but should be constrained to sentiment analysis. I have updated my sore following author response.
strengths_and_weaknesses: ### Strengths:
- This paper addresses an important problem of evaluating explainability methods. The paper is well motivated and well written.
- The conceptual framing offered by this paper as well as the resource contribution could be useful to future researchers.
- The authors have also provided an extensive experimental analysis of how well multiple model explanation methods are able to capture the true concept effect.
- I like the focus on direction of an intervention's effect on model behavior rather than the scale of effects as it simplifies the problem to knowing what causes what rather than the scale of a cause's contribution to the effect.


Weaknesses:
- This work could be much better positioned in prior work on benchmarks for explainability and counterfactual analysis for model explanations (such as the Language Interpretability Toolkit).
- The benchmark is only for sentiment analysis which limits its use while broader claims are made about NLP explainability. While that's still a good first step, it is arguably easier to "explain" a model on a sentiment analysis task versus say NLI or other more complicated NLP tasks using CEBaB, and there is no discussion offered in the paper on how this would extend to those tasks.



questions: - I don't understand what Table 4 is. The caption is very confusing so if you could share that, that'll be great.
limitations: I would like a greater discussion of how the method could be extended to other more complicated tasks. And if there are challenges associated with that, what are those?
review_title: null

Meta-review: metareview: 
The paper presents a new benchmark dataset for assessing explanation methods in NLP, on the sentiment analysis domain. The dataset is unique in that it focuses on the casual effects of modifying specific aspects, providing minimal pairs where only one of the aspects is different. After constructing the benchmark, the paper uses a causality-based metric (Section 2) to evaluate existing explanation methods. The reviewers (NcbX / dBGa) agree that the paper is well motivated and can provide useful resources for the community. The experimental results show a simple baseline they propose performs on par with existing explanation methods, which is already an interesting finding to the community. While the reviewer spotted some flaws (not providing important details, positioning of the work, weak discussion of related work, etc), most seem fixable by camera ready. I’d recommend acceptance. 



confidence: Certain
Decision: Accept