CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior
Authors: Eldar David Abraham
Karel D'Oosterlinck
Amir Feder
Yair Ori Gat
Atticus Geiger
Christopher Potts
Roi Reichart
Zhengxuan Wu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Explainability
Causality
Benchmark
Causal Explanation

Abstract: The increasing size and complexity of modern ML systems has improved their predictive capabilities but made their behavior harder to explain. Many techniques for model explanation have been developed in response, but we lack clear criteria for assessing these techniques. In this paper, we cast model explanation as the causal inference problem of estimating causal effects of real-world concepts on the output behavior of ML models given actual input data. We introduce CEBaB, a new benchmark dataset for assessing concept-based explanation methods in Natural Language Processing (NLP). CEBaB consists of short restaurant reviews with human-generated counterfactual reviews in which an aspect (food, noise, ambiance, service) of the dining experience was modified. Original and counterfactual reviews are annotated with multiply-validated sentiment ratings at the aspect-level and review-level. The rich structure of CEBaB allows us to go beyond input features to study the effects of abstract, real-world concepts on model behavior. We use CEBaB to compare the quality of a range of concept-based explanation methods covering different assumptions and conceptions of the problem, and we seek to establish natural metrics for comparative assessments of these methods.

CЕBaB: Estimating the Causal Effects of Real- World Concepts on NLP Model Behavior
Eldar David Abraham":!
Karel D'Oosterlink Barel.DDottrLLinckklGen...
eldar.a@camee a@campus, Deccampus.techiion.ce technion.ac.1. il
Amir Feder
Yair Gat
feder@campus. technion.ac. ac. i1
yairgat@campus. technion.cc. ac. il
Atticus Geiger* at atticus@@stanford..er
Christopher Potts* cgpotts@stanford. edu
Roi Reichart". roiri@technion.cm i1
Zhengxuan Wu*.2 wuzhengx@stanford.c edu
Technion Israel Institute of Technology
Stanford University
Ghent University
Abstract
The increasing size and complexity of modern ML systems has improved their predictive capabilities but made their behavior harder to explain. Many techniques for model explanation have been developed in response, but we lack clear criteria this paper, we cast model explanation as the
for assessing these techniques. causal inference problem of estimating causal effects of real-world concepts the output behavior of ML models given actual input data. We introduce CEBaB, a new benchmark dataset for assessing concept-basse maasett-baeed explanation methods in Natural Language Processing (NLP). CEBaB consists of short restaurant reviews with human-generated counterfactual reviews in which aspect (food, noise, ambiance, service) of the dining experience was modified. Original and counterfactual reviews annotated with multiply-validated sentiment ratings at the aspect-level and review-level. The rich structure of CEBaB allows us to go beyond input features study the effects of abstract, real-world concepts on model behavior. We CEBaB to compare the quality of a range of concept-bar based explanation methods covering different assumptions and conceptions of the problem, and we seek establish natural metrics for comparative assessments of these methods.
Introduction
Explaining model behavior has emerged as a central goal within ML. In NLP, models have grown in size and complexity, and while they have become increasingly successful, they have also become more opaque [28,36], raising concerns about trust [18,23], safety [1.34], and fairness [16,19]. These concerns will persist if these models remain "black-boxes".
Seeking open the black-box researchers have developed methods that try to explain model behavior 12, However, there is no consensus about how to evaluate such methods to allow robust
"Equal contribution. Author names alphabetical.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Table 1: Toy examples illustrating the structure of CEBaB (actual corpus examples are longer and more complex; a sample is given in Appendix B). Beginning from an OpenTable review, we give crowdworkers an actual restaurant review and they generate counterfactual restaurant reviews that would have been written if some aspect of the dining experience were changed and all else were held constant. Five different crowdworkers labeled each of the actual and counterfactual texts according to their aspect-level sentiment and overall sentiment. Aspect level sentiment labels are three way: '+' (positive sentiment), '_' (negative), and 'unk' (the aspect's s value is not expressed in the text). Overall sentiment labels are (worst) to 5 (best). Edited aspect labels are shown in blue.
food ambiance service noise overall
Original text: Excellent lobster and decor, but rude waiter.
unk 4
Edit Goal
food:
Terrible lobster, excellent decor, but rude waiter. Excellent decor, but rude waiter.
unk 2
food: unk
unk
unk
3
Excellent lobster, but lousy decor and rude waiter. ambiance: unk Excellent lobster, but rude waiter.
ambiance:
unk
3
unk
unk
3
service: + Excellent lobster and decor, and friendly waiter.
unk
5
service: unk Excellent lobster and decor.
unk
unk
5
noise: + Excellent lobster, decor, and music, but rude waiter.
noise:
Excellent lobster and decor, but rude waiter, and noisy.
comparisons. This not surprising, since such evaluations require very rich empirical data. Intuitively, we would like to (I) intervene on model inputs, to modify specific concepts without changing other correlated information, (2) observe the effects this has on model predictions, and, finally, (3) assess explanation methods for their ability to accurately predict these effects.
The absence of interventional data, or even an agreed-upon non-interventional benchmark, has created an environment in which explanation methods are often evaluated individually, and without comparison alternatives. Attempts have been made conduct comparative evaluations [11,17,77,, but only with synthetic, simplified datasets. Furthermore, these attempts do not define a unified evaluation approach, nor do they seek to contribute benchmark datasets that support such evaluations.
In this paper, we seek to overcome this obstacle by introducing CEBaB (Causal Estimation-Ban Benchmark). Table 1 summarizes the structure of CEBaB with a toy example: beginning with review text from the OpenTable website, we crowdsourced edits of the original text that are designed to meet specific goal, such as changing the food rating in the original text to negative unknown. All of the resulting edits were validated by five crowdworkers and each full text was evaluated by five crowdworkers for its overall sentiment. CEBaB is grounded in 2,299 original reviews, which were expanded via this editing procedure to a total of 15,089 texts, targeting four different aspect-level concepts (food, service, ambiance, noise) with three potential labels (positive, negative, and unknown, i.c., not expressed in the review), and each full text was labeled on five-star scale.
We focus on using CEBaB to compare concept- based explanation methods. This allows to go beyond the effect of individual tokens to study how more abstract concepts (in our case, aspect-level sentiment) contribute model predictions (about the overall sentiment of the text). Our proposed metrics center around assessing concept-basse explanation methods for their ability to accurately estimate causal concept effects [17], allowing to isolate the effect of individual concepts.
More specifically, we use CEBaB to measure the causal effects of particular variables in a causal graph, and we cast each explanation method as a causal estimator of these measurements. For example, suppose our causal graph of the data says that all four of our aspect-level categories will affect reviewer's overall rating. To estimate the effect of positive food quality the predicted overall rating from a classifier, we need to compare examples with high food quality to those with low quality, holding all other aspects constant. Such pairs of examples are normally not observed, but this is precisely what CEBaB provides. With CEBaB, we can directly compare the actual change in model predictions with the change that a concept- explanation method predicts.
In our experiments, we evaluate five leading concept- explanation methods: CONEXP [17], TCAV [26]. ConceptSHAP [57], INLP [40], M [11], and S-Learner [27]. These methods make a wide range of different assumptions about how much access we have to the model's 's internal structure, and they also diverge in the degree to which they account for the causal nature of the
2

---

concept effect estimation problem. Remarkably, CEBaB reveals that most methods cannot beat simple baseline. Indeed, this negative result emphasizes the value in our primary contribution of providing the data and metrics that enables a direct comparison of explanation methods.
2 Previous Work
Benchmarks for Explanation Methods Benchmark datasets have propelled ML forward by creating shared metrics that predictive models can be evaluated on (22,25,54.55], Unfortunately, benchmarks that are suitable for assessing the quality of model explanations still uncommon [10, 21J. Previous work on comparing explanation methods has generally only correlated the performance of given explainability method with others, without ground-truth comparisons [8,20,21,43].
Other works that do compare to some ground-truth either employ a non-causal evaluation scheme [26]. use causal evaluation metrics which do not capture performance on individual examples [52], evaluate on synthetic counterfactuals and rule-based augmentations [1 [1],, or are tailored for specific explanation method and hard to generalize [57]. To the best of our knowledge, CEBaB is the first sarge-scale naturalistic causal benchmark with interventional data for NLP.
Explanation Methods and Causality Probing is a relatively new technique for understanding what model internal representations encode. In probing, a small supervised [6,51] or unsupervised [5,32,44] model is used to estimate whether specific concepts are encoded at specific places in network. While probes have helped illuminate what models (especially pretrained ones) have learned from data, Geiger et al. [15] show with simple analytic examples that probes cannot reliably provide causal explanations for model behavior.
Feature importance methods can also be seen as explanation methods [33]. Many methods in this space are restricted input features, but gradient-based methods can often quantify the relative importance of hidden states as well [3,46,48.58]. The Integrated Gradients method of Sundararajan al. [50] has a natural causal interpretation stemming from its exploration of baseline (counterfactual) inputs [15]. However, even where these methods can focus on internal states, it remains difficult to connect their analyses with real-world concepts that do not reduce simple properties of inputs.
intervention-based methods involve modifying inputs or internal representations and studying the effects that this has model behavior [30,41] 41]. Recent methods perturb input or hidden representations to create counterfactual states that can then be used to estimate causal effects [9.12,47,53,15]] However, these methods are prone to generating implausible inputs or network states unless the interventions are carefully controlled [14].
Generating counterfactual texts automatically remains challenging and is still work-in-progress [4]. To overcome this problem, another class of approaches proposes to manipulate the representation of the text with respect to some concept, rather than the text itself [9, 19,11,40 1,40J. These methods fall into the category of concept-based explanations and we discuss two of them extensively in $3.
3 Estimating Concept Effects with CEBaB
We now define the core metrics that we use to evaluate different explanation methods. Figure 1 provides high-level view of the causal process we are envisioning. The process begins with an exogenous variable U representing a state of the world. For CEBaB, we can imagine that the value of U is state of affairs of person evaluating a restaurant a review variable X, with the value r of X mediated by and by mediating concepts C1.... C.....Cs Ck. which correspond to the four aspect- categories in CEBaB (food, service, ambiance, and noise), each of which can have values c E {positive, negative, unknown]. The review x is processed by model that outputs a vector of scores over classes (sentiment labels in CEBaB).
particular way, contributes to
Core Metric Our central goal is to use CEBaB to evaluate explanation methods themselves. CEBaB supports many approaches such evaluation. In this paper, adopt an approach based on individual- level rather than average effects. This makes very rich use of the counterfactual text and associated labels provided by CEBaB. The starting point for this metric is the Individual Causal Concept Effect: Definition (Individual Causal Concept Effect; ICaCE), For neural network N and feature function ó, the individual causal concept effect of changing the value of concept C. from to é' for
3

---

C1
C2
N(o(X))
C
Figure 1: A causal graph describing data generating process with exogenous variables U and V representing the state of the world, mediating concepts C1, C2.C.... Ck. and data X that is featurized with c. o(X) is input to classifier N, which outputs vector of scores output classes.
state of affairs u an underlying data generation process G is
C=c cd = Bung [(((x)) do (C (o(x)=)) ICaCEN, (9, 1
U =d=
(1)
ICaCE is theoretical quantity, In practice, we use the Empirical Individual Causal Concept Effect.
Definition 2 (Empirical Individual Causal Concept Effect; ICaCE). For a neural network N and feature function & the empirical individual causal concept effect of changing the value of concept C from e. to e' for state of affairs is
ICaCENs (0 "0==,,=_-d) - - -M(((====))
(2)
where (x _.-e) is a tuple of inputs originating from with the concept C set to the values e and d', respectively.
The ICaCEN, for pair of examples (I " vectors for the two cases. With CEBaB, we can easily calculate these values because we have clusters of examples that are tied to the same reviewing situation u and express different concept values.
C=c=)) is simply the difference between the output score
For assessing explanation method ε, we compare ICaCE values with those returned by E. Our core metric is the ICaCE-Error:
Definition 3 (ICaCE-Ervor). For a neural network N, feature function & and distance metric Dist, the ICaCF-Error of an explanation method & for changing the value of concept C. from co d' is:
D. (E) ICaCE-EEroo
1
Σ
Dist (ICaCTEN, ( ). ,6))
(3)
D
(It DED
We present results for three choices of Dist which vary in their ability to model the direction and magnitude of effects. These choices give subtly different but largely converging results, as detailed in Section and reported more fully in Appendix D.
Aggregating Individual Causal Concept Effect is often useful to also have a direct estimate of model's 's ability to capture concept-level causal effects. For this, we employ aggregating version of ICaCE, the Empirical Causal Concept Effect:
Definition (Empirical Causal Concept Effect; CaCE), For a neural network N and feature function 6, the empirical causal concept effect of changing the value of concept C' from e to c' in dataset 'D is
Σ
iCaCEx, ( 2
C=c ==
(4)
|Dg+e'
(=C= SGt=e,aCeee DED
This is an empirical estimator the Causal Concept Effect (CaCE) Goyal al. [17]. estimates, in general, how the classifier predictions change for given concept and intervention direction.

---

Table 2: The evaluated explanation methods and their attributes. Explainer Method denotes the complexity of the models used by each explanation method. Access to Explained Model denotes the degree of access an explainer method needs to the explained model. Concept Labels Needed indicates whether a method estimating the effect for an input -- needs the actual input label c and/or the intervened value e' at test time. Models with a Counterfactual Representation approximate
(( to estimate the effect. Finally, only M and S-Learner have Confounder Control to minimize the impact of confounding concepts. We predict these labels with classifier.
Explainer Method
Access
Concept Labels
Counterfactual Representation
Confounder
Explanation method
Explained Model
Needed (test time)
Control
Approx
None
None
All concepts and their labels"
X
CONEXP [17]
None
None
X
S-Learner (27)
Linear
Nome
All concepts and their labels"
X
TCAV [26]
Linear
Weights
None
X
ConcepISHAP (57)
Linear
Weights
None
INLP (40)
Linear
Weights
None
MausaLM [11)
Explained Model
Training Rogime
None
Estimating Real-World Causal Effect of Aspect Sentiment on Overall Sentiment We can also estimate ground truth causal effects in CEBaB by simply using its labels directly. There are again variety of ways that this could be done. We opt for the one that makes the richest use the structures afforded by CEBaB. For perspicuity, in parallel to the neural network-based ICaCE (Definition 2), we define the Empirical Individual Treatment Effect for our dataset:
Definition 5 (Empirical Individual Treatment Effects in CEBaB; ITE). The empirical individual treatment effect of changing the value of concept C from e' in CEBaB
CEBoB (I
(G==xx_- = ff= - f(x
ITE
(5)
where f is a simple look-up procedure that retrieves the overall sentiment labels for CEBaB examples.
We aggregate over these values by taking their average, in parallel to what we do for network predictions (Definition 4). This yields the Empirical Average Treatment Effect (ATE) for CEBaB.
Alternative Metrics In Appendix A in our supplementary materials, consider alternative formulations of the core metrics with causal concept effects and absolute causal concept effects, relating them to the different questions they engage with. We opt for the individual causal concept effect in our central metric (Definition 3), taking the central question to be what caused an ML model to produce an output for an actual input created from real-world process.
4 Evaluated Explanation Methods
We compare several model explanation methods that share three main characteristics. First, they are all suitable for NLP models and have been used the literature for generating model explanations the form of estimated effects on model predictions. Second, they all provide concept-level explanations, for a pre-defined list of intarniiterpreeabll concepts (e-g., how sensitive restaurant review rating classifier is to language related to food quality). This approach is also forward-looking allowing more researchers to construct new hypotheses (i.c., concepts we have not collected labels for) and estimate their effect the predictor. Third, all of the tested methods are model-agnostic, meaning that they separate the explanation from the model. At the same time, these methods differ in five important ways, summarized Table 2.
We now turn to reviewing the explanation methods that we later compare on CEBaB ($6). In our mathematical formulas, we employ unified notation for all methods, to make the definitions more accessible and easier to integrate into our experimental set-up. Assume we have a classifier N (which outputs probability vector) and feature function Φ, and we want to compute the effect on (x of changing the value of concept from to c' using an unseen test set (D,
Approximate Counterfactuals The gold labels of CEBaB are the difference between the logits for some original review C=c and ground ttuth counterfactual C=c' As a bascline, we sample
5

---

an original review zC-e with the same aspect-labels as the x; and use it as an approximate counterfactual:
Approx c, : = (( - (()()==))
(6)
We do this sampling using predicted aspect labels from the aspect-level sentiment analysis models described in Appendix C.
Conditional Expectation (CONEXP) Goyal al. [17] propose baseline where the effect of concept C' is the average difference in predictions examples with different values of C.
CONEXP N,, (C,,,,)) = IDC=c
Σ N()(())
Σ N(((z))
(7)
DC=
DEDC=d
REDCE
where DC=c and DD=c'' are subsets of DD where C takes values and é', respectively. To predict an effect, this method only relies on C, e, and d', resulting an estimate that does not depend on the specific input text itself.
Conditional Expectation Learner (S-Learner) We adapt S-Learner, a popular method for estimating the Conditional Average Treatment Effect (CATE) [27]. To estimate causal concept effects, our S-Leamnr trains a logistic regression model & to predict (z((z)) using the values of all the labeled concepts of example x, denoted by x!? Then, during inference, we compute an individual effect for example pair - by comparing the output of the model & on this pair:
S-Learner(C, c(;) E11'0 E(x
(8)
At inference time, S-Learner assumes access to all aspect-level labels x', which might not always be available. To alleviate this issue, instead predict the aspect-level labels x' from the original text using models described in Appendix C.
Kim et al. [26] use Concept Activation Vectors (CAVs), which are semantically meaningful directions in the embedding space of Φ. Our adapted version of Testing with CAVs (TCAV) outputs vector measuring the sensitivity of each output class k to changes towards the direction of a concept vg at the point of the embedded input. It is computed as:
TCAV
TCAV No (C; a)) (VN((o(x)) "c) k=1
where K is the number of classes and vo is linear separator learned to separate concept C' in the embedding space of ø.
ConCeptSHAP Yeh et al. [57] propose this expansion to SHAP [30], to generate concept- based explanation based on Shapley values [45]. Given a complete (i.e., such that the accuracy it achieves on a test set is higher than some threshold (3) set of m concepts {CI,.... Cm ConceptSHAP calculates the contribution of each concept to the final prediction. Our adapted version outputs a vector for each C E (C1,..., } and x. We justify this modification and provide implementation details in Appendix
CausaLM Feder et al. [11] estimate the causal effect of binary concept C' on the model's predictions by adding auxiliary adversarial tasks the language representation model in order to learn counterfactual representation cCP (z), while keeping essential information about potential confounders (control concepts). Their method outputs the text representation based individual treatment effect (TReITE), which is computed as:
TReITEN, (C; (0g -N((<x)) ((((c))))
(10)
where & denotes the learned counterfactual representation, where the information about concept C is not present, and A is classifier trained on this counterfactual representation. A key feature of CausaLM its ability to control for confounding concepts (if modeled), An inherent drawback of this technique is that it can only estimate interventions well for c' Unknown, since the counterfactual representation only trained to remove a concept C.
2This training approach, where an explainer model is fit to predict output the original model, shares the intuition LIME, the widely used explanation method [41], but for concept-level effects. As in Feder al. [11], control for the most correlated potential confounder.

---

Table 3: Dataset statistics of CEBaB combining train/dev/test splits.
1 star
1870 (12%)
Positive
Negative
Unknown
no maj.
Total
star
3056 (20%)
food
5726 (41%) 5526 (38%) 2605 (15%) 208 (31%)
14065
3 star
3517 (23%)
service
4045 (29%) 4098 (28%) 3877 (22%)
178 (27%)
12198
star
2035 (13%)
ambiance
2928 (21%) 2597 (18%) 5121 (29%) 203 (30%) 10849
5 star
2732 (18%)
noise
1365 (10%) 2215 (15%) 5883 (34%) 78 (12%) 9541
no maj.
1879 (12%)
(a) Aspect-level labels.
(b) Review-level ratings.
[Neg, Pos]
(Neg, Unk}
(Pos, Unk]
Neg to Pos Neg to Unk Pos to Unk
food
898
1316
1291
food
1.84
1.37
-1.02
service
851
857
938
service
0.98
0.91
-0.53
ambiance
947
585
472
ambiance
0.93
0.91
-0.50
noise
1 1145
208
260
noise
0.72
0.48
-0.47
(c) Edit pair distribution. Edit pairs are examples that come from the same original sourec text and differ only their rating for particular aspect.
(d) Empirical ATE for the five-way sentiment labels in CEBaB. The reverse of a given concept change is the negative of the value given - e.g., the ATÉ 'Pos to Neg' for food is -1.4. See Appendix B for the corresponding values for binary sentiment.
Iterative Nullspace Projection (LNLP) Ravfogel et al. [40] remove concept from representation vector by repeatedly training linear classifiers that aim to predict that attribute from the representations and projecting the learned representations their null-space. Similar CausaLM, INLP also estimates the TReATE (Equation 10) and can only estimate interventions for c' Unknown.
5
The CEBaB Dataset
Table provides an intuitive overview of the structure CEBaB. In the editing phase of dataset creation, crowdworkers modified an existing OpenTable review in an effort to achieve specific aspect-level goal while holding all other properties of the original text constant. Our aspect-level categories are food, ambiance, service, and noise. In the validation phrase, crowdworkers labeled each example relative to each aspect as 'Positive', 'Negative', or 'Can't tell' (Unknown). Having five labels per example allows us infer majority label or reason in terms of the full label distributions. In the rating phase, each full text was labeled using a common five-star scale, again by five crowdworkers.
We began with 2,299 original reviews from OpenTable (related to 1,084 restaurants) and expanded them, via the above editing procedure, into total of 15,089 texts. The distribution of normalized edit distances has peaks around 0.28 and 0.77, showing that workers made non-trivial changes to the originals, and even often had to make substantial changes achieve the editing goal. (See Appendix B for the full distribution.)
Table 3 summarizes the resulting label distributions, where an example has label y if at least 3 of the 5 labelers chose otherwise it is in the 'no majority' category. 99% of aspect-level edits have majority label that corresponds to the editing goal, and 88% of the texts have a review-level majority label on the five-star scale. Overall, these percentages show that workers were extremely successful in achieving their editing goals and that edits have systematic effects on overall sentiment.
The central goal of CEBaB to create edit pairs: pairs of examples that come from the same original text and differ only in their labels for particular aspect. For example, in Table the first two 'food edit' cases form an edit pair, since they come from the same original text and differ only in their food label. Original texts can also contribute to edit pairs; the original text in Table forms an edit pair with each of the texts it is related to by edits. Table 3c summarizes the distribution of edit pairs, and Table 3d reports the ground-trun truth ATE values ($3).
7

---

Table 4: CaCE (Definition 4) for bert-base-uncased fine-tuned as a 5-way sentiment classifier. Rows are concepts, columns are real-world concept interventions, and each entry indicates the average change in classifier output when the concept is intervened on with the given direction." 4 Results are averaged over distinct seeds with standard deviations. The CaCE value of changing concept C' from to c' is the negative CaCE value of changing concept C' from e' to c.
Negative to Positive
Negative to unknown
Positive to unknown
food service
1.90 (# 0.03)
1.00 (# 0.02)
-..2 (# 0.01)
1.42 (# 0.04)
0.89 (# 0.04)
-0.45 (# 0.01)
ambiance
1.27 (# 0.01)
0.79 (# 0.01)
-0.50 (# 0.03)
noise
0.75 (# 0.02)
0.44 (# 0.00)
-0.23 (# 0.02)
We release the dataset with fixed train/dew/test splits. In creating these splits, we enforce two high- level constraints. The first is our 'grouped' requirement: for each original review t, all texts that are related via editing occur in the same split as t. This ensures that models are not evaluated on examples that are related by editing to those they have seen in training. Second, if any text t in a group received 'no majority' label, then the entire group containing is put in the train set. This ensures that there is no ambiguity about how to evaluate models on dev and test examples.
Once these high-level conditions were imposed, the examples were sampled randomly to create the splits. This allows that individual workers can contribute edited texts across splits. This minor compromise necessary ensure that we could have large dev and test splits. Appendix C in our supplementary materials shows that worker identity has negligible predictive power.
There are two versions of the train set: inclusive and exclusive. The inclusive train set contains all original and edited non-dev/test texts (11,728 texts). The exclusive version samples exactly one train text from each set of texts that are related by editing (1,755 examples). The rationale is that models trained with an original review as well as its edited counterparts may explicitly learn causal effects trivially by aggregating learning signals across inputs. Our exclusive train split prevents this, which helps facilitate fair comparisons between explanation methods and better resembles a real-world setting.
Our dataset released publicly in JSON format and is available in the Hugging Face datasets library. It includes restaurant metadata, full rating distributions, and anonymized worker ids. Appendix B in our supplementary materials provides additional details on the dataset construction, including the prompts used by the crowdworkers, the number of workers per task, worker compensation, and sample examples with ratings to help convey the nature of workers' edits and the overall quality of the resulting texts and labels. In addition, Appendix C reports at the aspect-level and text-level that show that models perform well on CEBaB classification tasks, which bolsters the claim that CEBaB is reliable tool for assessing explanation methods.
wide range of classifier experiments
Experiments and Results
For each experiment, we fine-tune a pretrained language model to predict the overall sentiment of all restaurant reviews from our exclusive OpenTabl Table train set. Since the goal of our work is not to achieve state-of-the-ar performance, but rather compare explanation methods and demonstrate the usage of CEBaB, we test the ability of methods to explain commonly used models, trained with standard experimental configurations.
In the main text, we report results for bert-base-uncased fine-tuned as a five-way classifier. Appendix includes results for GPT-2, RoBERTa. and an LSTM, fine-tuned on binary. 3-way and 5 5-way versions of the sentiment task. All results, including the ground- effect that depends on the specific instance of model, are averaged across 5 seeds.
To evaluate the intrinsic capacity of a model to capture causal effects, we report the CaCE values, as in Definition& 4. The results for bert-base-uncal are given in Table 4. They are intuitive and
8

---

1.2
ICaCE-cosine
CcaC--nrmdiif
ICaCE-L2
0.8
0.6
0.0
Random
Approx
CONCAR
S-Leamer
TCAY
ConcoRiSAA
Causald
DNLP
Figure 2: ICaCE-Error (Definition 3) for bert-base-uncasd fine-tuned for five-way sentiment, averaged per aspect. We report values for cosine, L2, and normdiff. Lower is better. Stars mark the best result(s) per metric. Results averaged over 5 distinct seeds. "RandomExplainer takes the difference between two random probability vectors the predicted effect.
well-aligned with the ATE estimates in Table 3d, indicating that the model has captured the real-world effects.
Our primary assessment of the evaluation methods is given in Figure 2, again focusing five-way pert-base-uncased model as representative of our results. We provide values based on cosine, L2, and normdiff the value of Dist in Definition 3. The cosine-distance metric measures if the estimated and observed effect have the same direction but does not take the magnitudes of the effects into account. The L2-distance measures the Euclidian norm of the difference of the observed and estimated effect. Both the direction and magnitude of the effects influence this metric. To only compare the magnitudes, between the Euclidean norms of the observed and estimated effects, thus completely ignoring the directions of both effects.
use the formdiff-distanee which computes the absolute difference
Remarkably, our approximate counterfactual baseline proves to the best method at capturing both the direction and magnitude of the effects. The fact that simple baseline method beats almost all other methods indicates that we need better explanation methods we are going to capture even relatively simple causal effects like those given by CEBaB.
Recall from Table 2 that the compared methods require different levels of access to concept labels at inference time. Approximate counterfactuals and S-L.earner have access to both the direction of the intervention and the predicted test-time aspect labels, enabling them to outperform CONEXP, which has access to only the direction of the intervention, and TCAV, ConceptSHAP, and CausaLM, which have access to neither the intervention direction nor test-time aspect labels.
The INLP method ties with the best method for the cosine metric, despite having access to neither intervention directions nor test-time aspect labels. Perhaps this method could be extended make use of this additional information and decisively improve upon our approximate counterfactual baseline.
While CausaLM and INI.P both estimate the effect of removing concept from input, INLP uses linear probes to guide interventions on the original model, while Causal . trains an entirely new model with auxiliary adversarial objective. The direct of the original model is something INLP shares with the approximate counterfactual baseline; it seems that a tight connection to the original model may underlie success on CEBaB.
7 Conclusion
Our main contributions in this paper are twofold. First, we introduced CEBaB, the first benchmark dataset to support comparing different explanation methods against single ground-truth with concept labels for aspect-level and overall
human-created counterfactual texts and multiply
'Deiiniton 4 defines the CaCE values as vectors. In this table, we collapse the CaCE values to scalars by having '' output the most probable predicted class, instead of the class distribution.

---

sentiment, Using this resource, one can isolate the true causal concept effect of aspect-level sentiment on any trained overall sentiment classifier. CEBaB provides level playing field on which we can compare variety of explanation methods that differ in their assumptions about their access to the model, their computational demands, their access to ground-truth concept labels at inference time, and their overall conception of the explanation problem. Furthermore, the evaluated methods make absolutely no use CEBaB's counterfactual train set. In turn, we hope that CEBaB will facilitate the development of explanation methods that can take advantage of the very rich counterfactual structure CEBaB provides across all its splits.
Second, we have provided an in-depth experimental amalysis of how well multiple model explanation methods are able capture the true concept effect. A naive baseline that approximates counterfactuals through sampling achieves the best performance, with INLP and S-Learner being the only other methods that achieves state-of-the art on any metric. While CEBaB only grounded in one task, sentiment analysis alone is enough to produce starkly negative results that should serve as call to action for NLP researchers aiming to explain their models.
Acknowledgments and Disclosure of Funding
This research is supported in part by a grant from Meta AL. Karel D'Oosterlinck was supported through a doctoral fellowship from the Special Research Fund (BOF) of Ghent University. We thank our crowdworkers for their invaluable contributions to CEBaB.
References
[1] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in Al Safety. 1rXiv:/606.06560 [cs], 2016. URL httpp//arxiv org/abs/ 1606. 06565.
[2] Jasmijn Bastings, Yonatan Belinkov, Emmanuel Dupoux, Mario Giulianelli, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad. Proceedings of the fourth BlackboxNLP workshop on analyzing and interpreting neural networks for NLP. Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2021.
[3] Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, pages 63-71. Springer, 2016.
[4] Nitay Calderon, Eyal Ben-David, Amir Feder, and Roi Reichart. Docogen: Domain counterfactual generation for low resource domain adaptation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7727-7746, 2022.
[5] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? An analysis of BERT's attention. In Proceedings the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy, August 2019. Association for Computational Linguistics. URL https: aclweb. org/ urg/anthologgy/W 9--1-8...
[6] Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into single $& vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.1186//11/11---1 1198. URL https: /www. aclweb. aclweb.org/anthee 1ttp./////nnooooo//11---11111
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational
10

---

Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171- 4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1/111914222
https: /wwww//wwwww/////// www.wwwwwwb.bb..... htt.://ooo/aanooooo/////1 9-
[8] Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443-4458, Online, July 2020. Association for Computational Linguistics. doi: 18.53v3//v/ 2020.acl-main URL https://aclanthinn
. org wwwgg/222...mm acl mocl-main.40 408..
[9] Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 2021.
[10] Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood- Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. arXiv preprini arXiv: 109.00025 2021.
[11] Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. CausaLM: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):3333386, June 2021. URL https://aclantholloy) org.org////o.. 2021 cl-2.13 13.
doi: 10. 10.116//coli_aa000
[12] Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 1 11th International Joint Conference on Natural Language Processing \Volume| 1: Long Papers), pages 1828-1844 1843, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021..ee aoll.a 144. URL https:// aclanthology.org// org/ acl- 144.
[13] Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang. Yaron Singer, and Stuart Shieber. Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias. 20Xiv:2004.12266 [cs), November 2020. URL http://arxiv. arXiv: 2004. 12265.
org/abs/2004 12265.
[14] Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 163-173. Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ v 11/2020.blackboxnn 1.16. URL htttp:://aalantholooy org/ 2020. blackboxnlp-1.1 16.
[15] Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34, 2021.
[16] Bryce Goodman and Seth Flaxman. European Union regulations on algorithmic decision-making and "right to explanation". A/ Magazine, 38(3):50-57, October 2017. ISSN 2371 0738- 4602. doi: 10. 1009/amaam. .38.3.2741. URL http:://at//ee wwwpxiiiiorrreeeoo www.orggg/bb///ee/ee//e///// 08813. arXiv: 1606.08813.
[17] Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining Classifiers with Causal Concept Effect (CaCE), arXiv:1907.0m 000.007655... [cs, star], February 2020. URL http://arxivv org/ abs/ 1907. 67.65. arXiv: 1907.0716.
[18] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5 2018.
[19] Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of Opportunity in Supervised Learning. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URI http::///////oriiing../.. wew.iss.euips....o wwtpsssesrpspcpppprrrreeee.e.e wtt.ccpappp/////ii.... 99225228

---

[20] Peter Hase and Mohit Bansal. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior? arXiv preprint AXXi:20005.0181,, 2020.
[21] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019.
[22] Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411-4421. PMLR, 2020.
[23] Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computationnl Linguistics, pages 4198 4205, Online, July 2020. Association for Computational Linguistics. doi: 10. (0.18653/11/2020..0 mai..ma....8..... URL https://aclanthollgg. orgg//20.0.. acl-main. 386.
[24] Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv: XX1::6/2..0655/ 2016.
(25) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma. Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jía, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4110-4124 Online, June 2021. Association for Computational Linguistics. URL https: / /wwww aclweb. orgo hataannnhhloyyy///a 22alooyy/2222 2221 naacl-maiin 324.
[26] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres, Interpretability Beyond Feature Autribution: Quantitative Testing with Concept Activation Vectors (TCAV). In International Conference on Machine Leurning, pages 2668- 2677. PMLR, July 2018. URL http: /tttoe//pprceednggs. mlr. wwt.pprres//kkinne 2640-3498.
html. ISSN:
[27] Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10):4166414) 2019.
(28) Zachary C. Lipton. The mythos of model interpretability. Communications of the ACM, 61(10):36-43 43, September 2018. ISSN 0001-0782. doi: 10.1145/3233231. URL https: //doi.org/// 1145/3233231.
[29] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: Robustly Optimized BERT Pretraining Approach. arXiv:190..11922 [cs], July 2019. URL http://arxxvv.org/aheee 11692. arXiv: 1907. 1692.
[30] Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pages 4768-4777, Red Hook, NY, USA, December 2017. Curran Associates Inc ISBN 1188--5118-6000
[31] Thang Luong. Hieu Pham, and Christopher D Manning. Effective approaches to ditention-basee neural machine translation. In EMNLP, 2015.
[32] Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 1 117744))30046-3005 2020. ISSN 0027- 8424. doi: (0.1177//paas.1973671111 1907367117.
URL https: //wpp::///wwww. ppws///wwwwpppw.ppppa.aaaasa..ap org/ wtwp:/pa.....oooon////////e content/11//////
30046.
12

---

[33] Christoph Molnar. Interpretable machine learning. Lulu. com,, 2020.
[34] Clemens Otte. Safe and Interpretable Machine Learning: A Methodological Review. In
Christian Moewes and Andreas Nürnberger, editors, Computational Intelligence in Intelligent Data Analysis, Studies in Computational Intelligence, pages 111-122, Berlin, Heidelberg, 2013. Springer. ISBN 978-3-642-32378
doi: 1-10077978-3-6422-222
[35] Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4))669-688 1995.
[36] Judea Pearl. The limitations of opaque learning machines. Possible minds: twenty-five ways of looking at AI, pages 13-19, 2019.
[37] Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C Lipton, Graham Neubig, and William W Cohen. Evaluating explanations: How much do explanations from the teacher aid students? Transactions of the Association for Computational Linguistics, 10:359 3759933750040 2022.
[38] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 18))9, 2019.
[39] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv: 1910. 10683, 2019.
[40] Shauli Ravfogel, Yanai Flazar, Hila Gonen, Michael Twiton, and Yoav Goldberg, Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection, arXiv:2004.07667 [cs], April arXiv: 2004.07667.
2020. URL http: //axiv. http://rxv...orgg/b/a// 2000000047......
[41] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144, San Francisco California USA, August 2016. ACM. ISBN 978- 1781--4503-4232-2. doi: 10.1 10.1145/233972..299999777 URL httph::////o acm. org//oi/10.
29397778
[42] D. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66:688-701, 1974.
[43] Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J Anders, and Klaus- Robert Müller. Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, 109(3):477-278, 2021. 2021.
[44] Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3257-3267, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. N...8855/111/119113 URL https: http::://wwwwwwlll// aclweb. org/ anthology/N19 9- 1329.
[45] Lloyd Shapley. A value for n-person games. In H. W. Kuhn and A. W. Tucker, editors, Contributions to the Theory of Games I1, pages 307-317. Princeton University Press, 1953.
[46] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pages PMLR, 2017.
3
[47] Paul Soulos, R. Thomas McCoy, Tal Linzen, and Paul Smolensky. Discovering the compositional structure of vector representations with role learning networks. In Proceedings of the Third BlackboxNI_P Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 238 254, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.blackba wat.0oalacbbxolp--.... URL https::///// htlp//aaaaaaaooooooooooo org/2020. blackboxnlp-1.0k
[48] Jost Springenherg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. CoRR, 12 2014.
13

---

[49] Chi Sun, Luyao Huang, and Xipeng Qiu. Utilizing BERT for -spect-based sentiment analysis via constructing auxiliary sentence. arXiv preprint arXiv:1903.09588 2019.
[50] Mukund Sundararajan, Ankur Taly, and Qigi Yan. Axiomatic attribution for deep networks, In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' page 3319-3328. JMgR.org, 2017.
[51] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association Computational Linguistics, pages 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics. URL https://wwwt http://wwwwwwwwwbbbb.....m aclweb. wtg/anthology/P19-1455
[52] Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 107-118, Online, October 2020, Association for Computational Linguistics. doi: 10. 10.18653/v1/2000 http:///00020.mm/eemmemes. wwwg//20////2 emnlp-demos. 15.
URL
https://aclanthollee
[53] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. Investigating gender bias in language models using causal mediation analysis. In Hugo Larochelle, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12. 2020, virtual, 2020. URL https: / httpp::/proceee http///poceenn....e/////o cew.ips.ccppppprrrrrrr /tt////220////// h 11156022992177155fe122e6ffff wtw.eefaaabbodeee-bsrtttttteeeee html.
[54] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v 1/W1///W1/1//
URL http: / //wc://b....b.cc... org/ o 5444.
[55] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
[56] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, et al. Huggingface's transformers: State-of-the-an natural language processing. arXiv preprint arXiv:1910.0317 2019.
[57] Chih- Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-awe aware concept-based based explanations in deep neural networks. Advances in Neural Information Processing Systems,
2020.
[58] Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014, pages 818-833, Cham, 2014. Springer International Publishing.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Limitations that we found most significant are discussed in the supplementary material.
14

---

(c) Did you discuss any potential negative societal impacts of your work? [No] The work does not treat specific task where we see an immediate risk for negative societal impact. Though the methods described in this paper might be used by practitioners, and we suggest doing so with care since the suggested methods need be studied further in controlled settings before being used in such domains.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] The paper conforms with the provided guidelines.
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Code and Data for reproducing all of our results are submitted part of the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5 and supplementary material.
(c) Did you report error bars (c.g., with respect to the random seed after running experiments multiple times)? [Yes] All results are averaged across runs
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/rellasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [Yes] See supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're rr using/curating? [No] All assets we use are allowed for public use for the purposes of scientific research.
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] The data we use is taken from publicly available datasets, where all personally identifiable information have been removed.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes]] See Section for high-level description of the crowdsourced data and the supplementary material for the exact set of instructions given to crowd workers.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [Yes] See supplementary material for worker pay
and costs breakdown.
15


Reviews:
[{"review_id": "upAhuVrwWFQ", "reviewer": "Reviewer_T7tB", "summary": "The paper introduces CEBaB, a new benchmark dataset for assessing concept-based explanation methods in the domain of Natural Language Processing (NLP). \nCEBaB consists of short restaurant reviews with human-generated counterfactual reviews in which an aspect has been modified. \nCEBaB is leveraged to compare various concept-based explanation methods.\nThe purpose of the paper is to establish natural metrics for comparative assessments of these methods.\nMore precisely, five leading concept-based explanation methods: CONEXP, TCAV, ConceptSHAP, INLP, and CausaLM.\nThe dataset of reviews is composed of four aspects (food, service, ambiance, noise), associated with binary evaluation and an overall evaluation estimated from 1 to 5 stars.\nThe dataset is crowdsourced from an initial set of OpenTable reviews.\nAs a first step, the crowdsourcers have to expand the reviews to focus on one specific aspect.\nIn a second step, the crowdsourcers have labeled each aspect of the edited reviews.\nThe notion of the concept associated with the reviews is not clear, I presume it will be words.\nThe authors have used it as a classifier to explain a fine-tuned pretrained language model to predict the overall binary sentiment of all restaurant reviews", "questions": "Can you please detail the definition of the concept in the dataset? This point is central to all evaluated explanation methods.\nDid you consider augmenting the dataset through paraphrasing, do you think it would cause an additional challenge?", "limitations": "I do not see a strong limitation in regard to the purpose of the paper and the associated contribution and results.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.\n * The paper addressed a pertinent question of the explainability of large neural models\n * The paper is clear and properly federate in a unified description method a relatively large spectrum of explanation methods\n * The dataset is large and addressed the problem of explanation in NLP which is particularly known as tedious\n * The dataset has been crowdsourced in two separated steps which is a reasonable indicator of the limited noise in the data.\n * The paper already compares a large set of explanation methods with the resulting dataset\n\nWeakness\n * Unfortunately, the dataset is only in English, as most of the pretrained models are now multi-lingual it could have been interesting to consider this point.\n * The notion of concept is not clear in the context of the reviews, I assume it is the tokens.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kt22EulxD1M", "reviewer": "Reviewer_dBGa", "summary": "This paper introduces a new benchmark called CEBaB for evaluating concept-based explanation methods. This benchmark dataset consists of 15,089 restaurant reviews obtained by editing 2,299 original reviews from Opentable. Crowdworkers are asked to modify an original in order to alter a specific aspect (food, ambiance, service, or noise) in the review while holding all other aspects constant. This process creates pairs of examples where only one aspect of them is different, which allows estimating the causal effect of changing the particular aspect. The paper evaluates 6 different explanation methods by measuring the distance of the concept-based explanations w.r.t. the estimated causal effect.\n", "questions": "*Benchmarks:*\n\nHave the authors verified whether edits affect other aspects? The paper verifies that the edits successfully achieve the aspect-level goal (altering a particular aspect towards a specific direction), but there are no descriptions on whether other aspects are constant, which can be done with the help of crowdworkers on a small number of examples in the benchmark\n\nAre the final aggregated review and model score sensitive to different edits of a particular aspect-level goal? For each original review and a specific aspect-level goal, the paper collects one edited text. Ideally, it’s better to have multiple samples (as stated in Eq (2)) for estimating the ICaCE and ATE. It might be worthwhile to collect multiple edits for a relatively small set of examples and investigate the sensitivity.\n\n*Experiments:*\n\nThe paper benchmarks the performance of different explanation techniques and presents some interesting findings. E.g. TACV and ConceptShap performs worse (or on par) with the random baseline, even on the straightforward food aspect. It might be worth elaborating or providing qualitative analysis of the failures of these methods.\n\n*Related work*\n\nWhile the central contribution of this paper is benchmarking. The paper mainly discusses different explanation techniques. It might be good if there could be more discussion on the evaluation side, especially metrics and experimental findings. E.g., whether other work [11,56] uses the same or mostly similar evaluation metrics but evaluates on synthetic tasks.\n\n", "limitations": "See questions.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper collects a natural language dataset with pairs of human-written counterfactuals, which allows building.\n\nThe experiments cover an array of popular explanation techniques with varying requirements of access to the models.\n\nThe writing is clear and easy to follow.\n\nWeakness:\n\nWhile the collected benchmark and the experiments of evaluating different concept-based explanation methods are a central contribution, the paper does not provide some important details .\n\nThe paper does not provide thorough discussion of results.\n\nThe paper does not clearly discuss the relationship with other work on evaluating concept-based explanation methods.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "g5AYqy3g8Sc", "reviewer": "Reviewer_NcbX", "summary": "This paper introduces CEBaB, a new benchmark dataset to assess model explanations for NLP models. The key idea here is to edit short restaurant reviews with human generated counterfactual reviews in which one aspect of the review is modified. With aspect level and review level sentiment annotations on CEBaB, authors are able to approximate how a particular model might weigh some abstract concepts more than others when performing sentiment classification. The authors evaluate six (existing and proposed) explanation methods on this benchmark and find that there are significant differences between them. I like that this is a simple and intuitive idea that appears to work well in practice. However, there is a disconnect between the general claims the paper makes and the benchmark. The claims are about NLP model behavior in general but should be constrained to sentiment analysis. I have updated my sore following author response.", "questions": "- I don't understand what Table 4 is. The caption is very confusing so if you could share that, that'll be great.", "limitations": "I would like a greater discussion of how the method could be extended to other more complicated tasks. And if there are challenges associated with that, what are those?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- This paper addresses an important problem of evaluating explainability methods. The paper is well motivated and well written.\n- The conceptual framing offered by this paper as well as the resource contribution could be useful to future researchers.\n- The authors have also provided an extensive experimental analysis of how well multiple model explanation methods are able to capture the true concept effect.\n- I like the focus on direction of an intervention's effect on model behavior rather than the scale of effects as it simplifies the problem to knowing what causes what rather than the scale of a cause's contribution to the effect.\n\n\nWeaknesses:\n- This work could be much better positioned in prior work on benchmarks for explainability and counterfactual analysis for model explanations (such as the Language Interpretability Toolkit).\n- The benchmark is only for sentiment analysis which limits its use while broader claims are made about NLP explainability. While that's still a good first step, it is arguably easier to \"explain\" a model on a sentiment analysis task versus say NLI or other more complicated NLP tasks using CEBaB, and there is no discussion offered in the paper on how this would extend to those tasks.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2yMaQsx_TZ", "reviewer": "Reviewer_Jcm3", "summary": "The paper introduced CEBaB, a new  benchmark dataset for assessing concept-based explanation methods in Natural Language Processing (NLP). CEBaB consists of short restaurant reviews with human-generated counterfactual reviews in which an aspect (food, noise, ambiance, service) of the dining experience was modified. ", "questions": "How this paper help developing evaluation techniques for other problems apart from sentiment analysis?", "limitations": "They have described the limitations. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nEvaluating explanation methodology for causal effect is important for the research community. The dataset would be useful for that purpose. \n\nWeakness:\nThe dataset evaluates the model explanation methods based on aspects (food, noise, ambiance, service). This dataset can be used for sentiment analysis. I am not sure if this dataset can help in other classification problems for evaluating causal effects. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept