Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
Authors: Yair Ori Gat
Nitay Calderon
Amir Feder
Alexander Chapanin
Amit Sharma
Roi Reichart
Venue: ICLR 2024 poster (2024)
Keywords: NLP
LLMs
Interpretability
Explanation
Causal Inference
Matching
Primary area: visualization or interpretation of learned representations
TL;DR: Addressing challenges of black-box NLP model interpretability through two model-agnostic approaches of counterfactual explanation: generation and efficient matching.

Abstract: Causal explanations of the predictions of NLP systems are essential to ensure safety and establish trust. Yet, existing methods often fall short of explaining model predictions effectively or efficiently and are often model-specific. In this paper, we address model-agnostic explanations, proposing two approaches for counterfactual (CF) approximation. The first approach is CF generation, where a large language model (LLM) is prompted to change a specific text concept while keeping confounding concepts unchanged. While this approach is demonstrated to be very effective, applying LLM at inference-time is costly. We hence present a second approach based on matching, and propose a method that is guided by an LLM at training-time and learns a dedicated embedding space. This space is faithful to a given causal graph and effectively serves to identify matches that approximate CFs. After showing theoretically that approximating CFs is required in order to construct faithful explanations, we benchmark our approaches and explain several models, including LLMs with billions of parameters. Our empirical results demonstrate the excellent performance of CF generation models as model-agnostic explainers. Moreover, our matching approach, which requires far less test-time resources, also provides effective explanations, surpassing many baselines. We also find that Top-K techniques universally improve every tested method. Finally, we showcase the potential of LLMs in constructing new benchmarks for model explanation and subsequently validate our conclusions. Our work illuminates new pathways for efficient and accurate approaches to interpreting NLP systems.

Published as a conference paper at ICLR 2024
FAITHFUL EXPLANATIONS OF BLACK-BOX NLP MOD- ELS USING LLM-GENERATED COUNTERFACTUALS
Yair Ori Gat Nitay Calderon Amir Feder Alexander Chapanin', Amit Sharma" and Roi Reichart
T Facclty of Data and Decision Sciences, Technion, IIT Columbia University Data Science Institute, M Microsoft Research India
ABSTRACT
Causal explanations of the predictions of NLP systems are essential to ensure safety and establish trust. Yet, existing methods often fall short of explaining model predictions effectively or efficiently and are often model-specific. In this paper, we address model-agnostic explanations, proposing two approaches for counterfactual (CF) approximation. The first approach is CF generation, where a large language model (LLM) is prompted to change specific text concept while keeping confounding concepts unchanged. While this approach is demonstrated be very effective, applying L.M at inference-time is costly. We hence present a second approach based on matching, and propose a method that is guided by an LLM training-time and learns dedicated embedding space. This space is faithful to given causal graph and effectively serves to identify matches that approximate CFs. After showing theoretically that approximating CFs required order to construct faithful explanations, we benchmark our approaches and explain several models, including LLMS with billions of parameters. Our empirical results demonstrate the excellent performance of CF generation models as model-agnostic explainers. Moreover, our matching approach, which requires far less test-time resources, also provides effective explanations, surpassing many baselines. We also find that Top-K techniques universally improve every tested method. Finally, we showcase the potential of LLMs in constructing new benchmarks for model explanation and subsequently validate our conclusions. Our work illuminates new pathways for efficient and accurate approaches interpreting NLP systems.
INTRODUCTION
Providing faithful explanations for Natural Language Processing (NLP) model predictions is im-
perative to ensure safe deployment, establish trust, and foster scientific discoveries (Amodei et al.)
2016; Goodman & Flaxman 2017: Guidotti et al., 2019 Jacovi & Goldberg [200 Lyu et al. 2022).
These aspects are particularly significant in NLP, where the complexity of language and the opaque behavior of black-box models. For an explanation to be genuinely faithful and accurately depict model's underlying reasoning, it is crucial to establish causality. Recognizing this inherent link and following previous works (Vig et al. 2020 Geiger et al. 2020 Feder et al. 202 Wu et al. 2023a Feder et al. 2023 Wu et al. 2023b). ) in this paper we introduce theoretical framework that binds faithfulness and causal explanation together and propose two causal-inspine explanation methods.
In contrast to model explanation techniques that often conflate correlation with causation, causal- inspired methods usually contrast predictions for input example with those of its counterfactual (Soulos et al. 2020 Elazar et al.) 2021 Finlayson et al. 202T). Indeed, counterfactuals are at the highest level of Pearl's 's causal hierarchy (Pearl (2009). highlighting how changes lead to different prediction. However, they cannot be acquired without knowing the complete data-generating process (or structural model) of the text (Balke & Pearl 1994), which not practical for given real-world problem. Hence, we turn to approximated counterfactuals (CFs): Imagining how given text would look like a certain concept in its generative process were different (Calderon al. 2022).
"Equal contribution. Corresponding author: ni tay@campus. technion.cee ac. Code: https:: / //gi thub. com htt.///arrrattccuuaaleeeeaaaa http:///cccssaleeppppaaattttssss

---

Published as a conference paper at ICLR 2024
In $3.1 we propose simple, intuitive, and essential criterion for explanation methods: Order- faithfulness - If the explanation method ranks one concept's impact above another's s, its true causal effect should genuinely be greater. We then theoretically show that in contrast to non-causal expla- nation methods (that overlook confounders, for example), approximating CF methods are always order-faithful. Moreover, since CFs do not depend the explained model, they enable causal estima- tion in a model-agnostic manner. Model-agnostic explanation methods offer numerous ad advantages, especially during model selection, debugging, and deployment. For example, developers juggling multiple models can effortlessly rank them based on their vulnerability to confounding biases (such as gender bias). Hence, we focus only on -ades-agnostic explanations in this study.
Yet, acquiring CFs in past work was limited to simple local manipulations (Ribeiro et al.. 2020 Ross et al., 2021; Wu et al., 2021) or costly manual annotation (Gardner et al. 2020 Kaushik et al. 2020a), hindering practical causal effect estimation of high-level concepts on NLP models. To address the above limitations, we rely on Large Language Models (LLMs)) and introduce two practical approaches for model explanation. The first approach is Counterfactual Generation, in which an LLM is prompted to change a concept a text while holding confounding concepts fixed. We empirically demonstrate that LLM- LLM- generated CFss are the SOTA explanations. However, utilizing LLMs comes with a high computational and financial cost. Moreover, the slow generation process may limit the application of CF generation methods, especially in scenarios demanding real-time explanations or when explaining vast quantities of data (Calderon et al. 2023a).
Therefore, we introduce an efficient alternative: Matching, which is up to 1000 times faster (see Appendix B We use matching to estimate the causal effect, pairing each treated unit with one or more control units with similar observed characteristics, i.e., identifying CF approximations within a pre-defined candidate set. To enhance matching quality, we propose in $3.3 a novel method for learning a causal embedding space. Given a causal graph (e.g., Figure I we first employ an LLM for generating CFs (note that the LLM is used only during the learning phase). We also identify possible valid matches: Instances with identical values in observable adjustment variables (that satisfy the back-door criterion). Subsequently, we train our causal representation model by minimizing an objective composed of contrastive loss components. The causal representation model learns to encode examples similarly to their LLM-generated CFs matched examples. Conversely, it is trained to produce dissimilar representations for the query example and any misspecified CFs. As a result, representations remain faithful to the causal graph and can effectively identify matching candidates.
We undertake rigorous experiments to compare our causal language representation method against various generative and matching baselines. For this purpose, we employ CEBaB (Abraham et al. 2022], the only established benchmark for evaluating causal explanation methods. We estimate the causal effect of 24 concept interventions LMs, and two zero-shot LLMs with billions of parameters. In Appendix &Cwwe we conduct an ablation study examine the impact of each design choice within our matching method.
the predictions of five models, three small fine-tuned
Our empirical results reveal three key findings: (1) The counterfactual generation approach provides strong black-box model explanations. (2) Our method for learning causal representations for matching outperforms all the matching baselines, including the best-performing method from the CEBaB paper (see Figure (6). (3) Top-K K matching universally greatly enhances the explanatory capabilities of all examined methods, including generative models, when generating multiple CFs.
Finally, building on our findings that LLMs can produce high-quality CFs, we address a pressing challenge: The lack of benchmarks for NLP model explanation methods. In $D we demonstrate how to construct such a benchmark by leveraging GPT-4 to generate pairs of inputs and their CF' (used as "golden CF*). The new dataset focuses on Stance Detection, NLP task that determines the position expressed by the speaker towards a particular topic (c.g., abortions, climate change) (Küçük & Can 202I). By utilizing it, we observe similar patterns to our main conclusions. We hope this study will pave the way for safer, more transparent, and more accountable Al systems.
RELATED WORK
Explanation methods and causality. Feature importance tools (Molnar 2020) and gradient- based methods methods (Zeiler& & Fergus] often restricted input features, making it challenging connect their analyses with real-world
et al.) Shrikumar et al. [2017] Gat et al. (002))
2

---

Published as a conference paper at ICLR 2024
concepts that do not reduce simple properties of inputs. In this study, focus the explanation of high-level real-world concepts (Poeta et al. 2023). In probing, a small supervised (Conneau et al.
2018
Tenney al. 2019) unsupervised (Clark et al. 2019 Manning et al. 2020 Saphra & Lopez 2019) model is used to measure whether specific concepts are encoded at specific places in a network.
Geiger al. (202T) show that probes cannot reliably provide causal explanations. Other methods perturb input or hidden representations to create CF states that can then be used to estimate causal effects (Ribeiro al. 2016, Elazar et al., 2021: Finlayson et al. 2021: Soulos al. 2020; Vig et al.)
2020; Geiger et al. (021). However, these methods are prone to generating implausible inputs or network states unless the interventions are carefully controlled (Geiger et al., (020). Recently, Wu et al. (2023a) proposed the Causal Proxy Model, a novel explanation method inspired by S-learner (Künzel et al., 2019), which mimics the behavior of the explained model when the input is a CF Although CPM is effective, the need to train distinct explainer for each explained model is major disadvantage. Conversely, our study focuses on model-agnostic explanation methods.
Approximating counterfactuals. A common use-case for CF examples in machine learning is for data augmentation (Hong et al. 2023). These CFs involve perturbations to confounding factors (Garg et al., 2019), to the label (Kaushik et al.. 2019: 2020b [aa et al. 2020). CF examples can be generated through manual editing. heuristic keyword replacement, or automated text rewriting
(Kaushik et al. 2019 Gardner al. [2020] Shekhar al.]
2017: Garg et al. 2019) Feder al. 202 la
Zmigrod et al. 2019 Riley et al.] |020]] Wu et al. 2021 Mao et al. [202]] Rosenberg et al. 2021)
Manual editing is accurate but expensive, while keyword- -bsed methods can be limited in coverage and difficult to generalize across languages (Antoniak & Mimno 2021 Zhou & Wu (023). While
LLMs can overcome these challenges, they come with prolonged latency, high financial costs, or privacy constraints. Our matching approach is an efficient alternative to overcome these limitations.
3
METHOD
This study focuses on black-box NLP model explanations by estimating the causal effect of high-level concepts on model prediction. The first requirement for any causal estimation method is access to a causal graph that describes our causal beliefs, i.c., the concepts and the relationships between them (e.g., the causal graph in Figure D. See Appendix SF.T for further discussion about the requirements.
To provide an accurate explanation, we should estimate the Average Treatment Effect (ATE) Pearl 2009) or the Individual Treatment Effect (ITE) (Shpitser & Pearl 2006; Shalit et al.] 2017). Specif- ically, the treatment a high-level concept influencing the text (such as ambiance in a restaurant review), and the outcome variable is prediction of a text classifier. When discussing model explana- tion the sense of the causal effect of a concept on the model prediction, the common versions of the ATE and the ITE are the Causal Concept Effect (CaCE) (Goyal al. (019)) and the Individual Causal Concept Effect (ICaCE) (Abraham et al. 2022), which we next formally define.
Given a DGP G, intervention write T + f'), a model f, and a query example ot, the CaCE and ICaCE are defined to be:
treatment variable T' t + t' (for simplicity sometimes
CaCE, (T,t, = En'ng [f do([T = t')] - Exin [f(=') = t)]
(I)
(m,, T, Ea'ng [f(x)) 22, do = (') - f (ax)
One way to estimate the causal effect and explain the model is by using counterfactuals, which enables causal estimation in a model-agnostic manner (because they do not depend on the explained model). While we are aware of the causal graph that encodes our causal beliefs, for a given problem, we cannot access the complete data- generating process (DGP) nor the variable values of an example, and therefore, we cannot produce gold counterfactuals. Given this, we propose two approaches for approximating them: Counterfactual generation ($3.2) and Matching ($3.3)
Zy is an approximated counterfactual (CF) of the text D; resulting from the intervention T: + (can be either model-generated CF match), then the causal effect estimators are:
ICaCE, (z,, T't,t)) ICaCE, y (Ii, Ze) f(āv) - f(x))
(2)
/ (T, tT,t,t' D
1
Σ ICaCE, Zer) + Σ
ICaCE (2e, xv) +
Σ ICaCE (ā,, Fv))
3

---

Published as a conference paper at ICLR 2024
3.1 COUNTERFACTUALS AS AN IDEAL MODEL EXPLANATION
While definitions of faithfulness vary across literature, there is consensus that a faithful explanation method should accurately describe a model's 's underlying reasoning (Rudin) 2019 Dasgupta et al. 2022 Lyu et al. 2022). Relying on the causality literature, essential criterion: Ordee--aithfulnesss This implies that if the explanation method ranks the impact of one concept (or intervention) higher than another, the true causal effect of the first concept should exceed that of the second. Order-faithfulness is crucial in scenarios like model selection and deployment. For instance, determine the faimess of hiring model, it is essential know whether it incorrectly prioritizes non-relevant concepts (e.g.. gender) over genuine concepts (e.g., experience).
propose simple, intuitive, and
Definition (Order-Faithfulness) An explamation method S is order-faithful if:
BB~g(S(f= 11,,,,,
(S(, (2, + CaCE f1) > (T,, (3)
(T2,
Note that the definition above does not mandate the explanation method to estimate the causal effect accurately or generate scores proportional to the causal effects. Instead, it simply requires the method to be rank-preserving. In addition, it holds for comparing the importance of two distinct concepts, as many attribution methods like LIME do (Ribeiro et al., (016)) and it also holds when assessing the importance of changes in the same concept. We believe that order-faithe frder-faitffulness is a necessary condition for the broad definition of faithfulness. If an explanation method ranks the importance of one concept above another, but fact its causal effect smaller, then the method does not accurately reflect the actual reasoning process of the model being explained.
In Appendix |A| we provide detailed formal definitions of two explanation methods: Approximated CF explanation methods and Non-causal explanation methods (which we define as any function of the observed data). Furthermore, we prove the following theorem, which elucidates why approximated CF methods are always order-faithful, unlike non-causal ones.
Theorem. For explained model f. (1) The approximated CF explanation method Ser is order- faithful every DGP G and a pair of interventions; (2) For every DGP G, there exist a DGP g' resulting from small modification o a model f", and a pair of interventions Tí :11 + T2, t2 + i22 such that the non cauccllaal explanation method Sue is not order-faithful.
Proof sketch: We first prove that approximated CF methods are always faithful by showing the expected prediction of an approximated CF is equal to the interventional one (conditioned on the do operator). To prove the second part, we construct G' by introducing a new unobserved confounder variable (the "small modification") that reverses the order two interventions' causal effect on a new model f'. In addition, it produces for G since the generation process of x is the same in the two DGPs. Hence, Sye cannot be order-faithfin in G and G' at the same time.
ensure that the non-causal method produces the same explanations as
The theorem underscores the importance of developing causal-inspired explanation methods and suggests why our CF-based approaches are more effective than other baselines.
3.2 LLM-GENERATED COUNTERFACTUALS
Our first approach for CF approximation is Counterfactual Generation, in which a generative model (an LLM) prompted with to change an attribute conveyed in the text. We explicitly inject our causal beliefs into the prompt using the fundamental causal concept of adjustment (see Appendix (
Given a query example Ft, we instruct the LLM to intervene in the text by changing the value of the concept T from / to L'. Additionally, we instruct the L.M to consider the adjusted concepts (confounders) and hold them fixed. Optionally, we can specify the non-adjusted concepts (such as mediators and colliders) if they exist and ask the LLM to consider that the intervention on T might affect these concepts. This specification can increase the precision of the CF. For the Top- K technique, we simply generate multiple CFs. See Appendix F2 for examples of prompts and generated CFs.
Nevertheless, utilizing an LLM at inference time is frequently impractical because of its prolonged latency, high financial costs, privacy concerns that prevent data from being sent external servers (see Calderon et al. (202a) and our latency measurements in Appendix $B. Considering these challenges, in the next subsection we introduce efficient alternative, the matching technique.
4

---

Published as a conference paper at ICLR 2024
Table Toy illustration of examples (and their concept values) sampled from the four of a given query example and an intervention of changing service to positive (S + +). The misspecified CF (X.cr) resulted from a wrong intervention of removing the ambiance concept (A < ?).
Query
Excellent lobster and decor, but rude waiter.
F
S
N: A:+
Excellent lobster and decor, and friendly waiter.
%
F:
S + N:
A:+
700
A cozy atmosphere with great pizza and service.
U
F
5:+
N:? A:4
Excellent lobster and rude waiter.
Xx.F
F:+
8
A:?
Too expensive for a loud chaos with bland food.
Xx
F:
S:? N:
A:
Figure 1: Our illustration of the causal graph of the CEBaB benchmark (Abraham al. 2022). U and V are (unobserved) exogenous variables representing the state the world. For example, V may account for style, syntax, and length. The four concepts F (food), S (service), A (ambiance) and N (noise) affect both the textual review X and its five-star rating Y. The relationship between X and Y can be causal anti-causal. f is the model In this study, we are interested in estimating the causal effect of changing a concept on the predictive model (e.g., changing the value of the red variable S from positive to negative).
wish to explain, which is trained to predict Y from X.
3.3 CAUSAL REPRESENTATION LEARNING FOR MATCHING
Although counterfactual generation is valuable model explanation approach, employing LLMs during inference can be infeasible. Conversely, there is an opportunity to harness model-generated CFs train an efficient method that seeks approximations within pre-defined set of candidate texts, mitigating the need for direct generation. This approach is known as matching (Stuart (010) and endeavors to match each treatment unit with its most similar control unit. To allow effective matching, we introduce a novel method that learns to encode texts in space faithful to the causal graph, i.e., each treatment unit resonates closely with its CF counterparts and remains distinct from confounding texts. Formally, matched example of 2; is defined to be:
mi (z:)) arg max 8 (((x))) o(xx)) xy ED(==t)
The candidate set is denoted by DT t') and is also known as the control group, contains only texts whose treatment variable equals t'. and known in advance (i.e., both the treatment and control groups are pre-defined and there is no need to infer which text belong to which group). o(-) is feature extractor function (the language representation model we aim to learn) that maps discrete text the hypersphere space where matching is conducted by ranking candidates according their similarity with the query. In our experiments, the similarity function s(-) is the cosine similarity (defined in Eq, I]].
Notice that Eq 4 defines 1:1 matching, for Top-K matching we use: Σ (CaCE,(x)) m. (xx)
Our causal representation method is based on Encoder-only language model (e.g., RoBERTa (Liu et al., 2019) or a Sentence'
CeetenceTTrnsfoore Transformer (Reimers & Gurevych 2019)) fine-tuned to minimize an objective consisting of six contrastive loss components. The goal of the objective is to enhance the similarity with approximate CF candidates while reducing similarity with misspecified CFs.
Given a causal graph, an intervention on a treatment variable T - and an example æs, we define the four following sets (see toy examples in Table
1. XxF This set contains Counterfactuals, obtained by modifying x, through the intervention on T. while the other variables are held unchanged. We utilize LLM (ChatGPT) to simulate several CFs (up to ten).
2. X, This set contains Matched samples. These instances share identical values for the adjustment variables (that satisfy the back-door criterion, see Appendix şHp. Our study employs smaller model predict their values and construct set of valid matches.
5

---

Published as a conference paper at ICLR 2024
3. X-cr This set contains Misspecified Counterfactuals. These examples derived from editing 2; through interventions on variables other than the treatment variable. The term misspecification well-known in the causal literature and indicates wrong assumption of the DGP (Vansteelandt et al. 2012). In our case, a wrong intervention.
4. X This set contains Misspecified Matched samples. This the complementary set to XN, containing all those instances that do not qualify as valid matches due to differing values of the adjustment variables.
The causal representation model is trained by minimizing the sum of six contrastive loss components:
((x)) = [(os, Ker, Xx) + L( Ker, xx.r) ) + L(x,, Xcr, Xx)
+ C(x,,, Xx, x) + (xx,, Xn, Xxrr)
(5)
+ +(xx, Xx.F, X..)
Where the contrastive loss (Wang Liu 2021) is defined below and aims to attract some input æ to the "positive set" X4, and separate it from the "negative set" x_ (T is temperature hyperparameter);
En, exp(s((((r)), ((ax+))/T)
C(x,%,,, x_) - log Exe exp(s(x(())))) pl 11/T) + E.. (xp(s(o(x)).) 0(. })/T)
(6)
We hypothesize (and verify) that omitting components from the objective (Eq. 5 risks favoring inaccurate matches, as we highlight in the ablation study at Appendix $C Our objective prioritizes text matches that closely resemble CF approximations within a candidate set, defaulting to adjusting variable values in their absence. When choosing between misspecified CFs or misspecified matches, the methods favor the latter, given their potential shared traits with the CF, such as syntax or style linked to exogenous variables. Formally, given query example 2; and four matching candidates Tee E Xct, ; aw Xn, x-cF E X-cF, the candidates' ranking, which is based on their similarity with æt, should be Z-M I x--F 3 IH I ZcF. This desirable order not arbitrary and is a direct product of the six components, ensuring robustness to variations of the candidate set. In addition, we investigate in the ablation study (Appendix $C) the applicability of our matching method without pre-idente me-identified concept annotations in the training dataset (i.e., completely unsupervised setup). To this end, we utilize an LLM and predict the concept values in zero-shot manner.
In Appendix $G we provide additional implementation details, including the training procedure of our causal representation model G.2 and hyperparameters G.3
EXPERIMENTAL SETUP
4.1 MODEL EXPLANATION PIPELINE
Causal Iilinattion-Base Benchmark (CEBaB), CEBaB (Abraham et al. 2022) is an interventional dataset consisting of examples. interventions, and corresponding numan--ritter ground- -trtthhhhhhhh CF for each example and intervention. CEBaB originated from thousands of original restaurant reviews obtained from OpenTable. Each entry (original review or CF) in CEBaB received a 5-star sentiment rating, labeled by five annotators. Furthermore, every text was annotated at a conceptual level, as positive, negative or neutral w.r.t. four central mediating concepts: Food, Service, Ambiance, and Noise (see Figure D. CEBaB contains two train sets, exclusive (NV 1463) and inclusive (which we do not use), development (N = 1672), and test (N = 1688) sets. We randomly split the exclusive set into two equal sets: the train set and matching candidates set.
Explanation method evaluation. For a given example x; and an intervention T : - t', we can estimate the golden ICaCE using the corresponding human-written CF: ICaCE / (æ,, 3.). This is the difference between the outputs of the model f (a -ivedimensoon vector in CEBaB corresponding to the sentiment). Likewise, in a Ca-based explanation method, the estimation is computed with the approximated CF: ICaCEE (we, T We can then evaluate the explanation method using:
Err(f, m, M,T,T,TtR. T,t,t')
Σ (ED
Dist (CacEE / (x1, 2²),, ICaE xT-))
(7)
ID (T t t')l

---

Published as a conference paper at ICLR 2024
Where Dist can be, for example, the L2 distance, the cosine distance (1 - -csss or the norm difference (see Abraham et al. (2022) and Eq. [2). The Err scores we present in this study are the mean over 24 interventions: four concepts with three values (six interventions for each concept). See Appendix $G.1 and $G for full details about the causal estimation and explanation evaluation pipelines.
4.2
MODELS AND BASELINES
In Appendix we provide additional implementation details, including hyparameters and prompts.
Generative approach. We employ three different CF generation techniques: Zero-shot Generative, Few-shot Generative (with three demonstrations) and Fine-tune Generative. The first two are based on the Decoder-only ChatGPT (Ouyang et al., 2022), while the third is based on an Encoder-decode T5-base model (Raffel et al.) 2019) trained using parallel dataset, comprised of original reviews and their human-writter CF's (the inclusive train set of CEBaB). The zero-shot and few-shot prompts are described in Appendix $
Matching baselines. We benchmark against six matching methods: (1) Random Match, which randomly selects an example from the candidate set; (2) Propensity (Benedetto et al. 2018), which first estimates the propensity score P(T f'(x)) using concept predictor, and then conducts matching based on this score; (3) Approx, which randomly selects an example from Xy. Notice that Xy can be an empty set, thus, unlike the other baselines, matching may not be performed; (4) PT RoBERTa, which utilizes a pre-trained LM to represent texts and finds matches using their cos similarity; (5) PT S-Transformer, same as PT RoBERTa, expect the backbone model is SentenceTraansorre model (Reimers & Gurevych, 2019) trained to maximize semantic textual similarity; and (6) FT S S-Transformer, Sentence Transformer fine-tuned predict the five-star sentiment;
Notice that Abbraham et al. 2022) found that Approx outperforms all seven tested (non-matching) baselines (see Figure 60. Therefore we not include them this study.
Interpreted models. We interpret five different models of varying sizes. The first three are Encoder- only models fine-tuned using only the original reviews from the train set of CEBaB to predict the five-star sentiment. These models include DistilBERT (Sanh et al.) 2019), BERT (Devlin et al. 2019), and RoBERTa (Liuu et al. 2019). The other two are the zero-shot chat versions of the Decoder-only Llama-2 model (Touvron et al., 2023) with and 13 billion parameters. We extract the five-star sentiment distribution of the Llama models using the prompt described in Figure
5 RESULTS
Our main results are provided in Table 2 In Appendix $C we provide a thorough ablation study, demonstrating the effect of each of our design choices, including, among others, the impact of variations of the candidate set and the causal model objective (Eq. 5). Our three main findings are:
1. Counterfactual generation models are the SOTA model-agnostic explainers. The first promise of this study is that CF generation models are strong explainers. As can be seen from Table 2 the generative models achieve much lower errors compared the other methods, making them the SOTA model-agnostic explainers. In addition, the table provides two additional findings: (1) Few-shot prompts are better than zero-shot prompts for generating CFs; and (2) The small fine-tuned Encoder-decoder T5 is the best generative model. This suggests that when parallel dataset consisting of pairs of input examples and their human-writte CFs is available, then it is better fine-tune small model. Moreover, the small model is computationally (and financially) less expensive than LLMs (but much slower than matching methods). Nevertheless, consider that collecting a parallel dataset for fine-tuning is labor intensive.
The causal representation model is the best-pero matching method. Table also sheds light on the performance of our novel causal representation method (described in $3.3 our causal model in short) in comparison to various explainers and matching baselines. Notably, our causal model consistently outperforms all the matching baselines, achieving substantially lower errors across five explained models. Noteworthy, Abraham et al. (2022) proposed the Approx method (the only matching method they examined) and found it outperforms common NLP explainers. Our results show that all matching baselines (including Approx. excluding Random Match and Propensity) are competitive and perform similarly. Nevertheless, they fall short of our causal representation model.
7

---

Published as a conference paper at ICLR 2024
Table 2: Comparison between different methods and baselines. The columns present Err scores (Eq. 7 when explaining different five-class sentiment models. The sub-columns present three different measures: Euclidean distance (L2), Cosine distance (Cos), and norm difference (ND). The top table presents scores using single match (K = 1), and the bottom table when K = 10. The Generative rows are not matching methods and thus are not comparable to such methods. Another non-comparable row is the first, which presents the performance of our causal model with a candidate set that also includes ground -ruut (GT) CFs. Numbers are means over 24 interventions, is better.
DistiBBRT
BERT
RoBERTa
Llama2-7b
Llama2-13b
AVG
K=1
1.
Cos
ND
1.2
Cos
ND
1.2
Cos
ND
1.2
Cos
ND
1.2
Cos
ND
Causal Model (+GT)
.13
.09
.06
.14
.10
.07
.13
.09
.06
.12
.07
.06
.11
.07
.06
.09
Fine-tune Generative
.38
.28
.21
.42
.28
.23
.39
.27
.21
.36
.23
.22
.31
.21
.19
.28
Few-shot Generative
.42
.32
.23
.45
.29
.23
.41
.30
.21
.38
.26
.23
.33
.23
.20
.30
Zero-shot Generative
.43
.35
.24
.47
.35
.25
.44
.34
.23
.40
.27
.25
.35
.26
.21
.32
Random Match
.88
.64
.47
.96
.62
.49
.95
.63
.49
.84
.54
.50
.84
.55
.49
.66
Prupensity
.90
.68
.48
.99
66
.56
.99
.68
.56
.88
.61
.53
.86
.58
.52
.70
Approx
.74
.60
.41
.81
-58
.45
.79
.60
.44
.71
.49
.43
.70
.54
.49
.58
PT RoBERRa
.75
.61
.39
.83
-59
.42
.81
.61
.41
.74
.53
.43
.73
.51
.43
.59
PT S-Transformer
.75
.60
.40
.83
.60
.45
.81
.61
.43
.74
.55
.44
.69
.52
.41
.59
FT S-Transformer
.72
.78
.46
.22
.79
.59
.80
.84
.56
.73
.63
.45
.68
.63
.39
.66
Causal Model
.66
.55
.36
.70
.55
.39
.68
.56
.37
.64
.47
.39
.59
.45
.36
.52
DistiBBERT
BERT
RoBERTa
Llama2-7b
Llama2-13b
AVG
K K= 10
L2
Cos
ND
L2
Cos
ND
L2
Cos
ND
L2
Cos
ND
L2
Cos
ND
Fine-tune Generative
.32
.21
.20
.36
.23
.23
.34
.21
.21
.30
.18
.20
.27
.16
.18
.24
Few-shot Generative
.38
.30
.22
.42
.28
.24
.38
.29
.21
.34
.24
.22
.30
.22
.19
.28
Zero-shol Generative
.38
.32
.23
.42
.31
.25
.40
.31
.23
.36
.24
.23
.31
.22
.19
.29
Random Match
.70
.52
.40
.78
.51
.47
.77
.51
.45
.67
.45
.40
.67
.45
.40
.54
Propensity
.72
.54
.43
.80
.52
.51
.79
.52
.50
.68
.47
.42
.69
.47
.43
.57
Apprux PT RoBERTa
.61
.49
.37
.68
.48
.43
.66
.48
.41
.58
.40
.37
.57
.41
.36
.49
.64
.50
.36
.71
.49
.44
.70
.49
.42
.61
.44
.37
.60
.43
.36
.51
PT G-Transformer
.64
.51
.38
.72
.50
.45
.71
.50
.44
.61
.45
.38
.59
.44
.36
.51
FT S-Transformer
.61
.56
.43
.69
.66
.50
.67
.56
.47
.60
.48
.41
.56
.50
.36
.53
Causal Model
.56
.47
.35
.63
.46
.41
.60
.45
.38
.55
.40
.36
.52
.38
.33
.46
Although the generative approach exhibits better scores than the matching approach, it is important to note that the quality of the ICaCE, f estimation of any matching method is highly dependent on the matching set. Therefore, we also include the Err scores for our causal model when the candidate set contains the ground-truth human-written CFs (see the first row in Table In that case, our causal model outperforms even the generative models. This suggests that our method is potentially the SOTA explainer (although this is impractical since access to CFs is a strong assumption).
Finally, Figure 2 which plots the Err score when selecting the htth match my(x), reveals three desirable properties of our causal model: (1) The lowest Err score belongs to the first match, indicating that, on average, our causal model indeed selects the best match; (2) A monotonically increasing pattern (i.e., larger le results in higher Err) demonstrates strong association between representation similarity and explainability capacity: The more similar a match is, the more accurate the ICaCE, estimator becomes; and (3) The top-ranked matches selected by our causal model outperform other baselines, achieving notably lower Err scores.
3. Top-K K matching improves the explanation capabilities of every method. When comparing the top rows of Table T = 1) to the bottom rows (k- 10), it is easy to observe that Top- matching lowers the Err of every examined explanation method. This observation also holds for the generative models, which accordingly generate multiple CFs via sampling. The definitive results, combined with the simplicity and cost-effectiveness of the Top-K extension (see Appendix {B) suggest that top-K should be a standard for explainers. The question that arises is what should the value of K be?
In Figure 3 we plot the L2 Err as a function of & for our method and three other baselines when explaining DistilBERT (the trends are similar for the other metrics and explained models). The prominent '' shape curve of the causal model is desirable and indicates that the model selects good matches at first. Then, around A = 20, the error begins to surge and only after = 100 that matches the error at k- 1. Conversely, other baselines rank candidates less effectively and exhibit more
8

---

Published as a conference paper at ICLR 2024
L2 Error vs. the Match
L2 Error the Top-K
0.9
Approx
S-Trans Model
0.75
S-Trans
Score
Score
0.70
0.65
0.7
Approx
S-Trans
0.60
PT S-Trans
Causal Model
0.55
0
75
100
125
150
100
150
200
250
Figure 2: The 1.2 Err score for explaining the DistiBBERT model (Y-axis) as function se- lecting the hth match, (x)) For Figures 2 and 3 we present average Err scores only for inter- ventions with candidate set of size larger than 250, making the numbers differ from Table 2
The 1.2 Err score for explaining the DistilBERT model (Y-axis) as a function of Top- K matching. The figure illustrates the bene- ficial impact of considering multiple matches. However, beyond a certain point, adding more matches negatively affects causal estimation.
Figure 3:
gradual increase after their minimum. Nonetheless, the Top-K approach has beneficial effect for all methods and broad range of values can picked before the effect fades. We arbitrarily opted for k = 10 in this study, even though l: = 20 would have been more advantageous for our causal model. future direction could explore how to tune the k parameter for each individual example.
CONSTRUCTING A NEW BENCHMARK FOR MODEL EXPLANATION
While we aim to validate our earlier conclusions across multiple benchmarks, CEBaB remains the sole high-quality benchmark for NLP explainers. On the other hand, show that LLMs generate CFs that resemble human-written CFs. Consequently, we harness LLM capabilities to emulate CEBaB and propose benchmark for model explanation, eliminating the high cost of manually annotating such benchmarks. In Appendix DD, we thoroughly detail the new benchmark, including the construction process and the new experiments Detection, which is an important and well-studied NLP task that determines the position (support, oppose, neutral) expressed by the speaker towards particular topic (e.g.. abortions, climate change). We rely on the stance detection dataset from the SemEval 2016 shared task (Mohammad et al. 2016).
conduct. We decided focus on Stance
For simplicity of the new setup, we embrace the same causal graph of CEBaB (see Figure D. but replace the four review concepts with new relevant concepts: (1) Tweet's subject (Abortions, Atheism, Climate Change, Feminism, Hillary Clinton); (2) Speaker's age (Unknown, Teenager, Elder); (3) Speaker's gender (Unknown, Female, Male); and (4) Speaker's Job (Unknown, Farmer, Professor). Using GPT-4, we generate new tweets and golden CFs for the test set, enabling the evaluation of explanation methods. Furthermore, unlike CEBaB, this dataset allows evaluation in out-of- distribution setting, such as when the matching candidate set compromises texts discussing subjects differing from the subject of the test examples. We then test our methods under distribution shift, validating the robustness of our main results from the original CEBaB dataset (results Table 6).
7 DISCUSSION
In this paper, we explore the terrain of explaining the impact of real-world concepts on the predictions of NLP models. We focus on model-agnostic techniques, which do not require access to the explained model during the training time and hence can explain numerous models. We hope this paper may direct the attention of the NLP community toward the vital intersection of causal inference and model interpretability. We further motivate the utilization of LLM for model explanation, either for generating CFs or for guiding efficient approximation methods. Nevertheless, the gap that still exists between the two approaches demonstrates the need for further research. In Appendix Wewe discuss the requirements, limitations, and the applicability of our proposed methods to complex causal graphs. In Appendix g extend this discussion and emphasize the key implications of findings.

---

Published as a conference paper at ICLR 2024
Ethics statement. While explanation methods offer deeper transparency of NLP models, researchers and practitioners should remain vigilant and mindful. These methods, misused, could over- emphasize or amplify biases present in the model data. Our new dataset ($6), derived from tweets taken from SemEval16 (Mohammad et al. 2016), could inherently contain societal biases, including, but not limited to, social prejudices or racism. We acknowledge that tweets can sometimes reflect or amplify societal sentiments, both positive and negative. Furthermore, by leveraging GPT-4 to edit these tweets (e.g., changing the gender of the writer), we recognize the potential for introducing additional biases.
Reproducibility. We have made comprehensive efforts to ensure that researchers can effec- tively reproduce the results presented in our paper: (1) Theoretical Foundations: All pertinent theoretical results, encompassing definitions, assumptions, and proofs, are comprehensively de- tailed in Appendix A (2) Prompts: The specific prompts we employed for generating exam- ples and counterfactuals are laid out in Appendix | (3) Code: The code available in https: http://///aaaraaaeeessaaeeeeeeea
The hyperparameters we used are described in
REFERENCES
Eldar David Abraham, Karel D' Oosterlinck, Amir Feder, Yair Ori Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. Cebab: Estimating the causal effects of real-world con- cepts on NLP model behavior. NeurlPS. 2022. URL http: nips. wwtpcc/pppppaaiaaeeee files/ http:///520002220000033333333360
html
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in Al Safety. arXiv:1606.06555 [cs], 2016. URL http:://axxiv httt:://arxtw.or//abbs..
Maria Antoniak and David Mimno. Bad seeds: Evaluating lexical methods for bias measurement. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pp. 1889-1904, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 1/tp://///222122 http///2222aalllllooo acl-llng.. 148. UtLLhttts//acccaatholloo wwt.:/accaaanooolooooorooooaoe org 2021 -
ac1-long. 148
Alexander Balke and Judea Pearl. Probabilistic evaluation of counterfactual queries. In Barbara Hayes-Roth and Richard E. Korf (. Proceedings of the 12th National Conference on Artificial Intelligence, Seattle, WA, USA, July 31 August 1994, Volume pp. 230-237, AAAI Press / The MIT Press, 1994. URL http: / www. aaai. org wttgtibrarg///////oo/ee/ AAAI 4atpaaa///////99//aaa/4444---330 0400a1199-333.300 php
Umberto Benedetto, Stuart J Head, Gianni D Angelini, and Eugene H Blackstone. Statistical primer: propensity score matching and its alternatives. European Journal of Cardio-Thoracic Surgery, 53 (6):1112-1117, 2018.
Alexander Binder, Grégoire Montavon, Sehastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, pp. 63-71. Springer, 2016.
Nitay Calderon, Eyal Ben-David, Amir Feder, and Roi Reichart. Docogen: Domain counterfactual generation for low resource domain adaptation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7727-7746, 2022.
Nitay Calderon, Subhabrata Mukherjee, Roi Reichart, and Amir Kantor. A systematic study of knowledge distillation for natural language generation with pseudo-target training. In Anna Rogers, Jordan L. Boyd-Graber, Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9- 14, 2023, pp. 14632-14400 14659, Association for Computational Linguistics, 2023a. doi: 10.18653/v 1/133/112222... 1atb3///ooo......oooooog.g long-ong..1111 818. URL httthhhtss://// org/10. 18653/v1/2022 acl long. 818
Nitay Calderon, Naveh Porat, Eyal Ben-David, Zorik Gekhman, Nadav Oved, and Roi Reichart. Measuring the robustness of natural language processing models to domain shifts. CoRR,
ftl//300.00166 2023b. doi: 1044850/ARXIV2236600110 URL (ttts:::::/// //dt:s::///...../ http:://oooooooooo/// www.lo.rg/1100000.. 48550/ arXiv. 2306 00168
10

---

Published as a conference paper at ICLR 2024
Carlos Cinelli, Andrew Forney, and Judea Pearl. A crash course in good and bad controls. Sociological Methods & Research, pp. 00491241221009555,, 2022.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? An analysis of BERT's s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Florence. Italy, August
Analyzing and Interpreting Neural Networks for NLP, pp. 2019. Association for Computational Linguistics. URL http:: www.:aalwweb..rgg... ort///ooooontggy//////ooooo W19-4828
Alexis Conneau, German Kruszewski, Guillaume Lample, Loie Barrault, and Marco Baroni. What you can cram into a single $&!# ! vector: Probing sentence embeddings for linguis- tic properties. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp- 2126-2136, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1//18-11 7/ wwwweb.wwb.b.om org/ wttp://annnooooo/1--11111
URL https:
Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. Framework for evaluating faithfulness of local explanations. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári. Gang Niu, and Sivan Sabato (eds.), International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 4794 4815. PMLR, 2022. httttttts///ccceeeeet/e httpsiiinw......eeessseseee pttpss//vw.//aasspppsoaaass
html
Ruofan Deng, Li Pan, and Chloé Clavel. Domain adaptation for stance detection towards unseen target social media. In 10th International Conference on Affective Computing and Intelligent Interaction, ACHI 2022, Nara, Japan, October 18-2 21, 2022, pp. 1-8. IEEE, 2022. doi: 10. 109/ ACI155700.2022.. 202.0.99222....58118
http:t /ttp://ooii.oo w0w. 1109/ACI155700 2022. 9953818
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/v (01553/vln199111 14213399991.... URL https://doi. https:///doooor//000/ 10. 11853/v1/n19--1444
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9: 16755, 2021.
Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. Causalm: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333- 2021a.
Amir Feder, Naday Oved, Uri Shalit, and Roi Reichart. CausaL.: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333-388 June 2021b. doi: 10.1162/coli a .04000 URL htttps:://alaatthotota//
Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood- Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. Trans. Assoc. Comput. Linguistics, 10: 138-1118,,, 2022. URL htt.ttttt///tansaaee www.oorojj....... ojs/
hi////idexxx..e www.iinwpeeiiiiieeeeeeepep.. hhtp://aac/aattce/ieeeeeee 4105
Amir Feder, Yoav Wald, Claudia Shi, Suchi Saria, and David M. Blei. Causal-structure driven augmentations for text OOD generalization. CoRR, abs/2310. 12803, 2023. doi: 10.48550/ARXIV.
2310.12803. URL https:://oi..rr//1. 48550/0.885555a55 www.xiii..111..1. 12803
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Papers). pp. 1828-1843 Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/022 long. 144. URL http://tpp///ddaantoooooooo////p wttp:/or///21.....ooooooo 144
Natural Language Processing (Volume 1: Long
11

---

Published as a conference paper at ICLR 2024
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi. Dheeru Dua, Yanai Elazar, Ananth Gottumukkala,, Nitish Gupta, Hannanch Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. Evaluating models' local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1307-1323, Online, November (/06533//22200.fnnding----/ 2020. 11220.fimme findings emnlp. 117
2020. Association for Computational Linguisties. doi: URL
org/2020.
Sahaj Garg, Vincent Perot, Nicole I Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 219-226, 2019.
Itai Gat, Nitay Calderon, Roi Reichart, and Tamir Hazan. A functional information perspective on model interpretation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 7266-7278. PMLR, 2022. h htttt:::///proceddnngs... mlr. htt..//rreee//1///a22222222
html
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models par- tially embed theories of lexical entailment and negation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NL.P, pp. 163-173, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10.18653/v 1t200//0002000000ooo.o/ 1.16. URL
org/ 2020. blackboxnlp-1.16
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Bryce Goodman and Seth R. Flaxman. European union regulations on algorithmic decision-making and a "right to explanation". A/ Mag., 38(3):50-57, 2017. doi: 1aa.1//9immaa..338..2222.2. URL
https://doi. org/ 10. 1609/a a imag. v3813. 2741
Yash Goyal, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace).
CoRR, 01119907.76666. 2019.
www.:iiiii.oo////bae.p otgg/bs/////////// 07165
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM Comput. Surv., 51(5): 93:1-93:42, 2019. doi: 10. 10.1145/3236009. URL httpt:t///oooorrrr org/10 0145/333222 3236009
Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein. dooss- label- adaptive stance detection. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punia Cana, Dominicun Republic, 7-11 November, 2021, pp. 9011 9011-9019200 1-9028. Association for Computational Linguistics, 2021. doi: 10. 111138661122222 mailp-main.775 7.00. URL
18653/v1/22888 http:///ww2222....mmmammaaaaaaaa 710
Momchil Hardalov, Amay Arora, Preslav Nakov, and Isabelle Augenstein. A survey stance detection for mis- and disinformation identification. In Marine Carpuat, Marie-Catherine de Marn- effe. and Iván Vladimir Meza Ruíz (eds.), Findings of the Association for Computational Lin- guistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 1259-1277. Asso- ciation for Computational Linguistics, 2022. doi: 1/2155/1/20022.fii maddings-naacl.944 URL findings-naacl.99
Pengfei Hong, Rishabh Bhardwaj, Navonil Majumder, Somak Aditya, and Soujanya Poria. Re- mask: A robust information-masskinn approach for domain counterfactual generation. CoRR, 0al//305502888, 2023. doi: 10448550/ARXI1.235550022 URL htthpsts:://do. or arXiv. 2305.3305000 02858
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4198 4205, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2000 acl- 38.. URL https: www.lannhooogg.orggg/aaa.
acl-main.386
12

---

Published as a conference paper at ICLR 2024
Rohan Jha, Charles Lovering, and Ellie Pavlick. Does data augmentation improve generalization in nip? arXiv preprint arXiv:2004.15012 2020.
Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. Learning the difference that makes difference with founterfictually--gggeen data. arXiv preprint 1909.1244, 2019.
Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. Learning the Difference that Makes
Difference with Counteractually-Augmmneed Data. arXiv: 1909, 12434 [cs, stat], February 2020a.
URL http:://axii wwt.:o////bbb/////// htt.:////arrvv..r/////////999 12434. arXiv: 1909.12434.
Divyansh Kaushik, Amrith Setlur, Eduard Hovy, and Zachary Lipton. Explaining the efficacy of Nounterfactually-auggnee
data. arXiv preprint arXiv:20100001111 2020b.
Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening new frontier for causality. CoRR, abs/2305.000550 2023. doi: 10.48550/arXiv. 2305.00050. URL http:p///oooooooo..///////////// 23w...x..x..3..20....0
Dilek Küçük and Fazli Can. Stance detection: A survey, ACM Comput. Surv., 53(1):12:1-12:37, 2021. doi: 10. 145/3369026. URL htt.:tttss///oorrggtgge 114533339015
Sören R Künzel, Jasjeet S Sekhon, Peter Bickel, and Bin Yu. Metalearners for estimating heteroge- neous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10):41564111 2019.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. URL http::///xaiv..o/t// wwt..orrg/bbs//9//// 11692.
Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch Towards faithful model explanation in NIP: A survey. CORR, abs/2209. 1326, 2022. doi: 10.488550/rxxw.2220). 11326.. URL
https://doi org/rog///// 10. 48550/arXiv.....m 2209. 11326
Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(44) 2020. ISSN 0027-8424. doi: 10.1073/pnas.
1907367 117. /tt.:ttsss//////..p...o/////////
http://wwppa...oom pwtpppss...ggpcooteee//////////
Chengzhi Mao, Augustine Cha, Amogh Gupta, Hao Wang, Junfeng Yang, and Carl Vondrick. Generative interventions for causal learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3947-3956, 2021.
Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiao-Dan Zhu, and Colin Cherry. Semeval- 2016 task 6: Detecting stance in tweets. In Steven Bethard, Daniel M. Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch (eds.), Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval @ NAACL-HLT 2016, San Diego CA, USA, June 16-17, 2016. pp. 31-41 41. The Association for Computer Linguistics, 2016. doi: 10.18653// 16653/v1/s16-1000 1003. URL http:::://doii.rrr/
18653/v1/s16-100
Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Chris- tiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. In NeurlPS, 2022. URI, /tt./http://ppperss.... cttps//cpprnppscccppppppepee cc/paper.pppppp.eeee files//pppee//22///aaaa
html
Judea Pearl. Causality. Cambridge university press, 2009.
Elconora Poeta, Gabriele Ciravegna, Eliana Pastor, Tania Cerquitelli, and Elena Baralis. Concept- based explainable artificial intelligence: A survey. CoRR, abs/2312. 2023. doi: 10.48550/ ARXIV.2312.12936, URI httptttpss//.ioo//// 4005/////a...////iii. 2312.
13

---

Published as a conference paper at ICLR 2024
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10653 2019. /RLt:ttt://arrtt httpp://aaaxiv.oggabbb///// 1910. 10683
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNIPP 2019, Hong Kong, China, November 3-7, 2019, pp. 3980-3990. Association for Computational Linguistics, 2019. doi: 10.18653/v 1tt/3////D111111 1410, URL httpp:://oo/ wtt../////....1/1//111 18653/v1/D19-11111
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Pp. 1135-1144, San Francisco California USA, August 2016. ACM. ISBN 978- 4232--- doi: 10.1145/9999672.99977. URL. https://dl. acm. org/ /oi/10. /1111/1/99922.299999999
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4902-4912, Online, July 2020. Association for
Computational Linguistics. doi: 10.18653/112200...o acl- www-main.n4440 442. URL https:::: w/wp://wwwww//.cttt aclweb
htt/aanhhoooyy/22299 acl-main..an...m 442
Parker Riley, Noah Constant, Mandy Guo, Girish Kumar, David Uthus, and Zarana Parekh. Textsettr: Label-free text style extraction and tunable targeted restyling. arXiv preprint arXiv:2010.03802, 2020.
Daniel Rosenberg, Itai Gat, Amir Feder, and Roi Reichart. Are vqa systems rad? measuring robustness to augmented data with focused interventions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 61-70, 2021.
Alexis Ross, Tongshuang Wu, Hao Peng, Matthew E Peters, and Matt Gardner. Tailor: Generating and perturbing text with semantic controls. arXiv preprint 1rXiv:2107.07150, 2021.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nat. Mach. Intell., 155):206-215, 2019. doi: 10.
s42256-019-0048-xx URL httpss:/s///oi.oomm org/10 1038/
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, Abs//190.01 108, 2019. URL http: org/abs/1910. 01108
Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 3257-3267, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi:
1329. URL htt.hhttss///wwt acl.eb web. or http//atooooo/nn---111 1329
Uri Shalit, Fredrik D. Johansson, and David A. Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICMI. 2017, Sydney, NSW, Australia, 6-1 August 2017, volume 70 of P Proceedings of Machine Learning Research, pp. 3076-3085. PMLR. 2017. htRL/http://prceeeeeee
mlr wtwppprrss/////aaeaaas
html
Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. FOIL it! find one mismatch between image and language caption. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 255-265, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. N..18863/11/11111
httL:httss:////
/ http://///caaaahoolglolgo htt.a//annnnoolllooo//////---000 org/P17-1024
Ilya Shpitser and Judea Pearl. Identification of conditional interventional distributions. In 22nd Conference on Uncertainty in Artificial Intelligence, UAI 2006, pp. 2006.
14

---

Published as a conference paper at ICLR 2024
Ilya Shpitser, Tyler J. VanderWeele, and James M. Robins. On the validity of covariate adjustment for estimating causal effects. In Peter Grünwald and Peter Spirtes (eds.), UAI 2010, Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence. Catalina Island, CA. USA, July 8-11, 2010, pp. 527-536. AUAI Press, 2010. URL https::/ddllpitt.org / displayArticlebetail. http::/wwiiaaaaa/aaaaaaaaeeaaae0
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pp. 3145- 3153. PMLR, 2017.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Ilsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neurul Information Processing Systems 2020, NeurIPS 2020, December 12, 2020, virtmal, 2020. URL http:tt://::/eeeedheeee/ hww.eeediiinnnnnniii..... cc/ cc/paper/2020/ aashiccca9bbeeaaaaaeeeeooocobbb abaeraabbbbbbb6eebbbbbbaattaaa
Paul Soulos, R. Thomas McCoy, Tal Linzen, and Paul Smolensky, Discovering the compositional structure of vector representations with role learning networks. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. pp. 238-254, Online, November 2020. Association for Computational Linguistics. doi: 10. 18683/111222200 blackboxnlp- 1.23, URL Sttpsttp::/ / http///aaannooloorrrr//ooo
blackboxnlp-1.22
Elizabeth A Stuart. Matching methods for causal inference: review and a look forward. Statistical science: a review journal of the Institute of Mathematical Statistics, 25(1):1, 2010.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics. URI Ihtpps: wwww.aclweb org/
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrerr Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie. Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, 014/2307.09288. 2023. doi: 10.48550\axX...330.02222 .92288. URL /ttp::::///oi..o///..../.4 arrii.. 2307 09288
Stijn Vansteelandt, Maarten Bekaert, and Gerda Claeskens. On model selection and model misspeci- fication in causal inference. Statistical methods in medical research, 21(1):7-30, 2012.
Christina Vichmann, Tilman Beck, Mareus Maurer, Oliver Quiring, and Iryna Gurevych. Investigating opinions on public policies digital media: Setting up a supervised machine learning tool for stance classification. Communication Methods and Measures, 1 150-::50-184 2023.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. Investigating gender bias in language models using causal media- tion analysis. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Bal- can, and Hsuan-Tiee Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Decem- ber 6-12, 2020, virtual, 2020. URL https: httptttpssse/pprooceiinn..ng neur wweeuripsccc//eee wttpscccpppe////e/ httpp/ccccppee////////aaa// 22200//h/2220/ahh
15

---

Published as a conference paper at ICLR 2024
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In IEEE Conference on Computer Vision and Pattem Recognition, CVPR 2021, virtual, June 19- 25, 2021, pp. 2495-2504. Computer Vision Foundation / IEEE. 2021. doi: 109/
com/ Sttpm//connent//P222222 CVPR46437.2021 00252. [RL_httpps//openaccess www.w:fcwo.ccooeettttttttotoccee
/html/
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. Polyjuice: Automated, general-purpose counterfactual generation. arXiv preprint arXiv:2101.00288. 2021.
Zhengxuan Wu, Karel D'Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts. Causal Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 37313-37334 PMLR, 2023a. URL https://proceeedings wwaiigss..r....r.. wtt.smm...rees//////rrssss htmmllmmmmllmmll
proxy models for concept-based model explanations.
Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D. Goodman. Interpretability at scale: Identifying causal mechanisms alpaca. CoRR, abs/2305.0880000 2023b. doi: 10.48550/ARXIV. 2305.08809. URL https: 201.00885500//a// 2305 20...003222000 08809
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014, pp. 818- 833, Cham, 2014. Springer International Publishing.
Xiaoling Zhou and Ou Wu. Implicit counterfactual data augmentation for deep neural networks. arXiv preprint arXiv:2304. 1..00.1340 13431, 2023.
Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. Counterfactual data augmenta- tion for mitigating gender stereotypes languages with rich morphology. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1651-1661, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10. 10.18653/v1//11-11166. URL
16

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A faithful explanation
18
B Inference time efficiency
21
Ablation study
21
New setup: explaining stance detection models
23
E
Further Discussion
24
F
Requirements and limitations
25
F.1 Requirements
25
R2
Complex causal graphs
27
Implementation details
28
G.1
Causal effect estimation pipeline
28
.2
Training procedure
29
G.3
Hyperparameters)
29
G.4
Explanation method evaluation pipeline
30
Finding the adjustment set
Prompts and examples
32
17

---

Published as a conference paper at ICLR 2024
A A FAITHFUL EXPLANATION
In this section, we prove our main theorem from $3.1 We start by providing definitions for explanation methods and then continue to define the order- fnttrllatthfulnnes property. For simplicity, assume the image of the explained models is
Definition 1 (Approximated Counterfactual), Given DGP G, explained model f. an example a whose treatment value is T(x) = i, and an intervention t + t', we call By approximated counterfactual if:
ff(x)
='
((+)
E
(xrr=)) +
otherwise
Where or=v is the golden CF of x in 9 and is an approximation error.
This definition also suggests why the Top-K K technique improves the causal effect estimation: Aver- aging the prediction of K approximated CFs reduces the variance of the approximation error, making the estimator more robust.
Definition 2 (Approximated Counterfactual Explanation Method), Given dataset Do sampled from a DGP 9, an explained model and an intervention T = I', the approximated counterfactual explanation method SCF is defined to be:
S,, =
Σ [(xx) f(xx)
D
ZEB
Definition 3 (Non-causal Explanation Method), Given dataset DD sampled from DGP G, explained model f and an intervention T 1', let D K,T,J] be defined be a set that contains for each unit from D triplet of model input, model output, and treatment assignment:
D, = {<*(u,, (x(u), T(u))) |u E DD}
An explanation method is called non-causal explanation method Sxc if is a function of D(X, S.X,STTH Axc(f.I,t,t) = (D(X,
For simplicity, we notate the triplet ( (u(), T(u))) with (ax) T(x)). According to Def. 3 the training data of SNg is DI meaning it can be, for example, an unbiased estimator of: Eg [f()][T = t'] - [f (x) = t). However, it may also overlook other concepts than the treatment, such as confounding concepts, which may hint why SNc potentially fails.
Definition 4 Fader-FFathfulnness). Given an i.i.d. dataset D sampled from a DGP G, an explained model and two interventions C1 61 + cg C2 C2 + Ez. an explanation method S order- faithful if:
[S C1, C1, (()) > En-g [S(, C2 02, c2)l + CaCE,(Ci,, GG) > CaCE,(C2, C2, cb) (8)
Lemma 1. For an explained model f. SCF is order-faithful for any DGP G and any interventions.
Proof. We start by connecting the approximated CF to the do operator. From Def. [ follows that the expected prediction of an approximated CF is equal to the interventional one:
Ba-glf(;r)) = (f + c] = Ex-gl [f (øT=E) ) = Eang
= t')]
Combining this result with the Def. and the fact that Ph Pg. we get:
Bo-e[Scp(f,T) = Enng
f(āv) f(())
DD]
DED
Exwelf (( s(+:)
Exngl [ )] Ex (#t))
Enng [f(x)) (T t')] - Een [fx)) = t)]
(TT
Thus. ScF is an unbiased estimator of CaCE,, and it holds that:
Ep-g(ScF(J (1,0,,6)) c Eg-g(Scr(J C2 C22 cb) Ca, CaCE CaCaaaCCeCCCaaC b5)
Meaning that Scr is order- faithful for any G and any interventions.
18

---

Published as a conference paper at ICLR 2024
Theorem 1. For explained model f. (1) The approximated CF explanation method ScE is order- faithful for every DGP G and two interventions; (2) For every DGP G. there exist a DGP g resulting from a small modification of 9, model f'. and two interventions C1 ic - G. C2 C2 C2 such that the non-causal explanation method SN is not order-faithful.
Proof. From Lemma we know that SCF is order-faithful for any DGP and (I) holds.
For part (2) of the theorem, let G be a DGP with at least two concepts Ci...., Cg (in case G has only one concept, we can add concept to the DGP and continue with the proof). If SNe is not order-faithful for some two interventions, set f' and we are done. Therefore, we assume that SN is order-faithful for all interventions in 9 and w.l.o.g. assume that:
(a) C1 has parent nodes (except exogenous variables). It is guaranteed that at least one concept has no parents since the DGP is a DAG.
(b) The causal effect of C₁ is bigger than of C2: (CI, 4) > CC2 02, C2). Otherwise, inverse the corresponding signs of the proof.
Proof sketch of the second part of the theorem: We will construct a new DGP g' by introducing new unobserved confounding concept Co (this is the "small modification of g" from the theorem) and new explained model f", such that:
Vi 1 (x,
= fa(a,
C Ci 44) CaCE,r(C2, (2,
(10)
Notice that the first condition Eq. ensures that SNr produces the same explanations also in the new DGP. This is because for an intervention concept C, SNC is function of DIX, C.J. and the joint distribution from which it is sampled, Pgr Par (a, (xx C,), is the same in the original DGP G.
Morcover, according to the second condition Eq. 10 for the new g' and f'. the causal effect of C₁ and C2 is reversed (compared to the original G and f). However, SNe produces the same explanations as before (for G and f), meaning it cannot order-faithful also for G' and f".
We next show that such G' and f' exist.
Construction: Let f be a new explained model and G' be a copy of 9 with a new node Co, and new edges (Co, C1), (Co, (') (i.e., Co is confounder). Co accepts three values: 0, 1 and 2. Let p = (g ci) and p' = Po(C = d)). Define the following probabilities:
(C = 0) =
Pg = 1) pp'
(g 2) = =1 -1/
(g = ci(Co 0) = Pg.(Ci = C = =1
(g = cı/Co = 2) = Pg.( = dIG, = =) =0
Ve # on (C = cCo = 2) = Pg(C1 c)
1 1-p-1 p'
We conclude that the marginal distribution of C₁ is equal in both DGPs, Pg-))) Po(C)):
(C = c1) PC = e₁ (C = 0) (G, = 0) = 1= Pg(C1 = =q)
(C = 6)) =P = C/ = 1)P (Co = 1) =p' Po(Ci = =()
Ve + Gi (C = =Pg (C1 = <CC P3 22)
=(1 - p - p') Po(Ci c)
1-p - p' -Pg(Ci= =c)
Let d be a difference in causal effects between C₁ and C2 for f. which is positive following assumption (b) from the beginning of the proof.
CaCE,(C1,,) (1) CaCE, 02)
19

---

Published as a conference paper at ICLR 2024
Let f' be the new explained model. f' has an oracle acces to the underlying concepts and is defined to be:
f((x) f(x) 2d (=))=)) + 2d [())(=)=0)
Where is the indicator function. (x then (i = c1 with probability 1 and:
f( f(z) 2d [[rrue] 2d [[rrue] f(z)..
Conversely, if Co(=) then C₁ (a) <4 with probability 1 and:
(((x) f()) - 2d [[False] + 2d [[Falee] f())
Finally, if Co(x:) 2 then (a + c1 di and:
(( f()) - 2d [[Fatee] + 2d +2d-I[Faled, f(x)
From the above equations, it is clear that in the DGP g' without any interventions, for all values of Co given a we have f((x) = f(x). Meaning that:
P& x) = Po. (x Ps(S(X))X)
We have shown that (C1 Ps(C1). Since G' is copy of g (except Co). and since neither the marginal distribution of the concepts nor the generation process of X did not change, then:
Vi = L,.., = 1, K :Po Px
Finally, from definition of f, it depends only on thus: Vi: Po((())) C.) = Po(f(X)(X) (the same holds for f', and we have shown above that Pg C, = Pf: Therefore:
Vi I,... K (())))
R
(XC/CC))/((CCC)))0CX//EX (X,
Eq. 9 holds and accordingly, Pa (o Po(D[X, J, Ci)), meaning that Sxe produces the same explanations for and g'. Since SNG order-faithful in G, and we know from our assumptions on G that (CC di) > CaCE, C2, c2), in follows then that:
Ea C1, 01 4) > Eowg (S(f' C2.02,
We next show that Eq. holds (reversed causal effect), thus Sxe cannot be order-faithful in g':
(C,
Ca, (1, 2d CaCE:( 2d CaCEr(co(r)=1]]
C,,E,(C1, 22d
(2)) - CaCE, 02, (2) d CaCE,(C2, 02, (6) CaCE, (C2, 02, d2)
CaCE,(Ci, e4) -
The last equation is true f is not affected by C2 in any way except through the
'In practice. it can be that the unobserved variable Co controls for some modification (x) and the path X f(X) is replaced in the new DGP g' by X t (6(X) - (XX and Co (XX). Nevertheless, Sye has only to realizations of X and and therefore, for simplicity of the notations
oracle.
20

---

Published as a conference paper at ICLR 2024
Table 3: Comparison between the inference time latency (in seconds) of different methods. The top rows present counterfactual generation models, and the middle and bottom tables for matching methods. Notice that all the baselines described in $4.2 have the same latency as the Causal Model and thus are not specified.
Explaining single example
Explaining 100 examplees
Method
Backbone
Top-K 1 Top-K = 10
Top-K = 100
Top-K =
Top-K = 10
Top-K = 100
Fine-tune Generative
T5-Base
0.84
1.03
3.09
102.2
308.2
Zero-shot Generative
ChatGPT (turbo)
2.45
2,95
4.47
245.3
295.3
447.2
Few-shot Generative
CharGPT (turbo)
2.52
2.98
4.49
252.3
298.3
449.2
250 candidates
Top-K =
Top-K = 10 Top-K = 100
Top-K =
Top-K = 10
Top-K = 100
Apprex
0.86
0.86
0,86
1.95
1.95
1.95
Causal Model
S- S-ransformer
0.03
0.03
0:27
0.27
0.27
1000 candidates
Top-K
Top- - 10
Top-K 100
Top-
Top-K
10
Top- K
100
Approx
0.86
0.86
0,86
1.95
1.95
1.95
Causal Model
S- S-Trasffomme
0.03
0.03
0.03
0.27
0.27
0,27
B INFERENCE TIME EFFICIENCY
As discussed in $ | we seek three attributes in the explanation method: It should be model-agnostic, effective, and efficient. In this subsection, we focus on inference efficiency, an aspect that is vital for providing real- time explanations and explaining vast amounts of data.
To compare the computational efficiency of various explanation methods, we present the inference time latency in Table 3 This latency represents the time (in seconds) required to compute the
I for either single example batch of 100 examples. We measure this latency for Top- K = 1, 10, 100 using the following methods: (1) Generative baselines; (2) The Approx baseline, which employs three fine-tuned RoBERTa models. We first use these models to predict the three confounder concepts of the query examples. Following this, we randomly select matches that correspond to the values of each query example; and (3) Our causal matching method, which first represents the query examples, then computes their cosine similarity with the candidates and finally finds the most similar matches. Notice that other matching baselines have the same latency as our method. All the query and candidate examples are 50 tokens in length, which is also the length of the gener-ged CFs. For the matching baselines, we utilized a candidate set of 1000 examples.
As can be seen in T the latency of the explanation methods we evaluated varies dramatically between them. Interestingly, Top-K does not influence the latency of the matching methods since it merely involves impact on latency. Our method also outpaces the Approx baseline, as it utilizes fewer models and avoids the need to filter the candidate set based on the values of the confounder concepts.
argmax operation. Similarly, the size of the candidate set has no noticeable
The generative baselines are significantly slower than the matching methods, primarily due to the autoregressive voken-by-token generation process. For instance, when explaining single query example, the fine-tuned model is around 30 times slower than matching methods for K = and 100 times slower for K 100. For batch processing (explaining 100 queries), the benefits of parallelism offered by GPUs make matching methods exceptionally fast. Our method is 1000 times faster than the generative baselines.
C ABLATION STUDY
This subsection aims to shed more light on the effect of each of our design choices. To this end, we examine variations or omissions of components from our causal representation model. Noteworthy, with the original candidate set, which is relatively small, all of the ablation models are competitive, and the performance difference is insignificant. Therefore, we also experiment with variations of the candidate set that reveal the pitfalls of cach ablation model. Table presents the performance when using three candidate sets: (I) The original candidate set, containing only original reviews from the matching split; (2) The original set augmented by the ground-truth human- written CFs; and (3) The original set augmented by the misspecified model-generated CFs.
We start by examining the backbone encoder model of the causal model: Traneffrrrae and RoBERTa. Comparing the first two rows of Table 4 we observe that both encoders are competitive and can be utilized as the backbone. We decided to use Transforerr because its development
21

---

Published as a conference paper at ICLR 2024
Table 4: Ablation study. Row Our model. Row 2:
With a different backbone. Row 3: Causal S- S-Transformer (ours) without filtering misspecified CFs from Xcr- Row 4: Without numan-annotated concepts (i.e., concepts are predicted by a zero-shot LLM). Rows 5-10: Different vari- ations of the objective (Eq. 5 when discarding components that include the specified sets. Columns present different matching candidate sets: original, original with ground- truth CFs, and original with misspecified CFs. Numbers are the mean over 5 explained models and 24 interven- tions. is better.
CFs
Miss. CFs Miss. Matches
Matches
Causal Model
w/o
Filter
w/o
Ms
Original Cos
+
GT CFs
Míss. CFs
norm
L2
Cos
Cos
norm
w/o
-CFs
1
Causal S-Trans
.65
.52
.38
.13
.09
.06
.70
.58
46
2
Causal Roberta
.52
.37
.16
.70
.59
3
w/o filtering
.66
.51
38
.12
.08
.05
.77
63
52
0.00
0.25
0.50
0.75
4
w/o labels
.52
.36
.13
.09
.67
.59
.43
Similarity
5
w/o Xer
.67
.54
.39
.26
.18
.13
..9
.58
.45
6
w/o Xie
.66
.51
.37
.09
.04
81
.68
577
Figure 4: Average similarity be- tween query examples and exam- ples from the four sets (the bars), plotted for different variants of the causal model (Y-axis).
7
X
.38
.06
833
.88
8
w/o X-M
.66
.51
.38
.66
.11
007
.70
.59
46
9
w/o Xa Li Xay
.66
.39
.19
.13
.09
.74
.61
10
w/o X--F
.38
.15
.09
722
.60
.99
loss was lower. By comparing the first row to the third, we examine the impact filtering unsuccessful or misspecified CFs from XeF. Without filtering, the causal model prioritizes misspecified CFs above valid matches (see Figure 4) and as result, when misspecified CFs are present in the candidate set, the performance degrades (third row, right column, gray cells).
In row 4, we investigate the applicability of our method without human-annotatee annotated concepts in the training dataset (i.e., completely unsupervised setup). Utilizing an LLM (ChatGPT), we predicted concept values in a zero-shot manner. Row 4 reveals that the performance remains consistent and on par with models trained on human-annotated datasets. Surprisingly, it performs better some metrics (although insignificant), likely because the LLM predicts annotations that sometimes missed due to disagreements between annotators.
We next examine the impact of discarding components from the model objective Eq 5 To this end, for each set of examples, Xcr, Xy, X-cF, and X.M, we train an encoder model while ignoring the three components from the objective that utilize examples from the discarded set. In addition, we also train two encoder models using only model-generated CFs (w/o Xx and X_n), and without any (w/o Xer and X i.e., training only with human-written reviews- not CFs).
The behavior of each ablation model as one would expect. For example, when Ker is discarded from the objective (row 5), the ablation model struggles to identify CFs and the performance declines when the matching sets include ground-truth CFs. Conversely, when X-Fr or Xx are excluded (rows 7, 8), the model fails to distinguish between CFs and misspecified ones, often favoring the latter over valid matches. Consequently, the performance drops when the candidate set contains misspecified CFs. Only our causal model consistently performs well across both candidate sets. Notably, we can see that even without model-generater CFs (row 10), the causal model performs well. Although the causal estimation less precise, achieving a good explainer without utilizing an L.M is possible.
Finally, Figure presents the average similarity between query examples and the four sets for the causal and selected ablative models. As can be seen, our causal model learns the desirable ranking order: X-M x-C+ Zx Tg.. In contrast, the model learns to favor misspecified CF's over matches when discarding Xy or X-cF or when not performing filtering of misspecified CFs from XeF- Although these ablation models have competitive performance to model and some outperform when the candidate set contains ground-trus CFs, they fail when the set contains misspecified CFs, highlighting the importance of learning the right ranking order.
22

---

Published as a conference paper at ICLR 2024
Table 5: Examples from the new Stance Detection setup. In the first stage of the dataset construction (top table), we split the original tweets into the train, dev, matching, and test sets. In the second stage (middle table), we keep 30% of the original tweets, and for the remaining 70%, we randomly assign values for the three writer's s concepts, In this example, we ask GPT-4 to edit the tweet "to make it sound as if a teenage female wrote it", In the third (bottom table), we generate CFs for the test set (and are used for benchmarking). In this example, we ask GPT-4 to change the Job concept to Farmer.
S1: Original tweet. need feminism because "what were you wearing" shouldn't it be a question when I tell my stepmom was cat called. #SemST Subject: Feminism
Age: ?
Gender: ?
Job :
Label: Favor
S2: Modified tweet. OMG, like, totally need feminism cuz. "what were you wearing?" shouldn't 't even be
a thing when I tell my stepmom some guy shouted stuff me. #JustSaying
Subject: Feminism
Age + Teen
Gender +
Job
Label Favor
S3: CF (test set). Gosh, reckon I'd holler for feminism, cause "what were ya wearin?" ought be a
question when tell ma about some fella yelled stuff at me while was out the fields. #lustSaying Subject: Feminism
Age: Teen Gender :9
Job + Farmer Label Favor
D NEW SETUP: EXPLAINING STANCE DETECTION MODELS
While we aim to validate our earlier conclusions across multiple benchmarks, CEBaB remains the sole high-quality benchmark for NLP explainers. On the other hand, we show that LLMs generate CFs that resemble human- CFs. Consequently, we harness LLM capabilities to emulate CEBaB and propose a new benchmark for model explanation, eliminating the high cost of manually annotating such benchmarks. We decided to focus Stance Detection, which is an important and well-studied NLP task that determines the position (support, oppose, neutral) expressed by the writer towards particular topic (e.g., abortions, climate change) (Küçüük & Can 2021 Hardalov et al. 2022 Viehmann et al. (023). This task is intriguing because understanding public opinions and beliefs can guide decision-making policy formulation, and marketing strategies. Estimating the causal effect of high-level concepts like gender or age may shed light on the societal biases and perceptions that drive social trends.
We rely on the stance detection dataset from the SemEval 2016 shared task (Mohammad et al. 2016). This dataset contains tweets about five subjects: Abortion, atheism. climate change, feminism, and Hillary Clinton. Humans annotated the stance of the tweets with three possible labels: support, oppose, and neutral. For simplicity of the new setup, we embrace the same causal graph of CEBaB (see Figure D. but replace the four review concepts with new relevant concepts: (1) Tweet's subject (specified above); (2) Writer's s age - teenage, elder or unknown; (3) Writer's s gender - female, male or unknown; and (4) Writer's 's Job- farmer, professor unknown. The four concepts have exogenous variable U as their parent and are the direct cause of the text X and the label (stance) Y, which is naturally affected by the four concepts. The explanation methods should estimate the effect of interventions: changing the value of the three writer's 's concepts (age, gender, job) to three values (we do not explain the tweet's subject concept).
Constructing the new setup. We follow three steps for constructing the new benchmark. See the examples in Table 5 In the first stage, we randomly split the data into four sets: train (NV 1000), matching (NV = 2250), dev (N = 250), and test (N = 500). Each subject is precisely one-fifth of each split. For each set and subject, we keep 30% of the original tweet and assign an 'unknown' value to the writer's 's concepts. For the remaining 70%, we randomly sample two of the three writer's s concepts and assign new values (e.g., male, unknown age, professor). Noteworthy, we do not uniformly sample the writer's profile since we want to introduce correlations between the concepts and the label. 7 presents the joint probabilities of different concept and label values.
In the second stage, we utilize GPT-4 to generate new tweets according to the new value assignment of 70% of the tweets. The prompts we used are described in Figure I0 In the third and final stage, we utilize GPT-4 to generate the ground-truth CFs for the test set, which enables calculating the Err for evaluating explamation methods. Accordingly, for each test example, we prompt GPT-4 and generate single CF for each of the six possible interventions (3 writer's concepts with two possible values that are different from the assignment). The prompt is provided in Figure TT In addition, we ask GPT-4 to predict the new label of the tweet (after the edit) and add this information the dataset.
23

---

Published as a conference paper at ICLR 2024
Table 6: Results for the new stance detection setup introduce in an out-of-distributioo scenario (for each test example, the candidate set compromise texts with different subjects). We explain two stance detection models: New RoBERTa fine-tuned on new stance labels extracted by GPT-4; and Original- DistilBERT fine-tuned with the stance labels of the original tweets. The description of the rows given the caption of T Numbers are the mean over 18 interventions, better.
K
1
K 10
New Labels
Original Cos
New Labels
Original
AVG
L2
Cos
ND
L2
ND
L2
Cos
ND
L2
Cos
ND
Causal Model (+GT)
.08
.16
.06
.10
.13
.08
0.10
Zero-shot Generative
.22
.74
.16
.36
.30
.19
.15
.34
.30
.39
Random Match
.57
.85
.45
.85
.77
.71
.46
.82
.36
.76
.73
.65
.66
Propensity
.44
.70
.34
.74
.64
.66
Approx PT RoBERTa
.43
.73
.34
.89
36
.76
.75
.63
.84
.31
.70
.73
.60
.62
PT S-Transfommer
.47
86
36
.77
.75
.64
.40
81
.30
.73
.73
.63
.62
Causal Model
.46
.84
.36
.71
.74
.60
.40
.80
.30
.64
.71
.58
.59
Experimental details. We interpret two stance detection models. The first one is RoBERTa fine-tuned on the new stance labels extracted by GPT-4. The second model is DistilBERT fine-tuned on the stance labels of the original tweets. Noteworthy, the causal effect of the writer's concepts more pronounced in the second model. This arises because we designed the dataset have large correlation between the original labels and the concepts, which the fine-tuned model captures. In contrast, when fine-tuning with the new labels produced by GPT-4, this causal effect becomes weak. This is mainly because GPT-4 tends to predict the neutral label far more frequently than its representation within the original labels (64.8% vs. 25.5%, refer to Table 7). thereby diminishing the correlation we initially established.
In contrast to CEBaB, our new setup enables testing of our causal matching method in an out-of- distribution scenario (Calderon et al., 2023b). A prevalent approach in the stance detection literature is assessing model robustness against distribution shifts arising from changes in the subject of the texts (Hardalov et al. | Deng et al. 2022). For instance, a model might be trained texts discussing abortion and then tested on texts about climate change. In our context, such a distribution shift might happen when the subject of the test samples is distinct from the subjects within the matching candidate set. Consequently, we conduct experiments to gauge the performance of our method under cireumstances where the distribution shift is evident (i.e., for each test example, the matching set compromises only candidates with different subjects).
For the CF generation approach, we employ ChatGPT (and not GPT-4, which generates the ground- truth CF's of the test set) and only with zero-shot prompts. We benchmark our matching method with the same baselines from $4.2 and the same hyperparameters and training procedure described in
Results. We report the results on Table 6 As can be seen, we reproduce our main findings from the previous section on the new benchmark. confirming the robustness of our conclusions, even in out-of-distributio settings. Specifically, the generative approach outperforms the matching methods (although consider that this setup, the ground-truth CFs are also model-generatedl causal representation model surpasses the other matching baseline and outperforms the generative model when the candidate set contains ground -truth CFs, Finally, the Top- technique universally improves all the explainers.
addition, the
E FURTHER DISCUSSION
We hope our theoretical and empirical results might broadly impact the field, and suggest these four following directions:
Explanation methods should be causal-inspired. Throughout this paper, our discussions as well as the theoretical and empirical findings, collectively underscore pivotal message: True interpretability inextricably linked with causality. As NLP systems become part of our daily lives and influence critical decision- decision-making processes in sectors (like law, healthcare, politics, and education), the
24

---

Published as a conference paper at ICLR 2024
imperative for faithful explanations has never been greater. This realization calls for paradigm shift towards exploring and developing more causal -ausal-inspired explanation methods.
Understanding when LLM-generated counterfactuals fail. Our empirical results demonstrate that the best method for explaining the causal effect of high-level concepts on model prediction is by employing multiple (Top-KK) LLM-generated CFs. However, this does not mean that the problem of interpretability has been solved. There is an impending need for the community to design more challenging benchmarks identify the areas where the LLMs fall short, misunderstand causality, and fail to provide correct CF explanations (for example, see Kiciman et al. (2023). Moreover, as we emphasize throughout the paper, utilizing an LLM at inference time is frequently impractical because of its prolonged latency, high financial costs, or privacy concerns, which leads us to the next point.
Closing the gap of efficient explanation methods. As the pursuit of understanding model behavior intensifies, it is paramount that we do not trade off efficiency for quality. The noticeable performance gap between the two approaches we introduced in the paper (the generative and the matching approaches) underscores ripe avenue for research. For example, the candidate set is crucial for the success of the matching approach, and given the right examples within this set, has the potential to surpass its generative counterpart. Therefore, an interesting line of work could explore techniques for enriching the candidate set. Bridging this gap could lead to methods that encapsulate the best of both worlds: high quality and operational efficiency.
Constructing new CF-based benchmarks. As we shift to causal -nssal- explanation methods, how evaluate them must evolve in tandem. Accordingly, it is imperative benchmark explanations against the true causal effect. High-quality CFs, which can serve as a proxy for ground truth in this context, are the basis for such evaluation, exemplified by the CEBaB benchmark (Abraham et al.) 2022). Nevertheless, the traditional approach for constructing such benchmarks relies on human experts and both financially costly, labor-intensive and fraught with inherent difficulties due to the cognitive demands of the task. Our work demonstrates that LLMs can effectively facilitate this process. We encourage the community to craft benchmarks that resonate with real-world scenarios. These LLM-guided benchmarks can serve as catalyzers for appraising efficient methods as we work towards bridging the existing performance gap.
F REQUIREMENTS AND LIMITATIONS
In this section, we discuss the setup requirements for estimating the causal effect according to our two approaches and the limitations these requirements may pose. In addition, we discuss and demonstrate the applicability of our approaches to more complex causal graphs and setups than CEBaB.
F.1 REQUIREMENTS
Causal graph. The first requirement for any causal estimation method is access to a causal graph that describes our causal beliefs, i.e., the concepts and the relationships between them. Notice that we do not assume access to the complete data-gen notta-generating process (DGP), which besides the causal graph, also describes the exact mathematical model that quantitatively defines the relationships and interactions among concepts and the exogenous variables. Furthermore, the causal effect it must be identifiable from the graph (for an example of a causal graph that is non-identiffable, see Figure in Cinelli et al. (2022)) ), and therefore, in this study, we discuss only identifiable causal graphs.
To estimate the causal effect given a causal graph, we first need to find the adjustment set (see the back-door criterion in section { Complementary, the causal graph also implies which variables should not be adjusted: Mediators (variables that are part of the causal pathway) and Colliders (variables that are caused by both the treatment and the outcome). Adjusting these variables can lead to incorrect estimations. In the case we are interested in the direct causal effect (and not the total effect) of the treatment variable, we can add to the adjustment set the mediators (but not colliders! e.g., Sore Throat in Figure F2
This fundamental causal concept of adjustment is modeled in our approaches in the following ways. In the generative approach, we specify the adjusted variables in the prompt and optionally also specify non-adjusted variables that may or may not be changed due the intervention. In the causal
25

---

Published as a conference paper at ICLR 2024
Disease
Disease
An original health query written by a patient describing symptoms.
EL
EC
ES
Over the past few days, I've noticed that my cough has become quite severe. It's persis tent and seems to be more intense than usual cough. Along with this, I've started experiencing sore throat. It's uncomfortable and makes swallowing bit painful. Another strange thing I've noticed is a change in my sense of taste. Foods don't 't taste the same as they used to; it's like the flavors are muted or just little off.
Lack of Taste
Sore Throat
Cough
Er
Text
f(Text)
Figure 5: The causal graph of our motivating case study of a patient-doctor interaction in an online health consultation. text-based health query (Texl, see example on the right) is analyzed by doctor with the assistance of NLP classifier that outputs diagnosis of possible discases. In this causal graph, three symptoms affect the patient's 's text, with a notable interrelationship between Cough and SoreThroat. The exogenous variables impacting the symptoms are EL, ET.
representation learning method, the adjusted variables are used for constructing the Xy and X sets, which are part of the optimization objective (see 55
Since the causal graph is typically derived by domain expert, specifying it might seem like limitation preventing generalization and automation of our explanation methods. However, notice that causal graph specification is requirement for accurate causal estimation (Shpitser & Pearl 2006 Feder et al. 2021a 2022), this also aligns with our theoretical result: Explanation methods that are unaware of the causal mechanism might be unfaithful. In some fields (such as healthcare), causal explanations are crucial, making the effort to construct a causal graph a worthwhile endeavour. Consider NLP model that recommends medical treatments based on symptoms described in the medical record. In that case, the clinician can not rely on correlational interpretations of the model recommendations. Finally, following future research and technological advances, it is reasonable assume that the reliance on human experts for concept and causal graph discovery will reduce, and this process will gradually become more and more automatic.
Annotated training dataset In the generative approach, a training dataset is not essential, except when performing few-shot prompting. In this case, small selection of text examples, interventions, and counterfactuals is required.
Conversely, the general requirement of any matching method is the availability of a candidate set from which the method selects match. The candidate set should contain an annotation of the treatment concept. This is because the method needs to select match annotated with the corresponding target intervention value; otherwise, it will fail to provide an accurate estimation. In our setup, the candidate set is randomly sampled from training set.
For training our causal representation method, we also need training set annotated with the adjusted variables. This set is used for: (1) Constructing the Xer and X sets by prompting an L.LM to generate CFs of examples from the training set; (2) Training concept predictors which are used for filtering misspecified CFs; (3) Constructing the Xx and X sets.
Although candidate set and training dataset with annotations are used in our experiments, believe they are not required and can be easily generated by an LLM. We demonstrate this approach in our ablation study (Appendix (C where we train matching method with concept annotations predicted by an LLM. According to the results, the performance remains consistent and on par with models trained human annotations.
26

---

Published as a conference paper at ICLR 2024
F.2 COMPLEX CAUSAL GRAPHS
Our choice of CEBaB was driven by its unique status as the only non-synthetic interventional dataset tailored for benchmarking concept-level explanation methods in NLP. In this subsection, discuss and elaborate on the applicability of our theorem and methods to other causal graphs beyond CEBaB.
First, our theorem is designed be flexible and does not rely a specific type of causal graph. Our proof is based on minor modification to the graph, adding confounder, which can be applied to graphs of any complexity. The key observation is that while non-causal methods may struggle with this change, causal methods remain faithful. Thus, our theorem holds regardless of the complexity of the causal graph as long the causal effect can be identified from the graph.
Second, our methods are also applicable to any identifiable causal graph. The identification require- ment is the only limitation of applying our methods. However, without further assumptions on the causal mechanism, it is also limitation of any causal estimation method. For an example of a causal graph that is non-identifiable see Figure in Cinelli et al. al.] (202)) Given causal graph, we can find the adjustment set (see the back door criterion in subsection HD. the mediators descendants which are not colliders and lie in the causal paths between the treatment and the text, and the colliders. These sets are the only deliverable of the causal graph that impact the modeling of our methods (both the generative and the causal matching approaches), and we describe how to use these sets in subsections $3. and !E.
We next provide proof of concepts that strong LLMs (such as GPT-4) can succeed in generating CFs for graphs that differ from CEBaB, especially those involving mediators. By incorporating mediator into the causal graph, we aim to explore situations where an intervention leads to changes in more than just the outcome text. Our case study is based on the causal graph illustrated in Figure 5 and involves an online health consultation between a patient and doctor. The patient submits a written health query, and the doctor then reviews this query, aided by NLP classifier that estimates the likelihood of various diseases. Based on an explanation method that quantifies the effect of each expliane symptom on the prediction of the disease, the doctor can continue her interaction with the patient and concentrate on specific symptoms for a more accurate diagnosis.
According the causal graph Figure 5 there are three symptoms: Lack of Taste, Cough, and Sore Throat. While Disease 1 causes Lack of Taste and Cough, Disease 2 causes a Sore Throat. However, any combination originating from the exogenous variables. In addition, intensive cough can also cause Hence, when Cough is the concept we explain. Lack of Taste a confounder, while Sore Throat is mediator. The original query of the patient is provided on the right side of Figure describing patient suffering from all three symptoms.
disease and symptoms possible due to the involvement of noise
throat.
In the examples below, show how cough symptom to estimate the direct causal effect (red box) or the total causal effect (green box).
utilize GPT-4 to generate CFs by intervening the
GPT-4 prompt for direct effect estimation: Generate counterfactual by making the patient's ' cough weaker. Keep the symptom of lack of taste and sore throat fixed.
Over the past few days, I've noticed a mild cough. It's not very persistent and seems less intense than a usual cough. Along with this, I've started experiencing sore throat. It's uncomfortable and makes swallowing bit painful. Another strange thing I've noticed is change in my sense of taste. Foods don't 't taste the same as they used to; it's like the flavors are muted or just little off.
Exl:
Ex2: Over the past few days, I've noticed a mild cough. It's intermittent and not as intense as a usual cough. Along with this, I've started experiencing sore throat. It's uncomfortable and makes swallowing a bit painful. Another strange thing I've noticed is a change in my sense of taste. Foods don't 't taste the same as they used to; it's like the flavors are muted or just little off.
27

---

Published as a conference paper at ICLR 2024
GPT-4 prompt for total effect estimation: Generate a counterfactual by making the patient's cough weaker. Keep the symptom of lack of taste fixed. Notice that cough could impact the sore throat.
Ex1: Over the past few days, I've noticed a mild cough. It's not very persistent and seems less intense than a usual cough. Despite the mildness of the cough, I've started experiencing sore throat. It's uncomfortable and makes swallowing a bit painful, though not severely so. What's more peculiar is a change in my sense of taste. Foods don't 't taste the same as they used to; it's like the flavors are muted or just little off.
Ex2: Over the past few days, I've noticed that my cough present but not particularly severe. It's more like a mild, occasional cough rather than persistent or intense one. Along with this, there's slight soreness my throat, but it's not too uncomfortable and doesn't 't cause much pain when swallowing, Another strange thing I've noticed is change in my sense of taste. Foods don't 't taste the same as they used to; it's like the flavors muted just little off.
To generate approximate CFss for estimating the direct causal effect (red box), we adjust for the Lack of Taste and the Sore Throat concepts since we are only interested in the direct path between Cough and the text. We do this by prompting GPT-4 to keep these variables fixed. The red box shows GPT-4 successfully generates CFs by modifying the text describing the cough symptom while leaving the remaining text fixed.
In the case of total effect estimation (green box), we should not adjust for the mediator concept Sore Throat. This is because change in Cough may may not cause change in Sore Throat (which also affects the text). We can achieve this by informing GPT-4 that a change in cough could also impact sore throat. As the CFs in the green box show, GPT-4 sometimes ignores a potential change in the sore throat symptom (Ex1) and sometimes modifies the relevant text (Ex2). This demonstrates that strong LLMs like GPT-4 can handle complex situations when an intervention not only directly affects the text but also may impact other concepts.
Notice, however, that relying on single CF might lead a biased causal effect estimation. There- fore, generating multiple CFs is crucial. This perfectly aligns with our theoretical definition of an approximated CF explanation method which considers an approximation error. Accordingly, utilizing multiple approximated CF for estimating the causal effect may lower the variance of the estimator and make it more robust. This also explains why this technique (of Top-K matching) universally improves the performance of any CF-based explanation method in study. Notice that in the case of mediators, LLMS may not model accurately the conditional distribution of the mediator given the intervention (e.g.. P(SoreT ort orraat Cough) in our causal graph). We leave this challenge to future research, although a solution might be reweighing the CFs according to this probability.
Finally, work underscores novel contribution in utilizing LLMs for generating CFs. As demonstrate, this can facilitate the creation of new interventional datasets representing complex causal graphs and significantly advance the research in causal explanations and benchmark construction.
G IMPLEMENTATION DETAILS
G.1 CAUSAL EFFECT ESTIMATION PIPELINE
This study focuses on black-box NLP model explanations by estimating the causal effect of high- level concepts on model prediction. The aim of this subsection is to provide additional details about the causal estimation pipeline performed according to the two approaches for counterfactual approximation proposed in Section $ The generative approach and the matching approach.
Given a query (treated) example w,, an intervention T : + and an approximated CF 27, we use the following estimator for estimating the individual concept effect on a classifier f:
(Mr, ICaCE, T. T,t,t") ICaCE (m., xT) [ f(xx)
28

---

Published as a conference paper at ICLR 2024
In addition, can provide a more robust estimator (see the third findings in our results $ by performing Top-K K matching:
ICaCE ( Tt,,t)
ICaCE, zt..)
K
After calculating the ICaCEy, can also estimate the CaCE using:
(T,
1 Σ ICaCE (.)) +
Σ (CaCEy(ar) ar) +
Σ (CaCEy(0l" xb()
We next describe the two approaches for counterfactual approximation.
The generative approach. Given a query example It. and an intervention T t', prompt the LLM to generate a CF. The prompt instructs the LLM to intervene on the text by changing the value of the concept T from t f'. For the Top-K technique we simply generate multiple CFs.
The matching approach. Given a query example æt, and an intervention 7 : --> l', the matching approach aim select match from the candidate D(T = ("), which contains subset of examples such that their treated concept T is equal to the target intervention value t'.
In this study, the selection of match performed by computing the similarity between the represen- tations of the query example and each of the candidate set. An exception are the Random matching and the Approx baselines. Accordingly, the matching method rank the candidates according their similarity and selects the most similar one:
mi (ax:) = arg max (1b(a))) ((x))) ED(=D))
The nigh-dimensional representation ((-) of each text is extracted by text encoder and is the mean vector of the last layer hidden states of the text's tokens. The similarity function s(-) used in our study is the cosine similarity:
<(x,) ((ax.)T(eee (xxe)
s (1b()))) ((()))
(11)
[(((;)) (Φ(())))
The matching approach proposed in this paper utilizes a causal representation model that trained using the procedure described below (Appendix G.2).
G.2 TRAINING PROCEDURE
We start with a small set of textual examples (the train set) labeled with the adjusting variables (e.g., review concepts such as food and service). Initially, employ small Encoder-only model (RoBERTa) and train a concept predictor for each variable. The predictor models are then harnessed construct the four sets: By predicting the concept values of each training example, we divide them (according to their concept values) and construct the Xy and X-M sets.
For constructing Xę= and X-CF, we use few-shot LLM (ChatGPT) generate approximate CFs and misspecified CFs (see prompts at Ş.. We filter out misspecified CFs from Xçr if the concept predictors indicate an adjusted variable was also changed during the treatment intervention. Additionally, we use simple rules to filter unsuccessful generations (e.g., empty strings, "As an Al LM...").
To train our language representation model (aimed to explain specific concept), we proceed as For every training example æz, we randomly sample four examples: &er E Xer, 2x E XN, xF-F E X The training goal is to minimize the objective of Eq. 5 We repeat this process for several epochs (15) and select the checkpoint that achieves the lowest loss on the development set.
follows:
G.3 HYPERPARAMETERS
We use F = 0.1, train the causal representation models for 12 epochs, and select the model checkpoint that minimizes the objective of Eq. (or its modified version for the ablation models). Notice that
29

---

Published as a conference paper at ICLR 2024
when using the Err the selection criterion, one might be able to select a more robust checkpoint. However, this violates the model-agnostic property, as calculating the Err requires access to the explained model.
The backbone Tranf of our model is MPNet (all-mpett-base-V (Song et al. 2020). We learning rate of 5e-6 for training the causal representation models. We use learning rate le-5 and a batch size 16 for the fine-tuned baselines, explained models and concept predictors. The concept predictor models are used for filtering misspecified CFs (for constructing the X and X-c sets) and for the Approx and Propensity score baselines. We fine-tune RoBERTa model using the annotated train set for each of the four concepts.
For generating counterfactuals, we use the OpenAI API with the following models: gpt-3. 5.ttrro (referred as ChatGPT in the paper) and gpt-4 (GPT-4).
G.4 EXPLANATION METHOD EVALUATION PIPELINE
high-level explanation method that provides an estimator of the ICaCE can be evaluated using benchmarks like CEBaB, i.e., interventional dataset consisting of examples, interventions, and a corresponding human-written ground-truth CF for each example and intervention. For given example z; and an intervention T': - t', using the example and its corresponding ground-trute CF Tt. we can estimate the golden individual causal effect:
ICaCE,( -.) (())) f(x)))
For the same example and intervention, the explanation method estimates the individual causal effect y² In the case of CF-based explanation method, the estimation is computed with the approximated CF: y = ICaCE (Ze, Txp) this study, the approximated CF is an LLM-ggnnrated CF or match.
Given the two estimations, // and ym we then calculate the distance between the two using three metrics introduced in Abbraham al. (202) L2 distance, Cosine distance, and norm difference.
L2(1" ) = =Ily" y ll2
(12)
Cos(y" "os(y","") ,y²) =1 (u")"""" n
(13)
|
ND(y "² = |Ile 1|2 y [|
(14)
Finally, the distance is plugged into Eq to provide the estimation error of the explanation method:
m,, = [D[T t t')l
Σ (ED
Dist
ICaaEE (x,, 2t),, ICaCE, (x.,, x."" ",,))
H FINDING THE ADJUSTMENT SET
Causal paths. The following three triplets (or X, Y and Z) the main patterns of causal graph:
Z -+ Y. Conditioning on Z blocks the flow from X to Y. Forks (or common causes, confounders): X < Z Y. Conditioning on Z blocks the flow from X to Y.
Chains (or mediators):
Colliders (or common effects): X Z + Y. The flow from to is blocked by default. However, conditioning on Z opens the flow and induces an association between X and Y.
Back-door criterion (Pearl 2009). A set of variables Z satisfies the back-door criterion relative to (X, Y) in a directed acyclic graph G if:
No node in Z is descendant of X.
Z blocks every path between X and Y that contains into X.
30

---

Published as a conference paper at ICLR 2024
1.2
ICaCE:
Kact-nornddf
(CaCE-L2
1.0-
0.6
0.4
0.2
0.0
Bandomn
Apprax
CONEX
S-eammer
TDAY
ConcaptSHad
DRLD
Figure 6: A comparison between different explanation methods adapted from Abraham et al. 2022). The best performing methods are the simple Approx baseline and the S-learner.
The adjustment criterion (Shpitser et al. 2010) was later devised to handle cases in which Z may explicitly contain descendants X; however, it is unnecessary for our causal graph described in Figure We term the set Z, which satisfies the back-door criterion as the adjustment set.
The adjustment set CEBaB: In our causal graph (Figure D.. when estimating the causal effect of an aspect, for example, S, the model f. the following paths should be taken into account (F and N w.l.o.g):
S f(X)
F + x
- (xx)
S F +N + f())
(when
fX)
(when Y + +Y+ f(x)
The adjustment set is F, N, A, and Y must not be adjusted. Accordingly; the set of matches X₂ should contain texts with the same aspect values (excluding the treatment) as the query (treated) example. Moreover, it clarifies why we use the term misspecified to describe the sets X and X This is because they contain texts that least one of their aspect values is different from the query example or was changed to a different one.
Increased precision when conditioning on V. The exogenous variable V is direct cause of the text that mediates between concepts (e.g., S) and the model prediction f(X). For example, the variable V can account for the syntax, writing style, length of X. Controlling for V (hypothetically. since it is not observed) will not bias the causal effect estimation (of CaCE, and not of ICaCE, which by definition should control for V) since does not open any back-door paths from S to On the other hand, controlling for V can increase the estimation precision of CaCE (Cinelli et al. 2022)
Consider the case where the model learns spurious correlations between V and Y (e.g.. the sentiment of long texts tends to be negative). We use the Approx matching technique (which controls all the adjusted variables: F, N, and and sample unit from the control group which shares the same adjustment values as the query example) to calculate the čaCE, (see Eq Since V is independent of any other variable (except X), the CaCE, estimator is not biased asymptotically. However, when the candidate sets are finite (and small), the non coorreeresnntig sample might amplify the spurious correlations learns. Controlling for V (finding matches that also share the same attributes, e.g., writing style) mitigates this bias and increases the precision of the estimation.
Therefore, utilizing counterfactuals while training the causal representation model used for matching vital. In contrast to Xn, the set of counterfactuals Xer should share the same values of V, and the causal model learns to consider it when finding match.
31

---

Published as a conference paper at ICLR 2024
I
PROMPTS AND EXAMPLES
Rate the restaurant scale from to
review:
[TEXT]
rating:
Figure The prompt we use for rating CEBaB reviews and extracting the five-way sentiment distribution (of the Llamas models). We extract the next token probabilities of 1-5 tokens. [TEXT] is a review.
I'm providing a review from Open Table and your task edit the review according to my instruction.
---- Input Review
[TEXT]
Instruction
The reviewer gave a POSITIVE score to the FOOD aspect. You should edit the review such that there NEGATIVE score the FOOD aspect.
We consider four aspects FOOD, AMBIANCE, SERVICE, NOISE, make you change ONLY the rating FOOD.
Edited Review
The zero-shot prompt we use for generating CFs for CEBaB examples. [TEXT] is the review. The instruction is changed according the treatment (e.g., in this prompt, change the value of the food aspect from positive to negative.
Figure 8:
Table 7: Fractions in percentages of the variables (joint probability) in the new setup (Stance Detection,
SD. For example, according to row CC and column T, 7.3% of the data are tweets about Climate Change (Subject concept) written by teenager (Age concept). There are total of 4000 examples divided into the train (N = 1000), matching (N 2250), dev (N = 250), and test (N 500) sets. Orig. Label the label the original tweet before any modifications, and Label the label predicted by GPT-4 after modifying the tweet. Legend: Subject Ab (A (Abortions). At (Atheism), CC (Climate Change) Fe (Feminism) HC (Hillary Clinton); Age (Unknown), (Teenager), E (Elder); Job ? (Unknown), & (Female), &' (Male); Job- (Unknown), F (Farmer). P (Professor); Label (Neutral), X (Oppose), (Support); Orig, Label ? (Neutral), (Oppose), v (Support).
Subject CC
Age
Gender
Label
Orig. Label
Ab
At
Fe
HC
E
?
Ab
20.0
0.0
0.0
0.0
3.6
9.1
10.1
3.8
6.2
13.4
5,2
1.4
4,9
11.6
3.5
0.0
20.0
5.4
7.0
9.7
4.2
6.0
13.1
5.6
1.3
4.0
12.4
CC
0.0
0.0
20.0
0.0
0.0
8.7
7.3
4.0
9.5
6.2
4.3
8.4
4.0
7.6
12.5
1.0
6.5
7.6
0.8
11.5
HC
0.0
0.0
0.0
0.0
20.0
8.2
4.5
7.3
9.4
4.1
6.5
8.0
8.4
3.6
12.7
5.3
2.0
5.5
11.2
3.3
?
7.2
7.7
8.7
41.0
0.0
0.0
11.2
13.0
14.6
28.2
6,5
6.3
18.9
11.1
T
3.6
5.4
7.3
5.4
4.5
0.0
26.2
0.0
14.9
6.2
5.2
16.6
4.2
16.5
2.6
7.1
8.7
4,9
12.6
9.1
7.0
4.0
5,4
7.3
0.0
0.0
32.8
20.9
3.6
8.2
18.2
10.4
20.1
9.9
2.8
5.8
23.0
4.0
10.1
9.7
9.5
8.3
11.2
14.9
20.9
47.0
0.0
0.0
11.6
21.2
14.1
30.4
9.8
6.8
11.4
23.3
12.3
3.8
4.2
6.2
4.6
4.1
13.0
6.2
3.6
0.0
22.9
0.0
14.4
3.4
5.1
14.0
2.6
6.3
6.1
6.4
10.4
?
8.6
9.7
8.44
8.4
8.0
8.44
16.6
18.2
11.6
14.4
17.2
43.2
0.0
0.0
26.0
9.1
8.2
11.4
19.0
12.7
4.1
3.8
7.6
4.9
3.6
14.6
5.4
4.1
14.1
5.1
4.8
0.0
0.0
24.0
16.8
1.6
5.6
5.5
6.0
12.5
13.4
13.1
12.5
13.1
12.7
28.2
16.5
20.1
30.4
14.0
20.6
26.0
22.1
16.8
64.8
0.0
0.0
24.3
27.2
13.3
5,2
5.6
1.0
1.8
5.3
6.5
2.6
9.9
9.8
2.6
6.5
9.1
8.2
1.6
0.0
19.0
0.0
0.5
17.4
1.1
1.4
1.3
6.5
5.0
6.3
7.1
2.8
6.8
6.3
3.1
8.2
2.5
5.6
0.0
0.0
16.2
0.7
2.2
13.3
4.9
4.0
11.0
8.7
5.8
11.4
6.1
8.0
11.4
24.3
0.5
0.7
25.5
0.0
11.6
12.4
0.8
10.8
11.2
18.9
4.9
23.0
23,3
6.4
17.2
19.0
21.8
6.0
27.2
17.4
2.2
0.0
46.8
0.0
32

---

Published as a conference paper at ICLR 2024
N
I'm providing review from Open and task edit the review according instruction.
Here few examples:
Example
Review
(TEXT)
Instruction
The reviewer POSITIVE score the FOOD aspect. You should edit the review such that there is NEGATIVE score the FOOD aspect.
We consider four aspects FOOD, AMBIANCE, SERVICE, NOISE, make sure change ONLY the rating FOOD. Edited Review
[COUNTERFACTUAL]
Example 2
..... Review
[TEXT]
Instruction -....
The reviewer gave NEGATIVE score to the FOOD aspect. You should edit the review such that there is NO INFORMATION about the food aspect.
We consider four aspects FOOD, AMBIANCE, SERVICE, NOISE, make sure you change ONLY the rating of FOOD. Edited Review
[COUNTERFACTUAL]
Example 3 ---
Review
[TEXT]
Instruction .....
The reviewer geve & POSITIVE score to the FOOD aspect. You should edit the review such that there is NEGATIVE score to the FOOD aspect.
We consider four aspects FOOD, AMBIANCE, MEBBANCE,SERVICE, NOISE, make sure you change ONLY the rating FOOD. Edited Review
(COUNTERFACTO
--- Input Review
[TEXT]
Instruction .....
The reviewer gave POSITIVE score the FOOD aspect. You should edit the review such that there is NEGATIVE score the FOOD aspect.
We consider four aspects FOOD, AMBIANCE, SERVICE, NOISE, make sure you change ONLY the rating FOOD.
Figure 9: The few-shot prompt for generating CFs for CEBaB examples. [TEXT] is review and [COUNTERFACTUAL] is a numan-written CF Every prompt consists of three demonstrations where the aspect of the treatment changed. Since there are four aspects, only 12 human-written CFs required.
NI
The following tweet discuss (DOMAIN). Please edit the tweet to make sound as if [DESCRIPTION] Then, specify tweet favor, oppose, Respond only with this JSON format: '("now_tweet'. X, "label": Y)" where Y must be "favor", "oppose" or "neutral".
about [DOMAIN].
Tweet: (TEXT]
Figure 10: The prompt we use for generating examples for the new stance detection setup. Notice these examples are used for enriching the dataset with more writer profiles. [DOMAIN] is the tweet topic/subject (e-g., abortions), and [TEXT]
original tweet which we ask the LLM (GPT-4) to edit. The [DESCRIPTION] describes the new profile of the writer. For example, it can be: "an elder (60+) farmer wrote it". In addition, we ask the I.LM to predict the label of the new tweet.
33

---

Published as a conference paper at ICLR 2024
NI
The following tweet may discuss [DOMAIN]. The writer's profile is (DESCRIPTION]
DESCRIPTION).
Then, specify the new tweet favor, oppose, or remain neutral about [DOMAIN).
Respond only with this JSON format: ("new_tweet':X tweet":X, "label": Y) where must "favor", "oppose" "neutral".
Tweet: [TEXT]
for generating ground-truth CFs for examples from the stance detection setup. [DOMAIN] is the tweet topic/subject (e.g., abortions), and [TEXT] is the original tweet which we ask the LLM (GPT-4) to edit. The [DESCRIPTION] describes the old profile of the writer (e.g., "Age: Unknown, Gender: Female, Job: professor"). The [EDIT DESCRIPTION] describes the new profile, for example, "an elder (60+) farmer wrote it" or "the age, gender, or job of the writer is unknown". In addition, we ask the LLM to predict the label of the new tweet.
Figure 11: The prompt
The following tweet may discuss [DOMAIN]. The writer's profile is [DESCRIPTION] Please edit the tweet make sound [EDIT_ DESCRRTDESCI,,..NN. Respond only with the revised text.
Tweet: [TEXT]
Figure 12: The zero-shot prompt we use for generating CFs for examples from the stance detection setup. This prompt is used for benchmarking the LLM (ChatGPT), and not for generating instances for the test set. See Figure caption for more details.
34


Reviews:
[{"review_id": "157nx4HIZf", "reviewer": "Reviewer_9xHf", "summary": "The paper proposes two methods for approximating Counterfactuals in a model-agnostic way. The first one is to utilize an LLM to change attributes during inference time. The second method is to find Counterfactuals through efficient matching. In order to allow efficient matching, the paper developed a novel language representation learning method specifically for encoding counterfactuals. Such representation is learned through contrastive loss that maximizes the similarity of approximate counterfactuals and minimizes similarities of misspecified Counterfactuals. Both methods achieve better performances than prior works. The paper also released a dataset for evaluating NLP explanation techniques.", "questions": "1. If the matching candidate set doesn't exist, do you generate them given the concepts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper proposed an efficient and novel matching technique for finding Counterfactuals and provided strong theoretical and practical evidence that Counterfactuals are good explanations. \n2. Counterfactuals generated using this method are more order-faithful and comprehensive than prior work.\n3. Detailed ablation study to demonstrate the effect of each component in the method.", "weaknesses": "1. The results are only on one dataset CEBaB. The experimental section would be more convincing if more experiments were done on a wider range of datasets. \n2. The concepts are pre-defined, which can be a limiting factor to the comprehensiveness of the Counterfactuals generated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rsh1QVbsBs", "reviewer": "Reviewer_32rU", "summary": "This paper investigates the use of large language models (LLMs) for creating counterfactual examples to explain NLP classification model predictions. Two methods are proposed. The first involves directly using LLMs to generate counterfactual examples, altering only one aspect/concept of the input while maintaining the rest. The second method entails a matching process to discover approximate counterfactuals from a pre-defined candidate set. The study reveals that the matching approach, utilizing a specially trained feature extractor, outperforms strategies using pre-trained LMs as feature extractors and other baselines in the CEBaB benchmark.\n\n------------------------------------\nUpdate after discussion with authors:\n\nThe discussion with the authors and the information provided in the updated manuscript made me more convinced of the applicability of the proposed methods, so I raised my score to 6.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is overall well-written, with a well-defined research question;\n\n\n2. In addition to direct counterfactual generation via the LLM, the authors introduce an efficient, matching-based approach for identifying approximate counterfactuals from a pre-defined candidate set. Though this method doesn't perform as well as the direct LLM generation, it outperforms past baseline methods and is considerably more efficient than employing the LLM directly for each instance. The exploration of how to efficiently generate counterfactual examples using LLMs is an intriguing aspect of the paper.", "weaknesses": "1.\t\n\nThe paper mostly follows the setting of the work of CEBaB, including causal analysis, the approximated counterfactual method, and the evaluation. While some theoretical analysis is provided in Section 3.1, it mainly argues why the approximated counterfactual method which is initially proposed in the CEBaB is better than others. Underlining this, the first concern is that the paper's contribution appears to be limited to the proposal of two LLM-based approximated counterfactual methods that perform better in CEBaB's causal framework. Given the powerful ability of LLM, using it can better generate counterfactual examples (that only change one concept of the input while keep other aspects unchanged) is not very surprising.\n \nThe second concern is the limited applicability of the proposed matching method. The use of the matching method under the CEBaB setting requires pre-defined or pre-identified concepts/factors, such as Food (F), Service (S), Ambiance (A), and Noise (N) in restaurant reviews. However, these concepts may not always be available or readily identified in many real-world NLP scenarios. Given that the paper exclusively focuses on using LLMs to generate counterfactual examples in this particular setting, its broader applicability and contribution are questioned.\n\n2.\t\n\nAs indicated by the results in Table 2, the matching method (causal model) demonstrates only a slight improvement over the Approx baseline, especially when K=10. While the matching method is the most novel part of the paper IMO, the fact that its performance isn't significantly superior to the Approx baseline brings its practical importance into question.\n\nIn summary, the marginal performance improvement and the limited applicability of the \nproposed methods make me tend to reject the paper at this moment. \n\nHowever, I am open to further discussions and potential rebuttals from the authors that may address these concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FiPOyDn5uD", "reviewer": "Reviewer_wvZe", "summary": "This paper presents two methods for explaining the predictions of Natural Language Processing (NLP) models, focusing on the use of counterfactual approximations (CFs). The first is a Counterfactual Generation approach, where a large language model (LLM) is prompted to change a specific text concept while keeping others the same. The second is a Matching method that identifies text with similar properties within a dataset. The authors establish the value of approximating CFs for offering _faithful_ explanations and illustrate their techniques' applicability on several models. They further improve the ability to provide explanations using top-K matching. Furthermore, they highlight the potential of LLMs to create new benchmarks for NLP model explanations. The authors present theoretical and empirical evidence to support their research and propose further areas of study.", "questions": "I would appreciate if the authors discuss the issues I brought up in the previous section.\n\nAdditionally, could you explain the difference between Random Match and Approx again? It's not that clear from the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper contributes to the field of NLP model interpretability by introducing two practical methods for model-agnostic explanations, which could improve our understanding of model predictions.\n- The authors back their theoretical constructs with extensive experimental results, though the reliability of these methods depends on the specific conditions in which they are applied (such as access to a candidate set that offers good matching candidates).\n- The concept of Order-faithfulness is an innovative criterion for explanation methods, potentially providing valuable insights into the relative impact of different concepts on model predictions, although it would need to be tested across various contexts and model types to ensure its broad applicability.", "weaknesses": "- The first method proposed, Counterfactual Generation, is computationally expensive and may be infeasible in scenarios requiring real-time explanations.\n- Although Matching is faster than CF Generation, it might not be as accurate for all situations, especially when the matching candidate set does not sufficiently represent the input data. It would be great if the authors performed some ablation studies (reducing the quality of the matches in the candidate sets to show how much the performance degrades).\n- It's unclear how these techniques would perform on models trained on very niche tasks, which could inherently limit the possible counterfactuals, especially where such attributes may be hard to define beforehand. Both counterfactual generation and matching approaches assume that we have a set of attributes for which we wish to examine whether a model is paying attention to those. However, how does this approach work for open ended tasks (which is where LLMs are primarily being used) or tasks with a large number of classes, where generating counterfactuals (or finding matches) may be inherently difficult?\n- It's unclear how these methods would handle situations with multilayered complexities, such as nested counterfactuals, where counterfactual changes to one concept might trigger changes to other related concepts. The paper also does not extensively address scenarios where counterfactual approximations could result in impossibilities or logical contradictions, potentially limiting the breadth of their application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M0Nvtt7pZY", "reviewer": "Reviewer_GEWc", "summary": "This paper focuses on improving causal explanations, specifically the counterfactual explanation. The authors propose to leverage LLMs to generate the counterfactual input corpus, use the generated corpus to train a counterfactual representation model, and match the input and its corresponding counterfactual representation to generate the causal explanations. Experiment results show that the authors' method outperforms all previous matching baselines, representing a promising explanation ability.", "questions": "1. I cannot find an accurate definition of \"causal model\". Does the author use the representation generated by a language representation model optimized with Eq.(5) with generated counterfactual and matched examples, then calculate the matching value in Eq.(4), and use this value to calculate Eq.(2) as a causal model?\n\n2. How does the generative approach work? If (1) is true, does the causal model include the generative approach?\n\n3. There are two versions of ChatGPT (GPT-3.5 and GPT-4); which one did the authors use in the experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper provides detailed proofs and descriptions of the proposed method.\n2. The experiment results are reliable with the comparison between various baselines and models.\n3. The authors also construct a benchmark based on the findings of LLM's ability to generate counterfactual examples, which I think is a good contribution to the XAI community.", "weaknesses": "The description of the proposed method in Section 3 is confusing and not easy to understand. I think the authors should rephrase Section 3 with a general description of the proposed causal model. \n\nAbout Eq.(2), the authors use the difference in the model's prediction before and after the treatment as the treatment effect, which, in my opinion, is not robust when the model's output confidence is flat (or the uncertainty is high). This will affect the method's performance on small models like BERT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)