Causal Proxy Models For Concept-Based Model Explanations
Authors: Zhengxuan Wu
Karel D'Oosterlinck
Atticus Geiger
Amir Zur
Christopher Potts
Venue: Submitted to ICLR 2023 (2023)
Keywords: Explainability
Causality
Concept-Based Explanations
Causal Explanations

Abstract: Explainability methods for NLP systems encounter a version of the fundamental problem of causal inference: for a given ground-truth input text, we never truly observe the counterfactual texts necessary for isolating the causal effects of model representations on outputs. In response, many explainability methods make no use of counterfactual texts, assuming they will be unavailable. In this paper, we show that robust causal explainability methods can be created using approximate counterfactuals, which can be written by humans to approximate a specific counterfactual or simply sampled using metadata-guided heuristics. The core of our proposal is the Causal Proxy Model (CPM). A CPM explains a black-box model $\mathcal{N}$ because it is trained to have the same \emph{actual} input/output behavior as $\mathcal{N}$ while creating neural representations that can be intervened upon to simulate the \emph{counterfactual} input/output behavior of $\mathcal{N}$.  Furthermore, we show that the best CPM for $\mathcal{N}$ performs comparably to $\mathcal{N}$ in making factual predictions, which means that the CPM can simply replace $\mathcal{N}$, leading to more explainable deployed models.

Manuscript
CAUSAL PROXY MODELS FOR CONCEPT-BASED MODEL EXPLANATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Explainability methods for NLP systems encounter a version of the fundamental problem of causal inference: for given ground-truth input text, we never truly observe the counterfactual texts necessary for isolating the causal effects of model representations on outputs. response, many explainability methods make no use of counterfactual texts, assuming they will be unavailable. In this paper, we show that robust causal explainability methods can be created using approximate counter- factuals, which can be written by humans to approximate specific counterfactual or simply sampled using retadata-guided heuristics. The core of our proposal is the Causal Proxy Model (CPM). A CPM explains black-box model N because it is trained to have the same actual input/output behavior as N while creating neural representations that can be intervened upon simulate the counterfactual input/output behavior of A.. Furthermore, we show that the best CPM for A performs comparably to N in making factual predictions, which means that the CPM can simply replace N,, leading to more explainable deployed models.
INTRODUCTION
The gold standard for explanation methods in AI should to elucidate the causal role that a model's representations play in its overall behavior to truly explain why the model makes the predictions it does. Causal explanation methods seek to do this by resolving the counterfactual question of what the model would do if input X were changed to a relevant counterfactual version X'. Unfortunately, even though neural networks are fully observed, deterministic systems, we still encounter the fundamental problem of causal inference (Holland, 1986): for given ground-truth input X, we never observe the counterfactual inputs X' necessary for isolating the causal effects of model representations on outputs. The issue is especially pressing in domains where is hard to synthesize approximate counterfactuals. In response to this, explanation methods typically do not explicitly train on counterfactuals at all.
In this paper, we show that robust explanation methods for NLP models can be obtained using texts approximating true counterfactuals. The heart of our proposal is the Causal Proxy Model (CPM). CPMs are trained mimic both the factual and counterfactual behavior of a black-box model A. We explore two different methods for training such explainers. These methods share a distillation-style objective that pushes them mimic the factual behavior of A, but they differ in their counterfactual objectives. The input-based method CPMIN appends to the factual input new token associated with the counterfactual concept value. The hidden-state method CPMHI employs the Interchange Intervention Training (IIT) method of Geiger et al. (2022) to localize information about the target concept specific hidden states. Figure provides a high-level overview.
We evaluate these methods on the CEBaB benchmark for causal explanation methods (Abraham al., 2022), which provides large numbers of original examples (restaurant reviews) with human- created counterfactuals for specific concepts (e.g., service quality), with all the texts labeled for their concept-level and text-level sentiment. We consider two types of approximate counterfactuals derived from CEBaB: texts written by humans approximate specific counterfactual, and texts sampled using metadata-guie -uudedd heuristics. Both approximate counterfactual strategies lead to state-of-the-nn performance on CEBaB for both CPMIN and CPM
We additionally identify two other benefits of using CPMs to explain models. First, both CPMIN and CPMμ have factual performance comparable to that of the original black-box model W and can explain their own behavior extremely well. Thus, the CPM for V can actually replace N, leading to more explainable deployed models. Second, CPM models localize concept-level information in

---

Manuscript
their hidden representations, which makes their behavior specific inputs very easy to explain. We illustrate this using Path Integrated Gradients (Sundararajan et al., 2017), which we adapt to allow input-level attributions to be mediated by the intermediate states that were targeted for localization. Thus, while both CP and CPM1 are comparable as explanation methods according to CFBaB, the qualitative insights afforded by CPM1 models may given them the edge when it comes to explanations.
RELATED WORK
Understanding model behavior serves many goals for large-scale Al systems. including trans- parency (Kim, 2015; Lipton. 2018; Pearl. 2019; Ehsan al., 2021). trustworthiness (Ribeiro et al., 2016; Guidotti et al., 2018: Jacovi & Goldberg, 2020; Jakesch et al., 2019), safety (Amodei et al., 2016; Otte, 2013), and fairness (Hardt et al., 2016; Kleinberg al., 2017: Goodman & Flaxman, 2017: Mchrabi et al., 2021). With CPMs, our goal is to achieve explanations that are causally motivated and concept-based, and so we concentrate here on relating existing methods to these two goals.
Feature attribution methods estimate the importance of features, generally by inspecting learned weights directly or by perturbing features and studying the effects this has on model behavior (Molnar, 2020; Ribeiro et al., 2016). Gradient-basse based feature attribution methods extend this general mode of explanation to the hidden representations in deep networks (Zeiler & Fergus, 2014; Springenberg al., 2014; Binder al., 2016; Shrikumar et al., 2017; Sundararajan al., 2017). Concept Activation Vectors (CAVs; Kim et al. 2018; Yeh et al. 2020) can also be considered feature attribution methods, as they probe for semantically meaningful directions in the model's 's internal representations and use these to estimate the importance of concepts on the model predictions. While some methods in this space do have causal interpretations (e.g.. Sundararajan al. 2017; Yeh et al. 2020), most do not. In addition, most of these methods offer explanations in terms of specific (sets of) features/neurons. (Methods based on CAVs operate directly in terms of more abstract concepts.)
Intervention-baat based methods study model representations by modifying them in systematic ways and observing the resulting model behavior. These methods are generally causally motivated and allow for concept- based explanations. Examples of methods in this space include causal mediation analysis (Vig et al., 2020; De Cao et al., 2021; Ban et al., 2022), causal effect estimation (Feder et al., 2020; Elazar et al., 2021: Abraham et al., 2022; Lovering & Pavlick, 2022), tensor product decomposition (Soulos et al., 2020). and causal abstraction analysis (Geiger et al., 2020; 2021). CPMs are most closely related to the method of IIT (Geiger et al., 2021), which extends causal abstraction analysis to optimization.
Probing is another important class of explanation method. Traditional probes do not intervene the target model, but rather only seek to find information in it via supervised models (Conneau et al., 2018; Tenney et al., 2019) or unsupervised models (Clark al., 2019; Manning et al., 2020; Saphra & Lopez, 2019). Probes can identify concept-based information, but they cannot offer guarantees that probed information relevant for model behavior (Geiger et al., 2021). For causal guarantees, is likely that some kind of intervention is required. For example, Elazar et al. (2021) and Feder et al. (2020) remove information from model representations to estimate the causal role of that information. Our CPMs employ similar set of guiding ideas but are not limited to removing information.
Counterfactual explanation methods aim to explain model behavior by providing a counterfactual example that changes the model behavior (Goyal et al., 2019; Verma et al., 2020; Wu et al., 2021). Counterfactual explanation methods are inherently causal. If they can provide counterfactual examples with regard to specific concepts, they are also concept-based.
Some explanation methods train model making explicit use of intermediate variables representing concepts. Manipulating these intermediate variables inference time yields causal concept- model explanations (Koh et al., 2020; Künzel et al., 2019).
Evaluating methods in this space has been persistent challenge. In prior literature, explanation methods have often been evaluated against synthetic datasets (Feder et al., 2020; Yeh et al., 2020). In response, Abraham al. (2022) introduced the CEBaB dataset, which provides numan-validatte concept-based dataset to truthfully evaluate different causal concept-based model explanation methods. Our primary evaluations conducted on CFBaB.
2

---

Manuscript
V
Let Zu,e be text written in situation (u, e)):
=
G
Human-created &Ct-ee
Crowdworker edit dn,t to express that C₁ had value c', seeking to keep all else constant.
(a) A structural causal model leading to an actual text
Metadata-sampled :CC-ee Fu.e
Tu.o and its counterfactual text zn1e C U is an ex- ogenous variable over experiences, ..,.... diating concepts, and V exogenous variable cap- turing writing (and star-rating) experience. right, we create counterfactual in which concept C: takes on different value. Unfortunately, we cannot truly create such counterfactual situations and we never observe pairs texts like these. Thus, we must rely on approximate counterfactuals.
Sampled text expressing that C₁ has value d' but agreeing with Zu,e all other concepts.
me-
(b) Approximate counterfactuals. In the human- created strategy, humans revise an attested text try to express a particular counterfactual, secking to sim- ulate a causal intervention. In the metadata metadata-sampled sampled strategy, find separate text that aligns with the original for the value insofar expresses all the same concepts except the target concept Ct.
12
7811
T221
Y022
1822
Tu.e
:
Neat
1id1
T2.2
Mimmi
Cut
P11
P12
P1
Pi1
Piz
Plu
Pic
Fu.0
P21
P22
p21
Pout
1221
P2a
Phc
Pdi
Pd2
Pal
Bu.g
Pul
Pa2
Pi
Fust
(c) LMtiiit All CPMs (bottom) are trained to mimic the behavior of the neural model N to be explained (lop) for all factual inputs uu,u-
Pdl
pizz
Fa
Phi
Plaa
Pla
pla
P4
112
me
Poo
1122
T221
G.C.
Dout
12
Liu Examples Ju.y and SC,5-e mate counterfactual pair. The CPM (middle) is given input Ta,e. The objective is for it to mimic N (top) given cCtee' but under the intervention in which specific internal states are changed to those that the Pout
approxi-
CI
P12
Pit
P2a
Pat
Zupzi tc
:
CPM computes for input Fe "f' (bottom). which is distinct example that is sampled with the only cri- teria being that it express C: = c'. The effect of this intervention is to localize information about concept C: at the intervention site, since the only indication the CPM gets about C. t- via the intervention.
C,=e'
Pal
Paz
Pa
(d) Lix: Examples Th,e and CC. e are an approx- imate counterfactual pair. The CPM is given Tu," augmented with special token to,te' and trained to mimic the target model A when its input is a.ittee
Figure 1: Causal Proxy Model (CPM) summary. Every CPM for model N is trained to mimic the factual behavior of N (Lmimie). For CPMIN, the counterfactual objective is LIN. For CPMμss the counterfactual objective is LHI-
CAUSAL PROXY MODEL (CPM)
Causal Proxy Models (CPMs) are causal concept-based explanation methods. Given factual input Fa,v and a description of a concept intervention C, <- c', they estimate the effect of the intervention on model output. The present section introduces here on introducing the structure of these models and their objectives, and we save discussion of associated metrics for explanation methods for Section 4.
two CPM variants in detail. We concentrate
3

---

Manuscript
A Structural Causal Model Our discussion is grounded in the causal model depicted in Figure la, which aligns well with the CEBaB benchmark. Two exogenous variables U and together represent the complete state of the world and generate some textual data X. The effect of exogenous variable U on the data X is completely mediated by a set of intermediate variables C1 C2 ...., Cs. which we refer to concepts. Therefore, we can think of U as the part of the world that gives rise these concepts {C]-
Using this causal model, we can describe counterfactual data data that arose under a counterfactual state of the world (right diagram in Figure la). Our factual text is Zu,v. and we use xCCttc' for the counterfactual text obtained by intervening on concept C2 to set its value c'. The counterfactual C. describes the output when the value of C₁ is set to c', all else being held equal. Fu,e
0,0 Approximate Counterfactuals Unfortunately, pairs like (Iu,e: F d) are never observed, and
thus we need strategies for creating approximate counterfactuals 2 5C.600, Figure 1b describes the two strategies we use in this paper. In the human-created strategy, we rely on a crowdworker to edit Fu to achieve partieular counterfactual goal say, making the evaluation of the restaurant's ss food negative, CEBaB contains an abundance of such pairs (Zu, -C,5"))) However, CEBaB unusual in having many human-created approximate counterfactuals, so we also explore simpler strategy aC,td' is sampled with the requirement that it match u,, on all concepts but sets C2 to in which u,,""
c'. This strategy is supported in many real-world datasets for example, the OpenTable reviews underlying CEBaB all have the needed metadata (Abraham et al., 2022).
CPM Input-based CPM Given a dataset of approximate counterfactual pairs (I Zu,t &Cr-d) and a black-box -ox model N, we train a new CPMIN model P with a counterfactual objective as:
Li CEs (W (F P(ru,,i ((,,-e))
where Tu,vi tC,+-e in Eqn. 1 denotes the concatenation of the factual input and randomly initialized learnable token embedding tc describing the intervention C. t c'. CEs represents the smoothed cross-entropy loss (Hinton et al., 2015), measuring the divergence between the output logits of both models. The objective in Eqn. pushes P to predict the counterfactual behavior of N when a descriptor of the intervention is given (Figure Id).'
At inference time, approximate counterfactuals are inaccessible, To explain model M, we append the trained token embedding tc to factual input, upon which P predicts counterfactual output for this input, used to estimate the counterfactual behavior of A under this intervention.
CPM Hidden-state CPM Our CPM models are trained on the same data and with the same set of goals as CPMNN. to mimic both the factual and counterfactual behavior of N. The key difference is how the information about the intervention C, + c' is exposed to the model. Specifically, we adapt Interchange Intervention Training (Geiger et al., 2022) to train our CPMjn models for concept- based model explanation.
conventional intervention on a hidden representation H of neural network A fixes the value of the representation H to constant. In an interchange intervention, we instead fix H to the value it would have been when processing a separate source input s. The result of the interchange intervention is new model. Formally, we describe this new model as NH+H,, where <- is the conventional intervention operator and H₃ is the value of hidden representation H when processing input s.
Given a dataset of approximate counterfactual input pairs (zu, =Ci+d') and a black-box model NV, we train new CPMμ model P with the following counterfactual objective:
Ln = CEs (A P Prrcienng (Fa...)
Here H. are hidden states designated for concept C.. In essence, we train P to fully mediate the effect of intervening on C, in the hidden representation CC; The source input s is any input "e( that has C, = c'. As P only receives information about the concept-level intervention C, + c' via the
C,=c'
'Our objective is regard to single approximate counterfactual pair for the sake of clarity. At main-time, we aggregate the objective over all considered training pairs. We take C, to always represent the intervened-upon concept. The weights of A are frozen.
4

---

Manuscript
interchange intervention H < H the model is forced to store all causally relevant information with regard to C in the corresponding hidden representation. This process is described in Figure le.
C,=d and Zu.o share the same value only for C₁ and differ In the ideal situation, the source input ''' on all others, so that the counterfactual signal needed for localization is pure. However, we do not insist on this when we sample. In addition, we allow null effect pairs in which Ta, and x CC,tee' are identical. For additional details on this sampling procedure, see Appendix A.2.
At inference time, approximate counterfactuals are inaccessible, as before. To explain model W with regard to intervention C₁ + c', we manipulate the internal states of model P by intervening on the localized representation H for concept C, To achieve this, we sample source input x O. the train set any input that has C c' to derive H.
from
Training Objectives We include another distillation objective predict the same output as N under conventional circumstances as Limie CEs (N....)), P(zu,v). The overall training objective for our models can be written C >j Mimie + +2 Counterrraaau where Coounerfacuual can be either LIN or Liu. and we set λ1, A2 as 1.0 and 3.0 for simplicity.
4 EXPERIMENT SETUP
4.1 CAUSAL ESTIMATION- BASED BENCHMARK (CEBAB)
CEBaB (Abraham et al., 2022) is a large benchmark of high-quality, labeled approximate coun- terfactuals for the task of sentiment analysis on restaurant reviews. The benchmark was created starting from a set of 2.299 original restaurant reviews from OpenTable. For each of these original reviews, approximate counterfactual examples were written by human annotators; the annotators were tasked edit the original text to reflect a specific intervention, like 'change the food evaluation from negative to positive" or "change the service evaluation from positive to unknown'. In this way, the original reviews were expanded with approximate counterfactuals to total of 15,089 texts. The groups of originals and corresponding approximate counterfactuals are partitioned over train, dev, and test sets. The pairs in the development and test set are used to benchmark explanation methods.
Fach text in CEBaB was labeled by five crowdworkers with a 5-star sentiment score. In addition, each text was annotated at the concept level for four mediating concepts {Cambianse. Grood, Cnoise. and Oservice: using the labels (negative, unknown, positive), again with five crowdworkers annotating each concept-level label. We refer to Appendix A.1 and Abraham et al. 2022 for additional details.
As discussed above (Section 3 and Figure 1b), consider sources of approximate counterfactuals using CEBaB. For Luman-creeted counterfactuals, we use the edited restaurant reviews the train set. For vetadata-samples counterfactuals, sample factual inputs from the train that have the desired combination of mediating concepts. Using all the human-created edits leads to 19,684 training pairs of factuals and corresponding approximate counterfactuals. Sampling counterfactuals leads to 74,574 pairs. We use these approximate counterfactuals to train explanation methods. Appendix A.2 provides more information about our pairing process.
4.2 EVALUATION METRICS
Much of the value of a benchmark like CEBaB derives from its support for directly calculating the Estimated Individual Causal Concept Effect (ICaCEx) for a model W given a human -enaa-geeeeated approximate counterfactual pair (Zu,,, F0,5 (C,,,,
,CC+-e'):
(
aCred) N(( (x
This is simply the difference between the vectors of output scores for the examples.
We do not expect to have pairs (Nr 3Cte-d) at inference time, and this is what drives the devel- opment of explanation methods Exy that estimate this quantity using only a factual input ua, and description of the intervention C₁ d' To benchmark such methods, follow Abraham et al.
5

---

Manuscript
(2022) in using the ICaCB-ErTor:
(C) = [D]
Σ
Dist [iCaCEy((Tun)) 20 Ex c(dd) (4)
(Zu.vi ED
Here, we assume that D is dataset consisting entirely of approximate counterfactual pairs (Iu,,, -0,5-)) Dist measures the distance between the ICaCEN for the model N and the effect predicted by the explanation method. Abraham et al. (2022) consider three values for Dist: 1.2, which captures both direction and magnitude; Cosine distance, which captures the direction of effects but not their magnitude; and NormDiff (absolute difference of L2 norms), which captures magnitude but not direction. We report all three metrics.
4.3 BASELINE METHODS
BEST We compare our results with the best results obtained on the CEBaB benchmark. Crucially, BESTCEBaE consists aggregated best results from a set of methods including CONEXP (Goyal al., 2020), TCAV (Kim et al., 2018). ConceptSHAP (Yeh et al., 2020), INLP (Ravfogel et al., 2020), CausaLM (Feder et al., 2020), and S-Learner (Künzel et al., 2019).
S-Learner Our version of S-Learner (Künzel et al., 2019) learns to mimic the factual behavior of black-box model A while making the intermediate concepts explicit. Given a factual input, aspect-based sentiment
finetuned model B is trained to predict concept label for each concept classification task. Then, a logistic regression model LRw is trained to map these intermediate concept values the factual output of black-box model N,, under the following objective.
CS.B = CEs (NEs(N(2 (Z),,,),, (B(e,)))
(5)
By intervening on the intermediate predicted concept values at enference-time, we can hope to simulate the counterfactual behavior of N:
(Iu,ii C, + d') = LRv LBx(((((r, - (Bx
(6)
When using S-Learner in conjunction with approximate counterfactual inputs at train-time, we simply add this counterfactual data on top of the observational data that is typically used to train S-Learner.
GPT-3 Large language models such GPT-3 (175B) have shown extraordinary power in terms of in- context learning (Brown et al.. 2020).² 2 We use GPT-3 to generate new approximate counterfactual at inference time given factual input and descriptor of the intervention. This generated counterfactual is directly used to estimate the change in model behavior:
EXPT-3 (zn,o C. + c) = N (GPT-3( C. <
-
where ( C; + c') represents the GPT-3 generated counterfactual edits. We prompt GPT-3 with demonstrations containing approximate counterfactual inputs. Full details on how these prompts are constructed can be found in Appendix A.7.
4.4 CAUSAL PROXY MODELS
We train CPMs for the publicly available models released for CEBaB, fine-tuned as five-way sentiment classifiers on the factual data. This includes four model architectures: bert-bast bart-bas--uncsed (BERT; Devlin et al. 2019), RoBERTa-base (RoBERTa; Liu et al. 2019), GPT-2 (GPT-2; Radford al. 2019), and LSTM+GloVe (LSTM; Hochreiter & Schmidhuber 1997: Pennington et al. 2014). All Transformer- based models (Vaswani et al., 2017) have 12 Transformer layers. Before training, each CPM model is initialized with the architecture and weights of the black-box model we aim to explain. Thus, the CPMs are rooted the factual behavior of N from the start. We include details about our setup in Appendix A.3.
The inference time comparisons for these models are follows, where in Eqn. 8 and Eqn. refers to the CPM model trained under CPMIN and CPMHI objectives, respectively:
CPM (Ku,wi Ci c') = = (( (a-e) N (F
(8)
cCPMin (Zu,vi Cj t- =d== P = (Ka.v) - N ((va,))
(9)
We use the largest davinci model publicly available https: catpamaiicoomppaagrouuooo
com

---

Manuscript
no counterfactuas
sampled connterfactuols (ours)
Juman created counterfactuals
(ours)
(ours)
(ours)
Model
Metric
BESTemmn
S-Learner
S-Learner
GPT-3
CPM
CPM
S-Learner
GPT-3
CPMss
CPM
1.2
0.74
0.74
0.74
0.71 con)
(.63.cm
0.60
0.73
0.45
01.5
0.45
BERT
Cosime
0.59 Lu0y
0.63 (m
0.63 (mu
0.51 (00)
0.46 (00)
0.45 use
0.60 pm
0.36 (00)
0.35 (10)
0.36 (14)
NormDiff
0.44
0.54
0.53
0.35 (o1)
0.39 cm))
0.38
0.52 (m))
0.25
0.24
0.27
1.2
0.78 cony
0.78 (m))
0.78 (u))
0.74 con)
0.66 (m))
0.67 0o0
0.77 (m))
0.48 con
0.46 cm))
0.47 (1))
RoBERTa
Cneine
0.58
0.64
0.65 (m)
0.53
0.46.c0
0.47
0.63 (m))
0.39
0.38 cm))
0.39
NormDiff
0.45 (000
0.59 (m)
0.58 (J0)
0.36 (ot)
0.42 (1))
0.45 003g
0.56 (10)
0.28 com)
0.26 (m))
0.29 (s))
L2
0.60
0.60
0.61
0.65 (0n)
(.55
0.51
0.61 (00)
0.43
0.41 c00)
0.41
GPT-2
Cnsine
0.59 con
0.59 (m)
0.59 (m))
0.52 com
0.47 (m))
0.46 (08)
0.59 (m))
0.40 com
0.37 (m))
0.39 (0))
NormDiff
0.40
0.40
0.41
0.34
(.32
0.30 (08)
0.40 p0))
0.24
0.23
0.27 (m))
1.2
0.73 con
(o73
(.73
0.76 com
0.66
0.64 002)
0.72 (00)
0.49 c00)
0.52 (00)
0.54 (01)
LSTH
Cnsine
0.64
0.64
0.64 (m))
0.57
0.50
0.50
0.63 (m))
0.44
0.45 cm)
0.46
NorDDiff
0.50 cor>
0.53 (013
0.53 (00)
0.41 com
0.42 (00)
0.41 (00)
0.54 (00)
0.30 c00)
0.34 0.36 (90)
Table I: CEBaB scores measured three different metrics on the test set for four different model architectures five-class sentiment classification task. Lower is better. Results averaged over three distinct seeds, standard deviations in parentheses. The metrics are described in Section 4. Best averaged result is bolded (including ties) per approximate counterfactual creation strategy.
Here, is a source input with C. c' and H is the neural representation associated with C, which takes value H on the source input s. As HC; we use the representation of the [CLS] token. Specifically, for BERT we use slices of width 192 taken from the 1st intermediate token of the 10th layer. For RoBERTa, we use the 8th layer instead. For GPT-2, we pick the final token of the 12th layer, again with slice width of 192. For LSTM, we consider slices of the attention-gated sentence embedding with width 64. Appendix A.5 studies the impact of intervention location and size.
Following the guidance on IIT given by Geiger et al. (2022), we train CPMμ with an additional multi- task objective as LMulu = Ec,ec CE(MLP(H ), where probe is parameterized by multilayer perceptron MLP, and HC; the value of hidden representation for the concept C₁ when processing input z with a concept label of c for C..
RESULTS
We first benchmark both CPM variants and our baseline methods on CEBaB. We show that the CPMs achieve state-of-the-art performance, for both types of approximate counterfactuals used during training (Section 5.1). Given the good factual performance achieved by CPMs, we subsequently investigate whether CPMs can be deployed both as predictor and explanation method at the same time (Section 5.2) and find that they can. Finally, we show that the localized representations of CPM give rise to concept-award feature attributions (Section 5.3). Our supplementary materials report on detailed ablation studies and explore the potential of our methods for model debiasing.
5.1 CEBAB PERFORMANCE
Table 1 presents our main results. The results are grouped per approximate counterfactual type used during training. Both CPMIN and CPMj beat BESTсиBаB in every evaluation setting by a large margin, establishing -fate-of-the-art explanation performance. Interestingly. CPMμ seems to slightly outperform CPMIN using sampled approximate counterfactuals, while slightly underperforming CPMIN on human-created approximate counterfactuals. Appendix A.6 reports on ablation studies that indicate that, for CPMHt. this state-of-the-arr performance is primarily driven by the role of IIT in localizing concepts.
S-Learner, one of the best individual explainers from the original CEBaB paper (Abraham et al., 2022), shows only a marginal improvement when naively incorporating sampled and muman-created counterfactuals during training over using no counterfactuals. This indicates that the large perfor- mance gains achieved by our CPMs over previous explainers are most likely due to the explicit use of counterfactual training signal, and not primarily due to the addition of extra (counterfactual) data.
GPT-3 occasionally performs on par with our CPMs, generally only slightly underperforming our best explainer on human-creater counterfactuals, while being significantly worse sampled counter-
7

---

Manuscript
sampled counterfactuals
human-created counterfactuals CPM
Model
Metric
CPM
CPM
CPM
0.63 cn1)
0,52 cou
0.42 (m))
0.38 cos
BERT
Cosine
0.46 (20)
0.45
0.34 cm)
0.30 coo
NormDiff
0.39 (0l)
0.33 (m2)
0.23 (m))
0.22 cos)
0.66 (01)
0.63
0.40 cm))
0.37 co9
RoBERTa Cosine
0.46 (20)
0.48
0.33 (01)
0.29 (o9)
NormDiff
0.42 (91)
(19)
(01)
(m))
1.2
0.55 (01)
0.41
0.38 c013
0.36
GPT-2
Cosine
0.47 (21)
0.39 (m))
0.37 cmn)
0.35 cos
sumpled
hasman created
NormDiff
0.32 (01)
0.25
0.22 (m1)
0.24 (05)
Black-
counterfactuals
counterfactuals
Model
box
CPM
CPM
CPMN CPMun
L2
0.66 (01)
0.41
0.46 (00)
0.42 (o0)
LSTM
Cosine
0.50 (20)
0.42 (12)
0.50 (m))
0.40 (01)
BERT
0.70
0.70 0.67 (om
con 0.69
NormDiff
0.42 0.25
0.31 0.28 (02)
RoBERTa
0.70 (o0)
0.70 com) 0.69 son
0.71 (on) 0.71 com
GPT-2
0.65 (00)
0.65 co0
0.67 con
0.66 (01) 0.68 com
LSTH
0.60
0.60
0.56 (00)
0.54 (00) 0.59
Table 3: Self-explanation CEBaa scores mea- sured in three different metries on the test set for four different model architectures as a five-class sentiment classification task. Lower is better. Results averaged over three distinct seeds, stan- dard deviations in parentheses.
Table 2: Task performance measured as
Macro-F1 score on the test set. Results averaged over three distinct seeds; standard deviations in parentheses.
factuals. While the GPT-3 explainer also explicitly uses approximate counterfactual data, the results indicate that our proposed counterfactual mimic objectives give better results. The better performance of CPMs when considering sampled counterfactuals over GPT-3 shows that our approach is more robust to the quality of the approximate counterfactuals used. While the GPT-3 explainer is easy to set up (no training required), it might not be suitable for some explanation applications regardless of performance, due to the latency and cost involved in querying the GPT-3 API.
Across the board, explainers trained with human-created counterfactuals are better than those trained with sampled counterfactuals. This shows that the performance of explanation methods depends on the quality of the approximate counterfactual training data. While human counterfactuals give excellent performance, they may be expensive to create. Sampled counterfactuals are cheaper if the relevant metadata is available. Thus, under budgetary constraints, sampled counterfactuals may be more efficient.
Finally, CPMN is conceptually the simpler of the two CPM variants. However, we discuss in Section 5.3 how the localized representations of CPM lead to additional explainability benefits.
5.2 SELF-EXPLANATION WITH CPM
As outlined in Section CPMs learn to mimic both the factual and counterfactual behavior of the black-box models they are explaining. We show in Table 2 that our CPMs achieve a factual Macro-F1 score comparable to the black-box finetuned models.
We investigate if we can simply replace the black-box model with our CPM and use the CPM both factual predictor and counterfactual explainer. To answer this questions, we measure the self-explanation performance of CPMs by simply replacing the black-box model N in Eqn. 4 with our factual CPM predictions at inference time.
Table 3 reports these results. We find that both CPMIN and CPMju achieve better self-explanation performance compared to providing explanations for another black-box model. Furthermore, CPM provides better self-explanation than CPMIN. suggesting our interchange intervention procedure leads the model to localize concept-based information in hidden representations. This shows that CPMs may be viable replacements for their black-box counterpart, since they provide similar task performance while providing faithful counterfactual explanations both the black-box model and themselves.
8

---

Manuscript
Model
Prodeted Ornope
Score
Word Importance
antince
+0.13
kop
load
saad the decomstane MidtO
bst
Ekyy
Inuraity
and
Blackhto
frod
+4.11
1hc Ireee Stan
koad
1340
bex Dhyy
hoad Irumully
Scatoor
and
50
[1.5] Uh IEAMAC Roo
deccoration
taaie
tbay
Socndlyy
delatoon
panta
[SEP)
Sieellyy
10.23
[CLS] the mosk 10:0
100 koad
and the Deciraaionns Meui Taste
1xx Dvey Ttad
Fiendlly
water
Telicion
pana [SEP)
(1.5 the 1ot0ic
dbcovationa Midi
bot
(iayy
Sionlly
Waterr
delation
pada
decorrations
(577)
orviz
+4..5
[1.55] the menes 10a
500 koad
and
the decomttion Merre
1340 ealics
box tkay
Itad
stater oid
and
delutren
pa1s (M22))
[1.55]
Skay
fnsndly
pasa (563))
CIT
acammaa
Frod
[1..]]
500
kead
and the smal
decorsisi tsor euiros
Ekyy
thimnty
41344
[C.5] th (0600)
koad
the dicoostoons Vid
bus (bay
itad
waier
olt
and
delation
pasta (SEP)
[C.S] (000)) 100
decovations
tasic
Bryy
Blaxlly
wnterr
pada (SEP)
Table 4: Visualizations of word importance scores using Integrated Gradient (IG) by restricting gradient flow through the correpoondigg intervention site of the targeted concept. Our target class pools positive and very positive. Individual word importance is the sum of neuron-level importance scores for each input, normalized to to predicting the target class (red); +1 means the word contributes the most positively (green).
means the word contributes the most negatively
5.3 CONCEPT-AWARS FEATURE ATTRIBUTION WITH CPMиI
We have shown that CPM provides trustworthy explanations (Section 5.1). We now investigate whether CPMHI learns representations that mediate the effects of different concepts. We adapt Integrated Gradients (IG; Sundararajan et al. 2017) to provide concept- feature attributions, by only considering gradients flowing through the hidden representation associated with a given concept. We formalize this version of IG in Appendix A.8.
In Table 4, we compare concept-aware aware feature attibutions for two variants of CPMμi (IIT and Multi-tase and the original black-box (Finetuned) model. For IIT we remove the multi-task objective Luuli during training and for Multi-task we remove the the interchange intervention objective LIu- This helps isolate the individual effects of both losses on concept localization. All three models predict a neutral final sentiment score for the considered input, but they show vastly different feature attributions. Only IIT reliably highlights words that are semantically related to each concept. For instance, when we restrict the gradients to flow only through the intervention site of the noise concept, "loud" is the word highlighted the most that contributes negatively. When we consider the service concept, words like "friendly" and "waiter" are highlighted the most as contributing positively. These contrasts missing for representations of the Multi- and Finetuned models. Only the IIT training paradigm pushes the model to learn causally localized representations. For the service concept, we notice that the IIT model wrongfully attributes "delicious". This could be useful for debugging purposes and could be used to highlight potential failure modes of the model.
6 CONCLUSION
We explored the use of approximate counterfactual training data to build more robust causal ex- planation methods. We introduced Causal Proxy Models (CPMs), which learn to mimic both the factual and counterfactual behaviors of a black-box model W. Using CEBaB, a benchmark for causal concept-based explanation methods, we demonstrated that both versions of our technique (CPMIN and CPMμ) significantly outperform previous explanation methods without demanding the full causal graph associated with the dataset. Interestingly, we find that our GPT-3 based explanation method performs on-par with our best CPM model in some settings.
Our results suggest that CPMs can be more than just explanation methods. They achieve factual performance on par with the model they aim to explain, and they can explain their own behavior. This paves the way to using them as deployed models that both perform tasks and offer explanations. In addition, the causally localized representations of our CPMμi variant very intuitive, revealed by our concept-awe aware feature attribution technique. We believe that causal localization techniques could play vital role in further model explanation efforts.

---

Manuscript
REFERENCES
Eldar David Abraham, Karel D' Oosterlinck,, Amir Feder. Yair Ori Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. CEBaB: Estimating the causal effects of real-world concepts on NLP model behavior. Advances in Neural Information Processing Systems, 2022. URL httpt://arxiv...oom httprriiv.orggbb//2255555 14140.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané, Concrete problems in Al safety. 2016. URL http::://rrxiivorg/abbb////// 110./16666.666 06565.
Pangbo Ban, Yifan Jiang, Tianran Liu, and Shane Steinert- Steinert-Threlkeld. Testing pre- pre-trdi language models' understanding of distributivity via causal mediation analysis. arXiv:2209.0476.1 2022.
URL https: //ax:iv// www.ii.rr//bbba. absp/ http//2299.90422
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Alaus-Robed Müller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, 2016. URL https://doo org/10
00007/97-33319-220
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Pra- fulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in Neural Information Processing Systems, 2020. URL htttps:/prrcceedine merissenuriiss.. httpsssccppppe//000.. (B577od6bfccb497711888f888 1atfffaa142ff4a-PPaaae pa..r.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? An analysis of BERT's s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Florence, Italy, 2019. URL https:/ /www aclweb. anthology/WI1 918884
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties, Melbourne, Australia, 2018. URL https: //wp::://wwwwaaa http://wwwwwwww....//..oo: www.://.rrrgaaaooogg//--------
Nicola De Cao, Leon Schmid, Dieuwke Hupkes, and Ivan Titov, Sparse interventions in language models with differentiable masking. arxiv:2112.068377 2021. URL https://arxiv org/abs/
2112. 2112.06837.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Minneapolis, Minnesota, 2019. URL anthology/NI 94423.
org/
Upol Ehsan, Vera Liao, Michael Muller, Mark O Riedl, and Justin Weisz. Expanding explain- ability: Towards social transparency in Al systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021. URL https: //dtps:/ddl.cmm wrt//or/pdf/////////t /10. 1145/ 3411764. 1411774.444188.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Lin- guistics, 2021. URL https:://iirect mi mit..edee tdw//acl/////eeee//////////cll / //aatiiiicc wwt.tc/eecell....../////// 11tp:/1111eaaaaaaaaae_ee http:///annneeiccrooi-eneeeeeaee -atpaaaiioral--ppppaatiiio-o-ooo -xp---atioannnnnnnaaaeaa
Amir Feder, Naday Oved, Uri Shalit, and Roi Reichart. CausaLM: Causal model explanation through counterfactual language models. Computational Linguistics, 2020. URL https: https://aclanttholpg
org/2021.c1-22
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference mod- els partially embed theories of lexical entailment and negation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2020. URL https://aclanthologg.oom wrw.ho/oooyyoo//22200 blackboxnlp-1.1
10

---

Manuscript
Atticus Geiger, Hanson Lu, Thomas leard, and Christopher Potts. Causal abstractions of neural net- works. Advances Neural Information Processing Systems, 2021. URL https://proceedingss neurips,
Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Ieard, Noah Good- man, and Christopher Potts. Inducing causal structure for interpretable neural networks. In International Conference on Machine Learning, 2022. URL httpss:/pproceeeinggss. 22a. v162/ge 1/62/eiger222 22g............. html.
press/
Bryce Goodman and Seth Flaxman. European Union regulations on algorithmic decision and "right to explanation". Al Magazine, 2017. URL http://arxiv, wttg//b//////// 08813.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In International Conference on Machine Learning, 2019. URL. http: //proceedings mlr. wptppsese////yaalllaaa 9a. html.
Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining Classifiers with Causal Concept Effect (CaCE). (00i://99000077000 [cs, stat], February 2020. URL httphttp://axxiv www..oggg//..........oo....o.os. /aaaab/s/99/// 907.07165. arXiv: 1907.07165.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 2018. URL https://dl. www.m.org/oo//ab/////// 10000..145/// 1145/3236009.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems. Curran Asso- ciates, Inc., 2016. URL https: http:://proceedinng www.wps.cc//eepeeeeeeee httppppccccpape//11//////5 2016ffll/ 5 Padd.....
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. NeurlPS Deep Learning and Representation Learning Workshop. 2015. URL https://arxiv.
org/abs/1503. 152300011.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
URL https://ieeexplore www...erg.abseeeeeeeeeeee..eeeee orgg///bssraatt/oommme////59995
Paul W Holland. Statistics and causal inference. Journal of the American statistical Association, 1000/01000000 016621459)) 1986. 10478354.
1986. URL httpss//www tandf ond ine. www.oooooooooooi/////oooooooooii
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. URL https: / acc hatph//laaanttoogyyoyooooo org/ 22t./2220.c0 acl-main. mamm..i.... 386.
Maurice Jakesch, Megan French, Xiao Ma, Jeffrey T Hancock, and Mor Naaman. Al-mediated communication: How the perception that profile text was written by Al affects trustworthiness. In Proceedings of the 2019 CHIl Conference Human Factors in Computing Systems, 2019. URL https://dl. acm. org/do wwwp//b///////o//////////aa/bbee 1145/3290605 3300469.
Been Kim. Interactive and Interpretable Machine Learning Models for Human Machine Collabora- fion. PhD thesis, Massachusetts Institute of Technology, 2015. URL //dpsa/ mi t. edu/ handle/1721 1/98680.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). International Conference on Machine Learning, 2018. URL prress/800/Ke /im/ 8d. html.
http://proceedinns. mlr. http:/g........ps/aaaaaa/
Jon Kleinberg. Sendhil Mullainathan, and Manish Raghawan. Inherent trade-offs in the fair determi- nation of risk scores. 8th Innovations in Theoretical Computer Science Conference, 2017. URL
http:///rops... www.:sss..dshhhhh.cccc. det../s///p////////oo//// /tops//oollexte/2211191
Pang Wei Koh, Thao Nguyen, Yew Siang Tang. Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, 2020. URL https:: https://proceeeinngs mlr, press/vi 1atss///v1/kokhaa.... htmml.

---

Manuscript
Sören R Künzel, Jasjeet Sekhon, Peter J Bickel, and Bin Yu. Metaleamers for estimating heteroge- neous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 2019.
Zachary C. Lipton. The mythos of model interpretability. Communications of the ACM, 2018. URL attps://dol org/10 10. 1145/3333231.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. 2019. URL
11692.
Charles Lovering and Ellie Pavlick. Unit testing for concepts in neural networks, arXiv preprint arXiv:2208. 10244, 2022.
Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 2020. URL
pnas. org/ cot///nttn/t////tt/// 30046.
Ninarch Mchrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and faimess in machine learning. ACM Computing Surveys (CSUR), 2021. URL https://dL acm. org/doi /10. 1145/3457607.
Christoph Molnar. Interpretable Machine Learning. 2020.
Clemens Otte. Safe and interpretable machine learning: A methodological review. In Computational Intelligence in Intelligent Data Analysis, 2013. URL https://link. com wwt. coom/hapter// 10. 00011100777778 0117779978--62--28 -0122-2278---88
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 2019.
Judea Pearl. The limitations of opaque learning machines. Possible Minds: Twenty-Five Ways of Looking at AI, 2019. URL https:ttp////tttm / /tttp//ft......o... cs. cww.ca.....c..c...............l. wwt.pub/stat_ssr//rtteeee .ddf.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), 2014. URL https: /aap:////o/lathhooogy wow.oogg/////10/
Alec Radford, Jeffrey Wu. Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog. 2019. URL https://cdnt http::///doooooaii...c.m www.opp..../eeeetteeeeeeeeeeeeee unsupervised_mn waatttitsk__aaareeee pdf.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. URL org/2020 acl-main. 647.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should I Trust You?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco California USA, 2016. URL https:
//dl. acm. org//oi/1/. /011/1131///72 2ttp//9993977.2299778888
Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019. URL https: /tts:://wwwww /wwww aclweb. wgg/anthology//119-333
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning Important Features through Propagating Activation Differences. In International Conference on Machine Learning, 2017. URL http://proceediing... mlr. press//70/shsi www.shaarr11aaaaaaa
12

---

Manuscript
Paul Soulos, R. Thomas McCoy, Tal Linzen, and Paul Smolensky. Discovering the compositional structure of vector representations with role learning networks. In Proceedings of the Third BlackboxNLP Workshop Analyzing and Interpreting Neural Networks for NLP, Online, 2020. URL httph::p////..... /awpantolaaaayooyyooooooo org/2020 blackboxnlp-1..
Jost Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: httpg//bs/1412.... the all convolutional net. CoRR, 12 2014. URL http:////aaa...oor///////......
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, 2017. URL http://proceedings mlr. www.r.prees/v//11/ sundaaarajan htmml....
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Dipanjan Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
URL https:/ wwwww aclweb. org/ anthhlooy//P111---
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neu- ral Information Processing Systems 30. 2017. URL http://papers nips. ccc cc/paper/ 7181 7181-attention- aalss-lllllouueeee need. pad.
Sahil Verma, John Dickerson, and Keegan Hines. Counterfactual explanations for machine learning: A review. 2020. URL httpp://arxtte wrtp://///abbb////// 10596.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. Investigating gender bias in language models using causal mediation analysis. In Advances Neural Information Processing Systems, 2020. URL https::/proceedding Abstratt..bttttttccc..
neur cc/
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, al. Huggingface' transformers: State- of-the-art natural language processing. 2019. URL https: /ttp://///xvvvvoeee org.orgg/aaoo abs/1910.003
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. Polyjuice: Generating counterfactuals for explaining. evaluating, and improving models. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021.
Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. Advances in Neural Information Processsing Systems, 2020. URL httpp://///// pttpp//ooccen....////// wwwrieerrne...e cc/ Saaper/00200 h File/eeb2877f755cl11004ff2222222
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), European Conference on Computer Vision, 2014. URL https: springer hct//chapper/111.1007//// 3-319- 53.
A APPENDIX
A. CEBAB DATASET STATISTICS
Table 5 shows dataset statistics of CEBaB. The variants of CEBaB we consider only impact the train split. The top panel shows the number of observational samples and edits introduced in the CEBaB paper. The bottom panel shows our paired versions, where we create approximate counterfactual pairs. We explore two variants of approximate counterfactuals: (Section 4.1). The human setting considers all pairs made possible by using all data. The sampling setting considers pairs sampled from only the observational data, discussed in Section A.2.
and sampled counterfactuals
13

---

Manuscript
Dataset
train
dev
# test
CEBaB (observational)
1,755
1,673
1,689
CEBaB (all)
11,728
1,673
1,689
CEBaB (paired, human)
19,684
3,898
3,958
CEBaB (paired, sampling)
74,574
3,898
3,958
Table 5: Dataset statisties.
A.2 TYPES OF APPROXIMATE COUNTERFACTUAL PAIRS
Our approximate counterfactual training data comes in paired sentences of (original sentence, ap- proximate counterfactual sentence). The approximate counterfactuals differs from their original counterparts in only one concept value. We consider approximate counterfactual pairs to be sym- metric: we use both (original sentence, approximate counterfactual sentence) and (approximate counterfactual sentence, original sentence) as training pairs.
Human-created Counterfactuals CEBaB contains multiple counterfactual sentences for each original review. To achieve this, the dataset creators asked annotators to edit the original sentence to achieve a specified goal (c.g., "change the evaluation of the restaurant's food negative'). These originals and corresponding edits form our human pairs.
Metadata-samples Counterfactuals Human-created counterfactuals are not always available. With CEBaB, we simulate a second type of approximate counterfactuals by using metadata- -ggiddd heuris- ties: for given original sentence, sample counterfactual from the train set by matching concept labels while allowing only one label to be changed.
During training, also consider null effect pairs in our sampling setup. These pairs resemble cases where our approximate counterfactual sentence is identical to the original sentence. When training models these pairs, we expect models predict the same counterfactual and factual output.
A.3 TRAINING REGIMES
CPMIN To train CPMIN, we use the same model architecture as N, and initialize it with the model weights using weights from N. The maximum number training epochs is 30 with learning rate of 5e -5 and an effective batch size of 128. The learning rate linearly decays to 0 over the 30 training epochs. We employ an early stopping strategy for COSTeacE over the dev set for an interval of 50 steps with early stopping patience set to 20. We set the max sequence length to 128 and the dropout rate to 0.1. We take a weighted sum two objectives as the loss term for training CPM Specifically, we use [UMimie, WIN| [1.0,3.0] 3.0]. For the smoothed cross-entropy loss, use a temperature of 2.0.
CPM To train CPMHI we use the same model architecture as M, and initialize it with the model weights using weights from N. The maximum number training epochs is 30 with learning rate of 55 and an effective batch size of 256. We use a higher learning rate of 0.001 for the LSTM model it enables quicker convergence. The learning rate linearly decays epochs. We employ early stopping strategy for COSIcacE over the dev set for interval of 10 steps with early stopping patience set 20. We set the max sequence length to 128 and the dropout rate to 0.1. We take a weighted sum of three objectives as the loss term for training CPMHi- Specifically, use [uMimies WMltti, WHi] = [1.0, 1.0, 3.0]. In Appendix A.6, we conduct a set of ablation studies to isolate the individual contributions from each objective. For the smoothed cross-entropy loss, a temperature of 2.0.
over the 30 training
Our models all implemented in PyTorch (Paszke al., 2019) and using the HuggingFace library (Wolf et al., 2019). All of our results are aggregated over three distinct random seeds. To foster reproducibility, we will release code repository and model artifacts to the public.
14

---

Manuscript
L2
Cosine
NormDiff
Macro-FF
Layers
0.5
0.4
0.4
0.1
0.6
0.4
0.2
0.6
14
DEXT RuBERTT
GPT-2
Sizes
0.0
0.8
0.6
0.4
0.2
0.01
100 100
200
50
100
150 200 50 100 150
200
50
100
250
200
Figure 2: CEBaB scores for different intervention site locations and sizes for CPM The scores are measured in three different metrics on the test set for four different model architectures five-class sentiment classification task. Results averaged over three distinct seeds. Task performance as Macro-F1 score is reported when applicable. Shaded areas outline + SD.
A.4 ADDITIONAL BASELINE RESULTS
Table 6 shows baselines adapted from Abraham et al. (2022), which contains the present state-of-the- art explanation methods for the CEBaB benchmark. We report the best scores across these explanation methods in Table 1. These baselines are trained without using counterfactual data. Thus, we build additional baselines that use counterfactual data as shown in Table 7. S-Learner is selected as the best performing models and included in Table 1 for comparisons. The equations for the additional baselines are as follows:
EN Eapprox
ui C. + ()+c)=N((
=N(ss (xu,))
(10)
CN
crandom
(Iu,p C2 4 C)+d== =Nc==)(smmoo)) - ((nww))
(11)
ECACE (C₂ c)== |DCite
1
Σ
(N (2(ite) - (au.v)
(12)
uuu "EDD
(C, + cd)= DDC+t
(f (2210)) - (I
(13)
(Iu.D, JEDE,
where ganddom a randomly sampled training input, approx is training input sampled to match the concept-level labels of the true counterfactual under intervention C2 c DC:t' is the set of all approximate counterfactual training pairs that represent C, t- e' intervention, and f is look-up function that returns the ground-truth label associated with an input.
The signatures of ATE and ECCCE reflect that they are independent of the specific factual input Fa, considered. Furthermore, gATE is independent of N given that this explainer only uses ground-truth training labels to estimate causal effects.
Additionally, consider X-Learner, a variant of S-Learner (Künzel al., 2019). Our X-Learner consists of three steps. First, we cluster examples into groups by their concept and predicted concept label pairs (e.g., select all examples with food being positive)?. For each group, we fit logistic regression model (((C,.)) to predict the factual output of black-box model N using concept labels for each example except for labels for C, Next, we the models from the first step to build training sets for our individual treatment effect (ITE) estimators. To achieve this, we calculate ITE for each example as,
DC pPccee (B(2 -N (Z, D
(B
(14)
We use the finetuned concept-level sentiment analysis models B released by Abraham et al. (2022) for concept lahel prediction, which identical to the ones used in S-Leamer in Section 4.3.
15

---

Manuscript
Model
Metric
Approx!
S-Learnert
INLP8
L2
0.81 (01)
0.74 (.2)
0.80 (02)
BERT
Cosine
0.61 (01)
0.63 (01)
0.59 (03)
NormDiff
0.44 (00))
0.54 (02)
0.73 (02)
I.2
0.83 (01)
0.78 (00)
0.84 (01)
RoBERTa
Cosine
0.60 (01)
0.64 (00))
0.58 (01)
NormDiff
0.45 (.00)
0.59 (001)
0.81 (001)
L2
0.72 (12)
0.60 (.2)
0.72
GPT-2
Cosine
0.59 (00)
0.59 (.01)
1.00 (00)
NormDiff
0.41 (00))
0,40 (001)
0.58 (00)
L2
0.86 (01)
0.73 (00)
0.79 (01)
LSTM
Cosine
0.64 (01)
0.64 (.01)
0.74 (02)
NormDiff
0.50 (.01)
0.53 (.01)
0.60 (001)
Table 6: CEBaB scores measured in three different metrics on the test set for four different model architectures five-class sentiment classification task. Results are adapted from Abraham et al. (2022). Lower is better; standard deviations over 5 distinct seeds in parentheses. Results are aggregated over all aspects and all directional concept label changes. Details about these evaluation metrics can be found in Section 4. Results are based on Abraham et al. (2022), Künzel et al. (2019), and Ravfogel et al. (2020).
no connterfaciaals
sampled commerfactiil Cact-
human created counterfactuals
Alc-
AlL-
CaCl-
Model
Metric
X-Learner
Approx
Explainer
Explainer
Randon
X-Leamer
Apgrox
Explainer
Explainer
Randee
X-Learner
1.2
0.78 udi
0.81 :on
0.8)
0.81 cm
0.78 0.84
0.79 cm))
0.81 (25)
0.50 cm
0.75 0.84
BERT
Cosine
0.68 (1)
11.0 Jan
(572
0.72
0.53 (80)
0.68 um
01.66 /a0
0.69
11.9
0.53
0.64 umu
NornDiir
0.53.cm
0.44
(.62.10)
(162
(.55
0.53
0.43:00
0.62.
0.64
0.54
0.54.cm
12
0.82 (m0)
0183 (an
0.85 (00)
0.55 (30
0,87 (mb
0.82
0.31 /00
0,85 (00)
11.4
0.87
0.79 um
RORERTA
Cosine
0,20
0.1
0,73
(o73
0.53 (m)
0.70.vm)
0.57 (00
0,70
0.70
0.53
0.67
NermDiaf
0,57
0.46
0.58
0.57
0.44:30
0.57
0.68
0,59
0.58
12
(.65
0.72 (a%)
0.60
(368
0.76 (io
0.65.mn
0.72 (30
0.68 (19)
0:68
0,76
0.63.on
GPT-2
Cosine NernDiff
0.64
0.59
0.64.mm
0.86
0.65
0,56
0.62um
0.41
0.40:00
0.49
0.47
0.41
0.40
0.69
0.50
0,47
0.42
12
0,77
0.57
0.78
0:77
0.78
0.56
0.84
0.74
LSTH
NocmDDir
0.52
0.55
0.55
Table 7: CEBaB scores for additional baselines we considered. CEBaB scores are measured in three different metrics the test set for four different model architectures five-class sentiment classification task. Lower is better. Results averaged over three distinct seeds, standard deviations in parentheses. Details about these evaluation metrics can be found Section 4.
where B(x G, excludes the concept label for concept C₁ It measures the ITE for Tup when we change the concept label of C; from c to c'. We aggregate DCuiee over examples based on their editing concepts and concept labels. Next, we fit a set of linear regression models TCcce-e' to predict ITE for changing the concept labels for C; given concept labels of an example except for labels for C. Lastly, we use TC predict counterfactual output changes as,
Ex EX.Lemmer (Iu,wi C t d') =p TCuooee (B(xC, + (1 (1-p) p) To (B(a
(15)
where p is the propensity score which is calculated using B the probability of C. taking concept label el for an input example Fu by considering two potential concept labels e and d'
A.5 INTERVENTION SITE LOCATION AND SIZE
Previous work shows that neurons in different layers and groups can encode different high-level concepts (Vig et al., 2020: Koh et al., 2020). CPMμ pushes concept-related information to localize the targeted intervention site (the aligned neural representations for each concept). In this section, we investigate how the location and the size of the intervention site impact CPMjl performance. We use the optimal location and size found in this study for other results presented in this paper.
Location For Transformer- based models, we vary the location of the intervention site by intervening on the "[CLS]" token embedding layer 1. Specifically, we set 1 (2, 4, 6, 6,8, 10, 12). We skip this
16

---

Manuscript
Model
Ablation
L2
Cosine
NormDiff
Macro-F1
CPM
0.45 (.2) 0.36 (03)
0.27 (00)
0.69 (011)
Matli
0.47 (04)
0.38 (04)
0.30 (m)
0.69 (01)
BERT
Lu
0.79 (on)
0.60 (03)
0.64 (m)
0.60 (08)
random inif
0.81 (.02)
0.52 (00)
0.55 (02)
0.08 (00)
training
0.80 (.02)
0.86 (.04)
0.76 (.02)
0.70 (.01)
CPM
0.47 (03) 0.39 (.03)
0.29 (005
0.71 (.00)
Malli
0.49 (.05)
0.41 (05)
0.32 (06)
0.70 (.00)
RoBERTa
Luu
0.81 (,00)
0.53 (02)
0.63 (0))
0.39 (.06
random init
0.85 (.00)
0.51 (.00)
0.59 (01)
0.06 (.00)
+ no training
0.84 (00))
0.93 (.05)
0.83 (.00)
0.70 (.00)
CPMи
0,41 (04)
0.39 (.05)
0.27 (.05)
0,68 (.00)
Luuli
0.43 (.03)
0.41 (05)
0,,29 (04)
0.67 (.00)
GPT-2
Lin
0.66 ((01)
0.58 (04)
0,49 (01)
0.58 (.04)
random init
0.73 (.00)
0.54 (.00)
0,47 (0D)
0.16 (.00)
no training
0.65 (.00)
0.61 (.00)
0.57 (002
0.65 (.00)
CPMи
0.54 (.01)
0.46 (01)
0.36 (.00)
0.59 (.0))
Lmala
0.56 (.02)
0.47 (.0)
0.41 (.02
0.59 (.01)
LSTM
Lu
0.73 (.00)
0.64 (.02)
0.59 (.00)
0.59 (.01)
random init
0.82 (.00)
0.55 (.00)
0.55 (00)
0.13 (.04)
training
0.73 (01) 0.74 (.00) 0.59 (01)
0.60 (.01)
Table 8: Ablation study of our CPMμI method trained with human approximate counterfactual strategy. CEBaB scores measured in three different metrics on the test set for four different model
architectures as five-class sentiment classification task. Lower is better. Results averaged over three distinct seeds, standard deviations parentheses.
experiment for non-Transfone non-Transformer-basedd model (i.e., LSTM) since it only contains a single sentence embedding.
As shown in the top panel of Figure 2, intervention location significantly affects CPM performance. Our results show that layer 10 for BERT, layer 8 for RoBERTa, and layer for GPT-2 lead the best performance. This suggests layers have different efficacy in terms of information localization. Our results also show that intervening with deeper layers tends to provide better performance. However, for both BERT and RoBERTa, intervening on the last layer results in slightly worse performance compared to earlier layers. This suggests that leaving Transformer blocks after the intervention site helps localized information to be processed by the neural network.
Size For Transformer-based models, change the size of the intervention site de for each concept. Specifically, we set d. {1, 16, 64, 128, 192). For instance when de = 1, we use single dimension of the "[CLS]" token embedding to represent each concept, starting from the first dimension of the vector. For our non- Transrfooorrrrrr based model (LSTM), we intervene on the iisention-gatenn sentence embedding whose dimension size set to 300. Accordingly, we set de {1, 16, 64, 75).
As shown in Figure 2. larger intervention sites lead to better performance for all Transformer bassdd models. For LSTM, we find that the optimal size is the second largest one instead. On the other hand, our results suggest that the performance gain from the increase of size diminishes we increase the size for all model architectures.
A.6 ABLATION STUDY CPMHI
Geiger et al. (2022) show that training with multi-task objective helps IIT to improve generalizability. In this experiment, we aim to investigate whether the multi-task objective we added for CPMμ plays an important role in achieving good performance. Specifically, we conduct two ablation studies: removing the multi-task objective by setting WMllii 0.0, and removing the IIT objective by setting win 0.0.
17

---

Manuscript
sampled counterfactuals
human-creatte counterfactuals
Random
Probe-based
Random
Probe-based
Model
Metric
CPMm
Source
Source
CPMμ
Source
Source
L2
0.60 ca
0.74 (a5)
0.61 (ol)
0.45 (43) 0.70 (09)
0.43 (02)
BERT
Cosine
0.45 c.m
0.53 (am)
0.45 (00)
0.36 (04)
0.59 (04)
0.35 (01)
NormDitf
0.38 (.m)
0.54 (0z)
0.39 (ol)
0.27 (0))
0.53 (0)
0.25 (02)
1.2
0.67 (02)
0.79 (01)
0.66 (02)
0.47 (03))
0.72 (01)
0.44 (DI)
RoBERTa
Cosine
0.47 c.00
0.52 (01)
0.46 col)
0.39 (33)
0.57 (09)
0.37 (01)
NormDiff
0.45 (.3))
0.59 (00)
0.44 (03)
0.29 (.5)
0.55 (00)
0.25 (01)
1.2
0:51 (01)
0.65 (.0)
0.51 (m0)
0.41 (04)
0.58 (09)
0,39 (02)
GPT-2
Cosine
0.46 (.0)
0,55 (00)
0.46 (ol)
0.39 (05)
0.56 (10)
0.37 (01)
NormDiff
0.30 cam
0.46 (00)
0.31 (0t)
0.27 (.5)
0.44 (0))
0,25 (01)
1.2
0.64 (0z)
0.76 (.1)
0.65 (n0)
0.54 cm
0.69 (.99
0.55 (00)
LSTM
Cosine
0.50 (01)
0.57 (00)
0.50 (ol)
0.46 (00)
0.58 (00)
0.46 (01)
NormDiff
0.41 (011 0.54 (01)
0.41 (02)
0.36 (30)
0.52 (00)
0.38 (01)
Table 9: Ablation study of our CPMju method for different source input s sampling strategies inference time. CEBaB scores measured in three different metrics on the test set for four different model architectures as a five-class sentiment classification task. Lower better. Results averaged over three distinct seeds, standard deviations in parentheses.
Table 8 shows our results, which demonstrate that the IIT objective is the main factor that drives CPMи performance. Our results also suggest that the multi-task objective brings relatively small but consistent performance gains. Overall, our findings corroborate those of Geiger et al. (2022) and provide concrete evidence that the combination two objectives always results in the best-performing explanation methods across all model architectures.
Additionally, we explore two baselines for CPMиI. Firstly, we randomly initialize the weights of CPMμ1 Secondly, we take the original black-box model our CPMμ Compared to the results in Table 1, these two baselines fail catastrophically, suggesting the importance of our IIT paradigm.
As mentioned in Section 3, we sample source input Ci=e' from the train set any input that has C; = é' to estimate the counterfactual output. Furthermore, we explore two additional sampling strategics. First, we create a bascline where we randomly sample a source input from the train without any concept label matching. Second, we sample source input from the train set using the predicted concept label of our multi-task probe, instead of the true concept label from the dataset.
Fz'."'
As shown in Table 9, the quality of our source inputs impact our performance significantly. For instance, when sampling source input at random, CPM, fails catastrophically for all evaluation metrics. On the other hand, when sampling source based on the predicted labels using the multi-task probe, CPMμ maintains its performance.
A.7 GPT-3 GENERATION PROCESS
We use the 175B parameter davinci GPT-3 model (Brown et al., 2020) as few-shot learner to generate approximate counterfactual data. Let Zu.o be a review text with an original value for the mediating concept C, and an overall review sentiment y (e.g., restaurant review which is negative about the service, and felt neutral about their overall dining experience), and let c' be the target value of C., for which we would like to create a counterfactual review (e.g., change the text to become positive about the mediating concept service). In order to use GPT-3 an n-shot learner,
sample
o' ). where Tu' shares with Fu,v the same n 6 approximate counterfactual pairs (Iar,e's -.:tdd
value e for C, and the same overall sentiment, and the counterfactual review =C,--e' has the target "0, value ć' for C, We prompt the model with these pairs, and we also include the original review Ta..- We then collect the text completed by GPT-3 as the GPT-3 counterfactual review. An example for this shohot prompt and completion in Figure 3. In addition, we also prompt GPT-3 with pairs of original reviews and metadata-sampp sampled counterfactuals, and generate another set of GPT-3 counterfactual review for comparison. We sample n = 4 approximate counterfactual pairs in this case. An example of metadata-sampled counterfactual generation with GPT-3 can be seen in Figure 4.
18

---

Manuscript
Make the following restaurant reviews include POSITIVE mentions of SERVICE.
Original: had two casual dinners at State & Lake and three lunches. The food was great but the service was lacking. Everything was delicious. The interior is questionable, but not intrusive.
POSITIVE mentions of SERVICE: I had two casual dinners at State & Lake and three lunches. The food and the service were always great. Everything was delicious. The interior is questionable, but not intrusive.
Original: Food was excellent, but the service was not very attentive. Noise level was extremely high due to close proximity of tables and poor acoustics.
POSITIVE mentions of SERVICE: Food and service excellent. Noise level was extremely high due to close proximity of tables and poor acoustics.
Original: Great food, poor and very snobbish service.
POSITIVE mentions of SERVICE: Great food, very good service.
Original: My dining experince was excellent! However, the server was not nice.
POSITIVE mentions of SERVICE: My dining experince was excellent!
Original: Hae been here few times and it is just okay Entrees and wine list a bit pricey for what it is, inattentive staff.
POSITIVE mentions of SERVICE: Hae been here a few times and it is just okay Entrees and wine list bit pricey for what it is. Food comes out on time.
Original: Tables fairly close together, mushroom appetiser very good, pork entree fair, chicken good. The service was terrible.
POSITIVE mentions of SERVICE: Tables fairly close together, mushroom appetiser very good, pork entree fair, chicken good. The service was great however.
Original: Service very poor with the server unresponsive and misinformed on all requests. The food was very good with a good selection of entrees. The ambiance was romantic with quiet excellence.
POSITIVE mentions of SERVICE: Service was very good with the server attentive and responsive on all requests. The food was very good with a good selection of entrees. The ambiance was romantic with a quiet excellence.
Figure 3: Example GPT-3 prompt (gray) and GPT-3 completion (bold). Note that all original examples convey the same sentiment towards service negative) and same overall sentiment (y neutral), and that the counterfactual examples are all edited such that the sentiment towards service is the same (c' = positive).
For each few-shot learning prompt, we insert an initial string of the form of "Make the following restaurant reviews include c' mentions of C,.", where c' is expressed as one of ("POSITIVE" "NEGATIVE", "NOT" ("NOT" corresponds to making the review be unknown regarding the concept C.) and C; is one of ("AMBIANCE", "FOOD". "NOISE", "SERVICE"), J. We sample using a temperature of 0.9, without any frequency or presence penalties (since we expect the counterfactual review to be similar to the original review). In preliminary experimentation, the mediating concept and target value results and inserting line breaks between examples made for better completions, although there is room for future research in this area.
found that capitalizing
19

---

Manuscript
Make the following restaurant reviews include POSITIVE mentions of SERVICE.
Original: Been here several times. Always winner, except for the tasteless food!
POSITIVE mentions of SERVICE: I was very disappointed in the food but we did not wait long for each course and or waiter was very pleasant.
Original: food was decent but not great.
POSITIVE mentions of SERVICE; Lovely evening good service and wonderful food. Perfect for fresh fish fans
Original: The restaurant empty when arrived, reservation not necessary?
Wine list limited. Food was bland, presentation was very well done. I would not eat here again.
POSITIVE mentions of SERVICE: Abby provided the best service that we've ve had after probably two dozen visits. No thank you for making the risotto cake at lunch.... Two Stars!
Original: terrible place for lunch or dinner. All the food excellent with top notch ingredients
POSITIVE mentions of SERVICE: Excellent Valentine's menu. Excellent service and food. Would recommend this restaurant and will return.
Original:
The food was average for the cost. My husband and were so excited visit Bobby Flay's restraunt and were really disappointed. The food was average at best.
POSITIVE mentions of SERVICE: The service was amazing and the food was alright.
Figure 4: Example GPT-3 prompt (gray) and GPT-3 completion (bold). Note that all original examples convey the same sentiment towards service (c unknown) and same overall sentiment (y negative), and that the counterfactual examples are all metadata-sampled such that the sentiment towards service is the same (c' = positive).
We used the OpenAl API to access GPT-3. cost of creating our counterfactuals (around 4,000 examples) was approximately $50 per approximate counterfactuals creation strategy.
3. At the current price rate of $0.02 per 1,000 tokens, the total
A.8 INTEGRATED GRADIENTS
We adapt the Integrated Gradients (IG) method of Sundararajan et al. (2017) to qualitatively assess whether CPMj learned explainable representations of mediated concepts its intervention sites. The IG algorithm computes the average gradient from the model output to its input by incrementally interpolating from a "blank" input x' (consisting only of "[PAD]" tokens) to the original input æ. Eqn. 16 is the integrated gradients equation originally proposed in Sundararajan al. (2017), applied to CPM model p on input x.
8P(x' + (x x²)) do Øx,
IntegratedGrads, (x) -x; -a/=
(16)
0P(x) is the derivative of on the jth dimension of x. Here, day
In our implementation of IG, we wish to show the per-token attribution of input a on the model's final output P(x), mediated by the hidden representation of concept in P. That is, we'd like to ask,
20

---

Manuscript
Model
Prwllod Concept Soon
Woed
arbiacce
lood
Decocations:
Wnfles
had
Delicasss
SEP)
Black box sesmn!
fand
+10.1
(C1.5)
Tec meaee 1o0 (ie) load
and dicoraation korm toat ssico
Fer thry
had
Warton
+0.01
1C1 5] 1h musie Kic too
dicommatios worm toae estro
thay
Wather
post
(577)
artbeence
-0.61
(1 the (masi Mis 100) load
and the Recoration were Taste #nfes
bor (ley
bad
watter (9% and Belcions patt (SEP)
-0.08
[C1.5] the mude Vic 10s
for they
had
Wateer
patt
(112)
they mondlyy
pusts
+1.79
[C1.5] the
decomation
esico
had Incrally
and
(klicaoze
(52)))
Table 10: Additional visualizations of word importance scores using Integrated Gradient (IG) by restricting gradients flow through corresponding intervention site of the targeted concept. This table extends Table 4 in the main text.
"What the effect of the word 'delicious' in the input on the model's 's output, when we restrict our focus only on the model's representation of the concept food?"
To answer this question, we compute the gradient of the model output P(x) with respect to the input a but restrict the gradient to flow through the intervention site for particular concept. This allows us to capture the per-token attribution of the model's final output (whether particular words contributed to positive, negative, or neutral sentiment prediction), mediated by the concept that is represented by the specified intervention site. For example, Table 4, we can see that "delicious" has a positive attribution to the output of the model when we focus on its representation of the concept food.
Formally, consider trained CPM model P. an input x and mediating concept C. Let HCc be the activation of P at the intervention site for Cj. We define the gradient of P(x) along dimension mediated by C., as
0P(x) 0r,
0z(z) aH
mediated by C. 0HC
(17)
0r,
Eqn. 17 restricts the gradient to only flow through the hidden representation of the concept along which we'd d like to interpret our model.
We integrate these mediated gradients over straight path between input and baseline a', analogous to Eqn. 16. We implement our IG method using CaptumAI library. 4 We use the default parameters for runs with number of iterations set to 50, and set the integral method as gausslegende flag to True. To visualize individual word importance, we conduct
We set the mul multiply-by-inpue z-score normalization of attribution scores over input tokens per each concept, and then linearly scale scores between [-1, +1].
Table 10 extends Table in our main text with additional ablation studies on our training objectives.
A.9 MODEL DEBIASING
Being able to accurately predict outputs for counterfactual inputs enables explanation methods to faithfully debias model with regard to a desired concept. For instance, with CEBaB, debiasing a concept (e.g., "food") is equivalent to estimating the counterfactual output when we set the concept label for a concept to be unknown.
In this section, we briefly study the extent to which the CPMμt can function as debiasing method. To debias a concept, we enforce the sampled source input s as in Eqn. 2 to have unknown as its concept label for the concept to be debiased.
To show our methods can faithfully debias a targeted concept, we evaluate the correlations between the predicted overall sentiment label for sentences and the concept labels for each concept. Without any debiasing technique, we expect concept labels to be highly correlated with the overall sentiment label (e.g.. g., if food is positive, it is more likely that the overall sentiment is positive). We use CPM trained for the BERT model architecture as an example, and use examples in the test set.
"nttps://captuur ai/
21

---

Manuscript
ambiance
food
noise
service
Finetuned
10
1.0
10
cutE 00.3
curr=0
curr
0.33
1.0
14
1.0
1.0
CPMμi
corr-0.11
cor==0.31
corr
0.22
0.01
110
04
0.0
0.0
nog
pos
nog
unk
pos
neg
unk
pos
nog
pos
(a) Visualization for debiasing the ambiance concept.
amblance
food
noise
service
CPMHi Finetuned
1.0
14
1.0
1.0
cort
0.3
cocs 0
corr
corr-0.22
corr-0:30
0,0
6.0
0.0
1.0
comr-0337
corr=0.37
-011-0.28
corr-0.51
0.0
nog
unk
pos
neg
unk
pos
nog
unk
pos
nog
unk
pos
(b) Visualization for debiasing the food concept.
ambiance
food
noise
service
Finetuned
1.0
0.3
corr=
curr=0.2
cucr=0.39
0.5
8.5
0.0
CPM
corr-Ols
0.5
0.3
0.5
0.3
neg
unk
pos
neg
unk
unk
neg
unk
pos
(c) Visualization debiasing the noise concept.
amblance
food
noise
service
Finebuned
corr-0.22
0.5
0.3
8.5
0.3
CPМμ
0000
0orr-0.35
c0ry-6.28
corr-0.12
0.12
6.5
0.5
nog
unk
pos
neg
unk
pos
neg
unk
pos
neg
unk
pos
(d) Visualization for debiasing the service concept.
Figure 5: Debiasing visualizations for different concepts of CPMμ with BERT model architecture. Individual plots are correlation plots between concept labels concept and the overall sentence sentiment label.
Figure 5 shows correlation plots for the black-box model as well as CPMmn- expected, the correlation of the food concept is weakened through the debiasing pipeline by 57.50%. Our results also suggest that correlations of other concepts are affected, which suggests a future research direction focused minimizing the impact of the debiasing pipeline irrelevant concepts. We include results for the remaining concepts in the Appendix A.9.
Figure 5a Figure 5d show debiasing visualizations for three concepts: ambiance, noise and service. We use a CPMμt for the BERT model architecture as an example. We calculate the distributions with examples in the test set.
22

---

Manuscript
A.10
LEARNING DYNAMICS
L2
Cosine
Nofm
Macro-F1
0.7
0.2
best
0.6
Epoch Number
Figure 6: CEBaB scores measured in three different metrics on the dev and the test sets for CPMH with the BERT architectures for different training epochs. Task performance as Macro-F1 score is reported.
Figure 6 shows three different metrics measured on the dev and the test sets for a CPM trained for the BERT model architecture our training process, we find our CPMμt reaches a local minimum on COS while L2 and NormDiff are still trending downward. This suggests future research may need to choose desired metrics to optimize for during training, for early stopping to reach the best performing model.
example. Since we use COSIcacE on the dev set to early stop
Eyech
Peeiis
Reparence
fricady
sraino
Soal
+0.55
(CL.5
IEAiI vou Jon load and the docorndicss Wabe bake Mil
but they
Iad inieady
(S23))
sarvice
(C.55
IEARE 195A load and the docorndies Wase take pii
3od mendy
watir 48
Sdideas
(S22))
-125
(CL.5
Eocorations
kad Intaddy
(S3P))
docendicns wao
friadyy
wake
(SEP)
-121
docorations Ait
bad Iniedd
decoestioes Iote suz mioo
water
(S0P))
-B:49
(C5.5)
VAACE V06
Eecorstiies wase sia MiN
had
mindy
water ea
-1:77
(CT.5)
1r6d 196
285 Eecorstiies Wite min
had
mindy
water 00 and
Sikkeaa (933)
(C1.5)
RAME
t2 docorsdions KaN Mia
meads
ddiiioas
(802))
ba mendy
-102
(C.5)
feconsices knl Iode mis.
Mexdy
waler
Mikiiess posa
(SFP)
deliceas
Mindy
(919))
Soul
+1.25
(CT.5 mane
and O2 (ecoesices wel mio
the
hed
Mexdy
water
(SEP)
1.229
(C.55 Ieni
and Dz (ncordices wel0 hate 1Mo
ther
had
60. and ddikieas (SEP)
10022
(C..5 In RAAi via 1as load
lb: Coneedice NoW Lake vil.
Ind
Iniaay
64.
ddiddea pale (SEP)
deconsices
sarviz
12:4
(C.5) is MA6I via
100 loal
and t2: doneedice Wow Lake Ail. dic lad
waki
part (SEP)
-601
(CL.5 JEARE leal
kad
Iniadd Haki
(SEP)
doceedics Wew
(riedy
wake
patt (SEP)
-1.44
and
doomiicni R20
knd Inaadd
Adkiiaa
(SEP)
43.25
(C1.5)
994 100
and D2 docrentions KEN tant 19iz
thyy had
waker
(diideas pura (S83))
Table 11: Visualizations of word importance scores using Integrated Gradient (IG), using the same methods as in Table 4 and Table 10.
Table 11 visualizations of word importance scores using our version of Integrated Gradient (IG). Different from Table 4 and Table 10, which show the visualizations of our optimized model, we show a per-epoch result for for CPMHt, followed with our best model appended at the end. Our results suggest that early checkpoints in the training process focus at drastically different input words comparing to later checkpoints, though all models predict neutral for this given sentence. In addition, gradient aggregations over input words are rather stable towards the end the training. More importantly, CPMji learns how to highlight words that are semantically related to each concept gradually. For instance, we see clear trend of emphasising the word "decorations" for the ambiance concept throughout the training process. This suggests that our training procedure induces causally motivated gradients over input words gradually through the training process.
23


Reviews:
[{"review_id": "TWnatd9ffGy", "reviewer": "Reviewer_LSZt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a robust explanation method for NLP tasks using Causal Proxy Model (CPM). Given a black-box model to be explained, the CPM tries to simulate both the factual and counterfactual performance of that model. With CPM, one can have 1) an explanation of the black-box model, 2) comparable factual performance and 3) learn concept-level representation in the hidden code. The model is evaluated on the CEBaB benchmark. ", "summary_of_the_review": "Overall, the paper is well-written and the story follows naturally. The proposed method seems interesting and novel to me. In particular, the paper builds the insight that CPM can be directly used in the replacement of a black-box model. However, one potential limitation is that it builds on a known causal graph, which is usually unavailable in the world. ", "strength_and_weaknesses": "Strength:\n1) The paper proposes to use CPM to approximate the factual and counterfactual performance of a black-box model, which is an interesting idea. \n2) The learned CPM can perform comparably to the black-box model in terms of factual performance. \n3) CPM_HI generates a conceptual level of understanding of the black-box model. \n4) The paper provides the insight to directly use CPM as the deployed model and explainer. \n\nWeaknesses:\n1) The causal graph in Figure 1a is particularly for CEBaB data set. Does the method work without knowing the causal graph? My understanding is that one has to know the causal graph so as to use CPM. This can be its limitation. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "P10t85doJT1", "reviewer": "Reviewer_9V3t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to fine-tune a black-box model that is counterfactual consistent.\nFirst, they obtain the true counterfactual input via either: (1) from the dataset explicitly by a human annotator or (2) by sampling the nearest sentence from the dataset using intervention to a specific concept. Next, they employ the Causal Proxy Model (CPM) -- which is a copy of the Black-box to (1) mimic the BlackBox for the factual input, (2) create a neural representation to be intervened to approximate the BlackBox for the counterfactual input. While simulating the counterfactuals, they use the true counterfactual text as input to predict the counterfactual output from the BlackBox. However, to mimic the counterfactual output from the BlackBox, they use either of the following strategies: (1) append a token to the factual text using the intervened concept to approximate the true counterfactual input and train CPM, (2) employing interchange intervention training (IIT) by Geiger et al. (2022) to intervene at the hidden neurons of CPM that get affected due to the intervention of the concepts.\n", "summary_of_the_review": "As mentioned in the first point in the weakness, the authors generated consistent counterfactuals. However, they did not substantiate with their experiments how these counterfactuals will help in explanation. Some of the choice experiments are not well justified and may not be appropriate for the paper's goals. Overall, I disagree with the authors (1) about calling this method an explanation method , (2) calling the approach *causal* .\n", "strength_and_weaknesses": "# Strength\n1. Having a model that is counterfactually consistent is a reasonable goal when the model us to interact with humans. \n 2. the experiments demonstrated that the proposed CPM performs better than the baselines for various architectures.\n 3. Their experiments also showcased that the CPM can be consistent with the BlackBox and can even replace the BlackBox for estimating factual and counterfactual outcomes.\n\n# Weakness \n1. My biggest concern with this paper is motivation and how it is presented. The overall aim is to have counterfactual consistency, meaning the output should change reasonably by changing the counterfactual concept. It is not clear to this reviewer why they call their method an explanation. The so-called CPM is a fine-tuned version of the original model (with the same arch.), hence using the integrated gradient to explain CPM is not the same as explaining the original model.  Overall, I do not agree with the authors that this is an explanation method. They did not compare with some existing explanation strategies like TCAV, demonstrating how these counterfactual explanations serve a better purpose. They should rename the paper if the original aim is to predict good counterfactuals.\n\n 2. There are unnecessary notations in the paper that can be easily simplified. For example, they only use $U$ and $V$ in the causal diagram in figure 1, but they hardly use them in the paper. $x_{u, v}$ can simply be $x$. All the concept level interventions can be shown using $x^{C_i = \\hat{c}}$ and with similar form. They should mention this clearly in the paper if they have any specific reason to show $U$ and $V$. Beyond using the notion of counterfactual, causality also doe not play a role in this paper. Also, the name *proxy* is used for a different thing in causality literature, so it is quite confusing. \n\n 3. For figure 1-d) $\\mathcal{L}_{IN}$: It seems that the authors append a token to the factual text using the intervened concept to approximate the true counterfactual input. This concatenation can be problematic because of the conflicting concept in the input text and the generated token from the intervention. It is not clear to me why appending the counterfactual token is a good idea. If the authors are not appending a token and doing something else, it is not clear from the paper.\n\n 4. The notation in equation 8 is ambiguous. The authors should specify what does $C_i \\leftarrow c'$ indicate? If they meant intervened concept $C_i$ is appended to $\\mathcal{B}(x_{u,v})$, they should use consistent notation in equation (2) or figure 1-d. Do they mean to append using a special concept token?\n\n 5. The authors should clarify how they intervene in the intermediate layers of the transformer-based models. Did they intervene at the concept level in the CLS token of the intermediate layers? This is not clear from the paper.\n\n6. One of my main criticism is their choice of evaluation metric. Yes, it makes sense to ensure that the fine-tuned model does not hurt the performance of the original model (and there are experiments to show that in the paper) however, correct estimation of the ITE neither proves a better explanation nor a more useful fine-tuned model. So it is not clear to me why they chose to report ITE.\n\n\n 7. In many places (like table 4), they term the original BlackBox as 'Finetuned'. Can they specify the reason?\n\n 8. In fig 4, they should use a color bar.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NysZMa8kgfI", "reviewer": "Reviewer_MZ4t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method called Causal Proxy Model (CPM) for explaining a black-box model $\\mathcal{N}$ that makes use of available counterfactual training data.", "summary_of_the_review": "This paper is a well-written paper. However, its problem is not very significant and limited to certain cases, and its proposed solution is quite straightforward. Thus, I think the paper is below the acceptance bar of ICLR.", "strength_and_weaknesses": "### Strengths\n1) I think the main strength of the paper is that it is well-written and easy to follow.\n2) Learning a model that enables the explanation of a pretrained black-box model has a major practical impact. However, the proposed method is limited to a particular scenario in which counterfactual data is available.\n\n### Weaknesses\nI) About the motivation and idea:\n1) In the Abstract and Introduction, the authors argue that *“the fundamental problem of causal inference is that we hardly observe the counterfactual inputs”* but the proposed method, by design, assumes (approximate) counterfactuals are available. This limits the practical application of the proposed method to certain datasets (e.g., the CEBaB dataset in the paper) and cases.\n2) I do not see any discussion about the clear drawbacks of existing methods for explanation in comparison with the proposed method.\n\nII) About the proposed method:\n1) In my opinion, the proposed method is not very novel. Given counterfactual data $\\tilde{x}^{C_i \\leftarrow c'}\\_{u,v}$,  it is quite straightforward to think of matching $\\tilde{x}^{C_i \\leftarrow c’}\\_{u,v}$ with $x_{u,v}; t\\_{C_i \\leftarrow c’}$ for convenient intervention.\n2) A limitation of CPM is that it does not account for the stochasticity of $\\tilde{x}^{C_i \\leftarrow c’}\\_{u,v}$. $x_{u,v}; t\\_{C_i \\leftarrow c’}$ only yield one value but the corresponding counterfactual text $\\tilde{x}^{C_i \\leftarrow c’}\\_{u,v}$ can be abundant.\n\nIII) About the presentation:\n1) The overall writing is good but the descriptions of many (mathematical) terms in the paper are not clear, causing difficulty in understanding the method. I would like to discuss some of them below:\n- I don’t really understand what is the *“description of a concept intervention $C_i \\leftarrow c’$”*. Does it mean $C_i$ will take one of three categorical values {negative, unknown, positive} or a text that describes the concept $C_i$? Can we just call $C_i\\leftarrow c’$ a concept intervention for simplicity?\n- What is the token $t\\_{C_i \\leftarrow c’}$ used in the paper? Would the authors please provide some concrete examples of this variable as I could not find such thing in the paper? Is it possible to just write $t\\_{c’}$ instead of $t\\_{C_i \\leftarrow c’}$?\n- The objective $\\mathcal{L}\\_{\\text{mimic}}$ seems to be shared between $\\text{CPM}\\_\\text{IN}$ and $\\text{CPM}_\\text{HI}$ and should be put above the paragraph that describe $\\text{CPM}\\_\\text{IN}$. The authors should also write an overall loss, e.g., $\\mathcal{L} = \\lambda_1 \\mathcal{L}\\_{\\text{mimic}} + \\lambda\\_2 \\mathcal{L}\\_{\\text{counterfact}}$ where $\\mathcal{L}\\_{\\text{counterfact}}$ can be either $\\mathcal{L}\\_{\\text{IN}}$ or $\\mathcal{L}\\_{\\text{HI}}$\n2) The Structural Causal Model (SCM) described in Figure. 1a does not really make sense to me. I cannot figure out what are treatment, covariate, and outcome from this figure. From what shown in the figure, I assume $C_1$, …, $C_k$ are different treatments, X is an outcome, and no covariate. But it turns out X is actually a covariate, and the outcome is a 5-star sentiment score Y which does not appear in the causal graph. I suggest the authors to redraw the SCM to make it correct, adding Y and removing U, V as they do not have any contribution to the model.\n3) For the S-learner baseline described in Section 4.3, I don’t really understand how the authors actually perform intervention with the output of the BERT model $\\mathcal{B}$. In other words, how is  $\\mathcal{B}(x_{u, v})_{C_i \\leftarrow c’}$ implemented?\n\nIV) About experiments:\n1) Please provide references for methods used in $\\text{BEST}_{\\text{CEBaB}}$.\n2) S-learner considered in the paper seems not a strong baseline since the intervention is only done for the output of the BERT model on the FACTUAL text $x_{u, v}$ which means “late fusion” between covariate $X$ and treatment $C_i$. Meanwhile, CPM fuses $X$ and $C_i$ early, which allows the method to model the interaction between $C_i$ and $X$ better.\n3) The authors should compare their method with other causal inference methods such as X-learner [1], TARNet [2], and Balancing Linear Regression [3] in a fairer setting. The authors should also compare their method with other methods for explanation discussed in the paper. \n4) In this paper, the authors consider $\\mathcal{N}$ as a black-box model but in the experiment, they initialize $\\mathcal{P}$ from the weights of a pretrained $\\mathcal{N}$. This means $\\mathcal{N}$ is no longer black-box anymore. I suggest the authors to do experiments with $\\mathcal{P}$ different from $\\mathcal{N}$ (e.g., $\\mathcal{N}$ is BERT and $\\mathcal{P}$ is GPT-2 or LSTM). This will make the method more convincing.\n\n[1] Metalearners for estimating heterogeneous treatment effects using machine learning, Kunzel et al., PNAS-2019\n\n[2] Estimating individual treatment effect: generalization bounds and algorithms, Shalit et al., ICML-2017\n\n[3] Learning Representations for Counterfactual Inference, Johansson et al., ICML-2016", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KnhTRQ--Ewa", "reviewer": "Reviewer_aAEJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Causal Proxy Models, which are trained to mimic outputs of a given model for original inputs as well as for alternative (counterfactual) versions of those inputs that differ based on modification of a particular (sentiment) feature. They use data from the CEBaB dataset for creating human-annotated and sampled input pairs, and train their models on these data. The models involve either a special token indicating change of a particular feature, or representation intervention on representations trained to store causally-relevant information for a given feature. The authors report that their methods achieve state-of-the-art performance on the CEBaB dataset. They also show that the models can explain their own behavior, have performance on par with the original models, and that the representations trained to capture certain features do show reasonable feature attributions.", "summary_of_the_review": "The proposed methods are interesting, and show improvement on the specific benchmark being used. However, I have concerns about the narrowness in applicability of the method (not clear it's useful beyond that specific benchmark) and I'm not clear how much insight it provides as a contribution to explainability.", "strength_and_weaknesses": "Strengths: The paper presents an interesting set of methods, which are explained in the paper reasonably clearly, and which show state-of-the-art performance on the CEBaB dataset. The authors include a couple of follow-up analyses that help to flesh out the overall picture of the methods' outcomes and potential impact.\n\nWeaknesses: My main concern with the paper is that the methods feel very narrowly focused on improving on the CEBaB dataset, which seems like an interesting task, but which casts explanation in a very particular way. So there are two subparts to this concern. First, the way the proxy models are trained feels very specifically targeted toward improving on CEBaB -- aka, leveraging the fact that the task is focused on predicting how models will behave on inputs edited for particular sentiment features -- so although the methods are interesting, I'm not sure how surprising I find it that they were able to improve over other methods, and I'm also skeptical about how generalizable this method would be for explanation on other types of domains/tasks/explainability settings.\n\nThe second subpart of the concern relates to the last point: while the authors are targeting an existing benchmark that has been introduced for explainability, and they show improvement on that benchmark, it's just not clear to me from the paper in what way their method is actually resulting in interpretable/usable insights for explaining the behaviors of the models being approximated. What they show is that their methods allow them to use feature tokens or representational interventions to predict how target model predictions will change if an input has been changed along a particular sentiment feature dimension. But it's not clear to be how this help us to understand what strategies the models are using, what features are influential for prediction, what potential heuristics might cause untrustworthy behavior, etc -- the types of things that I expect from an explanation method. I'm happy to believe that there is more insight here than was made clear, but at the moment I don't feel the paper makes a strong argument for how it is providing insights into the target models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject