Explanations that reveal all through the deﬁnition of encoding
Authors: Aahlad Manas Puli
Nhi Nguyen
Rajesh Ranganath
Venue: NeurIPS 2024 poster (2024)
Keywords: feature attributions
model explanations
evaluating explanations
encoding the prediction
interpretability
Primary area: interpretability_and_explainability
TL;DR: We formalize the definition of encoding in explanation methods and provide two methods to detect encoding.

Abstract: Feature attributions attempt to highlight what inputs drive predictive power. Good attributions or explanations are thus those that produce inputs that retain this predictive power; accordingly, evaluations of explanations score their quality of prediction. However, evaluations produce scores better than what appears possible from the values in the explanation for a class of explanations, called encoding explanations. Probing for encoding remains a challenge because there is no general characterization of what gives the extra predictive power. We develop a deﬁnition of encoding that identiﬁes this extra predictive power via conditional dependence and show that the deﬁnition ﬁts existing examples of encoding. This deﬁnition implies, in contrast to encoding explanations, that non-encoding explanations contain all the informative inputs used to produce the explanation, giving them a “what you see is what you get” property, which makes them transparent and simple to use. Next, we prove that existing scores (ROAR, FRESH, EVAL-X) do not rank non-encoding explanations above encoding ones, and develop STRIPE-X which ranks them correctly. After empirically demonstrating the theoretical insights, we use STRIPE-X to show that despite prompting an LLM to produce non-encoding explanations for a sentiment analysis task, the LLM-generated explanations encode.

Explanations that reveal all through the definition of encoding
Aahlad Puli; Nhi Nguyen',
Rajesh Ranganath
New York University
Abstract
Feature attributions attempt to highlight what inputs drive predictive power. Good attributions or explanations are thus those that produce inputs that retain this pre- dictive power, accordingly, evaluations of explanations score their quality of pre- diction. However, evaluations produce scores better than what appears possible from the values in the explanation for a class of explanations, called encoding ex- planations. Probing for encoding remains a challenge because there no general characterization of what gives the extra predictive power. We develop a definition encoding that identifies this extra predictive power via conditional dependence and show that the definition fits existing examples of encoding. This definition im- plics, in contrast to encoding explanations, that non-encoding explanations contain all the informative inputs used produce the explanation, giving them "what you see is what you get" property, which makes them transparent and simple to use. Next, prove that existing scores (ROAR, FRESH. EVAL-X) do not rank non- encoding explanations above encoding ones, and develop STRIPE-X which ranks them correctly. After empirically demonstrating the theoretical insights, STRIPE-X to show that despite prompting an L.I.M to produce non-encoding expla- nations for sentiment analysis task, the L.MM-generated explanations encode.
Introduction
Artificial intelligence unlock information in data that was previously unknown. In medicine, for example, using AI, researchers have shown that electrocardiograms are predictive of structural heart conditions [1] or new-onset diahetes [2]. Good predictions often lead to ask what in the input is important for a prediction; this question is driving factor behind research in interpretability and explainability [3, 4]. One primary direction interpretability seeks produce explanations that are subsets of the input that retain the predictability of the label. These types of explanations and interpretations are called feature attributions and have been used to find factors associated with debt defaults [5], to demonstrate that detecting COVID- 19 from chest radiographs can rely on non- physiological signals [6], and discover a new class of antibiotics [7].
Several methods exist for producing feature attributions or explanations. While some methods com- pute functions of model gradients [8] or look predictability after removing features [9], other methods attribute scores to different inputs by treating them as players in a game [4, 10] or amor- tize their explanations by learning single model select subsets for each instance [11]. Choos- ing one from the many feature attribution methods requires an evaluation. There are, however, many approaches to evaluation itself: qualitative ones [12, 13, 14], which are limited to cases where humans have precise knowledge about the inputs relevant to prediction, and quantitative ones [2, 4, 15, 16, 17, 18, 19, 20], which do require human knowledge.
Intuitively, a good evaluation method for feature attributions should assign higher scores to explanations that select inputs that are score explanations based on the predictability of the label from the explanation face one major challenge: encoding. Informally, an encoding explanation is the label beyond what seems plausible from the values of the inputs themselves. The top left panel of Figure shows an explanation that predicts the label of dog or cat depending on whether the
predictive of the label. However, evaluations that
where the explanation predicts
'qqual contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Producing Explanations
Evaluating Explanations
Encoding Problem Explanation
Inputs
Explanation method
Selection
Explanation
Prediction model
Label pred.
Matric
Label
Selection
Acc.
10:0
given
Non-encoding Explanations
Encoding Detection
Label
Explanation
Label
Selection
Values
Non-ddetector
Weak detector
Strong detector (tte blil 1-40
HSAM (19) 108.3
Opt
Opt
Opt
Nen-ensoding
Inguuls
Soote
Soore
Scote
Label
Values
Norveno Nanccccciing
Enooding
Entodinng
Looodingg
Figure 1: Overview of the paper. Explanations produced to find inputs that relevant to predicting label. However, explanations can predict the label well due the selection being predictive of the label beyond the explanation's 's values. Such explanations are called encoding. In contrast, predicting instead from a non-encoding explanation equivalent to predicting from the values in the explanation. When explanations are evaluated purely based on the quality of prediction, encoding can undetected. We classify existing evaluations into non-detectors and weak detectors and develop strong detector, called STRIPE-X.
explanation a pixel on the right half or left half of the image respectively. Many explanation methods fit the description of encoding [20, 21]. Further, given that many evaluations only look at the quality of prediction, encoding can go undetected, rendering the evaluations ineffective at picking explanations. In contrast, non-encoding explanations predict the label well only when the values in the explanation do, making them easy to reason about.
In addressing encoding, this work makes the following contributions:
Develops a simple statistical definition of encoding via conditional dependence property.
Confirms the introduced definition captures all existing hoc encoding instances.
Shows that non-encoding explanations are easy to use because they retain all the predictive inputs used to build them, meaning that predictive non-en non-encoding explanations reveal imputs that predict the label to their users, and thus have a "what you see is what you get" property.
Formalizes evaluations' sensitivity to encoding weak detection (optimal scoring explanations are non-encoding) and strong detection (non-encoding explanations score above encoding ones).
Demonstrates that the evaluations ROAR [19] and FRESH [18] do not weakly detect encoding.
Proves that EVAL-X [20] weakly detects encoding, but does not strongly detect encoding.
Develops STRIPE-X and proves that strongly detects encoding.
Uses STRIPE-X to show that despite prompting an LLM to produce non-encoding explanations for sentiment analysis task, the LLM-generated explanations encode.
Figure provides an overview of this paper.
Evaluating explanations
We focus on explanation methods where the goal is to produce subsets the input that predict the label [22, 23]. Explanation methods of this form, also called feature attributions, saliency methods [4, 8, 24], or just "explanations, include thresholded rankings from Shapley values [25, 26]. LIME [24], and REAL-X [20]. With y as the label and x E Rd as the inputs, let g(y.x) x) be the joint distribution over them. An explanation method maps the inputs x to a binary selection mask e(x) the inputs: e Rd (0, 1)d. The explanation xe(x) is pair: the selection e(x) and the vector of explanation's values. For example, if x [a, b, is dimensiinnn and e(x) [0. 1, 1), Xe(x) consists of the binary mask e(x) and the values associated with the inputs that correspond to the indices in e(x) with value
Xe(x) (e(x), [6, c)).

---

We keep track of the indices because the same value can lead to different predictions depending on the index appears at: for example, predicting mortality from patient vital signs, a heart rate above 110 can occur in healthy patients but temperature of 110 is almost always fatal. Equivalently, like in existing work [15, 16, 17, 20), one can choose xe(x) to retain the values in the explanation in the same position and mask out those not selected: Xe(x) e(x) x (1- e(x)) mask-token. For concision, we overload the word "explanation" to mean the explanation method instead of the random variable Xx(x) when is clear from context.
Choosing between explanation methods requires evaluation. Explanation methods seek to return inputs that predict the label, so existing evaluations consider how well the explanation xx(x) predicts the label y [15, 16, 17, 20]. To score explanations based on predictive power, an evaluation method a()) takes as arguments both the explanation e(x) and the joint distribution g(y, x): a(g,e). e). Without loss of generality let higher be better.
2.1
Encoding: A disconnect between the predictiveness of explanations and the predictiveness of their values
We give simple example of encoding to build intuition for the disconnect between predicting the label from the explanation and predicting the label from the explanation 's values. Imagine that the goal is to explain which set of vital signs signal bacterial pneumonia as the diagnosis compared to the common cold. Consider the explanation method that selects the patient's 's height when the true probability of pneumonia is high given the whole set of observables (including labs, symptoms, and vital signs) and otherwise selects the patient's 's hair color. Physiologically, height and hair color do not indicate that the patient has pneumonia, meaning that this explanation should not be highly predictive of the label. However, by construction, pneumonia is likely exactly when the explanation selects height, and predicting the label from the explanation achieves the same accuracy as predicting x). Thus, despite the explanation
with the full conditional arg maxye (pneumonia, cold) q(y method only selecting physiologically irrelevant inputs, the explanation predicts the label well.
Encoding examples such as the one above are neither contrived nor unique. For example, Jethani et al. [20] show that certain procedures that learn to explain, when applied to MNIST digit classifi- cation, yield explanations that select background, black pixel that predicts the label at an accuracy 90%; (see Figure 1 in [20]). Other examples of encoding explanations that predict better than what is expected from the explanation's values exist [20, 21]. Encoding explanations should not score optimally under a good evaluation because the explanation selects inputs that do not appear to predict the label. However, without a general characterization of the discrepancy in predictive power for encoding, finding explanations whose values predict well remains challenge. The next section develops a definition of encoding.
3 Formalizing encoding
Intuitively, encoding a phenomenon where the information about the label in the explanation Xe(x) exceeds what is known from the explanation's values. As the input determines the explanation Xe(x), the quality of predicting the label y from the explanation relies on the information about the label transmitted from Xe(x)- There are two pathways for this transmission; we elaborate below.
Denoting the values in subset v by Xv, compare the event this subset takes the values a, i.e. x, to the event that the explanation's 's selection is v and that the explanation's 's values are i.e.,
xe(x) av.
1. Knowing that the explanation is xe(x) = (V, a) implies not only that the values in the explana- tion are determined as x,, = a, but also that the selection is determined as e(x) = v.
2. In reverse, knowing that the values of a subset of inputs are X, = a and knowing the selection e(x) = v implies that the explanation are Xc(x) (v, a).
Putting these two points together yields an equality between events:
{x Xx(x) = (v, a)} = fx e(x) = v) n {x Xy = a}.
(I)
Thus, the two pathways for information between x and the explanation Xe(x) are the selection e(x) and explanation's 's values x,; see Figure Existing work makes similar intuitive observations but stops short of formalizing the additional predictive power in an explanation xx(x) [20, 21].
3

---

To formalize this extra predictive power, define the explanation indicator E, = [[e(x) v.. A little algebra Appendix A.1 shows the explana- tion indicator Ey provides the extra information:
q(y xx(x) = (v, a)) = g(y Xv = a. Ev - gly Xv = a).
c(x)
Xv
extra information in Xe(x)
Building on this insight, we define encoding conditional dependence: Definition (Encoding). The explanation e(x) is encoding if there exists an S where (xxe(x) E S) > O such that for every a) E ES:
Xe(x)
Figure 2: Intuition for encoding: There are two ways the information in the inputs x about the la- bel y transmitted to the explanation Xe(x): (1) through the values in the explanation and (2) the selection e(x) (in red), When the latter happens, the explanation said to be encoding.
E,
(2)
An example mathematical construction of an encoding explanation pro- vided in Appendix B.1. The dependence in Def: Encoding means that for encoding explanations, there is a disconnect between how well the ex- planation Xe(x) predicts the label versus only the explanation's 'ss values Xv- This disconnect means that evaluations that score explanations based on predictions from the explanation or their transformations [15, 16, 17, 20] can favor explanation methods that select inputs whose values have little relevance to predicting the label.
Beyond the disconnect in prediction, encoding explanations are undesir- able as they conceal predictive inputs that nevertheless affect the explanation. This concealment can lead to incorrect conclusions, such as that inputs outside the selection are irrelevant, or bewilderment because predictive inputs outside the explanation drive changes in the selection in ways that cannot be understood from the explanation itself. An example is provided in Appendix B.2.
Non-encoding explanations. Conversely, for a non-encoding explanation, there exists no positive measure set explanations Xe(x), where the explanation indicator has conditional dependence given the explanation's 's values. That is, for set A where q(xe(x) A) 1, then for all (v, a) E A
y I Ev
Xv
which in turn guarantees
q(y Xe(x) = (v, g(y x, = a, E = 1) g(y xv, aa)
Appendix A.2 shows this. A simple example of a non-encoding explanation is a constant explanation that always picks the same subset of inputs, since constant E, is independent of any variable. The information for predicting the label in a non-encoding explanation lives in the explanation's values. Evaluations based on predictions from the explanation Xe(x) of non-encoding explanations will yield explanations where the input values xy predict the label. In other words, non-encoding explanations reveal all the informative inputs they depend on, and "what you see is what you get" in the explanation.
3.1 Encoding explanations in the wild
Def: Encoding encompasses examples in the existing literature beyond the example in Section 2. 1. In that example, the information about y lies in the positions in the selection e(x), which motivates the name position-based encoding (POST). This section describes two other informal examples from the literature of encoding explanations, prediction-based encoding (PRED) [21] and marginal encod- ing (MARG) [20]. and explains the intuition behind why they encode. In the appendix, we develop formalizations of these types of encoding and show that these formulations meet Def: Encoding.
Prediction- based encoding (PRED). To understand how prediction based encoding occurs, con- sider the task of sentiment analysis from movie reviews. Assume that reviews can either be of type "My day was terrible, but the movie was [ADJ1]." and "The movie was [ADJ2], but the day was not great." " where adjective ADJI can be "good" or "not great" and adjective ADJ2 can be "not great" or "terrible". Due to common English parlance, "terrible" indicates bad sentiment more often than "not great". Then, in the example setup above, only seeing that the fourth word is "terrible" yields bad sentiment with higher probability than when only seeing that the phrase is "not great". However, the fourth word does not always describe the movie. An explanation can look at "not great" describing the movie as bad but then selects "terrible" to encode the bad sentiment. This explanation encodes because the selected word may not describe the movie but the selection predicts the sentiment.

---

Marginal encoding (MARG). This type of encoding occurs when some inputs determine which other inputs determine the label. For example, in Figure 3, the color determines whether the top right patch produces the label or the bottom right patch. Inputs that control where the label comes from are named control flow inputs. For a real-world example, consider the following example from Jethani et al. [20], where the goal is to predict mortality for pa- tients with chest pain. A lab value that checks for heart injury and acts like a control flow input troponin. Abnormal troponin indicates that cardiac issues exist and cardiac imaging would inform mortality. Normal troponin the other hand can indicate that chest pain is unrelated to cardiac health and chest X-ray would instead inform mortality. Selecting one image or the other, but not the control flow input, conceals information about why the image was relevant to the label.
cat
dog
Figure 3: Left: Consider data where the color in the left half determines whether the label "cat", "dog") is produced from the top or bottom image on the right. Right: A MARG encoding explanation that produces only the top or the bottom animal image based the color. The animal image alone says less about the label than knowing the animal image and the color. Knowing the selection determines the color and thus provides ad- ditional information about the label.
Formalization. In Appendix B, provide mathematical formulations of each informal example and show that they fall under the definition of encoding in Def: Encoding: position- -based encoding (Appendix B.3). prediction-based encoding (Appendix B.4), and marginal encoding (Appendix B.5). The key intuition behind all of these is that the explanation e(x) varies with inputs other than the selected ones, and these additional inputs provide information about the label beyond the selected ones. Next, we turn to detecting encoding via quantitative evaluations.
4 Detecting encoding in explanations
This section develops notions of sensitivity to encoding for evaluation methods, and uses the math- ematical definition of encoding developed in the previous section establish which methods detect encoding and which do not. Hsia et al. [21] suggest that evaluation methods like EVAL-X can be gamed to produce high scores for encoding explanations by optimizing the evaluation. To study this case, we introduce the notion of weak detection. If the optimal score of an evaluation of explanations does not permit encoding, then that evaluation is said to weakly detect encoding:
Definition 2 (Weak detection of encoding). An evaluation or(g, e) of explanations weakly detects encoding if the optimal explanations e", i.e. alg, e*) max. alq, e), non-encoding.
Weak detection provides recipe for finding non-encoding explanations: find the explanation that achieves the maximum score of weak detector. However, such a recipe would only work when op- timizing without constraints because weak detection does require non-encoding explanations to have a better score than any encoding one. Requiring this leads to the definition of strong detection.
Definition 3 (Strong detection of encoding). An evaluation ag e) strongly detects encoding if for any encoding explanation e and non- non-encoding explanation e'. alq, e') > alg. e).
Evaluations that are not weak detectors cannot be strong detectors because they score some encoding explanation optimally.
4.1 Do existing evaluation methods detect encoding?
Here, we consider whether several techniques for evaluating explanations: ROAR [19], FRESH [18), and EVAL-X [20] can detect encoding. We analyze these evaluations on the following distribution y
[x
w.p. 0.9 else - X1
if x: = 1,
x = [x,, [2, Xy] ~ B(0.5)83
=
X2 w.p. 0.9 else - X2
if x3 = O.
(3)
Consider the explanation Cencode (x) encodes because X3 is used to create the explanation and X3 predicts the label conditional on x1 when Eg, 1. This is a MARG explanation (see Section 3.1).
[1,0,0 x3 1 and & = [0, 1, 0] otherwise; this
5

---

ROAR and FRESH do not weakly detect encoding. ROAR evaluates explanations by predict- ing the label from the inputs not selected by the explanation, denoted as x-c(x); ROAR scores explanations optimally if the predictions from the remaining covariates are as random as pre- dicting without any covariates at all. In other words, ROAR checks how informative x-e(x) is of and provides the highest score when y-lxx-(x)- In contrast, FRESH evaluates explana- tions by predicting the label from the explanation after removing all other inputs, denoted as explanationn
(x)(x) For example, assume we are given input x "Visually stunning. My fa- vorite movie ever" and an explanation e(x) that selects the words "stumning" and "favorite". Then, the explanation is xe(x) = ((0, 1, 1, 0j, ["]]["stumn ["stunning", "favorite")), whereas vxl(xx(x)) = ("stunning", "favorite", pad-token, pad pad-token, pad-token mation about where the selected words are in the input. See Appendix B.6 for a formal definition of val(Xx(x)) FRESH checks how predictive g(y vx]()x(x))) and assigns an optimal score if the prediction is good that of q(y x). These conditions hold for Cencode (x) in eq. (3):
had-token], which drops the infor-
Proposition 1. For the data generating process (DGP) in eq. (3), ROAR and FRESH assign their respective optimal scores to the encoding explanation Cencode (x).
The proof is in Appendix B.6. The intuition is that the encoding explanation Cencode (x) always se- lects the input that informs the label given the control flow X3; removing the only conditionally informative input means that x (x has no information about y. In turn, ROAR scores encod- ing explanation X X-eaxs(x (x) optimally, meaning it does not even weakly detect encoding. In addition, val (x)((x())) provides the exact same information about the label regardless of which position it came from, As result, x y (xe(x)))) so FRESH scores fencnde (x) optimally. Even though FRESH at- tempts to drop the information about the selection = =(xx during evaluation, (xe(())))) remains function of Xx(x) (v, a). so extra information can still be transmitted through the selection v.
Thus, ROAR and FRESH are not weak detectors of encoding.
EVAL-X weakly detects encoding but not strongly. EVAL-X [10, 26] is an evaluation method and is sometimes called the surrogate model score. The EVAL-X score with log- rg-probabilities is
EVAL-X(q, e) := (( (v(a)=(vaa) [log q (y x,, = a)).
(4)
This score measures the expected log-likelihood of the labels given the input values chosen by the explanation method e and is grounded in the sampling distribution q. Log-likelihoods are maximized by matching the true distribution, this leads to EVAL-X's weak detection:
Theorem 1. If exx) is EVAL-X optimal, then e(x) is not encoding.
Appendix A.4 gives a proof. The proof shows that at optimality, the prediction from the values of explanation has to match the prediction from the full inputs. In turn, given the values there is no additional information in x about y. which means the explanation indicator E, is independent y; this violates Def: Encoding, which proves the non-encoding nature of EVop-iaap explanations.
To test strong detection for EVAL-X, we consider explanations constrained to select one input. Such reductive constraints appear in practice because the goal of producing an explanation is often to aid humans who benefit from reduced complexity. Such constraints prohibit explanations from reaching EVAL-X's optimal score. Compare Cencade (x) with non-encoding constant explanation:
Proposition 2. Lei ea ex(x) Ea. Then, for the DGP in eq. (3), EVAL-X X((x Concnde) EVAL-X ec).
Thus, EVAL-X is not strong detector. The intuition is that the first two coordinates X1, X2 predict the label when selected by fencode, while the control flow feature does not predict the label. EVAL-X not being a strong detector means that optimizing EVAL-X over reductive set may yield an encoding explanation. In this case, Cencode is one of the EVAL-X- -poptmmll reductive explanations (Lemma 6).
4.2 STRIPE-X: a strong detector of encoding
Encoding explanations induce the dependence between the label y and the identity of the selection E, = 1 [e((x) = v] given the values in the explanation xy (Def: Encoding). This dependence can be tested for by building conditional independence tests [27, 28, 29]. Rather than testing, direct quantification of dependence can be useful for when combining with other scores, which can be done using instantaneous conditional mutual information:
(e(e) := (V y Xy = a) SINCODE-MTEERY
(5)

---

ENCODE-METER is 0 only when Def: Encoding does not hold:
Proposition ENCODE-METER 0y(e) if and only if e is not encoding.
The proof is Appendix A.5. Combining EVAL-X with
Method
Weak
Strong
ENCODE-METER weighed by a yields a method we call the
strongly information-penaaizee evaluator (STRIPE-X):
ROAR [19]
FRESH [18]
X
X
(6) For large enough a, the added penalty term pushes down the scores of encoding explanations below that of all non-encoding ones, meaning that STRIPE-X is strong detector of encoding:
STRIPE-X. (q: e) =- EVAL-X(q, e) - (e)
EVAL-X [20]
V
x
STRIPE-X
The weak and strong de- tection properties of different evalu- ation methods. Existing scores like ROAR [19] and FRESH [18], are not weak detectors, which in turn means they are not strong detectors either.
Table 1:
Theorem 2. With finite H(y x) and H(y), for any explanation that encodes and any that does not encode é', there exists an a" such that Vo > a" STRIPE-Xa (9, > STRIPE-Xa
The proof is in Appendix A.5. The intuition behind the proof is that for large enough a, the STRIPE-X scores for any encoding explanations will be dominated by the information term, and thus will become smaller than any non-encoding explanation whose score lower bounded by the negative marginal entropy. (y). Table summarizes the weak and strong detection properties of different evaluations.
Estimating STRIPE-X. The first component of STRIPE-X is EVAL-X. Computing EVAL-X (eq. (4)) requires an estimate of the predictive distribution of the label y given Xy. q(y xv) [20]. Estimation can be done in two ways. The first way makes use of a surrogate model trained predict the label from different random subsets using masked tokens [9, 20J. The second way to compute EVAL-X (eq. (4)) relies on conditional generative models [30, 31]. Both hyperparameters and a combination of the estimators can be chosen to maximize the average log- liglliieldoood on a held-out validation set across random input subsets.
To estimate the second part of STRIPE-X, the ENCODE-METER, first expand the mutual information terms in ENCODE-METER, (e)), in terms of expected KL:
(7) The outer expectation can be estimated using samples from the data and the inner expectation over can be estimated using the EVAL-X model q Xv). The distributions over Ey can be estimated using classifier of Ey that randomly masks the label and masks different subsets of the inputs. Further details and generative way to estimate STRIPE-X are in Appendix C.1 and Appendix C.3; full algorithms are given in Appendix D.
(e) (vv.))=((xxxx) (By By--tyy | xx.
(g x, a.y) q |x, a)]
STRIPE-X in practice. Using STRIPE-X to choose between explanations is straightforward: pick the one with the larger score. However, like other evaluations that use learned models, misestimation can pose a problem. With large a, non-encoding explanations with misestimated ENCODE-METER will have bad STRIPE-X scores, while with small some encoding explanations can have good scores. Across all experiments, we set a = 20, which yielded STRIPE-X scores for known encoding explanations worse than known non-encoding explanations.
5 Experiments
This section consists of two parts. The first part demonstrates the weak and strong detection ca- pabilities of the evaluations ROAR, EVAL-X, and STRIPE-X in simulated setting and on an image recognition task. To demonstrate these capabilities, we run these evaluations on instantiations of POSI, PRED, and MARG. Additionally, we evaluate existing method that learns explain under a reductive constraint, called REAL-X [20]. The second part shows how STRIPE-X enables discov- ering encoding explanations in the wild, without specific knowledge of the DGP or the method that produced the explanation. We employ STRIPE-X to uncover encoding in explanations generated by a large language model (LLM) for predicting sentiments from movie reviews.
5.1 Empirically studying the detection of encoding in simulated setting
We construct two examples with binary labels y: one discrete input x and one that is hybrid of continuous and discrete components. Both use one binary input in x E {0, 135 as a control flow
7

---

POSI
PRED
MARG
ary max T7 (xxs) if (x() > 0.5, MijMisI
Eu if T()) 0.5,
E1 if if = 1,
e(x)
c(x)=
c(x)
En else.
arg max - π (Xar) else.
{2 else.
MiJMiS1
Table 2: Here, л (x) = <(y 1| x). Different encoding explanation methods that we consider.
variable and switch the inputs that y depends on. In both DGPs, y only depends on x1 if x; 1, and only on X2 x3 = O; this means that X,, X; are purely noise. For both DGPs, y is sampled per the following distribution where X3 determines the subset the y depends on
a(y = x) [x3 1)g(y x1, [xx3 0Jg(y X2,
(8)
Thus, EVAL-X* is achieved by an explanation of size 2: See Appendix C.4 for details; the exact DGPS are given in eq. (36) and eq. (37).
<(x) E + <3 if X3 = 1 else e(x) = E2+
Encoding explanations. Table 2 describes the encoding explanations we consider for this setting. In Appendix C.4, we check that Def: Encoding holds for these explanations in the discrete DGP by estimating the role of the unselected inputs in affecting the explanation and the role of E, in predicting y beyond Xy; characterization of Def: Encoding to support this check is in Lemma
ROAR and FRESH fails to weakly detect encoding. To empirically test the analysis about ROAR and FRESH, we study whether the two evaluations weakly detect encod- ing. In this study, we compare each evaluation's 's score on the all-inputs explanation, which is optimal, to the score assigned MARG. MARG ignores X3 which is required produce the label y in eq. (8). ROAR log-likelihoods for
EVAL-X
STRIPE-X
004
100
EVALXX
-0.
PRED
POST
MARG
e =
PRED
POSI
MARG
e60 = Ei
MARG and the all all-inputs explanation are approximately -H(y) = -0.69 for both DGPs, In addition, the value of the input that MARG selects alone contains all the infor- mation about the label regardless of whether MARG selects X1 or X2. Thus, FRESH log-likelihoods for MARG and the all-inputs explanation are both approximately -0.29 for both DGPs. This result validates that ROAR and FRESH are not weak detectors because they do not separate the opti- mal explanation from all encoding explanations.
(a) Results: discrete DGP.
EVAL-X
STRIPE-X
-0.4
-10
-066
-20
EVALXX
-30
-088
PRED
POSt
MARG
E
PRED
POSI
MARG
e(x)=
(b) Results: hybrid DGP.
EVAL-X is weak detector of encoding but not strong detector. EVAL-X ligeeiiilhood scores are given in blue in Figures 4a and 4b. EVAL-X, being weak detec- tor, scores the encoding constructions (POSI, PRED, and MARG) strictly lower than the log-likelihood of the opti- mal explanation X However, the EVAL-X score for the MARG explanation is -0.4, which is above the score of -0.6 achieved by a non-encoding explanation e(x) = Er: thus, EVAL-X is not strong detector.
Figure 4: EVAL-X X and STRIPE-X X scores of the 3 encoding constructions and the non- encoding constant explanation (e(x) ε1), for hoth DGPS. EVAL--X, being only a weak detector, assigns suboptimal scores to all encoding explanations (<), but scores some encoding explanations above the con- stant explanation. STRIPE-X, x, being strong detector, pushes down the scores of all the encoding expla- nations below that of the non-encoding con- stant explanation that always selects x1.
On the other hand,
Strong detector STRIPE-X prices out all the encoding explanations. Figures 4a and 4b report STRIPE-X scores for the same set of explanations as above; STRIPE-X X scores are shown in red. Strong detector STRIPE-X scores the non encoding explanations above the nega- tive entropy Hy = 0.69 and scores every encoding construction under that threshold.
5.2 Detecting encoding on images of dogs and cats
The goal of this section is to study the encoding detection capabilities of ROAR, EVAL-X, and STRIPE-X on real data. We consider an image recognition task like the one in Figure 3 with la- bels and images from the cats_vs_dope dataset from the Tensorflow package [32]. We break images of size 64 x 64 into 4 patches each of size 32 x 32. In left-right then top-down order, let X1, X2, X3, X4 be the upper left, upper right, bottom left, and bottom right patches respectively; x1 X3 capture color, and X2, x, are the animal images. With annot ( (image) denoting the annota-
8

---

tion in the cats_vs_dogee _ogs dataset the image having dog or a cat, the label is assigned as:
y = =| = blue] x [annot (image x2)= + 1|x = red] x I [annot (image x4)=
We consider three encoding explanations (POSI, PRED, MARG) and two non-encoding ones: 1) optimal, which selects the color and the patch that produces the label as dictated by the color, and 2) denoted f fixed, which always outputs the bot- tom right patch X4. Appendix C.6 gives details.
ROAR
FRESH
EVAL-X
STRIPE-X
opt
0.69
-0.23
-0.27
-0.31
fixed
0.59
00.4
-0.4
-0.44
POSI
0.51
-0.69
-0.70
-5.98
PRED
0.69
-0.33
0..11
-1.40
We report the scores assigned to each explana- tion by ROAR, EVAL-X, and STRIPE-X in Table 3. ROAR scores two encoding explanations PRED and MARG as high as the optimal explanation, mean- ing it is not
MARG
0.69
-0.23
-0.53
-1.02
Table 3: ROAR, FRESH, EVAL.-X, and STRIPE-X scores for the image recognition experiment. Higher is better. ROAR and FRESH score two encoding ex- planations PRED and MARG as high as the optimal explanation, meaning they are not even weak de- tectors. EVAL-X being only weak detector scores POSI, PRED, and MARG all worse than the optimal explanation under both EVAL-X but not the non- encoding constant explanation (e(x) = Ea), denoted fixed. STRIPE-X being a strong detector scores the non-encoding explanations above the negative marginal entropy (y) = -0.69 and scores ev- ery encoding construction under that threshold.
weak detector. POSI, PRED, and MARG all score worse than the optimal expla- nation under both the weak detector EVAL-X and the strong detector STRIPE-X. However, EVAL- scores one non-encoding explanation (fixed) worse than two encoding ones, meaning it is not a strong detector. Being a strong detector, STRIPE-X scores the fixed explanation above the negative marginal entropy (y) -0.69 and scores every encoding construction under that threshold.
Evaluating explanations produced by REAL-X [20]. We REAL-X to learn explanations for the simulated setting and the image recognition task. In the simulated setting, REAL-X is run to select one input; Appendix C.4 gives details. In the image recognition task, REAL-X is run select one of the four patches as an explanation; Appendix C.6 gives details. In both the simulated setting (see Figures 4a and 4b) and the image recognition task (see Table 3), REALL-X fails achieve the optimal EVAL-X X score while achieving a STRIPE-X score below the threshold of negative marginal entropy H (v) -0.69. Upon investigation, we found that REAL-X produced an explanation that matched the MARG construction on at least 80% of the inputs in the simulated setting. On the image recognition task, REAL-X explanation matched the MARG explanation on the whole dataset. In both cases, STRIPE-X, being a strong detector, correctly alerts that the REAL-X explanation encodes.
5.3 Encoding in LLM-generated explanations
One can detect encoding in any explanation by checking if the STRIPF-X score falls below the negative marginal entropy. Recent work uses LLMS to produce explanations; c.g. [33] prompt an LLM to generate explanations for reasoning tasks which are later used to improve smaller models. If the LLM explanation encodes, the smaller model can falsely ignore the informative inputs the larger model's explanation depends and yet does not reveal. In this section, we evaluate explanations generated by an LLM, Llama 3, for a sentiment analysis task. We consider reviews that take one of two forms: with ADJ1 and ADJ2 as adjectives, the review is
'My day was <ADJ1> and the movie was <ADJ2>.
that is it
"My day was <ADJ1> and the movie was <ADJ2>, oh wait, reverse the adjectives'.
The second sentence in the review acts as a "control flow" input and determines whether ADJ1 or ADJ2 describes the sentiment about the movie. We prompt Llama 3 (see Appendix C.8) to predict the sentiment and select a few words from the review that were important for that sentiment; the selected parts form the generated explanation. To discourage encoding, the prompt explicitly instructs the LLM to select all the words that the LLM based the selection on; such an explanation, by Lemma 1, would be non-encoding, On the 5 most common selections e(x) generated by the LLM, we compute the EVAL-X X score and the ENCODE ENCODE-METER (c). The resulting STRIPE-X score is -2.78, falling short of the negative entropy - H. (y) -0.69, meaning the LLM encodes. We investigated why.
As an example, consider the review 'My day was resplendent and the movie was hollow. that is it. "; the LLM selects only hollow in the explanation. However, the LLM instead selects resplendent when that it is switched to oh wait, reverse the adjectives. Such oceur- rences are common. On > 70% of the data, the I.I selects the word that describes the movie

---

but does not select the second sentence in the review which controls which adjective describes the movie; this is akin to MARG encoding. Thus, the -LM-generated explanation encodes by looking at the control flow input in the second sentence to find the correct adjectives, but failing to select the control flow input. Such an explanation falsely indicates that only the adjectives are relevant to predicting the label. In contrast, a non-encoding explanation would, in addition to the adjective that describes the movie, reveal control flow words that indicate which adjective predicts the label.
In summary, despite being instructed to include all the words that were looked at when producing the explanation, the LLM encodes. Building non-encoding explanations with LLMS may require an extensive search over prompts or finetuning guided by scores from STRIPE-X.
6 Discussion
When an explanation is encodes, predictions from the explanation become disconnected from pre- dictions from the values in the explanations. Such explanations select values with little relevance to the label and yet score highly on the many existing predictive evaluations. We develop a simple statistical definition of encoding. Inverting this definition shows that when non-encoding explana- tions predict the label, users know the values of those inputs selected in the explanation predict the either non-detectors (00AR[19],FRES...
label. We then show that existing evaluations only weak detectors (EVAL--X [20]). Motivated by this, we introduce a new strong detector, STRIPE- x. After empirically demonstrating the detection capabilities (or lack thereof) of said evaluations, we use STRIPE-X to discover encoding in LLM-generated explanations.
[18)) or
More related work. Other investigations into evaluating explanations focused on label leakage [26, 34] and faithfulness [18, 35, 36, 37, 38]. Label leakage is similar to encoding in that additional information is in the explanation, but focuses on explanations that have access to both the inputs and the observed label; we leave extending Def: Encoding to leakage to the future. Faithfulness, intuitively, asks that the explanation reflect the process of how label is predicted from the inputs; formalization does not exist. Jacovi and Goldberg [38] note the need to define faithfulness formally. Encoding explanations are not faithful the process of making an explanation because predictive inputs outside those selected by the explanation control the explanation.
Limitations and the future. Using misestimated models in evaluations (like EVAL-X) may lead to mistakes (see Appendix B.8 for an example). The retinal fundus experiment from Jethani et al. [26] is an example where misestimation leads to reductive explanations scoring higher than using the full input. Misestimation can be due to poor uncertainty or due to dependence on shortcut features. One fruitful direction is to use better uncertainty estimates, like conformal inference [39] calibra- tion [40], or employ robustness methods [41, 42] to ameliorate errors due to misestimation. Another direction is tricks like REINFORCF-StYL gradients to construct non-encoding explanations by optimizing STRIPE-X. Explanations that output subsets may not always help humans interpret the mechanism of the prediction. For example, imagine one wants to understand why model correctly answers the question "Who won the ski halfpipe at the X-games 3 years after her debut in 2021?" with "Eileen Gu". A subset explanation may return "3 years after her debut in 2021" and "ski half- pipe". but that does not help human interpret how the model predicts. A better interpretation would be make the model output, "3 years after 2021 2024. Eileen Gu won in 2024. and debuted in 2021." Such explanations can also encode information about the prediction in the text produced as rationale [43]. An important direction here would be to extend the definitions of weak and strong detectors of encoding to evaluations of free-text rationales.
Data versus Model Explanations. Even with the formal definitions of explanation methods, there is a question about what is being explained: the data or the model. These two concepts often get blended together in the literature [11, 20]. We clarify this point and abstract the choice away as two different ways to produce the joint distribution q(y, x). In data explanation, the distribution under which a feature attribution method seeks to output subset of inputs that predict the label should be the population distribution of the data [23]. If, instead, the goal is model explanation, the goal should not be to highlight inputs that predict the label well in samples of the data; rather it should be to predict the label well in samples from the model. Formally, a model with parameters O is a conditional distribution, Po(y x). To target a model explanation, feature attribution method would aim to output a subset of inputs that predict the label under the distribution F F(x)pp (y x).
10

---

Acknowledgements
This work partly supported by the NIH/NHLBI Award ROTHL148248, NSF Award 1922658 NUT-HDR:FUTURE Foundations, Translation, and Responsibility for Data Science, NSF CAREER Award 2145542, NSF Award 2404476, ONR N00014-3-112634, Google DeepMind, and Apple. The authors would like to thank Yoav Wald, the NeurIPS 2024 reviewers and the NeurlPS 2024 area chair for helpful feedback.
References
[1] Pierre Elias, Timothy J Poterucha, Vijay Rajaram, Luca Matos Moller, Victor Rodriguez, Shreyas Bhave, Rebecca T Hahn, Geoffrey Tison, Sean Abreau, Joshua Barrios, al. Deep learning electrocardiographic analysis for detection of left-sided valvular heart disease. Jour- nal the American College of Cardiology. 80(6):613-628 6-66, 2022.
[2] Neil Jethani, Aahlad Puli, Hao Zhang, Leonid Garber, Lior Jankelson, Yindalon Aphinyanaphongs, and Rajesh Ranganath. New-onset diabetes assessment using artificial intelligence-enhanced electrocardiography, arXiv preprint arXiv:2205. 2tt.:22055.20000. 2022.
[3] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Visualising image classification models and saliency maps. Deep Inside Convolutional Networks, 2, 2014.
[4] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Ad- vances in neural information processing systems, 30, 2017.
[5] Kim Long Tran, Hoang Anh Le, Thanh Hien Nguyen, and Duc Trung Nguyen. Explainable machine learning for financial distress prediction: evidence from vietnam. Data, 7(11):160, 2022.
[6] Alex DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7))610-619, 2021.
[7] Felix Wong. Erica Zheng, Jacqueline A Valeri, Nina M Donghia, Melis Anahtar, Satotaka Omori, Alicia Li, Andres Cubillos Aarti Krishnan, Wengong Jin, al. Discovery of structural class of antibiotics with explainable deep learning. Nature, pages 2023.
[8] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. Proceedings of the IEEE international conference on computer vision, pages 618 616-626, 2017.
[9] Ian Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209):1-90 2021.
[10] Neil Jethani, Mukund Sudarshan, Ian Connick Covert, Su-In Lee, and Rajesh Ranganath. Fast- shap: Real-time shapley value estimation. In International Conference Learning Represen- tations, 2022.
[11] Jinsung Yoon, James Jordon, and Mihaela der Schaar. Invase: Instance- variable selection using neural networks. In International Conference on Learning Representations, 2018.
[12] Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Samuel Gershman, and Finale Deshi-Velez. Human evaluation of models built for interpretability. In Proceedings of the AAAI Conference on Human Compulation and Crowdsourcing, volume 7, pages 59-67, 2019.
[13] Adriel Saporta, Xiaotong Gui, Ashwin Agrawal, Anuj Pareek, Steven QH Truong, Chanh DT Nguyen, Van-Doan Ngo, Jayne Seekins, Francis Blankenberg, Andrew Y Ng, et al. Bench- marking saliency methods for chest x-ray interpretation. Nature Machine Intelligence, 4(10): 867-878, 2022.
[14] Jonathan Crabbé, Alicia Curth, Ioana Bica, and Mihaela van der Schaar. Benchmarking het- erogencous treatment effect models through the lens of interpretability. Advances in Neural Information Processing Systems, 15:1255--2 2022.
[15] Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus- Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(1 1):2660-2673 2016.
11

---

[16] V Petsiuk, A Das, and K Saenko. Rise: Randomized input sampling for explanation of black- box models. arXiv preprint arXiv:1806.07421, 2018.
[17] Piotr Dubkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in neural information processing systems, 30, 2017.
[18] Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and Byron C Wallace. Learning to faithfully rationalize by construction. arXiv preprint arXiv:2005.001 2020.
[19] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for in- terpretability methods in deep neural networks. Advances neural information processing systems, 32, 2019.
[20] Neil Jethani, Mukund Sudarshan, Yindalon Aphinyanaphongs, and Rajesh Ranganath. Have we learned to explain?: How interpretabiliity methods can learn to encode predictions in their interpretations. In International Conference on Artificial Intelligence and Statistics, pages 1459-1467. PMLR, 2021.
[21] Jennifer Hsia, Danish Pruthi, Aarti Singh, and Zachary C Lipton. Goodhart's law applies to nlp's 's explanation benchmarks. arXiv preprint arXiv:2308. 14272. 2023.
[22] Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. Journal of machine learning research, 3(Mar):1157-1182, 2003.
[23] Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An information -teeooetio--hhoottiic perspective on model interpretation. In International conference on ma- chine learning, pages 883-892. PMLR, 2018.
[24] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " "why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1 135-1144, 2016.
[25] Erik Štrumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems, 41:647-665, 2014.
[26] Neil Jethani, Adriel Saporta, and Rajesh Ranganath. Don't 't be fooled: label leakage in explana- tion methods and the importance of their quantitative evaluation. In International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2023.
[27] Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. semel-based condi- tional independence test and application in causal discovery. In Proceedings of the Twenty- Seventh Conference on Uncertainty in Artificial Intelligence, pages 804-813, 2011.
[28] Mukund Sudarshan, Wesley Tansey, and Rajesh Ranganath. Deep direct likelihood knockoffs. Advances in neural information processing systems, 33:5036-5046, 2020.
[29] Mukund Sudarshan, Aahlad Puli, Wesley Tansey, and Rajesh Ranganath. Diet: Conditional independence testing with marginal dependence measures of residual information. In Interna- tional Conference on Artificial Intelligence and Statistics, pages 10343-10367. PMLR, 2023.
[30] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language mod- els are few-shot learners. Advances in neural informution processing systems, 33:1 13718171 2020.
[31] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hier- archical text-conditional image generation with clip latents, 2022. URL https://arxii. org/abs/2204.06//55, 7, 2022.
[32] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Good- fellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL http://teettoor..... org/. Software available from tensorflow.org.
12

---

[33] Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, et al. Explanations from large language models make small reasoners better. arXiv preprint arXiv:2210.06726, 2022.
[34] Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language? arXiv preprint arXiv:2010.04119, 2020.
[35] Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Filippova. will you find these shortcuts?" protocol for evaluating the faithfulness of input salience methods for text classification. arXiv preprint arXiv:2/ 11222/11003300 1.07367, 2021.
[36] Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. Do feature attribution meth- ods correctly attribute features? In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pages 9623-9633, 2022.
[37] Yiming Ju, Yuanzhe Zhang, Zhao Yang, Zhongtao Jiang, Kang Liu, and Jun Zhao. Logic traps in evaluating attribution scores. arXiv preprint arXiv:2109.00 (10::22099.04633,, 2021.
[38] Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? arXiv preprint arXiv:2004.0300 2020.
[39] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-fiee predictive inference for regression. Journal of the American Statistical Asso- ciation, (13($23):1094-111 2018.
[40] Mark Goldstein, Xintian Han, Aahlad Puli, Adler Perotte, and Rajesh Ranganath. X-cal: Ex- plicit calibration for survival analysis. Advances neural information processing systems, 33: 18296-18307, 2020.
[41] Aahlad Manas Puli, Lily H Zhang, Eric Karl Oermann, and Rajesh Ranganath. Out-of- distribution generalization in the presence of nuisance-induced spurious correlations. ICLR 2022, 2021.
[42] Aahlad Manas Puli, Lily Zhang, Yoav Wald, and Rajesh Ranganath. Don't blame dataset shift! shorteut learning due to gradients and cross entropy. Advances in Neural Information Processing Systems, 36, 2023.
[43] Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. Faithfulness tests for natural language explanations. arXiv preprint arXiv:2305. 18029, 2023.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it accnowleddeed in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
14

---

All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple-
mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibilityy which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We only use public data. We use standard existing training techniques, de- scribe the hyperparameters in detail, and provided the Llama 3 prompts we used in our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https: (https://nipp tttsps://nips.co cc/ public/gule htglle//niiiess/ddedbbmissssiaap
for more details.
15

---

While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cttccc/ppu.o htte//pbllc/uuuudeseeeeeee/ wwwgguuddcccc/e /twee/gueeeeeeeeeeemmmmm Cttp://CoedSbbiiiicc ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: The error bars in 5.1 and 5.2 negligible because estimate means of metrics over datasets of 5000 samples. For 5.3, error bars are irrelevant because there is no comparison against existing method. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of emrors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
16

---

8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the cetpoliiie/tiissuieeeellelllll
NeurlPS Code of Ethics https::/ /ttt:s://neurip...
Answer: [Yes]
Justification: We do not use human subjects and only use public data and public models. Our work focuses on explainable machine learning and does pose additional ethical harms beyond what is standard for the field.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We investigate problem with explanations and fix it. There is no societal impact of our work that exceed that of the usage of explanations.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, Ict alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
17

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Question:
Justification: We do not release any data or models. Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 'ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given participants and screenshots, applicable, as well as details about compensation (if any)?
18

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19

---

A Theoretical Details
A.1 Expressing g(y Xe(x)) in terms of the values and the identity of the explanation
To express q(y Xe(x)), we use the following equivalence of events from eq. (1)
{x Xx(x) = (v, a)} = {x e(x) = v) n {x x, a).
(eq. (1))
Then, intuitively. conditioning on the event that xx(x) (v, a) gives you the same information as conditioning on the events exx) and Xv simultancously. We make this formal below.
For discrete for any a such that the probability g(Xe(x) (v, a)) 0, define the LHS and RHS of eq. (1) as Bv, Cv respectively. Then, the conditionals g Xe(x) = (v.a)) and g E 1, Xy = a) exist and can be written as follows:
q(y (x) = (v, a)) - gly x E B.) <(y Ev = 1, - a) = =q X C.).
These two conditionals are equal because By Cv.
The same kind of result holds for general random vectors (discrete or continuous) x but is little more involved because By may be non-empty while g(x E B.) = and the equality of conditional densities/probabiittes need to be written via measure theory. Assume the regular conditional proba- bilities q(y Xx(x)) and g(y, Ev xv), 7'Y Ev, Xv) and g(Ev Xv) are defined almost everywhere in their respective probability measures. Take any S, S {xy e(x) v) where (x, E S.)) and ((((x) = v) > O. Consider any measurable sets Y over y and (S,) == {(v, a) : a Sy} over xe(x). Now, by definition of regular conditional probability measures, joint probabilities are obtained by taking the expectation of the conditional with respect to marginal distributions over the conditioning set:
/ ay YY| xx(xxxxx= (v, (d)x)))(()x((())
<(y E xxx)(x) E Bv(Ss))
(B.)
1
S. gly E Xe(x) = (v.a))g(xxx)) = (v.a))da (9)
and
<(y E Y. = 1,,x E (S) = gly E Y,E, = x, = a)g(Xx = =a)da
Sy
gly E Y Ey = 1,,, = a (Eq = 1, Xv = a)da. (10) S.
Due to eq. (1), the LHS terms of the two equations above are equal and so are the probability measures over the integrating variables in eqs. (9) and (10). Letting xv be defined Borel sigma algebra, these two integrals eqs. (9) and (10) are equal and only if for any Borel set Sv. for almost every a E S,
gly E Y Xe(x) a)) q(y Y E, 1, a).
That is, in more plain terms, the conditional distributions are equal g(y xx(x) = (v. q(y E, 1, a).
A.2 With non-encoding explanations "what you see is what you get"
Def: Encoding says that an explanation e(x) is encoding if there exists an S where g(xe(x) E 0 such that for every (v, a) E S.:
E, X, = a.
(11)
For non-encoding explanation, Def: Encoding does not hold. Here, we derive the implications of violating Def: Encoding, Define the set A to contain all (v, a) where eq. (11) is violated:
A a(v.a): y 1L E, Xv = =a.
(12)
20

---

By definition, the complement A C is such that
(Vv a) EA y Ev Xv a.
Such set cannot have positive measure when Def: Encoding is violated which means
g(xx(x) E A =0
In turn,
((xx(x)) E A) g(xx(x) CA 1. Thus, A is such that (Xx(xx) E A) = 1, and by eq. (12) for all (v,a) E A
y 11 Ey Xy =
which in turn guarantees
g(y Xe(x) = (v,a)) = q(y Xv = a, Ey = =1) 9(y Xv = =a).
A.3
Helpful Lemmas and their proofs
A.3.1 Alternate conditions equivalent to Def: Encoding
The dependence in Def: Encoding occurs due to two reasons, understanding which sheds more light on the definition. First, for some selection e(x) provide enough information to reveal that the explanation should select the inputs denoted by v. In other words, the indicator of the selection is variable even after fixing the explanation's values themselves. Second, this indicator predictive of the label for the data with the explanation v. These two properties provide intuition on the definition of encoding:
the explanation's values Xv do not
Lemma 1. Def: Encoding holds for explanation e(x) if and only if there exists a selection such that q(e(x)= = =v > and set Sy [xx= e(x)= =v] such that g(xv Sv) > where both of the following conditions hold for almost every Sv:
Unpredictability of Explanation Additional Information from Explanation
(EEv Xv =a) +
ay X, = a, E, = + gly Xv = =a, a, E,, = 0).
Proof. First, Lemma 2 shows the Def: Encoding holds if only there exists a selection v such that q(e(x) v) > 0 and set Sy {xv e(x) v) such that g(x, E S.) > 0 where
Ya E Sv:
y -l Ey xv.
We use this alternate definition in what follows.
Given a non-measure zero Sv, by Lemma almost everywhere in Sy it holds that (EEv xxxx)= > 0.
Conditional dependence implies Unpredictability and Additional information (the only part). If q(E,, 1 xv) 1 almost everywhere (under ((x)), then E, is constant given Xv. and therefore independent of any variable given xv:
(E, = x+) =1
y.lEE, Xv-
Then, follows that conditional dependence implies the unpredictability property
E Xv
q(E, xv) 1.
Second, with the result from Lemma 3, we have (E, 1|x,) E (0, 1). Thus q(ylx,, E, 1) and q(y,x,, E, = 0) exist almost every where in S.. Then, by definition of conditional dependence, there is additional information about the label in the explanation:
E -
g(y Xv, E, 1) + gly Xv, E, = 0).
This shows that Def: Encoding implies the additional information property.
21

---

Conditional dependence implied by Unpredictability and Additional information (the if part). Now, gEvv 1 -1 (0, 1), then the following two conditional distributions exist almost everywhere in Sv
g(y Xv, E, 1)
g(y Xv: Ey = 0).
Then, by definition of dependence almost everywhere Sv:
g Xv, Ey = 1) # g(y Xv, Ey = 0) - Ev Xv. Thus, the unpredictability and the additional information properties imply Def: Encoding.
Lemma 2. Def: Encoding holds for explanation e(x) if and only if there exists a selection such that ((e(x) = vv > and a set S, S {xx, e(x) v} such that q(x, E S.) > 0 where
Ya E Sv: y H Ey X, a.
(13)
Proof. Def: Encoding says that the explanation exx) is encoding if there exists an where ((X.(x) E S) > 0 such that for every (v,a) E eq. (13) holds. This proof works by showing that having a positive measure implies the existence of v and Sy as in Lemma 2 such that eq. (13) holds.
Decompose (xx(xx) E S) by introducing an expectation over g(e(x)).
((xe(x) S) (x= ES e(x) = v).
As there are only finitely many v.
(Xe(x)) E S)
Ev s.t. q(((x)= v) >
and ((xx(x) E ES [(|((x) v) > 0.
The "only if" direction. Pick any v such that the RHS above holds and define S. = {a: (v. E S}. By definition,
S. = = (v, E 8} (x. e(x) v} S {xv :((x) = -v..
This proves that S. has positive measure:
<xx, E S.) = (xxe(x) E (S,((x) =v)= E e = v) * *g(e(x) =v) > 0.
Finally, as E S. = (v,a)= S. eq. (13) holds:
Va E Sv: y HL E, Xv =a
This completes the "only if" direction.
The "if" direction. Assume that there exists v such that g(e(x) = v) > and S. C {xv
e(x) v) such that (xv E Sv) >> 0 where
Va E Sv
Y. x, a.
Define S = a(v, E S.). By this construction, S has positive measure:
((xe(x) E S) = (((v, xx) SS
= q(e(x) = ")a((v.xu)) E e(x) = v)
q(e(x) v)g(xc E Sv |e(x) vv) = q(e(x) = v)g(xx E Sv) { as Sa S, S fxv : e(xx = v)
> 0,
where the last inequality holds because by assumption
q(e(x) = vv 0 g(xv E S.) 0.
Finally, (v,a) E a a Sv. eq. (13) holds:
V(v,a) S, y y E. Xv =a
This completes the "if" directions and with that the proof.
22

---

Lemma 3. For any set Sv [ fxv : c(x) = v} such that gxx, E Sv) > 0. then for almost every
E Sv. g(v Xv a) >
Proof. Define the set Av = fa: ((, = xv = a) = 03. Next compute the joint probability
g(xv Av nSv) = a(xv E Av)g(xv Sv Xv Av).
Now, noting that Sv a subset of (xv: = v). which is equivalent to {xv Ev 1]. thus
axxx E ES Xv E Av)
/ q(xx E Svv Xy a, Xv E Av)g(x, a X, Ay)da
/ g(xv E Svv Xv = a)q(xx = a Xv E Av)da
g(E 11 Xv a)g(x, a X, E Ay)da
/ ((Ex = 1 Xv = a) Xv E (a q(v = 1 Xv = = =O)dda
0.
The probability g(x, E Sy Xv E Av) is non-negative, so it must be zero. Plugging this conditional back into the joint gives g(xv E Av nS.) = 0. Then expanding yields
0 = q(xv E Av n (n) -((x Av xv E (.)((x E S.). Since (xx, E S,,)
> 0, (x, E Av X, E {Sv) must be zero and thus, g(x, & Av X, E (S.) =1, where expanding out the definition of Av gives the desired result that g Evv 1 almost a E S.:
a) > 0 for
= g(a E Av (.(aaS)))
= g(a & {a: (a:q(Pa (Ev = 1 xv =a) = =)== a
g(a (a (Ev = x, =a) > 0} a S.).
A.3.2 Optimal value and the optimal gap under EVAL-X
Lemma 4. The EVAL-X optimality gap value for e(-) is an averaged KL between gly x) and q(y xv): Eveyg(e(x) By )(x)=v) KLigly x))la(y xx)).. This gap zero, i.e. e(x) is optimal with the score EVAL-X* = E2 log q x)) for all such that g(e(x) vv) 0,
q(y x) q(y xv) a.e. in {x:e(x)=vV =v.
Proof. Let p be a generic conditional distribution and let x be the values outside the explanation.
max EVAL--((q,e) max E. ((v.a))=o(xin)) Byyy xxco)=(v.a)) log q (y xx = a) max E(v ( a)~o By(x x.s0 (vaa)) Butr (x00)) (v.a)xx v) log q (y xx a)
max E (lv.a)-o(xx. Byxx =v.a)=(vaa)) Eoyy x) a (y xx==a a)
max By (lv.a))e((x.o)) Bycx (xco) Baty 1x) boggry xx]
P
max ex(x) E x) plogppy xx]
max - Eg KL(g(y x))lp(y x)) + Elylog g(y x)]
E. [log q x)].
This upper bound achievable by an explanation that selects all inputs, so the maximum EVAL--X denoted as EVAL-X"- = Ey log q x).
23

---

As in the math above, the EVAL-X score for an explanation method can be expanded as
EVAL-X* l8 (lvva)>=((xx Bo(y (v.c))())))) log q (y Xv = a)
((v.))=((xxxx)) (xxx (v,aa) E By x) log g(y Xv a) [v)x(((())) Barg(xx (e())=)) By(x xx=a,e(x)=v]) Bucy x) loga(y Xv = a) Ev-g(e(x)) y v)x) Buy x) log g xv),
where in the last step. we dropped a because it equals Xy almost surely. Similarly, the optimal score
EVAL-X* expands to
Ex-gle(x)) Eytx e(x)=v)) Boty I x) log cly [x).
Let V be the set of values that explanations can take on, then taking the difference from optimality
EVAL-X* - EVAL-X
Bxwa(e(x) Boxx e(x)=v)) Boly x) log q(y x) g(y Xv = x) Evngle(x)) By(x e(x)=v)) KL igly x) g xv)]
Σ [e((() B) c(x)=v) VEV
(g x) g xx)]. KL
As each KL term is non non-negative, each term in the sum being set to 0 simultaneously achieves the optimum, which happens when for all v such that g(<(() = v) > 0,
aly x) = q xv) for almost every {x e(x) = v).
In Appendix A.4, we use the results from Lemma and Lemma to prove that the optimal score of EVAL-X can only be achieved by non-encoding explanations.
A.4 Proof of Theorem
Theorem 1. ffe(x) is EVAL-X optimal, then e(x) is not encoding.
Proof.
Note only q(((x) = v) > 0 are of interest, since q(e(x) = = = implies that Ev 0 almost surely and thus y Ey Xv.
Then if e(x) achieves OVAL-x", then by Lemma 4, for all v such that g(e(x) = v) > 0,
q(y x) = q x,) for almost every (x: e(x) v).
First, this optimality criteria incorporate E, 1 the lefthand side by first conditioning on
e(x) and then noting that the equality holds for x where e(x) v
<(y x) = g(y xv)
for almost every for almost every for almost every
{x : e(x) v}
g(y x, e(x)) =q(y xx)
{x e((x v}
- - gly x(x = = =q(y xx)
{x : e(x) =v}
4ly x, E, 1) =gyy xx) for almost every
(x e(xx) -v.
To understand if the optimality criterion disallows encoding, integrate the left and right-hand sides of this optimality criterion with the respect to complement of the inputs in xv. g(x;) xv: Ev 1) yields
i a(y x, E, 1)] Xv, E., 1)dx; = -f
/oly aly xx)g(x² xv, E, 1)dx;
for almost every {x e(x) = =v]
= / aly x E. = =1)g(x)
Xv: Ey 1)dx; = q(y xv) / ((x Xv. Ey =1)dx;
for almost every {x e(x) =v}
g(y Xv, Ey = 1) = xv)
for almost every (x e(x) = =v]
gly Xv, Ey= 1) q(y x,)
for almost every {x, e(x) = v].
24

---

Now expanding the right-hand side gives
<(y Xv) <(y, Ev - 1 xxxxv) + qy, Ev = 0 xv).
Combing the two equations gives
q(y Xv. Ey = 1) gly, E, = 1 Xv) + q(y, Ey, = xv)
for almost every fxv e(x) = v).
(14)
We show that this equality implies that y Ev Xv by splitting the analysis into cases based on (E,, = 1 xv) = and E(E, = xv) < I. turn, the condition Deft Encoding is violated and the explanation e(-) is not encoding.
Case EVAL-X optimality holds when the explanation is predictable. The first case to con- sider is when the event that the explanation takes the value v is determined by Xv for all samples with the explanation v. That is, q(E,, = 1 xx,,)= 1:
(Ev 1 xv) =1 + qEE, = 0 xv)
Then expanding this marginal into the joint shows that the joint q(y, Ev 0 xx) has to be zero as well.
q EE, = 0 x = - f
f [ly, o(yy, E, = 0 xx = O,
because integral of non-negative terms being zero implies that each term itself is zero almost surely.
Then, show that the determinism condition (Ev = 1 xx) = 1 is sufficient for the optimal- ity criterion eq. (14):
q(y Xv: Ey = 1) q(y Xv, E, = x 1
7(y Xv, Ev, = =1)9 Ev =
((y, E, =1
((y, E, = xx)) + (l, E,, = 0 xx) for almost every fxv : e((x) = v).
This shows that the EVAL-X optimality criteria is satisfied when the (Ev = 1 xv) = 1, thus the explanation is completely predictable from the explanation for examples with that explanation. By Lemma 1, we have
(E, = 1 xx, = = y yll Xv. which violates Def: Encoding. So there is no encoding in this case.
Case 2: When the explanation is unpredictable, EVAL-X optimality requires that the explana- tion provide extra information. Now consider the alternative case, gEE, 1 xx)) < 1. Here the explanation does not determine the explanation opening the possibility that the EVAL- XVot-m-optimal explanation method can encode information in the explanation.
Because q(e(x) = v) > 0, we have q(x,, E (xv : e(x) = v}) > O. Thus, by Lemma 3, for almost every (xv exx) v it holds that (E, 1 xx) > 0. Putting this result together with alternative
case (E, = 1 x < 1) gives: < E(E,
xx) < 1 for almost every {xv : e(x) = v).
Now, expanding out the optimality criterion cq. (14):
q(yl xv, Ey = 1) = <(y, E, = 1 xx) x 1 + q(y E = 0 xv) x
for almost every {xv : e(x) v}
- qly Xv. Ey = 1) = q(y E, = 1,,, )q = xv)
q(y E, 0, x1 (1- (EE,, = xx.)
for almost every {x, : e(x) = v]
- (1 - (E,, = | x Xv, E, = 1)= (1 9 (E, 1 xv) ky) xv: E, 0)
for almost every {x, : e(x) = v}
- q(y xv, E, 1) q(y xv. E, = 0) for almost every {x, e(x) v).
This equality says for all samples with the explanation v. knowing E, does not change the distri- bution of the label y. By Lemma 1, this equality implies that the independence y 1l E, xv holds which violates Def: Encoding.
D
25

---

A.5 Proof of Proposition 3 and Theorem 2
Proposition ENCODE-METER (y(e) if and only if not encoding.
Proof. First, Def: Encoding is violated if and only if there exists set A such that (xx(xx) A) =
and
a(v,a) E A
yy Xv
For the direction, note that if ENCODE-METER 6G(E) 0,
((vaa)>=(xxx0)) (y; Ey Xv = a) = 0.
To show the forward direction, the above equality means that if (g(e) = 0, almost surely for every ().a) (xx((x)) the instantaneous mutual information which implies the desired conditional independence
I(y; Ev X. = a)
yl Xv a.
By definition of almost surely, there exists set A such that g(Xe(x) E A) 1 the independence above holds; this completes the "if" direction.
To show the reverse direction, let there exist set A such that g(xe(x) E A) = 1, for every (v. a) A,
y Ell Xv.
In turn, for all (v, E A,
I(y: E, X, a) 0.
Then, the fact that g(xe(x) E A) = 1 implies that expectations with respect to ((Xe(x)) over the whole support equal expectations over ((x,(xx) Xx(x)) E A). which is ((xx(x) restricted to A:
((v.a)=((x)o))) [y; Ey Xv = a) = ((y (e Ey Xv = a)
This completes the "only if" direction.
Theorem 2. With finite H(y x) and H(y), for any explanation that encodes and any that does not encode é, there exists an a" such that Vo > a" STRIPE-Xa(q,el > STRIPE-X..ge
Proof. Recall that STRIPE-X is
STRIPE-XX.44 e) = TV (( Buty (v))= xx = =a]] adg(e),
where the ENCODE-METER
(c(e) = (l X, = a).
We first show bounds for the first term in STRIPE-X and then derive the STRIPE-X scores for encoding and non-encoding explanations.
Bounds on EVAL-X scores. We lower bound the EVAL-X score, which is the first term in STRIPE- X. for non-encoding explanations.
For non-encoding explanations, almost surely over v, g(xe(x))
q(y Xe(x) = (v) = q'y Xv = a).
Then,
EVAL--((q.e) Bolyxx==)) log
((v.a)))((xx)))) Egty (v))=(vva)) log a (y xv = a) -Fggyx log q(y)
((vaa) Bgyy =(w)) (log 9 (y (x(x)) = (v, (v. Eg(y. log g(y)
Bgly. [log (y xe(x))] E (g(y)))))) log g(v)
Balr) log (y Xe(x))
k(y)
I(y:
26

---

The above inequality implies that
EVAL.-X(q, e) (v (v B Byratr (v.x==(v.a)) log gly) I(y: xx(x)) 2
EVAL-X((, (y) 20
EVAL-X(Q. e) 2 (y).
Every inequality in the derivation above becomes strict when the explanation selects inputs that predictive of the label because
(x))) >
Thus, non-enen non-encoding explanations have EVAL-X scores that are at least Hg (y).
The optimal EVAL--X score for any explanation (see Lemma 4) equals the negative conditional en- tropy which is upper bounded by some finite number:
Egg [log qly x)] Hy x) =.
Comparing explanations via STRIPE-X. For any encoding explanation, by Proposition 3, for some
(,(e) >
Now, consider a* I(y:x) 2 0, which is finite because each term in the ratio is finite. Then, for all
a²
(0g(E) a 0e(E) x(y;x).
Thus,
-ad,(e) < x);
As EVAL-X scores are below = HH( x) for any encoding explanation,
STRIPE-X, (g. (g. EVAL-X(q. e) (e)
-H(y x) - Ily;
HH(y x) (H, H(y x))
-y)
Finally, for any non-encoding explanation, (cyc') by Proposition 3, STRIPE-X scores equal EVAL-X scores, which are lower bounded at (y)
Together, for every non-encoding explanation e' e((x) and encoding explanation e(x), it holds that
STRIPE-X. e') (y) > STRIPE-X.(q.e)
This proves that STRIPE-X is a strong detector of encoding.
B Encoding examples, non-detection of ROAR, and non-strong detection of EVAL-X
B.1 An illustrative DGP for Def: Encoding
With B(0.5) being a Bernoulli distribution, consider the following example
y ~ B(0.5) ~~ B(0.))
61, ~ N (0,
[ yy +6, 0,62
z 0,
x =
11 E3,y El, 1,62
z 1.
For this problem, if the third coordinate x3 0. all the information between the label and the covariates in the first coordinate X1, and if x; 1, the information is between the label and the second coordinate X2. The corresponding explanation function is e(x) 1 Ojξ + [Xx 1|62- This explanation is encoding because neither explanation's 's values xı nor X₂ determine the explanation function because it depends X3. Formally
g(y X3 1) # q(y X1, 0) y.LxS
E. X1,
27

---

which meets Def: Encoding. Consider an alternate ean--enooing explanation function e(x) [1[x, 0], 1 0]. 0 X2 do not determine e(x) that depends on the noise eg in X4. That means the unpredictabilltyy property in Lemma holds. However, by construction,
x1, -د 62 =
(1, IL Ev = y_ x1 and y X2.
So no additional information about the label is encoded:
y IL Ev Xv.
The additional information property in Lemma 1 avoids such cases where the explanations keeps additional information that is irrelevant to the label.
B.2 Encoding explanations conceal predictive inputs that affect the explanation Consider the following DGP
x X1, X2, Xa] ((0.5
X
X3 1,
(15)
X2
X3 = 0.
Let e be an encoding explanation that selects the first coordinate x3 = and the second coordinate otherwise. We never observe X3 when looking only at the explanation. Table 4 shows all possible values of this explanation. Notice that in the third and fourth rows, the value of Xe(x) changes to match the label exactly, stay constant. is impossible to understand the perfect predictiveness of xx(x)= as the encoding
though the values of the first two coordinates that we observe
explanation conceals the control flow feature X3 that determines which of the first features should be picked to predict the label.
Table 4: Possible values of the inputs, label, and explanation for the DGP eq. (15)
Xx(x) V
(v,
X1
X2
Xg
y
a
0
[0, 1, 0]
0
0
1
[1, 0, 0]
0
1
0
[0, 1, 0]
1
1
[1. 0,
0
0
0
[0, 1,
0
0
1
[1. 0.
1
0
[0, 1, 0]
1
[1, 0,
B.3 Position-based encoding fits Def: Encoding
Recall the perceptual task that classifying images of dogs versus classifying images of cats, and consider the encoding explanation Cposition (x) that is
eposition (x) E1 if
q = dog x) = 1,
eposition
E2
if
q(y
x)
Assume that the inputs in the top leftmost pixels are always background. meaning that the values of these inputs provide no information about the label yllx,, X2. Now we check if this intuitively- defined position-encoded explanation meets the definition for encoding (Def: Encoding). To condi-
tion on XG Eg, O. we need q(y dog) + 1. Note that
(Eg. 1 x g(y dog xx,) g(y dog) # Def: Encoding holds because the indicator of which explanation chosen Egg, determines the label.
gly = dog X Eg, = 1) = 1 0 q = dog Xx, Eg. = 0) = y 1 Eg, X5, This example shows how the encoding definition Deft Encoding captures the informally described position-bass based encoding from the literature.
28

---

B.4 Prediction-baid based encoding fits Def: Encoding
The informal example of prediction- based encoding from Section 3.1 selects a single input that makes the prediction from all of the input have the highest confidence when given the single input. One way to mathematically express such selection as follows:
eprediction (x) Sargmax, 9ly=1 xx) if gly = x) > 0.5,
(16)
Oprediction (x) Eargmin, ely=1 x:) if q(y = 1 x) - 0.5.
Here, we describe one set of conditions on the distribution g(y, x) for which the explanation in eq. (16) fits the definition of encoding in Def: Encoding. Assume that there exists non-measure- zero set U C {x g(y = 1 x) > 0.5} and an index & such that
x E U = q = 1 x) 2 P.
(17)
x E U = Vi g(y
xx,)) q(y 1 1xxx),,
(18)
x o U = q 1 x < P.
(19)
x & U + Ji gly x q(y xxx) > gly = 1 xx,)
(20)
Further, assume that X5, alone does not determine x E U:
0 < E U] x <
(21)
The assumptions above imply the facts below about eprediction (x) (x): 1. By eqs. (18) and (20)
x E U + eprediction (x) = Ek-
Define the explanation indicator E, = @prediction (x) = v).
2. By eqs. (18) and (20) and the definition of Ev
x + Ege =1
(22)
3. By eqs. (21) and (22)
q = xxx) = q(x E U xx.) < 1.
(23)
4. By eq. (23), q(y = 1 X Eg, = 1) and q(y = 1 X Egg- = 0) are well defined. Then,
by eqs. (17) and (19), for x E U
q(y=
X ES-- =1)
= Byyx Xey Ex=1) x)
2 By(x Xee, Bx=1)p
{as Eg =1 = x UEU)
p.
qly 1 XSx E₁ 0)
= Ey(x Xey Ox=0) x)
< Ey(x Xey By O)P
{as Eg =0 = x & U}
p.
Thus, for all elements of {xEx U}
q(y 1 X Eg, 1) > g(y = 1 X Ee, = -)).
(24)
By Lemma 1, the properties eqs. (23) and (24) imply that Va E fxç, : E U)
y EHL
XEL = a.
Finally, the set Up = (xe, : x E U) is non-measure-zero: as 1[x E U] =1 = I [ E U 1, accumulating qxx) with the restriction xs, U₁ leads to at least as much mass as accumulating with the stricter restriction x E U:
xxxx E Un) = / ((x)](xx, E Un|dx > f
f ((x))(xx E Ujdx g(x U) 0.
Together, the last two equations implies that Def: Encoding holds for Eprediction (x) from eq. (16).
29

---

xx(H)
Xe(x))
cat
dog
Xe(x))
Xe(se) (c) Marginal encoding
(a) Distribution y) (b) Non-encoding explanation
Figure 5: Example DGP and MARG encoding. (a) The color determines whether the label is pro- duced from the top or bottom image. (b) An explanation that correctly reveals that the label is generated based on both the color and, as dictated by the color, the top or the bottom image. The label is deterministic given the value of the explanation which means the label can be predicted per- fectly. (c) An encoding explanation would be one that produces only the top or the bottom animal image based on the color being red of blue respectively. This returned animal image does not indi- cate the fact that the data generating process depends on color. Now, the animal image selected by the explanation alone is insufficient to dictate the label because the color determines which image determines label. The identity of the image, whether top or bottom, provides additional infor- mation about the label beyond the values explanation, captured in Def: Encoding.
B.5 MARG explanations are encoding
We provide illustrative example of MARG explanation for the DGP in Figure 5, Here, we show how mathematical formulation MARG satisfied Def: Encoding.
Consider generic DGP with Bernoulli control flow input denoted x,: for some distinct sets U, V that do not include e and no combination of x Xy. Xy determine the rest
x X.
( x) q(y xu):
q x) = q xv).
Further, assume that the subsets leads different distributions over y such that non-zero measure subset Sy C [xt x such that x. 1}
q xu x. 1) # g 1 xu. X, 0).
(25)
Now, consider MARG explanation that looks at x, and outputs the corresponding sets U, V:
e(x) if x =1 else
e(x)
By definition the explanation only depends on the control flow input, not by X xv. Next, as Eg 1 is the same event as x, 1, e e(x) encoding because the assumption from eq. (25) implies:
q(y x Eu 1) + g(y xu, Eu =).
Then, this inequality holds for all elements of the non-measure-zero set Su, by Lemma 1, MARG is encoding.
B.6 Proof of Proposition
Definition 4. We denote val(xe(x) as the function that maps explanation Xe(x) (v, a) to the values the inputs take the selected indices, eight-padded to have the same dimension the input E Rd:
if < E V if - V; < d
val val(Xe(x)),
a, tadenoken
For example, i (a, 3, and e(x) [0, 1,, then Xe(x) (00, 1, 0, [8]) and
val vxl(x)(x)) [8, pad- podetoken, taa.p-ooeeen..pa-toeelll
30

---

Table 5: Probability table for the DGP in eq. (3). Conditional on the explanation Xs, does predict
the label. For example, given knowing x = 1, if X3 = implies ply = 1) = 0.9 but if X3 = 0,
ply = 1) = 0.5. The probability table Table shows this.
X1
X2
X3
p(y
x)
e(x)
val vXe(x))
0
0
0.1
E2
[0, pad-token, pad-token]
0
0
1
0.1
&1
[0, pad-token, pad-
0
1
0
0.9
&2
[1. pad-token, pad-token]
0
1
1
0.1
&1
[0, pad-token, pad-token]
0
O
0.1
E2
pad-token, pad-tooken]
D
0.9
&1
[1, pad-token, pad-token)
1
0.9
&
[1, pad-token, pad-token)
1
0.9
[1. pad-token, pad-token]
Proposition 1. For the DGP in eq. (3), ROAR and FRESH assign their respective optimal scores to the encoding explanation Pencode(X)
Proof. First, note that
X3 Xyl.y.
See Table 5 for the probability table for why this is true.
For this proof let e(x) Cencode (x). In the example eq. (3), masking out the inputs selected by e(x) would mean that
X3
c(x) (1 e(x), [x2,
X3 = 0 x-((x) (1 e(x)) xx)).
In turn, by the construction eq. (3)
X Lyy X3 = x-((x) X3 (x (x-e(x)) X3) Xay
ee( (26)
X x-e(x)
X3
where the conditional independence in the second step turns into the joint independence the third step due to x; l.
ROAR scores an explanation highly if x-exx) predicts the label poorly. So x x-e(x) independent of y. then e(x) would be scored optimally. Then, due to eq. (26), ROAR scores an encoding explanation optimally.
FRESH scores explanation highly if the selected value val vxl(xx(x)) (as defined in Definition 4) predicts the label well. From Table 5, we see that pfy
x) 0.9 for all cases with val(Xx((x) = [1, pad-token, tadetoken] and p(y = 1 x) = 0.1 for all cases with val((x(x)) = [0, pad-token, padetoken). Thus, if val (x)()) [1, nod-no pad-token] then
p(y x) 0.9
Ex )= tow.n.pad- [0.9]
Ex val(Xe(s) (l.paddtoooo..pdd-tttt [p(y = 1 x')]
ply val = [1, pad-token, pad-toen))
and if = [0, pad-token, pad- then ply x) 0.1
Ex'r<p(x' (x (0pppppdooootttoc.o.dddd [0.1]
(0a0://000.ddooooe.ppppeeeeeeepe P(yy 1 x')
ply vxl((x())) = [0, pad-token, pat.teennpddtooeen]
Therefore, in all cases, p(y = x) = ply = val val(Xx(())), Thus, predicting label from the selected value alone as good as predicting from the whole input. As result, FRESH scores this explanation optimally.
31

---

B.7 Showing that Cencode is the optimal reductive explanation for eq. (3) and scores better than a constant explanation under EVAL-X
We repeat the DGP in eq. (3) here
x
[1, X2, X3] ~ B(
Xi
w.p. 0.9 else 1 - X1
if X3 1, if X3 0.
y
X2
w.p. 0.9 else 1 X2
Lemma 5. In the DGP in eq. (3),
g(y X1 = 1) 0.7
9(y 1 X2 1) 0.7,
0(y 1 0) 0.3,
( 1 X2 0) 0.3.
q xs) 0.5.
Proof. We can compute these values from Table 5.
Proposition 2. Let ce(x) Es. Then, for the DGP in eq. (3), EVAL-X((g, Cencude) EVAL-X((q,ce Ce).
Proof. By Lemma 5, we have
EVAL-X(q, ee) = xx log g(y (x3) = xg log 0.5 x -0.69.
Now, denote Gencade E. for ease of reading
EVAL-X(Q, 6e)
B(v.a) By (va)) q(y Xv, a)]
q(x3- = 1) 1) Eggy |x, log x1)
q(x3 = () 1 gyy x2)
0.5 0.5 (0.9 log 0.7 0. log 0.3) #2
+ 0.5 * 0.5 * (0.9 0.7 0.1 0.3)
-0.44.
This concludes that EVAL-X(q Cencude) > EVAL-X(q,ee).
Lemma 6. In the DGP in eq. (3), (x)c is an EVAL-X-optimal reductive explanation and is encoding.
Proof. First, the following properties show for the DGP because when X3 1, y only depends X1, and if X3 = 0. only depends on X2!
X3
yilxi
X3 0.
These independencies imply that
<(y x ====x,,xxx0 = 9(y x2, X3 = 0). Then, the optimal explanation function that achieves EVAL--X* is e(x) = [1,0, 1] if X3 and [0, 1, otherwise.
gyy [x1, x2, 1)) gly x1, 1)
32

---

Reductive explanations of size 1. If the explanation is forced to have fewer than 2 inputs. the optimal reductive explanation e(x) is only allowed to be one of {1, E2. E3:
max e
B
[[e(x)) Ei]g(y x.). (E(12233
Σ
Rewriting this expression to split the support of based on X3 1 or 0:
xx)xx
1[e(x) = Ei] log gly x.)
(E(1,2,3)
=q(x;= 1)xx(xx Xx B x)m1)
Σ
1[e(x) Eil log q(y x;)
$E(1,233
+ +(xx3 = O) (x==))Boss Boly.x
Σ [[e(x) Ei] log gly xx))
x==0)
(E(1,2.3)
0.5 B x==1)
Σ
[e((x) Ei] logg(y x;)
KE (1,2333
+ 0.5 (gxxx Bycy X2 x0=
Σ
[[((x) 6, log xx)
(1,233))
xxx) Bolyi x3= [[e([x), x2, 1]) = E1] log <(v |x1)
2
+ 1xy xx=1)
Σ
[|(([x), 1J) lo lo xx))
$22.3
By((x))
Bycr.xa x==0) [e(([x) X2, 0)) E2] log x2)
Bglyy xa xs=0)
Σ [e((x), x2, 0)) Ed] log 4(y xx)
1,1
2 Byxxixx xx==)) [|(([x), X2, 1]) = E1] Byty x1) log g(y xx))
Σ
[e([x), 11) = Eg 1n ly xx)
$22.
(27)
+ - E (x xx=0)
[[e([x), x2, 0)) &2 gg x2) log q(y x22)
[e([[x, X2, 0I) Eilletty xxxx= log q(y [x,)
1,
(28)
We will now focus on the three terms within each of eq. (27) and eq. (28). Due to the following equality
q(y = 1 x1 = 1, 1) = g = x1 0, X3 = 1)
g(y
x2 1, = 0) = g(y 0 x; = 0,x; 0) = 0.9,
the expectations in the first terms in each of eq. (27) and eq. (28) are
Egly x1)x log g(y x1) = E
E,ty x2>>3=1) log q x²) = 0.9 log 0.7 + 0.g 0.3 = 0044.
Next turn to setting = 1 term in eq. (27). Due that q = 1 x2 = 1) = q(y = 0 x,2 = 0),
X1 X2
Ey(y xx log q(y x2) (0.9 log 0.7 + 0.1 0.3) -0.44
33

---

X1 $
Eggy x1.xa=1) log ((y x²) = (0.9 0. 1.12.
The same equalities hold for the i = 2 term in eq. (28) By x2,x,=) log g(y x1). Finally, regardless of X1,X2, the i terms in both eq. (27) and eq. (28) can be expressed as follows:
Boly x1xx=1) log gly x2) Byly xx.x==0) log gly x1) (0.9 log 0.5 + 0.1 0.5) x 0.69.
Now we can maximize the sum of eq. (27) and eq. (28), over e(x) such that [e(x)] 1.
Notice that setting 1 1ee(x) = {i] = when X3 = and 1 [e(x) = &2] = 1 when X3 0 achieves the highest score -0.44 in each of eq. (27) and eq. (28). This implies that optimal reductive
explanation is { = [1, 0, 0) if x3 = and { = [0, 1, otherwise. This is an encoding explanation
as we show below. Due to Egg X3+
<(y
x1, Egg 1) 0.9 # g(y 1 x1, Egg 0) 0.5.
In turn, + Eg, X for {x e(x) { and Def: Encoding holds, meaning that e(x) is encoding.
B.8 An example of misestimation of EVAL-X Consider the following example where
X1, X2,
X2. B(0.5) X1. N),,I), y= ① X3.
Assume that the misestimated EVAL-X X model satisfies these equalities
misestimatedd (y x1) for all X1,
misestimated (y x1) = for all x1.
There exists bad explanation that scores optimally under the misestimated EVAL-X:
E if X3 1, x + X3 = 0. if
e(x)
&
Then the EVAL-X score of this explanation under this particular misestimation is
EVAL-X misesiate (g, e) = E.g (lgg Ge(x)
misestimated (y Xe(x))]
= =yl 1)B, [log imisesstmatedd (y xe(x)) ly 1] + gly O,,,, [log misestimmaed (y xe(x)) y = 0] miscstimated (y xx1) y 0]
(e(x)
(e(x)
0.5 Eg (log gf, oe misestimaned (y xx1 y 0.5 Eg [log qE.
0.
Since y is deterministic given the maximum value of the EVAL-X score is also 0. So the bad explanation scores optimally due to misestimation. Deterministic y x is not necessary for estimation eиror to affect explanation quality. Here, with this incorrectly estimated EVAL-X, inputs that are pure noise, independent of everything, will be chosen.
B.9 Attention map explanations be encode.
Here, treating each of the coordinates of x as tokens, we consider cross-attention based predictive model of the following form: with y(a) softmax function over vector a, W as matrix, at, B as vectors, and a as the sigmoid function, and x. as the temperature, the predictive model f(-) is
f(x) = I B
Σ (xxx Wx),
34

---

We then show that using the highest attention score as the explanation produces an encoding expla- nation. For this example, we consider the following DGP:
11 22, 23 ~ (()5 193
zt 21 + 1, 0, +1],
2 0
12 - 1, -1],
zt if 23 = 1,
(29)
2 if 23 0,
(x1)
if 1,
y B(p) where
(x(-))
x; 1.
The following setting of parameters in f(x) produces function of x that converges to P as - xx:
0
9) (30)
a= = -1,0)
== [1, 1, 0).
W
0
By definition of W
1fx0,1]ifx3-= =1
Wx
[0,0,-1]ifx;
if = 1
[1,0,0] if 1
X, Wx
0,,0,0 x-
[0,0,00 if -1
[0,0.0]ffx =1
[0,0,0 =
= X2 Wx
(x2
[0,22 + 1f x===1=-
10, if = -1
Then, B3 0, the inner sum for 3 does not appear in the function f(x) Then, 0g
0,
01 = 1, 02 1,
X1 x = 10ifx3 -1
ΣT(xx * Wx)jayxx, >
00
(31)
f0ifx: X3 =
En(nxe Wx),ayx,
(32)
-X2 if -1
In turn, as B1 = 32
x if X3 =
Σ B,
Σ En<x= (Xx)xx+
K1000
-x2 if X3 -1
1,1
1)
Wx) < x=30 (x(x)) if = 1 (o(-x2) if -1
=x o
Σ 8x 12L.2
Σ <(xx; <
(33)
So, as K - cc, the function fxx), with the parameters in eq. (30), converges to p(x). meaning that this model will achieve the population likeeiill optimum under the DGP in eq. (29).
Now, the attention map as an explanation selects x; if x; and x2 otherwise; this comes from eq. (31) and eq. (32). This is an encoding explanation because Eg, = 1 if = 1 which gives
g(y x1) + gly X1, = 1) - E Xe:
C Experimental details
C.1 Estimating STRIPE-X
To compute the KL term in ENCODE-METER, we estimate (EE, x): and q(E, xx). To estimate these, we train a single model - to predict Ey from Xy and a new variable É that can equal
35

---

the label y or a dummy value null that is outside the support of y in the following way:
arg max Bx.xxx()))
po po (Ev = [[e(x) = v| xv, y)
0
+ log (Ev 1[e(x) = v| xv. = mull)
(v,f=nuul]
(34)
As log-likelihood is proper scoring rule and g(y = mull) = 0, the maximum above is achieved when
po EEv Xv, y) - q(Ev Xv,y = y) po (e xv mull) ((Ev xv).
In summary, to estimate ENCODE METER, solve eq. (34), its solution to estimate the KL term from the RHS in eq. (7) for each Xv. y, and then average this KL term over samples of Xy from the data such that e(x) = and samples of y from the EVAL-X model for g xx).
In practice, one does not need train a model for each v. We describe how estimate ENCODE- METER with a single model in Appendix C.2. We give the full STRIPE-X estimation procedure Algorithm Appendix D.
C.2 Estimating the encoding cost in STRIPE-X with categorical predictive models
STRIPE-X consists of the EVAL-X score and cost of encoding measured by ENCODE-METER De- fine V be the set of possible explanations and let Vijj denote the jth element of V. The EVAL-X model p- (y Xv) is trained to predict the label y from subsets Xv where is uniformly sam- pled from V [20]. Next is computing the ENCODE-METER (c(c) that is used in the encoding cost term in STRIPE-X. For each explanation, let F be the categorical variable (instead of an indicator E.)) that denotes, for each sample, which inputs were selected by the explanation e(x): j
Evyl = 1 [e(()) V[i] = 1. Let q(i) be the distribution over induced by g(e(x)). We train model pa (F Xv. v) with modification of eq. (34) that averages over ~ q(e(x)):
arg max (ex(((())) E xxx,))x)
Σ Vbjev
[[e(x))
VDii log = Xv, v.+)+
log (F = Xv, l null, vil)
(35)
The variable l takes values in {-1, 0, 1) where and correspond to y = 0 and y = respectively and 1 corresponds to the null value. For flexible enough model pe that achieves the population maximum of eq. (35), for any v = Vbil E V,
Po (F = xv, y, =) (F j Xv: = mull, m) x
xv.y=y),
=
This fact indicates how one can use the model to estimate ENCODE-METER, First, construct the explanation dataset D. = {(y, xe(x))} from D.. Define QD, to be the uniform distribution over D.. Define E(v.a) as the uniform distribution over K samples of y from the EVAL-X model:
E(v.a) = U [1S)eak (hheee 3 ~ py(y Xv =a).
Then, estimate ENCODE-METER follows:
Po ( = Xv. 1 y. v)
((g.e) =I(vva)rgo, (X.(a)) BynEr
PF = _x,,,,e 1= 8, log
Pe = Xv, null, v)
Pe F # x,, výv
pa OF Xv, l =,, log po (F + j Xv, fonf=nulllvv
C.3 Estimating ENCODE-METER with generative model
When estimating the STRIPE-X score with procedure above for many different explanations, the maximization in eq. (34) repeated for every explanation, which can be computationally expensive.
36

---

Table 6: Position-basede prediction- based, and marginal explanation schemes are all encoding. For samples in the set {x e(x) v) for one of the selections v that e produces, accuracy < 1 and the KL being non-zero means these explanations are encoding per Lemma
Encoding
Acc. Ey (t)
KL (4)
POST
0.61
0.88
PRED
0.51
0.18
MARG
0.51
0.20
This motivates a second procedure to estimate ENCODE-METER that avoids having to retrain models for each explanation by using generative model for q(x X): Formally, with X, fixed, the conditional mutual information term in eq. (5) can be computed as the marginal dependence between N samples of y from q(y Xv) and q( EE Y). The model for the former is available from EVAL--X estimation. Simulating from the later, namely q EEv Xv. y), is done by sampling from the generative model x Xv. y and then computing the indicator Ey as 1 [e(x) = v). Mechanically, with an estimator mutual information from samples ((a")isn {b' FigN) denoted MI((a') (b')) and with samples {a'} produced conditionally on values e' denoted by a subscript of the conditioned value {a') Jei, one can estimate SECODE-METER as follows: sample V(v.a) y Xy = a and
x vayy ~ q(x Xv a,y -ay==y' y²) repeatedly times and compute
((y MI (vy") (aama):f/eeee f1(e(x') = vl)vay.))
We give the full procedure in Algorithm 1.
C.4 Experimental details from the simulated study.
The tata-genn processes from the experiments. Let be the standard normal distribution and let ((a) be the Bernoulli distribution with 1 occurring with probability a. With p = 0.9, the discrete DGP is:
x X1. X3 X X3] B(0.5)8s
Xi
w.p. P
else
x1
if X3 1,
y
X2
w.p. else
X2
if X3 = 0.
(36)
The hybrid DGP is as follows: with 7 5 and (xx) Itexpl zz]
the sigmoid function
( X3x = 1, p (x()))
x X2, X2, X4, x5] ~ N((0.)511 X3 B(0.5),
y~ (37)
if = O,
Computing accuracy and KL to show that POSI, PRED, MARG are encoding. For each encod- ing type, we build two decision trees from 1000 samples from eq. (36): the first decision tree learns q(E, Xv) and the second learns g(y Xv, E, = b) for b E (0, We set the maximum depth to be Trees of this depth learn any function of binary digits; x with Ey as an additional column amounts to 6 binary digits. These decision trees are used to compute the accuracy of predicting E, with EEv Xv) and the KL between q(y Xy. E, = 1) and g Xv, Ey = 0). Within a set {x e(x) v} that is all x that have one of the possible selections v. Table 6 report the accuracy of predicting E, with (Ev x,) and the KL between q xx E, = 1) and q(y xy: E, 0), averaged only over samples in (x e(x) v).
EVAL-X. To estimate EVAL-X for the DGPS in eq. (36) and eq. (37), we compute conditionals ((y = 1 xv) via Monte Carlo approximation. Due to the different coordinates of x being inde- pendent, one can compute q(y xv) as a marginal expectation over the inputs except those in
glyl xv) = Egixy g Xv, x) = EExx x)..
We Monte Carlo estimate the RHS of this equation over 500 resamples of x.. We take 5000 samples from each DGP to estimate EVAL-X scores with respect to q(y, x). Appendix C.5 we also show experiment results where we use the EVAL--X accuracy and AUROC as the score instead.
37

---

1.0
1.0
1.0
1.0
-AURC
0.5
0.5
0.5
0.5
EVAL-X'AAC
EVAL-X' AUROC AUROC
EVAL-X
0.0
0.0
0.0
RRED
POS:
MARG
REALOX
PRED
MARC
REAL-X
PRED
POS
MARG
REAL-X
FRED
POSi
MARG
Re REAL
(a) Results: Discrete DGP.
(b) Results: hybrid DGP.
Figure 6: The EVAL-X-ACC and AUROC scores for the different explanations for the discrete DGP on the left and the scores for the hybrid DGP in the right. In both, multiple encoding explanations (PRED, MARG, and reductive one from REAL-X) all achieve Thus, ranking metrics like accuracy and AUROC are not sensitive to encoding explanations like EVAL-X log- likelihoods, and can fail to even weakly detect encoding. This stems from the fact that accuracy and AUROC only depend on the ranking of the datapoint, and therefore not sensitive to differences in log probabilities that do not change ranks.
same score as the corresponding EVAL--X score.
REAL-X. We solve REAL-X for any specified explanation size K as follows. In the case of the discrete DGP, for each possible value of x E supp(q(x)) (of which there are finitely many), we make e(x) output the subset of at most size K that achieves that maximum averaged log-likelihood over the samples that equal said value x = x. This produces the didimalll-scorinng explanation e(x) that maps each finite value in the support of q(x) to one subset of the coordinates of x. To do the same in the continuous DGP in eq. (37), we round x to integers and then use the same type of optimization as in the discrete case.
STRIPE-X. In estimating ENCODE-METER, the model po Xv, E) a decision tree of depth at most 5, which is then used to estimate averaged KL in the RHS of eq. (7) with a single sample from y Xy. The process is repeated for each v and averaged produce the ENCODE-METER, The simulated experiments were done on CPU with the whole runtime around 10 minutes.
C.5 Experiments with EVAL-X accuracy and EVAL-X AUROC
Here, instead of EVAL-X X log- lieiiiiiihoods, we the accuracy and AUROC of the EVAL-- model as the score. We call these EVAL-X-ACCC and EVAL-X-AUROC scores. These metrics only depend on the ranking of the datapoints, and therefore are sensitive to differences in log probabilities that do not change ranks. Figure 6 shows that, due to this insensitivity, multiple encoding explanations (PRED, MARG, and the excessively reductive one) all achieve the same score the correspond- ing EVAL-X" score. In summary, ranking metrics like accuracy and AUROC are not sensitive to encoding explanations like EVAL-X logliiiclsoooss
C.6 Classifying dogs and cats.
The POSI explanation selects the upper or the lower color patch depending on whether 4(y 1 x) 0.5 or not. The PRED explanation selects the patch predicting from which best matches the prediction from q = 1 x). The MARG explanation selects the top or the bottom image patch based on the color as in the DGP in Figure 5. We consider two non-encoding explanations. The first explanation, denoted optimal, selects exactly the features that occur in the DGP: [2], the color patch Xı is blue and {x1,xx otherwise. As y is determined by the explanation, meaning E. Xv for all v and values Xv, this explanation is non-encoding. The second one, denoted f fixed, always outputs the bottom right patch x;; this explanation is constant which violates the first criterion in Lemma meaning there is no encoding.
We also run the REAL-X method from [20] to produce explanation. REAL-X run over expla- nations that select one of the four quarter patches and exact marginalization over the selections.
The base cat and dog images were obtained from the cats_vs v_dgogs dataset from the Tensorflow datasets package. To construct images like in Figure 5, the color and the two images are sampled independently. The color being blue/red determines that the label associated with the top/bottom image becomes the label for the constructed image. The training, validation, and test dataset consist of 8000, 1000, and 1000 samples respectively.
38

---

We follow the procedure in Appendix C.2 to estimate STRIPE-X. The EVAL-X model is a pre-trained 18- 18-layer residual network. The model (F Xv. v) used in computing the ENCODE-METED term in STRIPE-X (eq. (6)) are 34-layer Residual neural networks. The EVAL.-X model is trained for 100 epochs with a batch size of 100 with the Adam optimizer, with the learning rate and weight decay parameters set to 10-3 and ( respectively. The Pa [F) (.v model is trained for 50 epochs with a batch size of 200 with the Adam optimizer, with the learning rate and weight decay parameters set to 5 x 10-s and 1 respectively. The pe model sees variable É through an entire extra channel where all the pixels take the value É. We used validation loss as the metric to early stop. The EVAL-X and STRIPE-X scores are computed on the test dataset. The cats vs. dogs experiment were done an A100 GPU where the whole training and evaluation ran in less than 20 minutes.
Remark on the gap between FRESH and EVAL-X scores for the optimal explanation. As the optimal explanation selects features sufficient to produce the label, meaning y_lxx Xy or y_lxx vx)(x) FRESH and EVAL-X log-likelihoods should be the same as predicting from the full feature set. One potential reason there is a gap between the two scores in table is that the FRESH and EVAL-X scores are computed with ResNet18 models that solve prediction problems of different levels of difficulty On one hand, FRESH is computed with a model trained for a single prediction task: predict from val(xx(x)) On the other hand, EVAL-X is computed with model trained for a more complicated task: for range possible v. predict from Xv. Using large models with appropriate regularization, such as weight decay, should mitigate the gap in scores.
C.7 LLM experiment details
We generate 10, 000 reviews of the following type: with ADJ1 and ADJ2 as adjectives, the review is
'My day was <ADJ1> and the movie was <ADJ2> that is it
'My day was <ADJ1> and the movie was <ADJ2> oh wait, reverse the adjectives'.
The second sentence in the review acts as a "control flow" input and determines whether ADJ1 or ADJ2 describes the sentiment about the movie. We prompted Llama to predict the sentiment and select words relevant to predicting the sentiment. In Appendix C.8, we give the prompts we used to make Llama 3 produce explanations from. For this problem, the inputs x are the reviews and Llama produces explanations e(x) that select a subset of words in the review. The summaries and explanations were generated for all 10, 000 samples but to estimate STRIPE-X, x, we only used data from the most common explanations (we restricted to inputs whose explanations v had high (e((x) v)). This resulted in a dataset of size 8136, which we split into a training, validation, and test datasets of sizes 6102, 1017, and 1017 respectively.
Both the EVAL-X model and the model for (F Xv, v. E) (see Appendix C.2) used in estimating the ENCODE METER term in STRIPE-X were finetuned GPT-2 models. For the EVAL-X model, used the Adam W optimizer with a batch size of 100 and trained for 50 epochs with the learning rate set to 5e - 5. weight decay set to 0, and Cosine learning rate scheduler with the number of cycles set to 1. For the pe model used in estimating ENCODE-METER we used the AdamW optimizer with batch size of 50 and trained for 25 epochs with the learning rate
5e weight decay set to 0, and a Cosine learning rate scheduler with number of cycles set to 1. The pe model sees variable l through the following word added to the input sequence of words: positive if L y 1, negative if l y = 0, and nothing if l = null. We used validation loss to early stop. We follow the procedure in Appendix C.2 to compute ENCODE-METER with po (F Xv, v, on the test data with the averaging over Xy estimated using a 5 samples per value of x.. All training and inference for this experiment was done on an A100. The explanation step and the estimation for both parts of STRIPE-X together took under 2 hours. The LLM-generated explanations achieves an EVAL-X score of -0.497 and an ENCODE-METER value was 0.114.
C.8 Prompts used to predict sentiment and produce explanation
In Figure 7, we provide the prompt we used to predict the sentiment from a review and generate an explanation for that prediction.
Figure 7: Llama 3 prompt used to predict sentiment and generate an explanation for that prediction.
39

---

System:
You are helpful and honest assistant. Please, respond concisely and truthfully.
You are asked to summarize movie revieus of the form "first sentence. second sentence".
The following are examples along with the reasoning.
Consider 'My day was moving and the movie was overblown that is it.' The second sentence means the second adjective 'overblown' describes the movie. Due to this description, the sentiment is negative.
Consider 'My day was moving and the movie was overblom. oh wait, reverse the adjectives."
The second sentence the first adjective 'moving' describes the movie. to this description, the sentiment is positive.
These all examples.
user: What is the sentiment about the movie in this review '<REVIEW>??
Think step-by-step about this latest review. If the second sentence instructs it, switch the adjectives and then based the descriptor of the movie, either 'positive' or 'negative'.
Explain why you chose those this sentiment by selecting as few words as possible from the review. Include all the words that you looked at.
Use this helpful format: "the sentiment <positive/negative> the explanation is <words, ...>. END.
D Algorithms
Algorithm describes an alternate way estimate the ENCODE-METER component of STRIPE-X with a conditional generative model. Algorithm 2 describes the predictive version of STRIPE-X estimation, which we used in experiments.
40

---

Algorithm 1: ENCODE-METER, generative version.
Input: Training data D ~ q(y, x) and test data Dr ~ q x), explanation function e(x), penalty weight EVAL-- model p, (y xy). Conditional generative model px Xv, y) and mutual information estimator that takes two sets arguments
MI [c {:
Result: Return estimate of STRIPE-X
Define J (v.a) as the set K random samples of y from the EVAL-X model:
J(va) (ý")&s where y (y x, a)}
2 Define L(v as the set K random samples of from pe conditioned on a and y:
[(v.a.J = {1(e(x*) = vllAsk { [wheee x ~ px xv =.y= }²)}
Construct the explanation dataset D. [(Xe(x) (e(x),a))} from D..
Define QD, to be the uniform distribution over De. Compute the following averaging of estimated mutual information between, J,
2 (e) [E (x) MI (((v.a) L(v.
Return ò(q. e) as the ENCODE METER estimate.
41

---

Algorithm 2: STRIPE-X, predictive version.
Input: Training data q(y, x) and test data D, q(y, explanation function e(x).
penalty weight A. Specifications for the models (y Xv) and Po Xv, ().
Result: Return estimate of STRIPE-X:
Define to be the uniform distribution over D)
2 Construct the explanation dataset D. = (x())) from Dr
Define qD. to be the uniform distribution over D..
Estimate EVAL-X()
Solve the following minimization problem to learn (y xv)
arg max E Bx. (y xx)]
Output: Py
Estimate ENCODE-METER METEE ()
Construct the set of possible selections V {v g(<(()) 0}
0}
Construct data of the form (x, F) where F = j if Eyy] 1
Fit the model po (F Xv, () via the following log-likelihood maximization:
arg max (exx)( Rxy (xx))
Σ VIEVV
[[((x) = [og = xx,, -..--yv))
+ log (F = Xv, e mull,
10
11
Define E(v.n) the uniform distribution over K samples of y from the EVAL-X model:
{(va)) U [[y]esk where 8* ~ ~p x, =a) a)}
Output:
The following nested expectation over QD, and E(-):
po ( = Xv. l = ÿ, v)
E (v.a)gn, (K.w)) E (po(F= =
- Xxy v,v) log
Pe (F = Xv, l mull, v)
pe + Xv. v.v)
+ Pa (F
(F * Xv. y, v) log
pa [ + j Xv. =mull,,)
12 Learn the EVAL-X model Py < EVAL-X
Estimate ENCODE-MENTE METER the (e(e) ENCODE-METER ().
14 Return the following as the STRIPE-X estimate:
((v.a)~gp, (Necse) Byrgpaly (vog)(v [logp, (y =y Xv =a)] (e)
42


Reviews:
[{"review_id": "6QCTHllSRz", "reviewer": "Reviewer_Fr3U", "summary": "How to best evaluate explanations is an important open question in the field, and one specific challenge that has so far received less attention is how to detect when explanations encode prediction in the identity of the selected inputs. This paper proposes a formal definition of encoding, that they later use to probe whether current evaluation protocols can detect encoding through experiments on simulated data. They find existing evaluation methods to be at best weak detectors of encoding, and they propose the evaluation method DET-X as a strong detector of encoding.", "questions": "- Fig.1C, \"the label is produced from the bottom image if the color is red\" then why is the explanation about the top image?\n- l.156 \"For example, consider reviews of the type \"My day was terrible, but the movie was [ADJ].\" where ADJ can be \"good\" or \"bad\" and let the explanation be \"terrible\" if ADJ=bad and \"Movie was good\" if ADJ=good. The sentiment about the movie comes from the second part which the explanation fails to isolate, meaning this explanation should be scored poorly.\" I am not sure the problem is that straightforward here. How would we differentiate a model that makes that mistake from an erroneous explanation?\n- l.165 \"control flow\" is not defined, hence it's hard to understand this section\n- l.366 \"We compute ENCODE-METER with q(Ev | xv) modeled by a ResNet34 trained the same way as EVAL-X but, instead of predicting the label, it predicts the identity of subset selected by the explanation\" I don't understand what this means, could you please clarify?", "limitations": "I do not see a limitation section in the paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The motivation for defining and properly evaluating the notion of encoding is sound\n- the experiments look sound\n- as far as I know, the formal definition is a novel contribution", "weaknesses": "- The main and major weakness of the paper is clarity.\n\t- The paper is very very dense (space between paragraphed tempered and heavily reduced) and this makes it at times intelligible. \n- Weak contextualization with regard to prior work. The related work in the main paper is very succinct. \n- Overall I believe that the substance of this paper is interesting, but in its current form, the paper is really hard to read and understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful feedback. If our response below addresses your primary concerns, would you kindly consider raising your score?\n\n**[paper is very very dense and this makes it at times intelligible]**\n\nWe thank the reviewer for this feedback. We modified the draft to use the standard paragraph spacing with the following changes to the writing\n- Made section 2.1 more concise, where each encoding example now only spans a single paragraph covering both intuitive example formal construction.\n- Moved lemma 1 and associated text to the appendix.\n- Made 5.1 and 5.2 concise by only listing results and conclusions, and moved experimental details about training models for the encode-meter to the appendix.\n\nIf the reviewer has any other concerns about the clarity/density, we'll be happy to address them.\n\n**[Weak contextualization with regard to prior work]**\n\nThere are limited studies in evaluating explanations and we discuss them. Due to space constraints, we had moved the related discussion about faithfulness and label-leakage to appendix B. We have moved it back and give it here:\n    \nOther investigations into evaluating explanations focused on label-leakage [14, 34] and faithfulness [11, 16, 35, 36, 37]. Akin to encoding, label-leakage is an issue that occurs with explanations that depend on both the inputs and the observed label; such explanations, when built naively, can yield attributed inputs that are more predictive of the label than the full input set. In this paper, we do not consider explanations that have access to the observed label. Faithfulness, intuitively, asks that the explanation reflect the process of how a label is predicted from the inputs. Typical faithfulness evaluations rely on the quality of prediction from the selected inputs [12, 16]. Jacovi and Goldberg [11], Jethani et al. [13] discuss how such evaluation methods are insensitive to encoding by checking how they score encoding constructions.\n\n    - [11] https://arxiv.org/abs/2004.03685\n    - [12] https://arxiv.org/abs/1806.10758\n    - [13] https://proceedings.mlr.press/v130/jethani21a.html\n    - [14] https://proceedings.mlr.press/v206/jethani23a.html\n    - [16] https://arxiv.org/abs/2005.00115\n    - [34] https://aclanthology.org/2020.findings-emnlp.390/\n    - [35] https://arxiv.org/pdf/2111.07367\n    - [36] https://ojs.aaai.org/index.php/AAAI/article/view/21196\n    - [37] https://arxiv.org/abs/2109.05463\n\nIf the reviewer sees a work we missed we would be happy to add it.\n\n**[Fig.1C, \"the label is produced from the bottom image if the color is red\" then why is the explanation about the top image]**\n\nThere may be a little misunderstanding here. In this example, the explanation selects the top left quarter patch only, which only contains color. This explanations encodes information about the label in the selection. The reviewer may also be asking why such a construction can be called an explanation. As the overarching goal in the paper is evaluating explanations, we consider any functions of inputs that outputs binary masks as candidates.\n    \n**[l.156 \"For example, How would we differentiate a model that makes that mistake from an erroneous explanation]**\n\nWe respond here assuming that the reviewer used the word \"erroneous\" to mean that the explanation selects inputs other than the ones that the model uses to predict. If the reviewer meant something else and clarify their comment, we would be happy to engage in further discussion.\n\nTo understand whether explanation selects the input the model depends on, one would have to look at how predictive the selected inputs are of the *labels produced by the model* . Let's look at evaluating a candidate explanation of a models predictions without assuming anything about how the explanation or the model were produced.\n\n- If the model makes mistakenly relies on the word \"terrible\" and the explanation correctly selects the word \"terrible\", then the selected features would have maximum information about the *model-predicted labels*.\n- If the model correctly depends on value that ADJ takes but the explanation selects the word \"terrible\", then the selected features would can only be informative of the *model-predicted labels* by encoding.\n    \nDue to these differences, an encoding explanation of a correct model and a non-encoding explanation of an incorrect model have different signatures that EVAL-X or DET-X would detect.\n\n**[l.165 \"control flow\" is not define]**\n\nThank you for pointing this out. Control flow is not common parlance and comes from the software engineering literature. We explain it below. We use the phrase \"control flow\" to indicate that the DGP looks at the value of one input to determine which other inputs to produce the label function; this process reflects an if-else statement where the condition being checks determines the flow of computation. We have added this paragraph to the paper.\n\n**[l.366 \"Trained the same way as EVAL-X but, instead of predicting the label, it predicts the identity of subset selected by the explanation?]**\n\nEVAL-X trains a model for $q(y | x_v)$ that predicts the label $y$ from the selected inputs $x_v$, by randomly choosing $v$ for every sample. Similarly, ENCODE-METER relies on the conditional distribution $q(E_v | x_v)$. To model $q(E_v | x_v)$, we replace $y$ with $E_v$ in the EVAL-X training procedure. Here, $E_v$ is the indicator of whether the explanation for the input $x$ is the subset $v$.\n\n\n**[I do not see a limitation section in the paper.]**\n\nWe discuss limitations in the Discussion section 6, paragraph \"Mis-estimated models, ... \". We elaborated on the limitation in the general response [LINK](https://openreview.net/forum?id=mkw6x0OExg&noteId=aAzltWtusG). If the reviewer can point to other questions or limitations that we should discuss, we are happy to add them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KvssAOT4o9", "reviewer": "Reviewer_A1S9", "summary": "The paper presents a novel approach to evaluate feature attribution methods in machine learning by addressing the issue of encoding in explanations. The authors define encoding as when the explanation's identity provides additional information about the target beyond the selected input values. They categorize evaluation methods into weak detectors (optimal for non-encoding explanations) and strong detectors (score non-encoding explanations higher). The paper introduces DET-X, a new score that strongly detects encoding, and empirically verifies its effectiveness through simulated and real-world datasets, including an image recognition task and sentiment analysis of movie reviews.", "questions": "- How does the DET-X score perform in different domains outside of image recognition and sentiment analysis?\n\n- Are there specific conditions or types of models where DET-X might not perform as expected?\n\n- How does the computational cost of DET-X compare to existing evaluation methods?", "limitations": "- The paper's findings are primarily validated on specific datasets and tasks, which may not generalize to all types of machine learning applications. \n\n- The DET-X score's implementation complexity might be a barrier for widespread adoption, especially in resource-constrained environments.\n\n- The paper highlights the risk of misestimation in model-based evaluations, which could impact the reliability of DET-X in certain scenarios.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a precise mathematical definition of encoding, addressing a significant gap in the interpretability literature.\n\n- The classification of evaluation methods into weak and strong detectors provides a clear framework for assessing the robustness of feature attribution methods. \n\n- The authors rigorously prove that their proposed DET-X score strongly detects encoding, differentiating it from existing methods.\n\n- The paper includes empirical validation on both simulated data and real-world applications, demonstrating the practical utility of DET-X.\n\n- By uncovering encoding in LLM-generated explanations for sentiment analysis, the paper shows the relevance of its contributions to current AI applications.", "weaknesses": "- The proposed DET-X score may require complex implementation and computational resources, which might limit its adoption in practical scenarios.\n\n- While the experiments are thorough, they are limited to specific datasets and types of tasks (image recognition and sentiment analysis). More diverse applications would strengthen the claims.\n\n- The paper acknowledges that misestimation of models used in evaluation could lead to incorrect conclusions, suggesting a need for robust estimation techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful feedback. If our response below addresses your primary concerns, would you kindly consider raising your score?\n\n**[The proposed DET-X score may require complex implementation and computational resources, which might limit its adoption in practical scenarios.The DET-X score's implementation complexity might be a barrier for widespread adoption, especially in resource-constrained environments.]**\n\nWhile the reviewer is correct that one needs to estimate the ENCODE-METER in addition to EVAL-X, the training process is standard supervised learning. The only difference is that the inputs are randomly masked like in EVAL-X. As training and evaluating predictive models via supervised learning is well-studied, even at scale, we do not foresee that training DET-X as a difficult task.\n\nThe reviewer is correct that DET-X requires twice as much computation as EVAL-X, which itself can take more computation than training a single model that predicts the label from the full inputs. This extra computation comes from have to learn to predict from different subsets. However, one cannot escape training to predict from different subsets when evaluating explanations based on how informative the selected inputs are. Using large pre-trained models does speed up this process. The estimation of encode-meter with GPT-2 in the LLM experiment, including hyperparameter tuning, took under a single day on a single GPU. Alternatively, given a conditional generative model for the full inputs given the masked ones, both components of DET-X reduce to averaging a single model's predictions over generated samples.\n\nCode for training models from masked images for EVAL-X already exists and we will release our implementation for DET-X with the camera-ready  version, if accepted.\n\n\n**[The paper acknowledges that misestimation of models used in evaluation could lead to incorrect conclusions, suggesting a need for robust estimation techniques. The paper highlights the risk of misestimation in model-based evaluations, which could impact the reliability of DET-X in certain scenarios.]**\n\nThank you for highlighting this point. The problem with misestimated models goes beyond our work and applies to all evaluation methods that build a model to score explanations, including ROAR, FRESH, and recursive ROAR. Avoiding misestimation is unavoidable, but post-hoc methods for capturing the uncertainty of predictions, via conformal inference or calibration, can help mitigate the errors in evaluation due to misestimation.\n\n\n**[While the experiments are thorough, they are limited to specific datasets and types of tasks (image recognition and sentiment analysis). More diverse applications would strengthen the claims. How does the DET-X score perform in different domains outside of image recognition and sentiment analysis?]**\n\nThe goal of the paper is to establish the mathematical definition of encoding and strong detection of encoding. So we focused on two popular tasks for which explanations are produced. \n\nWe have since added a tabular data experiment and report the results here. We ran an experiment on tabular CDS Diabetes dataset from the UCI repository and show the effectiveness of EVAL-X and DET-X on weakly and strongly detecting encoding respectively. See the PDF in the general response  [LINK](https://openreview.net/forum?id=mkw6x0OExg&noteId=aAzltWtusG).\n\nThe results show that\n1. EVAL-X scores the Optimal explanation above all the encoding ones, showcasing weak detection. \n2. However, EVAL-X scores the last non-encoding explanation, which selects features informative of the label, below the encoding ones PosEnc and MargEnc, showing its not a strong detector.\n3. DET-X correctly scores all the non-encoding explanations above all the encoding ones, demonstrating strong detection\n\n\n**[Are there specific conditions or types of models where DET-X might not perform as expected?]**\n\nDET-X is model-agnostic and would extend any type of model as long one can obtain input-output pairs from the model. DET-X depends on models that predict the label from subsets of the inputs. Learning to predict from subsets may take much larger models than predicting from the whole input set when the distribution of the label changes dramatically between conditioning  two similar input subsets. As we discuss in section 6, DET-X only works for explanations the select subsets of features. Future work can extend the notion of encoding to free-text (natural language) rationales.\n\n\n**[How does the computational cost of DET-X compare to existing evaluation methods?]**\n\nDET-X has twice the computational time of EVAL-X because it runs the EVAL-X training process twice, but the two models are independent and can be trained in parallel. Compared to methods like ROAR, the relative increase in computational context might depend on the problem at hand as the training procedures are different which mean optimization may converge at different rates.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3mwnAfmtQ9", "reviewer": "Reviewer_PabQ", "summary": "This paper claims that the main problem regarding the evaluation of explainability methods is encoding, which refers to the leakage of information in explanation (here defined as a boolean mask for selecting features) structure and not the value of the selected features. The paper tries to quantify the extent of this leakage and analyzes some previously designed methods from the proposed perspective, summarized in Table 1. They continue an intuitive approach to this quantity and use some definitions to build a measure termed \"ENCODE-METER\". This measure helps them to tweak the EVAL-X objective, they call DET-X, to gain \"strong\" encoding detection in definition 3. The claims, in this work, are supported by proofs and sometimes based on explanations that come from hand-designed data generating process for constructing counter examples. Finally, they evaluate the proposed framework on some data generating processes quantitatively.", "questions": "1. in figure 1 (c), the assumptions are unclear, why position based encoding explanation always selects the top-half? is it in the DGP that the cat always appear on the top? Or it is in the definition of encoding based explanation. More on that, how do we know \"The label is independent of the color in the selected features\"? Shouldn't you show this formally using definition of independence?\n2. line 164 \"may seem okay\" what does this mean? is it better to adopt a more formal writing style here?\n3. line 164-164 \"because the color is independent of the label,\" can you show this more formally using the definition of independence? I think that the label well depend on the color, at least here.\n4. lines 165-166 \"hides ... the predictions’ dependence on the control flow input\" how the control flow can be shown in an explanation? I guess this does not fit in the definition of salience based explanations?\n5. lines 209-210 \"only work when optimizing without constraints\" what is optimization referring to here? what are constraints?\n6. refer to weaknesses section \"Mathematics writing\" number 2.\n7. in line 220-221 I do not understand why it is written \"x3 predicts the label\". Maybe I'm wrong, but it seems that the label could be 1 or 0 when x3 is known and equal to 1 (for example). So knowing x3 does not tell anything about the label, and you still need to look at the other features (either x1 or x2). Can you make a table of conditional probabilities and reason why x3 tells the label?", "limitations": "Although authors have answered \"Yes\" to the checklist for \"discussion on limitations\", I didn't find a separate limitations section in the paper. Maybe they have referred to the limitations of their work indirectly.\n\nOne limitation that I see in this work is that the arguments are based on assumptions that might never come true on real world datasets like Imagenet. This is why I encourage the authors to go beyond toy datasets and evaluate their method in the wild.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper adopts a formal approach toward its claims and goals.\n2. The paper provides intuition and summarizes it in concise mathematical equations.\n3. The goal and problem being tackled is of high priority in the field of explainability, and affects many researchers in the field.\n4. The direction that the paper is proposing seems quite promising and must be better studied.\n5. The paper tackles encoding that is not formalized yet and underdeveloped, which adds to the novelty of the work.", "weaknesses": "I would like to thank authors for such a hard work and contribution to the community.\nI have learned a lot from this work, so the points mentioned here must not seem disappointing, but rather it should help for a work with higher quality. It seems that this work is very close to publication, yet with small potential errors I pointed out here (nonetheless, I might be wrong).\n\n## Important missing discussions\n1. It would be helpful to rank some **known explanation methods** and compare it with **known metrics** like ROAR. Then it would be easier for the reader to decide what are the differences between your evaluation method and other evaluation methods. ROAR for example ranks many methods and reaches to a bottom-line and raises some open questions. Toy datasets are good for formal proofs, simplicity and demonstration. But relying on toy datasets can create a gap between modelling and practice. I would recommend having a table similar to ROAR that ranks methods. Maybe we see that the results of DET-X, although being more elegant in theory, actually coincide with the results one gets with ROAR or some other heuristic method.\n2. It seems that the main idea of this paper and some referenced papers revolves around spurious correlations in the DPG. Could be useful if there is a discussion on how spurious correlations in the DGP affects explanations.\n\n## Formal writing\n1. definition 1, does not look like a definition, it is written like a proposition. It is easier to understand if a definition is like \"We call X an encoding if Y holds.\"\n2. Definition 2 is also like a proposition, or at least I do not understand what is being defined here. Also in $\\max_e$ what is the domain of e here? Is it over explanation methods or over explanations that are generated by a dataset. What is the domain of $e^*$?\n3. does definition 3, depend on the direction of the metric (higher the better or lower the better)?\n4. I cannot follow the argument in line 234-239. Maybe because I've not understood the definitions 1 and 2. In a proof, to show that a property satisfied, finally to show that X fits in some definition.\n\n## Punctuation, writing style and minor errors\n1. line 71 \"that Hooker et al.\" should be removed\n2. it is better for readability of the paper to provide a more intuitive definition of \"encoding explanations\" earlier in the paper I would recommend putting this line: \"Intuitively, encoding is when the binary mask output of the explanation itself provides information about the label beyond the selected values.\" earlier. Even in the abstract or early in the introduction when you want to mention encoding.\n3. lines 123-124, \"... to predict the label have not been selected ...\" shouldn't this proposition stated weaker? something like  \"... to predict the label **may** have not been selected ...\"\n4. line 139 punctuation should be: \"likely**,** exactly**,** when\"\n5. line 100 punctuation better be: \"method is good**,** requires\"\n6. lines 154-161 reads poorly, maybe I do not understand what it tries to convey!\n7. DGP in line 164 is defined later in 191.\n9. in Eq. 3 the word \"Different\" better reflect the mathematical form than \"Additional\"\n10. lines 196-198 this is too informal, or at least I cannot make sense of it. \n11. line 208 punctuation \"provides\" must be \"provide\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the generous comments and detailed feedback. We fixed the writing issues in the paper. If our response below addresses your primary concerns, would you kindly consider raising your score?\n\n\n**[rank explanations, compare with ROAR]**\n\nWe thank the reviewer for raising this point. Unlike ROAR, EVAL-X and DET-X provably at least weakly detect encoding on any distribution, not just on the toy examples. We provide a comparison on the image recognition experiment; see the PDF in the general response [[LINK](https://openreview.net/forum?id=mkw6x0OExg&noteId=aAzltWtusG).]. ROAR incorrectly ranks multiple encoding explanations as high as the optimal explanation.\n\n    \n**[Encoding is like spurious correlations?]**\n\nOne could think of encoding as a spurious correlation between the selection and the label. This correlation is not a fact about the DGP that produces $(X,Y)$. It is a consequence of the explanation depending on features that are informative of the label but only selecting a subset of them.\n\n**[Def 1 and 2 sound like propositions, domains of $e$]**\n\nWe have updated the writing to make definition 1 start with \"An explanation is called encoding if ... \". In definition 2, we define the notion of weak detection for a distribution $q(y,x)$ as a property an evaluation method can have. The domain of $e,^*$ is that of all functions that takes and input $x$ and return a binary mask over input. For example, considers explanations for images-label pairs; the domain is the space of images.  We have clarified this in the draft.\n\n**[definition 3, depend on the direction of the metric]**\n\nThe reviewer is correct. We assume higher scores mean better. If the direction flips, Definition 3 would say $\\leq$ instead of $\\geq$.\n\n**[the argument in line 234-239]**\n\nDefinition 1 and 2 are about weak and strong detection, which are properties of evaluation methods. Lines 234-239 only define EVAL-X. Theorem 1 proves that EVAL-X is a weak detector. In other words, EVAL-X is a weak detector. The sentence 239 leads to theorem 1. We have updated the draft to separate the definition of EVAL-X from the sentence that says it is a weak detector.\n    \n\n**[lines 123-124, \"... to predict the label have not been selected ...\"]**\n\nThe updated sentence reads \"An encoding explanation should not score optimally under a good evaluation because the requisite input values needed to predict the label have not been selected by the explanation **on a subset of the input values.**\"\n\n**[lines 154-161 unclear]**\n\nThis is an example of a \"bad\" encoding explanation. We flesh out the example here. We elaborate here. Consider reviews that can either be of type \"My day was terrible but the movie was [ADJ1].\" and \"The movie was [ADJ2], but the day was not great.\" where ADJ1 can be \"good\" or \"not great\" and ADJ2 can \"not great\" or \"terrible\". \n\nDue to english parlance, \"terrible\" indicates bad sentiment more often than \"not great\". Then, in the example setup above, only seeing that the fourth word is \"terrible\" yields bad sentiment with higher probability than when only seeing that the phrase is \"not great\". However, the fourth word but does not always describe the movie. An explanation can look at \"not great\" describing the movie as bad but then select terrible to encode the bad sentiment. Such an explanation is encoding because it selects a word that does not describe the movie but is informative of the sentiment.\n\n**[in Eq. 3, \"Different\" vs. \"Additional\"?]**\n\nThe word additional denotes that knowing the explanation provide extra information about the label not in the selected values.\n\n**[lines 196-198 informal.]**\n\nThe lines say that the these explanations are encoding due to lemma 1. By \"the explanation e(X) varies with inputs other than the selected ones\" we mean that $e(X)$ depends on inputs that weren't in the binary mask. By \"... inputs provide information about the label .. \" we mean that the inputs that are not selected in $e(X)$, are informative of the label which implies the second condition in lemma 1.\n\n\n**[fig1c unclear. Posenc selects the top-half?  $Y \\perp $ color?]**\n\nWe construct PosEnc to select the top-left-patch if label=cat. Intuitively, the label is independent of the color regardless of which patch it selects, the label is only determine the animal in the image. We added math to prove this in the draft.\n\n**[\"may seem okay\" informal]**[\n\nThe selected inputs (third panel, second row, left square for each input) show exactly the animal of the label. Selecting only the animal of the label seems desirable at first; this is what we meant by \"may seem okay\". We then explain why such an explanation is encoding.\n    \n**[how to show control flow?]**\n\nIn the figure 1 and figure 3 examples, the \"control flow feature\" is the color patch, because it branches the DGP where the top right or the bottom right patches produces the label. \n\n**[209-210 optimization? constraints?]**\n\nOptimization here refers to finding an explanation that maximize the evaluation score $\\max_e \\alpha(q, e)$. Various constraints can be put on explanations, such the explanation cannot select more than $K$ inputs.\n\n**[220-221 \"x3 predicts the label\"? why?]**\n\nThe reviewer is correct. However, conditional on the selected inputs, does predict the label. See the PDF in the general response for the math.\n\n**[Discuss limitations]**\n\nSee the general response [LINK](https://openreview.net/forum?id=mkw6x0OExg&noteId=aAzltWtusG).\n\n\n**[Based on assumptions that may not be true on real data like Imagenet.]**\n\nEvaluations like ROAR and EVAL-X have been used on explanations on imagenet and on chest x-rays. We prove such methods highly rank undesirable explanations that encode. In turn, conclusions about important inputs drawn from such high scoring explanations may not hold for the model being explained. Further, we experiment with a real sentiment analysis task where DET-X uncovers evidence that LLM-generated explanations encode.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General response

We thank the reviewers for their feedback. We are glad that the reviewers found the following strengths in our paper
- The paper is interesting (Fr3U), 
- Tackles a high priority problem in promising directions (PabQ),
- The definition of encoding is a novel and significant contribution (Fr3U, A1S9, PabQ)
- The experiments look sound and demonstrate practical utility to current AI applications with LLMs (Fr3U, A1S9).

Briefly, the paper studies the evaluation of feature attribution methods. Feature attribution evaluations typically check how well the label is predicted from the selected inputs returned by a feature attribution method. However, feature attribution methods can hide information about the label in the identity of the selection beyond what is available in the values of the selected variables. For example, an explanation of predicting pneumonia from a chest X-ray the can output the top right pixel when pneumonia is present but the bottom left pixel when there is no pneumonia. Such explanations are called ``encoding''. 
Encoding is a recognized problem that limits the utility of both explanations and their evaluations [3,4].

In the literature only specific constructions of encoding explanations exist, without a formal definition. Without such a formal definition, an evaluation method's ability to detect encoding cannot be tested beyond the recognized few constructions. To addres this gap, this paper makes the following contributions:

- Develop the first mathematical definition of encoding.
- Show that existing ad-hoc encoding constructions fall under the introduced definition.
- Formalize different notions of an evaluation’s sensitivity to encoding in terms of weak and strong detection.
- Show that existing evaluations ROAR [1] and FRESH [2] do not weakly detect encoding.
- Prove that EVAL-X weakly detects encoding, but does not strongly.
- Introduce DET-X and prove it strongly detects encoding.
- Use DET-X to uncover evidence of encoding in LLM-generated explanations for predicting the sentiment from movie reviews.


**[Rebuttal overview]**
In response to the reviewer feedback,

1. We have made the paper less dense by moving some details and technical parts (like lemma 1) to the appendix (Fr3U).
2. Evaluated EVAL-X and DET-X on Tabular data (A1S9).
3. Compared ROAR, EVAL-X, DET-X on the image experiment (PabQ).

Two reviewers also asked about limitations. We discuss limitations in the Discussion section 6, paragraph "Misestimated models, explanation search, and encoding for free-text rationales". Specifically, we point out that EVAL-X or DET-X scores may not retain their weak and strong detection properties when the scores are computed with misestimated models. We gave a formal example in Appendix D.4 but did not link it in section 6. Such problems from misestimation are not unique to EVAL-X or DET-X; they can occur in any evaluation method that build models to compute their score.


We responded to individual comments in separate responses.


[1] https://arxiv.org/abs/1806.10758

[2] https://arxiv.org/abs/2005.00115

[3] https://proceedings.mlr.press/v130/jethani21a.html

[4] https://arxiv.org/abs/2308.14272

Meta-review: This paper points out the encoding problem with the explanation methods, i.e., the explanation may be determined by information beyond the input values. This is a quite interesting idea and welcomed by all reviewers, but the presentation is a bit dense and needs to be polished.
Decision: Accept (poster)