Title: Explanations that reveal all through the deﬁnition of encoding
Venue: NeurIPS 2024
Accepted: yes
Keywords: feature attributions

model explanations

evaluating explanations

encoding the prediction

interpretability
Abstract: Feature attributions attempt to highlight what inputs drive predictive power. Good attributions or explanations are thus those that produce inputs that retain this predictive power; accordingly, evaluations of explanations score their quality of prediction. However, evaluations produce scores better than what appears possible from the values in the explanation for a class of explanations, called encoding explanations. Probing for encoding remains a challenge because there is no general characterization of what gives the extra predictive power. We develop a deﬁnition of encoding that identiﬁes this extra predictive power via conditional dependence and show that the deﬁnition ﬁts existing examples of encoding. This deﬁnition implies, in contrast to encoding explanations, that non-encoding explanations contain all the informative inputs used to produce the explanation, giving them a “what you see is what you get” property, which makes them transparent and simple to use. Next, we prove that existing scores (ROAR, FRESH, EVAL-X) do not rank non-encoding explanations above encoding ones, and develop STRIPE-X which ranks them correctly. After empirically demonstrating the theoretical insights, we use STRIPE-X to show that despite prompting an LLM to produce non-encoding explanations for a sentiment analysis task, the LLM-generated explanations encode.

Explanations that reveal all through the definition of encoding

Abstract

Feature attributions attempt to highlight what inputs drive predictive power. Good attributions or explanations are thus those that produce inputs that retain this predictive power; accordingly, evaluations of explanations score their quality of prediction. However, evaluations produce scores better than what appears possible from the values in the explanation for a class of explanations, called encoding explanations. Probing for encoding remains a challenge because there is no general characterization of what gives the extra predictive power. We develop a definition of encoding that identifies this extra predictive power via conditional dependence and show that the definition fits existing examples of encoding. This definition implies, in contrast to encoding explanations, that non-encoding explanations contain all the informative inputs used to produce the explanation, giving them a "what you see is what you get" property, which makes them transparent and simple to use. Next, we prove that existing scores (ROAR, FRESH, EVAL-X) do not rank nonencoding explanations above encoding ones, and develop STRIPE-X which ranks them correctly. After empirically demonstrating the theoretical insights, we use STRIPE-X to show that despite prompting an LLM to produce non-encoding explanations for a sentiment analysis task, the LLM-generated explanations encode.

1 Introduction

Artificial intelligence can unlock information in data that was previously unknown. In medicine, for example, using AI, researchers have shown that electrocardiograms are predictive of structural heart conditions [1] or new-onset diabetes [2]. Good predictions often lead one to ask what in the input is important for a prediction; this question is a driving factor behind research in interpretability and explainability [3, 4]. One primary direction in interpretability seeks to produce explanations that are subsets of the input that retain the predictability of the label. These types of explanations and interpretations are called feature attributions and have been used to find factors associated with debt defaults [5], to demonstrate that detecting COVID-19 from chest radiographs can rely on nonphysiological signals [6], and to discover a new class of antibiotics [7].

Several methods exist for producing feature attributions or explanations. While some methods compute functions of model gradients [8] or look at predictability after removing features [9], other methods attribute scores to different inputs by treating them as players in a game [4, 10] or amortize their explanations by learning a single model to select subsets for each instance [11]. Choosing one from the many feature attribution methods requires an evaluation. There are, however, many approaches to evaluation itself: qualitative ones [12, 13, 14], which are limited to cases where humans have precise knowledge about the inputs relevant to prediction, and quantitative ones [2, 4, 15, 16, 17, 18, 19, 20], which do not require human knowledge.

Intuitively, a good evaluation method for feature attributions should assign higher scores to explanations that select inputs that are more predictive of the label. However, evaluations that score explanations based on the predictability of the label from the explanation face one major challenge: encoding . Informally, an encoding explanation is one where the explanation predicts the label beyond what seems plausible from the values of the inputs themselves. The top left panel of Figure 1 shows an explanation that predicts the label of dog or cat depending on whether the

ProducingExplanations

EvaluatingExplanations

EncodingProblem

Inputs

Explanation

Selection

Explanation

Prediction

Label

Metric

Label

Explanation

method

model

pred.

00

90%

Label

Selection

Values

Acc.

Inputs

1

Definition:Labeldependsontheselectiongiventhevalues.

Non-encodingExplanations

Encoding Detection

Label Explanation

LabelSelection

Values

Non-detector

Weakdetector

Strongdetector

(e.g.,ROAR[19],FRESH[18])

(e.g., EVAL-X [20])

(Our STRIPE-X)

Opt.-.

Opt....

Opt....

Non-encoding

+

.,Inputs

Score

Score

Score

m.f

Label

Values

Encoding

0

Non-encoding

Non-encoding 2

Encoding 1

Encoding

2

Non-encoding

Non-encoding 2

Encoding 1

Encoding 2

0

Non-encoding 1

Non-encoding 2

Encoding 1

Encoding 2

Inputs

..f

explanation is a pixel on the right half or left half of the image respectively. Many explanation methods fit the description of encoding [20, 21]. Further, given that many evaluations only look at the quality of prediction, encoding can go undetected, rendering the evaluations ineffective at picking explanations. In contrast, non-encoding explanations predict the label well only when the values in the explanation do, making them easy to reason about.

In addressing encoding, this work makes the following contributions:

Develops a simple statistical definition of encoding via a conditional dependence property.

Confirms the introduced definition captures all existing ad hoc encoding instances.

Shows that non-encoding explanations are easy to use because they retain all the predictive inputs used to build them, meaning that predictive non-encoding explanations reveal inputs that predict the label to their users, and thus have a "what you see is what you get" property .

Formalizes evaluations' sensitivity to encoding as weak detection (optimal scoring explanations are non-encoding) and strong detection (non-encoding explanations score above encoding ones).

Demonstrates that the evaluations ROAR [19] and FRESH [18] do not weakly detect encoding.

Proves that EVAL-X [20] weakly detects encoding, but does not strongly detect encoding.

Develops STRIPE-X and proves that it strongly detects encoding .

Uses STRIPE-X to show that despite prompting an LLM to produce non-encoding explanations for a sentiment analysis task, the LLM-generated explanations encode.

Figure 1 provides an overview of this paper.

2 Evaluating explanations

We focus on explanation methods where the goal is to produce subsets of the input that predict the label [22, 23]. Explanation methods of this form, also called feature attributions, saliency methods [4, 8, 24], or just "explanations," include thresholded rankings from Shapley values [25, 26], LIME [24], and REAL-X [20]. With y as the label and x ∈ R d as the inputs, let q ( y , x ) be the joint distribution over them. An explanation method e maps the inputs x to a binary selection mask e ( x ) over the inputs: e : R d →{ 0 , 1 } d . The explanation x e ( x ) is a pair: the selection e ( x ) and the vector of explanation's values . For example, if x = [ a, b, c ] is three-dimensional and e ( x ) = [0 , 1 , 1] , x e ( x ) consists of the binary mask e ( x ) and the values associated with the inputs that correspond to the indices in e ( x ) with value 1 :

We keep track of the indices because the same value can lead to different predictions depending on the index it appears at; for example, in predicting mortality from patient vital signs, a heart rate above 110 can occur in healthy patients but a temperature of 110 ◦ F is almost always fatal. Equivalently, like in existing work [15, 16, 17, 20], one can choose x e ( x ) to retain the values in the explanation in the same position and mask out those not selected: x e ( x ) = e ( x ) × x +(1 -e ( x )) × mask-token . For concision, we overload the word "explanation" to mean the explanation method instead of the random variable x e ( x ) when it is clear from context.

Choosing between explanation methods requires evaluation. Explanation methods seek to return inputs that predict the label, so existing evaluations consider how well the explanation x e ( x ) predicts the label y [15, 16, 17, 20]. To score explanations based on predictive power, an evaluation method α ( · ) takes as arguments both the explanation e ( x ) and the joint distribution q ( y , x ) : α ( q, e ) . Without loss of generality let higher be better.

2.1 Encoding: A disconnect between the predictiveness of explanations and the predictiveness of their values

We give a simple example of encoding to build intuition for the disconnect between predicting the label from the explanation and predicting the label from the explanation's values. Imagine that the goal is to explain which set of vital signs signal bacterial pneumonia as the diagnosis compared to the common cold. Consider the explanation method that selects the patient's height when the true probability of pneumonia is high given the whole set of observables (including labs, symptoms, and vital signs) and otherwise selects the patient's hair color. Physiologically, height and hair color do not indicate that the patient has pneumonia, meaning that this explanation should not be highly predictive of the label. However, by construction, pneumonia is likely exactly when the explanation selects height, and predicting the label from the explanation achieves the same accuracy as predicting with the full conditional arg max y ∈{ pneumonia, cold } q ( y = y | x ) . Thus, despite the explanation method only selecting physiologically irrelevant inputs, the explanation predicts the label well.

Encoding examples such as the one above are neither contrived nor unique. For example, Jethani et al. [20] show that certain procedures that learn to explain, when applied to MNIST digit classification, yield explanations that select a background, black pixel that predicts the label at an accuracy > 90% ; (see Figure 1 in [20]). Other examples of encoding explanations that predict better than what is expected from the explanation's values exist [20, 21]. Encoding explanations should not score optimally under a good evaluation because the explanation selects inputs that do not appear to predict the label. However, without a general characterization of the discrepancy in predictive power for encoding, finding explanations whose values predict well remains a challenge. The next section develops a definition of encoding.

3 Formalizing encoding

Intuitively, encoding is a phenomenon where the information about the label in the explanation x e ( x ) exceeds what is known from the explanation's values . As the input x determines the explanation x e ( x ) , the quality of predicting the label y from the explanation relies on the information about the label transmitted from x to x e ( x ) . There are two pathways for this transmission; we elaborate below.

Denoting the values in a subset v by x v , compare the event this subset takes the values a , i.e. x v = a to the event that the explanation's selection is v and that the explanation's values are a , i.e., x e ( x ) = ( v , a ) .

Knowing that the explanation is x e ( x ) = ( v , a ) implies not only that the values in the explanation are determined as x v = a , but also that the selection is determined as e ( x ) = v .

In reverse, knowing that the values of a subset of inputs are x v = a and knowing the selection e ( x ) = v implies that the explanation are x e ( x ) = ( v , a ) .

Putting these two points together yields an equality between events:

Thus, the two pathways for information between x and the explanation x e ( x ) are the selection e ( x ) and explanation's values x v ; see Figure 2. Existing work makes similar intuitive observations but stops short of formalizing the additional predictive power in an explanation x e ( x ) [20, 21].

̸

To formalize this extra predictive power, define the explanation indicator E v = ✶ [ e ( x ) = v ] . A little algebra in Appendix A.1 shows the explanation indicator E v provides the extra information:

Building on this insight, we define encoding as a conditional dependence : Definition 1 ( Encoding ) . The explanation e ( x ) is encoding if there exists an S where q ( x e ( x ) ∈ S ) > 0 such that for every ( v , a ) ∈ S :

An example mathematical construction of an encoding explanation is provided in Appendix B.1. The dependence in Def: Encoding means that for encoding explanations, there is a disconnect between how well the explanation x e ( x ) predicts the label versus only the explanation's values x v . This disconnect means that evaluations that score explanations based on predictions from the explanation or their transformations [15, 16, 17, 20] can favor explanation methods that select inputs whose values have little relevance to predicting the label.

Beyond the disconnect in prediction, encoding explanations are undesir-

y

x

x

e

(

x

)

e

(

x

)

x

v

able as they conceal predictive inputs that nevertheless affect the explanation. This concealment can lead to incorrect conclusions, such as that inputs outside the selection are irrelevant, or bewilderment because predictive inputs outside the explanation drive changes in the selection in ways that cannot be understood from the explanation itself. An example is provided in Appendix B.2.

Non-encoding explanations. Conversely, for a non-encoding explanation, there exists no positive measure set of explanations x e ( x ) , where the explanation indicator has conditional dependence given the explanation's values. That is, for a set A where q ( x e ( x ) ∈ A ) = 1 , then for all ( v , a ) ∈ A

which in turn guarantees

Appendix A.2 shows this. A simple example of a non-encoding explanation is a constant explanation that always picks the same subset of inputs, since a constant E v is independent of any variable. The information for predicting the label in a non-encoding explanation lives in the explanation's values. Evaluations based on predictions from the explanation x e ( x ) of non-encoding explanations will yield explanations where the input values x v predict the label. In other words, non-encoding explanations reveal all the informative inputs they depend on, and "what you see is what you get" in the explanation.

3.1 Encoding explanations in the wild

Def: Encoding encompasses examples in the existing literature beyond the example in Section 2.1. In that example, the information about y lies in the positions in the selection e ( x ) , which motivates the name position-based encoding (POSI). This section describes two other informal examples from the literature of encoding explanations, prediction-based encoding (PRED) [21] and marginal encoding (MARG) [20], and explains the intuition behind why they encode. In the appendix, we develop formalizations of these types of encoding and show that these formulations meet Def: Encoding.

Prediction-based encoding (PRED). To understand how prediction-based encoding occurs, consider the task of sentiment analysis from movie reviews. Assume that reviews can either be of type "My day was terrible, but the movie was [ADJ1]." and "The movie was [ADJ2], but the day was not great." where adjective ADJ1 can be "good" or "not great" and adjective ADJ2 can be "not great" or "terrible". Due to common English parlance, "terrible" indicates bad sentiment more often than "not great". Then, in the example setup above, only seeing that the fourth word is "terrible" yields bad sentiment with higher probability than when only seeing that the phrase is "not great". However, the fourth word does not always describe the movie. An explanation can look at "not great" describing the movie as bad but then selects "terrible" to encode the bad sentiment. This explanation encodes because the selected word may not describe the movie but the selection predicts the sentiment.

Marginal encoding (MARG). This type of encoding occurs when some inputs determine which other inputs determine the label. For example, in Figure 3, the color determines whether the top right patch produces the label or the bottom right patch. Inputs that control where the label comes from are named control flow inputs . For a real-world example, consider the following example from Jethani et al. [20], where the goal is to predict mortality for patients with chest pain. A lab value that checks for heart injury and acts like a control flow input is troponin. Abnormal troponin indicates that cardiac issues exist and cardiac imaging would inform mortality. Normal troponin on the other hand can indicate that chest pain is unrelated to

cat

dog

x

e

(

x

)

e

(

x

)

x

e

(

x

)

e

(

x

)

cardiac health and a chest X-ray would instead inform mortality. Selecting one image or the other, but not the control flow input, conceals information about why the image was relevant to the label.

Formalization. In Appendix B, we provide mathematical formulations of each informal example and show that they fall under the definition of encoding in Def: Encoding: position-based encoding (Appendix B.3), prediction-based encoding (Appendix B.4), and marginal encoding (Appendix B.5). The key intuition behind all of these is that the explanation e ( x ) varies with inputs other than the selected ones, and these additional inputs provide information about the label beyond the selected ones. Next, we turn to detecting encoding via quantitative evaluations.

4 Detecting encoding in explanations

This section develops notions of sensitivity to encoding for evaluation methods, and uses the mathematical definition of encoding developed in the previous section to establish which methods detect encoding and which do not. Hsia et al. [21] suggest that evaluation methods like EVAL-X can be gamed to produce high scores for encoding explanations by optimizing the evaluation. To study this case, we introduce the notion of weak detection. If the optimal score of an evaluation of explanations does not permit encoding, then that evaluation is said to weakly detect encoding:

Definition 2 ( Weak detection of encoding ) . An evaluation α ( q, e ) of explanations weakly detects encoding if the optimal explanations e ∗ , i.e. α ( q, e ∗ ) = max e α ( q, e ) , are non-encoding.

Weak detection provides a recipe for finding non-encoding explanations: find the explanation that achieves the maximum score of a weak detector. However, such a recipe would only work when optimizing without constraints because weak detection does not require non-encoding explanations to have a better score than any encoding one. Requiring this leads to the definition of strong detection .

Definition 3 ( Strong detection of encoding ) . An evaluation α ( q, e ) strongly detects encoding if for any encoding explanation e and non-encoding explanation e ′ , α ( q, e ′ ) > α ( q, e ) .

Evaluations that are not weak detectors cannot be strong detectors because they score some encoding explanation optimally.

4.1 Do existing evaluation methods detect encoding?

Here, we consider whether several techniques for evaluating explanations: ROAR [19], FRESH [18], and EVAL-X [20] can detect encoding. We analyze these evaluations on the following distribution q

Consider the explanation e encode ( x ) = ξ 1 = [1 , 0 , 0] if x 3 = 1 and ξ 2 = [0 , 1 , 0] otherwise; this encodes because x 3 is used to create the explanation and x 3 predicts the label conditional on x 1 when E ξ 1 = 1 . This is a MARG explanation (see Section 3.1).

ROAR and FRESH do not weakly detect encoding. ROAR evaluates explanations by predicting the label from the inputs not selected by the explanation, denoted as x -e ( x ) ; ROAR scores explanations optimally if the predictions from the remaining covariates are as random as predicting without any covariates at all. In other words, ROAR checks how informative x -e ( x ) is of y and provides the highest score when y | = x -e ( x ) . In contrast, FRESH evaluates explanations by predicting the label from the explanation after removing all other inputs, denoted as val ( x e ( x ) ) . For example, assume we are given an input x = "Visually stunning. My favorite movie ever" and an explanation e ( x ) that selects the words "stunning" and "favorite". Then, the explanation is x e ( x ) = ([0 , 1 , 0 , 1 , 0 , 0] , [ "stunning" , "favorite" ]) , whereas val ( x e ( x ) ) = [ "stunning" , "favorite" , pad-token , pad-token , pad-token , pad-token ] , which drops the information about where the selected words are in the input. See Appendix B.6 for a formal definition of val ( x e ( x ) ) . FRESH checks how predictive q ( y | val ( x e ( x ) )) is and assigns an optimal score if the prediction is as good as that of q ( y | x ) . These conditions hold for e encode ( x ) in eq. (3):

Proposition 1. For the data generating process ( DGP ) in eq. (3) , ROAR and FRESH assign their respective optimal scores to the encoding explanation e encode ( x ) .

The proof is in Appendix B.6. The intuition is that the encoding explanation e encode ( x ) always selects the input that informs the label given the control flow x 3 ; removing the only conditionally informative input means that x -e encode ( x ) has no information about y . In turn, ROAR scores an encoding explanation x -e encode ( x ) optimally, meaning it does not even weakly detect encoding. In addition, val ( x e ( x ) ) provides the exact same information about the label regardless of which position it came from. As a result, x | = y | val ( x e ( x ) ) , so FRESH scores e encode ( x ) optimally. Even though FRESH attempts to drop the information about the selection v = e ( x ) during evaluation, val ( x e ( x ) ) remains a function of x e ( x ) = ( v , a ) , so extra information can still be transmitted through the selection v . Thus, ROAR and FRESH are not weak detectors of encoding.

EVAL-X weakly detects encoding but not strongly. EVAL-X [10, 26] is an evaluation method and is sometimes called the surrogate model score. The EVAL-X score with log-probabilities is

This score measures the expected log-likelihood of the labels given the input values chosen by the explanation method e and is grounded in the sampling distribution q . Log-likelihoods are maximized by matching the true distribution, this leads to EVAL-X's weak detection:

Theorem 1. If e ( x ) is EVAL-X optimal, then e ( x ) is not encoding.

Appendix A.4 gives a proof. The proof shows that at optimality, the prediction from the values of explanation has to match the prediction from the full inputs. In turn, given the values there is no additional information in x about y , which means the explanation indicator E v is independent of y ; this violates Def: Encoding, which proves the non-encoding nature of EVAL-X-optimal explanations.

To test strong detection for EVAL-X, we consider explanations constrained to select one input. Such reductive constraints appear in practice because the goal of producing an explanation is often to aid humans who benefit from reduced complexity. Such constraints prohibit explanations from reaching EVAL-X's optimal score. Compare e encode ( x ) with a non-encoding constant explanation:

Thus, EVAL-X is not a strong detector. The intuition is that the first two coordinates x 1 , x 2 predict the label when selected by e encode, while the control flow feature does not predict the label. EVAL-X not being a strong detector means that optimizing EVAL-X over a reductive set may yield an encoding explanation. In this case, e encode is one of the EVAL-X-optimal reductive explanations (Lemma 6).

4.2 STRIPE-X: a strong detector of encoding

Encoding explanations induce the dependence between the label y and the identity of the selection E v = ✶ [ e ( x ) = v ] given the values in the explanation x v (Def: Encoding). This dependence can be tested for by building on conditional independence tests [27, 28, 29]. Rather than testing, direct quantification of dependence can be useful for when combining with other scores, which can be done using instantaneous conditional mutual information:

ENCODE-METER is 0 only when Def: Encoding does not hold:

Proposition 3. ENCODE-METER ϕ q ( e ) = 0 if and only if e is not encoding.

The proof is in Appendix A.5. Combining EVAL-X with ENCODE-METER weighed by α yields a method we call the strongly information-penalized evaluator (STRIPE-X):

For a large enough α , the added penalty term pushes down the scores of encoding explanations below that of all non-encoding ones, meaning that STRIPE-X is a strong detector of encoding:

Theorem 2. With finite H ( y | x ) and H ( y ) , for any explanation that encodes e and any that does not encode e ′ , there exists an α ∗ such that ∀ α > α ∗ STRIPE-X α ( q, e ′ ) > STRIPE-X α ( q, e ) .

The proof is in Appendix A.5. The intuition behind the proof is that for a large enough α , the STRIPE-X scores for any encoding explanations will be dominated by the information term, and thus will become smaller than any non-encoding explanation whose score is lower bounded by the negative marginal entropy, -H q ( y ) . Table 1 summarizes the weak and strong detection properties of different evaluations.

Estimating STRIPE-X. The first component of STRIPE-X is EVAL-X. Computing EVAL-X (eq. (4)) requires an estimate of the predictive distribution of the label y given x v , q ( y | x v ) [20]. Estimation can be done in two ways. The first way makes use of a surrogate model trained to predict the label from different random subsets using masked tokens [9, 20]. The second way to compute EVAL-X (eq. (4)) relies on conditional generative models [30, 31]. Both hyperparameters and a combination of the estimators can be chosen to maximize the average log-likelihood on a held-out validation set across random input subsets.

To estimate the second part of STRIPE-X, the ENCODE-METER, first expand the mutual information terms in ENCODE-METER, ϕ q ( e ) , in terms of expected KL :

The outer expectation can be estimated using samples from the data and the inner expectation over y can be estimated using the EVAL-X model q ( y | x v ) . The distributions over E v can be estimated using a classifier of E v that randomly masks the label and masks different subsets of the inputs. Further details and a generative way to estimate STRIPE-X are in Appendix C.1 and Appendix C.3; full algorithms are given in Appendix D.

STRIPE-X in practice. Using STRIPE-X to choose between explanations is straightforward: pick the one with the larger score. However, like other evaluations that use learned models, misestimation can pose a problem. With large α , non-encoding explanations with misestimated ENCODE-METER will have bad STRIPE-X scores, while with small α some encoding explanations can have good scores. Across all experiments, we set α = 20 , which yielded STRIPE-X scores for known encoding explanations worse than known non-encoding explanations.

5 Experiments

This section consists of two parts. The first part demonstrates the weak and strong detection capabilities of the evaluations ROAR, EVAL-X, and STRIPE-X in a simulated setting and on an image recognition task. To demonstrate these capabilities, we run these evaluations on instantiations of POSI, PRED, and MARG. Additionally, we evaluate an existing method that learns to explain under a reductive constraint, called REAL-X [20]. The second part shows how STRIPE-X enables discovering encoding explanations in the wild, without specific knowledge of the DGP or the method that produced the explanation. We employ STRIPE-X to uncover encoding in explanations generated by a large language model (LLM) for predicting sentiments from movie reviews.

5.1 Empirically studying the detection of encoding in a simulated setting

We construct two examples with binary labels y : one discrete input x and one that is a hybrid of continuous and discrete components. Both use one binary input in x ∈ { 0 , 1 } 5 as a control flow

Method

Weak

Strong

ROAR [19]

✗

✗

FRESH [18]

✗

✗

EVAL-X [20]

✓

✗

STRIPE-X

✓

✓

variable and switch the inputs that y depends on. In both DGPs, y only depends on x 1 if x 3 = 1 , and only on x 2 if x 3 = 0 ; this means that x 4 , x 5 are purely noise. For both DGPs, y is sampled per the following distribution where x 3 determines the subset the y depends on q ( y = 1 | x ) = ✶ [ x 3 = 1] q ( y | x 1 , x 3 ) + ✶ [ x 3 = 0] q ( y | x 2 , x 3 ) . (8) Thus, EVAL-X ∗ is achieved by an explanation of size 2 : e ( x ) = ξ 1 + ξ 3 if x 3 = 1 else e ( x ) = ξ 2 + ξ 3 . See Appendix C.4 for details; the exact DGPs are given in eq. (36) and eq. (37).

Encoding explanations. Table 2 describes the encoding explanations we consider for this setting. In Appendix C.4, we check that Def: Encoding holds for these explanations in the discrete DGP by estimating the role of the unselected inputs in affecting the explanation and the role of E v in predicting y beyond x v ; a characterization of Def: Encoding to support this check is in Lemma 1.

ROAR and FRESH fails to weakly detect encoding. To empirically test the analysis about ROAR and FRESH, we study whether the two evaluations weakly detect encoding. In this study, we compare each evaluation's score on the all-inputs explanation, which is optimal, to the score assigned to MARG. MARG ignores x 3 which is required to produce the label y in eq. (8). ROAR log-likelihoods for MARG and the all-inputs explanation are approximately -H ( y ) = -0 . 69 for both DGPs. In addition, the value of the input that MARG selects alone contains all the information about the label regardless of whether MARG selects x 1 or x 2 . Thus, FRESH log-likelihoods for MARG and the all-inputs explanation are both approximately -0 . 29 for both DGPs. This result validates that ROAR and FRESH are not weak detectors because they do not separate the optimal explanation from all encoding explanations.

EVAL-X is a weak detector of encoding but not a strong detector. EVAL-X log-likelihood scores are given in blue in Figures 4a and 4b. EVAL-X, being a weak detector, scores the encoding constructions (POSI, PRED, and MARG) strictly lower than the log-likelihood of the optimal explanation EVAL-X ∗ . However, the EVAL-X score for the MARG explanation is -0 . 4 , which is above the score of -0 . 6 achieved by a non-encoding explanation e ( x ) = ξ 1 ; thus, EVAL-X is not a strong detector.

Strong detector STRIPE-X prices out all the encoding explanations. Figures 4a and 4b report STRIPE-X scores for the same set of explanations as above; STRIPE-X scores

(a)

Results: discrete DGP.

EVAL-X

STRIPE-X

0

-0.4

-0.6

-10

=-- EVAL-X

-0.8

PRED

P

MA

G

PRED

p

M

R

G

II

EVAL-X

STRIPE-X

0

-0.4

-10

-20

-0.6

==- EVAL-X’

-30

-0.8

PRED

POSI

PRED

ISO

M

p

M

Figure 4: EVAL-X and STRIPE-X scores of the 3 encoding constructions and the nonencoding constant explanation ( e ( x ) = ξ 1 ) , for both DGPs. EVAL-X, being only a weak detector, assigns suboptimal scores to all encoding explanations ( < ), but scores some encoding explanations above the constant explanation. On the other hand, STRIPE-X, being a strong detector, pushes down the scores of all the encoding explanations below that of the non-encoding constant explanation that always selects x 1 .

are shown in red. Strong detector STRIPE-X scores the non-encoding explanations above the negative entropy -H q ( y ) = -0 . 69 and scores every encoding construction under that threshold.

5.2 Detecting encoding on images of dogs and cats

The goal of this section is to study the encoding detection capabilities of ROAR, EVAL-X, and STRIPE-X on real data. We consider an image recognition task like the one in Figure 3 with labels and images from the cats_vs_dogs dataset from the Tensorflow package [32]. We break images of size 64 × 64 into 4 patches each of size 32 × 32 . In left-right then top-down order, let x 1 , x 2 , x 3 , x 4 be the upper left, upper right, bottom left, and bottom right patches respectively; x 1 , x 3 capture color, and x 2 , x 4 are the animal images. With annot(image) denoting the annota-

tion in the cats_vs_dogs dataset the image having a dog or a cat , the label is assigned as:

✶ ✶ We consider three encoding explanations (POSI, PRED, MARG) and two non-encoding ones: 1) optimal , which selects the color and the patch that produces the label as dictated by the color, and 2) denoted fixed , which always outputs the bottom right patch x 4 . Appendix C.6 gives details.

We report the scores assigned to each explanation by ROAR, EVAL-X, and STRIPE-X in Table 3. ROAR scores two encoding explanations PRED and MARG as high as the optimal explanation, meaning it is not even a weak detector. POSI, PRED, and MARG all score worse than the optimal explanation under both the weak detector EVAL-X and the strong detector STRIPE-X. However, EVALX scores one non-encoding explanation ( fixed ) worse than two encoding ones, meaning it is not a strong detector. Being a strong detector, STRIPE-X scores the fixed explanation above the negative marginal entropy -H q ( y ) = -0 . 69 and scores every encoding construction under that threshold.

Evaluating explanations produced by REAL-X [20]. We ran REAL-X to learn explanations for the simulated setting and the image recognition task. In the simulated setting, REAL-X is run to select one input; Appendix C.4 gives details. In the image recognition task, REAL-X is run to select one of the four patches as an explanation; Appendix C.6 gives details. In both the simulated setting (see Figures 4a and 4b) and the image recognition task (see Table 3), REAL-X fails to achieve the optimal EVAL-X score while achieving a STRIPE-X score below the threshold of negative marginal entropy -H q ( y ) = -0 . 69 . Upon investigation, we found that REAL-X produced an explanation that matched the MARG construction on at least 80% of the inputs in the simulated setting. On the image recognition task, REAL-X explanation matched the MARG explanation on the whole dataset. In both cases, STRIPE-X, being a strong detector, correctly alerts that the REAL-X explanation encodes.

5.3 Encoding in LLM-generated explanations

One can detect encoding in any explanation by checking if the STRIPE-X score falls below the negative marginal entropy. Recent work uses LLMs to produce explanations; e.g. [33] prompt an LLM to generate explanations for reasoning tasks which are later used to improve smaller models. If the LLM explanation encodes, the smaller model can falsely ignore the informative inputs the larger model's explanation depends on and yet does not reveal. In this section, we evaluate explanations generated by an LLM, Llama 3, for a sentiment analysis task. We consider reviews that take one of two forms: with ADJ1 and ADJ2 as adjectives, the review is

The second sentence in the review acts as a "control flow" input and determines whether ADJ1 or ADJ2 describes the sentiment about the movie. We prompt Llama 3 (see Appendix C.8) to predict the sentiment and select a few words from the review that were important for that sentiment; the selected parts form the generated explanation. To discourage encoding, the prompt explicitly instructs the LLM to select all the words that the LLM based the selection on; such an explanation, by Lemma 1, would be non-encoding. On the 5 most common selections e ( x ) generated by the LLM, we compute the EVAL-X score and the ENCODE-METER ϕ q ( e ) . The resulting STRIPE-X score is -2 . 78 , falling short of the negative entropy -H q ( y ) = -0 . 69 , meaning the LLM encodes. We investigated why.

As an example, consider the review 'My day was resplendent and the movie was hollow. that is it.' ; the LLM selects only hollow in the explanation. However, the LLM instead selects resplendent when that is it is switched to oh wait, reverse the adjectives. Such occurrences are common. On > 70% of the data, the LLM selects the word that describes the movie

but does not select the second sentence in the review which controls which adjective describes the movie; this is akin to MARG encoding. Thus, the LLM-generated explanation encodes by looking at the control flow input in the second sentence to find the correct adjectives, but failing to select the control flow input. Such an explanation falsely indicates that only the adjectives are relevant to predicting the label. In contrast, a non-encoding explanation would, in addition to the adjective that describes the movie, reveal control flow words that indicate which adjective predicts the label.

In summary, despite being instructed to include all the words that were looked at when producing the explanation, the LLM encodes. Building non-encoding explanations with LLMs may require an extensive search over prompts or finetuning guided by scores from STRIPE-X.

6 Discussion

When an explanation is encodes, predictions from the explanation become disconnected from predictions from the values in the explanations. Such explanations can select values with little relevance to the label and yet score highly on the many existing predictive evaluations. We develop a simple statistical definition of encoding. Inverting this definition shows that when non-encoding explanations predict the label, users know the values of those inputs selected in the explanation predict the label. We then show that existing evaluations are either non-detectors (ROAR[19],FRESH [18]) or only weak detectors (EVAL-X [20]). Motivated by this, we introduce a new strong detector, STRIPEX. After empirically demonstrating the detection capabilities (or lack thereof) of said evaluations, we use STRIPE-X to discover encoding in LLM-generated explanations.

More related work. Other investigations into evaluating explanations focused on label leakage [26, 34] and faithfulness [18, 35, 36, 37, 38]. Label leakage is similar to encoding in that additional information is in the explanation, but focuses on explanations that have access to both the inputs and the observed label; we leave extending Def: Encoding to leakage to the future. Faithfulness, intuitively, asks that the explanation reflect the process of how a label is predicted from the inputs; a formalization does not exist. Jacovi and Goldberg [38] note the need to define faithfulness formally. Encoding explanations are not faithful to the process of making an explanation because predictive inputs outside those selected by the explanation control the explanation.

Limitations and the future. Using misestimated models in evaluations (like EVAL-X) may lead to mistakes (see Appendix B.8 for an example). The retinal fundus experiment from Jethani et al. [26] is an example where misestimation leads to reductive explanations scoring higher than using the full input. Misestimation can be due to poor uncertainty or due to dependence on shortcut features. One fruitful direction is to use better uncertainty estimates, like conformal inference [39] or calibration [40], or employ robustness methods [41, 42] to ameliorate errors due to misestimation. Another direction is use tricks like REINFORCE-style gradients to construct non-encoding explanations by optimizing STRIPE-X. Explanations that output subsets may not always help humans interpret the mechanism of the prediction. For example, imagine one wants to understand why a model correctly answers the question "Who won the ski halfpipe at the X-games 3 years after her debut in 2021?" with "Eileen Gu". A subset explanation may return "3 years after her debut in 2021" and "ski halfpipe", but that does not help a human interpret how the model predicts. A better interpretation would be to make the model output, "3 years after 2021 is 2024. Eileen Gu won in 2024, and debuted in 2021." Such explanations can also encode information about the prediction in the text produced as a rationale [43]. An important direction here would be to extend the definitions of weak and strong detectors of encoding to evaluations of free-text rationales.

Data versus Model Explanations. Even with the formal definitions of explanation methods, there is a question about what is being explained: the data or the model. These two concepts often get blended together in the literature [11, 20]. We clarify this point and abstract the choice away as two different ways to produce the joint distribution q ( y , x ) . In data explanation , the distribution under which a feature attribution method seeks to output a subset of inputs that predict the label should be the population distribution of the data [23]. If, instead, the goal is model explanation , the goal should not be to highlight inputs that predict the label well in samples of the data; rather it should be to predict the label well in samples from the model. Formally, a model with parameters θ is a conditional distribution, p θ ( y | x ) . To target a model explanation, a feature attribution method would aim to output a subset of inputs that predict the label under the distribution F ( x ) p θ ( y | x ) .

References

Pierre Elias, Timothy J Poterucha, Vijay Rajaram, Luca Matos Moller, Victor Rodriguez, Shreyas Bhave, Rebecca T Hahn, Geoffrey Tison, Sean A Abreau, Joshua Barrios, et al. Deep learning electrocardiographic analysis for detection of left-sided valvular heart disease. Journal of the American College of Cardiology , 80(6):613-626, 2022.

Neil Jethani, Aahlad Puli, Hao Zhang, Leonid Garber, Lior Jankelson, Yindalon Aphinyanaphongs, and Rajesh Ranganath. New-onset diabetes assessment using artificial intelligence-enhanced electrocardiography. arXiv preprint arXiv:2205.02900 , 2022.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Visualising image classification models and saliency maps. Deep Inside Convolutional Networks , 2, 2014.

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems , 30, 2017.

Kim Long Tran, Hoang Anh Le, Thanh Hien Nguyen, and Duc Trung Nguyen. Explainable machine learning for financial distress prediction: evidence from vietnam. Data , 7(11):160, 2022.

Alex J DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence , 3(7):610-619, 2021.

Felix Wong, Erica J Zheng, Jacqueline A Valeri, Nina M Donghia, Melis N Anahtar, Satotaka Omori, Alicia Li, Andres Cubillos-Ruiz, Aarti Krishnan, Wengong Jin, et al. Discovery of a structural class of antibiotics with explainable deep learning. Nature , pages 1-9, 2023.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradientbased localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626, 2017.

Ian Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research , 22(209):1-90, 2021.

Neil Jethani, Mukund Sudarshan, Ian Connick Covert, Su-In Lee, and Rajesh Ranganath. Fastshap: Real-time shapley value estimation. In International Conference on Learning Representations , 2022.

Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Invase: Instance-wise variable selection using neural networks. In International Conference on Learning Representations , 2018.

Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Samuel J Gershman, and Finale Doshi-Velez. Human evaluation of models built for interpretability. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume 7, pages 59-67, 2019.

Adriel Saporta, Xiaotong Gui, Ashwin Agrawal, Anuj Pareek, Steven QH Truong, Chanh DT Nguyen, Van-Doan Ngo, Jayne Seekins, Francis G Blankenberg, Andrew Y Ng, et al. Benchmarking saliency methods for chest x-ray interpretation. Nature Machine Intelligence , 4(10): 867-878, 2022.

Jonathan Crabbé, Alicia Curth, Ioana Bica, and Mihaela van der Schaar. Benchmarking heterogeneous treatment effect models through the lens of interpretability. Advances in Neural Information Processing Systems , 35:12295-12309, 2022.

Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and KlausRobert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems , 28(11):2660-2673, 2016.

V Petsiuk, A Das, and K Saenko. Rise: Randomized input sampling for explanation of blackbox models. arXiv preprint arXiv:1806.07421 , 2018.

Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in neural information processing systems , 30, 2017.

Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and Byron C Wallace. Learning to faithfully rationalize by construction. arXiv preprint arXiv:2005.00115 , 2020.

Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems , 32, 2019.

Neil Jethani, Mukund Sudarshan, Yindalon Aphinyanaphongs, and Rajesh Ranganath. Have we learned to explain?: How interpretability methods can learn to encode predictions in their interpretations. In International Conference on Artificial Intelligence and Statistics , pages 1459-1467. PMLR, 2021.

Jennifer Hsia, Danish Pruthi, Aarti Singh, and Zachary C Lipton. Goodhart's law applies to nlp's explanation benchmarks. arXiv preprint arXiv:2308.14272 , 2023.

Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. Journal of machine learning research , 3(Mar):1157-1182, 2003.

Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An information-theoretic perspective on model interpretation. In International conference on machine learning , pages 883-892. PMLR, 2018.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135-1144, 2016.

Erik Štrumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems , 41:647-665, 2014.

Neil Jethani, Adriel Saporta, and Rajesh Ranganath. Don't be fooled: label leakage in explanation methods and the importance of their quantitative evaluation. In International Conference on Artificial Intelligence and Statistics , pages 8925-8953. PMLR, 2023.

Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. In Proceedings of the TwentySeventh Conference on Uncertainty in Artificial Intelligence , pages 804-813, 2011.

Mukund Sudarshan, Wesley Tansey, and Rajesh Ranganath. Deep direct likelihood knockoffs. Advances in neural information processing systems , 33:5036-5046, 2020.

Mukund Sudarshan, Aahlad Puli, Wesley Tansey, and Rajesh Ranganath. Diet: Conditional independence testing with marginal dependence measures of residual information. In International Conference on Artificial Intelligence and Statistics , pages 10343-10367. PMLR, 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. URL https://arxiv. org/abs/2204.06125 , 7, 2022.

Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL http://tensorflow. org/ . Software available from tensorflow.org.

Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, et al. Explanations from large language models make small reasoners better. arXiv preprint arXiv:2210.06726 , 2022.

Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language? arXiv preprint arXiv:2010.04119 , 2020.

Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Filippova. " will you find these shortcuts?" a protocol for evaluating the faithfulness of input salience methods for text classification. arXiv preprint arXiv:2111.07367 , 2021.

Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. Do feature attribution methods correctly attribute features? In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 9623-9633, 2022.

Yiming Ju, Yuanzhe Zhang, Zhao Yang, Zhongtao Jiang, Kang Liu, and Jun Zhao. Logic traps in evaluating attribution scores. arXiv preprint arXiv:2109.05463 , 2021.

Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? arXiv preprint arXiv:2004.03685 , 2020.

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association , 113(523):1094-1111, 2018.

Mark Goldstein, Xintian Han, Aahlad Puli, Adler Perotte, and Rajesh Ranganath. X-cal: Explicit calibration for survival analysis. Advances in neural information processing systems , 33: 18296-18307, 2020.

Aahlad Manas Puli, Lily H Zhang, Eric Karl Oermann, and Rajesh Ranganath. Out-ofdistribution generalization in the presence of nuisance-induced spurious correlations. ICLR 2022 , 2021.

Aahlad Manas Puli, Lily Zhang, Yoav Wald, and Rajesh Ranganath. Don't blame dataset shift! shortcut learning due to gradients and cross entropy. Advances in Neural Information Processing Systems , 36, 2023.

Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. Faithfulness tests for natural language explanations. arXiv preprint arXiv:2305.18029 , 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We only use public data. We use standard existing training techniques, describe the hyperparameters in detail, and provided the Llama 3 prompts we used in our experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The error bars in 5.1 and 5.2 are negligible because we estimate means of metrics over datasets of 5000 samples. For 5.3, error bars are irrelevant because there is no comparison against an existing method.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We do not use human subjects and only use public data and public models. Our work focuses on explainable machine learning and does not pose additional ethical harms beyond what is standard for the field. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We investigate a problem with explanations and fix it. There is no societal impact of our work that exceed that of the usage of explanations.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release any data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

A Theoretical Details

A.1 Expressing q ( y | x e ( x ) ) in terms of the values and the identity of the explanation

To express q ( y | x e ( x ) ) , we use the following equivalence of events from eq. (1)

Then, intuitively, conditioning on the event that x e ( x ) = ( v , a ) gives you the same information as conditioning on the events e ( x ) = v and x v = a simultaneously. We make this formal below.

For discrete x , for any v , a such that the probability q ( x e ( x ) = ( v , a )) > 0 , define the LHS and RHS of eq. (1) as B v , C v respectively. Then, the conditionals q ( y | x e ( x ) = ( v , a )) and q ( y | E v = 1 , x v = a ) exist and can be written as follows:

These two conditionals are equal because B v = C v .

The same kind of result holds for general random vectors (discrete or continuous) x but is a little more involved because B v may be non-empty while q ( x ∈ B v ) = 0 and the equality of conditional densities/probabilities need to be written via measure theory. Assume the regular conditional probabilities q ( y | x e ( x ) ) and q ( y , E v | x v ) , q ( y | E v , x v ) and q ( E v | x v ) are defined almost everywhere in their respective probability measures. Take any S v ⊆ { x v : e ( x ) = v } where q ( x v ∈ S v ) > 0 and q ( e ( x ) = v ) > 0 . Consider any measurable sets Y over y and B v ( S v ) := { ( v , a ) : a ∈ S v } over x e ( x ) . Now, by definition of regular conditional probability measures, joint probabilities are obtained by taking the expectation of the conditional with respect to marginal distributions over the conditioning set:

and

Due to eq. (1), the LHS terms of the two equations above are equal and so are the probability measures over the integrating variables in eqs. (9) and (10). Letting x v be defined on a Borel sigma algebra, these two integrals eqs. (9) and (10) are equal if and only if for any Borel set S v , for almost every a ∈ S v

That is, in more plain terms, the conditional distributions are equal q ( y | x e ( x ) = ( v , a )) = q ( y | E v = 1 , x v = a ) .

A.2 With non-encoding explanations "what you see is what you get"

Def: Encoding says that an explanation e ( x ) is encoding if there exists an S where q ( x e ( x ) ∈ S ) > 0 such that for every ( v , a ) ∈ S , :

For a non-encoding explanation, Def: Encoding does not hold. Here, we derive the implications of violating Def: Encoding. Define the set A to contain all ( v , a ) where eq. (11) is violated:

By definition, the complement A C is such that

Such a set cannot have positive measure when Def: Encoding is violated which means

Thus, A is such that q ( x e ( x ) ∈ A ) = 1 , and by eq. (12) for all ( v , a ) ∈ A

In turn, which in turn guarantees

A.3 Helpful Lemmas and their proofs

A.3.1 Alternate conditions equivalent to Def: Encoding

The dependence in Def: Encoding occurs due to two reasons, understanding which sheds more light on the definition. First, for some selection e ( x ) = v , the explanation's values x v do not provide enough information to reveal that the explanation should select the inputs denoted by v . In other words, the indicator of the selection is variable even after fixing the explanation's values themselves. Second, this indicator is predictive of the label for the data with the explanation v . These two properties provide intuition on the definition of encoding:

Lemma 1. Def: Encoding holds for an explanation e ( x ) if and only if there exists a selection v such that q ( e ( x ) = v ) > 0 and a set S v ⊆ { x v : e ( x ) = v } such that q ( x v ∈ S v ) > 0 where both of the following conditions hold for almost every a ∈ S v :

Unpredictability of Explanation Additional Information from Explanation

̸

Proof. First, Lemma 2 shows the Def: Encoding holds if only if there exists a selection v such that q ( e ( x ) = v ) > 0 and a set S v ⊆ { x v : e ( x ) = v } such that q ( x v ∈ S v ) > 0 where

We use this alternate definition in what follows.

Given a non-measure zero set S v , by Lemma 3, almost everywhere in S v it holds that q ( E v = 1 | x v ) > 0 .

Conditional dependence implies Unpredictability and Additional information (the only if part). If q ( E v = 1 | x v ) = 1 almost everywhere (under q ( x ) ), then E v is constant given x v , and therefore independent of any variable given x v :

Then, it follows that conditional dependence implies the unpredictability property

Second, with the result from Lemma 3, we have q ( E v = 1 | x v ) ∈ (0 , 1) . Thus q ( y | x v , E v = 1) and q ( y | x v , E v = 0) exist almost every where in S v . Then, by definition of conditional dependence, there is additional information about the label in the explanation:

̸

This shows that Def: Encoding implies the additional information property.

̸

Conditional dependence implied by Unpredictability and Additional information (the if part). Now, if q ( E v = 1 | x v ) ∈ (0 , 1) , then the following two conditional distributions exist almost everywhere in S v

Then, by definition of dependence almost everywhere S v :

̸

Thus, the unpredictability and the additional information properties imply Def: Encoding.

Lemma 2. Def: Encoding holds for an explanation e ( x ) if and only if there exists a selection v such that q ( e ( x ) = v ) > 0 and a set S v ⊆ { x v : e ( x ) = v } such that q ( x v ∈ S v ) > 0 where

Proof. Def: Encoding says that the explanation e ( x ) is encoding if there exists an S where q ( x e ( x ) ∈ S ) > 0 such that for every ( v , a ) ∈ S eq. (13) holds. This proof works by showing that S having a positive measure implies the existence of v and S v as in Lemma 2 such that eq. (13) holds.

Decompose q ( x e ( x ) ∈ S ) by introducing an expectation over v ∼ q ( e ( x )) ,

As there are only finitely many v ,

The "only if" direction. Pick any v such that the RHS above holds and define S v = { a : ( v , a ) ∈ S } . By definition,

This proves that S v has positive measure:

Finally, as a ∈ S v = ⇒ ( v , a ) ∈ S , eq. (13) holds:

This completes the "only if" direction.

The "if" direction. Assume that there exists v such that q ( e ( x ) = v ) > 0 and S v ⊆ { x v : e ( x ) = v } such that q ( x v ∈ S v ) > 0 where

Define S = { ( v , a ) : a ∈ S v } . By this construction, S has positive measure:

where the last inequality holds because by assumption

Finally, as ( v , a ) ∈ S = ⇒ a ∈ S v , eq. (13) holds:

This completes the "if" directions and with that the proof.

Lemma 3. For any set S v ⊆ { x v : e ( x ) = v } such that q ( x v ∈ S v ) > 0 , then for almost every a ∈ S v , q ( E v = 1 | x v = a ) > 0 .

Proof. Define the set A v = { a : q ( E v = 1 | x v = a ) = 0 } . Next compute the joint probability

Now, noting that S v is a subset of { x v : e ( x ) = v } , which is equivalent to { x v : E v = 1 } , thus

The probability q ( x v ∈ S v | x v ∈ A v ) is non-negative, so it must be zero. Plugging this conditional back into the joint gives q ( x v ∈ A v ∩ S v ) = 0 . Then expanding yields

Since q ( x v ∈ S v ) > 0 , q ( x v ∈ A v | x v ∈ S v ) must be zero and thus, q ( x v / ∈ A v | x v ∈ S v ) = 1 , where expanding out the definition of A v gives the desired result that q ( E v = 1 | x v = a ) > 0 for almost a ∈ S v :

A.3.2 Optimal value and the optimal gap under EVAL-X

Lemma 4. The EVAL-X optimality gap value for e ( · ) is an averaged KL between q ( y | x ) and q ( y | x v ) : ∑ v ∈V q ( e ( x ) = v ) E q ( x | e ( x )= v ) KL ( q ( y | x ) ∥ q ( y | x v )) . This gap is zero, i.e. e ( x ) is optimal with the score EVAL-X ∗ = E q [log q ( y | x )] if for all v such that q ( e ( x ) = v ) > 0 ,

Proof. Let p be a generic conditional distribution and let x -v be the values outside the explanation.

This upper bound is achievable by an explanation that selects all inputs, so the maximum EVAL-X denoted as EVAL-X ∗ = E q log q ( y | x ) .

As in the math above, the EVAL-X score for an explanation method can be expanded as

where in the last step, we dropped a because it equals x v almost surely. Similarly, the optimal score EVAL-X ∗ expands to

Let V be the set of values that explanations can take on, then taking the difference from optimality

As each KL term is non-negative, each term in the sum being set to 0 simultaneously achieves the optimum, which happens when for all v such that q ( e ( x ) = v ) > 0 ,

In Appendix A.4, we use the results from Lemma 1 and Lemma 4 to prove that the optimal score of EVAL-X can only be achieved by non-encoding explanations.

A.4 Proof of Theorem 1

Theorem 1. If e ( x ) is EVAL-X optimal, then e ( x ) is not encoding.

Proof. Note only q ( e ( x ) = v ) > 0 are of interest, since q ( e ( x ) = v ) = 0 implies that E v = 0 almost surely and thus y | = E v | x v .

Then if e ( x ) achieves EVAL-X ∗ , then by Lemma 4, for all v such that q ( e ( x ) = v ) > 0 ,

First, this optimality criteria can incorporate E v = 1 on the lefthand side by first conditioning on e ( x ) and then noting that the equality holds for x where e ( x ) = v .

To understand if the optimality criterion disallows encoding, integrate the left and right-hand sides of this optimality criterion with the respect to complement of the inputs in x v , q ( x c v | x v , E v = 1) yields

Now expanding the right-hand side gives

Combing the two equations gives

We show that this equality implies that y | = E v | x v by splitting the analysis into cases based on q ( E v = 1 | x v ) = 1 and q ( E v = 1 | x v ) < 1 . In turn, the condition in Def: Encoding is violated and the explanation e ( · ) is not encoding.

Case 1: EVAL-X optimality holds when the explanation is predictable. The first case to consider is when the event that the explanation takes the value v is determined by x v for all samples with the explanation v . That is, q ( E v = 1 | x v ) = 1 :

Then expanding this marginal into the joint shows that the joint q ( y , E v = 0 | x v ) has to be zero as well.

because an integral of non-negative terms being zero implies that each term itself is zero almost surely.

Then, we can show that the determinism condition q ( E v = 1 | x v ) = 1 is sufficient for the optimality criterion eq. (14):

This shows that the EVAL-X optimality criteria is satisfied when the q ( E v = 1 | x v ) = 1 , thus the explanation is completely predictable from the explanation for examples with that explanation. By Lemma 1, we have

which violates Def: Encoding. So there is no encoding in this case.

Case 2: When the explanation is unpredictable, EVAL-X optimality requires that the explanation provide no extra information. Now consider the alternative case, q ( E v = 1 | x v ) < 1 . Here the explanation does not determine the explanation opening the possibility that the EVAL-X-optimal explanation method can encode information in the explanation.

Because q ( e ( x ) = v ) > 0 , we have q ( x v ∈ { x v : e ( x ) = v } ) > 0 . Thus, by Lemma 3, for almost every { x v : e ( x ) = v } it holds that q ( E v = 1 | x v ) > 0 . Putting this result together with alternative case ( q ( E v = 1 | x v ) < 1 ) gives: 0 < q ( E v = 1 | x v ) < 1 for almost every { x v : e ( x ) = v } .

Now, expanding out the optimality criterion eq. (14):

This equality says for all samples with the explanation v , knowing E v does not change the distribution of the label y . By Lemma 1, this equality implies that the independence y | = E v | x v holds which violates Def: Encoding.

A.5 Proof of Proposition 3 and Theorem 2

Proposition 3. ENCODE-METER ϕ q ( e ) = 0 if and only if e is not encoding.

Proof. First, Def: Encoding is violated if and only if there exists a set A such that q ( x e ( x ) ∈ A ) = 1 and

For the if direction, note that if ENCODE-METER ϕ q ( e ) = 0 ,

To show the forward direction, the above equality means that if ϕ q ( e ) = 0 , almost surely for every ( v , a ) ∼ q ( x e ( x ) ) , the instantaneous mutual information is 0 which implies the desired conditional independence

By definition of almost surely, there exists a set A such that q ( x e ( x ) ∈ A ) = 1 the independence above holds; this completes the "if" direction.

To show the reverse direction, let there exist a set A such that q ( x e ( x ) ∈ A ) = 1 , for every ( v , a ) ∈ A ,

Then, the fact that q ( x e ( x ) ∈ A ) = 1 implies that expectations with respect to q ( x e ( x ) ) over the whole support equal expectations over q ( x e ( x ) | x e ( x ) ∈ A ) , which is q ( x e ( x ) ) restricted to A :

This completes the "only if" direction.

Theorem 2. With finite H ( y | x ) and H ( y ) , for any explanation that encodes e and any that does not encode e ′ , there exists an α ∗ such that ∀ α > α ∗ STRIPE-X α ( q, e ′ ) > STRIPE-X α ( q, e ) .

Proof. Recall that STRIPE-X is

where the ENCODE-METER

Wefirst show bounds for the first term in STRIPE-X and then derive the STRIPE-X scores for encoding and non-encoding explanations.

Bounds on EVAL-X scores. We lower bound the EVAL-X score, which is the first term in STRIPEX, for non-encoding explanations.

For non-encoding explanations, almost surely over v , a ∼ q ( x e ( x ) )

In turn, for all ( v , a ) ∈ A ,

Then,

The above inequality implies that

Every inequality in the derivation above becomes strict when the explanation selects inputs that are predictive of the label because

Thus, non-encoding explanations have EVAL-X scores that are at least -H q ( y ) .

The optimal EVAL-X score for any explanation (see Lemma 4) equals the negative conditional entropy which is upper bounded by some finite number:

Comparing explanations via STRIPE-X. For any encoding explanation, by Proposition 3, for some c > 0

Now, consider α ∗ = I ( y ; x ) c ≥ 0 , which is finite because each term in the ratio is finite. Then, for all α > α ∗

As EVAL-X scores are below C = -H ( y | x ) for any encoding explanation,

Finally, for any non-encoding explanation, ϕ q ( e ′ ) = 0 by Proposition 3, STRIPE-X scores equal EVAL-X scores, which are lower bounded at -H q ( y ) .

Together, for every non-encoding explanation e ′ ( x ) and encoding explanation e ( x ) , it holds that

This proves that STRIPE-X is a strong detector of encoding.

B Encoding examples, non-detection of ROAR,FRESH, and non-strong detection of EVAL-X

B.1 An illustrative DGP for Def: Encoding

With B (0 . 5) being a Bernoulli distribution, consider the following example

For this problem, if the third coordinate x 3 = 0 , all the information between the label and the covariates is in the first coordinate x 1 , and if x 3 = 1 , the information is between the label and the second coordinate x 2 . The corresponding explanation function is e ( x ) = ✶ [ x 3 = 0] ξ 1 + ✶ [ x 3 = 1] ξ 2 . This explanation is encoding because neither explanation's values x 1 nor x 2 determine the explanation function because it depends on x 3 . Formally

̸

Thus,

which meets Def: Encoding. Consider an alternate non-encoding explanation function e ( x ) = [ ✶ [ x 4 > 0] , ✶ [ x 4 ≤ 0] , 0 , 0] ; x 1 , x 2 do not determine e ( x ) that depends on the noise ϵ 2 in x 4 . That means the unpredictability property in Lemma 1 holds. However, by construction, ( y , x 1 , x 2 ) | = ϵ 2 = ⇒ ( y , x 1 , x 2 ) | = E v = ⇒ y | = E v | x 1 and y | = E v | x 2 .

So no additional information about the label is encoded:

The additional information property in Lemma 1 avoids such cases where the explanations keeps additional information that is irrelevant to the label.

B.2 Encoding explanations conceal predictive inputs that affect the explanation

Consider the following DGP

Let e be an encoding explanation that selects the first coordinate if x 3 = 1 and the second coordinate otherwise. We never observe x 3 when looking only at the explanation. Table 4 shows all possible values of this explanation. Notice that in the third and fourth rows, the value of x e ( x ) changes to match the label y exactly, even though the values of the first two coordinates that we can observe stay constant. It is impossible to understand the perfect predictiveness of x e ( x ) , as the encoding explanation conceals the control flow feature x 3 that determines which of the first two features should be picked to predict the label.

B.3 Position-based encoding fits Def: Encoding

Recall the perceptual task that classifying images of dogs versus classifying images of cats, and consider the encoding explanation e position ( x ) that is

̸

Assume that the inputs in the top leftmost pixels are always background, meaning that the values of these inputs provide no information about the label y | = x 1 , x 2 . Now we check if this intuitivelydefined position-encoded explanation meets the definition for encoding (Def: Encoding). To condition on x ξ 1 , E ξ 1 = 0 , we need q ( y = dog ) = 1 . Note that

̸

Def: Encoding holds because the indicator of which explanation was chosen E ξ 1 determines the label.

̸

This example shows how the encoding definition Def: Encoding captures the informally described position-based encoding from the literature.

B.4 Prediction-based encoding fits Def: Encoding

The informal example of prediction-based encoding from Section 3.1 selects a single input that makes the prediction from all of the input have the highest confidence when given the single input. One way to mathematically express such a selection is as follows:

Here, we describe one set of conditions on the distribution q ( y , x ) for which the explanation in eq. (16) fits the definition of encoding in Def: Encoding. Assume that there exists a non-measurezero set U ⊆ { x : q ( y = 1 | x ) > 0 . 5 } and an index k such that

Further, assume that x ξ k alone does not determine x ∈ U :

Define the explanation indicator E v = [ e prediction ( x ) = v ] .

By eqs. (18) and (20) and the definition of E v

By eqs. (21) and (22)

By eq. (23), q ( y = 1 | x ξ k , E ξ k = 1) and q ( y = 1 | x ξ k , E ξ k = 0) are well defined. Then, by eqs. (17) and (19), for x ∈ U

Thus, for all elements of { x ξ k : x ∈ U }

By Lemma 1, the properties in eqs. (23) and (24) imply that ∀ a ∈ { x ξ k : x ∈ U }

Finally, the set U k = { x ξ k : x ∈ U } is non-measure-zero: as ✶ [ x ∈ U ] = 1 = ⇒ ✶ [ x ξ k ∈ U k ] = 1 , accumulating q ( x ) with the restriction x ξ k ∈ U k leads to at least as much mass as accumulating with the stricter restriction x ∈ U :

✶ The assumptions above imply the facts below about e prediction ( x ) :

By eqs. (18) and (20)

cat

dog

(a)

Distribution

q

(

x

,

y

)

(b)

x

e

(

x

)

x

e

(

x

)

Non-encoding explanation

x

e

(

x

)

e

(

x

)

x

e

(

x

)

e

(

x

)

(c)

Marginal encoding

B.5 MARG explanations are encoding

We provide an illustrative example of a MARG explanation for the DGP in Figure 5. Here, we show how a mathematical formulation of MARG satisfied Def: Encoding.

Consider a generic DGP with a Bernoulli control flow input denoted x c : for some distinct sets U, V that do not include c and let no combination of x c , x U , x V determine the rest

Further, assume that the two subsets leads to different distributions over y = 1 such that on a non-zero measure subset S U ⊆ { x U : x such that x c = 1 }

̸

Now, consider a MARG explanation that looks at x c and outputs the corresponding sets U, V :

By definition the explanation only depends on the control flow input, not by x U or x V . Next, as E U = 1 is the same event as x c = 1 , e ( x ) is encoding because the assumption from eq. (25) implies:

̸

Then, this inequality holds for all elements of the non-measure-zero set S U , by Lemma 1, MARG is encoding.

B.6 Proof of Proposition 1

Definition 4. We denote val ( x e ( x ) ) as the function that maps explanation x e ( x ) = ( v , a ) to the values the inputs take at the selected indices, right-padded to have the same dimension as the input x ∈ R d :

For example, if x = [ α, β, γ ] and e ( x ) = [0 , 1 , 0] , then x e ( x ) = ([0 , 1 , 0] , [ β ]) and

Proof. First, note that

See Table 5 for the probability table for why this is true.

For this proof let e ( x ) = e encode ( x ) . In the example eq. (3), masking out the inputs selected by e ( x ) would mean that

In turn, by the construction in eq. (3)

where the conditional independence in the second step turns into the joint independence in the third step due to x 3 | = y .

ROAR scores an explanation highly if x -e ( x ) predicts the label poorly. So if x -e ( x ) is independent of y , then e ( x ) would be scored optimally. Then, due to eq. (26), ROAR scores an encoding explanation optimally.

FRESH scores an explanation highly if the selected value val ( x e ( x ) ) (as defined in Definition 4) predicts the label well. From Table 5, we see that p ( y = 1 | x ) = 0 . 9 for all cases with val ( x e ( x ) ) = [1 , pad-token , pad-token ] and p ( y = 1 | x ) = 0 . 1 for all cases with val ( x e ( x ) ) = [0 , pad-token , pad-token ] . Thus, if val ( x e ( x ) ) = [1 , pad-token , pad-token ] then

and if val ( x e ( x ) ) = [0 , pad-token , pad-token ] then

Therefore, in all cases, p ( y = 1 | x ) = p ( y = 1 | val ( x e ( x ) )) . Thus, predicting label from the selected value alone is as good as predicting from the whole input. As a result, FRESH scores this explanation optimally.

B.7 Showing that e encode is the optimal reductive explanation for eq. (3) and scores better than a constant explanation under EVAL-X

We repeat the DGP in eq. (3) here

Lemma 5. In the DGP in eq. (3) ,

Proof. We can compute these values from Table 5.

Proposition 2. Let e c ( x ) = ξ 3 . Then, for the DGP in eq. (3) , EVAL-X ( q, e encode ) > EVAL-X ( q, e c ) .

Proof. By Lemma 5, we have

Now, denote e encode as e e for ease of reading

This concludes that EVAL-X ( q, e encode ) > EVAL-X ( q, e c ) .

Lemma 6. In the DGP in eq. (3) , e encode ( x ) is an EVAL-X -optimal reductive explanation and is encoding.

Proof. First, the following properties show for the DGP because when x 3 = 1 , y only depends on x 1 , and if x 3 = 0 , y only depends on x 2 :

These independencies imply that

Then, the optimal explanation function that achieves EVAL-X ∗ is e ( x ) = [1 , 0 , 1] if x 3 = 1 and [0 , 1 , 1] otherwise.

Reductive explanations of size 1. If the explanation is forced to have fewer than 2 inputs, the optimal reductive explanation e ( x ) is only allowed to be one of ξ 1 , ξ 2 , ξ 3 :

Rewriting this expression to split the support of x based on x 3 = 1 or 0 :

We will now focus on the three terms within each of eq. (27) and eq. (28). Due to the following equality

the expectations in the first terms in each of eq. (27) and eq. (28) are

Next we turn to setting i = 1 term in eq. (27). Due that q ( y = 1 | x 2 = 1) = q ( y = 0 | x 2 = 0) ,

̸

The same equalities hold for the i = 2 term in eq. (28) E q ( y , | x 2 , x 3 =0) log q ( y | x 1 ) . Finally, regardless of x 1 , x 2 , the i = 3 terms in both eq. (27) and eq. (28) can be expressed as follows:

Now we can maximize the sum of eq. (27) and eq. (28), over e ( x ) such that | e ( x ) | = 1 .

Notice that setting ✶ [ e ( x ) = ξ 1 ] = 1 when x 3 = 1 and ✶ [ e ( x ) = ξ 2 ] = 1 when x 3 = 0 achieves the highest score -0 . 44 in each of eq. (27) and eq. (28). This implies that one optimal reductive explanation is ξ 1 = [1 , 0 , 0] if x 3 = 1 and ξ 2 = [0 , 1 , 0] otherwise. This is an encoding explanation as we show below. Due to E ξ 1 = x 3 ,

̸

In turn, y ̸| = E ξ 1 | x ξ 1 for { x : e ( x ) = ξ 1 } and Def: Encoding holds, meaning that e ( x ) is encoding.

B.8 An example of misestimation of EVAL-X

Consider the following example where

Assume that the misestimated EVAL-X model satisfies these equalities

There exists a bad explanation that scores optimally under the misestimated EVAL-X:

Then the EVAL-X score of this explanation under this particular misestimation is

Since y is deterministic given x so the maximum value of the EVAL-X score is also 0 . So the bad explanation scores optimally due to misestimation. Deterministic y | x is not necessary for estimation error to affect explanation quality. Here, with this incorrectly estimated EVAL-X, inputs that are pure noise, independent of everything, will be chosen.

B.9 Attention map explanations be encode.

Here, treating each of the coordinates of x as tokens, we consider a cross-attention based predictive model of the following form: with γ ( a ) as softmax function over a vector a , W as a matrix, α , β as vectors, and σ as the sigmoid function, and κ as the temperature, the predictive model f ( · ) is

We then show that using the highest attention score as the explanation produces an encoding explanation. For this example, we consider the following DGP:

The following setting of parameters in f ( x ) produces a function of x that converges to ρ as κ →∞ :

By definition of W

Then, β 3 = 0 , the inner sum for i = 3 does not appear in the function f ( x ) Then, as α 3 = 0 , α 1 = 1 , α 2 = -1 ,

In turn, as β 1 = β 2 = 1

So, as κ →∞ , the function f ( x ) , with the parameters in eq. (30), converges to ρ ( x ) , meaning that this model will achieve the population log-likelihood optimum under the DGP in eq. (29).

Now, the attention map as an explanation selects x 1 if x 3 = 1 and x 2 otherwise; this comes from eq. (31) and eq. (32). This is an encoding explanation because E ξ 1 = 1 if x 3 = 1 which gives

̸

C Experimental details

C.1 Estimating STRIPE-X

To compute the KL term in ENCODE-METER, we estimate q ( E v | x v , y ) and q ( E v | x v ) . To estimate these, we train a single model - to predict E v from x v and a new variable ℓ that can equal

the label y or a dummy value null that is outside the support of y -in the following way:

✶ As log-likelihood is a proper scoring rule and q ( y = null ) = 0 , the maximum above is achieved when

In summary, to estimate ENCODE-METER, solve eq. (34), use its solution to estimate the KL term from the RHS in eq. (7) for each x v , y , and then average this KL term over samples of x v from the data such that e ( x ) = v and samples of y from the EVAL-X model for q ( y | x v ) .

In practice, one does not need train a model for each v . We describe how to estimate ENCODEMETER with a single model in Appendix C.2. We give the full STRIPE-X estimation procedure in Algorithm 2 in Appendix D.

C.2 Estimating the encoding cost in STRIPE-X with categorical predictive models

STRIPE-X consists of the EVAL-X score and a cost of encoding measured by ENCODE-METER. Define V to be the set of possible explanations and let V [ j ] denote the j th element of V . The EVAL-X model p γ ( y | x v ) is trained to predict the label y from subsets x v where v is uniformly sampled from V [20]. Next is computing the ENCODE-METER ϕ q ( e ) that is used in the encoding cost term in STRIPE-X. For each explanation, let F be the categorical variable (instead of an indicator E v ) that denotes, for each sample, which inputs were selected by the explanation e ( x ) : F = j if E V [ j ] = ✶ [ e ( x ) = V [ j ]] = 1 . Let q ( j ) be the distribution over j induced by q ( e ( x )) . We train a model p θ ( F | x v , ℓ, v ) with a modification of eq. (34) that averages over v ∼ q ( e ( x )) :

The variable ℓ takes values in {-1 , 0 , 1 } where 0 and 1 correspond to y = 0 and y = 1 respectively and -1 corresponds to the null value. For a flexible enough model p θ that achieves the population maximum of eq. (35), for any v = V [ j ] ∈ V ,

This fact indicates how one can use the model p θ to estimate ENCODE-METER. First, construct the explanation dataset D e = { ( y , x e ( x ) ) } from D t . Define q D e to be the uniform distribution over D e . Define E ( v , a ) as the uniform distribution over K samples of y from the EVAL-X model:

Then, estimate ENCODE-METER as follows:

̸

C.3 Estimating ENCODE-METER with a generative model

When estimating the STRIPE-X score with procedure above for many different explanations, the maximization in eq. (34) repeated for every explanation, which can be computationally expensive.

̸

̸

This motivates a second procedure to estimate ENCODE-METER that avoids having to retrain models for each explanation by using generative model for q ( x | x v , y ) . Formally, with x v fixed, the conditional mutual information term in eq. (5) can be computed as the marginal dependence between N samples of y from q ( y | x v ) and q ( E v | x v , y ) . The model for the former is available from EVAL-X estimation. Simulating from the later, namely q ( E v | x v , y ) , is done by sampling from the generative model x | x v , y and then computing the indicator E v as ✶ [ e ( x ) = v ] . Mechanically, with an estimator of mutual information from samples ( { a i } i ≤ N , { b i } i ≤ N ) denoted MI ( { a i } , { b i } ) and with samples { a i } produced conditionally on values c i denoted by a subscript of the conditioned value { a i } c i , one can estimate ENCODE-METER as follows: sample y i ( v , a ) ∼ y | x v = a and

We give the full procedure in Algorithm 1.

C.4 Experimental details from the simulated study.

The data-generating processes from the experiments. Let N be the standard normal distribution and let B ( α ) be the Bernoulli distribution with 1 occurring with probability α . With ρ = 0 . 9 , the discrete DGP is:

The hybrid DGP is as follows: with γ = 5 and σ ( x ) = 1 1+exp( -x ) as the sigmoid function

Computing accuracy and KL to show that POSI, PRED, MARG are encoding. For each encoding type, we build two decision trees from 1000 samples from eq. (36): the first decision tree learns q ( E v | x v ) and the second learns q ( y | x v , E v = b ) for b ∈ { 0 , 1 } . We set the maximum depth to be 6 . Trees of this depth learn any function of 6 binary digits; x with E v as an additional column amounts to 6 binary digits. These decision trees are used to compute the accuracy of predicting E v with q ( E v | x v ) and the KL between q ( y | x v , E v = 1) and q ( y | x v , E v = 0) . Within a set { x : e ( x ) = v } that is all x that have one of the possible selections v , Table 6 report the accuracy of predicting E v with q ( E v | x v ) and the KL between q ( y | x v , E v = 1) and q ( y | x v , E v = 0) , averaged only over samples in { x : e ( x ) = v } .

EVAL-X. To estimate EVAL-X for the DGPs in eq. (36) and eq. (37), we compute conditionals q ( y = 1 | x v ) via Monte Carlo approximation. Due to the different coordinates of x being independent, one can compute q ( y = 1 | x v ) as a marginal expectation over the inputs except those in v :

WeMonte Carlo estimate the RHS of this equation over 500 resamples of x c v . Wetake 5000 samples from each DGP to estimate EVAL-X scores with respect to q ( y , x ) . In Appendix C.5 we also show experiment results where we use the EVAL-X accuracy and AUROC as the score instead.

(a)

Results: Discrete DGP.

(a)

Results: Discrete DGP.

(b)

Results: hybrid DGP.

1.0

1.0

1.0

1.0

EVAL-X*-AUROC

0.5

0.5

0.5

0.5

EVAL-X*-ACC

EVAL-X*-AUROC

EVAL-X*-ACC

0.0

0.0

0.0

0.0

POSI

R

R

POSI

MARG

R

ISO

POSI

R

G

REAL-X. We solve REAL-X for any specified explanation size K as follows. In the case of the discrete DGP, for each possible value of x ∈ supp ( q ( x )) (of which there are finitely many), we make e ( x ) output the subset of at most size K that achieves that maximum averaged log-likelihood over the samples that equal said value x = x . This produces the optimally-scoring explanation e ( x ) that maps each finite value in the support of q ( x ) to one subset of the coordinates of x . To do the same in the continuous DGP in eq. (37), we round x to integers and then use the same type of optimization as in the discrete case.

STRIPE-X. In estimating ENCODE-METER, the model p θ ( E v | x v , ℓ ) is a decision tree of depth at most 5, which is then used to estimate averaged KL in the RHS of eq. (7) with a single sample from y | x v . The process is repeated for each v and averaged to produce the ENCODE-METER. The simulated experiments were done on a CPU with the whole runtime around 10 minutes.

C.5 Experiments with EVAL-X accuracy and EVAL-X AUROC

Here, instead of EVAL-X log-likelihoods, we use the accuracy and AUROC of the EVAL-X model as the score. We call these EVAL-X-ACC and EVAL-X-AUROC scores. These metrics only depend on the ranking of the datapoints, and therefore are not sensitive to differences in log probabilities that do not change ranks. Figure 6 shows that, due to this insensitivity, multiple encoding explanations (PRED, MARG, and the excessively reductive one) all achieve the same score as the corresponding EVAL-X ∗ score. In summary, ranking metrics like accuracy and AUROC are not sensitive to encoding explanations like EVAL-X log-likelihoods.

C.6 Classifying dogs and cats.

The POSI explanation selects the upper or the lower color patch depending on whether q ( y = 1 | x ) > 0 . 5 or not. The PRED explanation selects the patch predicting from which best matches the prediction from q ( y = 1 | x ) . The MARG explanation selects the top or the bottom image patch based on the color as in the DGP in Figure 5. We consider two non-encoding explanations. The first explanation, denoted optimal , selects exactly the features that occur in the DGP: { x 1 , x 2 } if the color patch x 1 is blue and { x 1 , x 4 } otherwise. As y is determined by the explanation, meaning y | = E v | x v for all v and values x v , this explanation is non-encoding. The second one, denoted fixed , always outputs the bottom right patch x 4 ; this explanation is constant which violates the first criterion in Lemma 1 meaning there is no encoding.

We also run the REAL-X method from [20] to produce an explanation. REAL-X was run over explanations that select one of the four quarter patches and exact marginalization over the selections.

The base cat and dog images were obtained from the cats_vs_dogs dataset from the Tensorflow datasets package. To construct images like in Figure 5, the color and the two images are sampled independently. The color being blue/red determines that the label associated with the top/bottom image becomes the label for the constructed image. The training, validation, and test dataset consist of 8000 , 1000 , and 1000 samples respectively.

We follow the procedure in Appendix C.2 to estimate STRIPE-X. The EVAL-X model is a pre-trained 18-layer residual network. The model p θ ( F | x v , ℓ, v ) used in computing the ENCODE-METER term in STRIPE-X (eq. (6)) are 34-layer Residual neural networks. The EVAL-X model is trained for 100 epochs with a batch size of 100 with the Adam optimizer, with the learning rate and weight decay parameters set to 10 -3 and 0 respectively. The p θ ( F | x v , ℓ, v ) model is trained for 50 epochs with a batch size of 200 with the Adam optimizer, with the learning rate and weight decay parameters set to 5 × 10 -5 and 1 respectively. The p θ model sees variable ℓ through an entire extra channel where all the pixels take the value ℓ . We used validation loss as the metric to early stop. The EVAL-X and STRIPE-X scores are computed on the test dataset. The cats vs. dogs experiment were done on an A100 GPU where the whole training and evaluation ran in less than 20 minutes.

Remark on the gap between FRESH and EVAL-X scores for the optimal explanation. As the optimal explanation selects features sufficient to produce the label, meaning y | = x | x v or y | = x | val ( x e ( x ) ) , FRESH and EVAL-X log-likelihoods should be the same as predicting from the full feature set. One potential reason there is a gap between the two scores in table 3 is that the FRESH and EVAL-X scores are computed with ResNet18 models that solve prediction problems of different levels of difficulty On one hand, FRESH is computed with a model trained for a single prediction task: predict y from val ( x e ( x ) ) . On the other hand, EVAL-X is computed with a model trained for a more complicated task: for a range possible v , predict y from x v . Using large models with appropriate regularization, such as weight decay, should mitigate the gap in scores.

C.7 LLM experiment details

We generate 10 , 000 reviews of the following type: with ADJ1 and ADJ2 as adjectives, the review is

'My day was <ADJ1> and the movie was <ADJ2>. that is it' or

'My day was <ADJ1> and the movie was <ADJ2>. oh wait, reverse the adjectives'

The second sentence in the review acts as a "control flow" input and determines whether ADJ1 or ADJ2 describes the sentiment about the movie. We prompted Llama 3 to predict the sentiment and select words relevant to predicting the sentiment. In Appendix C.8, we give the prompts we used to make Llama 3 produce explanations from. For this problem, the inputs x are the reviews and Llama 3 produces explanations e ( x ) that select a subset of words in the review. The summaries and explanations were generated for all 10 , 000 samples but to estimate STRIPE-X, we only used data from the 5 most common explanations (we restricted to inputs whose explanations v had high q ( e ( x ) = v ) ). This resulted in a dataset of size 8136 , which we split into a training, validation, and test datasets of sizes 6102 , 1017 , and 1017 respectively.

Both the EVAL-X model and the model for p θ ( F | x v , v , ℓ ) (see Appendix C.2) used in estimating the ENCODE-METER term in STRIPE-X were finetuned GPT-2 models. For the EVAL-X model, we used the AdamW optimizer with a batch size of 100 and trained for 50 epochs with the learning rate set to 5 e -5 , weight decay set to 0 , and a Cosine learning rate scheduler with the number of cycles set to 1 . For the p θ model used in estimating ENCODE-METER, we used the AdamW optimizer with a batch size of 50 and trained for 25 epochs with the learning rate set to 5 e -5 , weight decay set to 0 , and a Cosine learning rate scheduler with number of cycles set to 1 . The p θ model sees variable ℓ through the following word added to the input sequence of words: positive if ℓ = y = 1 , negative if ℓ = y = 0 , and nothing if ℓ = null . We used validation loss to early stop. We follow the procedure in Appendix C.2 to compute ENCODE-METER with p θ ( F | x v , v , ℓ ) on the test data with the averaging over y | x v estimated using a 5 samples per value of x v . All training and inference for this experiment was done on an A100. The explanation step and the estimation for both parts of STRIPE-X together took under 2 hours. The LLM-generated explanations achieves an EVAL-X score of -0 . 497 and an ENCODE-METER value was 0 . 114 .

C.8 Prompts used to predict sentiment and produce explanation

In Figure 7, we provide the prompt we used to predict the sentiment from a review and generate an explanation for that prediction.

D Algorithms

Algorithm 1 describes an alternate way to estimate the ENCODE-METER component of STRIPE-X with a conditional generative model. Algorithm 2 describes the predictive version of STRIPE-X estimation, which we used in our experiments.

Algorithm 1: ENCODE-METER, generative version.

Input: Training data D ∼ q ( y , x ) and test data D t ∼ q ( y , x ) , explanation function e ( x ) , penalty weight λ . EVAL-X model p γ ( y | x v ) . Conditional generative model p θ ( x | x v , y ) and mutual information estimator that takes two sets as arguments MI [ { c i } , { d i } ] ;

Result: Return estimate of STRIPE-X :

1 Define J ( v , a ) as the set of K random samples of y from the EVAL-X model:

2 Define L ( v , a , J ( v , a ) ) as the set of K random samples of x from p θ conditioned on a and ˆ y :

4 Define q D e to be the uniform distribution over D e .

5 Compute the following averaging of estimated mutual information between, J , L

6 Return ˆ ϕ ( q, e ) as the ENCODE-METER estimate.

Algorithm 2: STRIPE-X, predictive version.

Input: Training data D ∼ q ( y , x ) and test data D t ∼ q ( y , x ) , explanation function e ( x ) , penalty weight λ . Specifications for the models p γ ( y | x v ) and p θ ( F | x v , ℓ ) .

Result: Return estimate of STRIPE-X :

1 Define q D to be the uniform distribution over D

2 Construct the explanation dataset D e = { ( y , x e ( x ) ) } from D t

3 Define q D e to be the uniform distribution over D e .

4 Estimate eval-x ()

5 Solve the following minimization problem to learn p γ ( y | x v )

Output: p γ

6 Estimate encode-meter ()

7 Construct the set of possible selections V = { v : q ( e ( x ) = v ) > 0 }

8 Construct data of the form ( x , F ) where F = j if E V [ j ] = 1 .

9 Fit the model p θ ( F | x v , ℓ ) via the following log-likelihood maximization:

10

11

Define E ( v , a ) as the uniform distribution over K samples of y from the EVAL-X model:

Output: The following nested expectation over q D e and E ( · ) :

̸

12 Learn the EVAL-X model p γ ← eval-x () .

13 Estimate ENCODE-METER as the ˆ ϕ q ( e ) ← encode-meter () .

14 Return the following as the STRIPE-X estimate:

̸

̸

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper claims that the main problem regarding the evaluation of explainability methods is encoding, which refers to the leakage of information in explanation (here defined as a boolean mask for selecting features) structure and not the value of the selected features. The paper tries to quantify the extent of this leakage and analyzes some previously designed methods from the proposed perspective, summarized in Table 1. They continue an intuitive approach to this quantity and use some definitions to build a measure termed \"ENCODE-METER\". This measure helps them to tweak the EVAL-X objective, they call DET-X, to gain \"strong\" encoding detection in definition 3. The claims, in this work, are supported by proofs and sometimes based on explanations that come from hand-designed data generating process for constructing counter examples. Finally, they evaluate the proposed framework on some data generating processes quantitatively.\n\nSTRENGTHS:\n1. The paper adopts a formal approach toward its claims and goals.\n2. The paper provides intuition and summarizes it in concise mathematical equations.\n3. The goal and problem being tackled is of high priority in the field of explainability, and affects many researchers in the field.\n4. The direction that the paper is proposing seems quite promising and must be better studied.\n5. The paper tackles encoding that is not formalized yet and underdeveloped, which adds to the novelty of the work.\n\nWEAKNESSES:\nI would like to thank authors for such a hard work and contribution to the community.\nI have learned a lot from this work, so the points mentioned here must not seem disappointing, but rather it should help for a work with higher quality. It seems that this work is very close to publication, yet with small potential errors I pointed out here (nonetheless, I might be wrong).\n\n## Important missing discussions\n1. It would be helpful to rank some **known explanation methods** and compare it with **known metrics** like ROAR. Then it would be easier for the reader to decide what are the differences between your evaluation method and other evaluation methods. ROAR for example ranks many methods and reaches to a bottom-line and raises some open questions. Toy datasets are good for formal proofs, simplicity and demonstration. But relying on toy datasets can create a gap between modelling and practice. I would recommend having a table similar to ROAR that ranks methods. Maybe we see that the results of DET-X, although being more elegant in theory, actually coincide with the results one gets with ROAR or some other heuristic method.\n2. It seems that the main idea of this paper and some referenced papers revolves around spurious correlations in the DPG. Could be useful if there is a discussion on how spurious correlations in the DGP affects explanations.\n\n## Formal writing\n1. definition 1, does not look like a definition, it is written like a proposition. It is easier to understand if a definition is like \"We call X an encoding if Y holds.\"\n2. Definition 2 is also like a proposition, or at least I do not understand what is being defined here. Also in $\\max_e$ what is the domain of e here? Is it over explanation methods or over explanations that are generated by a dataset. What is the domain of $e^*$?\n3. does definition 3, depend on the direction of the metric (higher the better or lower the better)?\n4. I cannot follow the argument in line 234-239. Maybe because I've not understood the definitions 1 and 2. In a proof, to show that a property satisfied, finally to show that X fits in some definition.\n\n## Punctuation, writing style and minor errors\n1. line 71 \"that Hooker et al.\" should be removed\n2. it is better for readability of the paper to provide a more intuitive definition of \"encoding explanations\" earlier in the paper I would recommend putting this line: \"Intuitively, encoding is when the binary mask output of the explanation itself provides information about the label beyond the selected values.\" earlier. Even in the abstract or early in the introduction when you want to mention encoding.\n3. lines 123-124, \"... to predict the label have not been selected ...\" shouldn't this proposition stated weaker? something like  \"... to predict the label **may** have not been selected ...\"\n4. line 139 punctuation should be: \"likely**,** exactly**,** when\"\n5. line 100 punctuation better be: \"method is good**,** requires\"\n6. lines 154-161 reads poorly, maybe I do not understand what it tries to convey!\n7. DGP in line 164 is defined later in 191.\n9. in Eq. 3 the word \"Different\" better reflect the mathematical form than \"Additional\"\n10. lines 196-198 this is too informal, or at least I cannot make sense of it. \n11. line 208 punctuation \"provides\" must be \"provide\".\n\nQUESTIONS:\n1. in figure 1 (c), the assumptions are unclear, why position based encoding explanation always selects the top-half? is it in the DGP that the cat always appear on the top? Or it is in the definition of encoding based explanation. More on that, how do we know \"The label is independent of the color in the selected features\"? Shouldn't you show this formally using definition of independence?\n2. line 164 \"may seem okay\" what does this mean? is it better to adopt a more formal writing style here?\n3. line 164-164 \"because the color is independent of the label,\" can you show this more formally using the definition of independence? I think that the label well depend on the color, at least here.\n4. lines 165-166 \"hides ... the predictions’ dependence on the control flow input\" how the control flow can be shown in an explanation? I guess this does not fit in the definition of salience based explanations?\n5. lines 209-210 \"only work when optimizing without constraints\" what is optimization referring to here? what are constraints?\n6. refer to weaknesses section \"Mathematics writing\" number 2.\n7. in line 220-221 I do not understand why it is written \"x3 predicts the label\". Maybe I'm wrong, but it seems that the label could be 1 or 0 when x3 is known and equal to 1 (for example). So knowing x3 does not tell anything about the label, and you still need to look at the other features (either x1 or x2). Can you make a table of conditional probabilities and reason why x3 tells the label?\n\nLIMITATIONS:\nAlthough authors have answered \"Yes\" to the checklist for \"discussion on limitations\", I didn't find a separate limitations section in the paper. Maybe they have referred to the limitations of their work indirectly.\n\nOne limitation that I see in this work is that the arguments are based on assumptions that might never come true on real world datasets like Imagenet. This is why I encourage the authors to go beyond toy datasets and evaluate their method in the wild.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach to evaluate feature attribution methods in machine learning by addressing the issue of encoding in explanations. The authors define encoding as when the explanation's identity provides additional information about the target beyond the selected input values. They categorize evaluation methods into weak detectors (optimal for non-encoding explanations) and strong detectors (score non-encoding explanations higher). The paper introduces DET-X, a new score that strongly detects encoding, and empirically verifies its effectiveness through simulated and real-world datasets, including an image recognition task and sentiment analysis of movie reviews.\n\nSTRENGTHS:\n- The paper introduces a precise mathematical definition of encoding, addressing a significant gap in the interpretability literature.\n\n- The classification of evaluation methods into weak and strong detectors provides a clear framework for assessing the robustness of feature attribution methods. \n\n- The authors rigorously prove that their proposed DET-X score strongly detects encoding, differentiating it from existing methods.\n\n- The paper includes empirical validation on both simulated data and real-world applications, demonstrating the practical utility of DET-X.\n\n- By uncovering encoding in LLM-generated explanations for sentiment analysis, the paper shows the relevance of its contributions to current AI applications.\n\nWEAKNESSES:\n- The proposed DET-X score may require complex implementation and computational resources, which might limit its adoption in practical scenarios.\n\n- While the experiments are thorough, they are limited to specific datasets and types of tasks (image recognition and sentiment analysis). More diverse applications would strengthen the claims.\n\n- The paper acknowledges that misestimation of models used in evaluation could lead to incorrect conclusions, suggesting a need for robust estimation techniques.\n\nQUESTIONS:\n- How does the DET-X score perform in different domains outside of image recognition and sentiment analysis?\n\n- Are there specific conditions or types of models where DET-X might not perform as expected?\n\n- How does the computational cost of DET-X compare to existing evaluation methods?\n\nLIMITATIONS:\n- The paper's findings are primarily validated on specific datasets and tasks, which may not generalize to all types of machine learning applications. \n\n- The DET-X score's implementation complexity might be a barrier for widespread adoption, especially in resource-constrained environments.\n\n- The paper highlights the risk of misestimation in model-based evaluations, which could impact the reliability of DET-X in certain scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nHow to best evaluate explanations is an important open question in the field, and one specific challenge that has so far received less attention is how to detect when explanations encode prediction in the identity of the selected inputs. This paper proposes a formal definition of encoding, that they later use to probe whether current evaluation protocols can detect encoding through experiments on simulated data. They find existing evaluation methods to be at best weak detectors of encoding, and they propose the evaluation method DET-X as a strong detector of encoding.\n\nSTRENGTHS:\n- The motivation for defining and properly evaluating the notion of encoding is sound\n- the experiments look sound\n- as far as I know, the formal definition is a novel contribution\n\nWEAKNESSES:\n- The main and major weakness of the paper is clarity.\n\t- The paper is very very dense (space between paragraphed tempered and heavily reduced) and this makes it at times intelligible. \n- Weak contextualization with regard to prior work. The related work in the main paper is very succinct. \n- Overall I believe that the substance of this paper is interesting, but in its current form, the paper is really hard to read and understand.\n\nQUESTIONS:\n- Fig.1C, \"the label is produced from the bottom image if the color is red\" then why is the explanation about the top image?\n- l.156 \"For example, consider reviews of the type \"My day was terrible, but the movie was [ADJ].\" where ADJ can be \"good\" or \"bad\" and let the explanation be \"terrible\" if ADJ=bad and \"Movie was good\" if ADJ=good. The sentiment about the movie comes from the second part which the explanation fails to isolate, meaning this explanation should be scored poorly.\" I am not sure the problem is that straightforward here. How would we differentiate a model that makes that mistake from an erroneous explanation?\n- l.165 \"control flow\" is not defined, hence it's hard to understand this section\n- l.366 \"We compute ENCODE-METER with q(Ev | xv) modeled by a ResNet34 trained the same way as EVAL-X but, instead of predicting the label, it predicts the identity of subset selected by the explanation\" I don't understand what this means, could you please clarify?\n\nLIMITATIONS:\nI do not see a limitation section in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}