Testing Semantic Importance via Betting
Authors: Jacopo Teneggi
Jeremias Sulam
Venue: NeurIPS 2024 poster (2024)
Keywords: Explainability
Semantic Concepts
Sequential Testing
Conditional Independence Testing
Primary area: interpretability_and_explainability

Abstract: Recent works have extended notions of feature importance to semantic concepts that are inherently interpretable to the users interacting with a black-box predictive model. Yet, precise statistical guarantees such as false positive rate and false discovery rate control are needed to communicate findings transparently, and to avoid unintended consequences in real-world scenarios. In this paper, we formalize the global (i.e., over a population) and local (i.e., for a sample) statistical importance of semantic concepts for the predictions of opaque models by means of conditional independence, which allows for rigorous testing. We use recent ideas of sequential kernelized independence testing to induce a rank of importance across concepts, and we showcase the effectiveness and flexibility of our framework on synthetic datasets as well as on image classification using several vision-language models.

Testing Semantic Importance via Betting
Jacopo Teneggi Johns Hopkins University j tenegg1 @iuu. edu
Jeremias Sulam Johns Hopkins University jsulam1 @juu. edu edu
Abstract
Recent works have extended notions of feature importance to semantic concepts that inherently interpretable to the users interacting with black-box predictive model. Yet, precise statistical guarantees such as false positive rate and false discovery rate control needed to communicate findings transparently, and avoid unintended consequences in real-world scenarios. In this paper, we formalize the global (i.e., over population) and local (i.e., for sample) statistical importance of semantic concepts for the predictions of opaque models by means of conditional independence, which allows for rigorous testing. We use recent ideas of sequential kernelized independence testing to induce rank of importance across concepts, and we showcase the effectiveness and flexibility our framework on synthetic datasets as well as on image classification using several vasion-language models.
1 Introduction
Providing guarantees on the decision decision-making processes of autonomous systems, often based on complex black-box machine learning models, is paramount for their safe deployment. This need motivates efforts towards responsible artificial intelligence, which broadly entails questions of reliability, robustness, fairness, and interpretability. One popular approach to the latter is to post-hoc explanation methods to identify the features that contribute the most towards the predictions of model. Several alternatives have been proposed over the past few years, drawing from various definitions of features (e.g.. pixels-or groups thereof-for vision tasks [41], words for language tasks [21], nodes and edges for graphs [84]) and of importance (e.g, gradients for Grad-CAM [57], Shapley values for SHAP [7, 15, 42, 70]. or informatiion-hhoreeee
quantities [43]). While most explanation methods highlight important features in the input space the predictor, users may care more about their meaning. For example, a radiologist may want to know whether model considered the size and spiculation of lung nodule to quantify its malignancy, and not just its raw pixel values.
To decouple importance from input features, Kim et al. [34] showed how to learn the vector repre- sentation of semantic concepts that are inherently interpretable to users (e.g., "stripes", "sky", or "sand") and how to study their gradient importance for model predictions. Recent vision-language (VL) models that jointly learn an image and text encoder, such CLIP [16, 52], have made these representations- commonly referred to as concept activation vectors (CAVs)-more easily acces- sible. With these models, obtaining the representation of concept boils down to a forward pass of the pretrained text encoder, which alleviates the need of dataset comprising images annotated with their concepts. Several recent works have defined semantic importance-bort with CAVs and VL models-by by means of concept bottleneck models (c.g.. CBM [36], PCBM [85], LaBo [83]). information-theereticc quantities (e.g., V-IP [37]), sparse coding (e.g., CLIP-IP-OMP [13], SpLiCe [6]), network dissection [2] (e.g., CLIP-DISSECT [47], TextSpan [26]. INVITE [14]), or causal inference (e.g.. DiConStruct [44], Sani al. [55]).
On the other hand, it is important to communicate findings of important features precisely and transparently in order avoid unintended consequences in downstream decision tasks. Going back to the example of a radiologist diagnosing lung cancer, how should they interpret two concepts with
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

different importance scores? Does their difference in importance carry any statistical meaning? To start addressing similar questions [10, 71] introduced statistical tests for the local (i.e., on a sample) conditional independence structure of a model's predictions. Framing importance by means of conditional independence allows for rigorous testing with false positive rate control. That is, for a user-defined significance level a: E (0. 1), the probability of wrongly deeming a feature important is no greater than a, which directly conveys the uncertainty in an explanation. Yet these methods consider features as coordinates in the input space, and it is unclear how to extend these ideas to abstract, semantic concepts.
In this work, we formalize semantic importance at three distinct levels of statistical independence with null hypotheses of increasing granularity: (i) marginally over a population (i.e.. global importance), (ii) conditionally over a population (i.e., global conditional importance), and (dáz) for a sample (i.e., local conditional importance). Each of these notions will allow us to inquire the extent to which the output of model depends on specific concepts-boit over population and on specific samples-and and thus deem them important. To test for these notions of semantic importance, instead of classical (or offline [58]) independence testing techniques [5, 10, 11, 28, 29, 69, 86]. which are based on p-values and informally follow the rule "reject if S a we propose to use principles of testing by betting (or sequential testing) [59], which are based on e-values [76] and follow the "reject when e 2 1//x" rule. As we will expand on, this choice is motivated by the fact that sequential tests data-efficient and adaptive to the hardness of the problem- which naturally induces rank of importance. We will couple principles of conditional randomization testing (CRT) [11] with recent advances in sequential kernelized independence testing (SKIT) [51, 62], and introduce two novel procedures to test for our definitions of semantic importance: the conditional randomization SKIT (C-SKIT) to study global conditional importance, and-following the explanation randomization test (XRT) framework [71]-the explanation randomization SKIT (X-SKIT) to study local conditional importance. We will illustrate the validity of our proposed tests on synthetic datasets, and showcase their flexibility on zero-shot image classification on real-world datasets across several and diverse VL models.
1.1 Summary of Contributions and Related Works
In this paper, we will rigorously define notions of statistical importance of semantic concepts for the predictions of black-box models via conditional independence- both globally over a population and locally for individual samples. For any set of concepts, and for each level of independence, we introduce novel sequential testing procedures that induce rank of importance. Before presenting the details of our methodology, we briefly expand on a few distinctive features of our work.
Explaining nonlinear predictors. Compared to recent methods based on concept bottleneck models [36, 83, 84], our framework does not require training a surrogate linear classifier because we study the semantic importance structure of any given, potentially nonlinear and randomized model. This distinction is not minor-training training concept bottleneck models results on explanations that pertain to the surrogate (linear) model instead of the original (complex, nonlinear) predictor, and these simpler surrogate models typically reduce performance [48, 85). In contrast, we provide statistical guarantees directly the original predictor that would be deployed in the wild.
Flexible choice of concepts. Furthermore, our framework does not rely on the presence of a large concept bank (but it can use one if it is available), Instead, we allow users to directly specify which concepts they want to test. This feature is important in settings that involve diverse stakeholders. In medicine, for example, there are physicians, patients, model developers, and members of the regulatory agency tasked with auditing the model each of whom might prefer different semantics for their explanations. Current explanation methods cannot account for these differences oef-.
Local semantic explanations. Our framework entails explanations for specific (fixed) inputs, whereas prior approaches that rely on the weights of linear model only inform on global notions of importance. Recently, Shukla et al. [64] and Pham al. [50] set forth ideas of local semantic importance by combining LIME [54] with T-CAV [34], and by leveraging prototypical part networks [46], respectively. Our work differs in that it does not apply to images only, it considers formal notions of statistical importance rather than heuristics of gradient importance, and it provides guarantees such as Type error and false discovery rate (FDR) control.
We adopt the distinction between global and local importance presented in [18].

---

Sequential kernelized testing. Motivated by the statistical properties of kernelized independence tests [62]. we will employ the maximum mean discrepancy (MMD) [29] as the test statistic in our proposed procedures. The recent related work in Shaer et al. [58] introduces the sequential version of the conditional randomization test (CRT) [11], dubbed e-CRT because of the use of e-values. Unlike our work, Shaer et al. [58] employ residuals of a predictor as test statistic, they do so in the context of global tests only, and unrelated to questions of semantic interpretability.
Background
In this section, we briefly introduce the necessary notation and general background. Throughout this work, we will denote random variables with capital letters, and their realizations with lowercase. For example, X Px is random variable sampled from Px, and indicates an observation of X.
Problem setup. We consider k-fold classification problems such that (X Pxy is a random sample X X' with its one-hot label Y E {0,1)*, and (I, y) denotes particular observation. We assume are given fixed predictive model, consisting of an encoder f: X' -> R and classifier : IRd + IR such that = f(x) is a d-dimensional representation of x, and ÿK- = g(h)x = (f(())) is the prediction of the model for a particular class k' (e.g., Dr is the output, or score, for class "dog"). Naturally, H, Y denote the random counterparts of and ÿ. Although our contributions do not make any assumptions on the performance of the model, f and g can be thought of as good predictors, e.g. those that approximate the conditional expectation of Y given X.
E Rddxm be a dictionary of m concepts such that Vj E [m]] ;= fl....... m), Cj E Rd is the representation of the jth concept-eithee obtained via CAVs [34] or a VL model's text encoder. Then, (c, h) is the projection of the embedding h onto the concepts c, and, with appropriate normalization, 2j E [-1, 1] is the amount concept сj in h. Intuitively, CBMs project dense representations onto the subspace of interpretable semantic concepts [36], and their performance strongly depends on the size of the dictionary [48]. For example, it is common for to be as large the embedding size (e.g., d = 768 for CLIP:ViT-L/14). In this work, instead, we let concepts be user-defined, allowing for cases where m < d (e... m = 20). This by design as (i) the contributions of this paper apply any set of concepts, and (ii) it has been shown that humans can only gain valuable information if semantic explanations are succinet [53]. However, we remark that the construction of informative concept
Concept bottleneck models (CBMs), Let =
especially for domain- specific applications- is subject of ongoing complementary research [20, 48, 78, 80, 81].
Conditional randomization tests. Recall that two random variables A, B are conditionally indepen- dent if, and only if, given a third random variable C, it holds that PAJB.C = PAIC (i.c., A C)). That is, B does not provide any more information about A with C present. Candes al. [11] introduced the conditional randomization test (CRT), based on the observation that if A C, then
the triplets (A, B, C) and (A, B, C) with B ~ PBic. are exchangeable. That is, PABC = P PABC and one can essentially mask B without changing the joint distribution. Opposite to classical methods, the CRT assumes the conditional distribution of the covariates is known (i.e., PBІc), which lends itself to settings with ample unlabeled data.
With this general background, we now present the main technical contributions of this paper.
3 Testing Semantic Importance via Betting
Our objective is to test the statistical importance of semantic concepts for the predictions of a fixed, potentially nonlinear model, while inducing a rank of importance. Fig. depicts the problem setup fixed model, composed of the encoder f and classifier g. is probed via set of concepts c. This figure also illustrates the key difference with post-hoc concept bottleneck models (PCBMs) [85], in that we do not train a sparse linear layer to approximate Y Z]. Instead, we focus on characterizing the dependence structure between Ý and Z. Herein, we will drop the Ok' notation and simply write ÿ, Y because always consider the output of the model for particular class individually.
3.1 Formalizing Statistical Importance of Semantic Concepts
We start by defining global semantic importance marginal dependence between y and Zj. j [m].
3

---

image X
prediction y
our contributions
encoder
embedding
classifier
planist 0.11
rank of statistical importance
cat 0.28
h
g
tiger 0.05
whiskers
predater 0.08
Testing by Betting
2. cute
Fixed model
3. wild
4. piano
concepts
5. music
Rλnno cutee whiskers music waldd
2 (c.
semantics
Figure 1: Overview of the problem setup and our contribution.
Definition (Global semantic importance). For a concept j E [m]]
Hog 1L
the global semantic independence null hypothesis.
Rejecting 0.y means that we have observed enough evidence to believe the response of the model depends on concept j. i.e. concept j globally important over the population. Note that both Ý and Z3 are fixed functions of the same random variable H, i.e. y = g(H) and Z₁ = (ej, H). Then, reasonable to wonder whether there is any point in testing H6, all-can obtain two independent random variables from the same one? For example, let g be linear classifier such that Y (w, H). Rd. Intuition might suggest that if (w, cy) = 0 then Y I. i.e. if the classifier is orthogonal to a concept, then the concept cannot be important. We show in the short lemma below (whose proof in Appendix C.I) that this is false, and that, arguably unsurprisingly, statistical independence is different from orthogonality between vectors, which motivates the need for our testing procedures.
Lemma 1. Let y (w, H), IRd Ifd 2 3, then Ho, is true c/ (w, 0.
The null hypothesis 11 above precisely defines the global importance of concept, but it ignores the information contained in the rest of them, and concepts may be correlated. For example, predictions for class "dog" may be independent of "stick" given "tail" and "paws", although "stick" is marginally important. To address this, and inspired by [11], we define global conditional semantic importance.
Definition (Global conditional semantic importance). For a concept j E [m], let -j = [m]] \ {j)
denote all but the jth concept. Then,
HCS Ý 1 Z1 Z-j
is the global conditional semantic independence null hypothesis.
Analogous to Definition rejecting H&C means that we have accumulated enough evidence believe the response of the model depends on concept even in the presence of the remaining concepts, i.e. there is information about Y in concept j that is missing from the rest.
We stress important distinction between Definition and PCBMs: the latter approximate EE Z] with a sparse linear layer, which inherently interpretable because the regression coefficients directly inform on the global conditional independence structure of the predictions, i.e. Ý = (8, Z). 3 E pT then Y YL Z-; + By = O. In this work, however, we do not assume any parametrization between the concepts and the labels because we want to provide guarantees on the original, fixed classifier g that acts directly on an embedding h. From this conditional independence perspective, PCBMs can be interpreted as (parametric) test of true linear independence (i.e., HPGBM, Y LLL 2- between the concepts and the labels (note that HPCBM has the true label Y and not the prediction Y), whereas we study the semantic structure the predictions of a complex model, which may have learned spurious, non-linear correlations of these concepts from data.
Akin to the CRT [1]], we assume we can sample from the conditional distribution of the concepts, i.e. Zz, Within the scope of this work, is small (m & 20) and we will show how to effectively approximate this distribution with nonparametric methods that do not require prohibitive computational resources. This is advantage of testing a few semantic concepts compared to input

---

features, especially for imaging tasks where the number of pixels is large (2 10") and learning a conditional generative model (e.g.. a diffusion model [67]) may be expensive.
Finally, we define the notion of local conditional semantic importance. That is, we are interested in finding the most important concepts for the prediction of the model locally a particular input x, i.c. g = g(f(x)) Recently, [10, 71] showed how to deploy ideas of conditional randomization testing for local explanations of machine learning models. Briefly, let B, C be random variables and BB C) a fixed, possibly randomized, real-valued predictor. For an observation (6, cb) the
explanation randomization test (XRT) [71] null hypothesis is 17(b, c) _ ry c). B ~ PBI-- That is, the observed value of B does not affect the distribution of the response given the observed value of C. We now generalize these ideas.
Definition 3 (Local conditional semantic importance). For a fixed 1-1, and any C' S [m]] denote YC = g( (Hc) with He ~ PHIZe=ze- Then, for a j E [m]] and a subset S [m]] \(),
Hoss YSU Y
(3)
the local conditional semantic independence null hypothesis.
In words, rejecting Hog.s means that, given the observed concepts in S, concept j g S affects the distribution of the response of the model, hence it is important. For this test, we assume we can sample from the conditional distribution of the embeddings given subset of concepts (i.e., PhIZo=sa). This is equivalent to solving an inverse problem stochastically, since : = (c, h) and c is not invertible (c E Rdxmm m < d). Hence, there are several embeddings h that could have generated the observed 2c. We will use nonparametric sampling ideas achieve this, stressing that it suffices to sample the embeddings H and not an entire input image X since the classifier directly acts on h and the encoder f is deterministic. Finally, we remark that Hb,.s differs from the XRT null hypothesis in that conditioning is performed in the space of semantic concepts instead of the input's.
With these precise notions of semantic importance, we now show how to test for each one of them with principles of sequential kernelized independence testing (SKIT) [51].
3.2 Testing by Betting
classical approach hypothesis testing consists of formulating null hypothesis Ho, collecting data, and then summarizing evidence by means of p-value. Under the null, the probability of observing a small p-value small. Thus, for a significance level a E (0, 1), we can reject Ho p S a. In this setting, all data is collected first, and then processed later (i.e., offline).
Alternatively, one can instantiate a game between a bettor and nature [59, 60]. At each turn of the game, the bettor places a wager against Ho. and then nature reveals the truth. If the bettor wins, they will increase their wealth, otherwise lose some. More formally, and as is commonly done [51, 58, 62]. we define a wealth process (K, her, with Ko- = 1 and K, = K₁-1 - (1 + UyKs) where Ut, Kr [-1, 1] are betting fraction and the payoff of the bet, respectively. It is now easy to see that when 02 2 0 (i.e., the bettor wins) the wealth increases, and the opposite otherwise. If the payoff' Ne guarantees the game fair, i.e. the bettor cannot accumulate wealth under the null, then we can use the wealth process to reject Ho with Type I error control (details in Appendix A). In particular, for a significance level a E (0, 1), we denote 5 = minft 2 K, 2 } the rejection time of Hu-
The choice of using sequential testing is motivated by two fundamental properties. First, sequential tests are adaptive to the hardness of the problem, sometimes provably [62, Proposition 3]. That is, the harder it is to reject the null, the longer the test will take, and vice versa. This naturally induces a rank of importance across concepts-it concept ey rejects faster than ey,, then cy more important (i.e., it is easier to reject the null hypothesis that the predictions do not depend on cÿ). We stress that this is not always possible by means of p-values because they do not measure effect sizes: consider two concepts that reject their respective nulls at the same significance level; one cannot distinguish which if any is more important. As we will show in our experiments, all tests used in this work are adaptive in practice, but statistical guarantees on their rejection times are currently open questions, and we consider them as future work. Second, sequential tests are sample-efficien because they only analyze the data is needed to reject, which is especially important for conditional randomization tests. the offline scenario, we would have to resample the entire dataset several times (which is expensive), but the sequential test would terminate in at most the size of the dataset [24].
5

---

Algorithm Level-a C-SKIT for concept j
Algorithm 2 Level-a x- X-SKIT for concept j
Input: Stream (Yr10, 2//0  Pyz,z-5'
Input: Observation 2, subset _ [m] \ {j}.
1: Ko +1
1: Ko f- 11
2: Initialize ONS strategy (Algorithm A.1)
Initialize ONS strategy (Algorithm A.1)
3: for = 1,... do
3: for t 1,... do
4: Compute P: in Eq. (4)
4: Compute Pt. as in Eq. (5)
5: D = (4(1) / -
5: Sample Sug
~ "H|[ss(c)==sutC
6: Sample // ~ P 7, 2z1Z.,==7'
6: Sample H (1) PH 7g
7: y-(1)
SJCY + ((H sug))
(1( t g
7:
D - (), /0 2 =
8
8: Ki Lanh (uD) pe (
8:
Ke + tanh(pr (D()) - Pr (D()))
9:
k - Ki-1 (1 v)Ki)
9: Kg t K₁-1 (1 0214)
10:
KG 2 1/a then
10: if K, 1/a then
11: return
11: return /
12: end if
12: end if
13: U++11 ONS step
13: U++1 + ONS step
14:
end for
14: end for
3.3 Testing Global Semantic Importance with SKIT
Podkopaev et al. [51] show how to design sequential kernelized tests of independence (i.e., Ho A B) by framing them as partieular two-sample tests of the form Hg : P P. with P p= PAB and P= PA PB. Similarly to [58, 62], they propose leverage simple yet powerful observation about the symmetry of the data under Hu [51, Section 4]. We state here the main result we will use in this paper (the proof included in Appendix A.2).
Lemma 2 (See [51, 58, 62]). VI. 2 1, let X ~ P and ~ P, and let Pt X - R be any fixed real-valued function on X. Then, K2 tanh(p:(X) (x)))) provides a fair game for Ho : P P.
That is, Lemma 2 prescribes how to construct valid payoff' for two-samples tests and, consequently, tests of independence. We note that the choice of tanh provides Ae E [-1, but any arbitrary anti-symmetric function can be used sign). Furthermore, any fixed function Pt is valid but, in general, this function should have a positive value under the alternative in order for the bettor to increase their wealth and the testing procedure have good power.
Going back to the problem studied in this work, note that the global semantic importance null hypothesis 0.j in Definition 1 can be directly rewritten as a two-sample test, i.e. HG. Y. is equivalent to H6. Pyz, Py Pz.. We follow [51] and use the maximum mean discrepancy (MMD) [29] to measure the distance between the joint and the product of marginals. In particular, let Ryy Rz, be two reproducing kemel Hilbert spaces (RKHSs) the domains of Y and Zj, respectively (recall that Ý and Zj are univariate). Then, Dلا is the plug-in estimate of the witness function of MMD(PyZ, Py x P2,) at time t.? We include the SKIT algorithm and technical details on computing SKIT and A-SKIT in Appendix B.I.
Computational complexity of SKIT. Analogous to the original presentation in Shekhar and Ramdas [62], the computational complexity of Algorithm B.1 is O(72), where 7 is the random rejection time.
We now move on to presenting two novel testing procedures: the conditional randomization SKIT (C-SKIT) for o. and the explanation randomization SKIT (X-SKIT) for HIC H.j.s-
3.4 Testing Global Conditional Semantic Importance with C-SKIT
Analogous to the discussion in the previous section, we rephrase the global conditional null hypothesis
HCS in Definition 2 as two sample test HCS Pyz,z_, = Pyz,z_, Z, P2, In contrast with other kernel-based notions of distance between conditional distributions [49, 63, and akin to the CRT [wi]-we assume we can sample from Pz,1z_,, which allows us to directly estimate
Recall that MMD(PAR, PA x Ps) is the Schmiit Independence Criterion (HSIC) [28].

---

MMD(Pý Py2,2_,' Pyz,z-,) in our testing procedure (we will expand on how to sample from this distribution shortly). Let Rg,,2z; Rz, be three RKHSs on the domains of Y, Zj, and Z-j (i.e., R. E.T,, where m is the number of concepts). Then, at time t, the C-SKIT payoff function is C-SKIT
(t-1) (t-1) - 2
where 1 2Y2,2_, 2-, 2, are the mean embeddings of their respective distributions in Ry Rz, X Rz-,, and & is the tensor product (see Appendix B.2 for technical details). Algorithm summarizes the C-SKIT procedure, which provides Type I error control for Hog as we briefly state in the following proposition (see Appendix C.2 for the proof).
(t-1)
(1-1)
Proposition 1. vt 2 L, let (Y, Z.. Z-,) Z-)) ~ Pyz,z-; and (Y, 2,. Z-j) Prz,z_; Z, Pzziz-3"
Then, == tanh(pp SKTT (Ý, Z-) Pi (Y.-SII(Y Zj, Z-j)) provides a fair game for H
Cannhpp-SKTTC
Computational complexity of C-SKIT. First note that Z-j y is (m 1 dimdeemssia vector (where m is the number of concepts). So, at each step of the test, the evaluation of the kernel associated with Rz=; requires an additional sum over (m)) terms. Furthermore, C-SKIT needs access to samples from Pziz... and we conclude that the computational complexity of Algorithm is OTT, m where Tn represents the cost of the sampler on samples, and it depends on implementation. For example, in the following, we will use non- parammeettiic samplers with T, (n ). Other choices of samplers, such as intaatiinallattoncooeeee may have constant cost (e.g., they are trained once and only used for inference).
3.5 Testing Local Conditional Semantic Importance with X-SKIT
Attentive readers will have noticed that the local conditional semantic null hypothesis Ho,. in Definition 3 is already a two-sample test where the test statistic / is the distribution of the response of the model with the observed amount of concept (i.e., Ysuts) = g H suts) )), and the null distribution P without (i.e., Ys = g(Hs)). Herein, we assume we can sample from He ~ PHIZe=ze for any subset S [m]] i.e. the conditional distribution of dense embeddings with specific concepts, which we will address via nonparametric methods. Then, for an RKHS Ry. the x-SKIT payoff function is
X- SKIT
(1-1) YouEs:
(1-1) Pys
with p(--1) (2-1) YYuts) Ys
mean embeddings of the distributions in Ry. That is, p X-SKI is the plug-in
estimate of the witness function of (YMD() Ys) -etchnical details are in Appendix B.3. Then, the X-KIT testing procedure, which is summarized Algorithm 2, provides Type I error control for S. as the following proposition summarizes (the proof is included in Appendix C.3).
HIC
Proposition 2. Vt 2 1, Kr tanh(pp-Skin X-KIT (Ysu()) -X-I (xx))((('s)) provides fair game for H.
Computational complexity of X-SKIT. Note that Algorithm assumes access to sampler PhIZo=ze: So its computational complexity is O(T,²²), where, similarly to above, Tù is the cost of the sampler. We briefly remark that, for the nonparametrie samplers used in this work, Tū = n² (compared to 7n² for C-SKIT) because we only need to estimate one conditional distribution.
So far, we have presented our tests for one concept at a time, but we are interested testing m 2 1 concepts. In this setting, it is well-known that multiple hypothesis testing requires appropriate corrections to avoid inflated significance levels. We use a result Wang and Ramdas [79] and devise greedy post-processor that guarantees false discovery control [3] (see Appendix A.4).
4 Results
First, we verify that our tests are valid and that they are adaptive to the hardness of their null hypotheses on two synthetic experiments Appendix D. Here, we showcase the flexibility and effectiveness of our framework zero-short image classification across several VL models on three
7

---

Table 1: Summaay of results for each dataset. Metries are reported as average across all VL models used in the experiments. See main text for details about the models and the metrics used.
Original model
Imagenette
AwA2
CUB
Method
Accuracy
Rank agreement
Accuracy
Rank agreement
f
Accuracy
Rank agreement
f1
SKIT C-SKIT
0.51
0.50
0.65
0.82
0.93
98.99%
0.54
99.50%
0,46
0.57
89.52%
X-SKIT
0.59
PCBM
95.85%
0.45
95.11%%
0.36
0.53
giant panda
tiger
glant panda
diger
plankton
flerce (0)
Black
hurter (0)
wh000 101
strong (9)
white (6)
hunter (0)
pawd
stripes (0)
biue (4)
(1)
blocda)
vegetaatio 101
b bipedal:0 (m)
rurry
meast
flope's
big
musrie (1)
importono
patemes [p]
virain teach
importance
halifioss
most tocth
meet (21
old world (0)
planiton (0]
world
walks
old world
Passess
plankton 10)
mcot to 101
yollow (0)
flopors (2)
5kg (0)
bulboux
Bpppers
back
desort 10)
hops
hores
ume (m]
hocves
skimmer
patches (0)
strain tooth 121
ned [n]
Iurnels
malts
tunks (6)
planitton (3)
desert (4)
hops (4]
1.0
(a) Global importance with SKIT.
(b) Global conditional importance with C-SKIT.
Figure 2: Importance results with CLIP: on 2 classes in the AwA2 dataset. Concepts are annotated with (p) if they are present in the class, or with (a) otherwise.
real-world datasets: Animal with Attributes 2 (AwA2) [82]. CUB-200-2011 I (CUB) [77]. and the Imagenette subset of ImageNet [22].². We compare performance and transferability of the ranks of importance provided by each method across 8 VI. models (see Appendix E for details) and, for all experiments, f is the image encoder of the model and g is the (linear) zero-shot classifier constructed by encoding "A photo of a <CLASS_ NAME> with the text encoder. Herein, we will always use RBF kernels to compute payoffs, and we repeat each test 100 times on independent draws of 7 samples to estimate each concept's expected rejection time and expected rejection rate at a significance level of a = 0.05 with the FDR post-processor described in Appendix A.4. That is, a (normalized) rejection time of 1 means failing to reject in maxx steps. Finally, recall that C-SKIT and X-SKIT need access to Pž,ız,, and PHi Zc=ze and that these distributions are not known in general. Since m is small, we use nonparametric methods estimate them (see Appendix F.
Table 1 summarizes the results of all experiments, which we now present and discuss individually.
4.1
AwA2 Dataset
Given the presence of global (i.e., class-level) annotations, we use SKIT and C-SKIT to test the global (and global conditional) semantic importance structure of the predictions for the top-10 best classified animal categories across all models (we describe the dataset, the concepts used, and the hyperparameters of the tests in Appendix E.2). We classify the top-10 concepts reported by each method as important, and we compute the fi score with the ground-trute annotations. We briefly remark that this choice is informed by the fact that most concepts have rejection rates larger than the significance level of a. When comparing with PCBM-sine. we use different concepts for each class- we train 100 independent linear models for each class, and we rank concepts based on their average absolute weights (instead of signed ones) because the null hypotheses presented in this work are two-sided, i.e. a concept is important both if increases the prediction for class if decreases it. Table 1 shows that both SKIT and C-SKIT outperform PCBM across all three metrics, with SKIT providing the best average rank agreement across different models and importance Si score (0.50 and 0.65, respectively). The fact that ranks provided by our tests have higher average agreement compared to PCBM suggests that VL models may share a similar semantic independence structure notwithstanding g their embedding size training strategy, i.c. semantic importance may be transferable across models (all individual pairwise agreements are included in Fig. E.4).
Finally, Fig, 2 shows ranks of importance with CLIP:ViT-L/14 on 2 animal categories (see Figs. E.5 and E.6 for all classes). In general, concepts are globally important (rejection rates are greater than a),
Coode reproduce all experiments is available at https h//gt www.mm/uamm-rruuppppcpm IBYDMTT.
8

---

joy
ka fors
spobedit 31 go
hopeed suabes (2)
us unde parta (o
Mau under
adid wing
bladi unde parts
onay underaant
sbou medium R 14 00
Back
yelox eye
insesrent undera
mutt-ccioned shs
blace 10
oragg
browrt
greas
indesgeer traat
(eo
1
(2rk cock-BB (4)
rounsed call tal (4)
poitted ca
Figure 3: Importance results with X-SKIT and VIT-L/I1 on 4 images in the CUB dataset. Concepts with (p) are present in the image according human annotations, and (a) otherwise.
and it is harder reject the global conditional null hypothesis (rejection rates are lower and rejection times larger), naturally reflecting the fact that conditional independence is a stronger condition.
4.2 CUB Dataset
This dataset (differently from AwA2) provides per-image annotations of semantic attributes. So, we use X-SKIT to test the semantic importance structure of VL models locally on particular images and validate its performance against such annotations (we include details about this experiment and extended results in Appendix E.3). The purpose of this experiment is to validate the performance of X-SKIT, hence we use the ground-truth binary semantic annotations as an oracle instead of predicting the presence of concepts. In practical scenarios where ground-truth is not available, one could-as done by previous works [12]- use LLMs to answer binary questions (e.g., "Does this bird have an orange bill? Yes/No"). Furthermore, note that for each concept j E [m]] there are exponentially many tests with null hypothesis Hb - one for each subset S C [m]] \ {)) (i)-which are intractable to compute. Thus, we report average results over 100 tests with random subsets with fixed size s.
Fig. 3 depicts prototypical results with CLIP:ViT-L/14 (Fig. E.9 includes results for all models on the same images). After running X-SKIT, we classify concepts as important by thresholding their rejection rates at level w-which is a statistically-vatid way of selecting important concepts. Results are included in Table 1, and we conclude that X-SKIT provides ranks of importance that are well well-aligned both across models (0.82 rank agreement) and with ground annotations (f1 score of 0.93). We remark that x- X-SKIT is the first method to provide local semantic explanations, hence why we cannot compare with alternatives.
4.3 Imagenette Dataset
Lastly, we use both SKIT, C-SKIT, and X-SKIT on the Imagenette subset of ImageNet [2ic- does not provide ground-truth semantic annotations to evaluate performance with. So, we use SpLiCe [6] to select which concepts to test (see Appendix E.4 for details), but we stress that any user-defined set of concepts would be valid-a unique feature of our proposed framework.
Figs. 4a and 4b show SKIT and C-SKIT results with CLIP:ViT-L/14 on classes in the dataset (Fig. E.11 includes all classes). We use SpLiCe to encode the entire dataset and test the top-20 concepts. Analogous to the experiment on AwA2, we can see that rejection rates are lower for C-SKIT (i.e., conditional dependence) compared to SKIT (i.e., marginal dependence). We evaluate rank agreement across all models and compare with PCBM in Table 1. These results confirm that not only are the ranks produced by our tests more transferable across models (rank agreement of 0.51 for SKIT, 0.54 or C-SKIT, 0.59 for X-SKIT, and 0.45 for PCBM), but also they retain the performance of the original classifier (98.99% for our methods vs 95.55% for PCBM). We refer interested readers to Fig. E. for all pairwise comparisons. Furthermore, we qualitatively study the stability of our tests as function of max in Fig. E. This is important because m represents the sample complexity of the tests. Our findings indicate that important concepts tend to exhibit greater stability in their ranks compared to less important ones, with SKIT showing overall more stability than C-SKIT.
To conclude, Fig. 4c shows X-SKIT results with VLIP:ViT-L/14 on three random images from the dataset (see Figs. 15 and 16 for all models and more images). We use SpLiCe to encode each image and obtain its top- 10 concepts, and then add the bottom-4 according to PCBM, for a total of 14 attributes per image. The choice of combining concepts both from SpLiCe and PCBM will highlight

---

tench
English springer
tench
English springer
battered
spariel
fish
Spartel
fishing
1019
bass
band
fish
sellin
brass
brat5
spaniel
Instrument
fishing
5955
bross
radio
jaz2
indtrument
obsolete
J822
battered
trumpet
importance
Instrument
putting
Importance
instrument
obsolete
exterior
obsolere
obsolete
dispenser
outting
fish
spaniel
fishing
cathedral
fishing
dispenser
fore
airshow
disporser
exterior
exterior
selling
Bew
hadredial
radio
racio
trampet
radio
flw airshow
fore
exterior
flww
dispenser
cothedral
airshow
band
cathedral
Ja22
brass
selling
selling
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
100
0.0
0.5
10
(a) Global importance with SKIT.
(b) Global conditional importance with C-SKIT.
opanial
trurper
Inunch
English springer
conductidd
nadmmingg
netbing
navel
chop
0N ky
bass
burge
par
(1]
Instrument Seatboce
cathedral
po
(c) Local importance with X-SKIT. The bottom-4 concepts according to PCBM are annotated with (*).
Figure 4: Results with CLIP.ViT-L/I4 on Imagenette.
the differences between these methods and our notion of local statistical importance, as will shortly expand on. Recall that we use nonparametric samplers to approximate PHI Zc=ee so the cost of using image-specific concepts boils down to projecting the feature embeddings with different matrix c, which is negligible compared to running the tests. We note that parametric generative models- such as variational autoencoders or diffusion models would have required retraining for each set of concepts, which is expensive. Overall, we find that ranks are well-aligned across models (0.71 rank agreement, see Table 1). We can appreciate how the bottom-4 concepts from PCBM, which are annotated with an asterisk, are not always last, i.e. concept may be locally important even if it is not globally important. For example, concept "fishing" may not be globally important for class "English springer", but it is locally important for an image of a dog in water. Conversely, a concept having a high weight according to SpLiCe does not imply will be statistically important for the predictions of the model, and these distinctions are important in order to communicate findings transparently.
Conclusions
There exist an increasing interest in explaining modern, unintelligible predictors and, in particular, doing so with inherently interpretable concepts that convey specific meaning to users. This work the first to formalize precise statistical notions of semantic importance in terms of global (i.e.. over a population) and local (i.e., on a sample) conditional hypothesis testing. We propose novel, valid tests for each notion of importance while providing rank of importance by deploying ideas of sequential testing. Importantly, by approaching importance via conditional independence (and by developing appropriate valid tests), we are able to provide Type I error and FDR control, a feature that is unique to our framework compared to existing alternatives. Furthermore, our tests allow to conlinear-classifie that would used in the wild, as opposed to
explain the training surrogate linear models as has been the standard so far.
Naturally, our work has limitations. First and foremost, the procedures introduced in this work require access to samplers, and there might be settings were learning these models is hard; we used nonparametric estimators in our experiments, but modern generative models could be employed, too. Second, kernel-based tests rely on the assumption that the kernels used are characteristic for the space of distributions considered. Although these assumptions are usually satisfied in IRd for RBF kernels, there may exist data modalities where this is not true (e.g., discrete data, graphs), which would compromise the power of the test. Finally, although we grant full flexibility to users to specify the concepts they care about, there is no guarantee that these are well-represented in the feature space of the model, nor that they are the most informative ones for a specific task. All these points are a matter of ongoing and future work.
10

---

Acknowledgments
We sincerely thank Zhenzhen Wang and the anonymous NeurIPS reviewers for useful conversations that strengthened the presentation of our experimental results. This research was supported by NSF CAREER Award CCF 2239787.
References
[1] Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American mathemati- cal society, 68(3)337--444 1950.
[2] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissec- tion: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6541-6549, 2017.
[3] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), (711):89-300, 1995.
[4] Alain Berlinet and Christine Thomas- Reproducing kemnel Hilbert spaces in probability and statistics. Springer Science & Business Media, 2011.
[5] Thomas B Berrett, Yi Wang. Rina Foygel Barber, and Richard Samworth. The conditional permutation test for independence while controlling for confounders. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(1):175-197, 2020.
[6] Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P Calmon, and Himabindu Lakkaraju. In- terpreting clip with sparse linear concept embeddings (splice). arXiv preprint 10376 2024.
[7] Beepul Bharti, Paul Yi, and Jeremias Sulam. Sufficient and necessary explanations (and what lies in between). arXiv preprint arXiv:2409.20422 2024.
[8] Gilles Blanchard and Etienne Roquain. Two simple sufficient conditions for fdr control. Electronic Journal of Statistics, 2:963- 992, 2008.
[9] Carlo Bonferroni. Teoria statistica delle classi calcolo delle probabilita. Pubblicazioni del istituto superiore di scienze economiche commericiali di firenze, 8:3-62, 1936.
[10] Collin Burns, Jesse Thomason, and Wesley Tansey. Interpreting black box models via hypothesis testing. In Proceedings of the 2020 ACM-IMS on foundations of data science conference, pages 47-57, 2020.
[11] Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for gold: model- x l'knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(3):551-57) 2018.
[12] Aditya Chattopadhyay, Kwan Ho Ryan Chan, and Rene Vidal. Bootstrapping variational information pursuit with large language and vision models for interpretable image classification. In The Twelfth International Conference on Learning Representations, 2024.
[13] Aditya Chattopadhyay, Ryan Pilgrim, and Rene Vidal. Information maximization perspective of orthogonal matching pursuit with applications to explainable ai. Advances in Neural Information Processing Systems. 36, 2024.
[14] Haozhe Chen, Junfeng Yang, Carl Vondrick, and Chengzhi Mao. Invite: Interpret and control vision-language models with text explanations. In The Twelfth International Conference on Learning Representations, 2024.
[15] Jianbo Chen, Le Song, Martin J Wainwright, and Michael Jordan. L-shapley and c-shapley: Efficient model interpretation for structured data. arXiv preprint arXiv: 0011180888011... 2018.

---

[16] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-iman learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 2818 2829, 2023.
[17] Thomas M Cover. Universal portfolios. Mathematical finance, 1(1):1-29, 1991.
[18] Ian Covert, Scott M Lundberg, and Su-In Lee. Understanding global feature contributions with additive importance measures. Advances in Neural Information Processing Systems, 33: 17212-17223, 2020.
[19] Ashok Cutkosky and Francesco Orabona. Black-box reductions for parameter-free online learning in banach spaces. In Conference On Learning Theory, pages 1493-1529, PMLR, 2018.
[20] Roxana Daneshjou, Mert Yuksekgonul, Zhuo Ran Cai, Roberto Novoa, and James Zou. Skincon: A skin disease dataset densely annotated by domain experts for fine- fine-grained debugging and analysis. Advances Neural Information Processing Systems, 35:18157-18111 2022.
[21] Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. A survey of the state of explainable ai for natural language processing. arXiv preprint arXiv:2010.00711, 2020..
[22] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages
Iece, 2009.
[23] Christiane Fellbaum. WordNet: electronic lexical database. MIT press, 1998.
[24] Lasse Fischer and Aaditya Ramdas. Sequential permutation testing by betting. arXiv preprint arXiv:2401.07365, 2024.
[25] Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. Kernel measures of conditional dependence. Advances in neural information processing systems, 20, 2007.
[26] Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image representa- tion via text-based decomposition. arXiv preprint arXiv:2310.05916, 2023.
[27] Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median heuristic. arXiv preprint arXiv:1707.07265 2017.
[28] Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Schölkopf. and Alex Smola. kernel statistical test of independence. Advances in neural information processing systems, 20, 2007.
[29] Arthur Gretton, Karsten Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[31] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong. John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip. July 2021. URL httpp:///det org/ 10.5281/ zenodo. 5143773. If you use this software, please cite it as below.
[32] Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904 4916. PMLR, 2021.
[33] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2)811995 1938.
12

---

[34] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pages 2668 2677. PMLR, 2018.
[35] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 2014.
[36] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning. pages 5338 -5388 PMLR, 2020.
[37] Stefan Kolek, Aditya Chattopadhyay, Kwan Ho Ryan Chan, Hector Andrade-Loarea, Gitta Kutyniok, and René Vidal. Learning interpretable queries for explainable image classification with information pursuit. arXiv preprint arXiv:2312. 1 2023.
[38] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))2778223 1998.
[39] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900. PMLR, 2022.
[40] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[41] Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. Explainable ai: A review of machine learning interpretability methods. Entropy, 23(1):18, 2020.
[42] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[43] Jan Macdonald, Stephan Wäldchen, Sascha Hauch, and Gitta Kutyniok. A rate-distortion framework for explaining neural network decisions. arXiv preprint 1001 2019.
[44] Ricardo Moreira, Jacopo Bono, Mário Cardoso, Pedro Saleiro, Mário AT Figueiredo, and Pedro Bizarro. Diconstruct: Causal concept-based explanations through black-box distillation. arXiv preprint arXiv:2401.08533 2024.
[45] Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in applied probability, 29(2):429 443, 1997.
[46] Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine one-grained image recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14933-14943, 2021.
[47] Tuomas Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron repre- sentations in deep vision networks. arXiv preprint arXiv:2204.10665, 2022.
[48] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. arXiv preprint arXiv:2304.06/29, 2023.
[49] Junhyung Park and Krikamol Muandet. A measure-theoreti approach to kernel conditional mean embeddings. Advances in neural information processing systems, 33:21247-21259, 2020.
[50] Thang M Pham, Peijie Chen, Tin Nguyen, Seunghyun Yoon, Trung Bui, and Anh Totti Nguyen. Peeb: Part-based image classifiers with explainable and editable language bottleneck. arXiv preprins arXiv:2403.0522 2024.
[51] Aleksandr Podkopaev, Patrick Blöbaum, Shiva Kasiviswanathan, and Aaditya Ramdas. Sequen- tial kernelized independence testing. International Conference on Machine Learning, pages 27957-27993. PMLR. 2023.
13

---

[52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages
PMLR, 2021.
[53] Vikram Ramaswamy, Sunnie SY Kim, Ruth Fong, and Olga Russakovsky. Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11932-109411 2023.
[54] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1 1135-1144, 2016.
[55] Numair Sani, Daniel Malinsky, and Ilya Shpitser. Explaining the behavior of black-box prediction algorithms with causal learning, arXiv preprint arXiv:2006.02482, 2020.
[56] David W Scott. Multivariate density estimation: theory, practice, and visualization, John Wiley & Sons, 2015.
[57] Ramprasaath Selvaraju. Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
(58) Shalev Shaer, Gal Maman, and Yaniv Romano. Model-x sequential testing for conditional independence via testing by betting. In International Conference on Artificial Intelligence and Statistics, pages 2054-2086. PMLR, 2023.
[59] Glenn Shafer. Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society Series A: Statistics in Society. 184()):07-433 2021.
[60] Glenn Shafer and Vladimir Vovk. Game-theoretic foundations for probability and finance, volume 455. John Wiley & Sons, 2019.
[61] John Shawe-Taylor and Nello Cristianini. Kernel methods for pattern analysis. Cambridge university press, 2004.
[62] Shubhanshu Shekhar and Aaditya Ramdas. Nonparametric two-sample testing by betting, IEEE Transactions on Information Theory, 2023.
[63] Tianhong Sheng and Bharath K Sriperumbudur. On distance and kernel measures of conditional dependence. Journal of Machine Learning Reseurch, 24(7):1-16, 2023.
[64] Pushkar Shukla, Sushil Bharati, and Matthew Turk. Cavli-using image associations to produce local concept- based explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3749-3754, -754, 2023.
[65] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 15638-15650, 2022.
[66] Le Song, Jonathan Huang, Alex Smola, and Kenji Fukumizu. Hilbert space embeddings of conditional distributions with applications to dynamical systems. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 961-968, 2009.
[67] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 133466, 2020.
[68] Bharath K Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert RG Lanckriet. Hilbert space embeddings and metrics on probability measures. The Journal of Machine Learning Research, 11:1517-1561, 2010.
14

---

[69] Wesley Tansey, Victor Veitch, Haoran Zhang, Raul Rabadan, and David M Blei. The holdout randomization test for feature selection in black box models. Journal of Computational and Graphical Statistics, (1(1):151-12, 2022.
[70] Jacopo Teneggi, Alexandre Luster, and Jeremias Sulam. Fast hierarchical games for image explanations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4494- 4503, 2022.
[71] Jacopo Teneggi, Beepul Bharti, Yaniv Romano, and Jeremias Sulam. Shap-xrt: The shapley value meets conditional independence testing. Transactions on Machine Learning Research, 2023.
[72] Isabella Verdinelli and Larry Wasserman. Feature importance: A closer look at shapley values and loco. arXiv preprint 20Xhv:2303.05981,,220
[73] Isabella Verdinelli and I Larry Wasserman. Decorrelated variable importance. Journal of Machine Learning Research, 25(7):1-27, 2024.
[74] Sebastiano Vigna. A weighted correlation index for rankings with ties. In Proceedings of the 24th international conference on World Wide Web, pages I 1176-117, 2015.
[75] Jean Ville. Etude critique de la notion de collectif. Gauthier- Paris, 1939.
[76] Vladimir Vovk and Ruodu Wang. E-values: Calibration, combination and applications. The Annals of Statistics, 49(3): 1736-114,, 2021.
[77] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 20rds-200-2011 dataset. 2011.
[78] Benyou Wang, Qianqian Xie, Jiahuan Pei, Zhihong Chen, Prayag Tiwari, Zhao Li, and Jie Fu. Pre-trained language models in biomedical domain: A systematic survey. ACM Computing Surveys, 56(3): 1-52, 2023.
[79] Ruodu Wang and Aaditya Ramdas. False discovery rate control with e-values, Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):822-852, 2022.
[80] Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:22 10..2233,, 2022.
[81] Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: Concept- aware mitigation of spurious correlation. In International Conference on Machine Learning. pages 37765-37786. PMLR, 2023.
[82] Yongqin Xian, Christoph H Lampert, Bernt Schicle, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):225- 2018.
[83] Yue Yang. Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19187-19197,200 2023.
[84] Hao Yuan, Haiyang Yu, Shurui Gui, and Shuiwang Ji. Explainability in graph neural networks: taxonomic survey. IEEE transactions on pattern analysis and machine intelligence, 45(5): 5782-5799, 2022.
[85] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. arXiv preprint arXiv: 15480, 2022.
[86] Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. arXiv preprint arXiv: 2012.
15

---

A Testing by Betting
In this appendix, we include additional background information on testing by betting that was omitted from the main text for the sake of conciseness of presentation. Recall that the wealth process (Ke) with No = NU {0} is defined as
Ko 1 and
K; Ki-1 (1 UgK4)
(6)
where 22 [-1, 1] the betting fraction and K₁ E [-1, 1] the payoff of the bet.
A.1 Test Martingales
We start by introducing the definition of test martingale (see, for example, Shaer et al. [58])
Definition A.I (Test martingale). A nonnegative stochastic process (St)teN a test martingale So = 1 and, under null hypothesis Ho, it is supermartingale, i.c.
Eh [S: Fi-1] S St-1,
where Fi-1 the filtration (i.e., the smallest o-algebra) all previous observations.
In the following, we will use Ville's inequality, which we include for the sake of completeness. Lemma (Ville's inequality [75]). If the stochastic process {St) is a nonnegative super- martingale,
P P 2 0: S. m) E/Sol/n, Vn > 0.
With this, we state condition under which we can use the wealth process to reject null hypothesis Ho with Type eτror control.
Lemma A.2 (See Shaer et al. [58]. Shekhar and Ramdas [62]). If
Eig [ive F,-1] =0,
where Fi-1 denotes the filtration (i.e., the smallest o alagebrra) of all previous observations, then the wealth process [3,) Jett describes a fair game and
PHO Bi 21: K, 2 1/a| s//a/oo Sa.
(10)
Proof. It suffices to show that if EH, [Kg [i-1] = 0, then the wealth process {KiheN, is a test martingale:
1. Ko 1 by definition, and
2. It is immediate to see that the wealth process is nonnegative because Uy [-1, and the bettor never risks more than their current wealth, i.c. they will never go into debt. Finally,
3. If En, [Kz. Fr = 0, then
EH, [K,] F(-1] = EHoo _K-11 (1 + veRee) F(-1]
(11)
K₁-1 EHo [1 + vetie F(-1]
(Kt-1 Fi-1 is constant)
(12)
< K,- (1 EHo Iee F(-1))
(v. S 1)
(13)
Kr-1,
(14)
and the wealth process is supermartingale under the null.
Then, by Ville's inequality, we conclude that for any significance level a E (0,1)
Pt. 1: K, > 1/a] < aE =
(15)
which is the statement of the lemma.
16

---

A.2 Symmetry-based Two-sample Sequential Testing
In this section, we show how leverage symmetry to construct valid sequential tests for two-sample null hypothesis of the form
Ho P-P.
(16)
Lemma A.3 (See [51, 58, 62)). Vt 2 1, let X P and X P be two random variables sampled from P and P. respectively. If P P, it holds that for any fixed function pr. X - IR
Px( px( _ px(X) - Px(X),
(17)
that is
(X) po(X)) = polps Px()))
(18)
where Po is the probability density function induced by Ho-
Proof. The proof straightforward. If P P., then and X are exchangeable by assumption. Proof of Lemma
Recall the lemma states that for any fixed function Pr X + R. the payoff
Kg tanh(p:(X) px()))
(19)
provides a fair game (i.e., it satisfies Lemma A.2) for two-sample test with null hypothesis
Ho P= P. We use Lemma A.3 above to prove a stronger result that implies the desired claim.
Lemma A.4 (See [51, 58, 621). For any 2 and any fixed syti-symmetric function { + R. holds that
EH. (X) - (x())) Ft-1] 0.
(20)
Proof. We can see that
E (X) px(x)) F(-1] = EH. (x(x(() (x())
(pe. & are fixed)
(21)
{(u)po(u) du
(change of variables)
(22)
S (E(u) +E(-u))po(u) du
(by Lemma A.3)
(23)
[ (E(u) E(u))po(u) du
(g is anti-symmetric)
(24)
B+
0,
(25)
which concludes the proof.
Proof of Lemma 2. Note that tanh an anti-symmetric function, so Lemma A.4 holds. Then,
Lemma A.2 implies that n4 (x)) -x(X)) provides test martingale for Hg P = P. C
A.3 Betting Strategies
So far, have discussed how construct valid test martingales in terms of the payoff K2- Then, it remains to define a strategy to choose the betting fraction V. In general, any method that picks ey before data is revealed maintains validity of the test, and briefly summarize few alternatives.
Constant betting fraction. Naturally, a fixed betting fraction v, = is valid. However, this strategy may be prone to overshooting, i.e. the wealth may go to zero almost surely under the alternative, and severely impact the power of the test [51, Example 2].
Mixture method [17, 58]. A possible way to overcome the limitations of setting a fixed betting fraction is average across distribution, i.c.
K₁ = /( K: p()) de,
p(c)
(26)
17

---

where K, is the wealth with constant betting fraction v2 and p(v) is prior over the choice of fractions (c.g.. uniform over [-1, 1).. This choice is valid, and motivated by the intuition that the mixture martingale will be driven by the term that achieves the optimal betting fraction [58, Theorem 1].
Online Newton step (ONS) [19]. Alternatively, one can frame choosing the betting fraction as an online optimization problem that finds the optimal 82 in terms of the regret of the strategy. We refer interested readers to [19, 58, 62] for theoretical analysis of this strategy and simply state here the wealth's 's growth rate. Algorithm A.I summarizes this strategy.
Lemma A.5 (See Shekhar and Ramdas [62]). For any sequence {ve E 1-1,11 2 1J, it holds that
log Ki
Σ
logt.
(27)
8t
Algorithm A.1 ONS Betting Strategy
Input: Sequence of payoffs {K1221
I: dg
2: e1 +
fort 1 do
4: + Kt/ + v(
5: a, a- 2f
6: e++1 max(0, min(1, 2/(2 - log(3)) =//oz)
7: end for
A.4 Controlling False Discovery Rate
Finally, we briefly present one way to provide false discovery rate (FDR) control when testing multiple hypotheses with sequential tests. Given m null hypotheses H ((m), denote e(!), e(!),... emm)
their respective e-values [60, 76] and let [0, xo)" - 2(ml be an e-testing procedure such that E(e(), ,..., e(m)) is the set of rejected null hypotheses. Then, FDR the expected proportion of false discoveries to the number of total findings, i.e.
S n Sol
FDR
(28)
|S|
where So = [ji [m] H is true} } is the set of true null hypotheses (i.e., the ones that should not be rejected). Following [8, 79], we say that & is self-consistent level if every rejected e-value satisfies e(d) 2 m/a[S), and we now state the lemma we use to construct our FDR post-processor in Algorithm A.2.
Lemma A.6 (See Wang and Ramdas (79]). Any self-consistemt e-testing procedure at level a controls FDR at level for arbitrary configurations of e-values.
Algorithm A.2 Level-a greedy FDR post-processor.
Input: Wealth processes (K(") ) ,...., (K! (m) J. 1 No-
1: š x- 0
2: for = 1..... m do
3: j', +
arg min jemm] re(00.0)
T s.t. K(() 2 m/as
4: if r' oo then
5: return S
6: end if
7:
(-SUU')
8: end for
9: return
18

---

Recall that the optional stopping theorem implies that for a test martingale {K₃ Hig,, the wealth K an e-value for any t 2 1. Then, intuitively, Algorithm A.2 transforms an e-testing procedure into a self-consistent one by greedily rejecting concepts as soon as they cross the adjusted threshold m/a[S]. Note that we do not know the number of rejections priori, and that m/a[S] is decreasing function of (S]. Hence, the adjusted threshold for the first concept will be m/a (which matches the common Bonferroni correction [9]), m/2a for the second one, then m/3a, and so on and so forth. The procedure stops when no more concepts reach the threshold, and concepts are sorted by their adjusted rejection times. We remark that Algorithm A.2 runs in (m)) time, and that it does not change the individual testing procedures- which important because concepts are tested concurrently in practice.
B Technical Details on Payoff Functions
In this appendix, we include technical details on how to compute the payoff functions of all tests presented in this paper. We start with brief overview of the maximum mean discrepancy (MMD) [29], and we refer interested readers to [1, 4, 61, 68] for rigorous introductions to the theory of reproducing kernel Hilbert spaces (RKHSs) and their applications to probability and statistics.
Definition B.1 (Mean embedding (see Gretton et al. (29]). Let P be a probability distribution on X and R an RKHS on the same domain. The mean embedding of P in R the element p. R with
vp E R. Bpla(X)] = (pP,
(29)
Furthermore, given ,..,..., (()) sampled i...d. from P, the plug-in estimate Pr is
(n))
(f - Σ x()
(30)
12
where po is the canonical feature map, i.c. (xX) k(X,-), and k is the kernel associated with R.
We now define the MMD between two probability distributions P. Q and show that it can be rewritten in terms of their mean embeddings.
Definition B.2 (Integral probability metric (see Müller [45])). Let P, Q be two probability distribu- tions over X. Furthermore, denote G {g- X' + R} hypothesis class of real-valued functions over X. Then,
Dp(P.Q) sup Erlg(X) Eq (xx]] gEg
(31)
the distance between P and Q induced by 9, and the function g" that achieves the supremum called witness function.
The MMD is defined as DB(R) (P. Q), where B(R) is the unit ball of R, i.e.
B(R)= { R: lll|R S 13.
(32)
Definition B.3 (Maximum mean discrepancy (see Gretton et al. (29])). For P, Q defined as above, let R be an RKHS on their domain. Then,
MMD(P,Q) = sup Ep[p(X)] - Exlp(X)].
(33)
PEB(R)
We note that we drop the absolute value because if p E B(R), then -p E B(R) also. From the definition of mean embedding, it follows that
MMD(P,
sup (uP, (uq, PtR PEB(R)
(34)
sup PEB(R)
(pP - m2,0)
(35)
I|up nell
(36)
and its witness function satisfies
0º pp PQQ
(37)
19

---

Algorithm B.1 Level-ox SKIT for the global importance of concept j
Input: Stream (810, ( P
1: Ko f-1 1
2: Initialize ONS strategy as in Algorithm A.I.
3: fort === 1.... do
4: Compute P: as in Eq. (47)
5: Observe D(21-1) (8(2(-1), _ 1) and D(20) = (8(20), 22,20)
6: Di21-1) (Y 2
and 2 (2/(2), 2 2
7: Compute Ne as in Eq. (48)
8: K2 < KG-1 (1 v2Kt))
9:
if K2 2 1/a then
10: return
11: end if
12: 1 ONS step
13:
end for
B.1 Computing I and SKIT
Recall that SKIT is the estimate of the witness function of (M,, Ps x Pz,) at time i.e.
Pt
SKIT
(2(t-1)) Yzz
Mg @ pz,
()(1-)) (2(e-1))
(38)
where
2(t-11 1)
192;
(2(1-13)
2(1 - 1]
1
Σ (p)(((1)) 2 p2, (2, (2
(39)
p==
2(1-1)
2(1- 1) Σ 92, (2)() r'=0
20
pp (P(1)))
(12(1-1)) 22
(40)
t
t t-
t'=0
and Py, are the canonical feature maps associated with Ry and R2,, respectively. We remark that / is an operator, and, for sample (Ú. 22),, its value p.KIT (j. 22) can be computed as
SKIT (ú, z3) = (Ryz) (2(t-11) 1)) - pyy (2()-))) 8 32
(2(t 1)) (0) 23)
(41)
(2((--1)) (0, 23) - (1 (2(--1)) & 122
(2(1-1)) ) 23)
-Py2,
(42)
with
2(1-1)
PYz, 1/2
(21--1)) (0.2,)
Σ
kip (t(1), k2, (2)(),2))
(43)
2(t 1)
t'=0
2((-1)
2(t-1)
(n ((t-))))) 2 2zz (0) = 1()
Σ
ky ()((),
Σ kzz (2,50) -2g)
(44)
#=0
where ky; kz, are the kernels associated with Ry and R.,, respectively.
Furthermore, note that, in practice, we only have access to samples from the test distribution PAz, (i.e.. the joint) and we swap elements of two consecutive samples to simulate data from the null distribution Ps. Pz,. More formally, let
(( (f(22),7,00) ((21)) PyZ,'
1) = (1122-11) 2.20-113 ~ Pyz,
(45)
such that
5(20 = (8/22),7((2))
1) (2121-1), (,,01)
(46)
20

---

Then.
Pt PYz,
SKIT
(22-11))
(2(t-1)) -Ky X Pz,
(2(1-1))
(47)
where 2y Py- juz are the mean embeddings of Pyz,: py, Pz, in Rÿ 2 Rz,, Ry. and R respectively, and 69 is the tensor product in Gretton al. [29]. We remark that / is real-valued operator, i.e. pakIr IR x - IR, and that we use data up to - to compute Pr in order to maintain validity of the test, i.e. Pt. is fixed conditionally on previous observations.
Following Lemma 2, we conclude
SKIT tanh (p. SKIT (D(2(-1) + P: SKIT (D(2t)) - SKIT (D(2(1)) - SKIT
p (i(20)
(48)
and Algorithm B.1 summarizes the SKIT procedure for the global semantic importance null hypothesis H6, in Definition 1.
B.2
Computing Pt P-SSKI and Ki
C-SKIT
Recall that Pi (K-SKIT is the estimate of the witness function of MMD MMD((22,,.. Pržz,_) with Zy
Pzzız-, at time t, i.c.
P1 PV7,7 192,2 5¹
C-SKIT
(1-1)
(49)
where
(1-1)
Fp (8 20 pz (211))
(50)
17,77
(1-1)
8 pz (2, (2 (p(((()))
oz (24))
(51)
22,22-;
and , (2, P2_, are the canonical feature maps associated with their respective RKHSs. We remark that p.-SKIT is defined as an operator, and, for triplet (0. 2j. x=z) its value can be computed
as
Pt (0,2),2-)) = (1477,2, - P42,2 , (0.2),
C-SKIT
(t-1)
2j. 2.. z-z)
(52)
- (0, ()1)) ( 2,,,,,, (0,
(53)
with
(1-1) 2z (0, (0,2).2-j) 1-1
kp (((1)))))>))) ,u)k (2 ) (2 =)) {==))==
(54)
f'=0
(t-1) 2, (9, 21,
ky (t) k))k (2 2z) 22) z, (2 (=-z)
(55)
where ky, kz, kz,, are the kernels associated with Rÿ, R2,, and Rz.,, respectively.
Following Lemma 2, we conclude
Hr SKIT
tanh(p; SKTI
COTTTý (8, 7,j7) &-SKIT -SKIT(Y', 18,7,,77-11
(56)
B.3 Computing p SKIT and n X-SKIT
Recall that, for particular sample 2, concept j E [m), and subset S C [m]] [ that does not contain j. Pi X-SKIT the estimate at time of the witness function of MMD(Ysug)) Ys) with
Ýc g(flc), He PhIzo=ze.1 i.c.
P
X-SKIT
P9uts "Vs
(t-1)
((-1))
(57)
21

---

where
-
Py su(i)
(P
(-11)
Σ
p (}5))
pp
(58)
Yours
t==0
and by is the canonical feature map of Ry. Then, for a prediction y. the value of p SKIT is
Pt.
SKIT
(0)
(1
Youts)
Pys (0)
(59)
(1-1)
(0)
Pyy (t-1)
(0)
(60)
Ysuts)
ky (P5h) SuG)))
Σ ky ( .0)
(61)
where ky is the kernel associated with Ry. To conclude, applying Lemma 2 implics
X-SKIT = tanh(p, X-SKIT (Ysu(s)) - - -A.-KKT (Ys)).
(62)
C
Proofs
In this appendix, include the proofs of the results presented in this paper.
C.1 Proof of Lemma
Recall that _dxm is dictionary of m concepts such that cj. E [m]] is the vector representation of the 1th concept. Then, Z (c, H) is the vector where after appropriate normalization- -Zj [-1, represents the amount of concept j in h.
We want to show that if y (w, H), w E Rd, and 2 3, then
Y L. + (w, cg) =
(63)
That is, w being orthogonal to does not provide any information about the statistical dependence between y and Zj. and vice versa.
Proof. Herein, for the sake of simplicity, we will drop the c, notation and consider single concept Furthermore, we will assume that all vectors are normalized, i.e. ||w||| ||h|| = ||c]| Note that the Eq. (63) above can directly be rewritten as
(w. Hc, H) +/ (w,c) 0.
(64)
We show there exist random vectors such that (w, c) = O but (w, H) 14 (c, H).
Let H U($"), i.e. H [H....., Ha) is sampled uniformly dimensions. It is easy to see that Vj E [d]. H3 (ej, H). where is the jth element of the standard basis. Furthermore, it holds that H₃ = 1- E H3. by definition. We conclude that v(j.j', let w = and c Cy', then (w, c) but (w, H) 1 H). That is, the fact that ej and ey orthogonal does not mean that their respective projections of are statistically independent.
the sphere in d
(+))
We show how to construct random vector // such that (w, H) H) but (w, c) * O.
Denote H. = (h E Sd (c = 7, r * 0} the linear subspace of unit vectors in Idd with the same nonzero inner product with Each vector h H. can be decomposed into parallel and an orthogonal component to c, i.e. Yh Hyp h lhe h1 = njc + h., where the last equality follows by definition of H.y.
22

---

Consider H, and M--p. it follows that for w, E Hy and h E H-
(w,h") (w. h
= (ne WT ne + hi) (ne + WL: -1i-ne+ hT)
(65)
u1² + (wi,1]) ,h1) =-1 + ( ,
(66)
(Ti, h - -hī) = 221
(67)
(w), A) = -21p.
( = := -
(68)
Denote = ((h+,h)) (w A) = 2n²3 the set of pairs of vector that satisfy Eq. (68),
and note that for each pair (h+,h² h)) there exists value 3 such that (me, l+)) (w, h h- = B.
Then, sampling from S is equivalent to sampling from the set of pairs of vectors in H., and H-. that attain the same correlation with w.
Note that when = 2, ht h E { by construction, hence E f0, +2w and (WI A) E (0, + - n/))}. Then, S because there are no pairs of vectors such that (Wi, A) = 21)² For 3, S is nonempty as long as !j I 11/2.
Then, we can construct H as follows:
Sample the component parallel to i.e. H. ~ u((ne))
- Sample the component orthogonal to c, i.e. (HI, HI) ~ U(S)
and note that by doing so, we have sampled (c, H) and (w, H) independently. Finally
He H1 if He m/c He + HI
H
(69)
if H -Nc
has (w, H) (c, H) by construction, but, since E Hay (w,, c) + O.
This concludes the proof of the lemma.
C.2 Proof of Proposition 1
Recall that Proposition states that the payoff function
C-SKT
(pt (Ý, Zj, Z-;) Pi C-SKIT (Ý, (8,z,,Z-)) Z-j))
(70)
provides a fair game for the global conditional semantic importance null hypothesis 0,J in Defini tion 2. That is, the wealth process provides Type error control.
Proof. Note that Hoy can directly rewritten as
H Pyz,z = Prz,2_,
(71)
where Z Pz, Then, under the null, the triplets (V 7y, able by assumption, and the result follows from Lemma 2.
/,, 7-z) and (Ý, 88,7..2 Z-z) exchange-
C.3 Proof of Proposition 2
Recall that Proposition claims that a wealth process with
X-SKII tanh(pp SKIT (fsu px SKIT (Ýs))
(72)
can be used to reject the local conditional semantic importance H&S,s in Definition with Type I enror control, i.e. the game is fair.
Proof. It is easy to see that H.S,s is already written as two-sample test. Then, under this null, Ys-o) and Ys are exchangeable by assumption, and Lemma implies the statement of the proposi- tion.
23

---

Z1
Zz
Z3
Figure D.I: Pictorial representation of the data-generating process for the synthetic dataset.
D Synthetic Experiments
In this section, we showcase the main properties of our tests on two synthetic datasets.
D.1 Gaussian Data
We start by illustrating all sequential tests presented in this work are valid, and that they adapt to the hardness of the problem. i.c. the weaker the dependence structure, the longer their rejection times. We devise a synthetic dataset with nonlinear response such that all distributions are known and we can sample from the exact conditional distribution.
The lata-generating process we consider is defined as
Z1 .MNN(H1 of
#1 1, = 1/2 = - -,,02 =
(73)
72 ~ NNNNN122 of)
(74)
Z3 11 ~ 03)
=
(75)
and
Y 2 2
+ +3Z33)
(76)
where S is the sigmoid funetion, (O0 of), op 0.01 is independent Gaussian noise, and B,, == 1, are coefficients that will allow us to test different conditions. Then, follows that
g(z) = E Z 2]
S(3121 + 822223 + B323))
(77)
and
o 0
03 o,
21
Zz ~ N
o3 2o +
0 iI:
(78)
Fig. D.1 depicts the getta-generatigg process. We remark that, for this experiment, we assume there exists a known parametric relation between the response Y and the concepts Z. This is only to verify our tests retrieve the ground-truth structure, and our contributions do rely on this assumption. With this data-generating process, it holds that:
1. If 32 then Zz.
2. If 91 then 71 Z-1, and
3. For an observation Z z, if 23 = then g g( _ g(Zs) with Zo ~ PZiZa=zo
We test each condition with SKIT, C-SKIT, and X-SKIT, respectively. We use both linear and RBF kernel with bandwidth set to the median pairwise distance between all previous observations (commonly referred how often a test rejects), and the expected rejection time (i.e., how many steps of the test it takes to reject) over 100 draws of 7 max 1000 samples, and with a significance level 0.05. We remark that a normalized rejection time of means failing to reject in max steps.
the median heuristic [27]). For each test, estimate the rejection rate (i.e.,
D.1.1 Global Importance with SKIT
First, we test that
=0 = YIL
(79)
24

---

8g =1
9g
Yu
bokcdn time Eiver
0.5
0.4
0.2
0.2
0.0
0.0
-2
0
2 -5.0
-25
0.0
2.5
0.00
0.05
0.10
0.15
0.20
(a)
(b)
Figure D.2: Global importance results for Ho Y Z- with SKIT. (a) Marginal distributions of Y and Zg for 32 = 1 and 0, respectively. The red dashed line is the linear regression between the two variables, and, as expected, the slope is 0 for (2 O. (b) Mean rejection rate and mean rejection time for SKIT with a linear and RBF kernel, as function of $2.
2111-1131
YuzIZa
0.0
Reyiliul Feteddon Neniiaali 340)
0.54
0.6
0.00
0.05
0.30
0.15
0.20
2,
P:
(a)
(b)
Figure D.3: Global conditional importance results for Ho Y Z1
Z-1 with C-SKIT. (a)
Z, ~ Pz, is independent of for Z_1 = [-1,3]. expected, slope of the linear regression between Y and Zı is 0. (b) Mean rejection rate and mean rejection time for C-SKIT with linear and RBF kernel, as a function of 31.
with the symmetry-based SKIT in Algorithm B.I. Fig. D.2a shows samples from the joint distribution Pyz, for [32- = and B3 O. As expected, when B2 = O, the slope of the linear regression (red dashed line) is Rs 0 because Y and Ze are independent. Fig. D.2b reports average rejection rate and average rejection time as a function of 33. As 82 increases, the strength of the dependency between Y and Ze increases, and the rejection time decreases-this adaptive behavior is characteristic of sequential tests.
We can verify that the rejection rate is below the significance level a 0.05 when 32 and that the SKIT procedure provides Type error control. Finally, we note that both kernels perform similarly for this test, with the linear kernel generally rejecting less than the RBF one, and with longer rejection times.
D.1.2 Global Conditional Importance with C-SKIT
Then, we test that
3
1L Z_1
(80)
with C-SKIT (Algorithm 1). We remark that can sample from the exact conditional distribution P2,12_, = P2, (2.a) because Z2 is independent of Z by construction, and the conditional Pz, 12, can be computed analytically as shown in Eq. (78). We verify the conditional distribution behaves as expected in Fig. D.3a. By construction, Z, is sampled without looking at Y, hence it is independent, and the slope of the linear regression (red dashed line) and time as function of 31. First and foremost, we can see that in this case the linear kernel always fails to snjee-iindeenndently of the value 31. This behavior highlights an important aspect of all kernel-based tests, that is the kernel needs to be characteristic in order for the mean embedding to be an injective function [25, 68]. If this condition not satisfidd, different probability distributions could share the same mean embedding in the RKHS, and it may not be possible disambiguate them
0. Fig. D.3b shows mean rejection rate
25

---

2, -.5
Z: Z
gl2 Eg(3))
1.0
plZp,
912j
0.8
Densey
0.6
0.4
2
0.2
0
0.0
0.3 0.4
0.5
0.6
0.7
0.8
0.3
0.4
0.5
0.6
0.7
0.8
-0.10
-0.05
0.00
0.10
g12)
(
23
(a)
(b)
Figure D.4: Local conditional importance results for Ho g 2242,,)
= (( with X-SKIT. (a)
Shows that, as expected, the test and null distributions overlap when 23 =
at all. Consequently, the test will not be consistent, and increasing 7 m will not increase power. For the RBF kernel-which satisfies the characteristic property for probability distributions on Rd-the test is valid (i.e., it provides Type error control for 31 = 0), and it is adaptive to the strength of the conditional dependence structure-ad 31 increases, the rejection time decreases.
D.1.3 Local Conditional Importance with X-SKIT
Finally, we test that for a fixed z,
g(Z d g(Za), Zo Pzize=zo: YC S [m]]
23 0
(81)
with X-SKIT (Algorithm 2). That is because of the multiplicative term 2223 in 9 the observed value of 22 does not change the distribution of the response of the model when 23 Fig. D.4a shows the test (i.e., g( (223)))) )) and null (i.e., g (Z))) distributions for different values of 2y when 22 = 1. As expected, we can see that when = O, the two distributions overlap, whereas when 23 0.5, the test distribution is slightly shifted to the right. Fig. D.4b shows results X-SKIT with both linear and RBF kernel as a function of 23. We use both positive and negative values of to show that x X-SIT has two-sided alternative, i.e. it rejects both when the test distribution is the right and to the left of the null. We can see that both the linear and RBF kernel provide Type I error control when 23
and that their rejection times adapt to the hardness of the problem.
Now that we have illustrated all tests in arguably the simplest setting, we move onto a synthetic dataset where the response is learned by means of neural network.
D.2 Counting MNIST Digits
In this section, we test the semantic importance structure a neural network trained to count numbers in synthetic images assembled by placing digits from the MNIST dataset [38] in 4 x 4 grid. Fig. D.5 depicts the lata-gcnerating process, which satisfies:
Blue zeros, orange threes, blue twos, and purple sevens are sampled independently with
Zulue zeros ~ U({0, 1,2)) +
Z Zeaane threes ~ ~4(10, 1,2)) +
(82)
Zulue twos (((1,2)) +e
Zuuple sevens ~ ((((1,2) +e
(83)
Green fives are sampled conditionally on blue zeros with
3 1/8
if N Viuee =
Zgreen Zoluse ~ Cat
(1,2,3),
[1/8, 3/8.3/4,1/8/
if N Nulue zeros = + (84)
3/88,1/8,3/4
if Nulue zerOs 2
That is, the number of blue zeros changes the probability distribution of green fives over 1,2,3.
26

---

blue zeros
orange threes
green fives
red threes
blue twos
purple sevens
5
O 3
0 3
5
3
3 S
3 2
Figure D.5: Pictorial representation of the data-generating process for the counting dataset.
Red threes are sampled conditionally on both orange threes and green fives with
Zred threes
Zorange threes
Zgroen lives 2 Bernoulli(p) +
(85)
Norange threes N Ngreen fives 2 3 = 0.9.
(86)
otherwise
That is, the product of the number of orange threes and green fives changes the distribution red threes over 1,2.
Finally, we remark that N denotes the nearest integer to Z, and is independent uniform noise (i.e., ~ ((-0.5, 0.5)) make the distribution of the concepts continuous. To summarize, in order to generate images, we first sample the concepts Z according to the distribution above, round their values to their respective nearest integers N, and finally randomly place digits from the MNIST dataset in a 4 x 4 grid according to their number. Note that this data-generating process adds color to original black and white MNIST digits, and that color matters for the counting task since orange threes and red threes have different distributions.
We remark that, with the data generating process above, we can sample from the true conditional distribution of the digits, and, consequently, of images. We omit details on the conditional distribution for the sake of presentation." We stress that this setting differs slightly from the general one presented in this paper, where we consider both an encoder f and a classifier such that ý = g( f (xx), and we sample from the conditional distribution of the dense embeddings H given any subset of concepts (i.e.. Prizz ). The scope of this experiment is to showcase the effectiveness of our tests when the response parametrized by complex, nonlinear, learned predictor, hence we train a neural network such that fxx and directly sample from the conditional distribution of images given any subset of digits (i.e., Pxize).
We sample a training dataset of 50,000 images and train a ResNet18 [30] to predict the number of all digits for 6 epochs with batch size of 64 and Adam optimizer [35] with learning rate equal 10 weight decay of 10-5, and a scheduler that halves the learning rate every other epoch (recall that the model needs learn disambiguate red and orange threes, color matters). To evaluate the model, we round predictions to the nearest integer and compute accuracy on a held-out set of 10,000 images from the same distribution (we use the original train and test splits of the MNIST dataset to guarantee no digits showed during training are included in test images), and the model achieves accuracy greater than 99%.
Herein, we study the semantic importance structure of the predicted number of red threes with respect the predicted number of other digits. Note that the ground-truit truth distribution satisfies the following conditions:
1. Red threes independent of blue twos and purple sevens, i.c.
7ned threes 11 7 twos
and
7 threes II 7 pupple sevens-
(87)
All code necessary reproduce experiments is available GitHub.
27

---

grounderthh predictions
Figure D.6: Distribution of ground-truth data and density estimation of the predictions of the trained model for the validation data in the counting digits experiments.
2. Red threes are independent of blue zeros conditionally on green fives, i.e.
7ed threes
II 7iluue zeros
Zegreen fives-
(88)
3. If-in specific image- there are no orange threes, then red threes are independent of green fives, i.e.
Zeed threes (NN greenn fives Ngreenn fives N Norange threes 0) Zied threes Norange threes 0.
(89)
D.2.1 Global Importance with SKIT
We start by testing whether the predictions of the model satisfy the ground-truth condition
Zrod threes 1l Zoloe twos
and
Zred threes perple sevens
(90)
with SKIT (Algorithm B.1). We remark that at inference, we round predictions to the nearest integer and add independent uniform noise e ~ ((.0.5, 0.5) make the distribution of the response of the model continuous. Fig. D.6 shows the ground-truth distribution of red threes as a function of other digits in the held-out set, and the kernel density estimation of the predictions of the model. As expected, we can see that the ground-trute truth distribution is marginally dependent on blue zeros, orange threes, and green fives, but is independent of blue twos and purple sevens.
We repeat all tests 100 times with both linear and RBF kernels with bandwidth set to the median of the pairwise distances of previous observations. We perform tests independent draws of data of size 7 max E {100, 200, 400, 800, 1600} from the validation set, and study the rank of importance as function of 7 max i.e. the amount of data available to test. Fig. D.7 includes mean rejection rate and mean rejection time for each digit, and the rank of importance of digits as function of maxx, We can that-as expected-both linear and RBF kernels successfully control Type error for "blue twos" and "purple sevens". and this confirms that the distribution of the predictions of the model agrees with the underlying ground-trutt data- generating process. Furthermore, we can see that the rank of importance is stable across different values of 7 with purple sevens and blue twos consistently ranked last.
D.2.2 Global Conditional Importance with C-SKIT
Then, we test whether the predictions of the model satisfy the ground conditional independence condition
Zred threes 11 Zolue zeros Z Zreen fives
(91)
with C-SKIT. Analogous to above, we repeat all tests 100 times with linear and RBF kernels, and Fig. D.8 includes results for max e { {100, 200, 400, 800, 1600). Here-ssimilarly the synthetic experiment presented in Appendix D. le can see that the linear kernel almost always fails to reject, ie. the mean rejection rates for all digits are close to O. As discussed earlier, this behavior due to the fact that the linear kernel is not characteristic for the distributions. On the other hand, the RBF is, and, as expected, it is consistent and it provides Type error control for the null hypothesis that red threes are independent of blue zeros conditionally on all other digits, which is true. Furthermore, we can see that the rank of importance is less stable compared to the one in Fig. D.2, and particular, maxx 100 seems not to be sufficient to retrieve the correct ground-trui structure (i.e., blue twos are ranked before green fives). This highlights how the amount of data available for testing may affect results and findings.
28

---

red threes
red Dhrees
grewn fivers
blue 2010s madder time peliaaene
orange threes
urange thress
orange threes
orange threes
green ween fives
green fives
green
Rank
blue zeros
blue poros
blue zeros
purple sevens
blue twcs
surple seowns
Blue twos
pupple sevens
purple sevens
0.5
3.0
1000
200
4D0 1800
1.0
100 200
800 1600
(a) Linear kernel.
(b) RBF kernel.
Figure D.7: Global semantic importance results for the predicted number of red threes with linear and RBF kernels. In each subfigure, the leftmost panel shows mean rejection rate and mean rejection time over 100 tests with a 0.05 and 7 800. The rightmost panel shows the rank of importance of digits for the prediction of red threes as a function of 7 max
thrroos
rcd threes
threes
red threes
orange threes
orange threes
o pango threes
orange threes
green
blue zeros
green
green fives
blue 20105
grean Evas
purple sevens
purple sevens
purple severs
vedio 110m
brue twos
blue took
blue twos
blue
purple savvens
blue
plue zoros
0.0
0.5
1.0
100
200 490 800
1600 TTar
0.0
0.5
1.0
100
200 400 200 1600
yru
(a) Linear kernel.
(b) RBF kernel.
Figure D.8: Global conditional semantic importance results for the predicted number of red threes with linear and RBF kernels. In each subfigure, the leftmost panel shows mean rejection rate and mean rejection time over 100 tests with a = 0.05 and max 800. The rightmost panel shows the rank of importance of digits for the prediction of red threes as function of max
D.2.3 Local Conditional Importance with X-SKIT
Finally, we test whether the predictions of the model satisfy the ground-trute -rrth condition that, for a particular image, if there are no orange threes (i.e., Rorange threes = 0) then red threes are independent of the observed green fives (i.e., Regeen lives), ) i.e.
Zred threes green fives = Rggrenn lives N Norange threes = 0) _ Z threes Norange threes 0.
(Ngreen
(92)
We remark that, in the equation above, conditioning is written in terms of the integer Morrage threes 0 because of its intuitive meaning, and that this is equivalent to conditioning on Zorange threes e (-0.5, 0.5). Similarly, we could replace Nggree fives with Zgreen fives, and, in practice, we run tests conditioning on the observed concepts z, and not their integer values n.
We use X-SKIT (Algorithm 2) with a linear and RBF kernel with bandwidth set to the median of the pairwise distances of previous observations. We repeat all tests 100 times on individual images with 0, 1, and orange threes, significance level a = 0.05, and T maxx = 400. Fig. D.9 shows results grouped by number of orange threes. As expected, we see that when Zorangg threes is grater than O, the number of green fives is important for the predictions of the model (i.e., rejection rate is close to 1, with short rejection rate). whereas when there are orange threes in the image, both the linear and RBF kernel control Type I error. We qualitatively compared our findings with pixel-level explanations with Grad-CAM [57]. and we can see that they only highlight red threes because that is the digit we are explaining the prediction of. That is, pixel-level explanations cannot convey the full spectrum of semantic importance for the predictions of a model- can be misleading to users. For example, in this case, user may not understand when the predictions of a model depend on the number of green fives, because they are never highlighted by pixel-level saliency maps. In real-worlk scenarios, digits may be replaced by sensitive attributes that cannot be inferred by the raw value of pixels. For example, a saliency map highlighting a face does not convey which attributes were used by the model, such as skin color, biological sex, or gender. It is immediate to see how being able to investigate the dependencies of the predictions of a model with respects to these attributes (which our definitions provide) is paramount for their safe deployment.
29

---

Candraning enange mines
CootiCA exsivaation
Lecal cond
prewn fiees
Conditin eragee stree
Local conditional Inportence grean flee
J 3
J
3
3.0
a
5
3
à
5
3
2
2
64
Adjaction cae
7
7
3
3
3
5
3
3
3
3
Irea
121
18ear
(a) Results for Foorange 2.
(b) Results for Thocuage
1.
importatce green tvee
orange threes 5
5
1.
7
7
Repectiondie dine
7 2
2
3 3
3 3
(c) Results for Toraage threes
0.
Figure D.9: Local conditional importance of N Ngreen fives conditionally on Norange threes- Each row contains the input image, the Grad-CAM explanation for the prediction of the model, and X-SKIT results for 100 repetitions of the test with - = 400, with linear and RBF kernel. Note that X-SKIT finds the observed number of green fives important whenever the number orange threes greater than zero, whereas Grad-CAM does not.
With these results on synthetic datasets, we now showcase the flexibility of our proposed tests on zero-shot image classification with several and diverse vision-language (VL) models.
E Experimental Details
In this appendix, we include further details about the real-world experiments that were omitted from the main text for the sake of presentation. All experiments were on private server with one 24 GB NVIDIA RTX A5000 GPU and 96 CPU cores with 500 GB of RAM memory.
List of VL models used in the experiments. We use 8 different VL models, both CLIP- and non-CLIP Mon-CLIP-based CLIP-RNSO, B/T-L/22 [52], OpenClip: ViT-B-32 V4T-L-14 [31], FLAVA [65], ALIGN [32], and BLIP [39].
Evaluating rank agreement. We use weighted version of Kendall's tau [33] introduced by Vigna [74] which assigns higher penalties to swaps between elements with higher ranks. This choice reflects the fact that concepts with higher importance should more stable across different models. We briefly remark that this notion of rank agreement is bounded in [-1, (-1 indicates reverse order, and perfect alignment) but not symmetric.
Evaluating importance agreement. We threshold rejection rates at level a to classify concepts into important and not important ones. Then, importance agreement is the accuracy between pairs of binarized vectors.
F.1 Estimating Conditional Distributions from Data
Here, we introduce nonparametric methods to estimate the conditional distributions necessary to run our C-SKIT and X-SKIT tests. Throughout this section, we will assume access to a training set {(h9, 20)) of n tuples of dense embeddings h E Ed with their semantics 2 E [-1, 1)
E.1.1 Estimating P2z,1z_, for C-SKIT
Here, we describe how to sample from the conditional distribution of a concept Z,, given the rest, Z-j, i.c. i.c
Z, P2,ız_,, which is necessary to C-SKIT test. In particular, for concept
30

---

Enghul
9330
1000
-4000
0000
nargral
sedd Fone
Aqund
Squit
Conone
French hom
margnal
neiginl
01
02
81
Zwry
Zwne
Zwan
Zween
Zoee
Zoen
Zoee
Zow
9/ PUTT
94
46
rengrel
coratinnl
FIREDO RES
03
0.1
01
02
0.1
03
87
0.1
82
0.1
0.2
0.5
0.1
0.2
0.1
0.1
0.8
0.8
0.7
Zoeesor
Zeoope
Figure E.1: Example marginal and estimated conditional distributions p(zy) and p( (22)|2-3) for two class-specific concepts on three images from the Imagenette dataset. Distributions are shown as function of the effective number of points in the weighted KDE (i.e., Reft).
j E [m]. and an observation 2 [-11 1)" we define the unnormalized conditional distribution
22
w
(93)
1/soou
by means of weighted kernel density estimation (KDE), where e is the standard normal probability density function, Msona is Scott's factor [56], and
W. á
V
(94)
are the weights. That is. the further (8) from ž-j, the lower its weight in the KDE. As for all kernel-basee methods, the bandwidth y important for the practical performance of the model. For our experiments, we choose v adaptively such that the effective number of points in the KDE (i.e. Near = (E - w Σ wf) is the same across concepts. This choice is motivated by the fact that different concepts have different distributions, and we want to guarantee the same number of points are used to estimate their conditional distributions. Furthermore, we note that Neft controls the strength of the conditioning- the larger Reff, the slower the decay of the weights, and the weaker the conditioning. That is, in the limit, the weights become uniform, the conditional distribution tends to the marginal p(z;), and tests presented become of decorrelated semantic importance [72, 73]. With this, sampling Zj Pz,lz_,==_,, boils down to first sampling i' according to the weights (8')
and then sampling Zj from the Gaussian distribution centered at 2,
Fig. E. shows the marginal (i.e., p(z,)) and estimated conditional distributions (i.e., p(æzl=-))) of two class-specific concepts as function of effective number of points Netr for three images in the Imagenette dataset. We can see how as Neff increases, the estimated conditional distribution becomes closer the marginal, and that the conditional distributions of class-specifie concepts tend to be skewed to higher values compared to their marginals. This behavior suggests that images from specific class have higher values of concepts that are related the class. We use Netr 2000 for all tests across all real-world experiments.
E.1.2 Estimating Priize=ao for X-SKIT
Here, we describe how to sample from the conditional distribution dense embeddings con- ditionally on any subset of concepts C C [m]] of particular semantic vector = E 1|m, i.e. He ~ PHIZe=ac which necessary run our X-SKIT test. We show how to achieve this by coupling the nonparametric sampler introduced above with ideas of nearest neighbors. This choice is
31

---

Ant
13
02 candiaiiy
03 Paysawiin
Foy candralii
Franch
cardinalit 3
cardinliy
caddralit
cadruiity
cadiaaloy
candrully +1
62
6J eceeer
aceeor
94 FIREGO
15
02
01
01 82
011 82
Fux
Taom
Tam
Figure E.2: Example test (i.e., ((H )) and null (i.e., g s(Hs) distributions for class-specific concept and a non-class specific one on three images from the Imagenette dataset as a function cardinality of S.
motivated by the need to keep samples in-distribution with respects to the downstream classifier Intuitively, we propose to:
1. Sample Ż ~ Pzizemza and then
2. Retrieve the embedding H(')) such that concept representation z(1) the nearest neighbor of Z.
Step 2 makes sure that samples are coming from real images, and overcomes some of the hurdles of sampling high-dimensional vector (H E Rd d ~ 102) conditionally low-dimensional one (&c e [-1, I|I C S [m], m x 20). More precisely, recall that ((h(),, )0)) is a set of n pairs of dense embeddings and their semantics, then
Hc = 00)
s.t.
i'= arg min || Z11. 7 Pzize=sci *E(n)
(95)
where Pz|Za=za is approximated with p(z-c =C), -C = [m] C' as in Eq. (93).
Fig. E.2 shows some example test (i.e., g ((H))())) and null distributions (i.e., g( (Hs)) for class- specific concept and a non-class specific one on the same three images from the Imagenette dataset in Fig. E. We remark that random subsets of increasing size. We can see that the test distributions of class-specific concepts skewed to the right, i.e. including the observed class-specific concept increases the output of the predictor. Furthermore, we see the shift decreases the more concepts are included in S, i.e. S is larger and it contains more information, then the marginal contribution of one additional concept will be smaller. On the other hand, including non-class-specine concept does not change the distribution of the response of the model, no matter the size of S --precciyly our local definition of importance (HLS (.j.s) demands.
be any subset of the remaining concepts,
show results for
E.2
AwA2 Dataset
Here, we include additional information, tables, and figures for the AwA2 dataset experiment. This dataset comprises 37,322 images (29,841 for training and 7,481 for testing) from 50 animal species along with class-level annotations of 85 attributes (some example figures are included in Fig. E.3). Concept annotations are reported both as frequencies (i.e., how often an attribute appears in images coming from a class) and as binary labels (i.e., means that an attribute is present in class, and otherwise). Table E.2 shows the zero-shot classification performance of all VL models used this experiment for the top- 10 classes, and, for each class, we test 20 attributes: the 10 most frequent, and a random subset of absent ones (concepts are included Table E.3). Finally we remark that, for each model, obtain the dictionary by encoding concepts with its text encoder.
32

---

giant panda
tiger
giratte
zebra
lion
squirel
sheap
horse
elephant
dolmatian
Figure E.3: Example images from the top-10 10 best classified classes in the AwA2 dataset.
Table E.2: Zero-shot classification accuracy on the AwA2 dataset.
Accuracy
Model
giant panda
tiger
girafle
zebra
lion
CLIPARN50
100.00%
100.00%
100.00%
100.00%
99.51%
CLIP-NiT-B/32
100.00%
100-00%
100.00%
100.00%
99.515%
CLIP: VIT-L44
100.00%
99.39%
100.000
100.00%
OpenClip-ViT---33
VIT-B-32
100.00%
100.00%
100. 100.00%%
100. 000%
100 100.00%
OpenClip-ViTLL.I
100.00%
100-00%
100.00%
100.00%
100.00%
FLAVA
100.00%
98. 98.86%
100.00%
100.00%
99.02%
ALIGN
100.000
100.00%
100.0009
99.57%
100.009%
BLIP
100.00%
100.00%
99.17%
99.15%
99.3 10.511%
average
100.000% = 0.010
999.8% 0.38
99.84% 0.29
919 99.849% 0.30
99.69% +: 0.33
Accuracy
Model
squirrel
shoop
horse
clephant
dalmatian
CLIP-RN50
90.17%
97.54%
98.18%
94.71%
96..16%
CLIP-VIT-B/B
99. 58%5
98.59%
99.399%
99.01%
98. 1876
CLIP-ViT-L/4
100.00%
99.63%
78%
100.00%
100.00%
OpenClip:
90. 58%
99.65%
98. 78%5
100.00%
97.27%
OpenCCii:ViT-LLII
100.00%
100.01%
100.00%
100.00%
100.00%
FLAVA
99.17%
100.00%6
99.3956
99.04%
98. 18%6
ALIGN
99.087
99.63%
99. 70%8
100.00%
99.09%
BLIP
17%
98.94%
99.30%
100.00%
100-00%
average
99.53% 1. 0.33
99.25% 1. 0.80
99.20% 0.55
99.10% 1.71
98.64% 1. 1.29
Table E.3: Class-level attributes tested on the AwA2 dataset.
Attributes (20)
Class
present (10)
absent (10)
patches. old world, furry, black, big. white, walks, paws, bulbous, vegetation
flies. flipeers, hooves, desert, hairless, red, blue, horns, plankton. ycllow
giant panda
stripes. stalker, meat, meat teeth, hunter.
strain teeth, tunnels, hops, plankton, bipedal,
tiger
strong. fieree, old world, muscle, big
tusks, flippers, flics, small, skimmer
long neck. long legs. big. quadrupedal, spots, vegetation, lean, old world, walks, ground
bipedal, hibernate, cave, mountains, ocean.
girafle
hunter, water, stripes, gray, fish
stripes, old world. quadrupedul, black. white. ground, group, grazer, walks, hooves
blue, spots, brown, water, coustal. patches, tusks, claws, scavenger, red
zebra
meat, stalker, strong, hunter, meat teeth.
tunnels. blue, horns, skimmer, long neck,
lion
big, fiecee, paws, furry, old world
wuter, Oippees, tusks, aretic, spots
tail, furry, forager, small, gray, tree, new world, forest, vegetation, brown
harns, blue, yellow, tusks, meat teeth,
squirrel
flippers, scavenger, desert, plankton, strain teeth
white, quadrupedal, walks, group, vegetation,
aretic, flippers, insects, paws, long neck,
sheep
grazer, ground, fields, furry, world
red, yellow, swims, plankton, hands
hooves, fast, grazer, hig, long legs
aretie, tree, bipedal, plankton, fish,
horse
tail, quadrupedal, fields, brown, strong
stripes, ocean, strain teeth, scavenger, orange
big, old world, gray, tough skin, quadrupedal
claws, flippers, orange, swims, ocean
elephant
tusks, hairless, strong, ground, walks
stripes, tunnels, plankton, coastal, strain teeth
big, old world, gray, tough skin, quadmupodal
claws, flippers, orange, swims, occan stripes, lunnels, plankton, coastal, strain teeth
dalmarian
tusks, hairless, strong, ground, walks
33

---

OpAr cot SIN- 14
OOME www.on---ttaaa.. 32
Cip- AEp-9FLLS44 NIT-802
Bava
open
Eava
1.40
3.80
3.2
2.5
331
11
1.00
3.00
0L7
CAD
1.80
341
0.50
0.75
CIAASRR/12
1.00
8.9
04E
0.54
0.44
dp MPVIUU
0.25
CIANT---8822222 52
+8.0
0.00
spoLd
3M
48-37
160
0-11
3.16
2.0
0.10
Tavo
1.1
3.9
6:14
0.93
3.07
0-11
0.16
3.99
a8gn
04p
4000
4000
40020
-1.00
SKIT
c-SKIT
PCBM
Figure E.4: Rank agreement comparison between SKIT, C-SKIT, and PCBM on the Aw dataset. Results are reported as mean and standard deviation over the 10 classes considered in this experiment.
giroffc
nock
ipos (01
block
Tunter
Strong
Stonng
vogusation
upod
Tarry
luny
tapozal
quadi podul
Importance
ket
stakkr
risss
wolks
ground
walk
dld
plansion
mout
1con
bettous
korp
hoooea
Scave 1pe
1ica
group
plankton
Inberrabe
0.5
2.0
0.5
1.0
wlephend
fiokds
frovess
spots
scavenger
harless
20rong
small
walls
quodkupeoll
old
pou
yollowr
vogetation on
ong log
Walko
tough
forager
0
ucc
quadrupedall
gray
tury quadrupedaa
bronn
Forest
vegetartoo
plackton
plamn
segatatioon
neeth
2rean
paart
diopers
orstge
DEC aoe 10)
noms
2013 (A)
domesti
0.5
1.0
0.0
0.5
1.0
0.0
0.5
Figure F.5: SKIT importance results with CLIP:VIT-L/14 on the AwA2 dataset.
ganntpands
girafe
plantton (8)
Teere 1p1
long werk
stripe (01
Serce 100
Ate1e 10)
huittr
1100
16g 10)
bipoaa
3p015
gro0e 101
lung kggs
Impor 201TE
mozt boath 101
yellow
Rlcpers
OM
Brock
Irg
ground
patches
Stroin z00th
nolis
wall (0)
moustarrs c
Scave-1e
skimmot
0.0
2.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
3.0
0.5
3.0
shrep
elechant
delmetion
tumy 16)
faids
hooves (p)
Su8ts (0)
spats
vegeration
quadssppca
ha nteee
vegebacion
(20
long neck
Saide
plarktin
frospee (10
strong
Rippars
Ieen
tucos
fury
world
tough
Mue
k5
striees
domestti
ground
Sroh bouth
plantgar
plaritonn
DRA 220i5
Blaiic
strain
plains (N)
Suodnepdd
Scaverger
orarge
fippers
grou0
stipes
0.0
0.5
2.0
0.0
0.5
0.0
0.5
1.0
0.0
0.5
1.0
0.5
Figure E.6: C-SKIT importance results with CLIP:ViT-L/I4 on the AwA2 dataset.
34

---

pelican
homed puffin
Tycaltche
flicker
cardinel
blue
cape glossy Starling
frigatebird
Figure E.7: Example test images from the CUB dataset with their respective classes.
Table E.4: Zero-shot classification accuracy on the CUB dataset.
Accuracy
Model
white pelican
brown
mallard
borned puffin
vermilion Byyatcher
CLIP-RNSO
82.00%
15.00%
00.0%
CLIP NIT--21-
96.675%
90.00%
33%
93.53%
CLIPNIT--LI1
98.00 1870
100.00%
100,0076
95.0KP
100.00%
PoT-Clip-VTT----
100. 00%
98. 33%
98 53.533
85 93%
95.075
Ope
100. 009S
100.009 00%
100,0050
6.%%
100 000%
FLAVA
100.0095
100.0056
BLIP
98.009%
80.00%
96.673%
65.00%
55.00%
average
17.75% 2.54%
83.85% 6.24%
06 3.125
10.08%
.1.9%%% 14.09%
Accuracy
Model
northem Blckkr
cardinal
jay
cape glossy
frigatebird
CLIP-RNSO
95%
13.1
00%
73.333%
83 43.33%
90.00%
CLIP-VIT-L/I
98. 33%
100.0095
85.005%
100
96.67%
OpenCTip VIE-B-32
96.67%
100.0055
88.33%
96. 06.6775
88.33%
VIE-L-
100.0/
100.0054
91.8756
50.0075
933.335%
FLAVA
76.07%
100.0095
81.6736
58-3355
30 0070
80.0075
55.00%
everage
87.08% à 12.90%
80, 18.18% 16.42%
84.79% 8.14%
87.08% =: 907%
82.08% d: 12.47%
E.3
CUB Dataset
Here, we include additional information and results for the experiment the CUB dataset [77], which contains 11 11,788 images of 200 different bird classes, and each image is annotated with the presence of 312 fine- fine-grained grained concepts that describe the appearance of the bird (c.g.. "has orange bill",
"has hook-shaped bill", "is small") with the labelers' confidence. Formally, the dataset is collection f(x(1) ()) 20) u (()) of images æ with class label y, binary semantic vector 2(0) {0, 1) and uncertainty values (1) E 2,,2,3,4 4}: "not visible" (1), "guessing" (2), "probably" (3), and "definitely" (4).
We randomly sample 10 images from the 10 classes with highest average accuracy across models: Table E,4 includes accuracies of all VL models used, and Fig. E.7 shows some example test images
Table E.5: X-SKIT results on CUB dataset as function of conditioning set size 8.
Rank agreement
Importance agreement
Importance fi score
1
0,81 + 0.14
0.96% + 0.06%
0.93 + 0.15
2
0.82 + 0.13
0.97% + 0.06% 0.95% + 0.08%
0.93 + 0.14
4
0.84 + 0.12
0.88 + 0.15
35

---

Hiiiiitt 32
dap 4774
GONE
1.00
0V5
0V
041
60.12
EXT
EHI
MOLUE
2.9
034
014
CIPATOO12
E31
101
0.0
000
0.0
300
380
034
031
dpMIFUIA
CIANT---32 32
4000
0.00
831
244
1.30
000
100
824
8.47
3.4
280
240
10.
100
024
124
844
Bava
831
2.37
8.67
2.56
146
020
634
1AD
234
0.84
3.20
286
3.4
246
240
2.0
146
034
31gn
-100
(a) Rank agreement.
saarry dip Ap.rtaa.
open dip Dip-9M--1 L214
qpen 14
opei clp 97 F14
Claa Wi 1214
Ci Raor
dip:RNSD
46
4000
M000
40.25
4607
40.0
43 44337
87
6605
46.00
4000
00
400E
40
0.8
126
01
3.6
30
153
0.14
040
D00
010
010
1.00
9.
0M
spon, dI
80
12000
12000
00
20.00
20/00
08.08
4000
000000
20.00 W8
Bavo E87
646
666
0.6
0.96
1.0
3.07
8.7
300
38
147
6.0
1.6
0.0
0.9
696
0.9
0.96
396
6.5
6.5
align
000
395
1.00
10.0
40.00
0.0
(b) Importance agreement.
Figure E.8: X-SKIT agreement results the CUB dataset function of conditioning set size, 8. Results are reported as means and standard deviations over the random 100 images used in the experiment.
for each class. For each image, we select 14 concepts to test: we first restrict ourselves to annotations with good confidence (i.c.. "probably" or "definitely"", and then, for each concept j. we estimate its marginal (i.e., Pj == P(z 1)) and class-condiiional (i.e., Pjly = P(Z, = Y y)) rates over the dataset. Finally, for each test image with label y. we score concepts by the difference between their class-conditional and marginal rates, i.e. 83 8y(U) = Patv Pj. Intuitively, large value of sy (v) indicates that concept j has a higher occurrence in class y compared the population, and we say that it is discriminative for class y. We test the 7 most discriminative concepts that are present in the observed image J, and a random subset of 7 concepts that are absent according to the ground-truth annotations. We remark that, since concepts are binary, we do not use the KDE-based methods presented in Appendix 1, and instead we approximate PhIze=ze by sampling uniformly from the entries in the dataset that match the conditioning vector zç. Finally, bandwidths set to the median of the pairwise distances of observations, and max = 200.
RBF kernels with
After running X-SKIT, we classify concepts important by thresholding their rejection rates at way of selecting important concepts. Table E.5 summarizes agreement and detection results as a function of conditioning set size (i.e., the number of concepts in S C [m] (.)). and Fig. E.8 includes all pairwise agreement values. We note that the fi score for 4 (i.e., when conditioning on concepts) is lower compared to 1, 2. This is expected, as the more concepts one conditions on, the smaller the effect of including one additional concept.
level w-which is statistically-var
Finally, Fig. E.9 shows ranks of importance for all models on the 4 example images used in the main body of the paper.
36

---

2000 80
5
2000
25
07
2%
21 00
01
22
20
four 1 for o example i in the
all ailli m models &
Figure E.9: S-SIT importance results results($ (s = 1) across a CUB dataset.
SOLTTimppraan
37

---

Engliish springer
player
French horn
church
parachute
golf
gas pump
garbage truck
chainsaw
Figure E.10: Example images from the Imagenette dataset with their respective labels.
Table E.6: Zero-shot classification accuracy on the Imagenette dataset.
Accuracy
Model
tench
English springer
Casseliic player
French burn
church
CLIPARN50
05.48%
89.45%
50%
93.45%%
100.00%%
CLIP-NIT-BB2
73%%
LLIP-VIT-L/14
99.2255
89.75%
89.723
99.75%
100. D0%
DpenClip-ViT-BB..
100.00%
100.0076
98. 32%
90. 90.24%
98.70%
OpenClip: V4
100.00%
100.00%
89.72%
99.75%
100.000
FLAVA
99.74%
99
ALIGN
99.7455
100.00%5
90. 41%4
90.75%
100.00%%
BLIP
100.00%
98.48%
90.50%
99.73%
100,00%
average
99.74%% 1: 0.26%
99 53% 0.50%
97. 55% 2.09%
99.33% 21 0.54%
90.97% 0.08%
Aucuraccy
Model
parachute
golf ball
gurhage trukk
chninsaw
65%
CLIP VTT-B/22
98 9755
99 2556
6136
19%
9H 22.2
CLIP VIE:L114
99.7456
98 7556
100.0056
74%
90.74%%
OpenClip-ViTBB-B.
90.4955
89 2136
97.61
99.23%
97.667%
OpenClip: VIT-L-14
100.00%
891.99%
08.7256
989 0006
966.37%
ALIGN
99.4955
88. 189.7556
98. 26.2
90.40%%
98.96%%
BLIP
99-4956
99 3055
98. 09%6
99.235%
98, 18%19
average
90.46% + 0.30%%
90 2295
0.61%
97 6445
+ 234%%
19) 3336 + 0.20%
18..22% 1.10%
E.4 Imagenette Dataset
Here, we present additional information and results on the Imagenette dataset presented in the main body of this manuscript." This dataset contains 13,394 images (9,469 for training and 3,925 for testing) from ten easily separable classes in ImageNet [22]. Fig. E0 includes example images from the classes in the dataset, and Table E.6 reports the classification accuracy across all vision-langee language models (98.99% + 0.01% % average accuracy).
Recall that the ImageNet dataset does not provide ground-truth semantic annotations, hence we use SpLiCe [6] to find which concepts to test. In particular, we use the 10,000 most frequent words in the vocabulary from the MSCOCO dataset [40], and we set the E regularization term in SpLiCe to 0.20. Following previous work [85], we filter the selected concepts such that they different from the classes the dataset. We use WordNet [23] to lemmatize both concepts and class names (e.g., "churches" becomes "church"), and we check that concepts are not contained in class names and vice versa. For example, the concept "churches" would be skipped because "church" " is already the name of class, and "gasoline" would be skipped because it contains part of the class "gas pump".
'The Imagenette dataset is available at https: : http:///gggbbbctatttaaaaa//aeeee
38

---

English
Franch
bottered
spanic
Cattedral
Indrummnt
Indrummett
15h
rxdio
Dans
obuneee
Indtrumer
arndiw
brwn
trumpet
disclato
Jace
3464
Rahing
radio
importance
batterrd
bass
catherra
putting
Ech
arshow
strstow
catnedaal
maning
endencr
band
bass
purting
10.0
Srumpes
fish1ng
Tice
battored
caltedral
fore
pumiing
J977
battened
sconie
form
0.0
0.5
1.0
0.5
parachute
gol ball
0a5 pump
gorbage truck
chaincaw
tore
dispenser eccenor
obsolete
ebsolete dispereer
acteri
New
inthumert
spaniel
trumper
eoterict
radi
putling
cathiciil
for
fihing
band
Importance
faringg
airsfow
soanic
lsh
buttered
disperser
ardhom
New
sinhow
exterior
19z2
putting
505s
brass
band
brass
cattedral
fshing
dispondor 100
Cathoora
fore
sparect
Ciunpot
spand
colheora
obeciete
Drank
seling
0.0
0.5
1.0
0.0
6.5
1.0
0.0
0.5
1.0
0.0
0.5
3.0
9.0
0.5
3.0
(a) Global importance with SKIT.
English
French
16h
obsolete
trurpst
band
6aterior
1r555
trss
Instrument
brns
band
00ng
Bross
battered
trumpet
trumpst
shoplde
Instrument
importance
Stamoct
fch
dispersor
1g
obsol.t0
for
diacenger
fching
depprae
Chan
dispenser
batteaed
cethecre
radio
10d.0
New
cathocral
1000
battored
sirsh
putting
arshow
battered
selling
selingg
botteree
Selling
putting
0.0
0.5
1.0
1.0
0.0
0.5
parachuto
golf ball
gas pump
garbage truck
chainsaw
fore bass
dispenser
abooktte
Instrumint
obsokcte
5066
[622
trass
band
mumpet
band
inttummnt
trumpe
Importance
obsclate
Instrmmett
Jazz
dlscenser
chpaaterr
instrumene
dispetser
extedor
fihing
fih
battered
557ing
spanicl
Spanto)
500n c
diparree
ectetor
cathecra
withoo
Foro
battered
cachedral
100/0
radlo
cathcdral
putting
baitaree
battered
arnhoa
purting
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
3.0
0.0
0.5
1.0
(b) Global conditional importance with C-SKIT.
Figure E.11: Global importance results with CLIP:VYT-L/14 on the Imagenette dataset.
open AAp
grem M44411--14
opee cip-imm-12
Openrt 11--9111-1
Cipy uipimin
city eip-mman aa
flave
Elave
di:RN50
2022
10.24
0.50
dPMITU14
642 544
30
2000
24
54
20/00
40000
0.25
Wing lendel's
spon spor_cisNIT..2
646
54L
1.00
0.60
0.30
3.8
642
3.9
3.50
3.54
6.46
646
2.3
4.00
053
0.93
2.09
3.56
fava
-050
3kgn
20
66
022
1.0
1874
10:00-000000000000
653
6-44
C44
830
H0/77
8.55
344
11
17
3.06
-075
455
142
344
3.47
0.47
620
6:42
649
84L
332
1.0
84
0.30
3:3
0.10
000
600
6:0
3.00
SKIT
C-SKIT
PCBM
Figure 1.12 Rank agreement comparison between SKIT, C-SKIT, and PCBM on Imagenette. Result are reported as mean and standard deviation over the 10 classes considered in the experiment.
39

---

tonch
English
cossente
French homn
church
bottered
Scanie
obsoleto
zumpet
cathedral
Fish
seting
radio
ptsolate
bass
radio
band
brass
band
brods
birthow
222
12h ng
rodio
Rank
Tiow
cbspleto
Ca53
cathedral
hathing
eatenne
Catiedrrl
Sparse:
finh
cospanser
fshing
arahow
diparspr
lww
New.
trumact
cathedda
putting
R5h
Tlw
batterod
yellng
band
Rah
Iome
buiting
azz
brass
batteree
spanial
fore
parachire
goff ball
garbagg truck
chainsaw
Sisprttee
stwolete nntrumont
stnoliete
cxcorior
new
catcriar
dispersor
Tore
Scan
Kumpet
grump os
rodio
putting
airshow
fore
Rsh ng
band
waller
Fatio
actentor
arshow
bathered
fore
rallo
Tow
72dio
cathedral
band
dispenser
brans
selling
Rchingg
Spanie
obzolete
cathedral
Now
Rish
bask
instrument
sunder
cathedral
Bass putting
Solling
bass
spantel
putting
200 200 400 850
200 200 400 8001500
200 200 420 5001800 yruc
200 200 400 50c 1620
100 200 402 502 1600
you
you
(a) Global importance with SKIT.
tench
Fnggish springer
Consellle player
French born
church
(sh
brass
radio
Band
octerior
instrument
bate
Iaad
crumoct
crumoct
band
aosalcto
mi
frrampe
Rant
Fosh
Fashing
dispertsar
Bthing
Rih
spanell
fish
exterior
cizppnner
72.10
spantal
celleee ched
radio
fore
calthedial
fore
Fine
catheddai
catteedral
balteered
putling
putting
battered
pullinng
puttinng
parachure
gull bul
gas pump
gurllage
Chaimaae
Fiow
dispenser
obsoleto baso
Insrument
strshow
putting
obsolete
0250
brnos
bross
band
Trumpet
band
trumoet
Obsolete
tuupeet
numenttnt
Dosolete
Pank
obpolete
instrumints
disponser
intrrmmnt
extorior
Bahingg
Filt
fichig
fish
forc
Pore
exterlor
sparie
catheorra
battered
spantel
spantel
bellaree
Tetho
odlo
Del0
putting
fore
How
Cottored
airshow
airshow
putting
putting
800
800
(b) Global conditional importance with C-SKIT.
Figure 13: Importance results with CLIP:VIT-L/14 on Imagenette function of 7 max
We run SKIT and C-SKIT with RBF kernels with bandwidths equal to the 9hth percentile of the pairwise distances of previous observations and max = 400, 800, respectively. We encode the entire dataset using SpLiCe and keep the top-20 concepts to test. Fig. E.11 shows global and global conditional importance results with CLIP: VIT-L/14 for all classes in the dataset. Furthermore, Fig. E. shows all pairwise rank agreement comparisons for SKIT, C-SKIT, and PCBM across all models used in the experiment. Lastly, Fig. E. 13 qualitatively shows ranks of importance function of 7 max with CLIP-ViT-L/14
We use X-SKIT on 2 random images from all classes in the dataset (20 images total). Recall that we use SpLiCe to encode each image and keep the top-10 concepts, and finally add the bottom-4 concepts according to PCBM, for total of 14 concepts per image. We set the bandwidth of the RBF kernel used in the test to the median of the distance of the observations, and max = 200. As in the CUB dataset experiment, we classify concepts as important by thresholding their rejection rates at
40

---

Table E.7: X-SKIT results on the Imagenette dataset function of conditioning set size
Rank agreement
Importance agreement
0.59 + 0.21
0.71 + 0.14
2
0.56 + 0.21
0.67 + 0.13
0.53 = 0.23
0.68 + 0.14
opeic dip 4.44
coNrr otin U088-3 A4
ctp vP 8-32
cop Ren.mLL
cip cipiteeg
clevmer2
4 Sputter
clevmuu
opent cien
cip
diS
0.75
claMTer32
3.55
329
0.0
845
10
1.0
0.50
0.6
030
0.83
0.0
106
055
649
643
159
3.50
655
100
3.54
0.11
0.4
0.54
04
1pen SIAATO--
643
2.55
0.1
0.7
340
6.6
3.00
643
133
1.39
0.54
0.8
OEPL ope VIRL-M
22033
20.11
20.00
10.37
23
12
30.12
17
20.70
28.77
00003
20.00
00.00
$0.00
00.38
0..0
64
1.3
3.5
0.52
0.5
140
647
0.56
0.60
456
650
6.9
641
340
2177
451
13
8.5
0.44
3,06
056
254
0.0
0.8
645
130
641
145
056
348
(a) Rank agreement.
operr clp 82
Wpeen cipo MRvVILL 34
coem dlp WIB--
coe Lap L411
Span Nip 14
Sle-V-- 802
opan Cit HPLLL
cid
dip:MAB0
26.30
2011
2A.0000000 30
12010
L0
6.00
2012
0.0
835
60
0.90
3.69
10
3.N
544
644
6:78
0.66
0.45
1.00
3.49
346
645
644
porrdlppi 8
0.4
LO 6m
0:1
1.30
2.70
14
2.96
8.3
0.45
644
1.00
0.57
0.99
3.50
3.47
3.03
Alan
+222
24
40130
H6:00 12.32
0.0
(b) Importance agreement.
Figure 1.: X-SKIT agreement results the Imagenette dataset across 8 different vision-languse models as function of conditioning set size, s. Results are reported as means and standard deviations over the random 20 images used in the experiment.
level a. Table E.7 summarizes rank and importance agreement as function of conditioning set size (i.e., the number of concepts in S), and Fig. E. 14 includes all pairwise agreement values. We can see that ranks are generally well-aligned across models, and that agreement slightly decreases as the number of conditioning concepts increases. Finally, Figs. E. and E. include results with X-STIT for 2 images from three classes in the dataset across all models used in the experiment, and Fig. E.17 summarizes ranks across models on the same images.
41

---

CUP-AN50
CUPNT88/22
CUPVITTU14
sprre
Spond
90016
10300
38889
1899
<20n
1555
24t2
ycorgater
tomppe
1sking
[ure
cothetral
cathesrel
Calfed
yomgsper
0.0
OsentipMii-4-4 34
0.5
FLAVA
0.5
NOTTA M
0.5
DUP
0.5
1.3
50on
000
tumy
cattedra
2113
Inin
Tatang
mac
cathedral
0.5 CUP-AN50
CUP-NIT
0.5
CLP VUR-VITUO 1
0.5
100
OpanC.loo VIVV
0.5
1.0
12:29
1000
mar
palch
Isunched
barge
helping
burgu
tening
tumpet
(oleng
recoue
145
mimort
las
Opanthp/M
2017
SLUP
barge
cachecral
help
cattedra
0.0
0.5
1.0
hel
0.0
0.5
1.0
0.5
11
Iumpett
0.5
1.0
CUPHINS0
52
band
3and
sand
(6h
Eathedreel
heilds
gradinte
cathecral 01
cboskots
cathed
gracuste
Icmma
obockate
squadl
0.0
0.5
0.5
1.0
0.5
0.5
5450
quiiss
usonrir
prom
bogg
Stom
bas
bost
bass
fymo
cothedta
dees
ctesieee
formal
obeoler
10
eathedral
graduste
cathecra
divertet
chiclere
CLP-
CLPPA11-11222
CLPMM T4//44
OpenC: L-VVT-B--
trurppr
t'rrppet
trumpet
(sh
cattedral
base
obovete
dbockere
(110
politinn
chip
2000 nd
mijor
20
5.3
10
1
0.5
F V/A
0.5
1.0
0.0
NIGG
0.5
1.0
0.0
SPP
0.5
14
Surppe
bumpst conbucted
trumpst conducted
Inst
contucted
Issdite
assdate
bas
Call
stockis
clockec
2400
conturted
75h
eattedrs
conredo (1)
0.0
0.5
1.0
0.
0.5
1.0
Figure E.15: Local importance results with SKIT and CLIP.ViT-L/14 for 2 images from three classes in the Imagenette dataset (part of II.
42

---

CJP-MT88222
CURVITUL4
OpenClesViT
Rew
decoending
despeedin
06w
departs
lunch
tander
instucter
osparts
londen
Eahing
tectbook
cathedral
80337
cathedrel
5pting
enghon
catichal
loting
insrumman
FLAVA
ALIGN
BLIP
luuch
luurch
Landon
london
desuarding
Bun
dtocording
Landum
irotructr
ong
10 rg
Inceumed
regand
Indrument
bend
ongland
Rihing
cathed's
bedbbok
Castedra
trothoot
england
teutbook
band
9/0
0.5
0.0
0.5 CUPMMT302
0.0
0.5
1.0
0.5
1.0
CLP AN50
CUPNIT-U14
OpenChiiNTTa
s88ng
50010
Sungoy
Ealerg
1olerg
sufbice
ANTUUS
bant
Entiig
Iead
catteetra
Bard
sciecce
aurgery
0.3
0-1
1.0
0.5
OpenCip/MMA-114
Instrumen:
Rakhing
cxtlud
Foting
Cothedtal
Insrumer
cettedral
bend
Figure F.16: Local importance results with SKIT and CLIP:VVT-L.14 for images from three classes in the Imagenette dataset (part II of II).
festrve
Sirk
Figure E.17: Summary of X- SKII ranks of importance across all models for 2 examples images from 3 classes in the Imagenette dataset.
43

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, we state that we introduce precise definitions of semantic importance for the predictions of black-box models, and novel procedures test for each of them. These are the contributions of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Throughout the paper, we clearly state what guarantees our proposed methods provide, and what they do not. We comment possible future directions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
44

---

Answer: [Yes]
Justification:
We provide complete proofs and background information for each claim.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide pseudocode for all algorithms used in the paper, detailed descrip- tions of the data generating processes for synthetic datasets, links the publicly available datasets that were used, description of hyperparameters used for training and testing models, and link GitHub repository to reproduce all experiments presented in the paper. Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
45

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide all code necessary reproduce the experiments presented in this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
CCodeeeodeeeeeeeeeeeeeee wwwissssssssssiiiiieooiooo onsiiniyoccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We dedicate an appendix the full experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report p-values and uncertainty estimates wherever they are appropriate.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
46

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify all computational resources used in this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: https:// /ttpar////eeu../eeeeeeeeeee wwwpeuuipp..cpppppieepepp wtw.:/hhhiiiiiiieieeeeenneeee
Answer: [Yes]
Justification: The work presented in this paper did not involve human subjects, and all data used was from current publicly available datasets.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The objective of this paper to develop statistically-valid methods to study the semantic importance of numan-interpretaal concepts for the predictions of black-box models. We envision this research will support better practices towards a responsible use of artificial intelligence in society.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
47

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The presented paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We stated all original references of papers and code repositories used for this paper, inclusive of their respective licenses.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
48

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
The submitted paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The submitted paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The submitted paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
49

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
50


Reviews:
[{"review_id": "30vPA3b1w6", "reviewer": "Reviewer_go3R", "summary": "The paper discusses the need for precise statistical guarantees in feature importance, especially for semantic concepts, to ensure transparency and avoid unintended consequences.\n\nIt introduces a framework using conditional independence for testing semantic importance and demonstrates its effectiveness on synthetic datasets and image classification tasks using models like CLIP.\n\nIt uses principles of testing by betting (or sequential testing), which are based on e-values, and MMD as a test statistic\n\nThe paper finds importance of concepts via conditional independence\n\nThe authors introduce two novel procedures: conditional randomization SKIT (C-SKIT) for global conditional importance and explanation randomization SKIT (X-SKIT) for local conditional importance.", "questions": "The practical implementation assumes a small set of concepts. Do you find it to be enough?\n \n What are the computational limits of the proposed method?  How do they scale with increasing data size and complexity? \n \n For what data modalities is the given choice of kernel functional?", "limitations": "The practical implementation assumes a small set of concepts. \n \n For certain tests, accurate generative models for the conditional distributions are required, which can be difficult to train.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Offers rigorous definitions and tests for global and local semantic importance\n\nIt emphasizes the importance of interpretable features in black-box models\n\nThe paper is well-written and smoothly explains its argumentation.", "weaknesses": "The paper does little comparison to other state-of-the-art techniques for feature importance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thank you for your comments and questions!\n\nWe address each point individually, and we are looking forward to discussing with the reviewer to answer any outstanding questions.\n\n---\n\n### **Comparison with SOTA**\n\nWe thank the reviewer for this comment, which has significantly strengthened our experimental results. We have described several additional experiments and comparisons with PCBM (which is the state-of-the-art for semantic explanations) in the general rebuttal. To summarize, we find that:\n\n- c-SKIT provides ranks that are more transferable across different vision-language models on both Imagenette and AwA2.\n- c-SKIT provides better semantic importance detection in terms of $f_1$ score on AwA2.\n\nFor local semantic explanations, we have included additional experiments on CUB-200-2011, which indicate x-SKIT is well-aligned with ground-truth ($\\approx 0.85$ average $f_1$ score). We note that, currently, there are no alternative methods that can produce local semantic explanations, hence why we could not compare.\n\nFor the sake of completeness, for global conditional importance, we tried comparing with LaBo (Yang et al., 2023), which, intuitively, adds a softmax activation to the weights of a PCBM classifier. This approach, however, fails to learn good predictors in our few-concepts setting ($\\approx 20\\\\%$ classification accuracy on AwA2 dataset compared to $\\approx 95\\\\%$ for PCBM).\n\n---\n\n### **Small number of concepts**\n\nWe are not sure we fully understand the question. Could the reviewer expand on what they mean by *``enough''*?\n\nOur proposed framework does not modify the original black-box predictor, whose accuracy does not depend on the number of concepts. So, defining a small set of concepts does not affect the performance of the predictor.\n\nOn the other hand, it is true that the number of concepts may thwart the ability to build effective samplers to instantiate our c-SKIT and x-SKIT tests. For example, in the case of limited data, it may not be feasible to train a conditional sampler on a large set of concepts. In our experiments, we focus on $\\approx 20$ concepts because previous work [1] has shown that humans prefer succinct explanations. We found that this assumption allows us to use non-parametric samplers which are fast, cheap, and do not require prior training.\n\nWe will clarify this in the revised version of the manuscript, thank you.\n\n---\n\n### **What are the computational limits? How do the tests scale with increasing data size and complexity?**\n\nWe thank the reviewer for these questions.\n\nFirst, the main computational limit is the need of conditional samplers. In certain domains, these models may be expensive both to train and run, such is the case for diffusion or language models. In our experiments, we strived to use methods that are effective but do not require prohibitive computational resources.\n\nSecond, the computational complexity depends on the specific test and the sampler. In particular:\n\n- SKIT: Following [2, Appendix F.2] the test runs in $O(\\tau^2)$, where $\\tau$ is the (random) stopping time of the test. This is because at each step $t$, computing the MMD requires summing over the previous $t-1$ terms.\n- c-SKIT and x-SKIT: both tests incur in a extra factor of $T_n$, which represents the cost of the sampler on $n$ data points.\n\nFinally, we note that the runtime does not depend on the number of concepts because different concepts can be tested simultaneously. We will include this in the revised version of the paper.\n\n---\n\n### **For what data modalities is the given choice of kernel functional?**\n\nThis is a great point!\n\nOur framework assumes the black-box predictor can be divided into an encoder and a classifier, and these need to be appropriate for the data modality at hand. Once inputs are mapped to an embedding space, one needs to decide on which kernel to use to test for semantic importance.\n\nIn general, when using kernel methods to test for a null hypothesis, we would like the kernel to be *characteristic* for the alternative (see [2]). That is, we want the kernel to be expressive enough to distinguish two distributions coming from the alternative. Over compact domains, this can be achieved by using *universal* (see [3]) kernels---such as the RBF kernel.\n\nIn practice, this means that:\n\n- for x-SKIT, the RBF kernel is appropriate whenever the classifier is a real-valued function (e.g., linear classifier with sigmoid activation). For discrete predictors (e.g., decision trees), other kernels may be necessary.\n- for c-SKIT and SKIT, the RBF kernel is appropriate whenever both the predictor and the concept bottleneck layer are real-valued functions (e.g., linear classifier with sigmoid activation and SVMs, respectively).\n\nWe remark that our tests are defined for any choice of kernel, and they can be instantiated directly for the desired data modality. \n\nFinally, we refer the reviewer to Fig. E.3 in the Appendix, where we precisely compare c-SKIT with a linear and RBF kernel on a synthetic dataset. This experiment shows that a linear kernel may fail to detect important concepts because it does not satisfy the universality property.\n\n---\n\n**References**\n\n[1] Ramaswamy et al.  \"Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability.\" (2023)\n\n[2] Podkopaev et al. \"Sequential Kernelized Independence Testing.\" (2023)\n\n[3] Sriperumbudur et al. \"Universality, Characteristic Kernels and RKHS Embedding of Measures.\" (2011)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ku9AoZH295", "reviewer": "Reviewer_mYwv", "summary": "The paper defines statistical importance of semantic concepts for black-box models such as CLIP via conditional independence. \nThis is motivated by the fact that users would be interested to know how they should interpret two concepts with different importance scores, and if the difference in such two concepts has any statistical significance.\nThe paper discusses that earlier work considers input features only, and not directly applicable to semantic concepts, which are represented in the internal layers of models. \nThe paper utilizes recent advances in sequential kernelized independence testing to develop statistical tests that can produce a rank of importance for each semantic concept.\nThe proposed method is experimentally verified using CLIP and a subset of ImageNet data.", "questions": "CLIP has an issue with compositional understanding (e.g. Hsieh et al. \"SugarCrepe\", Sec 5.3), since the embedding space that is learned through CLIP's contrastive loss is incentivised to only match up to the set of concepts between images and texts, rather than learning the relational / compositional structure between concepts in the image / text. I'm wondering if the ranking order discovered by this method also suffers from this issue of CLIP's representations.", "limitations": "- Experiments on real datasets are limited, and it would be interested to know how much transferable the results of CLIP to other vision-language models.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- How to measure statistical significance between different semantic concepts' importance is well-motivated (e.g. \"how should users interpret two concepts with different importance scores?\", \"Does their difference in importance carry any statistical meaning?\" are valid questions that users would be interested to explore)\n- Proposed two novel procedures to test for semantic importance: c-SKIT for global conditional importance, and x-SKIT for local conditional importance \n- Validated their method on zero-shot ImageNet classification with CLIP.", "weaknesses": "While the proposed solution for discovering feature/concept importance of black-box models is written as a general framework, it's not clear how applicable this method is to problems in practice, because the real-data experiment is only performed on CLIP. For example, users in practice would probably be interested in seeing how much the rankings produced by the proposed method agree across CLIP and other vision-language models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thank you for your comments and questions!\n\nHere, we address each point individually, and we are looking forward to discussing with the reviewer.\n\n---\n\n### **Limited experiments and transferability across different vision-language models**\n\nThis is a great question, and we thank the reviewer for raising this point. These additional experiments have strengthen the experimental results of our submission, and uncovered interesting aspects of transferability. \n\n**In particular, we have extended our experimental results to include 2 additional datasets (AwA2 and CUB) and 8 different models (both CLIP- and non-CLIP-based): CLIP:RN50, CLIP:ViT-B/32, CLIP:ViT-L/14, OpenClip:ViT-B-32, OpenClip:ViT-L-14, FLAVA, ALIGN, and BLIP.**\n\nWe evaluate agreement between pairs of models in terms of ranking of concepts, and whether they are classified as important or not. To compare ranks, we use a weighted version of Kendall's tau (see [1]) which assigns higher penalties to swaps at higher positions. That is, for example, a 1 -> 5 swap is worse than a 4 -> 5 swap. This reflects that higher positions should matter more and be more stable. To compare importance, we threshold rejection rates at level $\\alpha$ and compute the accuracy between the binarized vectors.\n\nWe briefly summarize here the findings of our experiments, which are described in the general response:\n\n- Ranks obtained with c-SKIT are more transferable than PCBM on both Imagenette and AwA2.\n- Both CLIP- and non-CLIP-based models are generally aligned in terms of ranks and importance, especially on local explanations.\n\n---\n\n### **Compositional understanding of CLIP's embedding space**\n\nThis is a very interesting point, which deserves further investigation outside of the current submission. \n\nFor example, following the SugarCrepe example of a photo with *\"a girl in white facing a man in black\"*, one could test whether the prediction of *\"girl\"* does depend on the concepts *\"white\"*, *\"facing\"*; while the prediction of *\"man\"* depends on *\"black\"*. We stress that this study would not be possible without the tools presented in this submission.\n\nWe would also note that our results on global (marginal) importance find that concepts are almost always important (i.e., their rejection rates are above $\\alpha$). As suggested by the reviewer, this finding may support the claim that CLIP's semantic space is entangled and overlapping. These aspects have also been considered by previous works (for example, MERU [2]), which focus on whether using an embedding spaces different from the unit sphere (e.g., the hyperbole) induce better hierarchical representations. In a similar fashion to above, one could also devise an experiment to retrieve and test semantic image traversal as in [2, Figure 5].\n\nThese example studies highlight that the framework presented in this submission has a broad reach beyond explainability and it will support other research efforts, thank you for raising this point.\n\n---\n\n**References**\n\n[1] Vigna. \"A weighted correlation index for rankings with ties.\" (2015)\n\n[2] Desai et al. \"Hyperbolic image-text representations.\" (2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9dSxjqZOQz", "reviewer": "Reviewer_nqVJ", "summary": "Recently, there has been a lot of interest in understanding the inner workings of deep neural networks. Most existing works learn semantic concepts that are inherently understandable to the user. Often, each semantic concept comes with an associated score, and in many cases, it is hard to interpret the scores. Even though some works address this issue, they only work for input features and don't easily apply to semantic concepts. The work aims to address this shortcoming and formalizes the notion of statistical importance for both local and global semantic concepts.", "questions": "Refer to Weaknesses.", "limitations": "Yes, the authors discuss the limitations of the proposed technique in detail. They have sufficiently addressed it, and the contributions outweigh the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The work formalizes the notions of global, local, and global conditional importance. \n- Most existing methods assume the presence of a large bank of concepts; the proposed method allows the stakeholders to specify the concepts they want to evaluate directly. This flexibility will enable explanations with diverse semantics for the model prediction on an example, as opposed to a single explanation.\n- In practice, most of the methods rely on the weights of a linear model over the concepts to convey the importance of the semantic concepts. In contrast, since the proposed method utilizes statistical significance, it can guarantee false positive rates.\n- Unlike existing methods, the proposed technique doesn't rely on training a surrogate linear model and can study the semantic importance of any given model.", "weaknesses": "The paper is generally sound regarding the contribution, but the authors should consider the questions below regarding the experiment.\n\n- First, I encourage the authors to evaluate their technique on diverse tasks and datasets. For instance, they can evaluate their method on datasets like AwA2 and CUB, which already have concept annotations. In addition, the authors could assess their methods on domains like NLP.\n\n- Why only consider a single backbone? I encourage the authors to include results from diverse backbones, strengthening the paper. \n\n- One of the paper's main contributions is the ease with which an end-user can understand the semantic concepts and their importance, but that aspect still needs to be evaluated. The authors should design & conduct a user study to assess it. \n\n- In addition, the authors can validate the concept's usefulness by intervening on the concepts and measuring the changes in model predictions. Another way to validate the concepts would be to measure their predictive ability; ideally, just using the important concepts to predict shouldn't hinder the model's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thank you for your comments and questions!\n\nWe now address each point raised by the reviewer individually, and we are looking forward to clarifying any outstanding questions.\n\n---\n\n### **Diverse datasets and models**\n\nWe thank the reviewer for their suggestions, which have significantly strengthen our experimental results and uncovered interesting findings.\n\n**As presented in the general response, we have now included additional experiments on both AwA2 and CUB-200-2011, comparing across 8 different models: CLIP:RN50, CLIP:ViT-B/32, CLIP:ViT-L/14, OpenClip:ViT-B-32, OpenClip:ViT-L-14, FLAVA, ALIGN, and BLIP.**\n\nIn particular:\n\n- Since AwA2 includes class-level annotations, we have included global conditional importance results with c-SKIT in comparison with PCBM.\n- Since CUB includes image-level annotations, we have included local conditional importance results with x-SKIT.\n\nWe describe all findings and experiments in the general response at the top of our rebuttal. To summarize, we find our c-SKIT outperforms PCBM both in terms of semantic importance detection and transferability across all models on both Imagenette and AwA2. Furthermore, x-SKIT importance ranks align well with ground-truth annotations ($\\approx 0.85$ average $f_1$ score) on CUB.\n\nWe consider NLP and language generation applications of our framework as important future directions. For example, it is still not completely clear how ideas of concept bottleneck models apply to encoders such as BERT. How do text encoders represent semantic information with standard pretraining techniques such as masked language modeling? Only very recent work [1] has started addressing these fundamental questions. Going beyond text encoders, how should one phrase questions of semantic importance for contrastive encoder-decoder architectures such as CoCa [2] or autoregressive models such as GPT? The structured nature of these models raises very important questions that currently remain unanswered, thank you for raising this point.\n\nWe will include these discussion points in the revised version of the manuscript.\n\n---\n\n### **User study to evaluate alignment with human intuition**\n\nAs stated in the general response, we agree with the reviewer that a user study would be necessary to claim alignment with human intuition, and we will rephrase those claims in the revised version of the manuscript. However, we remark that our tests *can* be readily applied to any set of concepts, including user-defined ones.\n\nWe believe the design of a robust user study to deserve its own investigation, and, in this submission, we focus on introducing the methods that will enable such a study. We stress that, currently, no alternative method *can* work with a few user-defined concepts. For this reason, we envision our method to enable the design of such studies.\n\n---\n\n### **Validation of concept usefulness**\n\nWe thank the reviewer for their suggestions.\n\nWe validate important concepts in our additional experiments on AwA2 and CUB using the ground-truth annotations. We refer the reviewer to our general response and the attached pdf for numerical results and comparison with PCBM.\n\nFurthermore, we would like to mention that our c-SKIT and x-SKIT tests precisely work by resampling concepts and measuring their effect on distributions of the output of the predictor. However, we remark that we use *observational* conditional distributions (i.e., $Z_j \\mid Z_{-j}$) and not *interventional* distributions (i.e., $Z_j \\mid \\text{do}(Z_{-j} = \\cdot)$). Characterizing the connections between our framework and causal inference is an interesting future line of research, thank you for raising this point.\n\nLastly, we kindly push back on the suggestion of validating concepts by training predictors, as our focus is to study which concepts are important for a fixed black-box model, and, in the general case, these may be different from the ones with highest predictive ability. This is a fundamental distinction between our approach and alternatives like PCBM. Intuitively, consider ImageNet classification. It might be the case that we want to test one concept only. But of course, even if that individual concept were truly used by the model, it will not be enough to train a good classifier from scratch. We can already see this behavior in the AwA2 experiments: restricting to 10 concepts, that we know from ground-truth annotations should be important, significantly reduces predictive power. We refer the reviewer to the \"average\" line in Table 1 of the attached pdf, where PCBM shows a drop in performance of around 4%.\n\n---\n\n[1] Tan et al. \"Interpreting pretrained language models via concept bottlenecks.\" (2024)\n\n[2] Yu et al. \"Coca: Contrastive captioners are image-text foundation models.\" (2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TSu1ko23aI", "reviewer": "Reviewer_FuBo", "summary": "The paper presents a method to test feature importance when a model makes its decision. The proposed method is based on hypothesis testing. The features concerned in this work is human-interpretable ones. For example, when CLIP makes a \"cat\" prediction to the image, the features tested are: \"whiskers\", \"pointy ears\", etc. The paper claims several novel contributions: (1) the method does not depend on the existence of dense features dictionary -- and by principle can take features as input from the users (2) the method keep the original predictor, unlike existing methods that train a separate predictor.", "questions": "Adding an actual \"feature from user input\" evaluation can significantly strengthen the claim. + see weakness", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "1. Conceptually very important and interesting, especially on the notion of sample-speficic vs. global vs global conditional\n2. *if* really works on input features from user, is a very important contribution", "weaknesses": "Although the proposed method's efficacy and claims are very interesting, i find several major concerns:\n1. the method part is very hard to understand -- especially on the equation part. I fins it hard to understand what does the equations mean, and how that connects to by rejecting the hypothesis, means that the prediction depends on the feature? the writing on method section can use a lot more explanation.\n2. The evaluation is not convincing. The result presented claims that the feature importance outputted by the algorithm agrees with intuition -- this is very subjective. The evaluation can be significantly strengthen by human evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thank you for your comments!\n\nWe thank the reviewer for their encouraging comments. Here, we address each weakness point individually, and we are looking forward to discussing more.\n\n> *if* really works on input features from user, is a very important contribution.\n\nWe do want to stress that our method *does* work with any set of concepts, and in particular it is the only method that can provide local semantic explanations for the predictions of a black-box model. So, we do appreciate the reviewer's comment that this is a very important contribution!\n\n---\n\n### **Clarity of presentation**\n\nCould the reviewer expand on their points of confusion? We would be more than happy to clarify methods and equations in the revised version of the manuscript.\n\n---\n\n### **Alignment with human intuition**\n\nWith agree with the reviewer that a user study needs to be performed to claim alignment with human intuition. As we state in the general response at the top of our rebuttal, we will smooth such claims in the revised version of the manuscript. We note that our contributions *can* readily work with any set of concepts, including user-defined ones, and that no alternatives currently exist.\n\n**We have included several additional experiments to strengthen the experimental section of our submission and validate the ranks of importance obtained with our methods.**\n\nIn particular, we have included the AwA2 and CUB datasets, which have ground-truth annotations of which concepts are present in images coming from certain classes or specific images, respectively. We refer the reviewer to our general response at the top of our rebuttal for a detailed description of the additional experiments and their results.\n\nTo summarize our findings:\n\n- c-SKIT has better semantic importance detection performance compared to PCBM on AwA2.\n- x-SKIT has good alignment with ground-truth annotations across all models ($\\approx 0.85$ average $f_1$ score) on CUB.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Thank you for your comments!

We sincerely thank all reviewers for their valuable comments and suggestions, which have strengthen our experimental results and the presentation of our contributions. 

**Following all reviewer comments, we have significantly extended our experiments on real world data: we now include additional results on both the AwA2 and CUB datasets, and we compare across 8 different vision-language models.**

In this general response, we address common questions raised by several reviewers, and briefly summarize additional results presented in the rebuttal pdf. We address comments from each reviewer in their individual response, and we are looking forward to discussing with the reviewers to answer any outstanding questions.

---

## Imagenette results on different models

**As suggested by RmYwv, RnqVJ, and Rgo3R, we have extended our analysis to include 8 different models: CLIP:RN50, CLIP:ViT-B/32, CLIP:ViT-L/14, OpenClip:ViT-B-32, OpenClip:ViT-L-14, FLAVA, ALIGN, and BLIP.**

We evaluate agreement between all pairs of models with the following metrics:

- *Comparison of ranks.* We use the method of [1] to compare ranks. A value of -1 means reverse order, and +1 means perfect alignment.

- *Comparison of importance.* We threshold rejection rates at level $\alpha$ to classify concepts as important or not. Importance agreement is the accuracy between the binarized vectors.

We summarize the results included in the pdf:

- Fig. 1, average global importance agreement equal to 0.57 (random baseline = 0.00).
- Fig. 2, comparison of c-SKIT and PCBM. **We find c-SKIT ranks to have higher agreement (0.64) compared to PCBM (0.52), which indicates c-SKIT is more transferable than PCBM.**
- Fig. 3, average local conditional importance agreement is 0.73 and rank agreement is 0.62. **These results indicate that different vision-language models share certain local semantic dependence structures.**

---

## Global conditional importance on AwA2

As suggested by RnqVJ, we include global conditional importance results on the AwA2 dataset. We use c-SKIT on the top-10 best classified classes for a fair comparison across models. For each class, we test 20 concepts: 10 present, and 10 absent according to the ground-truth annotations. 

We compute rank agreement and report $f_1$ scores between the ground-truth annotations and the top-10 concepts according to c-SKIT rejection times and PCBM absolute weights. We remark that the coefficients of a linear classifier are a heuristic notion of global conditional independence, whereas our tests provide precise statistical guarantees.

We summarize the results included in the pdf:

- Fig. 4, comparison of ranks obtained with c-SKIT and PCBM. **Similarly to above, we find c-SKIT ranks to be more transferable than PCBM's (0.54 vs 0.44 agreement)**.
- Table 1, $f_1$ scores for c-SKIT and PCBM. **c-SKIT consistently outperforms PBCM across all models (0.55 vs 0.48 average $f_1$ score). We stress this improvement in semantic importance detection does not reduce classification accuracy (99.50% vs 95.10%)**.

---

## Local importance on CUB

As suggested by RnqVJ, we include local importance results on the CUB-200-2011 dataset. We use x-SKIT on 2 test images from the top-10 best classified classes for a fair comparison across models. For each image, we test 14 concepts: 7 present, and 7 absent according to the ground-truth annotations. We threshold rejection rates at level $\alpha$ to classify concepts as important or not.

- Fig. 5, rank and importance agreement. **We find an average importance agreement of 0.97 and an average rank agreement of 0.86.**
- Table 2, $f_1$ scores as a function of size of conditioning set, $s$. **We find an average $f_1$ score of $0.84, 0.86, 0.83$ for $s \in \\{1,2,4\\}$. OpenClip:ViT-L/14 has the highest $f_1$ scores across all values of $s$, with a maximum of 0.89 for $s=2$.**

These results suggest models are aligned well with the ground-truth annotations.

- Fig. 6, example image with local importance ranks across different models.

---

## User study to evaluate alignment with human intuition

We agree with RnqVJ and RFuBo that a user study would be necessary to claim alignment of explanations with human intuition. **We will smooth these claims in the revised version of the paper in order to highlight that the scope of this work is to introduce a statistically-rigorous method that *can* work with any set of user-defined concepts**. 

Finally, we would like to remark that, currently, there are no alternatives to design such a study but our framework. In fact, we envision many studies to leverage the precise statistical guarantees provided by our methods.

---

## FDR control

For the sake of completeness, we have also addressed FDR control limitations, as we mentioned in the submitted manuscript. We have extended our results to report important concepts with FDR control at level $\alpha$. We will include this in the revised version of the manuscript.

---

**Details**

AwA2 classes: giant panda, tiger, giraffe, zebra, lion, squirrel, sheep, horse, elephant, dalmatian.

CUB classes: White Pelican, Brown Pelican, Mallard, Horned Puffin, Vermilion Flycatcher, Northern Flicker, Cardinal, Blue Jay, Cape Glossy, Starling, Frigatebird.

**References**

[1] Vigna. "A weighted correlation index for rankings with ties." (2015)

Meta-review: All reviewers unanimously recommended acceptance. It is a neat, well-written work that adds value to the literature on interpretability. The new experiments provided in the rebuttal have additionally improved the paper a lot, so please make sure to include them in the camera-ready version.
Decision: Accept (poster)