Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions
Authors: Hubert Baniecki
Maximilian Muschalik
Fabian Fumagalli
Barbara Hammer
Eyke Hüllermeier
Przemyslaw Biecek
Venue: NeurIPS 2025 poster ()
Keywords: explainability
feature interactions
Shapley interactions
Banzhaf value
CLIP
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We introduce faithful interaction explanations of CLIP and SigLIP models (FIxLIP), offering a unique perspective on interpreting image–text similarity predictions.

Abstract: Language-image pre-training (LIP) enables the development of vision-language models capable of zero-shot classification, localization, multimodal retrieval, and semantic understanding. Various explanation methods have been proposed to visualize the importance of input image-text pairs on the model's similarity outputs. However, popular saliency maps are limited by capturing only first-order attributions, overlooking the complex cross-modal interactions intrinsic to such encoders. We introduce faithful interaction explanations of LIP models (FIxLIP) as a unified approach to decomposing the similarity in vision-language encoders. FIxLIP is rooted in game theory, where we analyze how using the weighted Banzhaf interaction index offers greater flexibility and improves computational efficiency over the Shapley interaction quantification framework. From a practical perspective, we propose how to naturally extend explanation evaluation metrics, such as the pointing game and area between the insertion/deletion curves, to second-order interaction explanations. Experiments on the MS COCO and ImageNet-1k benchmarks validate that second-order methods, such as FIxLIP, outperform first-order attribution methods. Beyond delivering high-quality explanations, we demonstrate the utility of FIxLIP in comparing different models, e.g. CLIP vs. SigLIP-2.

Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions
Hubert Baniecki"
Maximilian Muschalik LMU Munich, MCML
Fabian Fumagalli Bielefeld University
University of Warsaw
Warsaw University of Technology
Barbara Hammer
Eyke Hüllermeier
Przemyslaw Biecek
Bielefeld University
LMU Munich, MCML, DFKI
University of Warsaw Warsaw University of Technology
Abstract
Language-image pre-training (LIP) enables the development vision-language models capable of zero-shot classification, localization, multimodal retrieval, and semantic understanding. Various explanation methods have been proposed to visu- alize the importance of input image-text pairs the model's similarity outputs. However, popular saliency maps are limited by capturing only first-order attribu- tions, overlooking the complex cross-modal interactions intrinsic such encoders. We introduce faithful interaction explanations of LIP models (FIXLIP) as a unified approach to decomposing the similarity in vision vision-language encoders. FIxLIP is rooted in game theory, where we analyze how using the weighted Banzhaf inter- action index offers greater flexibility and improves computational efficiency over the Shapley interaction quantification framework. From a practical perspective, we propose how naturally extend explanation evaluation metrics, such the pointing game and area between the insertion/delltion curves, to second-order interaction explanations. Experiments on the MS COCO and ImageNet-Ik bench- marks validate that second-order methods, such as FIXLIP, outperform first-order attribution methods. Beyond delivering high-quality explanations, we demonstrate the utility of FIXLIP in comparing different models, e.g. CLIP vs. SigLIP-22.
Introduction
Contrastive language-image pre-training [CLIP, 57] revolutionized computer vision by learning data representations well-performing in a plethora of downstream tasks. Scaling the training to predict similarity between image and text, combined with continuous architectural improvements [e.g. SigLIP, 75] and increasing the size of datasets, leads to the development of powerful vision-language encoders [VLEs, 13, 66, 68, 75]. These rather opaque encoders become the backbone components in large lasionnl models capable of actual conversation and reasoning [2, 36, 43], and are increasingly applied in high-stakes socision-making like in the case of medical imaging [29, 76, 78].
The limitations in the way CLIP represents the visual world have been extensively studied [14, 35, 67, 72, 73]. For example, recent study by Tong et al. [63] suggests vision- language models limited by the systematic shortcomings of CLIP, e.g. they fail when questioned about directions, quantity of objects in image, recognizing the presented text in visual form. The emergence of "CLIP-blind pairs", inputs that CLIP perceives as similar despite their clear differences, could lead to critical errors when applied in medical visual question answering and treatment recommendation [78].
Corresponding author: h. baniecki@uv edu.pl
Code: https: / thub. com com//bbanieck//fixl..
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

cross-modal sampling (m- m²)
explanation of cross-modal intra-modal interactions
input
vision enesder
regression--usse upprosimation
black
WLS
dog
hydr
peicaak
-23
enooder
dog
yellow
an:
O o-weighted masking
hydr
black dog next to a yellow hydrant similarity: 0.32
-B
dogg
black
black dog max: hydrran
sydrast
Ii yulley
p=0.3
p=0.5
p=0.7
positive negative subsets
Figure 1: Explaining similarity in language encoders with weighted Banzhaf interactions. We propose cross-modal sampling strategy to efficiently query the model for m² game values from m coalitions, and p-weighted masking to circumvent querying the model on out-of-distribution inputs. regression-basee bbased approximation with weighted least squares (WLS) of second-order attributions gives a faithful decomposition of the predicted similarity score. Explanations of cross- cross-modal and intra-modal interactions can be visualized and analyzed to interpret the CLIP's similarity prediction. The red values denote positive interactions contributing to an increase in similarity, while blue denotes interactions between tokens contributing decrease in similarity.
One way of validating similarity predictions at inference time explaining the encoders with saliency maps [12, 39, 49, 70, 77J. However, these first-order importance scores are limited by their unimodal view of the model's output [1]. Vision-language encoders cannot be faithfully explained without taking into account the cross-modal interactions between image-text pairs [32, 40].
We thus propose agdel-ag approach to interpreting VLEs based perspective where input tokens form coalitions of players in a cooperative game [19, 44). In doing that, extend the faithful Banzhaf interaction index [64] weighted case [47], addressing the emerging requirement for controllable sparsity in coalition sampling [6, 25, 27, 79]. Specifically, sparse inputs with many masked tokens become out-of-distribution, i.e. images become unrecognizable and text captions become ambiguous. Figure shows a high-level view of our approach. We sample uniformly m² variations of the input image-text pair. Each mask is sampled uniformly with probability p for each token in mask, and we take all pairwise combinations between m masked images and m masked texts. Then, we apply the explained model to predict similarity values for each of the input variations. Finally, we apply weighted least squares to approximate the model's predictions from sampled binary masks. The resulting explanation assigns attribution scores to each token and interaction scores to each pair of tokens in an image-texy text pair.
game-theoretical
Contributions. Our work advances literature in multiple ways: (1) game-theoretic explanation of vision-language encoders. We introduce faithful interaction explanations of LIP models (FIXLIP) that provide unique perspective decomposing the similarity predictions of VLEss. Our results highlight the necessity to consider cross-modal interactions between text and image inputs for an accurate model interpretation, beyond the first-order attribution. In fact, this the first work argue for using weighted Banzhaf interactions in this domain to overcome the out-of-distribution problem apparent in removal-bas based explainability. (2) Efficient computation allows scaling larger models. We propose cross-modal sampling strategy for approximating FIXLIP that improves its efficiency by 5-20x over traditional Shapley interaction quantification. We further scale the regression-bassee approximation of interactions to hundreds of tokens by prioritizing their subset for faithful explanation. (3) Evaluation metries for second-order interactions. We extend explanation evaluation metrics-tie the pointing game and area between the insertion/deletion curves-to explanation methods higher order. Experiments on the MS COCO and ImageNet- benchmarks validate that second-order methods, such FIxLIP, outperform first-order attributions. (4) FIXLIP facilitates

---

various approaches to understanding vision-language encoders. Finally, we demonstrate the utility of FIXLIP in applications like a model-aggostic comparison between CLIP and SigLIP-2.
2 Related Work
Attribution explanations of CLIP. Multimodal explanations have been studied in various tasks, including visual question answering [28, 55], visual reasoning [46], video classification [71], and sentiment analysis [69]. Our work considers explaining the bi-modal encoder prediction in the means of input attributions, specifically image and text token attributions of similarity predictions in CLIP. In this context, existing gradient-based [77], attention- [12, 39), and information-based [70, 80] methods approximate only first-order attributions visualized as saliency maps. Yet, Liang et al. [40] perform extensive user studies showing that visualizing the second-order attributions (pairwise interactions) is necessary for understanding complex multimodal models. Joukovsky et al. [32] propose to approximate these interactions with bilinear models [also for image-image encoders, 61]. We formalize the problem the means of game theory and propose an efficient approximation cross- modal and intra- modal interactions (Figure 1), discussing its several desirable theoretical properties. In concurrent work, Moeller et al. [48] propose to approximate only cross-modal interactions based on the Integrated Hessians methodology [30]. Our proposed evaluation metrics for interaction- explanations can be used to empirically cross-compare the above- mentioned attribution explanations.
Mechanistic interpretability of CLIP. Our research is orthogonal to work on the concept- based interpretability of CLIP's s internal representations. Research in this direction considers explaining particular neurons [22], concept-basee based image classification [60], linear probing [5], and training sparse autoencoders [41, 74). Gandelsman et al. [21] investigate how the individual model's s components affect its final representation. Beyond CLIP, Balasubramanian et al. [4] analyze interpreting image
representations via text in alternative vision transformer architectures [e.g. DINO, 11].
Shapley and Banzhaf interactions in machine learning. We build on the developments using game-theoretic interaction indices to understand complex machine learning models [9, 19, 56, 62, 64]. Our goal in this work is not to compare with the most recent advancements in approximating the interaction index [16, 33, 52], but rather to faithfully explain the similarity predicted by VLEs. From the first-order perspective, Parcalabescu and Frank [53, 54] apply the Shapley value to measure the importance each modality has on various tasks solved by CLIP. Tsai et al. [64] and Fumagalli et al. [20] propose weighted least squares regression approximation of higher-order interactions, which we incorporate as baseline our methodology. Recently, Kang et al. [34] applied a sparse Fourier transform to scale feature interaction explanations for large language models, while we explain vision- language encoders instead. Related to our contribution of weighted Banzhaf interaction explanations is work using weighted Banzhaf values estimate data valuation scores [37, 38], which are the importance that training data points or subsets have on the model's performance. Jin et al. [31] use Banzhaf interactions to enhance the training of video-language encoders. For comprehensive overview of work in this direction, see [50, 59] and references given there.
3 Game-theoretic Explanation of Similarity in Vision-Language Encoders
A rision-language encoder f consists of vision encoder fi Rd + Ed with nı E N image patches (tokens), and a language encoder fr : 1R" + gd with ny E N text tokens. For a given input pair (1z, the model computes cosine similarity of their d-dimensional embeddings
fI(ØT) (T(TT) [[5x(zz)] ([
[WI, = cos (oo(((/)))
In this context, we index the set of image tokens with N7 == {l,.... n and the set of text tokens with Ny {nz + L,..., T2 + n.). Related attribution methods construct explanations that assign importance values to all individual input tokens. To understand cross-modal and intra-modal relationships between tokens, we extend these explanations by adding all pairwise token interactions.
Definition 1 (Explanation). An explanation e RR assigns a constant en, individual attributions e, E R and pairwise interactions e(i,)) E IR, for an explanation basis
B
(0} Uti Nz UN u [[ i.j E Nr U i consiant
individual tokens
two-token interaction sets
3

---

An interaction explanation can be viewed as a complete graph with weighted nodes and edges, similar to the SI-Graph [51], or a symmetric quadratic matrix of dimension ny + ny, where the diagonal represents token attributions. To compute explanations, we mask tokens with pre-defined baseline, depending on the encoder's s architecture, e.g. attention masking for text tokens (we defer implementation details to Appendix B). The masking operator is used to define the FIXLIP game that captures all possible masks.
Definition 2 (Masking). We define the masking operator 4DM, RR" x R - R for n E (nz, nT) that compute for a subset of indices Mo C N with N E (NI, NT) and inputs I, b TR" as
MM b = Thi if E Mo. bis IfiEN
Definition 3 (Game). For an input image- text pair (II, xT) and a baseline (bz. by) indexed by NTU the FIXLIP explanation game v 2NTUNT - E. measures the similarity of the inputs given any mask M S NUUNT as v(M)
M MTNT) = f(xr @MNN2 FI @MON My bT).
The FIXLIP game measures the similarity of the masked image and text inputs for every possible mask. In the following, we will quantify attributions of individual tokens, as well as cross- and intra-modal interactions, that faithfully explain the FIXLIP game, and satisfy important axioms.
3.1 FIXLIP-p explanations via weighted faithful Banzhaf interactions
In this section, we formally define the FIXLIP explanation as an instance of Definition 1, which faithfully describes the similarity of all masked inputs v(M). We thus define as an additive explanation with interaction terms that recovers (MM) = (M) via a 2-additive game [24] as
ee = eg + Σ e; +
Σ (iJ)CMa
((, for all masks M S NNUNT.
iMM
A natural choice of faithfulness is measure the squared error (N(M) (M)) across all masks. To further distinguish in-distribution (few tokens masked) and out-of-distribution (many tokens masked) inputs, we associate mask weight according to its size |M in the following metric. Definition (p-faithfulness). For p E (0, and v, D: 2NNUNN - IR, we measure p-faithfulness 3p as 3p D) EMGNIUN,,'' (1- (o (
Remark Computing p-faithfulness requires evaluating 2 masks, which is infeasible in practice. Instead, we write By as an expectation over randomly generated masks 3p EMSP, [[((M - (. where Py(M) = M (1-)) -))" is a probability distribu- tion over NNUNN, This formulation allows us to estimate 8p using Monte Carlo integration.
The hyperparameter p determines the importance of each mask and can be viewed as the probability of a token being active, i.e. its index being included in M. The case p = 0.5 equally weights all masks, p > 0.5 prioritizes masks with many active tokens, i.c. preferring in-distribution inputs, whereas < 0.5 prioritizes sparse masks preferring more out-of-distribution inputs.
Definition 5 (FIxLIP-p). We define the FIXL.IP-P explanation as FIXLIP-p == argmin, Bp(M, ie)-
The FIXLIP-p explanation is the most faithful approximation of with respect .p. Its interactions correspond to the weighted Banzhaf interactions [47]. For = 0.5, FIXLIP-p is the faithful Banzhaf interaction index [64] with known analytical solution [26].
Remark 2. Weighted Banzhaf interactions define a cardinal pradinal-ppobabiissttcc interaction index that satisfies the linearity, symmetry, and dummy axiom [18]. Alternatively, faithful Shapley interactions optimize a variant of the faithfulness metric [64]. In our context, weighted Banzhaf interactions have two benefits: First and foremost, the hyperparameter p allows for prioritizing masks that are expected to better reflect in-distribution inputs, and provides a flexible explanation framework with intuitive weights. Second, it is unknown if the Shapley weights can be factored into independent distributions for each modality a crucial requirement for the cross cross-modal estimator introduced in Section 3.2.
FIXLIPP- explanations of VLES offer unique perspective interpreting image-text similarity predictions. Notably, the resulting second-order interaction explanation can always be simplified into first-order attribution explanation and visualized as traditional saliency map over input tokens.

---

Theorem 1 (First-order conversion), For E UT and e -I the first-order attribution values are given by e; + EEENNNNNTOHH (b.i), which are the weighted Banzhaf values of
We defer the proofs to Appendix A. While -Ix be computed analytically, it still requires 2 mask evaluations. We thus introduce a model-agnostic estimator for FIXLIP-p by estimating 8y via Monte Carlo integration with sampled masks and optimizing the least-squares objective.
Definition 6 (Model-agnostic estimator). Let M(), ....., ,...,,(m. ((m) fdd Pp. The model-agnostic estimator is then FIXLIP == argming ((m)) (v, ve), where {(m)) (v, De) = - 2 (V(M(0) )e(M())))
For p- 0.5, this estimator reduces Faith-Banzhaf [64] a variant of KernelSHAP [44].
3.2 Cross-modal sampling with p-weighted masking
We observe that the model-agnostic estimator uses naive sampling over masks M Pp, which be- comes prohibitive for models with a large number of input tokens. Since every token is independently active with probability p. we propose to separately sample my subsets for the image and my subsets for the text modality, i.e. to decompose PP, = P...I @ IP T. and sample independently. This allows for generating all possible combinations, obtaining a novel estimator of 3p.
Definition 7 (Cross-modal estimator). Let M
M(mx) iid
P.I and M4 ,.... MfMTT in PPPT. ((mr,,m)) (e). where
We define the cross-modal estimator as FIXLIP-P argming 8p
(m (v, W. UI Ex=16==1
1
(v(M (ez) UMff)) (o(M (ex) u U
Crucially, 87 ((M..MT) can be computed efficiently because f independently encodes images for mi samples with fz and texts for my samples with fr (ef. Equation 1). Then, similarity predictions computed in a vectorized manner between all the cross-modal combinations of masked inputs. In theory, given m = my + my samples, the model-agnostic estimator obtains m game values (similarity predictions), while our proposed cross-modal estimator obtains my my >>> m. The empirically observed computational speedup, which we analyze in Section 5.4, can depend on the model's batch size (closely related to the GPU VRAM hardware). Note that reusing the samples in text and image modalities inflicts dependence between them, and the following result establishes important theoretical properties.
Theorem 2. The cross-modal estimator 8p ((mz,T)) is unbiased, and its variance is bounded by
(m)) p (v, (v. S V (v. 0o)] S
mI + MT V My) De U Mr)) "..
mi my
In other words. Theorem 2 shows that the variance of the cross modal estimator is at most of the same order as the variance of the model-agnostic estimator with m z my Rs my samples. Moreover, the variance is at least the variance of the model-agnostic estimator with my my independent samples. In practice, we use the cross-modal estimator in higher budget scenarios to speed up the computation by decreasing the number of effective model inferences.
3.3 Large-scale adaptations for FIxLIP explanations
In contrast first-order explanations, the explanation basis of Definition grows quadratically as [ = 1 + dI + nIT ("It"T) R (nz + n²)²². For example, ViT-B/16 version of CLIP with 196 image and 30 text input tokens (players) results already in 25 425 interactions. From a practical perspective, we thus consider two heuristics as part of our complete methodology: (1) We employ a two-step filtering approach for prioritizing interactions to approximate when explaining games with a large number of players. A default is to pick clique subset with the highest (absolute) first-order attributions and compute interactions between them, e.g. top-72 with (?2) = 2 556 interactions. Another approach is to include only cross-modal interactions the approximation, explanation
e.g. 196 30 5 880. (2) We apply simple greedy subset selection algorithm e to find subgraphs M S Nz U Ny of the highest and lowest sums of De(M), which we use in evaluating (Section 5.1) and visualizing (Section 5.5) explanations. Further details are in Appendix B.
5

---

4 Evaluation Metrics for Interaction Explanations of VLE Predictions
In this section, derive three evaluation metrics for explanations that may include second-order interactions. First, we evaluate the faithfulness of the approximation Pe to v, since the basis of explaining VLEs is the masking operator that defines the FIXLIP game Note that conventionally this is done with an R coefficient [33, 50] or mean squared error [16, 19, 52], but we need to rely on rankings compare attribution methods like saliency maps. Second, generalize the area between insertion and deletion curves [AID, 25, 77], aka remove least/most important first [6]. a popular faithfulness metric for token attribution methods. Finally, extend the well-established pointing game evaluation of faithfulness from the image classification literature [3, 7, 8, 48]. In the latter two for the first-order attribution explanations without interactions.
metrics, we need to assume ((i))
We validate the faithfulness to the FIXLIP game V across sampled subsets. To this end, we compare the rankings of i/ and fú since related explanation methods are not normalized to estimate correctly, Definition 8 (p-faithfulness correlation). We define p-faithfulness correlation as the Spearman's 's rank correlation computed between v and t evaluated for m masks sampled from Pp (cf. Remark 1) as
Hoorr (e:p)= = correlation M1) M. (m) H.p
The rank correlation metrie Meuer yields insights into the general capability of recovering the ranking of similarity scores of uniformly masked inputs based on P,,
Beyond capturing the general p-faithfulness, we derive an insertion/deletion visualization with metric measuring the explanation's ability to identify token subsets (masks) of high and low similarity as evaluated by the game (model).
Definition 9 (Area between insertion/deletion curves (AID)). For each size k = 1.... ................................ nE nT, let Mo, == argmin N (M) and Me,max,k == arginax MCN : Po Insertion and deletion curves are computed Cinsert (e) (K (MM,,mu,,,)))) ) kmm.....z-nrr and Cáelete (e) (kk,
k= tor respectively. The metric is then defined as tr
(AID(e)= = f Cinsert (e) ff Cdeleto(e)
(v(Mo,max.x) - v ((Memiio.)))
MAID measures the difference between model predictions when including and excluding the most relevant tokens. For first-order methods, Me.min and Me are directly found by ranking e. Proposition 1. For token attribution values, i.e. when Vt,j e(ist = O, such as weighted Banzhaf values, Me,max,k and M Mo.min,k &. are given by the top-k and bottom-k coefficients of respectively. Consequently, Cdelete is found by deleting Me,max,k i.e. Me, Me. = (Nz UT) \ Me,max.t
Consequently, Cinsert and Cáelete generalize insertion and deletion curves to second-order interac- tion explanations. Note that for token attribution values, we have Me,max,k C Me,max, A++1 and Me,min,k C Me which does not hold when modeling interaction terms. We give a further description of the insertion/deletion process on single image- text input in Appendix B.2, Figure 9.
Lastly, we propose to evaluate an explanation e with designed pseudo ground-truth data. We measure the overlap between an explanation of inputs crafted as four combined images with a multi-object text prompt, and its "ground-truth" assumed prior. We show a visual example in Appendix B.3, Figure Definition 10 (Pointing game recognition (PGR)). For e, let C and Cout. denote the values of interactions belonging to, and not belonging to, object k text token with its corresponding image patches. We denote the positive and negative interactions by e>o, eco. The metric is then defined as
(e Ny) = ΣKENT (Ilein,k,20l1 + |leour,k,coll E [0, 1], where |ellı == Σ f
|e
EXENT (I[em, + |/oout,:l]1)
μPGR quantifies the ratio of absolute values of "correctly" identified cross-modal interaction terms to total interaction terms. Thereby, "correctly" identified cross-modal interactions refer to positive and negative interactions that are associated with and without object k in an image, respectively. We use μPPR. as a sanity check that interaction explanations are essential for explaining VLEs as compared to alternative attribution approaches. Scoring high PGR values across multiple objects present in an image-text pair denotes that explanation method can show the model distinguishes between them.

---

GAME
Grad-ECLIP
exCLIP
FIXLIP
ong yellow byde
B0g sellow Tyor 2000
bydr
BlACk non SIRlow SOAAE
Figure Visual comparison between FIXLIP and baselines. First-order attribution methods, e.g- GAME and Grad-ECLIP, lack the tools to faithfully explain complex similarity predictions of vision- language encoders like CLIP. Notably, in this example, the text token ant * is the most important for the similarity prediction. One of the differences from exCLIP that we include main effects in the approximation, which are crucial for obtaining faithful interaction explanations.
and
Experiments
Setup. In experiments, we empirically validate the performance of FIXLIP with three metrics defined in Section 4, measure its computational efficiency, and demonstrate utility visual explanation of VLEs. We mainly use the openly available pre-trained CLJP models [57] of two sizes: ViT-B/32 with 7 x 7 image patches and ViT-B/16 with 14 x 14. Moreover, we demonstrate the broader applicability of FIXLIP to explain Lig [75] and SigLIP-2 [66] up to the ViT-L/16 variant with 16 16 patches. We rely on two openly available datasets commonly used in explainability research: MS COCO [42] and ImageNet-1k, Ik [15]; the latter specifically to design the pointing game evaluation considering zero-shot classification.
In quantitative evaluation, we compare FIXLIP to few representative baselines: the first attribution method for CLIP abbreviated as GAME [12], a state-of-the-ar first-order attribution method Grad- ECLIP [77], and recently released second-order interaction method exCL.P [48]. For comparisons of Grad-ECIIIP and exCLIP with previous baselines, refer to the appropriate benchmarks [48, 77]. Figure 2 conceptually compares FIXLIP to baselines using the image-text input example from Figure Furthermore, we naturally demonstrate the improvement between FIXLIP and first- order Shapley/Banzhaf values. In that, we effectively scale FIXLIP to approximate second-order explanations with a budget of over 10 model inferences, which far exceeds related work. For FIXLIP-p, we use the cross-modal estimator with budget of 2 whereas FIXLIP with Shapley interactions uses the model-agnostic estimator with a budget of 2!7, yielding approximately similar runtime. We mainly experiment with p E {0.3, 0.5, 0.7}; note that there no computational overhead for different settings. Further details for the experimental setup are provided in Appendix C.
5.1 FIXLIP outperforms baselines as measured with insertion/deletion curves
Figure 3 shows insertion/deletion results for different explanation methods, where each line and metric value is average (+: sd.) over 1000 inputs. We observe that FIXLIP faithfully recovers the nonlinear importance rankings of token subsets (see Appendix Figure 9 for a visual example). It not only finds the most important tokens whose deletion results in significant drop in similarity (y-axis), but also finds the least important tokens whose deletion may even result in the prediction's increase, contrary to gradient-based methods. The general conclusion consistent for SigLIP-2 and a larger VIT-B/16 model (refer to Appendix D for additional results), albeit the method's ' faithfulness drops when increasing the size of input tokens, where further scaling to even higher image resolutions is a natural future work direction. As designed, increasing the masking weight to p 0.7 leads to an improved faithfulness the 0-50% range of input deleted (100-500 50% inserted), while decreasing to 0.3 improves faithfulness with 50-100% input deleted (50-0% inserted). Note that, in theory, all of the methods can obtain negative normalized values on the y-axis. We think that exCLIP fails to recover the appropriate ranking because it only approximates cross-modal interactions between tokens from the two modalities, omitting first-order effects and intra-modal interactions in the process (effectively constructing bipartite weighted graph without weights in nodes).
7

---

Prediction change (normalizedl
1.00
Method (Area between curves)
FixLIP p 0.7 (0.94 0.18)
0.75
FIXLIP p 0.5 (1.02 0.18)
0.50
FIXLIP 0.3 (0.95 0.18)
Shapley values (0.77 0.15)
0.25
Grad-ECLIP (0.37 i 0.16)
0.00
GAME (0.33 0.16)
exCLIP (0.25 0.18)
-0.25
delete
Curve
insert important (higher better)
-0.50
delete important (lower better)
100%
80%
60%
40%
20%
0%
Percentage input k (Ntt + mimg)
Figure 3: Insertion/deletion curves for CLIP (VIT-B/32) on MS COCO. AID score (higher is better) for FIXLIP against alternative explanation methods, where random baseline scores 0. The y-axis is normalized between the model's prediction on the original input (100%) and the fully removed one (0%), where negative values denote that the model is predicting the image- text inputs unsimilar. It means the similarity prediction prediction on the fully masked input. Methods such as Grad-ECLIP and exCLIP fail to recover nonlinear rankings of important tokens, while our method faithfully recovers the optimal subset explanation. Extended results for CLIP (VIT-B/16) and SigLIP-2 (ViT-B/32) are in Figures 10 & 11.
partially masked input is smaller than the
Table 1: Pointing game recognition for CLIP (ViT-B/32) ImageNet-Ik, PGR score (higher is
better) for FIXLII against alternative explanation methods, where a random baseline scores 0.25. First-order methods, such as Grad-ECLIP and Shapley values, fail distinguish between multiple objects at once, while second-order methods faithfully recover the appropriate explanation (up to the pointing game's irreducible non-optimality). Extended results for CLIP (VIT-B/16) are in Table 4.
Explanation Method
Recognition (t)
object
2 objects
objects
objects
GAME [12]
.61
.3
.33
.28..u1
Grad-ECLIP [77]
.68
.54.04
.33..0 .2
.28
Shapley values
.70
.66..66
.46..05
.371.04
Banzhaf values
.642.12
.52..08
.31.05
.35..04
exCLIP [48]
.73
.88..8
.891.08
.92
FIXLIP (Shapley interactions)
.82..88
.84..0
.86..00
FIXLIP (w. Banzhaf interactions p 0.3)
.78
.78
.80
.81
FIXLIP (w. Banzhaf interactions p = 0.5)
81
.802.08
2.1
.834.08
FIXLIP (w. Banzhaf interactions
0.7)
.83..12
.812.08
.83
.85
5.2 First-order attribution methods fail to pass sanity check with pointing game
Table I shows PGR results for different explanation methods, where each metric value is an average (#: sd.) over 500 inputs. We observe that first-order methods, e.g. Banzhaf values, fail to discriminate between multiple objects an image. Saliency maps can only highlight the important part of image-text inputs without disentangling the complex relationship between each separate word in caption and the corresponding image regions (Appendix Figure 8). Second-order methods such as FIXLIP and exCLIP pass our proposed sanity check; the latter scores slightly higher, as it specializes in approximating cross-mode interactions. The conclusion consistent for CLIP (ViT-B/16). We further analyze the potential application of pointing game evaluate different models in Section 5.6.
5.3 FIXLIP recovers a faithful decomposition of the similarity function
Figure demonstrates p-fattffulness results for different explanation methods, where each boxplot represents statistics aggregated from 1000 inputs. We confirm that FIXLIP optimizes faithfulness, which depends on the parameter p. Weighted Banzhaf interactions allow for precise controllability of the faithfulness optimization, whereas Shapley interactions perform well on average. Alternative

---

GAME
exCLPP
Shapley
Banchaf ues
Benzhel vause
RuLIP (WB1 p 0.3)
FIXLIP (MB) p 0.3)
FILI
(WBI 0.5)
FMLP (WBI p 0.7)
FkUP (WB) p
0.2
0.4
0.6
1.0
0.2
p-fathfulness correlation (p 0.7)
R² =p550
Figure 4: p-faithfulness correlation for CLIP (VIT-B/32) MS COCO. Correlation for different variants of FIXLIP against other explanation methods (left). Game-theoretice approaches can also be evaluated with the R coefficient (right). Extended results for CLIP (ViT-B/16) are in Figure 12.
attribution methods compute explanations unfaithful to the explained similarity function (Noorr R 0.5). Extended results for different models and values are given in Appendix D.
5.4 On the computational efficiency of the FIXLIP cross-modal estimator
Figure 5 demonstrates that the FIXLIP cross- modal estimator achieves over 20x speedup
Time in seconds (A100 GPU)
10a
Mode-agnostic
Explanation (total)
Crass moda
Game (inference)
when considering model inference time, i.e.
10a
game evaluations, and about 5x speedup when
accounting for the entire explanation pipeline im- plemented in Python. We did not necessarily op- timize latter, i.e. subset sampling, weighted least squares optimization, and other processing steps, expecting that the visible gap between the explanation and game time could be further de- creased. For context, related first-order attribu- tion methods take about 1 second to compute; Table 5 in [77], but note that it could be only for an image explanation without text attribution.
10c
45x
10a
20x
10
2
2
2
2020
2
Number of sampled masks (budget)
Figure 5: Computation time vs. budget for the FIXLIP explanation of SigLIP-2 (ViT-B/32), in- cluding game evaluations (model inference).
5.5 Visual explanation of vision-language encoders
We have already established that FIXLIP delivers state-of-the-art faithfulness performance across three diverse metrics. One of the explanations' applications function. Figure 6 demonstrates three types of visualizations that we envision can be used to broaden our understanding of VLEs. the interest of space, we provide further interesting examples in Appendix E. Specifically, provide comprehensive guide on interpreting interaction explanations Figures
interpret the model's output
like FIXLIP Appendix evaluation metries. Figure 16 shows a comparison between IIX of CLIP and SigL.P-22 (ViT-B/32), while Figure 17 shows a comparison between the ViT-B/32 and ViT-B/16 versions CLIP.
further compare FIXLIP to baselines the context of
5.6 Model-agnostic comparison of different vision-language encoder architectures
We use FIXLIP compare differ- ent VLE architectures with PGR
Table 2: Pointing game results as measured with FIXLIP.
Table 2. Surprisingly, SigLIP-2 can be more l'aitffully explained with cross-modal interactions than CLIP, for both model sizes. We observe that, general, smaller models (ViT-B/32) can be more faithfully explained than larger ones (ViT-L/16), which is consistent with prior work [9, 16). For more visual comparisons, Appendix E.3.
Size
Model
Recognition (f)
object
objects
o
4 obj.
CLIP
.83
.81
.824.77
.85..5
VIT-B/32
SigLIP-2
.90
.904.05
.894.05
.894.04
CLIP
81 $19.09
.81
.8
.824.04
VIT-B/16
SigLIP
.80
.82+.06
.844.00
.844.05
SigLIP-2
861.07
.88..04
.874.04
.871.03
SigLIP
81
.844.0
.84..4
.844.04
VIT-L/16
SigLIP-2
10
.854.06
.841.05
.851.04

---

(A) Faithful interaction explanations
(B) Conditioning tokens
poskthe
negaetve
QUARTER AND
condition
225 insaing not Chouter the chone
Seoning W Saassee BA0S ShAA
Bog
Sheaktee BULE in the E.S.
black
Call
pen
phonn
brown haired female doll suspended between two parking meters
and
(C) Visualizing subsets
Figure 6: FIXLIP facilitates various approaches to model understanding. (A) Interaction explanations allow answering the pivotal question: Why is it similar for the model? Here, the strongest interaction is between text token doll and image patch saying "dollar". One could say the model is right for the wrong reasons. (B) Each token can be selected for conditioning to visualize as a heatmap only the interactions (edges) outgoing from it. (C) complete graph can be traversed to find subsets of high positive and low negative similarity as approximated by the explanation.
Discussion
As vision laiguaee encoders are increasingly deployed in real-world applications, it becomes pivotal ensure that their predictions explainable. To this end, we introduced faithful interaction explana- tions of CLIP and SigLIP models, offering a unique perspective on interpreting image-text similarity predictions. Moreover, derived three evaluation criteria facilitating future work in this direction.
Limitations and future work. Our work faces three limitations, each with a clear path for future development. Although FIXLIP allows scaling to larger models with hundreds of players and efficiently computing a million model inferences, improvements could be made to its practical implementation. Specifically, emvision exploring the use of sparse linear regression, applying the Archipelago framework [65] to filter interactions for the approximation, and a non-greedy algorithm for closcr-to-optimal subset selection. Second, the visual properties and usability of FIxLIP should be further studied as a potential direction in human-computer interaction research [58], see e.g. a user study of cross- modaal interaction explanations by Liang et al. [40]. Finally, work is restricted to second-order interaction explanations, while efficiently approximating higher-order interactions (also in tri-modal settings beyond LIP models) becomes interesting challenge to overcome.
Broader impact. We believe FIXLIP can empower model developers debugging VLEs, under- standing their similarity predictions, and finding unwanted biases in image-text data. Especially when these models are used in high-stakes decision making, like in the case medical applications.
Code. We provide additional details on reproducibility in the Appendix, as well as the code to reproduce all experiments this paper is available at setps://gitaae com www.ccoomo/aeeeeeeeee htthh/aeiiniiiiii....
Acknowledgments and Disclosure of Funding
We gratefully acknowledge the Polish seighpeeffrmance computing infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH) for providing computer facilities and support within the computational grant no. PLG/2025/018333 Ministry of Education and Science program "Pearls of Science" project number PN/01/0087/0222. Barbara Hammer, Eyke Hüllermeier, Fabian Fumagalli, and Maximilian Muschalik gratefully ac- knowledge funding by the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation): TRR 3 318/1 2021 438445824.
Hubert Baniecki was supported from the state budget within the Polish
10

---

References
[1] Chirag Agarwal. Rethinking explainability in the era of multimodal Al. preprins,
130-0 2025. 1
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, et al. Flamingo: a visual language model for few-shot learning. In NewrIPS, 2022.
[3] Shreyash Arya, Sukrut Rao, Moritz Böhle, and Bernt Schiele. B-cosification: Transforming deep neural networks to be inherently interpretable. In NeurIPS, 2024. 4
[4] Sriram Balasubramanian, Samyadeep Basu, and Soheil Feizi. Decomposing and interpreting image representations via text in ViTs beyond CLIP. In NeurIPS, 2024. 2
[5] Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P. Calmon, and Himabindu Lakkaraju. Interpreting CLIP with sparse linear concept embeddings (SpLiCE). In NeurIPS, 2024. 2
[6] Stefan Bluecher, Johanna Vielhaben, and Nils Strodthoff. Decoupling pixel flipping and occlusion strategy for consistent XAI benchmarks. Transactions on Machine Learning Research, 2024. 1,
[7] Moritz Bohle, Mario Fritz, and Bernt Schiele. Convolutional dynamic alignment networks for interpretable classifications. In CVPR, 2021. 4
[8] Moritz Bohle, Navdeeppal Singh, Mario Fritz, and Bernt Schiele. B-cos alignment for inherently interpretable CNNs and vision transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(06):4004 4518, 2024. 4
[9] Sebastian Bordt and Ulrike Luxburg. From shapley values generalized additive models and back. In AISTATS, 2023. 2, 5.6
[10] Yue Cao, Yun Xing, Jie Zhang, Di Lin, Tianwei Zhang, Ivor Tsang, Yang Liu, and Qing Guo. SceneTAP: Scene-coherent typographic adversarial planner against vision-language models in real-world environments. In CVPR, 2025. E.2
[11] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supervupe vision transformers. In ICCV, 2021.
2
[12] Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In ICCV, 2021. 1, 2, 5, C.3,
[13] Sanghyuk Chun, Wonjae Kim, Song Park, and Sangdoo Yun. Probabilistic language-image pre-training. In ICLR, 2025. 1
[14] Ian Covert, Tony Sun, James Zou, and Tatsunori Hashimoto. Locality alignment improves vision-language models. In ICLR, 2025.
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 5, C.2
[16] James Enouen and Yan Liu. InstaSHAP: Interpretable additive models explain Shapley values instantly. In ICLR, 2025. 2, 4, 5.6
[17] Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott M Lundberg, and Su-In Lee. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature Machine Intelligence, 3(7):620-631, 2021. B.1
[18] Katsushige Fujimoto, Ivan Kojadinovic, and Jean-Luc Marichal. Axiomatic characterizations of probabilistic and probaiil- interaction indices. Games and Economic Behavior, 55(1):72-99 99, 2006. 2
[19] Fabian Fumagalli, Maximilian Muschalik, Patrick Kolpaczki, Eyke Hüllermeier, and Bar- bara Eva Hammer. SHAP-IQ: Unified approximation of any-order Shapley interactions. In NewrIPS, 2023. 1, 4, B.1

---

[20] Fabian Fumagalli, Maximilian Muschalik, Patrick Kolpaczki, Eyke Hüllermeier, and Barbara Hammer. KemelSHAP-IQ: Weighted least square optimization for Shapley interactions. In ICMI., 2024. 2
[21] Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting CLIP's image represen- tation via text-based decomposition. In ICLR, 2024.
[22] Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3):
e30, 2021. 2
[23] Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. FigStep: Jailbreaking large vision-language models via typographic visual prompts. In AAAI, 2025. E.2
[24] Michel Grabisch. Set Functions, Games and Capacities in Decision Making. Springer, 2016.
3.1 A.1
[25] Naofumi Hama, Masayoshi Mase, and Art B. Owen. Deletion and insertion tests in regression
models. Journal of Machine Learning Research, 24((90): 2023. 1, 4
[26] Peter L. Hammer and Ron Holzman. Approximations of pscudo-boolenn functions; applications to game theory. ZOR Mathematical Methods of Operations Research, 36(1):3-21, 1992. 3.1
[27] Peter Hase, Harry Xie, and Mohit Bansal. The out-of-distribution problem in explainability and search methods for feature importance explanations. In NeurlPS, 2021. 1, B.1
[28] Jack Hessel and Lillian Lee. Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think! In EMNLP, 2020. 2
[29] Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Stefan Chan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-IM: One million image-text pairs for histopathology. In NeurlPS, 2023. 1
[30] Joseph D. Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Machine Learning Research, 22(104): 2021.
[31] Peng Jin, Hao Li, Li Yuan, Shuicheng Yan, and Jie Chen. Hierarchical Banzhaf interaction for general video-language representation learning. IEEE Transactions Pattern Analysis and Machine Intelligence 2024. 2
[32] Boris Joukovsky, Fawaz Sammani, and Nikos Deligiannis. Model-agnostic visual explanations via approximate bilinear models. In ICIP, 2023. 2, C.3
[33] Justin Singh Kang, Yigit Efe Erginbas, Landon Butler, Ramtin Pedarsani, and Kannan Ramchan- dran. Learning to understand: Identifying interactions via the Möbius transform. In NeurlPS, 2024. 2,
[34] Justin Singh Kang, Landon Butler, Abhineet Agarwal, Yigit Efe Erginbas, Ramtin Pedarsani, Bin Yu, and Kannan Ramchandran. SPEX: Scaling feature interaction explanations for LLMs. In ICML, 2025.
[35] Martha Lewis, Nihal Nayak, Peilin Yu, Jack Merullo, Qinan Yu, Stephen Bach, and Ellie Pavlick. Does CLIP bind concepts? Probing compositionality large image models. In Findings of EACL, 2024.
[36] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In /CML, 2023.
[37] Weida Li and Yaoliang Yu. Robust data valuation with weighted Banzhaf values. In NeurIPS, 2023.7
[38] Weida Li and Yaoliang Yu. One sample fits all: Approximating probabilistic values simulta- neously and efficiently. In NewrIPS, 2024. 2
12

---

[39] Yi Li, Hualiang Wang, Yiqun Duan, Jiheng Zhang, and Xiaomeng Li. A closer look at the explainability of contrastive language-image pre-training, Pattern Recognition, 162: 162:111400 11409, 2025. 1,2
[40] Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis- Philippe Morency, and Ruslan Salakhutdinov. MultiViz: Towards visualizing and understanding multimodal models. In ICLR, 2023. 2,
[41] Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider. Sparse autoencoders reveal selective remapping of visual concepts during adaptation. In ICLR, 2025. 2
[42] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 5, C.2
[43] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023,
[44] Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In NeurlPS, 2017. 3.1.
[45] Daniel D Lundstrom, Tianjian Huang, and Meisam Razaviyayn. A rigorous stuy of integrated gradients method and extensions to internal neuron attributions. In ICML, 2022. B.1
[46] Yiwei Lyu, Paul Pu Liang, Zihao Deng, Ruslan Salakhutdinov, and Louis-Phillipe Morency. DIME: Fine-gres Fine-grained interpretations of multimodal models via disentangled local explanations. In AIES, 2022. 2
[47] Jean-Lue Marichal and Pierre Mathonet. Weighted Banzhaf power and interaction indexes through weighted approximations of games. European Journal of Operations Research, 21 12): 352-358, 2011. 1, 3.1, A.1
[48] Lucas Moeller, Pascal Tilli, Thang Vu, and Sebastian Pad6. Explaining caption-image inter- actions in CLIP models with second-order attributions. Transactions on Machine Learning Research, 2025. 2, 5, 1, C.3, 4
[49] Niels Morch, Ulrik Kjems, Lars Kai Hansen, Claus Svarer, Ian Law, Benny Lautrup, Stephen Strother, and Kelly Rehm. Visualization of neural networks using saliency maps. In ICNN, 1995.
[50] Maximilian Muschalik, Hubert Baniecki, Fabian Fumagalli, Patrick Kolpaczki, Barbara Ham- mer, and Eyke Hüllermeier. shapiq: Shapley interactions for machine learning. In NeurIPS. 2024. 2.4
[51] Maximilian Muschalik, Fabian Fumagalli, Paolo Frazzetto, Janine Strotherm, Luca Hermes, Alessandro Sperduti, Eyke Hüllermeier, and Barbara Hammer. Exact computation of any-order Shapley interactions for graph neural networks. In ICLR, 2025. 3
[52] Christopher Musco and R. Teal Witter. Provably accurate shapley value estimation via leverage score sampling. In ICLR, 2025. 2,
[53] Letitia Parcalabescu and Anette Frank. MM-SHAP: A performance-agnostic metric for measur- ing multimodal contributions in vision and language models & tasks. In ACL, 2023. 2
[54] Letitia Parcalabescu and Anette Frank. Do vision & language decoders use images and text equally? How self-consistent are their explanations? ICLR, 2025.
[55] Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. In CVPR, 2018. 2
[56] Guilherme Dean Pelegrina, Leonardo Tomazeli Duarte, and Michel Grabisch. A k-additive Choquet integral-based approach to approximate the SHAP values for local interpretability in machine learning. Artificial Intelligence, 104014, 2023. 2
13

---

[57] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI., 2021. 1,5, C.I
[58] Yao Rong, Tobias Leemann, Thai-Trang Trang Nguyen, Lisa Fiedler, Peizhu Qian, Vaibhav Unhelkar, Tina Seidel, Gjergji Kasneci, and Enkelejda Kasneci. Towards human- centered explainable Al: survey of user studies for model explanations. IEEE Transactions on Pattern Analysis and Machine Intelligence, (6((4):21042112) 2024. 6
[59] Benedek Rozemberezki, Lauren Watson, Péter Bayer, Hao-Tsung Yang, Olivér Kiss, Sebastian Nilsson, and Rik Sarkar. The Shapley value in machine learning. In IJCAI, 2022. 2
[60] Fawaz Sammani and Nikos Deligiannis. Interpreting and analysing CLIP's zero-shot image classification via mutual knowledge. In NeurlPS, 2024. 2
[61] Fawaz Sammani, Boris Joukovsky, and Nikos Deligiannis. Visualizing and understanding contrastive learning. IEEE Transactions on Image Processing, 33:541-555, 2024. 2, C.3
[62] Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The Shapley Taylor interaction
index. In ICML, 2020. 2
[63] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal LLMs. In CVPR, 2024. 1, E.2
[64] Che-Ping Tsai, Chih- Kuan Yeh, and Pradeep Ravikumar. Faith-Shap: The faithful Shapley interaction index. Journal of Machine Learning Research, 24(94):1-42, 2023. 2, 3.1, 2, 3.1
[65] Michael Tsang, Sirisha Rambhatla, and Yan Liu. How does this interaction affect me? Inter- pretable attribution for feature interactions. In NeurIPS, 2020.
[66] Michael Tschannen al. SigLIP 2:
Multilingual vision- vision-language encoders with improved semantic understanding, localization, and dense features. preprint 1422:22502.14786, 2025. 1,
C.I
[67] Vishaal Udandarao, Max F Burg, Samuel Albanie, and Matthias Bethge. Visual data-type understanding does not emerge from scaling vision-language models. In ICLR, 2024.
[68] Bo Wan, Michael Tschannen, Yongqin Xian, Filip Pavetic. Ibrahim Alabdulmohsin, Xiao Wang, André Susano Pinto, Andreas Peter Steiner, Lucas Beyer, and Xiaohua Zhai. LocCa: Visual pretraining with location-aware captioners. In NeurIPS, 2024.
[69] Xingbo Wang. Jianben He, Zhihua Jin, Mugiao Yang. Yong Wang, and Huamin Qu. M2Lens: Visualizing and explaining multimodal models for sentiment analysis. IEEE Transactions on Visualization and Computer Graphics, 28(1):802-812, 2022.
[70] Ying Wang, Tim Rudner, and Andrew G Wilson. Visual explanations of image-text representations via
information bottleneck attribution. In NeurIPS, 2023. 1,
[71] Jialin Wu and Raymond Mooney. Faithful multimodal explanation for visual question answering. In ACL Workshop BlackboxNLP.. 2019. 2
[72] Rui Xiao, Sanghwan Kim, Mariana-Iuliana Georgescu, Zeynep Akata, and Stephan Alaniz. FLAIR: VLM with fine-grained language-informed image representations, In CVPR, 2025. 1
[73] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like hags-of-words, and what to do about it? In ICLR, 2023.
[74] Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek. Interpreting CLIP with hierar- chical sparse autoencoders. In ICML, 2025.
[75] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lueas Beyer. Sigmoid loss for language image pre-training, In ICCV, 2023. 1, 5, C.I.
14

---

[76] Sheng Zhang et al. A multimodal biomedical foundation model trained from fifteen million image-text pairs. NEJM AI, 2(1), 2024. 1, B.1
[77] Chenyang Zhao, Kun Wang, Janet H. Hsiao, and Antoni B. Chan. Grad-ECLIP: Gradient-based visual and textual explanations for CLIP. In ICML, 2024. 1, 4, 1, 5.4, C.3, 4
[78] Zihao Zhao, Yuxiao Liu, Han Wu, Mei Wang. Yonghao Li, Sheng Wang. Lin Teng, Disheng Liu, Zhiming Cui, Qian Wang. and Dinggang Shen. CLIP medical imaging: A survey. Medical Image Analysis, 00210
[79] Xu Zheng, Farhad Shirani, Zhuomin Chen, Chaohao Lin, Wei Cheng. Wenbo Guo, and Dong- sheng Luo. F-Fidelity; A robust framework for faithfulness evaluation of explainable Al. In ICLR, 2025.
[80] Zhiyu Zhu, Zhibo Jin, Jiayu Zhang, Nan Yang, Jiahao Huang, Jianlong Zhou, and Fang Chen. Narrowing information bottleneck theory for multimodal image-text representations interpretability. ICLR, 2025.
15

---

Appendix
Table 3 summarises the mathematical notation used in the paper. In Appendix we derive proofs for Theorems & 2, and Proposition 1. Appendix B describes the implementation details of our methods introduced in Sections & 4. Appendix C provides further details on the setup of experiments conducted in Section 5. Finally, we discuss ablation results in Appendix D and visualize additional explanations in Appendix
A Proofs
18
A.1 Proof of Theorem
18
A.2 Proof of Theorem
19
A.3 Proof of Proposition
22
Implementation Details
23
B.1 Faithful interaction explanations of LIP models (FIXLIP)
................................
23
B.2 Area between the insertion/deletion curves (AID)
................................
24
B.3 Pointing game recognition (PGR)
24
C Experimental Setup
26
C.1 Models
26
C.2 Datasets
26
C.3 Baselines
26
C.4 Metrics
27
C.5 Compute resources
27
D Additional Experimental Results
28
E Visual Explanatons
31
E.1 Guidance on interpreting visualizations interaction explanations
E,2 Additional examples
33
E.3 Comparison between CLIP (ViT-B/32) and LigLIP-2 (ViT-B/32)
35
E,4 Comparison between CLIP (ViT-B/32) and CLIP (VIT-B/16)
36
16

---

Table 3: Summary of notation.
Notation
Description
N, EN
Image and text input dimensions (number of tokens) Image and text input
E R JT RMT
((I,,TT)
Vision-language encoder prediction for an image- text imput pair Encoder embedding dimension
dEN
(x(TT) ( R"
Vision and language encoder embedding vectors
Nr, NT
Sets image and text token indices
i.j
Token indices
B
Explanation basis set containing indices of tokens and token pairs Explanation object
e R18)
e, enr)) E IP
Token attribution and interaction values Subset of image text token indices Subset of image and text token indices Baseline image and vectors Input æ masked using the operator for subset M. and baseline b Game (value) function
M. C N with N E (Nz,N7)
M C UT
bz, bt
x BM, M(M)
D, De
Estimators of the game (value) function Probability parameter p-faithfulness error metric
E (0.
(,(
e PINLiP-P == argmin, Brlv, Da)
Explanation approximating the game function w.r., p-faithfulness Probability distribution of sampled masks over N Number of sampled mask subsets
(M(M) p (1 p) n uxy -MM
m
(1)) ...., ((())) (m))
Sampled mask subsets of image and text token indices Model-agnostic estimator of p-fatthfulness
36 (u. Vo)
Pp.I P..
Probability distributions of sampled masks 2xx and Nr Numbers sampled image and text mask subsets Sampled mask subsets of image token indices
mI, mt
(1
M (m²)
M@
M (m))
Sampled mask subsets of text token indices
( (v. vo)
Cross-modal estimator of p-faithfulness
Peore (e:p)
p- p-faithfulness Spearman's 's rank correlation evaluation metric Area between insertion/deletion curves evaluation metric Mask subset of size & for explanation minimizing ée PeMMe.iin
HAID (e)
M m
Me. Memmax. k
Mask subset of size le for explanation maximizing ée (Meeminn, Insertion curve; set of paired (k, ((Ma. values Deletion curve; set of paired (ks (M. Pointing game recognition evaluation metric Interaction values belonging to text token k Interaction values not belonging to text token &
Cinsert
Caelete (e)
values
(P
Cin,k
Cout.k
e>0, eco
Negative and positive interaction values in an explanation Absolute of explanation
||elli == Ee, (e(r))
17

---

A Proofs
A.1 Proof of Theorem
Proof. To prove this result, we combine alternative representations of the cooperative games using the Möbius transform a 2NNuNN + R, which is defined by
(MM) = (-1) for every M S NT
LCM
The approximated game De a 2-additive game, and hence its Möbius transform is restricted up to second-orde order interactions [24]. Moreover, the Möbius transform of the approximated game â FIXLIP satisfies
FINLIP P(0) (0) Co:
FIXLIP ((d)) (e(()) (0((0) C;
FIXLIP ((i,)) = (e(((,])) - -e((()))
where i, Nz N and i j. The approximated game by construction 2-additive game. and thus all higher-order Möbius coefficients are zero [24]. It was further shown by Marichal and Mathonet [47, Proposition 3] that the weighted Banzhaf values of ú can be computed as the optimal first-order approximation of D. The representation of the best k-th order approximation in the Möbius transform was given by Marichal and Mathonet [47, Proposition 6], and reads for i E Nr U Nr.
k 1, and equal probabilities
(I 11 - 1-1 ( 1-1D)MM
a((i)) (-1)2-1
E
MENTUNT: I.EM
= ={((i))+ +p
Σ
a((i.j))
JENrUNTOJH
Σ
e; +D
e(if)
JEN
which concludes the proof.
18

---

A.2
Proof of Theorem 2
Proof. We first compute the expectations
mi
E fl fmmmmmm mm) (1, Po)] =
Σ (w(Me UM(T) - ((M - u M4
MI
Ex Ér=1
mimor Σ
E [u(Me UM - D(M u M
((M,, [MMM U (T)))
=RMae (u(M)
=3p(1, De):
since Mz 1 My are independent.
We now proceed by computing the variance. We first introduce Ge 2Nx x 2NT - R for Mz S Nz. and Mf Ny
Me.Mr, My) = (M(Mz U My) Dc(MM
For the standard Monte Carlo estimator, we then obtain
V(š(m)) (v, (v) - (M),())
m
due m iid samples. For the cross-modal estimator, we first prove a result for the explicit form of the variance in Proposition 2.
Proposition We denote the expectation over the conditional variances as
TI = PM
[ ((v(M My) - (e M)] [[v U My) - (M U MT])
TTI = EM p.7
The variance of the estimator Sp (m) (v, Pe) is then given by
() mI + mT V (M MT)]
mI 1
MT TTI
T2/T
mImn
Proof. The variance is computed as
V m mT) (v, ve))
V
1
Σ
ma myT
Σ Ie (ML(t), -affrr))
MIMT
Fr=1
mg
mT
my
Σ
Σ
Σ
cov
(sa(M
2o (M1 MW(TT)
(3)
m
fn=1dy=lkre ky=1
To compute the covariance, distinguish four cases, namely
cov(go(M1 ( .M (M M
Vigo (M MT),,
if l kI, bT KT, if E = KI, lr $ KT, if l + KI, bT = KT,
cov(ge Cov(ge(MM, (Mr, go Me Mf)). cov(go (Mz, MT), 9o (M4, MT)). 0.
if lz + kz, br # kT,
where Mr, MG~P,, P. and MT, MT~PT w. The covariances can be further computed as (Nv( My), 9e (Mz, M)) = (Mg Mg. (Mz, My) - Elge Mr Mf) -EMe~Pa.z PMMT~P,.77 [ge [M,, .M [E pF Ig (M, My - E[ga(Mr, M7))
= EMarR EMy~P 9e My . EMerR [EMroR (g MT),
W [M (g MM, Mr)]]
19

---

where we have used Mr 1 Mr, My and My 1 My. Similarly, we obtain
cov (g MT), 9n (M4, My)) Combining these results into Eq. (3), we obtain
My [M ( My)]]
T
Σ
Σ
cov (Be(M ,M go (M1 .21
mmm
mz
mer
MieeMMI MT]
m
2
casc Ez=kn,
Mr~Ppit [EMT~Pp,7 (M My]]]
ET ky=1 lTTkT
case tz=
m
WT
My~Ba, P.T [EM 19 (M MT
ky.k==1lys 1y= 1z7
case kz
MInTT V/ (0a(MM (Mz, My) m2
mimr((m
V pixx 7B M7]]]
m
myMI - 1)
Mr~Pa.T [E (e
m
We now use the law of total variance and Mr 1 My to rewrite
V Mz T (M7, MT)1] V[a, MT]] EMZeP (V (90 MT)]],
and use a similar result for My to obtain
(v, mgmn + mIm7(mT 1) + mT 1) v Vigo (o (M7, MT)]
m²m²
MT - EMenPao V Mr~P, (M7, MT)11
111 EMy VV MM (M My)]]
MIMT
MT
mI + +MT 1 MM)
mIMT (mT - 11)EME= MymPa. ( MT]] + (mz 1)E (V WM (Mz, MT)]] (MT
MIMT
!nz + MT + Vlg MM)
m. 1
MT
MgmT TITT
TTIZ,
mgmT
miml
where we have used the definitions of TI/T = EM r [V (Mz, and TTiZ
EM (Vres>/Vaaaaa (VMy~E, (oz(M. My)]]. This concludes the proof.
By using Proposition we obtain
V
( mz .m (o) S
My + mT 1
V I9 MT)]
mi + mT
(Mg, MT),,
mIMnT
MIMT
since TIT.,TTI& 0.
20

---

Moreover, by the law of total variance we have TI/T. TTIT S Vige (10,(MM,, My), which implies
2m 1 Visfmmm) (v)
2(m 1)
2 m²
(Ma MT)]
Vl (MI, Mr)l
m²
1 Vloe(M,, Mr)
m²
-1 ( (e,De),,
which finishes the proof.
21

---

A.3
Proof of Proposition 1
Proof. For a token attribution explanation e, the additive approximation reads as io (M) +
Σ
iEM
Denote M the top-k coefficients of c, then for M S Nr U Ny with M = k, we have
E - Σ Ci:
iEM
and thus
Me,max,k
argmax M
(M
argmax Σ
e M
| REM
Conversely, for the bottom-k coefficients Mg. we have
E
Σ ei
REM
X M
for every M N NT with [M] = k. and thus
Me.min.k
argmin M
Po(M) =
argmin Σ
e M
M REM
Lastly, the top-k coefficients correspond to the complement of bottom-(nz + ny k)) coefficients, which yields
M tor M - = (Nr ME.
which finishes the proof of the deletion curve Cdelete
(Mo. -x)) k=
22

---

B Implementation Details
B.1 Faithful interaction explanations of LIP models (FIXLIP)
Because of how the model's forward function is implemented, without loss of generality, explain logit-scaled similarity output from Equation 1, i.c. C' f IT), which depends on the particular model's learnt constant, c.g. C' = 100 for CLIP, but = 117 for SigLIP and C' s 112 for SigLIP-2.
Masking. Following related work using input removal in attribution methods [17, 19, 27, 45], we apply simple baseline masking strategies, which were also proven successful based results. For vision encoders, we use a baseline after image normalization to propagate no signal forward. In the case of language encoders, encode active tokens with 1 and the deleted ones with O. It is important to leave the beginning/end sequence and padding tokens is. Note that in theory, the maximum possible context length "r is finite (e.g. 64, 77) and could be treated as constant across different inputs. However, we never explain these special tokens, treating parts of text inputs as the only features of interest. Alternativ implementations of masking worth considering are: removing words from the text inputs, or tokens from tokenized inputs, imputing with the [UNK] token, or [MASK], e.g. in the case of the BiomedCLIP architecture that is based on BERT [76]. Acknowledging the potential influence of the masking approach, as well as position encodings, on the value of an empty input, we check that remains about constant across different text input lengths in Figure 7.
empirical
native attention masking mechanism, where
40
Predicted logit
30
20
10
0
10
20
30
40
50
60
70
Abx
Figure Value of an empty input is about constant across different text lengths.
Budget split in the cross-modal estimator. The model-agnostic estimator (Definition 6) relies on m sampled masks for approximation; the number often called the approximation budget. The cross- modal estimator (Definition 7) introduces budget split, where we set my = min (2 max(44 [/m- =T/nnz)) }) and my == min m2 max(4, Lym .tt/nn///)) reasonable default, allowing for balanced exploration of both input spaces.
mm myz For all our experiments,
Two-step filtering approach. For the ViT-B/16 and ViT-L/16 model versions with a larger number of input image tokens (patches), we perform a cheap computation of the first-order attribution to prioritize interactions for approximation. We consider two strategies: picking a clique subset of the most interesting tokens, or focusing only on cross modsl interaction for explanation, which we specifically apply in the pointing game recognition (Definition 10, Appendix B.3). In the first strategy, given the desirable clique size k = 72, we take ky == max (5 [k n (nT + nz)] text tokens with the highest absolute feature attribution scores, and ky : lo image tokens accordingly. Similarly to the above considerations regarding the budget split between the two modalities, we found this to be empirically adequate. Future work can propose optimal strategics for both of these nuances.
Greedy subset selection. Even for the ViT-B/32 model version with fewer than 100 input tokens (nodes in graph), finding cliques of the highest and smallest sums using brute force is computation- ally prohibitive. Still, we want to use these subsets denoted as Me.min, and Me k in evaluation with insertion/deletion curves (Definition 9) and explanation visualization. Thus, we implemented simple greedy algorithm starting the search from each (or subset) of the tokens (nodes). Its general goal is to add consecutive tokens to the subset based on minimizing/maxiiizing the subset's value. For models with larger number of inputs, e.g. + WT = 196 + 30, the greedy strategy remains applicable; however, may require a few minutes to complete when considering all subset sizes k L,..., RZ + NT.
23

---

B.2 Area between the insertion/deletion curves (AID)
Figure describes visually the process of computing insertion/deletion curves and how can differ between the first-order attribution methods and second-ord order interaction explanations. Consider the following example shown in (bottom, Ours, 73% ), where the faces of smiling man and child are masked with the masked caption saying "a man model predicts the input to be dissimilar (helow-average similarity), which means enters negative values in our normalized case. Contrarily, in (bottom, Baseline), the unmasked restaurant text token keeps the model's similarity near the average level. Another interesting phenomenon happens for the insertion curve in Figure 9 (top). Faithfully masking the redundant information with our method causes the model's s similarity to increase above the original prediction. Contrarily, in (top, Baseline), the baseline gradient-basee based method is unable to faithfully recover such redundant tokens.
child smil smiling at a in a a-". Here, the
B.3 Pointing game recognition (PGR)
Figure illustrates an example of the pointing game evaluation for interaction explanations like FIXLIP, giving additional context to why first-order attribution methods fail to pass this desirable sanity check. A saliency map can only highlight important parts of image/text inputs, but whenever more objects/concepts appear in the input, they are unable disentangle the basic rela- tionship between the two modalities. PGR metric measures the ratio of absolute values of "correctly" identified cross-modal interaction terms to total interaction terms. For example, in Figure 8 (2nd row, 2nd column), we sum the positive interactions between "banana" image tokens and the banana/cat text tokens, as well as the absolute negative interactions between "tractor"/ image tokens and the banana/cat text tokens. Then, we divide this sum by the total absolute sum of all interactions, which gives normalized PGR score. In Appendix C.2, further describe the specific combinations of image and object/class labels used to create our exemplary benchmark in this paper, although the overall methodology should be treated in generic manner.
1 object
2 objects
3 objects
4 objects
banana
banana 3c
banana
banana
banana
banana caat
banana cat' tractor
banana cat tractor bail
Banana
Ganana cat
Gananal cat tractor
Bananal cat tractor ball
Figure 8: Visualization of a first-order attribution (top row), e.g. Shapley values, compared to cross-modal interactions in FIXLIP (middle row) for an exemplary pointing game. FIXLIP can be conditioned on any token, e.g. the first text input token (bottom row) for a simpler visualization. Consecutive columns here correspond to consecutive columns in Tables 1, 2 & 4.
24

---

Ours
92%
73%
5%
mur und undd and did 920 lesourtt
thn and thi dlarringg at piz moal
than and ceM pizaimo)
maat pizzi
Baseline
and pitteimo
PI2 meal F1
p022 med 1
pizze mext
Prediction change (normalized]
Explanation method
second-order interactions
first-order attributions
Evaluation
0.5
insert important (1)
delete important (4)
0.0
-0.5
100%
80%
60%
40%
20%
0%
Percentage of input / (Pea + Dmg)
Baseline
Smarg
real in
restaurant
Festarr 11
Ours
92%
73%
40%
15%
59
piza inia restautann
p:200 meal
0220 Heil
RcAarart
Figure Exemplary insertion/deletion curves for two explanations e computed with FIXLIP and another first-order method. We show five subset visualizations per curve per method. In the case of the baseline method, each consecutive subset needs to include the preceding one, which is not the case for FIxLIP, capturing the complex cross-modal and intra-modal interactions. For example, chile with the corresponding face patches (top right corner) gets overvalued by the man-pis san-pizza tokens appearing jointly with the corresponding image part (child disappears). Similarly, in the deletion curve, the subsets of text and image tokens interchange with each other at around 50% (bottom row).
25

---

Experimental Setup
C.1
Models
We use the following openly available models from Hugging Face with default hyperparameters:
CLIP VIT-B/32 [57]: openal/clip-ritt-ba--paaaaaa (MIT License)
CLIP ViT-B/16 [57]: opena//alip-xtttbbaeppptcctl (MIT License)
SigLIP ViT-B/16 [75]: go0gl//ssiiii--aase-ptttt1---112 (Apache License 2.0)
IPgLIP VIT-I/16 [75]: gogg///sss111g11aggggaahhhh22222 (Apache License 2.0)
SigLIP-2 ViT-B/32 [66]: goggg///iigglgg---aaaaaaaa222333 (Apache License 2.0)
SigLIP-2 ViT-B/16 [66]: gogggs//aiggllg--aaasaatthh2222 (Apache License 2.0)
SigLIP-2 ViT-L/16 [66]: google//sggl1p--aaggeeataclllll (a (Apache License 2.0)
We additionally rely on CLIP VIT-B/32 and ViT-B/16 model versions from the clip Python library [57, MIT License], not to modify the official implementation of the baselines (Appendix C.3).
We set the batch size to 64 for the base models and to 32 for the large models, performing computation on A100 GPUs with 40GB VRAM.
C.2 Datasets
We use the following openly available datasets from Hugging Face:
MS COCO [42]: clip-benchhmrrk//ds_.neeee cttpsass_cccccooccccccccce (CC BY 4.0)
ImageNet- k validation set [15]: ILSVRC/imagenet-11: (ImageNet Agreement)
Experiments with the CLIP models are performed using 1000 image-text pairs from the MS COCO test set, for which each of the models predicted the highest similarity scores. Experiments with the SigLIP-2 models are performed using 100 image-text pairs to save computational resources, since we are not using it to compare with baseline methods.
Regarding ImageNet-Ik, we use all 50 images from each of the following 10 class labels for constructing the pointing game evaluation: goldfish (1), husky (248), cat (282), plane (404), church (497), ipod (605), ball (805), tractor (866), banana pizza these images with labels into 10 varied games follows: golddffsh-hhusk---izee--eee aat-goldfiish-plae----eee
pizza (963). We combine
Danana-cat-traatoo-bba pizza-ippo--olldfiiehb-aaaaa pl sane-churchh-all--ee
ipod-cat-husky-pleee husay-baanaa--plane--huulel
tractor-ball--baaaa---pd
ball-busky-bbmana-traee In each case, we distinguish four scenarios by gradually adding each consecutive token from left
church-pizza-ipod-cae
right, resulting in 40 scenarios, each with a total of 50 images, which constitutes reasonable case.
C.3 Baselines
We the following publicly available source code:
GAME http::////thhh..oo///aaaaaa/aaaa EMxa-aaaaa--llllaiiiiiyyy
(MIT License)
Grad [77]: https://githht: cow.uuan.c///aagg.... 2hao http.///yyyaggg-aaoaaaa----1111- (License unknown)
exCLIP [48]: Supplementary material at https://peenreriee.eee/oommm
p1krRJt98c (License unknown)
We were unable to run exCLIP on the CLIP (ViT-B/16) model version for inputs with more than 24 text tokens, which are common in MS COCO, running into ofmomo errors. Furthermore, we refrain from attempting to run these implementations on SigLIP models, which were not part of the original experiments. For context, we did not find an openly available implementation of InteractionI [32], while it not clear how to adapt InteractionCAM [61] (proposed for image- image encoders) to the text/language domain.
26

---

C.4 Metrics
For p-faithfulness correlation (Definition 8), we set m 1000 for reasonable sample size and analyze p = {0.3, 0.7, as well as Shapley sampling weights to give a broader context.
Note that the insertion/delction curves (Definition 9) have different lengths for each input (explana- tion), depending on the number of tokens (players). We thus interpolate each curve over a fixed range of 51 points between 0-100% input and then aggregate them obtain the final visualization.
Pointing game recognition has no additional hyperparameters beyond the pre-defined dataset described in Appendix C.2. For the example presented in Table 2, we omit the (sub)games with either goldfiss or ipod for SigLIP models, because their corresponding text tokenizer divides these class labels into multiple tokens.
C.5 Compute resources
Experiments described in Section 5 and Appendix D were computed on cluster consisting of 4x AMD Rome 7742 CPUs (256 cores), 4TB of RAM, and 16x A100 GPUs for about 15 days combined. envision that preliminary and failed experiments have required the same amount of compute resources.
27

---

D Additional Experimental Results
We report the insertion/deletion results for CLIP (ViT-B/16) in Figure 10 and for S (VIT-B/32) in Figure 11, where we also analyze the gradual relationship between p-masking in FIxLIP and the model's prediction change more gradually. Table reports pointing game results for CLIP (ViT-B/16), where interestingly FIXLIP with 0.5 outperforms FIxLIP (Shapley interactions).
Figure 12 shows further ablations based the different distributions in the p-faithfulness correlation metric. Interestingly, FIXLIP with p 0.7 evaluated on masks from 0.3 performs much better than FIXLIP with p 0.3 evaluated on masks from p 0.7, which further confirms motivation to increase p in machine learning applications where out-of-distribution sampling is not desirable.
Prediction change (normalized)
Method (Area between curves)
1.00
FIXLIP p = (0.91 = 0.19)
0.80
FIXLIP p = 0.5 (0.96 0.18)
0.60
FIXLIP p = 0.3 (0.84 0.20)
Banzhaf values (0.88 0.15)
0.40
Grad-ECLIP (0.56 0.15)
0.20
GAME (0.58 0.16)
Curve
0.00
insert important (higher better)
0.20
delete important (lower better)
100%
80%
60%
40%
20%
0%
Percentage of input k// (Next nimg)
Figure 10: Insertion/deletion curves for CLIP (VIT-B/16) on MS COCO. Extended Figure 3. AID score (higher is better) for FIXLIP against alternative explanation methods, where a random baseline scores O. The y-axis is normalized between the model's prediction the original input (100%) and the fully removed one (0%), where negative values denote that the model is predicting the inputs unsimilar. Gradient-basee based methods such as Brad-ECLIP fail recover nonlinear rankings of important tokens, whereas FIxLIP faithfully recovers the optimal subset explanation. exCLIP does not appear in the comparison as its implementation returns OOM error when querying the larger model (14 x 14 image tokens) with texts longer than 24 tokens, which are common in MS COCO.
Prediction change (normalized)
1.25
Method (Area between curves)
1.00
FIXLIP = (1.07 = 0.20)
FIXLIP p 0.6 (1.12 0.19)
0.75
FIXLIP p 0.5 (1.15 0.18)
0.50
FIXLIP p 0.4 (1.15 0.17)
FIXLIP 0 (1.10 = 0.17)
0.25
FIXLIP SI (1.12
0.00
Curve
insert important (higher better)
-0.25
delete important (lower better)
-0.50
100%
80%
60%
40%
20%
0%
Percentage of input k/ (Nex + nimg)
Insertion/deletion curves for SigLIP-2 (VIT-B/32) on MS COCO. Extended Figure 3. AID score (higher is better) for model-agnostic estimator of FIXLIP with different masking weights. The y-axis is normalized between the model's 's prediction on the original input (100%) and the fully removed one (0%), where negative values denote that the model is predicting the inputs are unsimilar. We observe a gradual relationship between p and the steepness of the curves at particular fraction of input deleted. We omit comparison with other baselines in this case as they were originally implemented for CLIP and not SigLIP.
Figure 11:
28

---

Table 4: Pointing game recognition for CLIP (VIT-B/16) on ImageNet-Ik.. Extended Table 1.
PGR score for FIXLIP against alternative explanation methods, where a random baseline scores 0.25. First-order methods, such as Grad-ECLIP and Shapley values, fail distinguish between multiple objects simultancously, while second-orde methods faithfully recover the appropriate explanation (up the irreducible non-optimality of the pointing game).
Explanation Method
Recognition (t)
object
objects
3 objects
4 objects
GAME [12]
.60
.41 1.04
..0
.27
Grad-ECLIP [77]
.721.13
..4
13
.261.01
Shapley values
664.07
.57+.05
.50..04
.42..08 .03
Banzhaf values
.7+..8
.584.05
.50+.05
.11..04
exCLIP [48]
.734.15
.82..07
.854.05
.87..05
FIXLIP (Shapley interactions)
.76+.08
.78+.05
.79+.05
.79+.05
FIXLIP (w. Banzhaf interactions p 0.3)
.764.10
.754.08
.774.07
.772.06
FIXLIP (w. Banzhaf interactions = 0.5)
.814.08
.804.08
.814.05
.81..05
FIXLIP (w. Banzhaf interactions p 0.7)
.751.09
.751.07
.774.06
.784.05
29

---

CLIP (VIT-B/32)
1.0
p-faithfuiness correlation
0.6
0.4
p-faithfulness metric distribution
p 0.3
0.5
0.7
exCLIP
GAME
Grad-ECLIP
Shapley values
Banzhaf values
FixLIP (SI)
FiXLIP
FIXLIP
FIXLIP (WBIp- 10llpoo0) 0.7)
(WBIp- 0.3) (WB), 10ppmm. 0.5)
CLIP (VTT-B/32)
1.0
0.8
0.6
0.4
0.2
p-fatthfulnees metric distribution
0=3
p 0.5
p= 0.7
0.0
Shapley values
Banzhaf values
FIXLIP (SI)
FiXLIP (WBI,= oa) (WBIp=o.) (WBIp=07)
FIXUIP
FiXLIP
CLIP (MT-B/16)
1.0
D-faithfulness correlation
0.8
0.2
p-falthfulnees metric distribution
p
p 0.7
GAME
Grad-ECLIP
Shapley values
Banzhaf values
FIXLIP (SI)
FIXLIP
FIXLIP (WBIp-o.s) (WBIp-07)
FIXLIP
(WBIp-o3)
CLIP (VT-B/16)
1.0
0.8
0.6
0.4
0.2
p-faithfulness metric distribution
p= 0.3
P=05
0.7
0.0
Shapley values
Banzhaf
FIXLIP (SI)
FIXLIP (WBIp- oa) (WBIp (o.) (WBIp-
FIxUP
FixLIP
values
Figure 12: Faithfulness evaluation for CLIP (VIT-B/32 and VIT-B/16) on MS COCO. Extended
Figure 4. We evaluate the faithfulness of different FIXLIP variants and baselines in terms of p- faithfulness correlation and the Ri2 coefficient (where is possible for game-theoretic approaches). For 000 inputs, we sample 1 000 evaluation masks according to different p-fatthfulness distributions:
p 0.3, p 0.5, p 0.7. For each f-itthffulnees distribution, the corresponding FIXLIP (w.
Banzhaf interactions) outperforms the baselines.
30

---

E
Visual Explanatons
E.1
Guidance on interpreting visualizations of interaction explanations
In all of our visualizations, explanations are highlighted using a red-blue color scheme, where red indicates a positive contribution to the model's output and blue negative one. The transparency of the color reflects the strength of the contribution- morei more opaque elements correspond to higher absolute explanation values. Pairwise interactions are represented as edges, main effects (or first-order attributions) are displayed as colored boxes around words or overlayed as heatmaps on top of images. In addition to the color and transparency, edges also vary in thickness based on the interaction strength. In general, interactions may link elements within the same modality, e.g. text-text interactions or image-image interactions, or across modalities, e.g. text-image interactions.
Figure 13 illustrates how to interpret visualizations in multimodal setting involving image and text inputs. Provided an input image and text pair (Figure 13A), we compute FIXLIP and retrieve an explanation containing interactions and main effects. The FIXLIP output (Figure 13B) can be visualized by plotting the image and text together, and overlaying the main effects and interactions. This allows for a quick screening of interesting parts in the input as judged by an explanation. In this particular example, the elephant text token positively linked to all image tokens containing the elephant. The two text tokens white and bird are linked to the one image token containing the bird. Here, elephant red because removing would decrease the predicted similarity score, while bird is blue because removing it might even increase the predicted similarity score (depending on cross-modal interactions). Based on the FIXLIP, more detailed visualizations can be created.
Conditioning on tokens. To investigate the role of a specific token i E NT U Nz, we can condition the interaction space on i and retrieve all interactions of the form e(r,}} for E N \ {i}. This conditional view highlights how token i interacts with all other tokens and is especially useful for analyzing its contextual relevance across modalities. Then, we project the interactions onto the first-order and visualize them with a heatmap (Figure 13C). Conditioning on elephant shows the greatest positive interactions with the body in the image tokens and the word small. Conditioning on the image token containing the bird mostly interacts with white and bird.
Finding interesting subsets. FIXLIP explanations allow searching for interesting subsets M like Me.ax. or min. for varying sizes of k. We can search the space of interactions and visualize only the explanations for tokens in these subsets, or overlay the masks on top of the input. In Figure 13D, we illustrate Me, m containing tokens related to elephant. Starting the search from the bird token retrieves the k subset with information about the bird. While the elephant subset increases the similarity, the bird subset could slightly decrease the similarity, presumably because the model would find the isolated image- text similarity of the elephant clearer.
Visualizing heatmaps. Based on Theorem 1, second-order explanations provided by FIXLIP can be converted into first-order explanations. Hence, FIXLIP explanations can also be visualized with heatmaps (Figure 13E). Similar to baselines such as Grad-FECLIP or Shapley values, we overlay the converted first-order FIXLIP explanations onto the input. The scores are scaled jointly across both modalities since we retrieve the explanation scores across both modalities, An alternative could be scale scores independently with different maximum values (one per modality). Joint scaling allows for relative comparisons, which is not the case for independent scaling, where two tokens are visualized with full color intensity. Figure 13E (left), the independently scaled heatmap shows that the vision token containing the bird and the elephant text token have the highest scores in their respective modality. Jointly scaling the heatmap reveals that the elephant text token of higher score than the vision token containing the bird.
31

---

(A) Input
(B) FIxLIP output
a small elephant standing next white bird
a small elephant standing next to a white bird
(C) Conditioning on tokens
sma next
felephant standing white bird
small elephant standing next to white birdi
small elephant standing next to white bird
(D) Finding interesting subsets
Subset with (M) Ps 8.7
Subset with (M) s -0.9
small
bird
ellephant
white
small elephant
white bird
(E) Visualizing heatmaps
Scaled independently
Scaled jointly
small elephant standing white bird
small elephant standing to white bird
Figure 13: Showcase of explanation visualization. A comprehensive description is in Appendix E.
32

---

E.2
Additional examples
Figure 14 demonstrates the utility of conditioning on tokens. Figure 15 shows intriguing examples of interactions in inputs with text written in an image, which is related to the typographic capabilities of large vision- language models [10. 23, 63].
First-order explanation
Second-order explanation
Conditioning on tokens
pineapple banana piece of meat placed next to other
pineapple
piece of meat placed
Diece of meatt placed next to each other
pineapple
to each otter
positive
negative
condition
piineaple)) piece
banana and meat other
8. pinespale Gananail and piece placed next each other
each
Figure 14: FIXLIP provides a utility of conditional heatmap generation. Comparison between first-order attribution explanation and second-order interaction explanation, which is able to recognize complex nuances of the presented scene.
33

---

ABBEY ROAD NW8
CITY
sign that says
abbey road nw city of westminster
FEB. 28
Tifeee THEORY GALTRA INCUUSIVE
The Pantry
sign
reading
juice
theory
near a traffic light
HAIGH
AGHO RRY
the streets signs at the intersection of ha iihh and ash bury
Figure 15: Intriguing examples. Top interactions in inputs with text written in image.
34

---

E.3
Comparison between CLIP (VIT-B/32) and SigLIP-2 (ViT-B/32)
See Figure 16. Note that these models have different text tokenizers and imput image resolutions, i.e. 224 x 224 vs. 256 x 256, which is why the images differ slightly.
CLIP (VIT-B/32)
S (VIT-B/32)
aduR bata
adaR dealis and SaNy diath wuiting aganat
SOME stalte
Srale wa king against
BOSSO in neWrot DORTO S00 W plass
BOISRE RE-R os eating swack with 1968
8 poosen hebrel eatng Sncc
STASE with MENCN soAee
TOOM FAe parkes town
MOTER In font of for LN2 parter
water g frone hor bis rarked
3.EOOK OL AAM &es DANG AAMOOL DORE
TocK AT 2005 s6N0 EMTE AIFURVL
THORE
SOUNS BAM MaNK
wit Soagle the R90S
with Smagle the tratt
BRaTTOu EOGELY 33600 Julti IAss Use (rOUITaTT
Bosebalt
MAGSTA gUTTIC titioaal BTEKS the
amirest Sacecel DWTH WiTh BRX Booss the Bacsgroust
BEOR
SmGiTs mrntt
Sand down
tend BEAn for eriic WOOEE at 8000
water B00
Figure 16: Visual comparison between FIXLIP of CLIP (left) and SigLIP-2 (right).
35

---

E.4 Comparison between CLIP (VIT-B/32) and CLIP (VIT-B/16)
Visual examples between CLIP (ViT-B/32) and CLIP (ViT-B/16) are in Figure 17. Notably, in the example including the clock (Fig. 17, row 3), conditioning on the vision token containing the letters and part of reveals a strong interaction with another vision token of this word the "sign" and the text token town. This may suggest the presence of thinl-onder interaction between these tokens.
Input
CLIP (VIT-B/32)
CLIP (ViT-B/10 //)
dle a conn made chocae
BIG Wthh a RRAE made
BUEER WiTh NE SAZE FACH TAAE
BXGD 3 Cowe Det madd
pate with a Pote moze
0244
boo Mil SUR
860 wh
4338 Wi:: Wootts
SUl
TOWN N
TOWN HAXTUN
TDi HAXTUN
TOW N $1 X
HAXTUN
XTUU
lange
WAY ELAaEE BOER with reslardo W1 sontro
WILL BRRIGOS DARAEO
DOS With EECOON Potls
MATIC with Fenlando Wn contort
S00TC WAT FerTORIOO C3
gAnI with a rentande M0
gamte WiTh Fiieend
1003
wooden
black standing atlffed bears.
BRKKR 22115 BENKERRS 1988 WELLS FM B HEVRRAND DO
EcEient Maaniing
723 stuffos beans
stuffed
Figure 17: Visual comparison between FIXLIP of CLIP ViT-B/32 (left) and VIT-B/16 (right). Each heatmap is conditioned on the selected input token (in black), also for vision tokens (in yellow).
36

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: validate our claims empirically and provide proofs Appendix
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In Section 6.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: In Section A.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yss]
Justification: All models and datasets used are openly available. We share code imple- mentation for our method, experiments, and analyzes on GitHub. We add details on the implementation and experimental setup in Appendices B & C, respectively.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All models and datasets used are openly available. We share code implementa- tion for method, experiments, and analyzes on GitHub.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes].
Justification: We state experimental setting at the beginning Section 5, and elaborate on in Appendix C.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report metric values with standard deviations and via boxplots.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Answer: [Yes]
37

---

Justification: We estimate compute resources in Appendix C.5,
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http///nuripp..pp htcc/public/Btiicccueeeeeeeeeeee
Answer: [Yes]
Justification: We do not violate the NeurIPS Code Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We describe the broader impact in Section 6.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release new data or models.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original papers, state versions, include URLs, and list asset licenses in Appendix C.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not release new assets.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our method does not involve using LLMs.
38


Reviews:
[{"review_id": "KIbfpIJX8x", "reviewer": "Reviewer_gzAn", "summary": "This paper presents an explanation method for image-text dual encoder models that is based on interactions between image patch features and text token features. In practice, the method works by sampling many masks for the image patch and text token positions, computing for each match the image-text similarity according to the model, and fitting a least-squares regression that predicts the image-text similarity from the weighted sum of interactions between pairs of image and text features. The method is evaluated in several ways: an insertion/deletion setting, a pointing game setting, and a qualitative analysis. The method generally performs better than first-order attribution methods. The experiments also include an analysis of how the computational requirements of the method scale with the number of masks sampled.", "questions": "- Can you explain the method with less notation?\n- In the insertion/deletion evaluation, the proposed method is not uniformly better than the Shapley baseline for all choices of p. Are there hyperparams that could be tuned in the Shapley case and if so, is the reported Shapley line the best of those?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The experiments section is thorough. There are several applications or ways of evaluating the method, and confidence intervals/error bars are provided for at least some of the quantitative evaluations. In addition, a running-time analysis is provided.\n- The method does provide value over first-order attribution methods according to the insertion/deletion and pointing-game evaluations.\nWeaknesses:\n- Due to the heavy use of notation, the method section is quite challenging to understand and is overwhelming.\n- Isn’t Definition 2 circular, since we use b_i to define b?\n- What does it mean for a mask to be high or low similarity (line 202)?\n- Explanation of the insertion/deletion evaluation could be dramatically improved\n- Line 308 needs ref (“consistent with prior work”)", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank the reviewer for their time and effort in reviewing our paper.**\n\n> **Strengths:** The experiments section is thorough. There are several applications or ways of evaluating the method, and confidence intervals/error bars are provided for at least some of the quantitative evaluations. In addition, a running-time analysis is provided. The method does provide value over first-order attribution methods according to the insertion/deletion and pointing-game evaluations. \n\nThank you for highlighting **the thoroughness of our experiments**!\n\n> **W1:** Due to the heavy use of notation, the method section is quite challenging to understand and is overwhelming.\n\nThank you for your valuable feedback. To address it, we will add a notation table describing each symbol at the beginning of the Appendix. Our method is rooted in game theory, where we introduce the necessary mathematical notation to prove **Theorems 1 & 2**. In that, we extend the notation used in related work on applying Shapley and Banzhaf interactions in machine learning [16, 28, 44, 58].\n\n> **W2:** Explanation of the insertion/deletion evaluation could be dramatically improved\n\nTo make the insertion/deletion evaluation metric clearer, we provide its comprehensive description with a visual example aiding such an explanation in **Appendix B.2, Figure 7**.\n\n> **Q1:** Isn’t Definition 2 circular, since we use b_i to define b?\n\n**Definition 2** is not circular and introduces the operator $\\oplus_{M_o}$ that requires two inputs $x$ and $b$. We consider $b$ as a vector $b \\in \\mathbb{R}^n$, while we use its values indexed with $b_i$ to define the masking operator applied to $x$ and $b$.\n\n> **Q2:** What does it mean for a mask to be high or low similarity (line 202)?\n\nThank you for pointing out this mental shortcut; we mean token subsets (masks) of high and low similarity as evaluated by the game (model).\n\n> **Q3:** Line 308 needs ref (“consistent with prior work”)\n\nThank you for this valuable suggestion; we will add references [8, 13] to this sentence.\n\n> **Q4:** Can you explain the method with less notation?\n\nOur method can be described as a 3-step algorithm demonstrated visually in **Figure 1**. \n\n**Input:** The method for explaining a similarity value predicted by a vision–language encoder (e.g. CLIP) takes an image–text pair as input. It also takes a float probability parameter $p \\in (0,1)$ and an integer budget parameter $m^2 \\in \\mathbb{N}$. \n\n1. We sample uniformly $m^2$ variations of the input image–text pair (**Definition 2**). Each mask is sampled uniformly with probability $p$ for each token in a mask (**Section 3.1, line 126+**), and we take all pairwise combinations between $m$ masked images and $m$ masked texts (**Section 3.2,  line 155+**).\n2. We apply the explained model to predict similarity values for each of the $m^2$ input variations (**Definition 3**).\n3. We apply a regression-based least squares approximator to predict the model’s predictions from sampled binary masks (**Section 3.1, lines 146+**). \n\n**Output:** An explanation assigning attribution scores to each token and interaction scores to each pair of tokens in an input image–text pair (**Definition 1**). \n\nWe can add such an algorithm in the Appendix or in the paper, given additional space.\n\n> **Q5:** In the insertion/deletion evaluation, the proposed method is not uniformly better than the Shapley baseline for all choices of p. Are there hyperparams that could be tuned in the Shapley case and if so, is the reported Shapley line the best of those?\n\nThis is a great question. In the Shapley case, there is no such hyperparameter, while we report a line for the ‘best’ Shapley weights. The goal of our method (FIxLIP) is to allow for arbitrary sparseness of sampled masks, which is not possible in the Shapley case (see **Remark 2**). We introduce a flexible framework where the parameter $p$ can be used to achieve the best $p$-faithfulness metric results. For example, FIxLIP with $p=0.7$ outperforms FIxLIP (Shapley interactions, SI) as measured with 0.7-faithfulness correlation in **Figure 3 (left)**. Analogously, in **Figure 3 (right)**, FIxLIP with $p=0.5$ outperforms FIxLIP (SI) based on $R^2$ with $p=0.5$. In **Appendix D, Figure 10**, we show another interesting scenario analysing the SigLIP-2 (ViT-B/32) model, where FIxLIP with $p=0.4$ and $p=0.5$ outperforms FIxLIP (SI)."}, {"review_id": "cogxSDQ9qE", "reviewer": "Reviewer_Raws", "summary": "This paper proposes FIXLIP for interpreting similarity scores in vision–language encoders through second-order interaction attributions grounded in weighted Banzhaf interaction indices. The authors address the limitations of existing first-order saliency methods by modeling cross- and intra-modal token interactions, thus capturing richer and more faithful explanation structures. Experiments show that FIXLIP produces more faithful and informative explanations than prior state-of-the-art attribution methods.", "questions": "See weakness.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Authors extend explainability to second-order interactions using weighted Banzhaf indices, offering more expressive and faithful attribution than first-order approaches.\n2. Strong theoretical formulation.\n3. The experimental results show the superiority of the proposed second-order estimator,\n\nWeakness:\n1. What is WLS in Figure 1?\n2. Simply masking the patches and subtokens could be a bad choice as the final representations of patches and subtokens are entangled with each other. Also I assume masking subtokens means remove them from the sentence. But what does masking mean for patches? Do authors mask them as a whole black or white patch?\n3. Although second-order interactions are insightful, the method does not scale to third-order or beyond, which might limit expressivity in complex VLMs. A simple case could be just have the third-order established by edge transformation.\n4. It would be interesting to see how FlixLIP compared to previous works given the same budget. \n5. How does the subset selection affect performance (Sec. 3.3)? \n6. What are the computational overhead for different p settings in Table 1? Also what is the budget for other methods.", "was_revised": true, "final_justification": "This is a borderline paper to me while I lean to accept this paper. My biggest concerns are 1) the proposed method seems to be a  simple extension and 2) the budget might be a trouble in explanining models. An okay/good enough pape to me personally.", "rebuttal": "**We sincerely thank the reviewer for their appreciation of our work and the constructive feedback.**\n\n> **Strengths:** Authors extend explainability to second-order interactions using weighted Banzhaf indices, offering more expressive and faithful attribution than first-order approaches. Strong theoretical formulation. The experimental results show the superiority of the proposed second-order estimator,\n\nThank you for highlighting **our theoretical contribution**!\n\n> **W1:** Simply masking the patches and subtokens could be a bad choice as the final representations of patches and subtokens are entangled with each other. Also I assume masking subtokens means remove them from the sentence. But what does masking mean for patches? Do authors mask them as a whole black or white patch?\n\nWe agree that optimizing the masking technique for transformers is an open research question. We view this as an implementation detail of our proposed faithful interaction explanations, where one can implement it with any masking technique proposed in the literature. Kindly note that we mention implementing masking text tokens with *attention masking* (**lines 103–104**), and *not removing them from the sentence*, which indeed could be a bad choice. Regarding image patches, we mask them with a $0$ baseline value (after image normalization) to propagate no signal forward. These and further implementational details can be found in **Appendix B.1**.\n\n> **W2:** Although second-order interactions are insightful, the method does not scale to third-order or beyond, which might limit expressivity in complex VLMs. A simple case could be just have the third-order established by edge transformation.\n\nWhile we (truly) think that this is indeed an interesting direction, we think that it does not invalidate our method and positive results. Our work takes a first big step in offering a unique perspective on interpreting image–text similarity predictions through second-order interactions. While the weighted Banzhaf interactions can be naturally extended to third-order explanations, this entails computational and perceptual challenges. Note that we state *efficiently approximating higher-order interactions* as an important future work direction in **line 323**. Take for example the image–text pair with the clock and the sign in **Appendix E.2, Figure 17 (Row 3, Column 4)**: Conditioning on the vision token containing the letters “T” and part of “O” reveals a strong interaction with another vision token of this word in the sign and the text token “town”, which suggests *the presence of a third-order interaction between these tokens*.\n\n> **W3:** It would be interesting to see how FlixLIP compared to previous works given the same budget.\n\nMethods from previous works have no budget since they do not use model inferences to explain the similarity prediction. We compared FIxLIP to Shapley and Banzhaf values given the same budget in all experiments.\n\n> **Q1:** What is WLS in Figure 1?\n\nIt means a regression-based approximation with weighted least squares (WLS), which we will add to the caption.\n\n> **Q2:** How does the subset selection affect performance (Sec. 3.3)?\n\nThis is a great question. We only use the subset selection for insertion/deletion evaluation and visualizing explanation graphs. While the greedy subset selection is not optimal, we already achieve *state-of-the-art faithfulness performance*, where optimizing weighted clique finding could further improve our results. Regarding efficiency performance, it takes about two seconds to find subsets of all sizes for one explanation instance. Having said that, *efficiently detecting* and *analyzing* the subsets (explanation graphs) is a very interesting line of future work.\n\n> **Q3:** What are the computational overhead for different p settings in Table 1? Also what is the budget for other methods.\n\nTo clarify, there is no computational overhead for different $p$ settings. Shapley values use the model-agnostic estimator with a budget of $2^{17}$, while other methods have no budget hyperparameter (number of sampled masks) since they do not perform model inferences."}, {"review_id": "AcY875AZic", "reviewer": "Reviewer_WGoS", "summary": "This paper points out that the current explanation methods with saliency maps are limited by the unimodal view of the model’s output. The vision-language encoders should be explained by considering the cross-modal interactions between image-text pairs. Therefore, the paper proposes an explanation method to measure the importance of each token in the image and text encoders via weighted faithful Banzhaf interactions.", "questions": "- It is confusing that the explanation for the two-modality interaction is called second-order. If there is a definition, it should mention the reference.\n- It said that this is the first work using weighted Banzhaf interactions to overcome the out-of-distribution problem apparent in removal-based explainability in line 46. How to understand the out-of-distribution? It’s better to have a clear demonstration in the introduction. \n- I know the attribution-based methods like Grad-ECLIP and GAME are totally different from the methods with game theory. Then, what’s the main difference and the advantages of using Banzhaf compared with Shapley to estimate the token importance? It will be clearer to have a figure to show the comparison of the two kinds of algorithms.\n- For the explanation graph in Figure 1, why do the tokens of object parts have a negative contribution to the similarity, such as the two dog patches and the three hydrant patches? More demonstrations about the results in Figure 1 should be added. It’s hard to understand.\n\nAbout experiments:\n- In Figure 2, negative y-axis values can be obtained with the Sharpley and FIXLIP methods, while the minimum values for the attention heat map methods (Grad-ECLIP, GAME, and XCLIP) are zero. This will greatly increase the performance gap. The reason given in the paper is “negative values denote that the model is predicting the inputs are unsimilar.” What does that mean? Another image is also used in the deletion/insertion experiments? I'm still confused about why the similarity drop can be negative. \n- What is the concrete process in the PGR metric and sanity check in Section 5.2? I know that the point game is that a hit shows when the maximum value on the heat map is located on the correct object. So, what’s the operation when multiple objects are involved? And what is the sanity check with a pointing game? \n\nI may increase my rating after my concerns are solved.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Very comprehensive algorithm demonstration.\n- It's the first work to use game theory to generate explanations for cross-modal interaction in the vision-language models.\n\nWeaknesses:\n- The clarity of the paper needs to be improved. Concretely, please see the questions.", "was_revised": true, "final_justification": "Thank the authors for their thorough replies. My concerns are basically solved, so I would like to raise my rating to 4.", "rebuttal": "**We gratefully thank the reviewer for a thorough and insightful review of our work.** Below, we provide point-by-point responses to each of the questions.\n\n> **Q1:** It is confusing that the explanation for the two-modality interaction is called second-order. If there is a definition, it should mention the reference.\n\nFollowing the closely related literature [35, 46], we use *second-order* to denote interaction scores between *two* tokens (features). Please note that a *second-order* interaction is a subset of a more general term called *higher-order* or *any-order* interaction scores between feature subsets of *higher* or *any* size, following the related work [16, 28, 44, 58]. \n\n> **Q2:** It said that this is the first work using weighted Banzhaf interactions to overcome the out-of-distribution problem apparent in removal-based explainability in line 46. How to understand the out-of-distribution? It’s better to have a clear demonstration in the introduction.\n\nThank you for this valuable feedback, which we will use to improve the introduction. We understand out-of-distribution data as inputs with many masked tokens (**line 118**), where images become unrecognizable and text captions are ambiguous. We show an example of such an out-of-distribution input in **Figure 1**, where for $p=0.3$ most of the image–text pair is masked and unrecognizable to the model. Our method addresses the emerging requirement for controllable sparsity in mask sampling [5, 21, 23, 71] (**line 39**).\n\n> **Q3:** I know the attribution-based methods like Grad-ECLIP and GAME are totally different from the methods with game theory. Then, what’s the main difference and the advantages of using Banzhaf compared with Shapley to estimate the token importance? It will be clearer to have a figure to show the comparison of the two kinds of algorithms.\n\nWe highlight the two main differences between Banzhaf and Shapley in **Remark 2**. FIxLIP with $p$-weighted (Banzhaf) masking allows using the cross-modal estimator (**Definition 7**), which is far more efficient with respect to budget, i.e. obtains $m^2$ model inferences from $m$ input samples, as depicted with $m \\rightarrow m^2$ in **Figure 1**. Therefore, its *main advantage is computational efficiency*. Furthermore, it allows for prioritizing masks that are expected to better reflect in-distribution inputs. Yet, both Banzhaf and Shapley provide faithful explanations. We implement FIxLIP with Shapley using the baseline model-agnostic estimator (**Definition 6**), which obtains only $m$ model inferences from $m$ input samples instead. Note that we consider both approaches to be a contribution of this paper and compare them in experiments as appropriate.\n\n> **Q4:** For the explanation graph in Figure 1, why do the tokens of object parts have a negative contribution to the similarity, such as the two dog patches and the three hydrant patches? More demonstrations about the results in Figure 1 should be added. It’s hard to understand.\n\nCritically questioning the model’s reasoning, as demonstrated by the Reviewer’s curiosity in this point, is precisely the motivation for this paper. *The behaviour of vision–language encoders like SigLIP-2 can be highly complex*. Thus, although our contribution is to improve their understanding through interaction explanations that are measurably faithful to the similarity prediction, we cannot expect the model to be aligned with human intuition. In the case from **Figure 1**, we see that intra-modal interactions between image tokens are relatively unimportant as compared to the intra-modal interactions between text tokens and cross-modal ones. We hypothesize that the dog and hydrant image patches carry a lot of redundant information. Thus, interactions between these tokens can be slightly negative as compared to the strong ‘hydr’–‘ant’ text interaction and cross-modal interactions. Note that the graph-like visualization of explanations is *only one of the three approaches* we showcase in **Figure 5**. We also provide a visual comparison between FIxLIP and baseline methods using the dog/hydrant example in **Appendix E.2, Figure 13**. Finally, to aid in the understanding of the introduced visualizations, we provide a comprehensive guide on interpreting them in **Appendix E.1**. \n\n> **Q5:** In Figure 2, negative y-axis values can be obtained with the Sharpley and FIXLIP methods, while the minimum values for the attention heat map methods (Grad-ECLIP, GAME, and XCLIP) are zero. This will greatly increase the performance gap. The reason given in the paper is “negative values denote that the model is predicting the inputs are unsimilar.” What does that mean? Another image is also used in the deletion/insertion experiments? I'm still confused about why the similarity drop can be negative.\n\nThis is a great question. In theory, all of the methods (Grad-ECLIP, GAME, XCLIP, FIxLIP) can obtain negative normalized values on the Y-axis. *Negative values denote that the model is predicting the image–text pair to be unsimilar*. It means that the prediction on a masked input is smaller than the prediction on the empty input (see **Appendix B.1, Figure 6**). We give a supplementary explanation of the insertion/deletion process on a single image in **Appendix B.2, Figure 7**. Consider the following example shown in **Figure 7 (bottom, Ours, 63%)**, where the faces of a smiling man and child are masked with the masked caption saying ‘a man — a child smiling at a — in a —’. Here, the model predicts the input to be dissimilar (below-average similarity), which means going into negative values in our normalized case. Contrarily, in **Figure 7 (bottom, Baseline)**, the unmasked ‘restaurant’ text token keeps the model’s similarity near the average level. Another interesting phenomenon happens for the insertion curve in **Figure 7 (top)**. Faithfully masking the *redundant* information with **our** method causes the model’s similarity to *increase* above the original prediction. Contrarily, in **Figure 7 (top, Baseline)**, the baseline gradient-based method is unable to faithfully recover such redundant tokens. Finally, note that **Figures 2, 9 & 10** show an average over a set of such examples given in **Figure 7**. \n\n> **Q6:** What is the concrete process in the PGR metric and sanity check in Section 5.2? I know that the point game is that a hit shows when the maximum value on the heat map is located on the correct object. So, what’s the operation when multiple objects are involved? And what is the sanity check with a pointing game?\n\nKindly note that we provide a comprehensive description of this process in **Appendix B.3, Figure 8**. When multiple objects are involved, the pointing game recognition (PGR) metric measures the ratio of absolute values of “correctly” identified cross-modal interaction terms to total interaction terms. For example, in **Figure 8 (2nd row, 2nd column)**, we sum the positive interactions between ‘banana’/‘cat’ image tokens and the ‘banana’/‘cat’ text tokens, as well as the absolute negative interactions between ‘tractor’/‘ball’ image tokens and the ‘banana’/‘cat’ text tokens. Then, we divide this sum by the total absolute sum of all interactions, which gives a normalized PGR score. By performing the *sanity check*, we check that interaction explanations are essential for explaining vision–language encoders as compared to baseline attribution approaches. Scoring high PGR values across 1, 2, 3, and 4 objects in **Table 1** denotes that an explanation method can distinguish between multiple objects at once, thus passing the sanity check.\n\n> I may increase my rating after my concerns are solved.\n\n**Thank you for taking this into consideration.** We would greatly appreciate an increase in rating if you feel we adequately addressed your questions. Else, we would appreciate an opportunity to address any remaining concerns."}]

Author rebuttal:

Meta-review: The paper proposes a novel approach for interpretability of CLIP-style image-text tokens, that gives insight into the cross-modal interactions between image-patch tokens and text tokens (called “second-order” in the paper).

All reviewers point out the novelty of the proposed approach and commend the thoroughness of both the technical analysis of the method, and the empirical evaluation. All reviewers recommend (weak) acceptance of the submission, and the authors were able to clear up all questions during rebuttal.

It should be noted that all reviewers provided relatively low confidence values for their reviews. Judging from the reviews themselves, this is in part because of challenges in understanding the writing of the paper. While all reviewers seem to have grasped the proposed approach in the end, multiple reviewers pointed out that reformulating the approach section of the paper could go a long way in making it easier to digest / understand. Thus, while I think that the technical novelty and thoroughness of the paper warrants its acceptance, I strongly encourage the authors to consider rewriting their approach section, and providing more intuitive explanations in addition to precise notation, which could go a long way towards strengthening the impact of the paper.
Decision: Accept (poster)