Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributions
Authors: Lucas Moeller
Pascal Tilli
Thang Vu
Sebastian Padó
Venue: Accepted by TMLR ()

Abstract: Dual encoder architectures like Clip models map two types of inputs into a shared em-
bedding space and predict similarities between them. Despite their wide application, it is,
however, not understood how these models compare their two inputs. Common first-order
feature-attribution methods explain importances of individual features and can, thus, only
provide limited insights into dual encoders, whose predictions depend on interactions be-
tween features.
In this paper, we first derive a second-order method enabling the attribution of predictions
by any differentiable dual encoder onto feature-interactions between its inputs. Second, we
apply our method to Clip models and show that they learn fine-grained correspondences
between parts of captions and regions in images. They match objects across input modes
and also account for mismatches. This intrinsic visual-linguistic grounding ability, however,
varies heavily between object classes, exhibits pronounced out-of-domain effects and we can
identify individual errors as well as systematic failure categories. Code is publicly available:
https://github.com/lucasmllr/exCLIP

Published in Transactions on Machine Learning Research (07/2025)
Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributions
Lucas Moeller
lucas. moeller@ims. uni-stuttgart. de pascal. filli@iissmm.m uni-stuttgare de wu@ims ani-stullgart.c de pado @ims, uni-stullgart. de
Pascal Tilli*
Ngoc Thang Vu
Sebastian Pado
University of Stutigart
Reviewed on OpenReview: https: //
// openrevie.. net/ id- HUUHHHHHU1990
Abstract
Dual encoder architectures like CLIP models map two types of inpuls into shared em- bedding space and predict similarities between them. Despite their wide application, it is, however, not understood how these models compare their two inputs. Common first-order feature-attribution methods explain importances of individual features and can, thus, only provide limited insights into dual encoders, whose predictions depend on interactions be- tween features.
In this paper, we first derive a second-order method enabling the attribution of predictions by any differentiable dual encoder onto feature-interactions between its inputs. Second, apply method to CLIP models and show that they learn fino-grained correspondences between parts of captions and regions in images. They match objects across input modes and also account for mismatches. This intrinsic visual- linguisin grounding ability, however, varies heavily between object classes, exhibits pronounced out-of-domain effects and we can identify individual errors as well as systematic failure categories. Code is publicly available: https:/ h/gp::///ttttn tigtu......... wtt.tttw...ooomoeeeeeeeeotte com//ucaamllr/exxL..I
1 Introduction
Dual encoder models use independent modules to represent two types of inputs in a common embedding space and are optimized to predict a scalar similarity measure for them. The training objective is typically triplet or contrastive loss (Sohn, 2016; van den Oord et al., 2019). Popular examples include Siamese transformers for text-text pairs (SBERT) (Reimers & Gurevych, 2019) and Contrastive Language-Image Pre- Training (CLIP) models (Radford et al., 2021; Jía al., 2021) for lext-image pairs. The learned representations have proven to be highly informative for downstream applications, such as image classification (Zhang et al,, 2022a), visual question answering (Antol et al., 2015; Tilli & Vu, 2025), image captioning and visual entailment (Shen et al., 2021), as well as text or image generation (Chen et al., 2023a; Yu et al., 2022; Rombach et al., 2022). In (multi-modal) information retrieval, dual encoders enable efficient semantic search (Baldrati et al., 2022; Zhu et al., 2024; Formal et al., 2021; Xiong et al., 2021; Johnson et al., 2019), serving e.g. Retrieval- Augeevall--gmmeneeddee Generation (RAG) Gao al. (2023)
Despite the wide-spread application of dual encoder models, open question remains how these models compare their two inputs. Common first-order attribution methods like Shapley values (Lundberg & Lee, 2017) or Integrated Gradients (IG) (Sundararajan et al., 2017) can only provide limited insights into dual encoders because they attribute to individual features (Zheng et al., 2020; Ramamurthy et al., 2022; Janizek et al., 2021; Sundararajan et al., 2020). However, similarity fundamentally depends on comparisons and, therefore, on interactions of features (Tversky, 1977; Lin, 1998). In dual encoders this manifests in the final
*These authors contributed equally.

---

Published in Transactions on Machine Learning Research (07 (025))25)
Our second-order attributions
First-order attributions
Attribution slicing
Visualization
CLIPSurgery
ECLIP
imagee
Itusge paoo projiction
projection
Caption
MacKCLIP
IG
span selection
kid with headphones
kid with headphones feeding birds.
with headphones feeding birds.
headphones feeding birds.
feeding birds.
Caplion projection
caspiio
IG Deer next to a woman with an umbrella ECLIP
W
hounding-box
Deet next to woman
Deet next to a woman
Deer next to woman
Deet next to a woman with an umbrella
selection
with an umbrella.
with an umbrella.
with an imbrella
Slicing:
Selection
Summation
Heatmass:
positive
negative
Figure 1: (Left column) Our second-order attributions can point out interactions between arbi- trary spans in captions and regions in images. We can visualize them by slicing (yellow selection) our 3d attribution tensor with image dimensions (H, W) and caption dimension (details in Section 3). A selection can be projected onto the image (top-left) the caption (bottom-left) by summation (green arrows). Heatmaps for these projected attributions in shades of red/blue for positive/negative /negative values. (Right column) In contrast, first-order attributions only attribute the overall similarity between captions and images onto either the image (lop-right) the caption (bottom--igght). They canmol assess underlying interactions.
cosine-similarity of the two embeddings, resulting in all terms contributing to the output similarity score to contain multiplicative dependencies between the two inputs. In such multiplicative terms, a change in one involved feature aflects the contribution others; hence, these features interact.
Only few works have studied feature interaction in symmetric Siamese encoders (Eberle al., 2020; Möller et al., 2023; 2024; Vasilciou & Eberle, 2024) and they have remained almost entirely unstudied in non- symmetric dual encoders like CLIP (Joukovsky et al., 2023).
In this work, we address this research gap and aim at means to analyze which aspects in two given inputs dual encoders compare in order to predict a similarity for them. Our contributions are the following: (1) Motivated by the theory behind IG (cf. Appendix F), we derive a general second-order feature attribution method that can explain interactions between inputs any differentiable dual encoder model. The method does not rely on any modification of the trained model, nor on additional optimization. (2) We apply our method to a range of CLIP models and demonstrate that they can capture fine-grained interactions bet ween corresponding parts of captions and regions in images. They identify matching objects across the input modes and also penalize mismatches. Using imagc-captioning datascts with object bounding-bal box annotations, we evaluate the extent and limitations of this intrinsic visual-linguistic grounding ability in a wide range of CLIP models.
Figure illustrates our interaction attributions showing how they can point out corresponding parts of captions and regions images. In contrast, first-order alternatives cannot access these interactions but only provide insights into aspects important to the overall similarity between text and image input.
2

---

Published in Transactions on Machine Learning Research (07/2025)
2
Related work
Metric learning refers to the task of producing embeddings reflecting the similarity between inputs (Kaya & Bilge, 2019). Applications include face identification (Guillaumin et al., 2009; Wojke & Bewley, 2018) and image retrieval (Zhai & Wu, 2018; Gao et al., 2014). Siamese networks with cosine similarity of embeddings were carly candidates (Chen & Hc, 2021). The triplct-loss (Hoffer & Ailon, 2015) involving negative examples has been proposed as an improvement but requires sampling strategies for the large number of possible triplets (Roth al., 2020). Qian al. (2019) have shown that the triplet-loss can be relaxed to a softmax variant. Sohn (2016) and van den Oord et al. (2019) have proposed the batch contrastive objective which has been applied in both unsupervised (Caron et al., 2020) and supervised representation learning (Khosla et al., 2020). It. has led to highly generalizable semantic text (Reimers & Gurevych, 2019) and image embeddings (He et al., 2020) and ultimately to the CLIP training paradigm Radford et al. (2021).
Vision-language models process both visual and linguistic inputs. Zhang et al. (2022b) were the first to train a dual-encoder architecture with a contrastive objective on image-text data in the medical domain. Radford et al. (2021) and Jia et al. (2021) have applied this principle to web-scale image captions and alt-text data. In the following, the basic inter-modal contrastive loss has been extended by intra-modal loss terms (Goel et al., 2022; Lee et al., 2022; Yang et al., 2022a), self-supervision (Mu et al., 2022), non- contrastive objectives (Zhou et. al., 2023), incorporating classification labels (Yang et al., 2022b), textual augmentation (Fan et al, 2023), unified multi-modal encoder architecture (Mustafa et al., 2022) and retrieval augmentation (Xie et al., 2023). Next to more advanced training objectives, other works have identified the training data distribution to be crucial for performance: Gadre et al. (2023) have proposed the DataComp benchmark focusing on dataset curation while fixing model architecture and training procedure, Xu et al. (2024) have balanced metadata distributions and Fang et al. (2024) have introduced data filtering networks for the purpose. The strictly separated dual-encoder architecture has been extended to include cross-encoder dependencies (Li et al., 2022a; Pramanick et al., 2023), and multi-modal encoders have been combined with generative decoders (Chen et al., 2023a; Lu et al., 2023; Li et al., 2021; Koh et al., 2023; Alayrac et al., 2022; Yu et al., 2022).
Local feature attribution methods aim at explaining given prodiction by assigning contributions to individual input features (Murdoch et al., 2019; Veshi- & Kim, 2017; Lipton, 2018; Atanasova et al., 2020). First-order gradients can approximate a prediction's 's sensitivity to such features (Li et al., 2016) and gradientxinput saliencies can approximate feature importance (Simonyan et al., 2014). In transformer archi- tectures, attention weights have been analyzed (Abnar & Zuidema, 2020), but were subsequently contested as explanation because they are only one component. of the model (Jain & Wallace, 2019; Wiegreffe & Pin- ter, 2019; Bastings & Filippova, 2020). Layer-wise relevance propagation (LRP) defines layer-specific rules to back-propagate attributions to individual features (Montavon et al., 2019; Bach et al., 2015). In contrast, Shapley values (Lundberg & Lee, 2017) and IG (Sundararajan et al., 2017) treat models holistically and can provide a form of theoretical guaranty for correctness. This has recently been challenged by Bilodeau et al. (2024) who proved fundamental limitations of attribution methods. A widely used attribution method in the vision domain is GRADCAM (Selvaraju et al., 2017), which Chefer et al. (2021) and Bousselham et al. (2024) extended to transformer architectures.
Assigning importances to individual features, first-order attribution methods cannot capture dependencies on feature interactions. Tsang et al. (2018) have proposed to detect. such interactions from weight matrices in feed-forward neural networks, Cui et al. (2020) investigated them in Bayesian networks. The Shapley value has been extended to the Shapley (Taylor) Interaction Index (Grabisch & Roubens, 1999; Sundararajan et al., 2020; Fumagalli et al., 2024) and Janizek et al. (2021) have generalized IG to integrated Hessians. Plummer et al. (2020) and Zheng et al. (2020) have assessed interactions underlying similarity predictions in Siamese image encoders. Eberle et al. (2020) extended LRP for this class of models (Vasileiou & Eberle, 2024), and our prior work extended IG to Siamese language encoders (Möller et al., 2023; 2024). In this work, we further generalize this method to multi-modal dual encoders.
CLIP explainability. Several works have previously pursued the goal of better understanding CLIP mod- els and contrastive image encoders. Wang et. al. (2023) and Kazmierczak et al. (2024) have proposed in-
3

---

Published in Transactions on Machine Learning Research (07/2025)
formation bottleneck approaches. Bhalla al. (2025) identified interpretable sparse concepts in the shared embedding space. Rasekh et al. (2024) predicted human-understanddale rationales for images. Quantmeyer et al. (2024) localized where the text encoder processes negation. Giulivi & Boracchi (2024) created saliency maps for WORDNET concepts. Chen et al. (2022) proposed an improved CAM variant and analyzed which objects the model looks at. Materzyúiska et al. (2022) were interested the entanglement of image rep- resentations. Gandelsman et al. (2023) identified the roles of individual attention heads in CLIP's image encoder and later investigated second-order effects of neurons (Gandelsman al., 2025). Lewis al. (2024) analyzed whether CLIP models adequately handle compositional concepts. Sam et al. (2024) investigated the model's reasoning ability about differences in images and Tu et al. (2024) examined safety objectives. The unseen performances of CLIP have motivated number of authors to identify the reasons behind its ostensible generalization ability and robustness towards domain shifts (Xue el al., 2024; Nguyen el al., 2022; Fang et al., 2022; Tu et al., 2023; Mayilvahanan et al., 2024; 2025). Zhao et al. (2024) explored a wide range of first-order methods to attribute similarity scores to images and captions independently and Li et al. (2023) proposed the CLIPSURGERY method. Sammani al. (2023) and Lerman et al. (2021) independently introduced a second-order variant of GRADCAM that can assess feature interactions. It can be applied to CLIP; in Appendix G, show that it is special case of our method. Most closely related to our work, INTERACTIONLIME (Joukovsky et al., 2023) pioncered the attribution of interactions between captions and images in CLIP models. However, relying on a local bilinear approximation of CLIP, it does not explain the original model and requires additional optimization as well as hyper-paramett tuning (cf. Appendix H), Last, ITSM by Li et al. (2022c) and the method by Black et al. (2022) are forward-facing saliency moth that compute importance values through pair- wise embedding multiplication. We compare these approaches against ours in Section 4.2.
Visual-linguistic grounding refers to the identification of fine- fineggraiedd relations between text phrases and corresponding image parts (Chen et al., 2023b). Specialized models predict regions over images for a corresponding imput phrase (Sadhu et al., 2019; Ye et al., 2019). This objective has been combined with contrastive caption matching (Li et al., 2022b; Datta et. al., 2019), and caption generation (Yang el. al., 2022c). The VoLTA model internally matches latent image-region and text-span representations (Pramanick et al., 2023). In multi-modal text generative models, grounding has been included as an additional pretraining task (Li al., 2020; Su al., 2019; Chen et al., 2020) and can be unlocked with visual prompt learning (Dorkenwald et al., 2024). At the intersect of grounding and explainability, Hendricks et al. (2016) have generated textual explanations for vision models and have grounded them to input images (Hendricks et al., 2018; Park et al., 2018). In this paper, we do not optimize models to explicitly ground predictions, but aim at analyzing to which extent purely contrastively trained dual encoders acquire this ability intrinsically.
3 Method
We first derive general second-order attributions for dual encoder predictions enabling the assessment of feature-interactions between their two inputs. In the following, we then describe their realization in trans- former models, specifically CLIP.
Derivation of second-order attributions. We begin from the definition of a dual encoder f,
= =ffaa (h()),
= ((a.b)=g(a
with two vector-valued encoders and h, respective inputs a and b and scalar output 8. We also define two reference inputs Ia and rb. whose role will be discussed later. With these definitions, we can write the following expression,
f(a, b) - f(r., b) - f(a, rb) + f(ra, rb),
(2)
which will serve as a rigorous starting-poii of our derivation. In the following, we first proceed by showing the equality of this initial starting-poine to Eq. 10. We then reduce this equality to our final result in Eq. 11 using the approximations discussed below. At this point, we are also discussing an intuitive interpretation of the final result.
4

---

Published in Transactions on Machine Learning Research (07/2025)
As first step, see as an anti-derivative and reformulate the above expression into an integral over its derivative:
[f(a.b] b) - fr b)] - [f rb) - f(ra. rb)]
0 θy, y) - ((r: dy, = 110 0y,0x: f yx, dx, dy,
b
0²
(3)
Here, x and y are integration variables for the two inputs. This step can be seen as the second-order equivalent to Equation 12 in the theory behind IG (ef. Appendix F). We use component-wise notation with indices i and for the imput dimensions and omit sums over double indices for clarity. We then plug in the model definition from Equation 1,
8²
Ox,dy, ga (x) hy( dx: dy
(4)
again using component-wise notation for the dot-product with & indexing the dimension of the shared embedding space. Since noither embedding depends on the other integration variable, we can separate both integrals and derivatives applying a product rule:
(g)(x) 0x,
(h:(y) dy;
dx;
(5)
Jr. 0y;
Both terms are line integrals from the references the actual inputs in the respective input representation spaces; (g) and (y) are the Jacobians of the two encoders. To proceed with these integrals, define integration paths and substitute. We follow Sundararajan et al. (2017), and use the straight lines between both references and inputs,
x(a) = ra + a(a ra); y(ß) = rb + 3( rb),
(6)
(7)
parameterized by and B, respectively. For the integral over encoder g substituting the path x(a) yields an integral over the scalar integration variable
L. 8gg (x(a)) 0x;(a) da (a-rr)) ("
Ōga (x(a)) do ix;
0x, da
Since 0x(a)/8a = (a - ra) is a constant w.r.t a, we can pull it out of the integral. integrated Jacobian for the encoder g,
We then define the
N (x(x)) (' 8g (x((x) da Jo Ox;
J ki
Ox,
as the analogon to integrated gradients for vector-valued models. The integral over encoder h can be processed in the same way by substituting y(3) to obtain J In practice, these integrals are calculated numerically by sums over N steps, with on n/N. This introduces an approximation error which must, however, converge to zero for large N by definition of the Riemann integral. We plug the results from Equation and the definitions of the integrated Jacobians into Equation 5:
(a Fa): (b ro); = Σ A
(10)
After computing the sum over the output embedding dimension k this yields interaction terms, each involving a feature pair (i,j) with feature from input a and feature j from input b. We can write the values of these terms for all fcature pairs into a matrix with index on one side and on the other, which we refer as the attribution matrix Aij. the last step, we write out the omitted sum over and j explicitly. Note that except for the numerical integration, the equality to Equation 2 still holds. Hence, the sum over all feature
5

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
A hot dog sitting on a table covered confetti. Surrounded by glitter, there is sausage in bun.
A dog sitting on a table covered confetti. Surrounded by glitter, there is sausage in bun.
Figure 2: (Left) Intra-modal text-text attributions between top and bottom captions (top: selections in yellow, bottom: corresponding attributions red/blig for positive/negnne /ngateive. (Right) Intra-modal image- image attributions between left and right image (left: bounding-box selection in yellow, right: heatmaps above). More examples can be found in Figure 10.
pair attributions in A is an exact reformulation of our starting-point. At this point, we return to the references, Ia and rb, defined above. We require them to be approximately dissimilar to any other input, c.g. a black image or a caption consisting of padding tokens for respective encoders. If this is the case all three terms involving r, and r, Equation 2 approxinately vanish, i.e. f(ra, b) = f(a, 16) 0, f(r, 0. This reduces the equality between Equations and 10 our final result:
f(a, b)
Σ A.j.
(11)
Intuitively, this provides an approximate decomposition of the model prediction s= f(a, b) into additive contributions of feature-pair interactions between the two inputs. Throughout this paper, we evaluate the attribution matrix A.
Interaction attributions in transformer models. In the derivation above, we treat image and text representations vectors. In transformer-basee encoders, text inputs are represented Sx dimensional tensors, where S is the length of the token sequence. Image representations with H and W being height and width of the image representation; in vision-transformers both equal the number of patches P. Dy and Dj, are the encoders' embedding dimensionalities. Our pair-wise image-text interaction attributions thus have the dimensions x W x Da X Dy, which quickly becomes intractably large. Fortunately, the sum over dimensions Equation 11 enables the additive combination of attributions in A. We sum over the embedding dimensions of both encoders D, and D₁. and obtain an H W x S dimensional attribution tensor, which estimates for each pair of text token and an image patch how much their interaction contributes to the overall prediction. These attributions still threo-dimensional and thus not straightforward to visualize. However, again we use their addilivity, slice the 3d attribution tensor along text or image dimensions and project onto the remaining dimensions by summation. This projection is demonstrated in Figure 1 schematically and with examples, both for a selection over token range in the caption (top) and a selection over a bounding-box in the image (bottom). Albeit CLIP models are typically trained to match images against captions, we can also compute intra-modal attributions for image-image text-text pairs by applying the same encoder both inputs. Appendix B discusses this in more detail. Figure 2 shows two examples.
of shape H x W x D
4 Experiments
In our experiments, we apply our feature-interaction altributions to CLIP models. We focus on evaluating the interactions between mentioned objects captions and corresponding regions in images by selecting token- ranges in captions and analyzing their interactions with image patches. In the first series of experiments, we compare our attributions against baselines (Section 4.2). The second series in Scction 4.3 then utilizes our method and analyzes CLIP models.

---

Published in Transactions on Machine Learning Research (07/ (025)
4.1 Experimental setting
We base our evaluation on three image-caption datasets that additionally contain object bounding- annotations in images, Microsoft's Common Objects in Context (Coco) (Lin et al., 2014), the FLICKR30K collection (Young et al., 2014) with entity annotations (Plummer et al., 2015), and the Hard Negative Captions (HNC) dataset by Dönmez et al. (2023). HNC generates captions from scene graphs based on templates. We use basic subject predicate object template to align with the domain of the other two datasets. We use HNC for evaluation only, on F'LICKR3KK we use the test split, and on Coco we use the validation split for our analysis as the test split does not contain
ca captions'.
We work with CLIP dual encoders (Radford et al., 2021) trained with the standard inter-modal contrastive objective and analyze the original OPENAL models, as well as METACLIP (Xu et al., 2024) and the OPEN- CLIP reimplementations trained on the LAION (Schuhmann et al., 2022), DEN (Fang et al., 2024), COMMON- POOL, and DATACOME (Gadre et al., 2023) datasets?. If not mentioned otherwise, our experiments are basod on the ViT-B-16 architecture. In addition to the unmodified models, we evaluate variants finc-tuned on the Coco and FLICKR30K training splits. We run all trainings for five epochs using AdamW (Loshchilov & Hutter, 2018), starting with initial learning rate of 1 x 10-? that exponentially increases to 1 10-5. Weight decay is set to 1 x 10-4 and the batch size is 64 on a single 50GB NVIDIA A6000.
4.2 Attribution evaluation
In the first series of experiments, compare our attributions against baselines. Figure 1 includes qualita- tive comparison of our second-order interaction attributions against first-order variants. A detailed compari- son between first-order methods has been presented by Zhao et al. (2024). We closely follow their evaluation protocol and extend it to second-order methods. Unless stated otherwise, we attribute to the second-last hidden representation in the models' image and text encoders and use = integration steps (cf. Equa- tion 9), with a black image as the image reference and a padding token sequence as the text reference. In Appendix D, we include additional experiments on the accuracy of our attributions as function of N, well as different reference choices.
Baselines. We compare our method against four baselines: Iiteraction-AAM (ICAM) (Sammani et al., 2023) is also gradient-based and can be seen as a special case of our approach as shown in Appendix G. Interaction-Lime (ILIME) is a bilinear extension of LIME for dual-encoder models (Joukovsky el al., 2023). Code is not available, therefore, we reimplement it; details are Appendix H. TrSM (Sammani et al., 2023) follows the simple approach of pair-wise multiplication of token and image patch embeddings after applying CLIP'S final projection layer to the individual embeddings. Originally, it is applied to output representations and we refer to this variant as TrSM-O. We also apply it to the same hidden representations that our hod attributes to and refer to this variant as ITSM-H. A qualitative comparison between all methods is included in Figure 11. None of the used methods including our own modifies the model architecture, its parameters, embeddings or gradients.
Input perturbation. Following Sammani al. (2023), we perform conditional perturbation experiments by ileratively removing or inserting the most attributed features in one input while keeping the other input unmodified. Figure 3 plots the decrease in similarity score for CID. Our method produces the steepest score decline as function of the number of patches removed, indicating its ability to identify the most relevant interactions. Next to CID, we also evaluate conditional image patch insertion (Cn) as well as text token deletion (CTD) and insertion (CTI). All plots are shown in Figures 16 and 17. Table provides a summary and reports the area under the curve (Auc) for the four variants. With the exception of LLMME on the text side, our method consistently results in the highest Auc values for the insertion experiments and the lowest for deletion. While [IME. performs well conditional text. attribution, interestingly, its image attributions are not competitive. We discuss this in Appendix H. Insertion and deletion experiments have been criticized for producing oft-of-domain inputs (Hooker et al., 2019). Therefore, we also construct in-domain perturbations through hard negative captions and evaluate their effects in Section 4.3.
http://wwwwmaaaaaaaaaaayyp.ooooo spplps
2CIIP family: http:///sthuihccoo/oqena////// Open family: http://gthhhb.commmmlaanadassss
7

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Ours
ICAM
ITSM-O
InsM-11
LLME
Random
1.00
LAION (tuned)
Cosine Similarity
ITSM-O ITSM-H
0.75
CLD 4
65.94
91.39
101.35
88.85
83.43
85.04
Cu
113.35
80.21
69.01
87.53
86.48
85.18
0.50
Random
ILIME
CTD
4.32
7.86
6.28
7.41
4.05
7.52
Cn f
9.26
7.77
7.82
6.72
9.89
7.47
0.25
OPENA1
CtD L
16.33
21.04
23.97
20.91
20.25
20.47
0.00
Cu f
24.43
20.47
16.24
20.60
20.58
20.47
CTD
1.06
1.35
1.15
1.30
1.07
1.31
-0.25
On f
1.06
0.95
0.99
0.92
1.04
0.96
0
25
50
125 150 175 Deleted Patches
75 100
Figure Decline of average similarity scores for iterative image patch deletions according to attributions for the LAION model fine-tuned on Coco. Uncertainty intervals are standard deviation over the evaluation split.
The AUC for CID, CIL, CTD, and CTI, on Coco for the finc-tuned LAION and the original OPENAT model. 4: lower is better, fo higher is better. Corresponding plots in Fig. 16.
Table I:
1.0
commonpool
Cumul. density
0.8
laion
datacomp
0.6
dfn
meta
0.4
0.2
A dog sitting inside the cab of a truck. (Pag=0.1, PCΛ=0)
A woman holding her
gny fording an
0.0
0.0
0.2
0.4
0.6
0.8
1.0
tennis racket. (PGk=0.4, PGA= I)
elephant over fence, (Poe=0.9, PGA=1)
Point Game Energy (PGE)
Figure 4: (Left) Examples for attributions between sclected objects in the caption (yellow) and the image logether with corresponding Coco bounding-boxes (red), PGE and PGA values as described in Section 4.2. (Right) Cumulative PGE distributions for the OPENCLIP models on Coco before (dashed) and after (solid) in-domain fine-tuning.
Object localization. Following Zhao el al. (2024), we employ the Point Game (PG) framework by Zhang et al. (2018) to evaluate how well attributions between objects mentioned within captions and images, correspond to human bounding-box annotations. In FLICKR30K, spans in captions that correspond to bounding-boxes are already annotated. In HNC, object classes exactly match sub-strings in captions and for Coco, we identify objects in captions through dictionary based synonym matching. For this experiment, we include all object annotations that correspond to single instance of its class in the image, and whose bounding-bow is larger than one patch. This results in 3.5k imago-caption pairs from Coco, 8k pairs from FlaCKR30K, and 500 pairs from HNC. Within the PG-framework, Point Game Accuracy (PGA) defines the fraction of cases for which the most attributed patch falls within the objects' bounding-box, and Point Game Energy (PGE) is the fraction of positive attributions within the bounding- box relative to the total attribution (Zhao & Chan, 2023; Wang et al., 2020). For PCE, compare both full distributions (Figure (right)) and median values (mPGE). Figure 4 shows examples from different PGE-ranges and the corresponding cunulative distributions. Very high or low values, unambiguously indicate good correspondence or clear failure cases, respectively. Intermediate values often result. from attributions exlending to contextual elements beyond actual bounding boxes, such as the tennis court in the second example.
Table 2a compares our method against the baselines. Full results are in Table 5. Our attributions outper- form the baselines by large margins. Figure 15 includes corresponding cumulative Par-distributions.
8

---

Published in Transactions on Machine Learning Research (07/2025)
Coco
FLICKR3OK
Coco
HNC
FLICKR30K
Training
Method
mPGG
PGA
mPGE
PGA
Training
Tuning
mPGR
PGA
mPGF
PGA
mPGE
PGA
ITsM-O
18.1
21.4
19.5
23.3
OPENAI
No
72.3
79.0
57.0
65.0
64.4
72.1
ITSM-H
29.8
38.1
29.5
37.4
Yes
78.0
82.9
73.4
79.0
OPENAI
ILIME
27.9
34.9
25.8
33.1
Laion
No
49.4
63.3
40.0
51.6
38.2
52.0
ICAM
38.6
54.6
33.5
51.4
Yes
71.1
83.2
54.6
61.8
Ours
72.3
79.0
64.4
72.1
(b) Results of the PG-based grounding evaluation
ITSM-O
22.8
30.3
24.5
28.7
for the OPENAI and LAION models. Tuning indicates
ITSM-H
30.5
28.8
36.6
LAION (tuned)
whether a model was fine-tuned on the respective train
ILMEE
28.8
37.8
25.8
34.5
ICAM
32.5
58.4
33.5
51.4
split of dataset. Improvements upon fine-tuning in bold.
Ours
71.1
83.2
54.3
61.8
(a) PG-based comparison of our attributions against all baselines described above.
Table 2: PGA: Point Game Accuracy, mPGE: median Point Game Energy. Extensive results for Table 2a including additional models are shown in Table 5. Full results of Table 2b can be found in Tables 3 and 4.
Based on these distributions, we test whether improvements are statistically significant using the framework of stochastic order (Dror et al. (2019); details in Appendix E). At the strict criterion of p 0.001 and c= 0.01, our method consistently results in significantly better PcE-slatistics.
4.3 Model analysis
We now turn to applying our method to gain insights into how CLIP models match images and captions.
Intrinsic grounding ability. Many of the tested models achieve good performances on the object local- ization task. On Coco, the off-the-shelf OPENAL (fine-tuned LAION) points to the correct objects in images in 79.0% (83.2%) the cases (PGA) and their high PGE values show that overall, attributions are dis- tributed to the correct image regions. Table 2b as well as Table and 4 include the results for all models and datasots. We emphasize that all models including the fine-tuned ones, have only been trained on contrastive caption-image matching. Therefore, the strong intrinsic grounding abilities that we observe here show that the coarse contrastive objective can induce fine-grained correspondence between caption parts and image regions in CLIP models. However, we also observe large differences between the original models and the fine-tuned variants, especially in the OPENCLIP models.
Out-of-domain effects. The off-the-shedd models were trained on large web-based captioning datasets but have (presumably) not been exposed to the FLICKR30K and Coco train splits. To assess domain effects of their grounding abilities, we compare the versions ine-tuned on respective train splits to the original models in Table 2b and the Appendix. Figure also plots cumulative PGE distributions for both. While the unmodified OPENAI model already demonstrates strong grounding abilities on Coco and FLICKR30K, the off-the-shelf OPENCLIP counterparts perform notably worse. Upon in-domain fne-tuning the OPENCLIP models improve by an average of 21.7+8.33 41..4+4.3) percentage points (p.p.) in median PGE and by 18.7 (9.18+4.4) p.p. in PGA on Coco (FLICKR30K). All changes are significant at p < 0.001 and c 0.01. These large improvements indicate that this fine-grained connection between captions and images, however, struggles to generalize beyond the training domain.
Class-wise evaluation. The generalization issue of the models' grounding ability becomes apparent in the examples shown in Figure 5. The off-tho-shelf model fails to identify the clock and even assigns negative altribution to the surfboard, whereas the fine-tuned version clearly identifies both. To examine the models' understanding of individual visual-linguistic concepts in more detail, we break the above analysis down to individual classes. The right side of Figure 5 shows average PGE-values and their standard deviations for Coco classes in the OPENCLIP LAION model. The classes are ordered from left right based on their average grounding ability in the unmodified model (blue). The model effectively identifies the leftmost classes,
9

---

Published in Transactions on Machine Learning Research (07 (07/2025)
unmodified
fine-tuned
1.2
unmodified model fine tured model
Grounding abiliy (PGE)
0.8
0.6
0.4
A clock the intersect of two streets.
0.0
sheep
Cappaant
mlephaan
microwaut cow
zobra
hot
truck
drann
giraff
Indtrro Sustarrouuu
owen
birg
suitt
lila
fire Saaaaann
bi
trann
so
500
stop
parking mete
kex
skateb
Loothbrg
Bacyea yay Son More Trie
Beoeggee bre
CoCo object casssss
A surfer riding wave on yellow surfboard
Figure (Teft) Class-wise average PGE before and after in-domain fine-tuning the OPEN- CLIP LAION model on Coco. Error bars are standard deviations over all class instances. (Right) Two explicit examples of how the model's grounding ability changes upon tuning. The corresponding classes are emphasized with red arrows on the left.
off-the-shetf
Froction
actua abject
2,50
girafle A desk with zebra checking each other out.
long bridge with brain going underneath.
A couple siiting on a bench
Sign of total attribution within bounding-boo
cellphone
and pocket change.
looking at the sea.
Figure 6: (Left) Examples of negative attributions for mismatches. Attributions for yellow selections in captions. Mismatching objects (underlined) receive negative attributions (blue). The histogram on the right shows the distribution over the sign of such cross-attributions..
while grounding is notably weaker for the rightmost. Upon fine-tuning (orange), most classes show clear improvements. By means of the standardized mean difference between the two PGE values, we observe the largest improvements for the classes horse, bench, giraffe, airplane and clock. This shows that contrastive finc-tuning can sharpen the visual-linguistic conception of individual object classes. In Appendix C (Figure 18), we replicate this experiment for DEN and COMMONPOOL yielding similar results.
Object Discrimination. We frequently observe that aftributions between a given object in the text and non-matching one in the image or vice versa are not only neutral but negative. Figure 6 includes four explicit examples. To systematically evaluate this effect, two distinct object classes, each appearing exactly once in the image. We then compute attributions between the two corresponding bounding- boxes and text spans and also across them, which we refer to as cross- altribution. Attribution to the actual object's bounding-box is positive in 94.1% (94.1%) of all cases, while aoossattributions to the other object are negative in 68.4% (69.7%) instances in the original (fine-tuned) model (cf. Figure (right)). This implics that CLIP models do not only match corresponding objects across the input modes but can actively penalize mismatches by assigning them negative contributions.
sample instances from Coco that include at least
10

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
COCO HNC
150
HNC
Count
Court
100
1.0
50
-0.10
-0.05
0,00
0.00
-0.00
0.00
0.01
holding
A holding cup of coffee
Score difference
Attribution difference
Slice of pizza.
Figure 7: Attribution changes in hard negative captions. (Left) Histograms for score (6s) and allribution (6Λ) differences. (Right) An example with the true caption on the left. and a hard negative caption on the right. The true object marked in yellow, and the replaced negative one in magenta.
correlation
locally exceeding
coverage
misidentiiication
umusual
comclated
loca y exceeding bounding
coverager partial visiaiity
object misidenciiiation
OpenAl
unusual scene
difficut
DFN
Someone on Burf
waiting person behind a bicycle.
hahy
guy holding up toothbrusli
0.0
Fraction of fellure
02
0.4
0.6
sink and toilet.
belly down.
spoon cupcake.
Figure 8: Examples for the five failure categories that we can identify (left) and their relative occurrence in three models (right). More examples for all categories are in included in Figure 12.
Hard negative captions. On the text side, it is straightforward to produce in-domain perturbations. We create hard negative captions that replace a single object in a positive caption with a reasonable but different object to receive negative counterpart. To this end, we leverage the automatic procedure by Dönmez al. (2023) logether with our simplified (emplate (ef. Section 4) and additionally create second resource from Coco by manually annotating small yet high-quality evaluation sample of 100 image-caption pairs. We check whether our negative captions actually result in decrease of the predicted similarity pared with their positive counterparts and define the difference as 6g. It is negative in 95.2% (89.1%) of the Coco (HNC) pairs. We then compute attributions between the token range of the original or replaced object and the object bounding-boo in the image and define the attribution difference as δ is also negative 95.2% (74.1%) of the Coco (HNc) examples. Full histograms for 8g and ĐΛ as well as an example for a change attributions is included in Figure 7. These results show that the model mostly reacts correctly to mistakes in captions and decreases the correspondence between affected image regions and caption spans.
Qualitative failure analysis. To identify cases where the models' grounding abilities are systematically weak, extract objects with PGE < 0.2 from the Coco validation set and categorize them qualitatively. For the LAION, OPENAL, and DEN models, this results in approximately 200 image-caption pairs each. We can identify five major failure categories: (1) Visually correlated scenes like bascball courts, bathrooms, offices, elc., (2) altributions locally exceeding bounding boxes, (3) coverage or partial visibility of objects, (4) actual object misidentiffcationss, and (5) difficult or unusual scenes. Figure shows the distribution among these categorics and an example for cach. More examples are included Figure 12. Category (1), correlated scenes, accounts for approximately half of all failures in all three models, indicating that CLIP models may struggle to differentiate between objects that commonly appear together.
11

---

Published in Transactions on Machine Learning Research (07/ (07/2025)
5 Discussion
Interpretation of results and future work. While prior work has already established that CLIP models can ground full text inputs onto images and vice versa (Zhao et al., 2024), our sccond-order attributions take these insights a step further and show that this lingut-ing correspondence is more fine- gne-grained connecting individual parts of captions and images.
Our evidence for this intrinsic grounding ability to be significantly reduced on data outside the initial training domain complements recent efforts towards an understanding of CLIP'S ostensible out-of-domain generaliza- tion (Xue et al., 2024). While Fang et al. (2022) and Nguyen et al. (2022) identified the training distribution as the critical component, Mayilvahanan et al. (2024; 2025) recently showed that it must be assigned to domain cont contamination of web-scale training datasets and CLIP models do nol actually generalize to unseen image domains, like renditions.
The finding that CLIP models can actively assign negative contributions to mismatches reveals non-trivial mechanism in their prediction computation. The fact that this is not consistently the case and we also observe positive cross-attributions in correlated scenes like tennis courts, bathrooms, kitchens, streets, etc., however, is yet to be understood. It suggests the contrastive objective may not provide sufficient supervision to learn to tell apart objects that frequently co-occur, Future work should establish a detailed understanding of this phenomenon. A solution may be to augment the training data with negatives targeting such correla- tions (Yuksckgomul et al., 2023; Patel et al., 2024).
Our baseline experiments show that analyzing interactions in CLIP models is not trivial. Neither simpli- fied gradient- based approaches (ICAM), pair-wise wise embedding multiplication (ITSM) nor surrogate modeling (ILImE) are sufficient for the purpose. It may still be possible to further enhance our method accounting for discrete text representations (Sanyal & Ren, 2021), incorporating non-uniform interpolation (Bhat & Raychowdhury, 2023) or inlegrating along non-linear paths (Kapishnikov et al., 2021; Zhuo & Ge, 2024).
Limitations. As stated explicitly in Equation 11, our interaction attributions are an approximation. Throughout this work, we altribule to intermediate representations of inpuls, which both efficient and informative (Möller et al., 2023). In transformers, intermediate representations have undergone multiple contextualization steps and are technically not strictly tied to input features at a given position. Finally, re- cently proven fundamental limitations of attribution methods urge caution in their interpretation especially regarding monterfactuul conclusions about feature importance (Bilodeau et al., 2024).
Despite these considerations, our consistent results on vaption-image interactions across a variety of models and datasets provide strong empirical evidence for the evolvement of fine-grained inter-modal correspondence in CLIP models through contrastive training. While we must be more careful drawing definite conclusions about specific failure cases, we argue that explainability methods like ours can be used to formulate hy- potheses about mistakes and biases in models. We cannot regard them as guaranteed robust and faithful, but they provide insights that have the potential to improve models further.
6 Conclusion
We derive general second-order attributions in dual encoder architectures, enabling the attribution of sim- ilarity predictions onto interactions between input features. Our method is applicable to any differentiable dual-encoder and requires no modifications of the initial model. We believe it can also provide valuable insights into more complex relations between images and text (Krishna et al., 2017), models for different modalities (Guzhov et al., 2022) and applications like retricval (Mueller & Macdonald, 2025; Vast et al., 2024). Our experiments with CLIP models provide strong evidence for them capturing fine-grained inter- actions between corresponding visual and linguistic concepts despite their coarse contrastive objective. At the same time, also observe pronounced out-of-domain effects. These results complement recent findings identifying limitations in the generalization capabilities of CLIP (Mayilvahanan et al., 2025). Finally, an er- ror analysis revealed that CLIP models can struggle with covered or partially visible objects, unusual scenes, and correlated contexts like kitchens, offices, or sports courts.
By enabling the analysis of interactions between caption and image features, our approach contributes to an emerging interest in understanding higher-order dependencies in CLIP models (Gandelsman et al., 2025; Joukovsky et al., 2023), reaching beyond well-understood first-order effects (Zhao et al., 2024).
12

---

Published in Transactions on Machine Learning Research (07 (07))
Acknowledgements
Funded by Deutsche Forschunggemminschaatt (DFG, German Rescarch Foundation) under Germany's Excel- lence Strategy - EXC 2075 390740016. We acknowledge the support by the Stuttgart Center for Simulation Science (SimTTec)).
References
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguisties, 2020.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikol aj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for few-shot learning. In Advances in Newral Information Processing Systems, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference compuler vision, 2015.
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. A diagnostic study of explainability techniques for text classification. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), 2020.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeicch Samek. On pixel-wise explanations for non-linear classificr decisions by layer-wise relevance propagation. PloS one, 2015.
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Effective conditioned and com- posed image retrieval combining clip-based features. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognation (CVPR), pp. 11466-214777 June 2022.
Jasmijn Bastings and Katja Filippova. The elephant in the interpretability room: Why atlention explanation when we have saliency methods? In Proceedings of the Third BlackbozNLP Workshop Analyzing and Interpreting Newral Networks for NLP, 2020.
Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio Calmon, and Himabindu Takkaraju. Interpreting clip with sparse linear concept embeddings (splice). Advances in Neural Information Processing Systems, 2025.
Ashwin Bhat and Arijit Raychowdhury. Non-uniform interpolation in integrated gradients for low-latency explainable-si. In 2023 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 5, 2023. doi: 10.1
202...22...0000 202..0118882.,, 10181829.
Blair Bilodcau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility theorems for feature attribu- tion. Proceedings of the National Academyy of Sciences, 2024.
Samuel Black, Abby Stylianou, Robert Pless, and Richard Souvenir. Visualizing paired image similarity transformer networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022.
Walid Bousselham, Angie Boggust, Sofian Chaybouti, Hendrik Strobelt, and Hilde Kuehne. Legrad: An explainability method for vision transformers via feature formation sensitivity. arXiv preprint arXiv:2404.0322 2024.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 2020.
13

---

Published in Transactions on Machine Learning Research (07/)025)
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVE /CV conference on computer vision and pallern recognilion, 2021.
Peijie Chen, Qi Li, Saad Biaz, Trung Bui, and Anh Nguyen. gscorecam: What objects is clip looking at? Tn Proceedings of the Asian Conference on Computer Vision, 2022.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Good- man, Adam Gryener, Basil Mustafa, Lueas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, We- icheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme Ruiz, Andreas Peter Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLl: A jointly-scaled multi- lingual language-image model. In The Eleventh International Conference Learning Representations, 2023a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021.
Ven-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, The Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, 2020.
Zhihong Chen, Ruifei Zhang, Yibing Song, Xiang Wan, and Guanbin Li. Advancing visual grounding with scene knowledge: Benchmark and method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), 2023b.
Tianyu Cui, Pekka Marttinen, and Samuel Kaski. Learning global pairwise interactions with bayesian neural networks. In ECAI 2020, pp. 1087 1094. IOS Press, 2020.
Samyak Datta, Karan Sikka, Anirban Roy, Karuna Ahuja, Devi Parikh, and Ajay Divakaran. 2lign2groudd: Weakly supervised phrase grounding guided by image-caption alignment. In Proceedings of the IEEE/CH /CVF international conference on computer vision, 2019.
Eustasio del Barrio, Juan A. Cuesta-Albertee and Carlos Matrán. An Optimal Transportation Approach for Assessing Almost Stochastic Order, pp. 33-44. Springer, 2018.
Esra Dönmez, Pascal Tilli, Hsiu- Yu Yang, Ngoc Thang Vu, and Carina Silberer. Hnc: Leveraging hard nega- tive captions towards models with fino-grained visual-linguistic comprchension capabilitios. In Proceedings of the 27th Conference on Compulational Natural Language Learning (CoNLL), 2023.
Michael Dorkenwald, Nimrod Barazani, Cees GM Snoek, and Yuki M Asano. Pin: Positional insert unlocks object localisation abilities in vlms. arXiv preprint 3rXiv2242.08665 2024.
Finale Doshi-Veler and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint Xrxiv: 08608, 2017.
Rotem Dror, Segev Shlomov, and Roi Reichart. Deep dominance how to properly compare deep neural models. In Proceedings of the 57th ACL, 2019.
Oliver Eberle, Jochen Büttner, Florian Kräutli, Klaus-Robert Müller, Mattoo Valleriani, and Grégoire Mon- tavon. Building and interpreting deep similarity models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.). Advances in Neural Information Processing Systems, 2023.
Alex Fang, Gabriel Nharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). In Kamalika. Chaudhuri, Stefanie Jegelka, Le Song, Csaba. Szepesvari, Gang Niu, and Sivan Sabato (eds.).
14

---

Published in Transactions on Machine Learning Research (07/2025)
Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 62166666666 6234. PMTRR, 17-23 Jul 2022. URL https://proceedingg mml..
preas/v1622/ang22a..ca. html.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference on Learning Representations, 2024.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International A ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, pp. 2288 2292, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380379. doi: 10. 011144//444445346300000 URL https://doi. org/ 10. 1111/11//444901 33635/334808833333888. 3463098.
Fabian Fumagalli, Maximilian Muschalik, Patrick Kolpaczki, Eyke Hüllermeier, and Barbara Hammer. Shap- iq: Unified approximation of any-order shapley interactions. Advances in Neural Information Processing Systems, 2024.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Gian- nis Daras, Sarah M Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Veneu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datascts. In Thirty-seennth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Yossi Gandelsman, Alexei Efros, and Jacob Steinhardt. Interpreting clip's image representation via texi- based decomposition. arXiv preprint Stf::23310009111 2023.
Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting the second-order effects of neurons in CLIP. In The Thirteenth International Conference on Learning Representations, 2025.
Xingyu Gao, Steven CH Hoi, Yongdong Zhang, Ji Wan, and Jintao Li. Soml: Sparse online metric learning with application to image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, 2014.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Retrieval-angmmeted generation for large language models: A survey. ArXio, 2023.
Loris Giulivi and Giacomo Boracchi. Concopt visualization: Explaining the clip multi-modal embedding using wordnet. In 2024 International Joint Conference on Neural Networks (IJCNN), 2024.
Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive eenguage-image pretraining, In Advances in Newral Information Processing Systems, 2022.
Michel Grabisch and Mare Roubens. An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of game theory, 1999.
Matthicu Guillaumin, Jakob Verbook, and Cordelia Schmid. Is that you? metric learning approaches for face identification. In 2009 IEEE 12th international conference on computer vision, 2009.
Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-200 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976 980. IEEE, 2022.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xic, and Ross Girshick. Momentum contrast for unsupervisod visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
15

---

Published in Transactions on Machine Learning Research (07/2025)
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In Computer Vision-F:C FCCCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 14, 2016, Proceedings, Part IV 14, 2016.
Lisa Anne Hcndricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Grounding visual explanations. Proceedings of the European conference on compuler vision (ECCV), 2018.
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In Similarity- Based Pattern Recog- nition: Third International Workshop, STMBAD 2015, Copenhagen, Denmark, October 12-14 14, 2015. Pro- ceedings 3, 2015.
Sara Hooker, Dumitru Erhan, Picter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In Adoances in Neural Informalion Processing Systems, 2019.
Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), 2019.
Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Mochine Learning Research, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Jelf Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535 547, 2019.
Boris Joukovsky, Fawaz Sammani, and Nikos Deligiannis. Model-agnostic visual explanations via approxi- mate bilinear models. In 2023 IEEE International Conference Image Processing (ICIP), 2023.
Andrei Kapishmikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Boluk- basi. Guided integrated gradients: An adaptive path method for removing noise. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5050-5058, June 2021.
Mahmut Kaya and Hasan Şakir Bilge. Deep metric learning: A survey. Symmetry, 2019.
Rémi Kazmierczak, Eloise Berthier, Goran Frehse, and Gianni Franchi. CLIP-QDA: An explainable concept bottleneck model. Transactions on Machine Learning Research, 2024.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neuraa information processing systems, 2020.
Jing Yu Koh, Ruslan Salakhutdinov, and Danicl Fried. Grounding language models to images for multimodal inputs and outputs. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32 73, 2017.
Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Uniclip: Unified framework for contrastive language-image pre-training. In Advances in Newral Information Processing Systems, 2022.
Samuel Lerman, Charles Venuto, Henry Kautz, and Chenliang Xu. Explaining local, global, and higher-order interactions in deep learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1224-1233, 1233, 2021.
16

---

Published in Transactions on Machine Learning Research (07 2025)
Martha Lewis, Nihal Nayak, Peilin Yu, Jack Merullo, Qinan Yu, Stephen Bach, and Ellic Pavlick. Does CLIP bind concepts? probing compositionality in large image models. In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguistics: EACL 2024, pp. 1487 1500, St. Julian's, 's, Malta, March 2024. Association for Computational Linguistics. URL org/2024. findings-eacl.102 101/.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: universal encoder for vision and language by cross-modal pre- pre-trrining. In Proceedings of the AAAI conference on artificial intelligence, 2020.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in NLP. Tn Proceedings of the 2016 Conference of the North American Chapter of the Association, for Compulational Linguistics: Human Language Technologies, 2016.
Jumnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joly, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Newral Information Processing Systems, 2021.
Jumnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning, 2022a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language- image pre-training. In 2022 IEFE/CV Conference on Computer Vision and Poltern Recognition (CVPR), 2022b.
Yi Li, Hualiang Wang, Yiqum Duan, Hang Xu, and Xiaomeng Li. Exploring visual interpretability for contrastive language-image pre-training. arXiv preprint or Sft:2209007044 2022c.
Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhance- ment in open- ppcabuocay tasks. arXiv preprint arXiv:2304. 05653, 2023.
Dekang Lin. An information-theorede definition of similarity. In Proceedings of the Fifteenth International Conference on Machine Learning, ICML, '98, pp. 296-2063-000 -30, San Francisco, CA, USA, 1998. Morgan Kanf- mann Publishers Inc. ISBN 1558605568.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C LLwrencc Zitnick. Microsoft coco: Common objects in context. In Computer SeioonFECCVV 2014: 19th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, 2014.
Zachary C. Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 2018
Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2018.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. UNIFIED-IO: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations, 2023.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Newral Information Processing Systems, 2017.
Joanna Materzyíska, Antonio Torralba, and David Bau. Disentangling visual and written concepts in clip.
In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Prasanna Mayilvahanan, Thaddäus Wiedemer, Evgenia Rusak, Matthias Bethge, and Wieland Brendel. Does CLIP's generalization performance mainly stem from high train-test similarity? In The Twelfth International Conference on Learning Representations, 2024. URL https:// wtw.rrvviii..et/ouuuii..... tnBaiidobu.
17

---

Published in Transactions on Machine Learning Research (07/2025)
Prasanna Mayilvahanan, Roland S. Zimmermann, Thaddäus Wiedemer, Evgenia Rusak, Attila Juhos, Matthias Bethge, and Wieland Brendel. In search of forgotten domain generalization. In The Thirleenth International Conference on Learning Representations, 2025. URL https://openreview..mt tetprrrriieeeeeetttuuuimiii Fk3eod9aaD.
Grégoire Montavon, Alexander Binder, Scbastian Lapuschkin, Wojeiech Samck, and Klaus-Robee Müller. Layer- Wise Relevance Propagalion: An Overview, pp. 193 209. Springer, 2019.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xic. SLIP: Suu--suponiiiion meets language- image pre-training. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviu, Israel, October 23-27, 2022, Proceedings, Part XXVI, 2022.
Ariane Muellerr and Craig Macdonald. Semantically proportioned ndeg for explaining colbert's learning process. In Claudia Hauff, Craig Macdonald, Dietmar Jannach, Gabriclla Kazai, Franco Maria Nardini, Fabio Pinelli, Fabrizio Silvestri, and Nicola Tonellotto (eds.), Advonces in Information Retrieval, pp. 241-356, Cham, 2025. Springer Nature Switzerland.
W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences, 2019.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal con- trastive learning with limoe: the language-image mixture of experts. In Advances in Neural Information Processing Systems, 2022.
Lucas Möller, Dmitry Nikolaev, and Sebastian Padó. An attribution method for siamese encoders. In Proceedings of EMNLP, 2023.
Lucas Möller, Dmitry Nikolaev, and Sebastian Padó. Approximate attributions for off- af-tthe-hhelf Siamese transformers. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of CLIP. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). ), Advonces in Neural Information Processing Systems, 2022. URL
https://openrreiiet het/forum?kdd-TTTarraa
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing the evidence. In 2018 IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2018.
Maitreya Patel, Naga Sai Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, et al. Tripletclip: Improving compositional reasoning of clip via synthetic vision-land languale negatives. Advances in neuraa information processing systems, 37:32731 1-37600, 2024.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia. Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- tosettnn-ee models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
Bryan A. Plummer, Mariya I. Vasileva, Vitali Petsiuk, Kate Sacnko, and David Forsyth. Why do these match? explaining the behavior of image similarity models. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI, 2020.
Shraman Pramanick, Li Jing, Sayan Nag, Jiachen Zhu, Hardik J Shah, Yann LeCun, and Rama Chellappa. VoLTA: Vision-language transformer with weakly-supervised local-feature alignment. Transactions Machine Learning Research, 2023.
Qi Qian, Lei Shang, Baigui Sun, Juhma Hu, Hao Li, and Rong Jin. Softtriple loss: Deep metric lcarning without triplet sampling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
18

---

Published in Transactions on Machine Learning Research (07/2025)
Vincent Quantmeyer, Pablo Mosteiro, and Albert Gatt. How and where does CLIP process negation? In Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Nya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Karthikeyan Natesan Ramamurthy, Amit Dhurandhar, Dennis Wei, and Zaid Bin Tariq. Analogics and feature attributions for model agnostic explanation of similarity learners. Xi preprint 2rxiv:2222.01522 2022.
Ali Rasekh, Sepehr Kazemi Ranjbar, Milad Heidari, and Wolfgang Nejdl. Ecor: Explainable clip for object rocognition, arXiv preprint or Stv:2404.12899 2024.
Nils Reimers and Iryna Gurevych. Sentence-BERT, Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IINNLP) 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "hy should i trust you?' explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pallern Recognition (CVPR), 2022.
Karsten Roth, Timo Milbich, Samarth Sinha, Pratock Gupta, Bjorn Ommer, and Joseph Paul Cohen. Revisiting training strategies and generalization performance in deep metric learning. In International Conference Machine Learning, 2020.
Arka Sadhu, Kan Chen, and Ram Nevatia. Zero-shot grounding of objects from natural language queries. In Proceedings of the IEEE/CVE International Conference on Computer Vision, 2019.
Dylan Sam, Devin Willmott, Joao D Semedo, and J Zico Kolter. Finctuming clip to reason about pairwisc differences. arXiv preprint arXiv:2409.09721, 2024.
Fawaz Sammani, Boris Joukovsky, and Nikos Deligiannis. Visualizing and understanding contrastive learning. IEEE Transactions on Image Processing, 2023.
Soumya Sanyal and Xiang Ren. Discretized integrated gradients for explaining language models. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10285 10299, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10.18653/ v1/2021.emmlp-maall emnlp-main8805. URL https: haap::////caathlogg// otg/2021. maannlppmmii..000
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaezmarezyk, and Jenia Jitsev. Laion-ōb: An open large-scale dataset for training next generation image-lext models. In Advances in Newral Information Processing Systems, 2022.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basod localization. In 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and- language tasks? In International Conference on Learning Representations, 2021.
19

---

Published in Transactions on Machine Learning Research (07 (07/2025)
K. Simonyan, A Vedaldi, and A Zisserman. Deep inside convolutional networks: visualising image classifica- tion models and saliency maps. In Proceedings of the International Conference on Learning Represenialions (ICLR). ICLR, 2014.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems, 2016.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguitiic representations. In International Conference on Learning Representations, 2019.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The shapley taylor interaction index. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Pascal Tilli and Ngoe Thang Vu. Discrete subgraph sampling for interpretable graph based visual question answering. In Proceedings of the 31st International Conference on Computational Linguistics, 2025.
Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. In International Conference on Learning Representations, 2018.
Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language-iman pre-traiming (CLIP). In T'irtt--eventth Conference on Neural Information Processing Systems, 2023. URL https://openrevier mett/forum?iddsMipMeoee
Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness contrastive language-image pre-training (clip). Advances in Neural Information Processing Systems, 2024.
Amos Tversky. Features of similarity. Psychological reviem, 84(4):327, 1977.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807. 0377., 2019.
Alexandros Vasileiou and Oliver Eberle. Explaining text similarity in transformer models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024.
Mathias Vast, Basile Van Cooten, Laure Soulier, and Benjamin Piwowarski. Which neurons matter in ir? applying integrated gradients- methods to understand cross-cncoders. In Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '24, pp. 133 143, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706813.. doi: 10. 1..145/3664190. 3672528. URL https://doi. or0/10.....1...0 1145/3664190.3672288
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang. Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In 2020 IEEE/CVP Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020.
Ying Wang, Tim GJ Rudner, and Andrew G Wilson. Visual explanations of image-text representations via multi-modal information bottleneck attribution. Advances in Neural Information Processing Systems, 2023.
Sarah Wiegrelfe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 2019.
Nicolai Wojke and Alex Bewley. Deep cosine metric learning for person re-identification. In 2018 IEEE winter conference on applications of computer vision (WACV), 2018.
20

---

Published in Transactions on Machine Learning Research (07 (225))
Xie, S. Sun, X. Xiong, Y. Zheng, D. Zhao, and J. Zhou. Ra-clip: Retrieval augmented contrastive language- image pre-training, In 2023 IEEE/CV /CF Conference on Computer Vision and Poltern Recognilion (CVPR), 2023.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Represenlations, 2021. URL zeFrigyZIn.
Hu Xu, Saining Xie, Xiaoqing Tan, Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In The Twelfth Inter- national Conference on Learning Representations, 2024.
Yihao Xue, Siddharth Joshi, Dang Nguyen, and Baharan Mirzasolciman. Understanding the robustness of molti-modal contrastive learning to distribution shift. In The Twelfth International Conference on Learning Representations, 2024. URL https: ttttps://openreviert http://ppnrreee..eeeeetoerrraaaa
J. Yang, J. Duan, Tran, Y. Xu, S. Chanda, L. Chen, B. Zeng, T. Chilimbi, and J. Huang, Vision- lasgualangu pre-training with triple contrastive learning. In 2022 IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
J. Yang, C. Li, P. Zhang, B. Xiao, C. Liu, L. Yuan, and J. Gao. Unified contrastive learning in image- text-label space. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. In Computer Vision ECCV 2022: 17th European Conference, Tel Arte, Israel, October 23-27, 2022, Proceedings, Part XXXVI, 2022c.
Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-altention network for referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual de- notations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Young, Mojtaba Seyedhosscini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022.
Mert Yuksekgomul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what do about it? In The Eleventh International Conference on Learning Represenlations, 2023. URL https:/ /ttp://eeeppeeeeeeeeemm wwt./ee//oteto /ttp
Andrew Zhai and Hao-Yu Yu Wu. Classification is a strong bascline for deep metric learning. arXiv preprint arXiv:1811.19649 2018
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Selaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 2018
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, 2022a.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, 2022b.
21

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Chenyang Zhao and Antoni B. Chan. ODAM: Gradient-bascle instance-specific visual explanations for object detection. In The Fleventh International Conference. on Learning Regresenlations, 2023.
Chenyang Zhao, Kun Wang, Xingyu Zeng, Rui Zhao, and Antoni B. Chan. Gradient-based visual explanation for transformer-based CLIP. In Forty-first International Conference on Machine Learning, 2024.
Meng Zheng, Srikrishna Karanam, Terrence Chen, Richard Radke, and Ziyan Wu. Towards visually explaining similarity models. arXiv preprint or 2Xix:2008.00033 2020.
J. Zhou, L. Dong, Z. Gan, L. Wang, and F. Wei. Non-contrastive learning meets language-image pro-training. In 2023 IBEE/CVE Conference on Compuler Vision and Pollern Recognilion (CVPR), 2023.
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey, 2024.
URL
org/ hwtggaab/23008.. 271007
Yue Zhuo and Zhiqiang Ge. Ig2: Integrated gradient on ilerative gradient path for feature attribution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(11):7173 7190, 2024. doi: 10.1109/
2PAMM.202433388002
A Additional Examples
Figure 9 shows two additional examples for inter-modal attributions, one for text-span selection and image projection and one for bounding-box selection and caption projection.
Figure 11 shows a qualitative comparison between our attributions and the baselines described in Section 4.2.
Figure 12 shows five examples for each of the five failure categories that we identified in Section 4.3 under Qualitative failure analysis.
B Intra-modal attributions
This section doscribes intra-modal model attributions for text- or imago-pairs exemplified in Figure For text-text attributions, after summation over embedding dimensions, the at altributions take the form of an S XS22 dimensional matrix, with S and S2 being token sequence lengths of the two texts. For image-image pairs, attribution tensors become four dimensional taking the shape (HxW), (H x W)2, containing contribution for every pair of two patches from either image. Figure 10 includes additional examples.
(a) A couple sitting on a (b) A couple sitting on bench looking at the sea. bench looking the
(c) A dog is jumping for (d) dog is jumping for frisbee.
frisbee.
Figure 9:
Additional examples for inter-modal attributions of token-range selection with image projections
(left) and bounding-bor box selection with caption projection (right). The visualization is identical to Figure 1.
22

---

Published in Transactions on Machine Learning Research (07 (07/2025)
PREPARED F
Figure 10: Image-image attributions between the yellow bounding-bos in the left image and the one to its right as described in Section 3. Visualisation is identical to Figure 2 (right)
Coco
HNC
FLICKR30K
Model
Tuning
mPGE
PGE>0.8
PGA
mPGE
PGE>0.8
PGA
mPGE
PGE>0.8
PGA
RN50
No
66.3
28.8
76.9
50.1
22.6
61.8
60.1
25.5
71.2
VYT-B/32
No
63.5
33.3
69.1
52.8
28.5
58.5
50.4
23.4
58.1
VIT-B/16
No
72.3
35.7
79.0
57.0
31.7
65.0
64.4
28.4
72.1
ViT-B-16
Yes
78.0
48.4
82.9
73.4
40.7
79.0
Table 3: Summary of the Point-Game evaluation for different CLIP models by OPENAI as described in Scction 4.2. Model refers to the investigated architecture, Tuning is whether the model fine-tuned the train split of the respective dalaset. Best overall results are in bold, best results of unmodified models are underlined.
C Extended results
Table 3 shows full results for our Point-Gamo evaluation on different OPENAI models. Next to the ViT- B/16 architecture, we also evaluate the RN50 and ViT-B/32 variants. Table includes the full evaluation for all OPENCLIP models, In addtion to the median PGE (mPGE), in these tables we also report cumulative PGE densitites for the 80th percentile (PGE>0.8). Full cumulative PGE- PGE-histogrmms for additional models are included in Figures 13 and 14.
Table 5 presents full results of our Point-Game baseline experiments extending Section 4.2. Correspond- ing cumulative densities of the PGE-metric are shown in Figure 15. Figures 16 and 17 show the plots of the conditional insertion and deletion experiments for the OPENCLIP LAION model and the original OPENAT model, respectively. The corresponding Auc values are contained in Table
Figure 18 extends the class-wise PGE-evaluation from Section 4.3 to the OPENCLIP DEN and DATA- CоMP models.
D Additional Experiments
A Approximation Error. In Section 3, have shown the equality between Eq. 2 and Eq. 10. The only approximation affecting this equality is the numerical integration in Eq. 9 to calculate the two integrated Jacobians J and Jhh by sum over N bins. We can evaluate how good this approximation is by explicitly calculating the four similarity predictions between the references and inputs f(a, f(r,, b), f(a, rb) and fr,,, rb), as well as the attribution matrix A. The approzimation error can then be defined as the absolute difference between Eq. 2 and Eq. 10. Figure 19, we plot this error as a function of different magnitudes for N. For larger N, it converges as expected.
Choice of references. The choice of the references ra and rs is ambiguous as long as they are uninforma- tive. We try different options and evaluate their approximation errors as defined above. For the image input.
23

---

Published in Transactions on Machine Learning Research (07/ 2025)
Ours
ICAM
ILIME
IrSMout
ITSM adden
A guyy
with hat
and frisbee
in front of & cad
by & lake
ICAM
ILME
frs
with imbrella sitting bench Figure 11: Qualitative comparison between attributions, the ICAM, ILMME and both ITSM variants. Heatmaps over images in a given column are for the marked parts of the captions in yellow below.
Street
24

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
correlated scene
A baseball player awinging bat.
living room with vase with flowers.
Au office with desk, chall and laptop.
Bathroom area, tub, toilet and slik.
Kitchen with stove, and refrigerator.
locally exceeding bounding boxes
A man riding a skateboard up ramp.
A person skies files through the air.
An airline status hoard with a clock.
A person holds pizza on poel.
Two people sitting on a small boat floating.
coverage or partial visibility
lonking picutre book.
A dog next to eat lying on bed.
man drinking
A person feeding sheep behind fence.
A man sitting in chade cuting a banana.
bottle
object misidentification
PEP
A baby with spoon
living room with coucli and rug.
parking meter decorated with house.
plate food next to
with knifé
looking at cupcake.
& cup of pepsi.
about to cut some cake.
QUIMMUN
difficult or unusual scene
A closcup of Bork holding, some broccoli.
A tan teddy bear wearing a bow tle.
Someone holding an umbrellà on sidewalk.
Cars and bus stopped for train.
A dog that is sniffing baseball bat
Figure 12: Five examples for each of the five identified failure categories as described in Section 4.3.
25

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
COCO
FLICKR30K
Training
Tuning
mPGE
PGE>0.8
PGA
mPGE
PGE>0.8
PGA
No
49.4
22.0
63.3
38.2
15.9
52.0
LAION
Yes
71.1
47.3
83.2
30.6
61.8
No
43.0
18.2
58.8
36.7
15.5
53.0
COMMONPOOL
Yes
57.7
28.7
67.1
44.6
20.8
56.2
No
38.5
56.0
11.8
48.9
DATACOMP
Yes
72.4
50.0
75.1
50.7
27.3
56.0
No
46.5
19.6
54.3
35.4
12.3
43.3
DFN
Yes
71.4
53.3
74.6
53.1
33.5
58.3
No
44.2
16.8
52.3
37.0
14.5
46.4
Meta-CLIP
Yes
57.5
49.8
77.1
49.2
24.1
57.2
Table 4: Summary of the Point-Game evaluation for OPENCLIP models on Coco and Flickr30k. The Training column refers the dataset the model was initially trained on, Tuning is whether the model was additionally fine-tuned on the train-split of the respective evaluation dataset. All models implement the ViT-B-16 architecture except Meta- CLIP that uses quickgelu activations. Best overall results are in bold, best results for unmodified models are underlined.
1.0
1.0
RN50
0.8
ViT-B/32
0.8
ViT-B/32
Cumul. density
Cumul. density
VIT-B/16
ViT-B/16
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.8
1.0
0.0
0.2
0.6
0.8
1.0
Point Game Energy (PGE)
Point Game Energy (PGE)
Cumulative PcE-distribution plots of the unmodified (dashed) fined-tuned (solid) OPE- NAI models on Coco (Ieft) and FLICKR30K (right) dataset as described in Scction 4.2.
Figure 13:
1.0
0.8
0.8
Cumul. density
Cumul density
0.6
0.6
commonpool
0.4
OpenAl RN50
0.4
laion
OpenAl VIT-B/32
datacomp
OpenAl ViT-B/16
0.2
dfn
OpenCLIP VIT-B-16
meta
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Point Game Energy (PGE)
Attribution within bounding- boxx (y)
Figure 14:
Cumulaatve PGE-distribution plots for the different models on HNC (left) and the unmodified (dashed) / fine-tuned (solid) OPENCLIP models on FLICKR30K (right) as described in Section 4.2.
we use black image and zero-centered gaussian noise. For the text input, we use sequence of padding tokens and the empty sequence consisting only of the CLS and EOS tokens. Figure 19 includes approximation
26

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Coco
Flickr30k
Training Method
mPGE
PGA
mPGE
PGA
ITSMout
18.1
21.4
19.5
23.3
ITSMMidden
29.8
38.1
29.5
37.4
OpenAT
ILIME
27.9
34.9
25.8
33.1
ICAM
38.6
54.6
33.5
51.4
Ours
72.3
79.0
64.4
72.1
ITSMout
22.8
30.3
24.5
28.7
ITSMhidden
30.5
34.6
28.8
36.6
LAION (tuned)
ILMME
28.8
37.8
25.8
34.5
ICAM
32.5
58.4
33.5
51.4
Ours
71.2
83.2
56.3
63.6
FTSMout
24.2
34.5
25.1
31.4
ITSMaidden
27.4
23.0
27.7
36.5
DEN (tuned)
ILIME
27.9
39.2
25.7
33.5
ICAM
33.3
46.5
24.2
42.2
Ours
71.4
74.6
53.1
58.3
ITSMout
25.5
38.7
26.5
33.9
ITSMaideen
35.0
42.3
22.6
28.4
DATACOMP (tuned)
ILMME
28.4
39.3
25.7
34.1
ICAM
36.9
49.5
23.2
37.3
Ours
72.4
75.1
50.7
60.0
Table 5: PGE--valuation results of our method compared against the ITSM and InteractionCAM (ICAM) baselines for different models as described in Section 4.2 under Object localization. Best results for every model are in bold.
1.0
0.8
Cumul. density
0.6
0.4
dfn
laion
0.2
datacomp
openai
0.0
0.2
0.4
0.6
Attribution within bounding-box (у)
Figure 15: Cumulative PGE-diisributions for our baseline experiment in Section 4.2. Our method is in solid, InteractionCAM is dashed and ITSMout dotted. ITSMRidden is excluded for uncluttered visualization.
errors for all four combinations of these references function of N. Combinations with gaussian noise for the image reference appear to converge slightly faster than the black image.
For different references, attributions can vary slightly. However, these differences are small, even for large objects like the cow in Figure 20 (left), for which attributions tend to spread out. The absolute difference of attributions between any two combinations of references on the order of 10-5 and tends to decrease for larger N. The plot in Figure 20 (right) shows this for the difference between black- back--pading and gaussian- emply reference combination.
27

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
1.00
1.00
Cosine Similarity
ITSM-O
Cosine Similarity
0.75
ITSM-H
0.75
ILIME
0.50
Random
0.50
0.25
0.25
ITSM-0
0.00
0.00
Random
25
75
100
150 175
10
Inserted Patches
Inserted Tokens
(a) Conditional Image Insertion.
(b) Conditional Text Insertion.
Ours
1.00
ICAM
1.00
Cosine Similarity
ITSM-O
Cosine Similarity
0.75
ITSM-H
0.75
ILIME
0.50
Random
0.50
Ours
ICAM
0.25
0.25
ITSM-0
0.00
0.00
ILIME
Random
-0.5
-0.25
0
25
50
150 Deleted Patches
75
100
125
175
2
6
8
10
Deleted Tokens
(c) Conditional Image Deletion.
(d) Conditional Text Deletion.
Figure 16: Average change in similarity score upon conditional insertion and deletion performed on either the caption or the image using a ViT-B-16 model pretrained on LAION Confidence intervals are standard deviations over the evaluation dataset. Table summarizes the AUC of these plots.
E Stochastic Dominance
Stochastic dominance defines an order relation between probability distributions based their cunulatives. del Barrio et al. (2018) have proposed significance test building the principle and Dror et al. (2019) have identified it being particularly suitable to compare deep neural models. The test's e-parameter is the maximal percentile range where the inferior distribution is allowed to dominate the superior one and Dror el al. suggest to set it to e < 0.4. The smaller the stricter the criterion. a is the significance level.
F Integrated Gradients
We derive IG for model ((a) = a with vector-valued input a and a scalar prediction 8, e.g. classification score. We define the reference input r, begin from the difference between the two predictions and reformulate it integral over the integration variable
S(a) - 5(r) ((r)= 1.h 0s(x)
0x; dx;
(12)
Again we do not write out sums over double indices. To solve the resulting line integral, we substitute with the straight line x(a) = r + a(a - r) and pull its derivative ox(a)/da = (a (a-r) r) out of the integral:
L Of(x(a)) @x,(a) da = (a-r); /' v.f(x(a))
(13)
0xi(a) da
28

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Cosine Similarity
0.15
Cosine Similarity
0.10
0.10
0.05
0.05
ICAM
ITSM-O
ITSM-O
0.00
ITSM-H
ILME
0.00
-0.05
Random
Random
25
75
100 125
150 175
10
Inserted Patches
Inserted Tokens
(a) Conditional Image Insertion.
(b) Conditional Text Insertion.
0.15
0.15
Cosine Similarity
Cosine Similarity
0.10
0.10
Ours
0.05
ICAM
ICAM
ITSM-O
0.05
ITSM-O
0.00
ITSM-H
ITSM-H
-0.5
ILIME
Random
0.00
Random
ILIME
25 50
100 125 150 175
2
6
8
10
Deleted Patches
Deleted Tokens
(c) Conditional Image Deletion.
(d) Conditional Text Deletion.
Figure 17: Conditional insertion and deletion performed either the caption or the image using the original VIT-B/16 model by OPENAl without fine-tuning.
In practice, we approximate the integral by (i)) x 0, the equality between Eq. 12 and Eq. 13 can be reduced to the final approximation of IG:
N steps. If the reference is uninformative, that
N r);
f(a) = (a (a-r))
v.f(x((n)),
(14)
N
which decomposes the model prediction f(a) into contributions of individual feature in a.
G Relation to interactionCAM
Here, we first discuss the relation of integrated gradients and GRADCAM and then show how our second- order method can be reduced the ICAM bascline.
We start from the right-hand-side of Equation 14, the final form of IG. can reduce these this result further by setting N = 1 and using the zero vector as reference, (r=0. These simplifications yield,
a, (a).
(15)
which often referred as gradientx input and the basic form of GradCam. The method typically attributes to deep image representations in CNNs, so that a has the dimensions Cx x W, the number of channels, height and width of the representation. To reduce attributions to two-dimensional map, it sums
29

---

Published in Transactions on Machine Learning Research (07/202) 2025)
unmodified model fine-tuned
1.0
0.8
0.6
Grounding
0.4
0.2
0.0
sheep cow
giraffo
teddy t beat microwatth sandad with
bowi
truck
elephara
heat
motors
oven
caks
chair
cal
train
remate
horse
airplene
bicyele
scases
rollet
scisccrs
bira
lapoop
person
meter
hydrant
clack
radiet
orange
keyborg beaar
scop
parking
tennis
fire
CoCo object classes
unmodifled model
1.0
fine-tuned model
0.8
Grounding ability
0.6
0.4
0.2
sheep
bow
banens
zebra
sandwichh ch
micro aarowaae vve
tediy bear-
bear
eloph hantt
bed
truck
bear
car
refri eefrgerabol
pizze
grainn
imborala
scis sors
bus
matorcycie
chair
laptop
orange
clock
remose
sultccss
hot dog
couch
let
dog
hydrant
herse
tv
bench
bird
racket
bottle
De
keeyboard
cat
book
bicycle-
vass
sase
kile
sppon
beak
dell phone-
surfboand
fork
knire
skis
frisboe
gink-
snawbbrrd rend toothorish
stop
baseball
parkingg
tennis
fire
CoCo object clesses
Figure 18: Class-wise PCE-evaluation for the OPENCLIP LAION (top) and DATACOMP (bottom) models
before and after in-domain moe-tunin as discussed in Section 4.2.
over the channel dimension and applics relu-activation to the outcome. The original version also average pools the gradients over the spatial dimensions, however, this is technically not necessary.
As discussed earlier, neither integrated gradients nor GRADCAM can explain interaction in dual encoder predictions. Following the logic from above we can, however, reduce our second-order attributions from Eq. 10 by setting N = 1 in the computation of the integrated Jacobians in Fq. and using ra = rg = 0. For our attribution matrix from Equation 10 we then receive the simplified version
0g Ohe a, Da, Jb
by.
(16)
This simplification could be termed Jacobiansx inputs and equivalent to the ICAM by Sammani et al. (2023). Note, however, that setting N the worst possible approxination to the integrated Jacobians. Therefore, it is not surprising that empirically this version performs worse than our full attributions.
30

---

Published in Transactions on Machine Learning Research (07 (07/2025)
gaussian-empty
black-empty
0.15
gaussian-padding
Approximation
black-padding
0.10
0.05
0.00
100
10
102
10a
N
Figure 19: Approximation errors for different reference choices function of the number of integration steps N. The image references are abbreviated 'gaussian' for gaussian noise and 'black' for the black image. Text references are 'padding' and 'empty' for a padding sequence and the empty sequence, respectively, Exemplatory standard deviations over the evaluation sample are shown as shades of the respective plots.
black-padding
black-empty
gaussian-padding
kaussian-empty
5
Altribution difference
4
Bird sitting atop gattle on snow covered field.
10a
10'
10'
10'
N
Figure 20: (Left) Attribution differences for all four combinations of references as indicated above the images for the yellow selection in the caption below. (Right) The mean absolute attribution difference between a black padding and goussian-emply reference combination as a function of the number of approximation steps N and its standard deviation over the evaluation smaple (blue shade).
H Interaction LIME
We reimplement the ILMME method proposed by Joukovsky et al. (2023) that extends the principle LIME Ribeiro al. (2016) to dual encoder models with two inputs.
The core idea of LIME is to locally approximate the actual model f around a given input with interpretable surrogate model sp. The local neighborhood of the input is approximated by a sample of perturbations. The surrogate model is typically linear and operates on latent representations of the input. Further, there needs to be a mapping from latent representations to input representations, so that we can generate corresponding inputs that the actual model can process.
In the image domain latent representations za are typically binary variables indicating the presence or absence of super pixels in the input. To enable a direct comparison our method and the other baselines, use the vision transformer's patches super pixels. Analogously, in the text input we define latent representations z as binary variables indicating the presence of input tokens. Disabled image patches are replaced with the mean over the image, disabled tokes are replaced with the padding token.
The local neighborhood of a given input pair (a, b) is approximated by sampling N such latent representations (zf,, 2,) from two Bernoulli distributions. For the corresponding input perturbations (a,, by), we then compute the CLIP scores s; f(a,, b,) and fit the surrogate model reproduce these predictions.
To account for interactions between the two inputs dual encoder models, Joukovsky (2023) propose to use bilinear form surrogate model:
31

---

Published in Transactions on Machine Learning Research (07 2025)
(2"" x." Wzz"
(17)
with a weight matrix W and a scalar bias which is then optimized according to the following MSE
objective:
N taa a, b, by) (f(a,:b)) - 2( w.:
min W.c
(18)
Here, IT a function that weights individual neighborhood samples (a,, b;) according to their similarity the original input (a, b). We use the cosine similarities between perturbed and original captions and image inputs, respectively, and following Joukovsky et al. (2023), define the total similarity weight the average of the caption and image similarity:
1 T a, ] (a) (a) +h (b)(b)((b.)) (b))))))))
(19)
To fit sp, we use stochastic gradient descent with learning rate of 10-2 and weight-decay of 10-3 over N 1000 samples with Bernoulli drop-out probabilities of p = 0.3 for both caption and image representations. These parameters closely align with Joukovsky et al. (2023). Additionally, representations z and 2 with the square root of the numbers of tokens VS and image patches vH x W, respectively, helps to stabilize convergence.
find that scaling the latent
Finally, the fitted weight matrix W models interactions between image patches and caption tokens. There- fore, we can evaluate and visualize in the same way as our attribution matrices A.
In Section 4.2 we found that IIME performs well and even slightly better than our method on conditional caption attribution. At the same time its conditional image attributions are not competitive. Consequently, its grounding ability as evaluated by the PG-metrics is also weak (ef. Table 2a).
We believe the reason for this imbalance of attribution quality may be due to the different magnitudes in the number of caption tokens and image patches. While captions typically have ~ 10 tokens, image representations in ViT-B-32 architectures consist of ~ 200 patches. Therefore, the ratio of the number of samples N and tokens much better than for image patches and the surrogate model yp might. be able estimate their importances better.
Overall, we find that the optimization of LLMME is quite sensitive to dyper-parameter choices and requires extensive tuning to find a sctting that leads to stable convergence. In contrast, our method does not require additional optimization and involves no dyper-parametedd except the number of integration steps N, whose increase must, however, improve attributions due to Equation 9.
Implementation Details
For the implement of our method, we make use of the auto-differentiation framework in the PyTorch package. For a give input x(an), g(x(a,)) is the forward pass through the encoder g, and the Jacobian (x(n))) is the corresponding backward pass. For an efficient computation of all N interpolation steps Eq. 9, we can batch forward and backward passes since individual interpolations are independent of another. In practice, we altribute to intermediate representations, thus, the interpolations in Eq. 6 are between latent representations of the references and inputs. We use PyTorrc hooks to compute these interpolations during the forward pass. Algorithm 1 sketches Porocch-like pscudo-code of the implementation.
The application of our method to a different model or archilecture only requires the implementation of a single forward hook. Registering hooks into models is a standard feature auto-differentiation frameworks and does not require any modification of the given model's original code. The remaining steps generate our attributions are differentiation through standard backpropagation and, finally, simple matrix multiplication to compute Eq. 10.
32

---

Published in Transactions on Machine Learning Research (07 (07/2025)
J Computational Complexity
Since the computation of the interpolated inputs x(o,,) can be performed in parallel, N is constant with regard to time complexity. To build the full Jacobians of the encoders, however, we need to compute a separate backward pass for each output dimension, because auto-Lifferentiation can only compute backward passes for scalar-valued outputs. Time complexity dominated by this aspect and is thus on the order of O(D), with being the embodding dimensionality of the output. The intergrated method in Algorithm 1 sketches this computation.
Due to the fact that we typically attribute to intermediate representations, however, do not need to compute full backward passes. Backpropagation can be stopped once it reaches the representation we attribute to, which results in this operation to be cheaper the deeper the representation of interest is. Le. attributing to layer eleven is cheaper than attributing to layer five.
After building the Jacobians, the final attributions are computed through the matrix multiplications in Eq. 10, which outlined at the bottom of Algorithm 1. Its computation time is negligible when calculated on GPU, but can substantially add the total time when performed on CPU.
Space-wise, our method requires storing two Jacobians with the dimensions Dx and D (D)HxW), since input/intermedie /ntermedaaae representations are still sequential (S) on the text side and patch-based (H x W) on the image side. Thus, memory consumption scales quadratically on the order of O(D?), and we require large VRAM to handle the computation efficiently on GPU.
In contrast, first-order methods only require backpropagation of single scalar output value, i.e. the simi- larity score, whose result is gradient vector as opposed a Jacobian matrix. Hence, the cost of obtaining sccond-order interaction attribution is the computation and handling of these Jacobians, which is sub- stantially more expensive but enables a different level of insight into models that is not accessible through first-order methods.
33

---

Published in Transactions on Machine Learning Research (07/2025)
Algorithm PyTorch-like pseudocode sketching the computation of our attributions. The syntax is sim- plified and not consistent. For fully functional implementation, please refer to GitHub repository. Comments in the pseudocode refer to the corresponding equations in Section 3.
from torch import Tensor, arange, stack, autograd import ExplainableCLIP, mmageppreparatin, tokenize
model
ExplainablecLIIP....
image_ing input image_ paagee_rrpraation _load_image image ("path/to/Lmage (mg.imaagg)g)))))) caption_input tokenize some caption describing the image")
# Equations 6 and 7
def interpoloatex:: Tensor, ref: Tensor, n_steps: int):
Compute n_ steps linear interpolations between a reference ref and input æ. ..1
step a_steps
alphas arange 0, step) interpolation coefficients
x <_interp ref + alphas * (x ref) # interpolated representations
return x interrp
# Equation
def jategrated_jacobbaanniinet
Tensor, intermediate: Tensor):
Compute the integrated Jacobian for embedding w.r.t. intermediate representation.
gradients = []
for dim in range (embedding. size s0))
grad_d d autograd, (rabeddeedi intermediate) gradients. aopend(grad...
jacobians stack (grcdegrai))tts)
Integration over interpolations stacked along the first dimensiion int int_jacobian jacobians. (dim=00) return int_jacobian
# p place hooks in the model to compute interpolations (not actual hook syntax) image_hoo hook model register _ookk_hoooooo hww.:hooooknoto (interpolate, srage_layer, image_ reference, n_steps) caption_hood hook model register Soogistterhhoo((ti wwwpr/hoooooineeee (interpollte, text_layer,
n_ n_steps)
# Compute embeddings and retrieve intermediate representations from hooks image_eenl eeeeeddebednnggg = model. aacode_image (image input)
caption embedding = model. encode_ encode_caption image inter, image_ref_inter inter caption (caption (naptoonnntt))))))n input))
hook. get get_
peption_inntr, caption_ref inter caption_hood socloonhoook.o get www.rmedddaeeeeeeeeeetttttttttta
# Equation 10
image_jacobian interaate__jccoiiinnmeeeeeeeeeee
image_inter) inter)
caption_jacobiar
hagproociiaccppttooooeeeeeggggge embedding, caption_inds inte_inter
# Matriz multiplication
JJ cawttionjacoobiieecccaaa jacobian. @ image_jacoblan
image_delle delta image inter image_ree inter poption_inter delta caption inter caption_res ref _itter # Element-wise multiplication with broadcast ing attributions caption_delta JJ * delta
caption_delta
34


Reviews:
[{"review_id": "cv5qK4JuAg", "reviewer": "Reviewer_6F2A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The method's derivation is clear and explained straightforwardly and the resulting cross-modal attributions are intuitive\n- The methods is validated quite extensively, and the authors provide a large number of experiments to show the method's performance in terms of faithfulness of the provided attributions compared to the main alternatives\n- The method is computationally parsimonious, making for a practical attribution algorithm\n- The method is used to provide convincing insights on the effects of out-of-domain deployment vs fine-tuning\n\n### Weaknesses\n- Relation to Integrated Gradients is not sufficiently discussed\n- The computational complexity of the method scales with H x W x S x d (the encoder output dimension) x N (the integration steps) x the cost of computing a gradient through the encoder. This might be efficient, as the paper indicates, but it might still be considerable, especially for large images and long captions. The paper might benefit from a more detailed discussion on the computational cost of running the argument.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new attribution method develop to explain the functioning of dual-encoder models like CLIP by computing pair-wise attributions between cross-modal features (like text and images). The method is derived as a second-order method, but the factorized formulation of dual-encoders allows the authors to obtain an expression that is relatively simple and computationally manageable, and results formally very similar to the Integrated Gradients method (in practice, the expression of the final cross-modal attributions is essentially the product of Integrated Gradients).\n\nIn the case of CLIP, the final attribution map is a 3D tensor of size caption sequence length x image height x image width, which sliced in the text sequence length dimensions provides a 2D attribution map indicating the pixel contributions to explaining the text tokens in the slice, and sliced in the image height and width dimensions provides a 1D attribution map indicating the text token contributions.\nThe method is extensively empirically evaluated by applying it on OpenAI's CLIP model, as well as the open models MetaCLIP and OpenCLIP, and compared against a host of alternative baseline explainability methods on multiple experiments to quantify faithfulness of the proposed method through patch deletion, correspondence of the localization of the attribution maps with human bounding-box annotation, and testing the effect of hard negative captions.\nIn most experiments, the proposed methods outperforms the baselines.\n\nFinally, the paper showcases the use of the method to diagnose the functioning of CLIP-like models, by evaluating the effect of fine-tuning to mitigate out-of-domain shifts, and using it to uncover and classify failure modes which presumably diagnose issues in how the CLIP model is trained.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Since the feature-pair interaction sum to f(a,b), the total attributions for one modality (obtained by marginalizing pairwise interactions over the other) should sum to 0 (under the assumption of uninformative references used in the Methods section). Could you discuss the implications of this? For instance, it seems that for each group of positive pixels, there will have to be a corresponding group of negative pixels, which seems like a constraint of sort, since one could imagine an image consisting exclusively of positive pixels and an uninformative background. However, the proposed method does not contemplate this scenario. Could you please elaborate on this? The paragraph 'Object Discrimination' notices this phenomenon, but discusses it as seemingly accidental and diagnostic of the model's functioning, while in fact it presumably should be discussed as an inherent property of the attribution method. If this is indeed the case, then even some of the main conclusions of the paper should be rephrased to reflect the fact that for instance the uncovered penalization of mismatches by assigning negative attributions discussed in the Discussion and Conclusion sections might not indeed necessarily be a property of CLIP models, but a phenomenon of the attribution method.\n* Provide a discussion on the computational complexity of the method, and consider adding pseudocode to show in practice how attributions are computed.\n* Discuss the choice of reference inputs and the choice in integrating along a linear path between references and inputs (eq. 6 and 7).\n* Explain what uncertainty intervals mean, e.g. in Fig. 3.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XAxe9zWcyv", "reviewer": "Reviewer_zVe1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nA derivation of a new second-order feature attribution approach that is principally grounded (although remaining an approximation). \n\nThe authors apply this technique to highlight existing failure modes of CLIP models.\n\n## Weaknesses\n\nThere’s somewhat of a lack of how these empirical findings can be leveraged to improve dual encoder architectures (and their training) and their downstream usage.\n\nThere’s no real discussion on the impact of this approximation — how good / bad is this approximation in practice?\n\nLack of discussion with other works that seem to make quite similar findings of issues in CLIP models [1, 2]\n\n\n\n[1] Lewis, et. al. Does CLIP Bind Concepts? Probing Compositionality in Large Image Models\n\n[2] Sam, et. al. Finetuning CLIP to Reason about Pairwise Differences.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on performing feature attribution on dual-encoder based models to improve their overall explainability. The authors address this by deriving a decomposition of a model prediction into additive contributions from features for each individual modality (Eq 11). When averaging over feature dimensions, this produces a 3D attribution tensor. The authors compare this approach against other feature attribution baselines that only look at first-order interactions.\n\nThe authors provide evaluations for CLIP models on a variety of different tasks, including attribution evaluation and model analysis. Their attribution evaluation setting is comprised of multiple tasks: (1) perturbing most important features and evaluating the resulting change in similarity scores; (2) visualization object localizations of texts mentioned in the caption – both qualitatively and quantitatively through the Point Game framework. Their model analysis evaluation is comprised of measuring OOD generalization, class-wise grounding, and object localization.\n\nTheir empirical results show that (1) CLIP models show more fine-grained interactions between the inputs of two different modalities and (2) CLIP models struggle to differentiate between objects appearing together and performs poorly out of distribution. The authors also show that (2) can be addressed through domain-specific finetuning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some critical points to address:\n* The authors should explore more about the approximate error in the attribution method, specifically discussing how the choice of reference inputs and the number of integration steps affect the accuracy of attributions\n* The authors should include a discussion on the computational overhead of the method compared to first-order attribution techniques, as this is a natural concern when increasing the complexity of measured feature interactions\n\nRecommended change to strengthen the work:\n* One of my main concerns is the practicality or pragmatic uses of this evaluation. Wat are some of the use cases of these new insights that have been gained? How do they inform how CLIP models should be trained?\n* Added discussion of related works mentioned above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TxPpwk5QDE", "reviewer": "Reviewer_gQKD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- To the best of my knowledge the paper tackles an underexplored research domain\n- Method design is principled (although difficult to read). The inter- and intra-model attribution method is flexible and effective\n- Strong empirical results on different tasks.\n- Convincing example of using the method to detect model failures (i.e., out-of-domain effects and subsequent qualitative failure analysis)\n\n**Weaknesses:**\n- I really like Figure 1. However, it is a bit confusing and can be slightly improved. It would be helpful to notate the individual images with a), b), etc. to make it easier to point the reader to the specific image in the caption. \"Left\" is ambiguous for two rows for example.\n- The method section lacks intuition and is really difficult to read. No single step is motivated, which makes it hard to understand why certain things are introduced in the first place. Here, some high-level intuition would make the paper easier to read. The end of the first paragraph contains some intuition. The authors could try to state the goal of their derivation at the beginning of the section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors note a research gap in attribution methods that are designed for non-symmetric dual-encoder models, such as CLIP. \nThey propose a novel second-order attribution method that can analyze interactions between modalities in dual encoders. Notably, their approach does not require any model fine-tuning or test-time optimization. They evaluate their method using CLIP on common benchmarks, ablations using input perturbations, and object localization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Implement the proposed changes to Figure 1 or other ideas to make the Figure more readable \n- Add high-level intuition to the method section to motivate the derivation (making the paper easier to follow)\n\nI want to note that I am not well familiar with the related work in this area.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)