Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution
Authors: Ying Wang
Tim G. J. Rudner
Andrew Gordon Wilson
Venue: NeurIPS 2023 poster (2023)
Keywords: Interpretability
Attribution Maps
Information Bottleneck
Multi-Modal Learning
Vision-Language Pretrained Models
TL;DR: This paper presents an attribution method for identifying critical features of image-text representations in vision-language pretrained models using the information bottleneck principle.

Abstract: Vision-language pretrained models have seen remarkable success, but their application to safety-critical settings is limited by their lack of interpretability. To improve the interpretability of vision-language models such as CLIP, we propose a multi-modal information bottleneck (M2IB) approach that learns latent representations that compress irrelevant information while preserving relevant visual and textual features. We demonstrate how M2IB can be applied to attribution analysis of vision-language pretrained models, increasing attribution accuracy and improving the interpretability of such models when applied to safety-critical domains such as healthcare. Crucially, unlike commonly used unimodal attribution methods, M2IB does not require ground truth labels, making it possible to audit representations of vision-language pretrained models when multiple modalities but no ground-truth data is available. Using CLIP as an example, we demonstrate the effectiveness of M2IB attribution and show that it outperforms gradient-based, perturbation-based, and attention-based attribution methods both qualitatively and quantitatively.

Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution
Ying Wang* New York University
Tim G. J. Rudner*
Andrew Gordon Wilson New York University
New York University
Abstract
Vision-language lanou-la pretrained models have seen remarkable success, but their applica- tion to safety-crition settings is limited by their lack of interpretability. To improve the interpretability vision-language models such as CLIP, we propose multi- modal information bottleneck (M2IB) approach that learns latent representations that compress irrelevant information while preserving relevant visual and textual features. We demonstrate how M2IB can be applied to attribution analysis of vision-language pretrained models, increasing attribution accuracy and improving the interpretability of such models when applied to safety-critical domains such healthcare. Crucially, unlike commonly used unimodal attribution methods, M2IB does not require ground truth labels, making possible to audit representations of vision- langugge pretrained models when multiple modalities but no ground-tre -trut data is available. Using CLIP M2IB attribution and show that it outperforms gradient- graddeet-bbsed, perturbation-bast and attention-baa based attribution methods both qualitatively and quantitatively.
example, demonstrate the effectiveness of
Introduction
Vision-Language Pretrained Models (VLPMs) have become the de facto standard for solving a broad range of vision-language problems [15]. They are pre-trained on large-scale multimodal data to learn complex associations between images and text and then fine-tuned on given downstream task. For example, the widely used CLIP model [21], which uses image and text encoders trained on 400 million image-tex pairs, has demonstrated remarkable performance when used for challenging vision-language tasks, such as Visual Question Answering and Visual Entailment [27].
VLPMs are highly overparameterizee black- box models, enabling them to represent complex relation- ships in data. For example, Vision Transformers [ViTs; 7] state-of-the-ar ransformer-based vision model and are used as the image encoder of CLIP [21], containing 12 layers and 86 million pa- rameters for ViT-Base and 24 layers and 307 million parameters for ViT-Large. Unfortunately, VLPMs like CLIP are difficult to interpret. However, model interpretability is essential in safety-critical real-world applications where VLPMs could be applied successfully, such clinical pocision-makinn or image captioning for the visually impaired. Improved interpretability and understanding of VLPMs would help us identify errors and unintended biases VLP and improve the safety, reliability, and trustworthiness of VLPMs, thereby allowing us to deploy them in such safety-critical settings.
To tackle this lack of transparency in deep neural networks, attribution methods, which aim to explain a model's predictions by attributing contribution scores to each input feature, have been proposed for post-hoc interpretability. For example, for vision models, attribution methods be used create heatmaps that highlight features that are most responsible for a model's prediction. Similarly, for language models, scores are assigned to each input token. While the requirements for interpretability vary depending on the task, dataset, and model architecture, accurate and reliable attribution is an important tool for making emerging and stale-of-the-art models more trustworthy.
"Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Original
GradCAM
Saliency
KemeiSHAP
RISE
Chefer et al.
Ours
hungry mouse mouse trap
hungry mouse mouse trap
hungry mouse
mouse STa
hungry mouse 1190
mousS
TUnnTy mouse
MTL-5
TungTy mouse
PrungTy mouse MID
IOGUEE
IIl defined op
dofined 68 hies largety
def neddned
defined
defined mes Iargely upper
II
defined
oc lnrelly
oc kies
Inrgety
nst rile
aareey upper
Itles
largety upper
ac itees
Iargely
consetidation
consolidation
consoridanion
consolidation the left
consoliaation in
consonaation
consondatton
the left mid lung
the left mid wna
the left mid Kina
M00 Nung
the left mid lung
the In mid unn
the loft mid AUN
image relative importance
fext relative importance
Figure I: Example Attribution Maps For Image and Text Inputs. The red rectangles second and third rows show the ground-truth bounding boxes associated with the text, provided in the MS-CXR dataset [3]. Multi-modal information bottleneck (M2IB) attribution maps successfully identify relevant objects in the given image-text pairs, while other methods provide less precise localization and neglect critical features in the inputs.
While existing attribution methods focus primarily on unimodal models, we propose attribution method for VLPMs that allows us to identify critical features in image and text inputs using the information bottleneck principle [31]. Unlike standard unimodal attribution methods, the proposed multi-modal information bottleneck (M2IB) attribution method does not require access to ground- truth labels.
To do this, we formulate a simple multi-modal information bottleneck principle, use a variational approximation to derive tractable optimization objective from this principle, and optimize this objective with respect to a set of attribution parameters. Using these parameters, we are able to "turn off" all irrelevant features and only keep important ones and generate attribution maps. In contrast unimodal information bottleneck attribution [24], we aim to find attribution parameters that maximize the likelihood of observing features of one modality given features associated with the respective other modality. We perform qualitative and quantitative empirical evaluation and find that M21B is able successfully identify important features relevant both image and text inputs. We provide an illustrative set of attribution map examples in Figure 1.
Our key contributions summarized as follows:
1. We propose multi-modal information bottleneck principle and use it to develop multi-modal information bottleneck attribution method to improve the interpretability Vision-Langure Pretrained Models.
2. We perform extensive empirical evaluation and demonstrate several datasets incluuiing healthcare data that can be used in safety-critical settings -tht that multi-moda, information bot- tleneck attribution significantly outperforms existing maddeent-based perturbation-based. and attention- based attribution methods, quantitatively and qualitatively.
The code for our experiments is available at:

---

2
Related Work
Attribution methods enable post-hoc interpretability of trained models by allocating significance scores to the imput features, such as image pixels or textual tokens.
2.1 Attribution methods
Gradient-Based Attribution. The use of gradients as a foundational component in attribution methods has been widely explored in the interpretability literature. Simonyan et al. [28] directly use the gradient of the target with respect to inputs as the attribution score. Building on this, Grad- CAM -rredhent-weiighed Class Activation Mapping; 25J weights convolutional layer activations by average pixel-wise gradients to identify important regions in image inputs. Integrated Gradients [30] introduces two essential axioms for attribution methods- sensitivity and implementation invariance, which motivates the application of integrated gradients along paths from baseline input to the instance under analysis.
Berterbation-Basee Attribution. Perturbation approaches involve modifying input features to observe the resulting changes in model predictions. Unlike gradient-based techniques, they eliminate the need for backpropagation, allowing the model to be treated as a complete black box. However, such methods can be computationally intensive, especially for intricate model architectures, given the need to re-evaluate predictions under many perturbation conditions. The LIME [Local Interpretable Model- agnostic Explanations: 22] algorithm leverages perturbation in local surrogate model. Lundberg and Lee [17] propose SHAP (SHapley Additive exPlanations), which employs game-theoreti, approach to attribution by utilizing perturbations to compute Shapley values for each feature. Furthermore, they propose a kernel- based estimation approach for Shapley values, called KernelSHAP, drawing inspiration from local surrogate models.
Attention-based Attribution. The rise of transformers across multiple domains in machine learning has necessitated the development of specialized attribution methods tailored to these models. Using attention as an attribution method offers a straightforward approach to deciphering and illustrating the significance of various inputs transformer's decision-making process. Nevertheless, solely relying on attention is inadequate as it overlooks crucial information from the value matrices and other network layers [5]. To address this issue, Chefer al. [6] propose a method that learns relevancy maps through a forward pass across the attention layers with contributions from each layer cumulatively forming the aggregated relevance matrices.
Attribution. Schulz et al. [24] use information bottleneck attribution (IBA), where an information bottleneck is inserted into a layer of trained neural network distill the essential features for prediction. IBA is a model-agnostic method and shows impressive results on vision models including 16G-16 [14] and ResNet-50 [10]. Subsequently, IBA has been applied to language transformers [12] and was shown to also outperform other methods on this task. However, IBA has thus far been focused only one modality and has only been adopted in supervised learning. To the best of our knowledge, there is no previous work on applying the information bottleneck principle to multi-modee models like VLPMs.
Information-Theolr
2.2 Evaluation of Attribution Methods
Despite active research on attribution methods, we still lack standardized evaluation metrics for attribution due to the task's 's inherent complexity. For vision models, attribution maps are frequently juxtaposed with ground-truth bounding boxes for a form of zero-shot detection. However, the attribution of high scores to irrelevant areas might arise from either subpar attribution techniques or flawed models, making it challenging to isolate the root cause of such discrepancies.
To tackle this issue, previous studies have resorted to degradation-based metrics [4. 33J. The underly- ing principle is that eliminating features with high attribution scores should diminish performance, whereas the removal of low-attributed features, often seen as noise, should potentially enhance performance. Additionally, Hooker et al. [11] introduce ROAR (Remove and Retrain), methodology that deliberately degrades training and validation datasets in alignment with the attribution map, thereby accounting for potential distribution shifts. Should the attribution be precise, retraining the model using these altered datasets would result in pronounced performance decline. Rong et al. (23) argue that mere image masking can cause a leak of information via the mask's 's shape, and thus propose a Noisy Linear Imputation strategy that replaces pixels with the average of their neighbors.
3

---

It is important to note that a visually appealing saliency map does not guarantee the efficacy of the underlying attribution method. For example, an edge detector might generate a seemingly plausible saliency map, yet it does not qualify as attribution method because it is independent of the model under analysis. Adehayo et al. [1] propose a sanity check designed to assess whether the outputs of attribution methods genuinely reflect the properties of the specific model under evaluation. Notably, several prominent models, including Guided Backprop [29] and its variants appear insensitive to model weights and consequently do not pass the sanity check.
3 Attribution via a Multi-Modal Information Bottleneck Principle
In this section, introduce a simple, multi-modal variant of the information bottleneck principle and explain how to adapt it to feature attribution.
3.1 The Information Bottleneck Principle
The information bottleneck principle [32] provides a framework for finding compressed represen- tations of neural network models. To obtain latent representations that reflect the most relevant information of the input data, the information bottleneck principle seeks to find a stochastic latent representation Z of the input source X defined by a parametric encoder Pz xx z; 8) that is max- imally informative about a target Y while constraining the mutual information between the latent representation and the input X. For representation parameterized by parameters 0, this principle can be expressed as the optimization problem
max, I (Z () s.t. (() II
where +)) is the mutual information function and Í is compression constraint. We can equivalently express this optimization problem as maximizing the objective
F(0) = T(Z,Y;
X;;)
(2)
where B is a Lagrange multiplier that trades off learning latent representation that is maximally informative about the target Y with learning a representation that is maximally compressive about the input X [2].
3.2 A Multi-Modal Information Bottleneck Principle
Unfortunately, for VLPMs, the loss function above is not suitable since we wish to learn interpretable latent representations using only text and vision inputs without relying on task-specific targets that may not be available or expensive to obtain. To formulate multi-modal information bottleneck principle for VLPMs, we need to develop an optimization objective that is more akin to optimization objectives for self-supervised methods for image-text representation learning that only use (text, image) pairs [21, 18].
This learning problem fundamentally differs from supervised attribution map learning for unimodal tasks, For example, we may have an image of a bear, Xbear and a corresponding label, Ybear = "bear". For unimodal classification task, can simply maximize I I(Ybear; Zbear; 0) 3I Z 0) with respect to 6, where Zbear is the latent representation of X Xbear-
In contrast, in image-text representation learning, we typically have text descriptions, such as "This a picture of a bear" (T/ear) instead of labels [21]. In this setting, both X and Ihear are "inputs" without pre-defined corresponding label. To obtain task-agnostic image-text representation independent from any task-specific ground-truth labels, we would like to use both input modalities and define multi-modal information bottleneck principle and whereas the outputs are closely dependent on the specific downstream task. This requires defining an alternative to the "fitting term" [[2 0) of the conventional information bottleneck objective.
Fortunately, there is a natural proxy for the relevance of information in multi moaltll data. If image and text inputs are related (e.g.. text that describes the image), good image encoding should contain information about the text and vice versa. Based on this intuition, we can express multi-moda information bottleneck objective for X, with E .M = {modalityl, modality2). as (0.... I(Zm, Emy; 6m) - (T Xm em.)
where m' = Mmmm the complement of m, and Fm is embedding of modality m'. Next, we will show how use this multi-modal variant of the information bottleneck principle for attribution.

---

3.3 A Multi-Modal Information Bottleneck for Attribution
To compute attribution maps for image and text data without access to task-specific labels, we will define an information bottleneck attribution method for multi-modal data.
To restrict the information flow in the latent representation with simple parametric encoder, adapt the masking approach in Schulz et al. [24]. For clarity and brevity, we henceforth represent the Vm -m-dimensional latent representation Z. in its vectorized form in R with J = Vm W... Assuming independence across latent representation dimensions, we then define
pm x.. (2m Imm mm) (=m him (Ti; Am) @ fin (Imm), am diag((1, Immm (Tim; )))) (4)
where for a pair of modalities M = {image, text) } with m M,, pm = {>mm =m: em are parameters. hmm (Im; Am) RR is mapping parameterized by Am. f (Xm) E R. is the vectorized output of the tn layer of modallty-speccife neural network embedding Jm. on E RRso is a hyperparameter, diag[-] represents an operator that transforms a vector into a diagonal matrix by placing the vector's elements along the main diagonal, 1, E IR' is an all-ones vector, and @ is the Hadamard product. To avoid overloading notation, we will drop the subscript in probability density functions except when needed for clarity. Based on Equation (4), we can express the stochastic latent representations as
Zm Xmi 6m = Nm (Zmi Am) ① f'r (zm) + o (1, hm (Tmi Am)) O where E (O, Iy). From this reparameterization, we can see that [hm (Imi Am for i E J} means that no noise is added at index so [2m]; will be the same as the original I'm (Im)i. whereas [lmm (Tmi Am) )| = O means that [Zm]] will be pure noise.
(5)
We can now express the multi-modal information bottleneck attribution (M2IB) objectives as Fumage (Ounage) I( (Zimage+ Etext: uumage) Sumage I (Zimage: Ximage: Oumage) Frext (frext) [Ztext Einage; frexa) - Stext I (Zrext, X Xtext; Brext),
(6)
which we can optimize with respect to the modality-specific sets of parameters Ammage and Atext= respectively. {8iaage Timage i Limgee } and (Beext; Ttext, } are each sets of hyperparameters.
3.4 A Variational Objective for Multi-Modal Information Bottleneck Attribution
To obtain tractable optimization objectives,
variational approximation. First, note that T(Zm, xm 0m) = Exx... [DKi, X.m. (- Xm; 0m)) || pz... (=;0)))] where Zm Xm; Om can be sampled empirically whereas p(Zmi Om) does not have analytic expression because the in- tegral p(zm; mm) = [[(2m I.m;; Om p(Fm) drmm is intractable. To address this intractability, we approximate p(zm) by (2m)) [2 0,15). This approximation leads the upper bound T( T(?m, Xm, 0m) = F Pxx. [F2,, m. Xm Xm; Om)
qz... (-)) DKK. (Pz... qz..)
< E Ep DKr. (pzm. x., (- Xmi 6m)) 1z... (-))
(8)
Fcompression (0m).
m
Next, while the unimodal information bottleneck attribution objective uses ground-truth labels to compute the "fit term" in the objective, the multi-modal information bottleneck attribution objectives require computing the mutual information between the aligned stochastic embeddings.
f pleme, Ami 0m) log plem's 7mi 0m) dem" dām p(em )p(zm)
1(2m, Emi; 0m) =
p(eme Zm)
= p(eme, Am; Om) log p(em')
dem' dām.
(10)
which is not in general tractable. To obtain an analytically tractable variational objective, we approx-
imate the intractable p(em' (m)) by variational distribution g(em') (mm) = (em's 9m (Zm) Lk),
where Im is a mapping that aligns the latent representation of modality m with Eny and K is the dimension of the embedding, and get
(2 Env; 0m)) 2 / plem', Zmi 0m) log q(emm) 2mm) dem džm
/ p(em) Im)p(zm xmi 0m) log q(em 2m) deem dem dēm
(11)
(0m))
5

---

With this approximation, we can obtain a tractable variational optimization objective by sampling Xm and En from the empirical distribution
1 Σ 60 - du(em' - fmme (4/m/ (n)
fa eml
)),
(12)
where J (Tme) is the embedding input Imm under the VIPM.. With these approximations, we can now state the full variational objective, which is given by
Fapprox (fm) = F (fm)) - F Frompressin (fm).
(13)
m The derivation of this objective has closely followed the steps in [2]. In practice, the objective can be computed using the empirical data distribution so that
N
ʃ p(zm a:in) 0m log ((em' (2m) d2m
Fempirical Fm
(0m)
N
(14)
-Bm (P2. (xm ( (aa || qzm (-).
Final Variational Optimization Objective. Finally, we assume that hm (arl1) Am ) (z(())))m)
(ni)
(i.e.,
each input point has its own set of attribution parameters), that In is the mapping defined by the post- bottleneck layers of a VLPM for modality m, and that for each evaluation point the final embeddings for each modality get normalized across the embedding dimensions (i.e., both mapping, Im and fmr, contain embedding normalization transformations). For normalized gm (2m) and em', the log of the Gaussian probability density ff (Im²) ) 9m (Zm)) simplifies and is proportional to the cosine similarity between Sm (I'm') and Im (2mm)) giving the final optimization objective
fempirical (0m)
1
Σ
f
pmzm (x(() Scosine (em's Im (2m) d2m
N
(15)
Bm (pzm 1 C: 3 ;n) 0m)) || qzm.. (-)),
where (tosiue(::)) is the cosine similarity function. For gradient estimation during optimization, the remaining integrals can be estimated using simple Monte Carlo estimation and reparameteriza- tion gradients. For Om (mmm mn, (,). the objective function is maximized with respect to Am independently for each modality, and Bm. om, and fm are modality-specife hyperparameters.
Empirical Evaluation
We evaluate the proposed attribution method using CLIP [21] on four image-caption datasets, includ- ing widely-used image captioning datasets and medical datasets. Our main datasets are (i) Conceptual Captions [26] consisting of diverse images and captions from the web, and (ii) MS-CXR (Local Alignment Chest X-ray dataset; [3]), which contains chest X-rays and texts describing radiological findings, complementing MIMIC-CX (MIMIC Chest X-ray; [13]) by improving the bounding boxes and captions. In addition, also include some qualitative examples from the following radiology and remote sensory datasets to show the potential application of our model in safety-critical domains. Namely, we have (iii ROCO (Radiology Objects in COntext; Pelka et al. [19]) that includes radiology image-caption pairs from the open-access biomedical literature database PubMed Central, and (iv) RSICD (Remote Sensing Image Captioning Dataset; Lu al. [16]), which collects remote sensing images from web map services including Google Earth and provides corresponding captions.
4.1 Experiment Setup
For all experiments, we use pretrained CLIP model with ViT-B/32 [7] as the image encoder and 12-layer self-attention transformer as the text encoder. For Conceptual Captions, we use the pretrained weights of openai/clip--ee prrit--ase-patctte
For MS-CXR, we use CXR-RePaiR [8] which is CLIP finetuned on radiology datasets, and compare the impact of finetuning in Section 4.5. For each (image, caption) pair. we insert an information bottleneck into the given layer of the text encoder and image encoder of CLIP separately, then train the bottleneck using the same setup as the Per-Sample Bottleneck of original IBA [24], which duplicates a single sample for 10 times to stabilize training and runs 10 iterations using the Adam optimizer with a learning rate of 1. Experiments show no significant difference between different learning rates and more training steps. We conduct hyper- pyper-parameter tuning on the index of the layer I, the scaling factor 3, and the variance o², For a discussion of these hyperparameters in the multi-modal information bottleneck objective, see Appendix A.
PCr. https: bttps://hune httph//ggg.gg.ac..coopeeeaaaaaae
wit--ase--paaeeeeee batch33.

---

key
while planes
ground
(a) Different Highlights with Respect to Different Text.
(b) Multiple Occurrences of Same Object.
Figure 2: Example Saliency Maps when Involving Multiple Objects. Our method can successfully detect all occurrences of all relevant objects in image and text.
4.2 Qualitative Results
We qualitatively compare our method with 5 widely used attribution methods, including gradient- based GradCAM [25] and Saliency method [28]. perturbation- Kernel SHAP [17] and RISE [20], and tranefirmer-specffic method [6]. We show one image-caption example and two radiology examples in Figure 1 and provide more qualitative comparisons in the Appendix B. As shown, our method is able to capture all relevant objects appearing in both modalities, while other methods tend to focus on one major object. As illustrated in Figure 2a, the highlighted areas in the image change according to different inputs, and our method can capture complicated relationships between image and text when involving multiple objects. Our method is also able to detect multiple occurrences of the same object the image, as illustrated in Figure 2b.
4.3 Localization Test
We quantitatively measure the effectiveness of our proposed attribution method by evaluating its accuracy in zero-shot detection for images. We binarize the saliency map such that the areas with scores higher than the threshold (75%) are assigned while the rest are assigned O. We denote the resulting binary map as Spred. We also construct a ground-truth binary map, Sgt. using the bounding boxes provided by MS-CXR [3], where the region inside the bounding boxes assigned to 1 while the outside is assigned to O. Note that some samples have multiple bounding boxes, and we consider all of them test the method's 's multi-occurrence detection ability. Then, we calculate the IoU (Intersection over Union) of S pred and Sgt. Namely, for images with height of and width of m, the score is calculated by
E -==1 1 o5 Sg
Localization
(16)
E Ej= "pred St
where 1 is the indicator function, A is the logical AND and V is the logical OR operator.
We found that M2IB attribution attains an average IoU of 22.59% for this zero-shot detection task. outperforming all baseline models, as shown in Table 1. Recognizing that a localization score of 22.59% appears somewhat low in absolute terms, we briefly note that there are two potential causes that could lead to the low absolute values in the localization scores: (i) M2IB attribution indeed generates segmentation instead of bounding boxes, so evaluation by bounding boxes would underestimate the quality of the saliency map. (ii) The model under evaluation [8] is not finetuned for detection and may only have learned a coarse-grained relationship between X-rays and medical notes.
4.4 Degradation Tests
While the localization test shows that M2IB attribution may be a promising zero-shot detection and segmentation tool, the localization test may underestimate the accuracy of attribution since even perfect attribution map can produce a low localization score if the (finetuned) VLPM under evaluation poor at extracting useful information chest X-ray classification.
is very likely for challenging specialized tasks like
To get more ine-grained picture of the usefulness of M2IB, we use three additional evaluation metrics to compare M2IB to competitive baselines. The underlying idea of all three evaluations is that removing features with high attribution scores should decrease the performance, while discarding features with low attribution scores can improve the performance as noisy information is ignored. We randomly sample 2,000 image-text pairs from Conceptual Captions and 500 image-text pairs from MS-XXR. For each dataset, we perform ten experiments for each evaluation metric (five for ROAR+) and report the average score with the standard error in Table 1.
7

---

Table 1: Quantitative Results. The boldface denotes the best result per Means and standard
errors were computed over ten random seeds.
Methods
GradCAM (25]
Saliency [28]
KS [17]
RISE [20]
Chefer al. [6]
Ours
CC image
Conf. Drop
4,96 0.01
1,99 = 0.01
1994 = 0.01
1.12 0.01 0.01
1.63 0 0.01
0.01
Conf. Incr.
17.84 = 0.08
22.95 2 0.12
25.18 = 0.28
35.72 0.14
37,41 = 0.12
41.55
0.19
% ROAR+1 f
2.29 = 0.41
6.88 = 0.88
1.56 0.88
3,15 = 0.97
7.66 = 0.55
10.59
0.85
CC text
Drop
1.78
1.30
1,06
1,06
68. Coní. Incr f
29.71 1: 0.19
18,96 1: 0.15
46.87 :: 0.21
38.11 4: 0.48
38.42 E 0.11
38,55 0.20
ROAR+ 1
43.23
43.74 0.65
+
49.04 + 1.12
53.57 1.26
1.12
Conf. Drop
2.76
0.81 + 0.01
187
0.01
MSCXR image
Conf. Incr. ROAR+ f
12.64 0.46
35.08 + 0.44
10:24 = 0.68
1.28 = 0.44
21.44 = 0.46
45.92 + 0.70
3.54
1.35
12.67
16.79
24.42 1.19
38.7 0.86
% Localization
5.56 1. 0.13
21.6 21.6.100 0.16
1.77 1 0.13
10.97 1 0.24
21.65 1. 0.25
22.59
0.14
MSCXR
Conf. Drop
2.26 + 0.04
3.35 0.03
2.4 = 0.05
1.16 = 0.02
2.93 0.03
2.28 0.04
Conf. Incr, % ROAR+ 1
36.24
0.54
34.12 =
57.2 0.68
28.08 0.34
35.48
0.69
11.07 = 0.62
15.79 + 0.92
14.28 + 1.09
12.09 + 1.52
9.11 + 0.6
16.31 + 0.75
Original
Saliency Map
Attribution Weighted Image
Corrupted (binarized M)
man holds what is believed be some the debria that caused damage vehicles..
man Bati What is beieved some the WeaKEE that caused Hamaage to Subicles
holds <B> <Br <ão debr's <Bo damage <Bs vehicles.
<B> belleved <B> of <Bo dio <Bs <Bs-
Figure 3: Visualization of Degradation. The third column is obtained by calculating the element-wise product of the original image and saliency map, while the text with attribution scores lower than 50% percentile is masked by a blank token <B>. It is used in the Increase in Confidence metric and Drop in Confidence metric. The fourth column shows an example the training data in ROAR+. We replace the image pixels with attribution scores higher than 75% percentile by the channel mean and replace the text tokens with attribution scores higher than 50% by a blank token <B>, The results in Table 1 use the padding token as the blank token <B>
Drop in Confidence [4]. An ideal attribution method should only assign high scores to important features, thus we should not observe drop in performance if only the high-attribution parts allowed in the input. For images, we use point-wise multiplication of the saliency map and the image input. Since scaling token ids is meaningless, with attribution scores in the top 50% are kept. We provide an example of distilled image and text in Figure 3. Formally, define this score by
binarization similar [33] where only tokens
N
Confidence Drop
max(0, 9. - st),
(17)
N
where o; is the cosine similarity of features of original images and texts, and 8; is the cosine similarity when one modality is distilled according to the attribution. The lower this metric is, the better the attribution method is. This metric implemented in the gytorch-gradee
repository. 3 Increase in Confidence [4]. Similarly, removing noisy information in the input might increase the model's confidence. We compute
N Σ 1 < 8i),
Confidence Increase
(18)
where 1 is the indicator function and the definition of o; and s; is the same as above. Higher values indicate better performance. This metric is also implemented in the -orodcaagaa repository."
'GradCAM and its variants are usually applied to image classification and use the softmax outputs of each class as confidence scores. Since our setting does not contain any labels, we use cosine similarity with the other modality instead.

---

Remove and Retrain + (ROAR+, an Extension of ROAR [II].. We finetune the base model on the degraded images and texts where the most important parts are replaced by uninformative values (i.e.. channel means of images or padding tokens for texts, see Figure 3) and evaluated on a validation - set of original inputs. If the attribution method is accurate, a sharp decrease in performance is expected because all useful features are removed, and the model cannot learn anything relevant from the degraded data. We split the testing dataset into 80% training data and 20% validation data. We use the same contrastive loss used for pretraining CLIP and define the score by (le - lo/lo.. where lo is the validation loss retraining using the original data, and Le is the validation loss when retraining with corrupted data. We repeat the process five times and report the average score.
The results are summarized in Table 1. M2IB attribution outperforms baseline models in almost all numerical metrics, except for perturbation based methods, which achieve better Increase/Drop in Confidence scores for texts. Perturbation- baasaa methods perform better for short text because they can scan all possible binary masks on the text and then find the optimal one with the highest confidence score. However, this kind of method is very computationally expensive. We use 2,048 masks for image and 256 masks for text in RISE, where each mask of each modality requires one forward pass to get the confidence score, resulting in 2.3k forward passes (7.8s on RTX8000 with a batch size of 256). In contrast, M2IB attribution only requires 100 forward passes and takes 1.2s for one image-text pair.
In general, removing pixels or tokens with lower attribution scores using. M2IB attribution generally increases the mutual information with the other modality, while masking by our attribution map generally decreases the relevance with the other modality and makes the model perform worse when retraining on the corrupted data. This confirms that our method generates useful attribution maps.
4.5
Sanity Check
We conduct a sanity check to ensure our method is, in fact, sensitive to model parameters. We follow the sanity check procedure proposed by Adebayo et al. [1], where parameters in the model are randomized starting from the last to the first layer. As shown in Figure 4, M2IB passes the sanity check as the attribution scores of image pixels and text tokens change as the model weights change. Our method also produces more accurate saliency maps for finetuned models compared to pretrained models, which further confirms that the resulting attribution can successfully reflect the quality of the model. Since we insert the information bottleneck after a selected layer (layer 9 in this case), the randomization of this and previous layers appears to have a larger influence on the output.
4.6 Error Analysis and Limitations
We notice that our proposed attribution method generally performs well on text but sometimes shows less satisfying performance on images. By inspecting the qualitative examples, we observe that M2IB sometimes fails to detect the entire relevant regions in images. As shown in the fourth ("sea" example) and fifth ("bridge" example) rows in Figure our method only highlights a fraction of the object in the image, although it should include the whole object. This is probably due to the fact that the model under evaluation only relies on a few patterns in the image to make its prediction. Increasing the relative importance of the fitting term (i.e., using smaller (3) enlarges the highlighted area. However, we don't recommend using extreme |3 because will break the balance between fitting and compression and thus make the information bottleneck unable to squeeze information.
We also note that M2IB is sensitive to the choice of hyperparameters. As shown in Figure 5, different combinations of hyperparameters will generate different saliency maps. We show how to use the ROAR+ score to systematically select the optimal hyperparameters and also provide visualization to illustrate the effect of different hyperparameters in Appendix A. Since there is no convention on evaluating the attribution method, we suggest considering various evaluation metrics, visualization of examples, and the goal of the attribution task when choosing hyperparameters. We emphasize that M21B should be used with eaution since attributing the success or failure of model solely to a set of features can be overly simplistic, and different attribution methods might lead to different results.
"The original ROAR also corrupts the validation data, which means that different methods might use varied training and validation datasets, making direct comparisons difficult. To ensure fair comparison, we consistently the original data as the validation set for all methods.

---

Finetaned
Pretrained
Prujection
Layer
Layer 6
Luyer
Random
Ours
focal breas of
TOCM Sreas of corisaiitation
Bocal Broes consoliaaaion
Tocal BrORS sorscciaiiinn
tocal WERRS consonssation
Roca 300RS carselication
Sccal bress
EX
GradCAM
muNI
b bioterrl Escal consolidation
MuRI
DUNTAAOI MUM local Das consolidation
MUR
ST Gccal RERASS consolisation
mutii
claatrro
MUN
conssiidation
EigenCAN AM
(osa)
O
MI
Gcal IEA
Dxall
ul
lctal MENNS of
local SIti of
Ictal VESAES o
Figure 4: Saliency Maps for Sanity Checks. "Finetuned" represents the model that is finetuned on MIMIC-CXR [13], Chest X-ray dataset. "Pretrained" represents pretrained CLIP [21] from OpenAl. "Projection" represents CLIP with randomized projection layer, which is the last layer of the image encoder or text encoder that projects image or text features into the shared embedding space. "Random" means that all parameters in the model are randomly initiated. The remaining columns represent models with weights randomized starting from the last to the given layer. The results suggest that the saliency maps M2IB attribution are sensitive to model weights, as desired, meaning that M2IB passes the sanity check.
5
Discussion
We developed multi-modal information bottleneck (M2IB) attribution, an information-theoretie approach multi-modal attribution mapping. We used CLIP-VTT-B/32 in our experiments, but M2IB attribution can be directly applied to CLIP with alternative neural network architectures and compatible with any VLPM for which the features of all modalities are projected into shared embedding space-a commonly used approach in state-of-the-at multi-modal models.
Going beyond vision and language modalities, Girdhar et al. [9] recently proposed new multi- modal model, ImageBind, which aligns embeddings of five modalities to image embeddings through contrastive learning pairs of images with each modality and uses similar architecture as CLIP, where the encoder for each modality is a transformer. We applied M2IB attribution to pairs six different modalities using ImageBind-which required minimal implementation steps-and provided qualitative examples in Appendix C to illustrate that M2IB attribution can be applied successfully to more than just vision and text modalities.
In this paper, we provided exhaustive empirical evidence that M2IB increases attribution accuracy and improves the interpretability of state-of-the-ar vision-language pretrained models. We hope that this work will encourage future research into multi-modal coormation-theorede attribution methods that can help improve the interpretability and trustworthiness of VLPMs and allow them to be applied to safety-co safety-critical critical domains where interpretability is essential.
Acknowledgments
This work was supported in part through the NYU High-Performance Computing services well as by NSF CAREER USS-244599, NSF I-DISRE 193471, NSF IIS-1910266, BigHat Biosciences, Capital One, and Amazon Research Award.
10

---

References
[1] Julius Adebayo, Justin Gilmer, Michael Muelly, lan J. Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. CoRR, abs/18 10.03292, 2018.
[2] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2017.
[3] Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez- Valle, Hoifung Poon, and Ozan Oktay. Making the most of text semanties to improve biomedical vision-language processing. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022, pages 1-21, Cham, 2022. Springer Nature Switzerland. ISBN 778-3-031-2005921
[4] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 839-847,
2018. doi: 10.1109/WACV 20188.0097.
[5] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. CoRR, abs/2012.09838, 2020.
[6] Hila Chefer, Shir Gur, and Lior Wolf. Generic attention- explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 397-406, October 2021.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[8] Mark Endo, Rayan Krishnan, Viswesh Krishna, Andrew Y. Ng, and Pranav Rajpurkar. Retrieval- based chest x-ray report generation using pre-trained contrastive language-image model. In Proceedings of Machine Learning for Health, volume 158 of Proceedings of Machine Learning Research, pages 209 209-219, 2021.
[9] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One embedding space bind them all, 2023.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10. 10.1109/CP...
[11] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for inter- pretability methods in deep neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[12) Zhiying Jiang, Raphael Tang, Ji Xin, and Jimmy Lin. Inserting information bottlenecks for attribution transformers. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 3850-3857 Online, November 2020. Association for Computational Linguistics. doi: 10. 00111866//v11 102.533/112200..mdiin... htw.22....nniing-en.....44
[13] Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Roger G. Mark, and Steven Horng. MIMIC-XR: A large publicly available database of labeled chest radiographs. CoRR, abs/1901 007042, 2019.
[14] Shuying Liu and Weihong Deng. Very deep convolutional neural network based image clas- sification using small training sample size. In 2015 3rd IAPR Asian Conference on Puttern Recognition (ACPR), pages 730-734, 2015. doi: 10. 201.......0157866500
11

---

[15] Siqu Long, Feiqi Cao. Soyeon Caren Han, and Haiqin Yang. Vision-and indon-and--ae -are-aand--aggga language pretrained models: survey. In Luc De Raedt and Luc De Raedt, editors, Proceedings of the 31st International Joint Conference on Artificial Intelligence, IJCAI 2022, IJCAI International Joint Conference on Artificial Intelligence. pages 5530-5537. International Joint Conferences on Artificial Intelligence, 2022. doi: on Artificial Intelligence, IJCA 2022; ; Conference date: 23-07-2022 Through 29-07-2022.
(T0.66//JCAA.22222222222 31st International Joint Conference
[16] Xiaoqiang Lu, Binqiang Wang, Xiangtao Zheng, and Xuelong Li. Exploring models and data for remote sensing image caption generation. IEFE Transactions Geoscience and Remote Sensing, 56(4):2183-2195, 2017. doi: 10.1109/TGRS 20..
[17] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett,, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[18] Norman Mu, Alexander Kirillov, David A. Wagner, and Saining Xie. SLIP: self- supervuuervii meets traguage-image pre-training. CoRR, abs/21 12. 12.1220,,, 2021.
[19] Obioma Pelka, Sven Koitka, Johannes Rückert, Felix Nensa, and C. Friedrich. Radiology Objects in cOntext (ROCO): A multimodal image dataset. In CVI-STENTIABELS 2018.
[20] Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference (BMVC), 2018.
[21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021.
[22] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135-1144, 2016.
[23] Yao Rong, Tobias Leemann, Vadim Borisov, Gjergji Kasneci, and Enkelejda Kasneci. A consistent and efficient evaluation strategy for attribution methods. In Proceedings of the 39th International Conference on Machine Learning, pages 18770-18795. PMLR, 2022.
[24] Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. In International Conference on Learning Representations, 2020.
[25] Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-CAM: Why did you say that? visual explanations from deep networks via gradient-based localization. CoRR, abs/1660.02391, 2016.
[26] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.11853/vvl 1/P18 1238.
[27] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can CLIP benefit assion-and-lanuuage tasks? In International Conference Learning Representations, 2022.
(28) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference Learning Representations, 2014.
[29] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for Simplicity: The All Convolutional Net. arXiv e-prints, art, arXiv: 1ee1141...00 412.6806, December
2014. doi: 10.48550/arXiv. 1412.22....111.6666666
12

---

[30] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319-3328. PMLR, 06-11 Aug 2017.
[31] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1-5. IEEE, 2015.
[32] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, 2000.
[33] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-CAM: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 24 25, 2020.
13

---

Appendix
Table of Contents
Experimental Details and Further Experimental Results
15
17 Extension to Additional Modalities With ImageBind
Attribution Maps for Additional Examples
19
14

---

Appendix A Experimental Details and Further Experimental Results
controls the relative importance of the compression term. The larger the is, the less information is allowed to flow through this layer. As shown in Figure 5a, too large and too small & generates similar attribution maps in terms of relative importance. However, too small / allows nearly everything through the bottleneck, whereas too large / nearly discards everything.
controls the values of the noise added the intermediate representations. When is very small, the values the noise will be close to O, thus having minimal impact on the intermediate representations. This effect similar to the situation when 3 is very small and IB will add almost noise (Figure 5b). also directly affects the compression term as smaller o will lead to higher KL divergence. Thus, and [ are correlated with each other and we perform grid search find the best combination.
Layer l where the information bottleneck inserted also impacts the attribution. Inserting the bottleneck too early will prevent the model from learning informative features while inserting the bottleneck too late reduces the impact the bottleneck (Figure 5c). We also observe that the attribution of texts is usually more stable than images.
These hyperparameters can chosen according to numerical metrics mentioned in Section 4.4. We randomly sample 500 image-text pairs from Conceptual Captions and 500 image-text pairs from MS-CXR, which is ensured to not overlap with the test set Section 4. We then perform grid search for the best combination of 3 {1, 0.1, 0.01} and a² {1, 0.1, 0.013 and l {7,8,9) (indexing from O), and find the best combination of hyperparameters shown Table 2.
Table 2: Hyperparameter Tuming Results for L, 3 and a: We calculate the ROAR+ score for different combinations for 3 runs and report the average. For each table, the highest score is in bold and indicates the performance is optimal for this set of hyperparameters. For texts of Conceptual Captions, we select f = 9, 8 = 0.1 and a Since ROAR+ uses binarized saliency maps (75% threshold for image pixels and 50% threshold for text tokens, it mainly focuses on the features with high attribution scores and neglects the change the attribution for less important features. Thus, sometimes hyperparameters with slightly lower ROAR+ scores might generate more visually appealing results as in Figure 5, though the numerical results are consistent with qualitative examples general.
(a) Conceptual Captions Image
(b) Conceptual Captions Text
d²=1
a² 201
2 =0001
d²=1
===0.1
22=0.01
8=1
10.13
10.78
7.11
8=1
15.49
10.74
10.78
t=7
8-0.1
8.00
7.96
6.66
è=7
8=0.1
13.31
13.73
14.18
8=0.01
7.12
6.69
8.32
8=0.01
12.96
15.61
12.78
8=1
12.08
8.25
9.49
8=1
11.29
16.47
8.82
t=8
8=0.1
10.33
6.33
t=
8=0.1
13.26
13.86
8=0.01
10.05
6.77
8.23
3=0.01
8.99
11.09
13.72
8=1
7.75
9.44
5.63
B=1
13.85
18.16
9.87
t=9
8-0.1
12.72
8.91
9,94
6=
8=0.1
18.16
15.03
8.97
8=0.01
8.88
6.42
9.08
8=0.01
11.72
8.46
9.80
(c) MSCXR Image
(d) MSCXR Text
d²=1
=0=0.1
200001
o²=1
-²=0.1
200001
8=1
35.24
29.56
25.20
8=1
15.77
10.91
9.22
t=7
8=0.1
34.15
30.44
30.28
6=7
8=0.1
12.60
11.15
8.92
8=0.01
30.12
30.21
29.32
8=0.01
13.89
12.31
13.11
8=1
34.59
29.35
30.07
B=1
10.76
14.34
12.13
t=8
8=0.1
32.33
30.68
28.04
t=8
3=0.1
12.67
10.53
11.09
8=0.01
31.99
33.24
36,99
8=0.01
10.58
12.49
15.48
3=1
31.76
28.43
32.94
B=1
11.70
11.01
12.24
1=9
8=0.1
37.17
35.06
32.22
1=9
8=0.1
10.39
13.54
12.48
8=0.01
36.19
34.59
30.18
8=0.01
12.93
14.64
12.57
15

---

0.001
A
B
consalid dation
In
consalidation
in
consolidation
in
conso ldetion Ient
in
consolidetiion
in
the
left
Iung the left lung the Ie Jung the
lung
the Im kung
Junny Bnimals of the week pictures
Tunny animals of the week anima pictures
funny animats of the week, animal pictures
runny animats of the week, animal pictures
tunny animats of the week antimal pictures
animal
(a) Impact of Scaling Factor 3. Higher 8 means higher weight of the compression term.
o²
o² 0.01
o²
o²=1
o²
consolidation
In
consolidation
In
consolidation left lung
In
consolldation
In
consolliation
In
the
Belt
lung
left lung
left lung
left kung
Junny week pictures
funny week pictures
Tunny week pictures
funny animals week pictures
funny animals pictures
animal
animal
animal
(b) Impact of noise variance a².
Smaller o² means lower impact of compression.
1=6
1=7
1=
10
consolidation the lung
consolidation lung
consolidation lung
lung
the
lung
Runny animals of the week, animal pictures
funny animals of the week animal pictures
funny animals of the week animal pictures
funny snimals of the week animal pictures
funny animats of the week animal pictures
(c) Impact of the Layer where IB inserted f. Larger
means deeper layer in the network.
Figure 5: Visualization of the Impact of Different Hyperparameters. B and o² that make the fitting and compression terms at similar scale and deeper layer é usually give better performance.
16

---

Appendix B Attribution Maps for Additional Examples
Original
GradCAM
Saliency
KemeISHAP
RISE
Chefer et
Ours
noted
noted
noted
noted
noced
noted
local tre
BGG consolicat
l0cal
consolication
the left mis Lung
the
left
tnit
left
lung
SHI1
mid
TURS
BHTt
m1/
DH11 11165
TATIS
consolication mas5
consolicatlo mass
comsolicat.oo on IOGSI
consolisat On R02I5
comsotical o1 mmoss
comsoliiat
musss
WARSSUS
versess
mass
petchry
patthy ddction
DARSTIV consolastioon
Deticty CONSSORRSEE.OR
peNcshy consonsaution
pacchr consonddation the
ERIETTV Consonfrssionn
middee and lower
middle
BOM lower
middle and lower
micole and BEWER
MOSOE and lower
dole and lower
moddte
and lower
mild
cardi
mmme gai
Gardl
OrT gall
GRTEI
EXTRRI gAl
cArdi
CRTIR gal
GAnSI
CXTIR gal
midd
BXTR gAll
candii
omne SAI
middle 1000
meddle lbc pncumooia
midilee and lower MIB Anskmmon
and kbb pnsumonis
MISEM
lower
middl
and lower pncumorta
middd
and Inwer
ancumonia
lobe pneummoni
1808
000
Ancumonia
relative importance
relative importonce
Figure 6: Attribution maps for randomly picked examples from the MS-CXR chest X-ray dataset [3]. Note that the attribution score is assigned to each token, instead of each word, due to tokenization.
17

---

Original
GradCAM
Saliency
KemeiSHAP
RISE
Chefer al.
Ours
grandfather grandson setting chees chees
arofatter grandson setting board
Wwwrraattin oronsssn sctting the
and
Grerdfaatio at grandson setting Loard
SF TINE Dranddoo Lhtess
Rnsersatiattn annd Grondsoo Sctting Uhe
Betting
Bronndoo setting the LITuSS
board
boant
Jomy
feid
feld
fold
0f CEOOS
Teld
0T CHOPS
Redd
penson anc. grant the christmas
person and grant front
person and DVINE Fiont
percon RIGH
and
nenson fron
ETTNNT IRN
person
annd grent Front
penson
and GEAMT Chw
the
Brane
the SM0
MBM SRIM
17
17 B00
the MMH
the BMN
welking
walking
giri
911
gir
over
lnidge
bridge
ta callonss on wooden stool
bolloons woodenn stool
ballcons wooden SRO0I
balicons We00dees 50008
looms on
BNllIoons wooden sto0t
RNITIonn mooden
Wooden 8000
SEOOI
cherry
aheng Tailing
charry BaTING
WNNT milk
anamy
falling
milk
3B0
wedsing daisy
wedding Blnsy
& daisy
Barsy
dany
Saisy
image relative importance
test relative importance
Figure 7: Attribution maps for randomly picked examples from the Conceptual Captions dataset [26].
18

---

8 ==1
1=1e-5
k0
V0%
K00
10V
K00
V1vw
view
00005
Figure 8: Attribution maps for MRI and CT examples from the ROCO dataset [19]. The highlighted areas increase when 3 decreases. Due to the lack of segmentation masks in ROCO, we are unable to perform quantitative evaluation.
Appendix C Extension to Additional Modalities With ImageBind
Our method can be easily extended to representation learning of modalities other than image and text, as long as features different modalities are projected into shared embedding space. We use ImageBind [9] as example showcase the effectiveness of our method in interpreting audio-image and audio-text representation learning. Since ImageBind aligns other modalities (audio, sensors that record depth (3D), thermal (infrared radiation), and inertial measurement units (IMU)) embedding to image embeddings through contrastive learning, the embeddings of all modalities are in one shared embedding space. Thus, can insert the information bottleneck into certain layers of the feature encoders of ImageBind respectively and adopt the M2IB objective similar to Equation (6) and Equation (7).
(a) Audio: Dog Barking
(b) Audio: Cat Mcowing
bankna
deg
(c) Audio: Dog Barking
(d) Audio: Cat Mcowing
Figure Attribution map on image and text when the other modality is audio.
19


Reviews:
[{"review_id": "0Cv3bkvXMG", "reviewer": "Reviewer_xLqC", "summary": "This paper proposes an adjusted information-bottleneck-based approach for interpreting multi-modal models. Specifically, the approach was appropriately adapted from the information bottleneck method that has been widely used for interpreting image models. Through various approximations, the method was successfully adjusted for the multimodal case. The experimental results presented in the paper demonstrate the successful identification of important shared information between images and text.", "questions": "1) Lines 135-137 seem to explain the unimodal information bottleneck, so why does Z_text appear?\n\n2) I may have missed it, but what do m and s represent in line 168? Are they the mean and standard deviation? Does it mean that multiple texts are involved?", "limitations": "It seems that there are no particular specific limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Understanding the mutual information between images and text in a multi-modal space is a crucial problem. This paper successfully adjusts the information bottleneck-based explainability method, primarily used in the vision unimodal field, to the multimodal space using various approximation techniques. The qualitative results demonstrate the method's ability to capture the mutually related information between text and images, while the quantitative results reaffirm its superior performance compared to the existing methods. Although there were concerns about cherry-picking due to the limited number of examples, the additional examples provided in the appendix alleviate those concerns.", "weaknesses": "Regarding the approximation part, as it is not within my expertise, I cannot make a definitive judgment. However, I don't have any major concerns about other aspects of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and for your thoughtful and constructive questions and suggestions!\n\nWe were pleased to read that you found that **\"qualitative results demonstrate the method's ability to capture the mutually related information between text and images\"**, that **\"the quantitative results reaffirm its superior performance compared to the existing methods,\"** and that you agree that our work tackles a **\"crucial problem\"**.\n\nPlease let us know if you have any remaining questions!\n\n---\n\n> Lines 135-137 seem to explain the unimodal information bottleneck, so why does Z_text appear?\n\nYes, lines 135--137 are about the unimodal case and $Z\\_{\\text{text}}$ is a typo and should be $Z\\_{\\text{image}}$. The correct description should be “for an image classification task, we can simply minimize the I(V_bear;Z_image) and maximize I(L_bear;Z_image) where Z_image is the latent representation.” Thank you for pointing this out! We have corrected in the revised version.\n\n> I may have missed it, but what do m and s represent in line 168? Are they the mean and standard deviation? Does it mean that multiple texts are involved?\n\nYes, $m_T$ is the mean and $s_T$ is the variance. $T$ here refers to the masked intermediate representation in eq (4), which is multi-dimensional. In our current setting, we only consider a pair of one image and one text, while the text can have arbitrary length. We understand that $T$ can be confused with “Text” and we are considering replacing it with other symbols to avoid confusion.\n\n---\n\nAs noted in the *General Author Response*, we have made a significant effort to take all reviewers' feedback into account. As part of this, we added additional state-of-the-art baselines for transformer models and further improved the quantitative and qualitative results of the proposed method. We would greatly appreciate it if you could take these additions into account as you make your final assessment.\n"}, {"review_id": "PSLCLCKzao", "reviewer": "Reviewer_We26", "summary": "This paper introduces a feature attribution method for vision language models. The method is based on previous works on information bottleneck attributions.  It incorporates Gaussian moment-matching estimator techniques to make the objective function tractable. The experimental results show that the method outperforms CAM-based methods quantitatively and qualitatively. ", "questions": "1. The legend for the attribution maps is missing. The legend should show which color means which value. Why do only image attribution maps have red regions?\n2. The approximation technique assumes each dimension follows Gaussian distributions. Could you please provide any evidence on this?\n3. In Line 236, the threshold of 75% seems arbitrary. Also, I wonder if 17.6% of the average IoU is high enough to show the effectiveness of the method.\n\nTypo\n1. Line 54 dind -> find\n2. Line 139 indepdent -> independent\n3. Line 249 missing citations.", "limitations": "The method involves the modification to the training process; The pretrained models should be finetuned. This will limit the utility of the method. Furthermore, recent studies have shown that the pretrained models become less accurate and robust when finetuned. ", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Applying the information bottleneck attribution approach to multi-modal settings is novel. \n2. The evaluation used diverse evaluation metrics such as Conf. Drop, Incr., ROAR, and ROAR+. It even included a sanity check", "weaknesses": "The baselines included in experiments are very limited; they did not include any explanation methods developed for ViTs. The baselines were GradCAM and its variants only. GradCAM is not a good method for explaining ViTs. In transformer architectures, since the information can go across patches (long-range attention), locational information is not preserved. This violates the assumption of GradCAM that the location in the penultimate layer can be mapped to the input layer. I would recommend comparing the methods with recent state-of-the-art methods developed for ViTs. I found a GitHub repo that well listed recent ViT papers that would be helpful for curating the methods (https://github.com/cmhungsteve/Awesome-Transformer-Attention)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive questions and suggestions!\n\nWe were pleased that you appreciated that our contribution is **\"novel\"** and that our evaluation uses a set of **\"diverse evaluation metrics\"**.\n\nWe conducted several additional experiments to address your comments.\n\nPlease let us know if you have any remaining questions!\n\n---\n\n> The method involves the modification to the training process; The pretrained models should be finetuned. This will limit the utility of the method.\n\nThere appears to be a misunderstanding. As noted in our general comment, **the model under evaluation, CLIP, is entirely frozen**. We insert an information bottleneck into the pretrained and frozen CLIP and only the parameters in the information bottleneck (i.e., $\\lambda\\_{X}$ in our manuscript) are updated. This allows us to have a small number of trainable parameters and makes our method very fast and lightweight in practice.\n\nThe goal of the proposed method is to explain the inner decision-making mechanism of a given, pretrained model, thus **no modifications are made to the pretrained model** and we only train the information bottleneck, which can be viewed as an add-on to the pretrained, frozen model.\n\nWe appreciate your comment and have made this subtlety clearer in the manuscript.\n\n---\n\n> I would recommend comparing the methods with recent state-of-the-art methods developed for ViTs.\n\n**We followed your suggestion** and searched the repo you shared and also searched the proceedings of top computer vision conferences over the past three years, **and compared to an attention-based method** [1] that is designed for bi-modal transformers. **We present the results in Table 1 in the pdf included in the *General Author Response*. The proposed method outperforms this strong baseline in almost all metrics (12 out of 13), which reaffirms the effectiveness of the proposed method.** (Note that lower confidence drop (Conf. Drop.) indicates better performance whereas higher values are better for all other metrics.)\n\nWe believe that the additional comparison makes our empirical evaluation more robust and nicely complements the CAM-based baselines, which have been widely applied in previous works (including to transformers, e.g. in [2][3])\n\n---\n\n> In Line 236, the threshold of 75% seems arbitrary. Also, I wonder if 17.6% of the average IoU is high enough to show the effectiveness of the method.\n\nThe threshold was indeed chosen heuristically. In response to your comment, we conducted experiments using different thresholds (the reported number is the average over five runs) and show that the ranking is almost consistent across thresholds (the number in **bold** is the best). Since for the purposes of this evaluation, we primarily care about the relative value of the IoU (i.e., we do not necessarily care how high the IoU is in absolute terms but are primarily interested in the IoU values relative to other methods' values), the choice of threshold is only important insofar as it may show that different thresholds may produce different rankings across methods.\n\nAs can be seen from the table below, **the proposed method in our paper (M2IB) consistently performs best for all thresholds chosen**.\n\n|Threshold|GradCAM|EigenCAM|Chefer et al.|RISE|Ours|\n|---|---|---|---|---|---|\n|50|9.26|7.3|15.05|8.68|**15.13**|\n|60|9.26|6.49|16.75|8.46|**16.9**|\n|75|8.95|8.47|18.13|7.79|**20.26**|\n|90|6.62|3.69|21.27|5.88|**21.28**|\n\nThe IoU scores should be considered as a lower bound on the performance of each method. Since we only have ground-truth bounding boxes---instead of the actual segmentation masks---the IoU scores will by construction underestimate the accuracy of the saliency maps because the bounding boxes will include irrelevant pixels. Although the absolute value of the localization score might seem insignificant, we include this metric because the relative value is informative for comparing the performance of different methods, as discussed above.\n\n---\n\n> The approximation technique assumes each dimension follows Gaussian distributions. Could you please provide any evidence on this?\n\nWe approximate $p(T_1)$, $p(Z_2)$, and $p(T_1, Z_2)$ by multivariate Gaussian distributions (without factorization). Since $Z$ is defined as a function of the random variable $X$, how close this approximation is to the distributions implied by the model depends on the distribution of $X$ and the model architecture. Schulz et al. (2020) make a similar approximation and justify it by appealing to [4], which states that activations after linear or convolutional layers tend to have a Gaussian distribution. While this may be true for specific distributions over $X$, it is unlikely to be true in general. However, we do not believe that this is a problem in practice. As our experiments with the cosine similarity estimator (e.g., in Appendix B.2) demonstrate, similarity between deterministic representations alone is sufficient to obtain an effective fitting term in the objective.\n\n---\n\n> The legend for the attribution maps is missing. [...] Why do only image attribution maps have red regions?\n\nThank you for catching this! We added the legend to Figure 1 in the attached pdf. We use spectral colors to indicate the attribution scores for images (where red represents the highest value and blue represents the lowest value) and use blue with different shades for texts (where darker blue represents higher scores).\n\n---\n\n### Typos\n\nThank you for pointing out the typos in the manuscript. We have fixed them.\n\n[1] Chefer et al. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. ICCV, 2021\n\n[2] Paul and Chen. Vision transformers are robust learners. AAAI, 2022.\n\n[3] Yang et al. Mitigating spurious correlations in multi-modal models during fine-tuning. ICML, 2023.\n\n[4] Klambauer et al. Self-normalizing neural networks. NeurIPS, 2017.\n\n---\n\nPlease let us know if you have any further questions or concerns!"}, {"review_id": "gCeIl1J5zD", "reviewer": "Reviewer_PE9L", "summary": "This paper aims to enhance the interpretability of vision-language pretrained models like CLIP. To achieve this, the authors propose a multi-modal information bottleneck (M2IB) objective that reduces irrelevant and noisy information while retaining important visual and textual features. M2IB is used for attribution analysis, increasing attribution accuracy and interpretability. M2IB audits representations when only multiple modalities are available without ground truth data. The effectiveness of M2IB is demonstrated using CLIP as an example, showing that it outperforms CAM-based methods both qualitatively and quantitatively.", "questions": "1. M2IB has been qualitatively and quantitatively compared with CAM-based attribution methods. Could the authors also compare it with some mask-based explanation methods like [RISE](https://arxiv.org/abs/1806.07421)? In terms of explanation faithfulness, mask-based explanation methods usually perform better.\n\n\n2. Could the authors discuss the difference between M2IB and this Neurips 2022 paper [Fine-Grained Semantically Aligned Vision-Language Pre-Training]( https://arxiv.org/abs/2208.02515)? Note that I am not one of the authors of this paper and do not have any relationship with them but find out that both M2IB and that work tries to increase the relevance between image and text feature.\n", "limitations": "I do not have any particular limitations to highlight.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is tackling a significantly important research question: the interpretability of vision-language pretrained models;\n\n2. The paper is well-structured, with clear organization and sufficient methodological details provided, enabling readers to comprehend the proposed method;\n\n3. The evaluation is thoroughly conducted. For example, in Table 1, numerous faithfulness metrics, including Confidence Drop/Increase, ROAD, and ROAR, are employed for assessment. \n", "weaknesses": "I do not have any particular weaknesses to highlight; however, I still have some questions that I hope the authors can clarify. Please see the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive questions and suggestions!\n\nWe were pleased to read that you agree that our paper **\"is tackling a significantly important research question\"** and that you found our paper to be **\"well-structured, with clear organization and sufficient methodological details provided, enabling readers to comprehend the proposed method\"** and noted that it contains **\"thoroughly conducted\"** evaluations with **\"numerous faithfulness metrics\"**.\n\nFollowing your suggestion, we added experiments for RISE and discuss the differences between the method proposed in our manuscript and the paper you shared.\n\nPlease let us know if you have any remaining questions!\n\n\n---\n\n> M2IB has been qualitatively and quantitatively compared with CAM-based attribution methods. Could the authors also compare it with some mask-based explanation methods like RISE? In terms of explanation faithfulness, mask-based explanation methods usually perform better.\n\nWe agree that RISE is a useful baseline and followed your suggestion to include a comparison between the proposed method and RISE.\n\nRISE estimates attribution scores empirically by probing the model with randomly masked versions of the input image and obtaining the corresponding outputs. The final saliency map is the weighted average of the masks where the weights are the confidence scores. Since the method is designed for image classification (i.e., it requires labels!), we use the cosine similarity between the feature of masked input and the feature of the other modality as the confidence score (same as how we adapt GradCAM). The main weakness of this method is its high time complexity. The accuracy of the generated saliency map is dependent on the number of masks. For example, in our experiment with RISE, we used 8,192 masks for images and 256 masks for texts (the paper that proposed RISE used an even larger number of masks), where each mask requires one forward pass to get the confidence score. This is a significant amount of computational overhead, especially if we wish to apply the method to a large number of samples. In contrast, our method only requires 100 forward passes. On one RTX8000, RISE takes 26.26s per sample with a batch size of 256, while ours only takes 1.27s. This means that for 10,000 samples, RISE would need roughly 72-75h, whereas our method would only need approximately 2-3h.\n\n\n**We present an empirical comparison between RISE and the method proposed in our paper in Table 1 in the attached pdf.**\n\nWe find that **our method outperforms RISE in the majority of metrics (11 out of 13)** with the fixed hyperparameter search described above and remains very close to the top performer for the rest (2 out of 13). We also included further qualitative examples in Figure 1.\n\n| Dataset | Modality | Metric          | RISE          | Ours      |\n| ------- | -------- | --------------- | ------------- | --------- |\n| CC      | image    | Conf.Drop     | 1.18          | **0.80**  |\n| CC      | image    | Conf.Incr.    | 35.80         | **39.80** |\n| CC      | image    | ROAR+         | 13.88         | **69.97** |\n| CC      | text     | Conf.Drop     | 1.48          | **1.08**  |\n| CC      | text     | Conf.Incr     | **34.00**     | 32.30     |\n| CC      | text     | ROAR+         | 21.75         | **53.66** |\n| MSCXR   | image    | Conf.Drop     | 1.07          | **0.44**  |\n| MSCXR   | image    | Conf.Incr.    | 16.20         | **50.2**  |\n| MSCXR   | image    | ROAR+         | 44.95         | **50.74** |\n| MSCXR   | image    | Localization  | 7.79          | **20.26** |\n| MSCXR   | text     | Conf.Drop     | 5.37          | **5.33**  |\n| MSCXR   | text     | Conf.Incr.    | 17.20         | **16.60** |\n| MSCXR   | text     | ROAR+         | 20.83         | **29.80** |\n\n(Note that lower Conf.Drop indicates better results, while higher values are better for other metrics. The table above is identical to Table 1 included in the *General Author Response*.)\n\n---\n\n> Could the authors discuss the difference between M2IB and this Neurips 2022 paper Fine-Grained Semantically Aligned Vision-Language Pre-Training? Note that I am not one of the authors of this paper and do not have any relationship with them but find out that both M2IB and that work tries to increase the relevance between image and text feature.\n\nOur approach and LOUPE have different goals. LOUPE is a pretraining strategy: It explicitly learns an alignment between visual regions and textual phrases and aims to improve the precision of image-text retrieval and downstream tasks. Our method, M2IB, is an attribution method: It aims to give a visualization of the alignment between visual regions and textual phrases implicitly learned during the pretraining of CLIP and does not modify the training pipeline of CLIP. Instead, it helps us understand the inner mechanism of vision-language pre-training and enhance the faithfulness of vision-language pretrained models like CLIP. \n\nIn LOUPE, it argues that \"Existing methods mainly model the cross-modal alignment by the similarity of the global representations of images and texts... However, they fail to explicitly learn the fine-grained semantic alignment between visual regions and textual phrases, as only global image-text alignment information is available.\" Our method quantitatively and qualitatively shows how well these models _implicitly_ learn semantic alignment between visual regions and textual phrases with only global image-text alignment information. Unfortunately, we cannot compare LOUPE and CLIP using our M2IB because the is no publicly available code for LOUPE. We believe our paper could inspire future works that further improve vision-language pre-training like LOUPE.\n\n---\n\nPlease let us know if you have any further questions!"}, {"review_id": "kFPCg1AEsL", "reviewer": "Reviewer_6SYf", "summary": "Visual-language models (VLMs) are increasingly used for multiple vision-language problems like object recognition, image classification, image captioning, etc. Formally, these models are trained in a self-supervised fashion using large datasets comprising millions of image-text pairs, such that they learn generic image and text representations during the training, which can be fine-tuned for different downstream tasks. However, VLMs are intrinsically very complex and are hard to interpret. In this work, the authors use CLIP, a state-of-the-art visual-language model, and propose a multi-modal information bottleneck (M2IB) objective that compresses irrelevant and noisy features and retains visual and text features relevant to the underlying VLM without requiring ground truth labels. Further, the authors show the effectiveness of M2IB in generating attribution maps with increased attribution accuracy across several high-stakes datasets, such as medical diagnosis.", "questions": "1. The authors mention that they use a \"variational approximation upper bound to estimate the information in the target layer and aim to minimize this upper bound during training with respect to embedding-specific attribution parameters.\" Does this mean that they need access to the training pipeline to generate explanations for an intermediate target layer?\n\n2. The authors do propose M2IB that does not require ground-truth labels **y** and show how we can generate attribution maps without the knowledge of **y**. However, the authors never discuss the possibility of using the zero-shot prediction of the VLM and making the problem simpler. It would be interesting to discuss the utility of zero-shot predictions as VLMs achieve decent zero-shot performance.\n\n3. Any intuition behind how the layer index $m$ affects the resulting qualitative and quantitative performance of the attribution maps.\n\n4. Should we even expect high scores in Eqn. 13? Why do we hypothesize that a VLM will look at all the pixels in the bounding box to make its respective decision?\n\n5. Any comments on why there is no attribution to the vehicle in Figure 3?", "limitations": "Please refer to the weakness and open questions in the above sections. \n\nSuggestion: The authors extensively discuss the limitation of the work in the supplementary document and do not discuss anything in the main draft. It would be good from a reading perspective to include certain aspects of the limitation in the main document.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. One of the main highlights of the work is that it does not require the ground-truth label for generating attribution maps and approximates its impact using mutual information between the image and text representations.\n\n2. In contrast to existing works that have focused on a single modality (images or texts) for supervised learning, the authors propose applying the information bottleneck principle to aid our understanding of the inner mechanisms of VLMs using both image and textual inputs.\n\n3. The proposed solution is clear and appears to be a simple extension of the information bottleneck paper referred to in the paper.", "weaknesses": "1. The hypothesis that a good image encoding should contain information about the text, while a good text encoding should include information about the image is unclear. This is fundamentally questionable as the model never learns a unique mapping between the **Z_image** and **Z_test** representations, i.e., the representations in VLMs are learned in a compact manner and it may be difficult to disentangle representations between similar concepts, say **Z_bear_image** and **Z_panda_text**.\n\n2. The proposed technique comprises a large set of hyperparameters for tuning M2IB and generating attribution maps as explanations and the authors do not provide a clear analysis of how to tune or fix them for different models and datasets.\n\n3. In Lines 214-216, the authors mention that \"for each {image, caption} pair, we insert an information bottleneck into the given layer of the text encoder and image encoder of CLIP separately, then train the bottleneck using the same setup as the Per-Sample Bottleneck of original IBA\". This sounds overly computationally expensive as we need to tune all the hyperparameters for each <image, caption> pair!\n\n4. In Lines 228-229, the authors state that \"our method is able to capture all relevant objects appearing in both modalities, while other methods tend to focus on one major object\" --- is that the aim of the results? It is totally possible that the model is only using limited information to learn a particular image-caption pair. I can see why it's logically unintuitive to humans, but that can totally be what the model is actually doing. A good check would be how different the mutual information on using the attribution generated using CAM-based methods and the proposed method.\n\n5. Minor comment: the paper has several typos and the writing can be improved. For example, \"dind\" in Line 54, \"emoirical\" in Line 183, missing references CXR-RePaiR [] in Line 249, and \"test encoder of CLIP\" in Figure 4 caption. Also, the paper discusses a lot about future works in Section 5. It would be better to talk about the limitations of the work instead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive questions and suggestions!\n\nWe agree that **\"one of the main highlights of the work is that it does not require the ground-truth label\"** and were pleased you found our method **\"clear\"** and **\"simple\"**.\n\nPlease let us know if you have any remaining questions!\n\n---\n> The hypothesis that a good image encoding should contain information about the text [and vice versa] is unclear. [...] It may be difficult to disentangle representations between similar concepts, say Z_bear_img and Z_panda_txt.\n\nWhen Tishby proposed information bottleneck theory [1], he defined relevant information as the information that a signal $X$ provides about another signal $Y$. His examples include \"the information that face images provide about the names of the people portrayed.\" Borrowing his definition of \"relevant information\", we think it's reasonable to hypothesize that a good image encoding should contain information about the text, while a good text encoding should include information about the image, given that image and text are relevant and **both may contain shared high-level information about the object of the two modalities**.\n\nThe goal of M2IB is to visualize how \"good\" the image-text representation is (i.e. how well the model captures and encodes the relevant shared high-level information between image and text). We agree that a pretrained model such as CLIP might find it hard to disentangle representations between similar concepts and thus may make wrong predictions. M2IB may help alleviate such shortcomings by providing an error analysis tool for VL-PMs.\n\nFor example, CLIP struggles with the \"yellow apple\" example (Figure 3 in the attached pdf) because apples are often red whereas pears are usually yellow. Using M2IB to generate saliency maps for this example, the attribution score of \"pear\" is higher than that of \"apple\", explaining the low zero-shot prediction performance of CLIP. M2IB can help us understand VL-PM behavior in more challenging settings.\n\n---\n> the authors do not provide a clear analysis of how to tune or fix [the hyperparameters] for different models and datasets.\n\n> In Lines 214-216, the authors mention [...]. This sounds overly computationally expensive as we need to tune all the hyperparameters for each <image, caption> pair!\n\n**We do not tune the hyperparameters for every single image-text pair**, and you are correct that doing so would be computationally expensive.\n\nFor a given set of hyperparameters and a dataset, we sample $M$ points (in our experiments, $M = 500$) from the datasets, compute the optimal masks (i.e., lambdas) for each of the $M$ image-text pairs using M2IB (which takes ~1.3s per image-text pair), and then use the resulting lambdas to construct degraded data points, which are then used to compute a ROAR+ score for the given set of hyperparameters. We repeat this for all sets of hyperparameters we wish to evaluate. The hyperparameters with the best ROAR+ score are then selected. For a single set of hyperparameters, performing a ROAR+ based hyperparameter search can be performed within a few hours using an RTX8000 GPU.\n\nThis procedure is simple, automatic, and task-driven: It finds attribution maps, which---when used to degrade a data point---lead to the greatest loss in information for downstream prediction tasks.\n\nFor further details on the hyperparameter selection procedure, see the *General Author Response*.\n\n---\n> [...] Does this mean that they need access to the training pipeline to generate explanations for an intermediate target layer?\n\n**No, we do not perform any training on the VL-PM under evaluation (i.e., CLIP in this case); the model is frozen at all times.** \"Training\" refers to the training of the mask parameters ($\\lambda\\_X$ in Eqn. (4) in the information bottleneck, which has the same size as intermediate representation. For further details, please see our *General Author Reponses*.\n\n> It would be interesting to discuss the utility of zero-shot predictions as VLMs achieve decent zero-shot performance.\n\nWhen using zero-shot prediction to generate labels for images, the problem becomes unimodal (image only). The results are similar to our method (as shown in Figure 2(a) in attached pdf) but the approach is **inherently unimodal and requires ground-truth categories**.\n\n**We seek to model a more intricate relationship between _image_ and _text_, where there can be more than one object and the free-form text conveys more information than a class label**. For example, we use CLIP for zero-shot prediction on the image containing a snake and an apple (Figure 2(b) in the attached pdf) and it predicts \"apple\" with probability of 46.61%, \"snake\" with probability of 39.87%. Given the image and text of \"this is a snake and an apple\", our method correctly highlights both snake and apple in image and text. \n\n---\n### Layer depth\n\nWe observe larger highlighted regions in shallow layers (Figure 5c in the Supplementary Material). Our method tends to compress less in shallow layers because masking early will prevent the model from learning strong global features.\n\n---\n### Localization\n\nWe agree that not all pixels in the bounding boxes are relevant. We use bounding boxes due to the lack of exact segmentation masks for MS-CXR, and use bounding-box-based IoU to compare methods.\n\n---\n### Vehicle in Figure 3\n\nThe vehicle appears to be a blurry background in the image, so the penalty from the compression term outweighs the gain from the fitting term by including it. We provide Figure 4 in the attached pdf that highlights both debris and car.\n\n---\n### Typos & Limitations\n\nThank you for pointing out the typos! We have fixed them. We have also moved our discussion of limitations to the main text.\n\n[1] Tishby et al. The information bottleneck method, 1999. \n\n[2] Schulz et al. Restricting the flow: Information bottlenecks for attribution. ICML, 2020. \n\n---\n\nPlease let us know if you have any further questions or concerns!"}]

Author rebuttal:
# General Comment to All Reviewers and Area Chair

We thank all reviewers for their thoughtful and constructive feedback!


## Summary

In this general response, we will
1. clarify that **the proposed method, M2IB, does not require (re-)training a pretrained model**: The parameters of the pretrained model are frozen.
2. highlight a **further improvement in performance** after fixing an issue in our hyperparameter tuning protocol.
3. present **a comparison to two state-of-the-art attribution methods** that were requested by reviewers.

Your questions and comments have been very valuable, and we believe that the added empirical results and clarifications meaningfully improve the manuscript.


## Clarification: M2IB **does not** require (re-)training a pretrained model


Attribution methods are used to explain the predictions of trained machine learning models. They aim to understand the influence of the inputs as per the learned parameters of the model, providing meaningful insights into the model's decision-making process. Therefore, model's parameters are always fixed in the context of attribution methods. In our paper, we use M2IB to understand pretrained/finetuned CLIP and the parameters of CLIP are frozen in all experiments. What we train is the mask parameter ($\lambda\_X$ in Eqn. (4)) in the information bottleneck.

For example, in line 46-48 of our manuscript, we write
> We use a variational approximation upper bound to estimate the information in the target layer and aim to minimize this upper bound during training with respect to **embedding-specific attribution parameters**. 

Here, "training" refers to the training of the mask parameters ($\lambda\_X$ in Eqn. (4)) in the information bottleneck. The mask parameter has the same size as the intermediate representation (e.g. for CLIP model with ViT-B/32, the size is 50 * 768 for image and token_length * 512 for text). That is, we train only around 50K "embedding-specific attribution parameters", which is fewer parameters than would be trained in a relatively small MLP. **All parameters in CLIP are kept frozen.**



## Further improvements in performance with M2IB!

**We were able to further improve the attribution maps obtained with M2IB by performing a more thorough grid search.** Previously, we fixed the layer index and then performed a grid search over beta and variance, then fixed the best beta and variance and searched over different layer indices. We realized that this is not a very effective search, since the selected beta and variance were optimal for the pre-selected layer index but may not be optimal for other layer indices. Thus, we perform a grid search over three hyperparameters -- beta = {1, 10, 100} and variance = {1, 0.1, 0.01} and layer index = {7, 8, 9} -- and select the best combination according to ROAR+ scores, as described in line 283 of the manuscript. We also provide a detailed explanation of the hyperparameter selection procedure in our response to Reviewer 6SYf. **The updated results are shown in Table 1 in the attached pdf.**



## Requested Additions: We added a transformer-specific baseline and a perturbation-based baseline

We followed your suggestions and added two additional strong baselines:

1. **Transformer-specific method** [1]: This method is specifically designed for bi-modal transformers and shows promising results in interpreting CLIP. This method can be only applied to transformers whereas our method does not have any constraints on the model class. We included this baseline in our comparison.
2. **Perturbation-based method** (RISE; [2]): This method estimates attribution scores empirically by probing the model with randomly masked versions of the input image and obtaining the corresponding outputs. The final saliency map is the weighted average of the masks where the weights are the confidence scores. Since the method is designed for image classification (i.e., it requires labels!), we use the cosine similarity between the feature of masked input and the feature of the other modality as the confidence score (same as how we adapt GradCAM). The main weakness of this method is its high time complexity. The accuracy of the generated saliency map is dependent on the number of masks. E.g., we use 8,192 masks for images and 256 masks for texts, where each mask requires one forward pass to get the confidence score. In contrast, our method only requires 100 forward passes. On one RTX8000, RISE takes 26.26s per sample with a batch size of 256, while ours only takes 1.27s. 

**We present the results for these two baselines in Table 1 in the attached pdf.**

Neither of the baselines requires separate training steps. For RISE, the number of masks chosen is a hyperparameter. We used 8,192 masks for images. (For comparison, the paper in which RISE was proposed used 8,000 masks for ResNet.) We used 256 masks for text, since 256 masks can almost cover all possible binary masks for the text in the task.

We find that the new baselines slightly outperform the results reported in our manuscript but that **our method outperforms the baselines in the majority of metrics (10 out of 13)** with the fixed hyperparameter search described above and remains very close to the top performer for the rest (3 out of 13). We also included further qualitative examples in Figure 1. Note that when interpreting the result, lower Conf. Drop is better, while higher values are better for other metrics.)

[1] Chefer et al. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. ICCV, 2021

[2] Petsiuk et al. Rise: Randomized input sampling for explanation of black-box models. BMVC, 2018.

---

Please find a pdf with Table 1 and additional qualitative results attached.

---

**Thank you for reviewing our work!**

Meta-review: This work adapts the explainability method for pre-trained multimodal models via a multimodal information bottleneck. This provides significant research values with the recent arising of multimodal research. The method is thoroughly evaluated in multiple metrics and shows consistent advantage over existing CAM-style explainability methods. Reviewers all agree on the importance of the work and value its technical depth and contribution. Rebuttal addressed reviewers’ concerns. Accepted.
Decision: Accept (poster)