Title: Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution
Venue: NeurIPS 2023
Accepted: yes
Keywords: Interpretability

Attribution Maps

Information Bottleneck

Multi-Modal Learning

Vision-Language Pretrained Models
Abstract: Vision-language pretrained models have seen remarkable success, but their application to safety-critical settings is limited by their lack of interpretability. To improve the interpretability of vision-language models such as CLIP, we propose a multi-modal information bottleneck (M2IB) approach that learns latent representations that compress irrelevant information while preserving relevant visual and textual features. We demonstrate how M2IB can be applied to attribution analysis of vision-language pretrained models, increasing attribution accuracy and improving the interpretability of such models when applied to safety-critical domains such as healthcare. Crucially, unlike commonly used unimodal attribution methods, M2IB does not require ground truth labels, making it possible to audit representations of vision-language pretrained models when multiple modalities but no ground-truth data is available. Using CLIP as an example, we demonstrate the effectiveness of M2IB attribution and show that it outperforms gradient-based, perturbation-based, and attention-based attribution methods both qualitatively and quantitatively.

Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution

Abstract

Vision-language pretrained models have seen remarkable success, but their application to safety-critical settings is limited by their lack of interpretability. To improve the interpretability of vision-language models such as CLIP, we propose a multimodal information bottleneck (M2IB) approach that learns latent representations that compress irrelevant information while preserving relevant visual and textual features. We demonstrate how M2IB can be applied to attribution analysis of vision-language pretrained models, increasing attribution accuracy and improving the interpretability of such models when applied to safety-critical domains such as healthcare. Crucially, unlike commonly used unimodal attribution methods, M2IB does not require ground truth labels, making it possible to audit representations of vision-language pretrained models when multiple modalities but no ground-truth data is available. Using CLIP as an example, we demonstrate the effectiveness of M2IB attribution and show that it outperforms gradient-based, perturbation-based, and attention-based attribution methods both qualitatively and quantitatively.

1 Introduction

Vision-Language Pretrained Models (VLPMs) have become the de facto standard for solving a broad range of vision-language problems [15]. They are pre-trained on large-scale multimodal data to learn complex associations between images and text and then fine-tuned on a given downstream task. For example, the widely used CLIP model [21], which uses image and text encoders trained on 400 million image-text pairs, has demonstrated remarkable performance when used for challenging vision-language tasks, such as Visual Question Answering and Visual Entailment [27].

VLPMs are highly overparameterized black-box models, enabling them to represent complex relationships in data. For example, Vision Transformers [ViTs; 7] are a state-of-the-art transformer-based vision model and are used as the image encoder of CLIP [21], containing 12 layers and 86 million parameters for ViT-Base and 24 layers and 307 million parameters for ViT-Large. Unfortunately, VLPMs like CLIP are difficult to interpret. However, model interpretability is essential in safety-critical real-world applications where VLPMs could be applied successfully, such as clinical decision-making or image captioning for the visually impaired. Improved interpretability and understanding of VLPMs would help us identify errors and unintended biases in VLP and improve the safety, reliability, and trustworthiness of VLPMs, thereby allowing us to deploy them in such safety-critical settings.

To tackle this lack of transparency in deep neural networks, attribution methods, which aim to explain a model's predictions by attributing contribution scores to each input feature, have been proposed for post-hoc interpretability. For example, for vision models, attribution methods can be used to create heatmaps that highlight features that are most responsible for a model's prediction. Similarly, for language models, scores are assigned to each input token. While the requirements for interpretability vary depending on the task, dataset, and model architecture, accurate and reliable attribution is an important tool for making emerging and state-of-the-art models more trustworthy.

GradCAM

Saliency

RISE

Ours

Original

KernelSHAP

image - relative importance

text - relative importance

hungry

hungry

hungry

hungry

hungry

hungry

eating

cheese

ina

eating

cheese

eating

cheese

ina

eating

cheese

目司

eating

cheese

目

eating

cheese

ina

eating

cheese

ina

mousetrap

mousetrap

mousetrap

mousetrap

ill

definedop

山

definedop

山

definedop

definedop

山

defined

op

ill

definedop

definedop

acitieslargely

acities

largely

largely

itieslargely

s

largely

s

largely

acities

largely

intheupperlobe

intheupper lobe

inthe

upperlobe

intheupperlobe

theupperlobe

intheupperlobe

in the

upperlobe

local

local

local

local

local

local

local

consolidation

in

consolidation

回

in

consolidation

in

consolidation

in

consolidation

in

consolidation

in

theleftmidlung

the left

the left

theleft

theleft

theleft

midlung

the left

0.0

0.2

0.4

0.6

80

1.0

0.0

0.2

0.4

0.6

0.8

10

While existing attribution methods focus primarily on unimodal models, we propose an attribution method for VLPMs that allows us to identify critical features in image and text inputs using the information bottleneck principle [31]. Unlike standard unimodal attribution methods, the proposed multi-modal information bottleneck (M2IB) attribution method does not require access to groundtruth labels .

To do this, we formulate a simple multi-modal information bottleneck principle, use a variational approximation to derive a tractable optimization objective from this principle, and optimize this objective with respect to a set of attribution parameters. Using these parameters, we are able to 'turn off' all irrelevant features and only keep important ones and generate attribution maps. In contrast to unimodal information bottleneck attribution [24], we aim to find attribution parameters that maximize the likelihood of observing features of one modality given features associated with the respective other modality. We perform a qualitative and quantitative empirical evaluation and find that M2IB is able to successfully identify important features relevant to both image and text inputs. We provide an illustrative set of attribution map examples in Figure 1.

Our key contributions are summarized as follows:

We propose a multi-modal information bottleneck principle and use it to develop a multi-modal information bottleneck attribution method to improve the interpretability of Vision-Language Pretrained Models.

We perform an extensive empirical evaluation and demonstrate on several datasets-including healthcare data that can be used in safety-critical settings-that multi-modal information bottleneck attribution significantly outperforms existing gradient-based, perturbation-based, and attention-based attribution methods, quantitatively and qualitatively.

The code for our experiments is available at: https://github.com/YingWANGG/M2IB .

2 Related Work

Attribution methods enable post-hoc interpretability of trained models by allocating significance scores to the input features, such as image pixels or textual tokens.

2.1 Attribution methods

Gradient-Based Attribution. The use of gradients as a foundational component in attribution methods has been widely explored in the interpretability literature. Simonyan et al. [28] directly use the gradient of the target with respect to inputs as the attribution score. Building on this, GradCAM [Gradient-weighted Class Activation Mapping; 25] weights convolutional layer activations by average pixel-wise gradients to identify important regions in image inputs. Integrated Gradients [30] introduces two essential axioms for attribution methodssensitivity and implementation invariance , which motivates the application of integrated gradients along paths from a baseline input to the instance under analysis.

Perterbation-Based Attribution. Perturbation approaches involve modifying input features to observe the resulting changes in model predictions. Unlike gradient-based techniques, they eliminate the need for backpropagation, allowing the model to be treated as a complete black box. However, such methods can be computationally intensive, especially for intricate model architectures, given the need to re-evaluate predictions under many perturbation conditions. The LIME [Local Interpretable Modelagnostic Explanations; 22] algorithm leverages perturbation in a local surrogate model. Lundberg and Lee [17] propose SHAP (SHapley Additive exPlanations), which employs a game-theoretic approach to attribution by utilizing perturbations to compute Shapley values for each feature. Furthermore, they propose a kernel-based estimation approach for Shapley values, called KernelSHAP, drawing inspiration from local surrogate models.

Attention-based Attribution. The rise of transformers across multiple domains in machine learning has necessitated the development of specialized attribution methods tailored to these models. Using attention as an attribution method offers a straightforward approach to deciphering and illustrating the significance of various inputs in a transformer's decision-making process. Nevertheless, solely relying on attention is inadequate as it overlooks crucial information from the value matrices and other network layers [5]. To address this issue, Chefer et al. [6] propose a method that learns relevancy maps through a forward pass across the attention layers with contributions from each layer cumulatively forming the aggregated relevance matrices.

Information-Theoretic Attribution. Schulz et al. [24] use information bottleneck attribution (IBA), where an information bottleneck is inserted into a layer of a trained neural network to distill the essential features for prediction. IBA is a model-agnostic method and shows impressive results on vision models including VGG-16 [14] and ResNet-50 [10]. Subsequently, IBA has been applied to language transformers [12] and was shown to also outperform other methods on this task. However, IBA has thus far been focused on only one modality and has only been adopted in supervised learning. To the best of our knowledge, there is no previous work on applying the information bottleneck principle to multi-modal models like VLPMs.

2.2 Evaluation of Attribution Methods

Despite active research on attribution methods, we still lack standardized evaluation metrics for attribution due to the task's inherent complexity. For vision models, attribution maps are frequently juxtaposed with ground-truth bounding boxes for a form of zero-shot detection. However, the attribution of high scores to irrelevant areas might arise from either subpar attribution techniques or flawed models, making it challenging to isolate the root cause of such discrepancies.

To tackle this issue, previous studies have resorted to degradation-based metrics [4, 33]. The underlying principle is that eliminating features with high attribution scores should diminish performance, whereas the removal of low-attributed features, often seen as noise, should potentially enhance performance. Additionally, Hooker et al. [11] introduce ROAR (Remove and Retrain), a methodology that deliberately degrades training and validation datasets in alignment with the attribution map, thereby accounting for potential distribution shifts. Should the attribution be precise, retraining the model using these altered datasets would result in a pronounced performance decline. Rong et al. [23] argue that mere image masking can cause a leak of information via the mask's shape, and thus propose a Noisy Linear Imputation strategy that replaces pixels with the average of their neighbors.

It is important to note that a visually appealing saliency map does not guarantee the efficacy of the underlying attribution method. For example, an edge detector might generate a seemingly plausible saliency map, yet it does not qualify as an attribution method because it is independent of the model under analysis. Adebayo et al. [1] propose a sanity check designed to assess whether the outputs of attribution methods genuinely reflect the properties of the specific model under evaluation. Notably, several prominent models, including Guided Backprop [29] and its variants appear insensitive to model weights and consequently do not pass the sanity check.

3 Attribution via a Multi-Modal Information Bottleneck Principle

In this section, we introduce a simple, multi-modal variant of the information bottleneck principle and explain how to adapt it to feature attribution.

3.1 The Information Bottleneck Principle

The information bottleneck principle [32] provides a framework for finding compressed representations of neural network models. To obtain latent representations that reflect the most relevant information of the input data, the information bottleneck principle seeks to find a stochastic latent representation Z of the input source X defined by a parametric encoder p Z | X ( z | x ; θ ) that is maximally informative about a target Y while constraining the mutual information between the latent representation Z and the input X . For a representation parameterized by parameters θ , this principle can be expressed as the optimization problem

where I ( · , · ; θ ) is the mutual information function and ¯ I is a compression constraint. We can equivalently express this optimization problem as maximizing the objective

where β is a Lagrange multiplier that trades off learning a latent representation that is maximally informative about the target Y with learning a representation that is maximally compressive about the input X [2].

3.2 A Multi-Modal Information Bottleneck Principle

Unfortunately, for VLPMs, the loss function above is not suitable since we wish to learn interpretable latent representations using only text and vision inputs without relying on task-specific targets Y that may not be available or are expensive to obtain. To formulate a multi-modal information bottleneck principle for VLPMs, we need to develop an optimization objective that is more akin to optimization objectives for self-supervised methods for image-text representation learning that only use (text, image) pairs [21, 18].

This learning problem fundamentally differs from supervised attribution map learning for unimodal tasks. For example, we may have an image of a bear, X bear, and a corresponding label, Y bear = 'bear' . For a unimodal classification task, we can simply maximize I ( Y bear ; Z bear ; θ ) -βI ( X bear ; Z bear ; θ ) with respect to θ , where Z bear is the latent representation of X bear .

In contrast, in image-text representation learning, we typically have text descriptions, such as 'This is a picture of a bear' ( L ′ bear ) instead of labels [21]. In this setting, both X bear and L ′ bear are 'inputs' without a pre-defined corresponding label. To obtain a task-agnostic image-text representation independent from any task-specific ground-truth labels, we would like to use both input modalities and define a multi-modal information bottleneck principle and whereas the outputs are closely dependent on the specific downstream task. This requires defining an alternative to the 'fitting term' I ( Z, Y ; θ ) of the conventional information bottleneck objective.

Fortunately, there is a natural proxy for the relevance of information in multi-modal data. If image and text inputs are related (e.g., text that describes the image), a good image encoding should contain information about the text and vice versa. Based on this intuition, we can express a multi-modal information bottleneck objective for X m with m ∈ M = { modality1 , modality2 } , as

where m ′ = M\ m is the complement of m , and E m ′ is embedding of modality m ′ . Next, we will show how to use this multi-modal variant of the information bottleneck principle for attribution.

3.3 A Multi-Modal Information Bottleneck for Attribution

To compute attribution maps for image and text data without access to task-specific labels, we will define an information bottleneck attribution method for multi-modal data.

To restrict the information flow in the latent representation with a simple parametric encoder, we adapt the masking approach in Schulz et al. [24]. For clarity and brevity, we henceforth represent the V m × W m -dimensional latent representation Z m in its vectorized form in R J with J ˙ = V m · W m . Assuming independence across latent representation dimensions, we then define

p Z m | X m ( z m | x m ; θ m ) = N ( z m ; h m ( x m ; λ m ) ⊙ f m m ( x m ) , σ m diag[( 1 J -h m ( x m ; λ m )) ]) , (4) where for a pair of modalities M ˙ = { image , text } with m ∈ M , θ m ˙ = { λ m , σ m , ℓ m } are parameters, h m ( x m ; λ m ) ∈ R J is a mapping parameterized by λ m , f ℓ m ( X m ) ∈ R J is the vectorized output of the ℓ m th layer of modality-specific neural network embedding f m , σ 2 m ∈ R > 0 is a hyperparameter, diag[ · ] represents an operator that transforms a vector into a diagonal matrix by placing the vector's elements along the main diagonal, 1 J ∈ R J is an all-ones vector, and ⊙ is the Hadamard product. To avoid overloading notation, we will drop the subscript in probability density functions except when needed for clarity. Based on Equation (4), we can express the stochastic latent representations as

where ε ∼ N (0 , I J ) . From this reparameterization, we can see that [ h m ( x m ; λ m )] i = 1 for i ∈ { 1 , ..., J } means that no noise is added at index i , so [ Z m ] i will be the same as the original f ℓ m ( x m ) i , whereas [ h m ( x m ; λ m )] i = 0 means that [ Z m ] i will be pure noise.

We can now express the multi-modal information bottleneck attribution (M2IB) objectives as

which we can optimize with respect to the modality-specific sets of parameters λ image and λ text , respectively. { β image , σ image , ℓ image } and { β text , σ text , ℓ text } are each sets of hyperparameters.

3.4 A Variational Objective for Multi-Modal Information Bottleneck Attribution

To obtain tractable optimization objectives, we use a variational approximation. First, we note that I ( Z m , X m ; θ m ) = E p Xm [ D KL ( p Z m | X m ( · | X m ; θ m ) ∥ p Z m ( · ; θ m ))] , where Z m | X m ; θ m can be sampled empirically whereas p ( z m ; θ m ) does not have an analytic expression because the integral p ( z m ; θ m ) = ∫ p ( z m | x m ; θ m ) p ( x m ) d x m is intractable. To address this intractability, we approximate p ( z m ) by q ( z m ) ˙ = N ( z m ; 0 , I J ) . This approximation leads to the upper bound

Next, while the unimodal information bottleneck attribution objective uses ground-truth labels to compute the 'fit term' in the objective, the multi-modal information bottleneck attribution objectives require computing the mutual information between the aligned stochastic embeddings,

which is not in general tractable. To obtain an analytically tractable variational objective, we approximate the intractable p ( e m ′ | z m ) by a variational distribution q ( e m ′ | z m ) ˙ = N ( e m ′ ; g m ( z m ) , I K ) , where g m is a mapping that aligns the latent representation of modality m with E m ′ and K is the dimension of the embedding, and get

With this approximation, we can obtain a tractable variational optimization objective by sampling X m and E m ′ from the empirical distribution

where f m ′ ( x m ′ ) is the embedding input x m ′ under the VLPM. With these approximations, we can now state the full variational objective, which is given by

The derivation of this objective has closely followed the steps in [2]. In practice, the objective can be computed using the empirical data distribution so that

Final Variational Optimization Objective. Finally, we assume that h m ( x ( n ) m ; λ m ) ˙ = λ ( n ) m (i.e., each input point has its own set of attribution parameters), that g m is the mapping defined by the postbottleneck layers of a VLPM for modality m , and that for each evaluation point the final embeddings for each modality get normalized across the embedding dimensions (i.e., both mapping, g m and f m ′ , contain embedding normalization transformations). For normalized g m ( z m ) and e m ′ , the log of the Gaussian probability density q ( f m ′ ( x m ′ ) | g m ( z m )) simplifies and is proportional to the cosine similarity between f m ′ ( x m ′ ) and g m ( z m ) , giving the final optimization objective

where S cosine ( · , · ) is the cosine similarity function. For gradient estimation during optimization, the remaining integrals can be estimated using simple Monte Carlo estimation and reparameterization gradients. For θ m = { λ m , σ m , ℓ m } , the objective function is maximized with respect to λ m independently for each modality , and β m , σ m , and ℓ m are modality-specifc hyperparameters.

4 Empirical Evaluation

We evaluate the proposed attribution method using CLIP [21] on four image-caption datasets, including widely-used image captioning datasets and medical datasets. Our main datasets are (i) Conceptual Captions [26] consisting of diverse images and captions from the web, and (ii) MS-CXR (Local Alignment Chest X-ray dataset; [3]), which contains chest X-rays and texts describing radiological findings, complementing MIMIC-CXR (MIMIC Chest X-ray; [13]) by improving the bounding boxes and captions. In addition, we also include some qualitative examples from the following radiology and remote sensory datasets to show the potential application of our model in safety-critical domains. Namely, we have (iii) ROCO(Radiology Objects in COntext; Pelka et al. [19]) that includes radiology image-caption pairs from the open-access biomedical literature database PubMed Central, and (iv) RSICD (Remote Sensing Image Captioning Dataset; Lu et al. [16]), which collects remote sensing images from web map services including Google Earth and provides corresponding captions.

4.1 Experiment Setup

For all experiments, we use a pretrained CLIP model with ViT-B/32 [7] as the image encoder and a 12-layer self-attention transformer as the text encoder. For Conceptual Captions, we use the pretrained weights of openai/clip-vit-base-patch32 . 2 For MS-CXR, we use CXR-RePaiR [8] which is CLIP finetuned on radiology datasets, and compare the impact of finetuning in Section 4.5. For each {image, caption} pair, we insert an information bottleneck into the given layer of the text encoder and image encoder of CLIP separately, then train the bottleneck using the same setup as the Per-Sample Bottleneck of original IBA [24], which duplicates a single sample for 10 times to stabilize training and runs 10 iterations using the Adam optimizer with a learning rate of 1. Experiments show no significant difference between different learning rates and more training steps. We conduct a hyper-parameter tuning on the index of the layer l , the scaling factor β , and the variance σ 2 . For a discussion of these hyperparameters in the multi-modal information bottleneck objective, see Appendix A.

a

manis

lying

onthe

floor

an

elephant

isabove aman

an

elephant

iS

lifting

itsleg

white

planes

on the ground

white

planes

onthe

punob

4.2 Qualitative Results

We qualitatively compare our method with 5 widely used attribution methods, including gradientbased GradCAM [25] and Saliency method [28], perturbation-based Kernel SHAP [17] and RISE [20], and transformer-specific method [6]. We show one image-caption example and two radiology examples in Figure 1 and provide more qualitative comparisons in the Appendix B. As shown, our method is able to capture all relevant objects appearing in both modalities, while other methods tend to focus on one major object. As illustrated in Figure 2a, the highlighted areas in the image change according to different inputs, and our method can capture complicated relationships between image and text when involving multiple objects. Our method is also able to detect multiple occurrences of the same object in the image, as illustrated in Figure 2b.

4.3 Localization Test

We quantitatively measure the effectiveness of our proposed attribution method by evaluating its accuracy in zero-shot detection for images. We binarize the saliency map such that the areas with scores higher than the threshold (75%) are assigned 1 while the rest are assigned 0. We denote the resulting binary map as S pred . We also construct a ground-truth binary map, S gt , using the bounding boxes provided by MS-CXR [3], where the region inside the bounding boxes is assigned to 1 while the outside is assigned to 0. Note that some samples have multiple bounding boxes, and we consider all of them to test the method's multi-occurrence detection ability. Then, we calculate the IoU (Intersection over Union) of S pred and S gt . Namely, for images with a height of n and a width of m , the score is calculated by where ✶ is the indicator function, ∧ is the logical AND and ∨ is the logical OR operator. We found that M2IB attribution attains an average IoU of 22.59% for this zero-shot detection task, outperforming all baseline models, as shown in Table 1. Recognizing that a localization score of 22.59% appears somewhat low in absolute terms, we briefly note that there are two potential causes that could lead to the low absolute values in the localization scores: (i) M2IB attribution indeed generates segmentation instead of bounding boxes, so evaluation by bounding boxes would underestimate the quality of the saliency map. (ii) The model under evaluation [8] is not finetuned for detection and may only have learned a coarse-grained relationship between X-rays and medical notes.

4.4 Degradation Tests

While the localization test shows that M2IB attribution may be a promising zero-shot detection and segmentation tool, the localization test may underestimate the accuracy of attribution since even a perfect attribution map can produce a low localization score if the (finetuned) VLPM under evaluation is poor at extracting useful information-which is very likely for challenging specialized tasks like chest X-ray classification.

To get a more fine-grained picture of the usefulness of M2IB, we use three additional evaluation metrics to compare M2IB to competitive baselines. The underlying idea of all three evaluations is that removing features with high attribution scores should decrease the performance, while discarding features with low attribution scores can improve the performance as noisy information is ignored. We randomly sample 2,000 image-text pairs from Conceptual Captions and 500 image-text pairs from MS-CXR. For each dataset, we perform ten experiments for each evaluation metric (five for ROAR+) and report the average score with the standard error in Table 1.

Original

Saliency Map

Attribution Weighted Image

Corrupted (binarized M)

a man holds what is believed to be some of the debris that caused damage to vehicles .

a <B> <B> what is believed <B> be some of the <B> <B> caused <B> to <B> <B>

<B> man holds <B> <B> <B> to <B> <B> <B> <B> debris that <B> damage <B> vehicles .

amanholds

whatisbelieved

tobesomeofthedebristhat

causeddamagetovehicles

Drop in Confidence [4]. An ideal attribution method should only assign high scores to important features, thus we should not observe a drop in performance if only the high-attribution parts are allowed in the input. For images, we use point-wise multiplication of the saliency map and the image input. Since scaling token ids is meaningless, we use binarization similar to [33] where only tokens with attribution scores in the top 50% are kept. We provide an example of distilled image and text in Figure 3. Formally, we define this score by

where o i is the cosine similarity of features of original images and texts, and s i is the new cosine similarity when one modality is distilled according to the attribution. The lower this metric is, the better the attribution method is. This metric is implemented in the pytorch-gradcam repository. 3

Increase in Confidence [4]. Similarly, removing noisy information in the input might increase the model's confidence. We compute

where ✶ is the indicator function and the definition of o i and s i is the same as above. Higher values indicate better performance. This metric is also implemented in the pytorch-gradcam repository. 3

Remove and Retrain + (ROAR+, an Extension of ROAR [11]). We finetune the base model on the degraded images and texts where the most important parts are replaced by uninformative values (i.e., channel means of images or padding tokens for texts, see Figure 3) and evaluated on a validation set of original inputs. 4 If the attribution method is accurate, a sharp decrease in performance is expected because all useful features are removed, and the model cannot learn anything relevant from the degraded data. We split the testing dataset into 80% training data and 20% validation data. We use the same contrastive loss as used for pretraining CLIP and define the score by ( l c -l o ) /l o , where l o is the validation loss of retraining using the original data, and l c is the validation loss when retraining with corrupted data. We repeat the process five times and report the average score.

The results are summarized in Table 1. M2IB attribution outperforms baseline models in almost all numerical metrics, except for perturbation-based methods, which achieve better Increase/Drop in Confidence scores for texts. Perturbation-based methods perform better for short text because they can scan all possible binary masks on the text and then find the optimal one with the highest confidence score. However, this kind of method is very computationally expensive. We use 2,048 masks for image and 256 masks for text in RISE, where each mask of each modality requires one forward pass to get the confidence score, resulting in 2.3k forward passes (7.8s on RTX8000 with a batch size of 256). In contrast, M2IB attribution only requires 100 forward passes and takes 1.2s for one image-text pair.

In general, removing pixels or tokens with lower attribution scores using M2IB attribution generally increases the mutual information with the other modality, while masking by our attribution map generally decreases the relevance with the other modality and makes the model perform worse when retraining on the corrupted data. This confirms that our method generates useful attribution maps.

4.5 Sanity Check

We conduct a sanity check to ensure our method is, in fact, sensitive to model parameters. We follow the sanity check procedure proposed by Adebayo et al. [1], where parameters in the model are randomized starting from the last to the first layer. As shown in Figure 4, M2IB passes the sanity check as the attribution scores of image pixels and text tokens change as the model weights change. Our method also produces more accurate saliency maps for finetuned models compared to pretrained models, which further confirms that the resulting attribution can successfully reflect the quality of the model. Since we insert the information bottleneck after a selected layer (layer 9 in this case), the randomization of this and previous layers appears to have a larger influence on the output.

4.6 Error Analysis and Limitations

We notice that our proposed attribution method generally performs well on text but sometimes shows less satisfying performance on images. By inspecting the qualitative examples, we observe that M2IB sometimes fails to detect the entire relevant regions in images. As shown in the fourth ('sea' example) and fifth ('bridge' example) rows in Figure 7, our method only highlights a fraction of the object in the image, although it should include the whole object. This is probably due to the fact that the model under evaluation only relies on a few patterns in the image to make its prediction. Increasing the relative importance of the fitting term (i.e., using smaller β ) enlarges the highlighted area. However, we don't recommend using extreme β because it will break the balance between fitting and compression and thus make the information bottleneck unable to squeeze information.

We also note that M2IB is sensitive to the choice of hyperparameters. As shown in Figure 5, different combinations of hyperparameters will generate different saliency maps. We show how to use the ROAR+ score to systematically select the optimal hyperparameters and also provide visualization to illustrate the effect of different hyperparameters in Appendix A. Since there is no convention on evaluating the attribution method, we suggest considering various evaluation metrics, visualization of examples, and the goal of the attribution task when choosing hyperparameters. We emphasize that M2IB should be used with caution since attributing the success or failure of a model solely to a set of features can be overly simplistic, and different attribution methods might lead to different results.

Pretrained

Projection

Layer 9

Layer 6

Layer 3

Random

Finetuned

Ours

GradCAM

bilateral

multi

bilateral

multi

bilateral

multi

bilateral

bilateral

multi

bilateral

bilateral

n

focalareas

of

focalareas

of

focal

areasof

focal

areas

f

focalareas

of

focal

areas

of

focalareas

f

consolidation

consolidation

consolidation

consolidation

consolidation

consolidation

consolidation

bilateral

multi

bilateral

multi

bilateral

bilateral

multi

bilateral

multi

bilateral

multi

bilateral

multi

focalareas

of

focalareas

of

focalareas

focal

areasof

focalareasof

focalareas

focal

areas

of

consolidation

consolidation

consolidation

consolidation

consolidation

consolidation

consolidation

bilateral

multi

bilateral

multi

bilateral

bilateral

multi

bilateral

multi

bilateral

multi

bilateral

focal..

areasof

focalareas

focalareasof

focalareasof

focalareas

of

focalareas

of

focalareas

of

consolidation

consolidation

consolidation

consolidation

consolidation

consolidation

consolidation

5 Discussion

We developed multi-modal information bottleneck (M2IB) attribution, an information-theoretic approach to multi-modal attribution mapping. We used CLIP-ViT-B/32 in our experiments, but M2IB attribution can be directly applied to CLIP with alternative neural network architectures and is compatible with any VLPM for which the features of all modalities are projected into a shared embedding space-a commonly used approach in state-of-the-art multi-modal models.

Going beyond vision and language modalities, Girdhar et al. [9] recently proposed a new multimodal model, ImageBind, which aligns embeddings of five modalities to image embeddings through contrastive learning on pairs of images with each modality and uses a similar architecture as CLIP, where the encoder for each modality is a transformer. We applied M2IB attribution to pairs of six different modalities using ImageBind-which required minimal implementation steps-and provided qualitative examples in Appendix C to illustrate that M2IB attribution can be applied successfully to more than just vision and text modalities.

In this paper, we provided exhaustive empirical evidence that M2IB increases attribution accuracy and improves the interpretability of state-of-the-art vision-language pretrained models. We hope that this work will encourage future research into multi-modal information-theoretic attribution methods that can help improve the interpretability and trustworthiness of VLPMs and allow them to be applied to safety-critical domains where interpretability is essential.

Acknowledgments

This work was supported in part through the NYU High-Performance Computing services as well as by NSF CAREER IIS-2145492, NSF I-DISRE 193471, NSF IIS-1910266, BigHat Biosciences, Capital One, and an Amazon Research Award.

References

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. CoRR , abs/1810.03292, 2018.

Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations , 2017.

Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, Hoifung Poon, and Ozan Oktay. Making the most of text semantics to improve biomedical vision-language processing. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 , pages 1-21, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-20059-5.

Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 839-847, 2018. doi: 10 . 1109/WACV . 2018 . 00097.

Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. CoRR , abs/2012.09838, 2020.

Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 397-406, October 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Mark Endo, Rayan Krishnan, Viswesh Krishna, Andrew Y. Ng, and Pranav Rajpurkar. Retrievalbased chest x-ray report generation using a pre-trained contrastive language-image model. In Proceedings of Machine Learning for Health , volume 158 of Proceedings of Machine Learning Research , pages 209-219, 2021.

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One embedding space to bind them all, 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016. doi: 10 . 1109/CVPR . 2016 . 90.

Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Zhiying Jiang, Raphael Tang, Ji Xin, and Jimmy Lin. Inserting information bottlenecks for attribution in transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 3850-3857, Online, November 2020. Association for Computational Linguistics. doi: 10 . 18653/v1/2020 . findings-emnlp . 343.

Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Roger G. Mark, and Steven Horng. MIMIC-CXR: A large publicly available database of labeled chest radiographs. CoRR , abs/1901.07042, 2019.

Shuying Liu and Weihong Deng. Very deep convolutional neural network based image classification using small training sample size. In 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR) , pages 730-734, 2015. doi: 10 . 1109/ACPR . 2015 . 7486599.

Siqu Long, Feiqi Cao, Soyeon Caren Han, and Haiqin Yang. Vision-and-language pretrained models: A survey. In Luc De Raedt and Luc De Raedt, editors, Proceedings of the 31st International Joint Conference on Artificial Intelligence, IJCAI 2022 , IJCAI International Joint Conference on Artificial Intelligence, pages 5530-5537. International Joint Conferences on Artificial Intelligence, 2022. doi: 10 . 24963/IJCAI . 2022/773. 31st International Joint Conference on Artificial Intelligence, IJCAI 2022 ; Conference date: 23-07-2022 Through 29-07-2022.

Xiaoqiang Lu, Binqiang Wang, Xiangtao Zheng, and Xuelong Li. Exploring models and data for remote sensing image caption generation. IEEE Transactions on Geoscience and Remote Sensing , 56(4):2183-2195, 2017. doi: 10 . 1109/TGRS . 2017 . 2776321.

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Norman Mu, Alexander Kirillov, David A. Wagner, and Saining Xie. SLIP: self-supervision meets language-image pre-training. CoRR , abs/2112.12750, 2021.

Obioma Pelka, Sven Koitka, Johannes Rückert, Felix Nensa, and C. Friedrich. Radiology Objects in cOntext (ROCO): A multimodal image dataset. In CVII-STENT/LABELS@MICCAI , 2018.

Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference (BMVC) , 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR , abs/2103.00020, 2021.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016 , pages 1135-1144, 2016.

Yao Rong, Tobias Leemann, Vadim Borisov, Gjergji Kasneci, and Enkelejda Kasneci. A consistent and efficient evaluation strategy for attribution methods. In Proceedings of the 39th International Conference on Machine Learning , pages 18770-18795. PMLR, 2022.

Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. In International Conference on Learning Representations , 2020.

Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-CAM: Why did you say that? visual explanations from deep networks via gradient-based localization. CoRR , abs/1610.02391, 2016.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10 . 18653/v1/P18-1238.

Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can CLIP benefit vision-and-language tasks? In International Conference on Learning Representations , 2022.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations , 2014.

Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for Simplicity: The All Convolutional Net. arXiv e-prints , art. arXiv:1412.6806, December 2014. doi: 10 . 48550/arXiv . 1412 . 6806.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 3319-3328. PMLR, 06-11 Aug 2017.

Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw) , pages 1-5. IEEE, 2015.

Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, 2000.

Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-CAM: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 24-25, 2020.

Table of Contents

A

Experimental Details and Further Experimental Results

B

C

Attribution Maps for Additional Examples

Extension to Additional Modalities With ImageBind

Appendix

15

17

19

Appendix A Experimental Details and Further Experimental Results

β controls the relative importance of the compression term. The larger the β is, the less information is allowed to flow through this layer. As shown in Figure 5a, too large and too small β generates similar attribution maps in terms of relative importance. However, too small β allows nearly everything through the bottleneck, whereas too large β nearly discards everything.

σ controls the values of the noise added to the intermediate representations. When σ is very small, the values of the noise will be close to 0, thus having minimal impact on the intermediate representations. This effect is similar to the situation when β is very small and IB will add almost no noise (Figure 5b). σ also directly affects the compression term as smaller σ will lead to higher KL divergence. Thus, σ and β are correlated with each other and we perform a grid search to find the best combination.

Layer ℓ where the information bottleneck is inserted also impacts the attribution. Inserting the bottleneck too early will prevent the model from learning informative features while inserting the bottleneck too late reduces the impact of the bottleneck (Figure 5c). We also observe that the attribution of texts is usually more stable than images.

These hyperparameters can be chosen according to numerical metrics mentioned in Section 4.4. We randomly sample 500 image-text pairs from Conceptual Captions and 500 image-text pairs from MS-CXR, which is ensured to not overlap with the test set in Section 4. We then perform a grid search for the best combination of β = { 1 , 0 . 1 , 0 . 01 } and σ 2 = { 1 , 0 . 1 , 0 . 01 } and ℓ = { 7 , 8 , 9 } (indexing from 0), and find the best combination of hyperparameters is shown in Table 2.

Table 2: Hyperparameter Tuning Results for ℓ , β and σ : We calculate the ROAR+ score for different combinations for 3 runs and report the average. For each table, the highest score is in bold and indicates the performance is optimal for this set of hyperparameters. For texts of Conceptual Captions, we select ℓ = 9, β = 0.1 and σ = 1. Since ROAR+ uses binarized saliency maps (75% threshold for image pixels and 50% threshold for text tokens, it mainly focuses on the features with high attribution scores and neglects the change in the attribution for less important features. Thus, sometimes hyperparameters with slightly lower ROAR+ scores might generate more visually appealing results as in Figure 5, though the numerical results are consistent with qualitative examples in general.

𝛽 = 1

𝛽 = 10

𝛽 = 0.1

𝛽 = 0.001

𝛽 = 0.01

(a) Impact of Scaling Factor

β

. Higher

β

means higher weight of the compression term.

&

"

=1

&

"

=10

&

"

=0.1

&

"

=0.001

&

"

=0.01

(b) Impact of noise variance

σ

2

. Smaller

σ

2

means lower impact of compression.

( = 9

( = 10

( = 8

( = 6

( = 7

consolidation

in

consolidation

in

consolidation

in

consolidation

in

consolidation

in

the left

lung

the left

lung

the left

lung

the left

lung

the left

lung

funnyanimals

of

funnyanimals

of

funny

animals

of

funny

animals

of

funnyanimals

of

the week,animal

the week,animal

the week,animal

the week，animal

the week,animal

pictures

pictures

pictures

pictures

pictures

consolidation

consolidation

in

consolidation

in

consolidation

in

consolidation

in

theleft

lung

the left

lung

the left

lung

the left

lung

the left

lung

funnyanimals

of

funnyanimals

of

funnyanimals

of

funny animals

of

funnyanimals

of

the week,animal

the week,animal

the week, animal

the week，animal

the week，animal

pictures

pictures

pictures

pictures

pictures

consolidation

in

consolidation

in

consolidation

in

consolidation

in

consolidation

in

theleft

lung

the left

lung

the left

lung

the left

lung

the left

lung

funny

animals

of

funny

animals

of

funny animals

of

funnyanimals

of

funnyanimals

of

theweek，animal

theweek，animal

theweek，animal

the week,animal

theweek,animal

pictures

pictures

pictures

pictures

pictures

(c) Impact of the Layer where IB is inserted

ℓ

. Larger

l

means deeper layer in the network.

Appendix B Attribution Maps for Additional Examples

GradCAM

Saliency

RISE

Ours

Original

KernelSHAP

image - relative importance

text - relative importance

ome gal

mildcardi

omegal

mildcardi

omegal

mildcardi

omegal

mildcardi

ome gal

mildcardi

ome gal

mildcardi

omegal

ynoted

ynoted

ynoted

noted

ynoted

noted

ynoted

marked cardi

ome

marked

cardiome

markedcardiome

markedcardiome

markedcardiome

marked cardi

ome

markedcardiome

galyis present

galyis

present

galyispresent

galyispresent

galyispresent

galyis present

galis present

local

local

local

local

local

local

consolidation

in

consolidation

回

consolidation

in

consolidation

in

in

consolidation

in

in

the left

midlung

the left

the left

the left

the left

theleft

midlung

theleft

left

ar

left

perihil

ar

left

perihil

ar

left

peri.hil

ar

left

perihil

ar

left

perihil

ar

left

perihil

ar

consolidation

consolidation

consolidation

consolidation

consolidation

consolidation

consolidation

versus mass

versus mass

versus mass

versusmass

versusmass

versus

mass

versus mass

patchy

patchy.

patchy

patchy

patchy

patchy

patchy

consolidation

in

consolidation

in

consolidation

in

consolidation

consolidation

in

consolidation

in

in

the mid left

lung

the mid left

lung

the midleft

lung

the midleft

lung

the mid left

lung

the midleft

lung

the mid left

lung

middleand lower

middle

andlower

middleandlower

middleandlower

middleandlower

middleandlower

middle

andlower

lobe pneumonia

lobe

pneumonia

lobe

pneumonia

ome gal

mild

cardi

ome gal

cardi

omegal

mild

cardi

ome gal

mild

cardi

ome gal

mild

cardi

omegal

cardi

omegal

y

y

y

y

middleand lower

middleandlower

middleandlower

middleandlower

middleandlower

middleandlower

middle

andlower

lobe pneumonia

obepneumonia

lobe

pneumonia

00

0.2

0.4

0.6

80

1.0

0.0

0.2

0.4

0.6

80

1.0

GradCAM

Saliency

RISE

Ours

Original

KernelSHAP

image - relative importance

text - relative importance

grandfather

and

grandfather

and

grandfather

and

grandfather

and

grandfather

and

grandfather

and

grandfather

and

grandson  setting

grandson setting

grandsonsetting

grandson setting

grandsonsetting

grandson setting

grandsonsetting

a chess pieceon

achesspieceon

achesspieceon

achesspieceon

achesspieceon

achesspieceon

achesspieceon

the chessboard

the chess board

the chess

board

the chess board

the chessboard

the chess board

the chess board

huene

scarecrow

ina

scarecrow

ina

scarecrow

同a

scarecrow

na

scarecrow

ina

scarecrow

na

field

ofcrops

field

of

crops

field

of

crops

field

ofcrops

of

crops

field

of

crops

field

ofcrops

person

andgrant

personandgrant

personandgrant

personandgrant

personandgrant

personandgrant

personandgrant

infront

christmas

ofthe

tree

christmas

infront

ofthe

tree

infront

christmas

ofthe

tree

infront

christmas

ofthe

tree

infront

christmas

ofthe

tree

infront

christmas

ofthe

tree

infront

christmas

ofthe

tree

warmevening

on

warmevening

on

warm eveningon

warmevening

on

thesea

thesea

thesea

thesea

thesea

thesea

thesea

abridge

girlwalking

over

abridge

girl walking

over

abridge

girl

walking

over

abridge

walking

over

walking

over

abridge

girl

walking

over

abridge

girl

walking

over

birthday

cake and

birthday

cakeand

birthday

cake

and

birthday

cake

and

birthday

cake

and

birthday

cake and

birthday

cakeand

ona

s

ona

ona

ona

ona

s

ona

ona

woodenstool

wooden stool

wooden stool

wooden stool

wooden stool

woodenstool

woodenstool

cherryfalling

cherryfalling

cherry

falling

cherry

falling

cherry

falling

cherry

falling

cherry

falling

intothemilk

into'themilk

intothemilk

intothe milk

into

themilk

into

themilk

intothemilk

still

life

with

still

life

with

still

life

with

still

life

with

still

life

with

still...

life

with

still.

life

with

weddingringsand

a daisy

weddingrings

and

weddingrings

and

wedding

ringsand

weddingrings

and

wedding

ringsand

wedding

ringsand

00

0.2

0.6

0.8

1.0

00

80

1.0

𝛽 = 1

𝛽 = 1𝑒 - 5

leixe

onal

view).

axial

mri

（cor onalview ）.

axial

mri ( cor onal

view）.

axial

mri( cor

onal

view.

axial

onal

view.

axial

euo

view).

tscan showing a large

left

ct scan showing a large left

scan showing a largeleft

scan showing a large

left

Ctscan showing alarge

left

tscan showing a large

left

ovarian

S

ovarian

cy st

ovarian

cy

ovarian

ovarian

SK

ovarian

cyst

Appendix C Extension to Additional Modalities With ImageBind

Our method can be easily extended to representation learning of modalities other than image and text, as long as features of different modalities are projected into a shared embedding space. We use ImageBind [9] as an example to showcase the effectiveness of our method in interpreting audio-image and audio-text representation learning. Since ImageBind aligns other modalities (audio, sensors that record depth (3D), thermal (infrared radiation), and inertial measurement units (IMU)) embedding to image embeddings through contrastive learning, the embeddings of all modalities are in one shared embedding space. Thus, we can insert the information bottleneck into certain layers of the feature encoders of ImageBind respectively and adopt the M2IB objective similar to Equation (6) and Equation (7).

(a) Audio: Dog Barking

(b) Audio: Cat Meowing

(c) Audio: Dog Barking

(d) Audio: Cat Meowing

a

cat

is

me

owing

or

a

bop

is

barking

cat

is

me

owing

or

a

6op

is

barking

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nVisual-language models (VLMs) are increasingly used for multiple vision-language problems like object recognition, image classification, image captioning, etc. Formally, these models are trained in a self-supervised fashion using large datasets comprising millions of image-text pairs, such that they learn generic image and text representations during the training, which can be fine-tuned for different downstream tasks. However, VLMs are intrinsically very complex and are hard to interpret. In this work, the authors use CLIP, a state-of-the-art visual-language model, and propose a multi-modal information bottleneck (M2IB) objective that compresses irrelevant and noisy features and retains visual and text features relevant to the underlying VLM without requiring ground truth labels. Further, the authors show the effectiveness of M2IB in generating attribution maps with increased attribution accuracy across several high-stakes datasets, such as medical diagnosis.\n\nSTRENGTHS:\n1. One of the main highlights of the work is that it does not require the ground-truth label for generating attribution maps and approximates its impact using mutual information between the image and text representations.\n\n2. In contrast to existing works that have focused on a single modality (images or texts) for supervised learning, the authors propose applying the information bottleneck principle to aid our understanding of the inner mechanisms of VLMs using both image and textual inputs.\n\n3. The proposed solution is clear and appears to be a simple extension of the information bottleneck paper referred to in the paper.\n\nWEAKNESSES:\n1. The hypothesis that a good image encoding should contain information about the text, while a good text encoding should include information about the image is unclear. This is fundamentally questionable as the model never learns a unique mapping between the **Z_image** and **Z_test** representations, i.e., the representations in VLMs are learned in a compact manner and it may be difficult to disentangle representations between similar concepts, say **Z_bear_image** and **Z_panda_text**.\n\n2. The proposed technique comprises a large set of hyperparameters for tuning M2IB and generating attribution maps as explanations and the authors do not provide a clear analysis of how to tune or fix them for different models and datasets.\n\n3. In Lines 214-216, the authors mention that \"for each {image, caption} pair, we insert an information bottleneck into the given layer of the text encoder and image encoder of CLIP separately, then train the bottleneck using the same setup as the Per-Sample Bottleneck of original IBA\". This sounds overly computationally expensive as we need to tune all the hyperparameters for each <image, caption> pair!\n\n4. In Lines 228-229, the authors state that \"our method is able to capture all relevant objects appearing in both modalities, while other methods tend to focus on one major object\" --- is that the aim of the results? It is totally possible that the model is only using limited information to learn a particular image-caption pair. I can see why it's logically unintuitive to humans, but that can totally be what the model is actually doing. A good check would be how different the mutual information on using the attribution generated using CAM-based methods and the proposed method.\n\n5. Minor comment: the paper has several typos and the writing can be improved. For example, \"dind\" in Line 54, \"emoirical\" in Line 183, missing references CXR-RePaiR [] in Line 249, and \"test encoder of CLIP\" in Figure 4 caption. Also, the paper discusses a lot about future works in Section 5. It would be better to talk about the limitations of the work instead.\n\nQUESTIONS:\n1. The authors mention that they use a \"variational approximation upper bound to estimate the information in the target layer and aim to minimize this upper bound during training with respect to embedding-specific attribution parameters.\" Does this mean that they need access to the training pipeline to generate explanations for an intermediate target layer?\n\n2. The authors do propose M2IB that does not require ground-truth labels **y** and show how we can generate attribution maps without the knowledge of **y**. However, the authors never discuss the possibility of using the zero-shot prediction of the VLM and making the problem simpler. It would be interesting to discuss the utility of zero-shot predictions as VLMs achieve decent zero-shot performance.\n\n3. Any intuition behind how the layer index $m$ affects the resulting qualitative and quantitative performance of the attribution maps.\n\n4. Should we even expect high scores in Eqn. 13? Why do we hypothesize that a VLM will look at all the pixels in the bounding box to make its respective decision?\n\n5. Any comments on why there is no attribution to the vehicle in Figure 3?\n\nLIMITATIONS:\nPlease refer to the weakness and open questions in the above sections. \n\nSuggestion: The authors extensively discuss the limitation of the work in the supplementary document and do not discuss anything in the main draft. It would be good from a reading perspective to include certain aspects of the limitation in the main document.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to enhance the interpretability of vision-language pretrained models like CLIP. To achieve this, the authors propose a multi-modal information bottleneck (M2IB) objective that reduces irrelevant and noisy information while retaining important visual and textual features. M2IB is used for attribution analysis, increasing attribution accuracy and interpretability. M2IB audits representations when only multiple modalities are available without ground truth data. The effectiveness of M2IB is demonstrated using CLIP as an example, showing that it outperforms CAM-based methods both qualitatively and quantitatively.\n\nSTRENGTHS:\n1. The paper is tackling a significantly important research question: the interpretability of vision-language pretrained models;\n\n2. The paper is well-structured, with clear organization and sufficient methodological details provided, enabling readers to comprehend the proposed method;\n\n3. The evaluation is thoroughly conducted. For example, in Table 1, numerous faithfulness metrics, including Confidence Drop/Increase, ROAD, and ROAR, are employed for assessment.\n\nWEAKNESSES:\nI do not have any particular weaknesses to highlight; however, I still have some questions that I hope the authors can clarify. Please see the questions section.\n\nQUESTIONS:\n1. M2IB has been qualitatively and quantitatively compared with CAM-based attribution methods. Could the authors also compare it with some mask-based explanation methods like [RISE](https://arxiv.org/abs/1806.07421)? In terms of explanation faithfulness, mask-based explanation methods usually perform better.\n\n\n2. Could the authors discuss the difference between M2IB and this Neurips 2022 paper [Fine-Grained Semantically Aligned Vision-Language Pre-Training]( https://arxiv.org/abs/2208.02515)? Note that I am not one of the authors of this paper and do not have any relationship with them but find out that both M2IB and that work tries to increase the relevance between image and text feature.\n\nLIMITATIONS:\nI do not have any particular limitations to highlight.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a feature attribution method for vision language models. The method is based on previous works on information bottleneck attributions.  It incorporates Gaussian moment-matching estimator techniques to make the objective function tractable. The experimental results show that the method outperforms CAM-based methods quantitatively and qualitatively.\n\nSTRENGTHS:\n1. Applying the information bottleneck attribution approach to multi-modal settings is novel. \n2. The evaluation used diverse evaluation metrics such as Conf. Drop, Incr., ROAR, and ROAR+. It even included a sanity check\n\nWEAKNESSES:\nThe baselines included in experiments are very limited; they did not include any explanation methods developed for ViTs. The baselines were GradCAM and its variants only. GradCAM is not a good method for explaining ViTs. In transformer architectures, since the information can go across patches (long-range attention), locational information is not preserved. This violates the assumption of GradCAM that the location in the penultimate layer can be mapped to the input layer. I would recommend comparing the methods with recent state-of-the-art methods developed for ViTs. I found a GitHub repo that well listed recent ViT papers that would be helpful for curating the methods (https://github.com/cmhungsteve/Awesome-Transformer-Attention)\n\nQUESTIONS:\n1. The legend for the attribution maps is missing. The legend should show which color means which value. Why do only image attribution maps have red regions?\n2. The approximation technique assumes each dimension follows Gaussian distributions. Could you please provide any evidence on this?\n3. In Line 236, the threshold of 75% seems arbitrary. Also, I wonder if 17.6% of the average IoU is high enough to show the effectiveness of the method.\n\nTypo\n1. Line 54 dind -> find\n2. Line 139 indepdent -> independent\n3. Line 249 missing citations.\n\nLIMITATIONS:\nThe method involves the modification to the training process; The pretrained models should be finetuned. This will limit the utility of the method. Furthermore, recent studies have shown that the pretrained models become less accurate and robust when finetuned.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposes an adjusted information-bottleneck-based approach for interpreting multi-modal models. Specifically, the approach was appropriately adapted from the information bottleneck method that has been widely used for interpreting image models. Through various approximations, the method was successfully adjusted for the multimodal case. The experimental results presented in the paper demonstrate the successful identification of important shared information between images and text.

STRENGTHS:
Understanding the mutual information between images and text in a multi-modal space is a crucial problem. This paper successfully adjusts the information bottleneck-based explainability method, primarily used in the vision unimodal field, to the multimodal space using various approximation techniques. The qualitative results demonstrate the method's ability to capture the mutually related information between text and images, while the quantitative results reaffirm its superior performance compared to the existing methods. Although there were concerns about cherry-picking due to the limited number of examples, the additional examples provided in the appendix alleviate those concerns.

WEAKNESSES:
Regarding the approximation part, as it is not within my expertise, I cannot make a definitive judgment. However, I don't have any major concerns about other aspects of the paper.

QUESTIONS:
1) Lines 135-137 seem to explain the unimodal information bottleneck, so why does Z_text appear?

2) I may have missed it, but what do m and s represent in line 168? Are they the mean and standard deviation? Does it mean that multiple texts are involved?

LIMITATIONS:
It seems that there are no particular specific limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06