SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Final Variational Optimization Objective. Finally, we assume that  $h_m(x_m^{(n)}; \lambda_m) \doteq \lambda_m^{(n)}$  (i.e., each input point has its own set of attribution parameters), that  $g_m$  is the mapping defined by the post-bottleneck layers of a VLPM for modality  $m$ , and that for each evaluation point the final embeddings for each modality get normalized across the embedding dimensions (i.e., both mapping,  $g_m$  and  $f_{m'}$ , contain embedding normalization transformations). For normalized  $g_m(z_m)$  and  $e_{m'}$ , the log of the Gaussian probability density  $q(f_{m'}(x_{m'}) | g_m(z_m))$  simplifies and is proportional to the cosine similarity between  $f_{m'}(x_{m'})$  and  $g_m(z_m)$ , giving the final optimization objective
$$
\begin{array}{l} \hat {\mathcal {F}} _ {m} ^ {\text {e m p i r i c a l}} \left(\theta_ {m}\right) \doteq \frac {1}{N} \sum_ {n = 1} ^ {N} \int p \left(z _ {m} \mid x _ {m} ^ {(n)}; \theta_ {m}\right) S _ {\text {c o s i n e}} \left(e _ {m ^ {\prime}}, g _ {m} \left(z _ {m}\right)\right) d z _ {m} \tag {15} \\ - \beta_ {m} \mathbb {D} _ {\mathrm {K L}} (p _ {Z _ {m} \mid X _ {m}} (\cdot ; x _ {m} ^ {(n)}; \theta_ {m}) \| q _ {Z _ {m}} (\cdot)), \\ \end{array}
$$
where  $S_{\mathrm{cosine}}(\cdot ,\cdot)$  is the cosine similarity function. For gradient estimation during optimization, the remaining integrals can be estimated using simple Monte Carlo estimation and reparameterization gradients. For  $\theta_{m} = \{\lambda_{m},\sigma_{m},\ell_{m}\}$ , the objective function is maximized with respect to  $\lambda_{m}$  independently for each modality, and  $\beta_{m},\sigma_{m}$ , and  $\ell_{m}$  are modality-specific hyperparameters.
We evaluate the proposed attribution method using CLIP [21] on four image-caption datasets, including widely-used image captioning datasets and medical datasets. Our main datasets are (i) Conceptual Captions [26] consisting of diverse images and captions from the web, and (ii) MS-CXR (Local Alignment Chest X-ray dataset; [3]), which contains chest X-rays and texts describing radiological findings, complementing MIMIC-CXR (MIMIC Chest X-ray; [13]) by improving the bounding boxes and captions. In addition, we also include some qualitative examples from the following radiology and remote sensory datasets to show the potential application of our model in safety-critical domains. Namely, we have (iii) ROCO (Radiology Objects in COntext; Pelka et al. [19]) that includes radiology image-caption pairs from the open-access biomedical literature database PubMed Central, and (iv) RSICD (Remote Sensing Image Captioning Dataset; Lu et al. [16]), which collects remote sensing images from web map services including Google Earth and provides corresponding captions.
For all experiments, we use a pretrained CLIP model with ViT-B/32 [7] as the image encoder and a 12-layer self-attention transformer as the text encoder. For Conceptual Captions, we use the pretrained weights of openai/clip-vit-base-patch32. For MS-CXR, we use CXR-RePaiR [8] which is CLIP finetuned on radiology datasets, and compare the impact of finetuning in Section 4.5. For each {image, caption} pair, we insert an information bottleneck into the given layer of the text encoder and image encoder of CLIP separately, then train the bottleneck using the same setup as the Per-Sample Bottleneck of original IBA [24], which duplicates a single sample for 10 times to stabilize training and runs 10 iterations using the Adam optimizer with a learning rate of 1. Experiments show no significant difference between different learning rates and more training steps. We conduct a hyper-parameter tuning on the index of the layer  $l$ , the scaling factor  $\beta$ , and the variance  $\sigma^2$ . For a discussion of these hyperparameters in the multi-modal information bottleneck objective, see Appendix A.
$^{2}$ Cf. https://huggingface.co/openai/clip-vit-base-patch32.
(a) Different Highlights with Respect to Different Text.
Figure 2: Example Saliency Maps when Involving Multiple Objects. Our method can successfully detect all occurrences of all relevant objects in image and text.
(b) Multiple Occurrences of Same Object.
We qualitatively compare our method with 5 widely used attribution methods, including gradient-based GradCAM [25] and Saliency method [28], perturbation-based Kernel SHAP [17] and RISE [20], and transformer-specific method [6]. We show one image-caption example and two radiology examples in Figure 1 and provide more qualitative comparisons in the Appendix B. As shown, our method is able to capture all relevant objects appearing in both modalities, while other methods tend to focus on one major object. As illustrated in Figure 2a, the highlighted areas in the image change according to different inputs, and our method can capture complicated relationships between image and text when involving multiple objects. Our method is also able to detect multiple occurrences of the same object in the image, as illustrated in Figure 2b.
4.3 Localization Test
We quantitatively measure the effectiveness of our proposed attribution method by evaluating its accuracy in zero-shot detection for images. We binarize the saliency map such that the areas with scores higher than the threshold  $(75\%)$  are assigned 1 while the rest are assigned 0. We denote the resulting binary map as  $S_{\mathrm{pred}}$ . We also construct a ground-truth binary map,  $S_{\mathrm{gt}}$ , using the bounding boxes provided by MS-CXR [3], where the region inside the bounding boxes is assigned to 1 while the outside is assigned to 0. Note that some samples have multiple bounding boxes, and we consider all of them to test the method's multi-occurrence detection ability. Then, we calculate the IoU (Intersection over Union) of  $S_{\mathrm{pred}}$  and  $S_{\mathrm{gt}}$ . Namely, for images with a height of  $n$  and a width of  $m$ , the score is calculated by
$$
\text {L o c a l i z a t i o n} = \frac {\sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} \mathbb {1} _ {S _ {\mathrm {p r e d}} ^ {i j} \wedge S _ {\mathrm {g t}} ^ {i j}}}{\sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} \mathbb {1} _ {S _ {\mathrm {p r e d}} ^ {i j} \vee S _ {\mathrm {g t}} ^ {i j}}}, \tag {16}
$$
where  $\mathbb{1}$  is the indicator function,  $\wedge$  is the logical AND and  $\vee$  is the logical OR operator.
We found that M2IB attribution attains an average IoU of  $22.59\%$  for this zero-shot detection task, outperforming all baseline models, as shown in Table 1. Recognizing that a localization score of  $22.59\%$  appears somewhat low in absolute terms, we briefly note that there are two potential causes that could lead to the low absolute values in the localization scores: (i) M2IB attribution indeed generates segmentation instead of bounding boxes, so evaluation by bounding boxes would underestimate the quality of the saliency map. (ii) The model under evaluation [8] is not finetuned for detection and may only have learned a coarse-grained relationship between X-rays and medical notes.
4.4 Degradation Tests
While the localization test shows that M2IB attribution may be a promising zero-shot detection and segmentation tool, the localization test may underestimate the accuracy of attribution since even a perfect attribution map can produce a low localization score if the (finetuned) VLPM under evaluation is poor at extracting useful information—which is very likely for challenging specialized tasks like chest X-ray classification.
To get a more fine-grained picture of the usefulness of M2IB, we use three additional evaluation metrics to compare M2IB to competitive baselines. The underlying idea of all three evaluations is that removing features with high attribution scores should decrease the performance, while discarding features with low attribution scores can improve the performance as noisy information is ignored. We randomly sample 2,000 image-text pairs from Conceptual Captions and 500 image-text pairs from MS-CXR. For each dataset, we perform ten experiments for each evaluation metric (five for ROAR+) and report the average score with the standard error in Table 1.
Table 1: Quantitative Results. The boldface denotes the best result per row. Means and standard errors were computed over ten random seeds.
Methods	GradCAM [25]	Saliency [28]	KS [17]	RISE [20]	Chefer et al. [6]	Ours
CC image	% Conf. Drop ↓	4.96 ± 0.01	1.99 ± 0.01	1.94 ± 0.01	1.12 ± 0.01	1.63 ± 0.01	1.11 ± 0.01
% Conf. Incr. ↑	17.84 ± 0.08	22.95 ± 0.12	25.18 ± 0.28	35.72 ± 0.14	37.41 ± 0.12	41.55 ± 0.19
% ROAR+ ↑	2.29 ± 0.41	6.88 ± 0.88	1.56 ± 0.88	3.15 ± 0.97	7.66 ± 0.55	10.59 ± 0.85
CC text	% Conf. Drop ↓	2.19 ± 0.01	1.78 ± 0.01	1.71 ± 0.01	1.30 ± 0.01	1.06 ± 0.01	1.06 ± 0.01
% Conf. Incr ↑	29.71 ± 0.19	38.96 ± 0.15	46.87 ± 0.21	38.31 ± 0.48	38.42 ± 0.11	38.55 ± 0.20
% ROAR+ ↑	43.23 ± 0.66	43.74 ± 0.65	47.46 ± 3.62	49.04 ± 1.12	53.57 ± 1.26	60.41 ± 1.12
MSCXR image	% Conf. Drop ↓	2.76 ± 0.03	0.81 ± 0.01	2.37 ± 0.04	3.94 ± 0.03	1.87 ± 0.02	0.55 ± 0.01
% Conf. Incr. ↑	12.64 ± 0.46	35.08 ± 0.44	10.24 ± 0.68	7.28 ± 0.44	21.44 ± 0.46	45.92 ± 0.70
% ROAR+ ↑	3.54 ± 0.80	25.46 ± 1.35	12.67 ± 1.02	16.79 ± 0.76	24.42 ± 1.19	38.7 ± 0.86
% Localization ↑	5.56 ± 0.13	21.6 ± 0.16	7.77 ± 0.13	10.97 ± 0.24	21.65 ± 0.25	22.59 ± 0.14
MSCXR text	% Conf. Drop ↓	2.26 ± 0.04	3.35 ± 0.03	2.4 ± 0.05	1.16 ± 0.02	2.93 ± 0.03	2.28 ± 0.04
% Conf. Incr. ↑	36.24 ± 0.54	18.88 ± 0.54	34.12 ± 0.77	57.2 ± 0.65	28.08 ± 0.34	35.48 ± 0.69
% ROAR+ ↑	11.07 ± 0.62	15.79 ± 0.92	14.28 ± 1.09	12.09 ± 1.52	9.11 ± 0.6	16.31 ± 0.75
Saliency Map
Attribution Weighted Image
Figure 3: Visualization of Degradation. The third column is obtained by calculating the element-wise product of the original image and saliency map, while the text with attribution scores lower than  $50\%$  percentile is masked by a blank token  $\langle \mathrm{B}\rangle$ . It is used in the Increase in Confidence metric and Drop in Confidence metric. The fourth column shows an example of the training data in ROAR+. We replace the image pixels with attribution scores higher than  $75\%$  percentile by the channel mean and replace the text tokens with attribution scores higher than  $50\%$  by a blank token  $\langle \mathrm{B}\rangle$ . The results in Table 1 use the padding token as the blank token  $\langle \mathrm{B}\rangle$ .
Corrupted (binarized M)
Drop in Confidence [4]. An ideal attribution method should only assign high scores to important features, thus we should not observe a drop in performance if only the high-attribute parts are allowed in the input. For images, we use point-wise multiplication of the saliency map and the image input. Since scaling token ids is meaningless, we use binarization similar to [33] where only tokens with attribution scores in the top  $50\%$  are kept. We provide an example of distilled image and text in Figure 3. Formally, we define this score by
$$
\text {C o n f i d e n c e} \quad \operatorname {D r o p} = \frac {1}{N} \sum_ {i = 1} ^ {N} \max  \left(0, o _ {i} - s _ {i}\right), \tag {17}
$$
where  $o_i$  is the cosine similarity of features of original images and texts, and  $s_i$  is the new cosine similarity when one modality is distilled according to the attribution. The lower this metric is, the better the attribution method is. This metric is implemented in the pytorch-gradcam repository. $^3$
Increase in Confidence [4]. Similarly, removing noisy information in the input might increase the model's confidence. We compute
$$
\text {C o n f i d e n c e I n c r e a s e} = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {1} \left(o _ {i} <   s _ {i}\right), \tag {18}
$$
where  $\mathbb{1}$  is the indicator function and the definition of  $o_i$  and  $s_i$  is the same as above. Higher values indicate better performance. This metric is also implemented in the pytorch-gradcam repository. $^3$
3GradCAM and its variants are usually applied to image classification and use the softmax outputs of each class as confidence scores. Since our setting does not contain any labels, we use cosine similarity with the other modality instead.
Remove and Retrain + (ROAR+, an Extension of ROAR [11]). We finetune the base model on the degraded images and texts where the most important parts are replaced by uninformative values (i.e., channel means of images or padding tokens for texts, see Figure 3) and evaluated on a validation set of original inputs. If the attribution method is accurate, a sharp decrease in performance is expected because all useful features are removed, and the model cannot learn anything relevant from the degraded data. We split the testing dataset into  $80\%$  training data and  $20\%$  validation data. We use the same contrastive loss as used for pretraining CLIP and define the score by  $(l_c - l_o) / l_o$ , where  $l_o$  is the validation loss of retraining using the original data, and  $l_c$  is the validation loss when retraining with corrupted data. We repeat the process five times and report the average score.
The results are summarized in Table 1. M2IB attribution outperforms baseline models in almost all numerical metrics, except for perturbation-based methods, which achieve better Increase/Drop in Confidence scores for texts. Perturbation-based methods perform better for short text because they can scan all possible binary masks on the text and then find the optimal one with the highest confidence score. However, this kind of method is very computationally expensive. We use 2,048 masks for image and 256 masks for text in RISE, where each mask of each modality requires one forward pass to get the confidence score, resulting in 2.3k forward passes (7.8s on RTX8000 with a batch size of 256). In contrast, M2IB attribution only requires 100 forward passes and takes 1.2s for one image-text pair.
In general, removing pixels or tokens with lower attribution scores using M2IB attribution generally increases the mutual information with the other modality, while masking by our attribution map generally decreases the relevance with the other modality and makes the model perform worse when retraining on the corrupted data. This confirms that our method generates useful attribution maps.
4.5 Sanity Check
We conduct a sanity check to ensure our method is, in fact, sensitive to model parameters. We follow the sanity check procedure proposed by Adebayo et al. [1], where parameters in the model are randomized starting from the last to the first layer. As shown in Figure 4, M2IB passes the sanity check as the attribution scores of image pixels and text tokens change as the model weights change. Our method also produces more accurate saliency maps for finetuned models compared to pretrained models, which further confirms that the resulting attribution can successfully reflect the quality of the model. Since we insert the information bottleneck after a selected layer (layer 9 in this case), the randomization of this and previous layers appears to have a larger influence on the output.
4.6 Error Analysis and Limitations
We notice that our proposed attribution method generally performs well on text but sometimes shows less satisfying performance on images. By inspecting the qualitative examples, we observe that M2IB sometimes fails to detect the entire relevant regions in images. As shown in the fourth ("sea" example) and fifth ("bridge" example) rows in Figure 7, our method only highlights a fraction of the object in the image, although it should include the whole object. This is probably due to the fact that the model under evaluation only relies on a few patterns in the image to make its prediction. Increasing the relative importance of the fitting term (i.e., using smaller  $\beta$ ) enlarges the highlighted area. However, we don't recommend using extreme  $\beta$  because it will break the balance between fitting and compression and thus make the information bottleneck unable to squeeze information.
We also note that M2IB is sensitive to the choice of hyperparameters. As shown in Figure 5, different combinations of hyperparameters will generate different saliency maps. We show how to use the ROAR+ score to systematically select the optimal hyperparameters and also provide visualization to illustrate the effect of different hyperparameters in Appendix A. Since there is no convention on evaluating the attribution method, we suggest considering various evaluation metrics, visualization of examples, and the goal of the attribution task when choosing hyperparameters. We emphasize that M2IB should be used with caution since attributing the success or failure of a model solely to a set of features can be overly simplistic, and different attribution methods might lead to different results.
4The original ROAR also corrupts the validation data, which means that different methods might use varied training and validation datasets, making direct comparisons difficult. To ensure a fair comparison, we consistently use the original data as the validation set for all methods.
Figure 4: Saliency Maps for Sanity Checks. "Finetuned" represents the model that is finetuned on MIMIC-CXR [13], a Chest X-ray dataset. "Pretrained" represents pretrained CLIP [21] from OpenAI. "Projection" represents CLIP with randomized projection layer, which is the last layer of the image encoder or text encoder that projects image or text features into the shared embedding space. "Random" means that all parameters in the model are randomly initiated. The remaining columns represent models with weights randomized starting from the last to the given layer. The results suggest that the saliency maps of M2IB attribution are sensitive to model weights, as desired, meaning that M2IB passes the sanity check.
We developed multi-modal information bottleneck (M2IB) attribution, an information-theoretic approach to multi-modal attribution mapping. We used CLIP-ViT-B/32 in our experiments, but M2IB attribution can be directly applied to CLIP with alternative neural network architectures and is compatible with any VLPM for which the features of all modalities are projected into a shared embedding space—a commonly used approach in state-of-the-art multi-modal models.
Going beyond vision and language modalities, Girdhar et al. [9] recently proposed a new multimodal model, ImageBind, which aligns embeddings of five modalities to image embeddings through contrastive learning on pairs of images with each modality and uses a similar architecture as CLIP, where the encoder for each modality is a transformer. We applied M2IB attribution to pairs of six different modalities using ImageBind—which required minimal implementation steps—and provided qualitative examples in Appendix C to illustrate that M2IB attribution can be applied successfully to more than just vision and text modalities.
In this paper, we provided exhaustive empirical evidence that M2IB increases attribution accuracy and improves the interpretability of state-of-the-art vision-language pretrained models. We hope that this work will encourage future research into multi-modal information-theoretic attribution methods that can help improve the interpretability and trustworthiness of VLPMs and allow them to be applied to safety-critical domains where interpretability is essential.
This work was supported in part through the NYU High-Performance Computing services as well as by NSF CAREER IIS-2145492, NSF I-DISRE 193471, NSF IIS-1910266, BigHat Biosciences, Capital One, and an Amazon Research Award.
[30] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319-3328. PMLR, 06-11 Aug 2017.
[31] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE information theory workshop (itw), pages 1-5. IEEE, 2015.
[32] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, 2000.
[33] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-CAM: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 24-25, 2020.
A Experimental Details and Further Experimental Results 15
B Attribution Maps for Additional Examples 17
C Extension to Additional Modalities With ImageBind 19
Appendix A Experimental Details and Further Experimental Results
$\beta$  controls the relative importance of the compression term. The larger the  $\beta$  is, the less information is allowed to flow through this layer. As shown in Figure 5a, too large and too small  $\beta$  generates similar attribution maps in terms of relative importance. However, too small  $\beta$  allows nearly everything through the bottleneck, whereas too large  $\beta$  nearly discards everything.
$\sigma$  controls the values of the noise added to the intermediate representations. When  $\sigma$  is very small, the values of the noise will be close to 0, thus having minimal impact on the intermediate representations. This effect is similar to the situation when  $\beta$  is very small and IB will add almost no noise (Figure 5b).  $\sigma$  also directly affects the compression term as smaller  $\sigma$  will lead to higher KL divergence. Thus,  $\sigma$  and  $\beta$  are correlated with each other and we perform a grid search to find the best combination.
Layer  $\ell$  where the information bottleneck is inserted also impacts the attribution. Inserting the bottleneck too early will prevent the model from learning informative features while inserting the bottleneck too late reduces the impact of the bottleneck (Figure 5c). We also observe that the attribution of texts is usually more stable than images.
These hyperparameters can be chosen according to numerical metrics mentioned in Section 4.4. We randomly sample 500 image-text pairs from Conceptual Captions and 500 image-text pairs from MS-CXR, which is ensured to not overlap with the test set in Section 4. We then perform a grid search for the best combination of  $\beta = \{1,0.1,0.01\}$  and  $\sigma^2 = \{1,0.1,0.01\}$  and  $\ell = \{7,8,9\}$  (indexing from 0), and find the best combination of hyperparameters is shown in Table 2.
Table 2: Hyperparameter Tuning Results for  $\ell$ ,  $\beta$  and  $\sigma$ : We calculate the ROAR+ score for different combinations for 3 runs and report the average. For each table, the highest score is in bold and indicates the performance is optimal for this set of hyperparameters. For texts of Conceptual Captions, we select  $\ell = 9$ ,  $\beta = 0.1$  and  $\sigma = 1$ . Since ROAR+ uses binarized saliency maps (75% threshold for image pixels and 50% threshold for text tokens, it mainly focuses on the features with high attribution scores and neglects the change in the attribution for less important features. Thus, sometimes hyperparameters with slightly lower ROAR+ scores might generate more visually appealing results as in Figure 5, though the numerical results are consistent with qualitative examples in general.
(a) Conceptual Captions - Image
σ2=1	σ2=0.1	σ2=0.01
l=7	β=1	10.13	10.78	7.11
β=0.1	8.00	7.96	6.66
β=0.01	7.12	6.69	8.32
l=8	β=1	12.08	8.25	9.49
β=0.1	10.33	6.96	6.33
β=0.01	10.05	6.77	8.23
l=9	β=1	7.75	9.44	5.63
β=0.1	12.72	8.91	9.94
β=0.01	8.88	6.42	9.08
(b) Conceptual Captions - Text
σ2=1	σ2=0.1	σ2=0.01
l=7	β=1	15.49	10.74	10.78
β=0.1	13.31	13.73	14.18
β=0.01	12.96	15.61	12.78
l=8	β=1	11.29	16.47	8.82
β=0.1	13.26	14.80	13.86
β=0.01	8.99	11.09	13.72
l=9	β=1	13.85	18.16	9.87
β=0.1	18.16	15.03	8.97
β=0.01	11.72	8.46	9.80
(c) MSCXR - Image
σ2=1	σ2=0.1	σ2=0.01
l=7	β=1	35.24	29.56	25.20
β=0.1	34.15	30.44	30.28
β=0.01	30.12	30.21	29.32
l=8	β=1	34.59	29.35	30.07
β=0.1	32.33	30.68	28.04
β=0.01	31.99	33.24	36.99
l=9	β=1	31.76	28.43	32.94
β=0.1	37.17	35.06	32.22
β=0.01	36.19	34.59	30.18
(d) MSCXR - Text
σ2=1	σ2=0.1	σ2=0.01
l=7	β=1	15.77	10.91	9.22
β=0.1	12.60	11.15	8.92
β=0.01	13.89	12.31	13.11
l=8	β=1	10.76	14.34	12.13
β=0.1	12.67	10.53	11.09
β=0.01	10.58	12.49	15.48
l=9	β=1	11.70	11.01	12.24
β=0.1	10.39	13.54	12.48
β=0.01	12.93	14.64	12.57
$\beta = 0.001$
consolidation
$\beta = 0.01$
consolidation in
$\beta = 0.1$
consolidation i