SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ying Wang* New York University
Tim G. J. Rudner* New York University
Vision-language pretrained models have seen remarkable success, but their application to safety-critical settings is limited by their lack of interpretability. To improve the interpretability of vision-language models such as CLIP, we propose a multimodal information bottleneck (M2IB) approach that learns latent representations that compress irrelevant information while preserving relevant visual and textual features. We demonstrate how M2IB can be applied to attribution analysis of vision-language pretrained models, increasing attribution accuracy and improving the interpretability of such models when applied to safety-critical domains such as healthcare. Crucially, unlike commonly used unimodal attribution methods, M2IB does not require ground truth labels, making it possible to audit representations of vision-language pretrained models when multiple modalities but no ground-truth data is available. Using CLIP as an example, we demonstrate the effectiveness of M2IB attribution and show that it outperforms gradient-based, perturbation-based, and attention-based attribution methods both qualitatively and quantitatively.
Vision-Language Pretrained Models (VLPMs) have become the de facto standard for solving a broad range of vision-language problems [15]. They are pre-trained on large-scale multimodal data to learn complex associations between images and text and then fine-tuned on a given downstream task. For example, the widely used CLIP model [21], which uses image and text encoders trained on 400 million image-text pairs, has demonstrated remarkable performance when used for challenging vision-language tasks, such as Visual Question Answering and Visual Entailment [27].
VLPMs are highly overparameterized black-box models, enabling them to represent complex relationships in data. For example, Vision Transformers [ViTs; 7] are a state-of-the-art transformer-based vision model and are used as the image encoder of CLIP [21], containing 12 layers and 86 million parameters for ViT-Base and 24 layers and 307 million parameters for ViT-Large. Unfortunately, VLPMs like CLIP are difficult to interpret. However, model interpretability is essential in safety-critical real-world applications where VLPMs could be applied successfully, such as clinical decision-making or image captioning for the visually impaired. Improved interpretability and understanding of VLPMs would help us identify errors and unintended biases in VLP and improve the safety, reliability, and trustworthiness of VLPMs, thereby allowing us to deploy them in such safety-critical settings.
To tackle this lack of transparency in deep neural networks, attribution methods, which aim to explain a model's predictions by attributing contribution scores to each input feature, have been proposed for post-hoc interpretability. For example, for vision models, attribution methods can be used to create heatmaps that highlight features that are most responsible for a model's prediction. Similarly, for language models, scores are assigned to each input token. While the requirements for interpretability vary depending on the task, dataset, and model architecture, accurate and reliable attribution is an important tool for making emerging and state-of-the-art models more trustworthy.
hungry mouse eating cheese in a mouse trap
ill - defined op ac ities largely in the upper lobe
hungry mouse eating cheese in a mouse trap
ill - defined op acities largely in the upper lobe
hungry mouse eating cheese in a mouse trap
ill - defined opac ities largelyin the upper lobe
hungry mouse eating cheese in a mouse trap
ill - defined opacities largelyin the upper lobe
local consolidation in the left mid lung
Figure 1: Example Attribution Maps For Image and Text Inputs. The red rectangles in the second and third rows show the ground-truth bounding boxes associated with the text, provided in the MS-CXR dataset [3]. Multi-modal information bottleneck (M2IB) attribution maps successfully identify relevant objects in the given image-text pairs, while other methods provide less precise localization and neglect critical features in the inputs.
While existing attribution methods focus primarily on unimodal models, we propose an attribution method for VLPMs that allows us to identify critical features in image and text inputs using the information bottleneck principle [31]. Unlike standard unimodal attribution methods, the proposed multi-modal information bottleneck (M2IB) attribution method does not require access to ground-truth labels.
To do this, we formulate a simple multi-modal information bottleneck principle, use a variational approximation to derive a tractable optimization objective from this principle, and optimize this objective with respect to a set of attribution parameters. Using these parameters, we are able to "turn off" all irrelevant features and only keep important ones and generate attribution maps. In contrast to unimodal information bottleneck attribution [24], we aim to find attribution parameters that maximize the likelihood of observing features of one modality given features associated with the respective other modality. We perform a qualitative and quantitative empirical evaluation and find that M2IB is able to successfully identify important features relevant to both image and text inputs. We provide an illustrative set of attribution map examples in Figure 1.
1. We propose a multi-modal information bottleneck principle and use it to develop a multi-modal information bottleneck attribution method to improve the interpretability of Vision-Language Pretrained Models.
2. We perform an extensive empirical evaluation and demonstrate on several datasets—including healthcare data that can be used in safety-critical settings—that multi-modal information bottleneck attribution significantly outperforms existing gradient-based, perturbation-based, and attention-based attribution methods, quantitatively and qualitatively.
The code for our experiments is available at: https://github.com/YingWANGG/M2IB.
Attribution methods enable post-hoc interpretability of trained models by allocating significance scores to the input features, such as image pixels or textual tokens.
2.1 Attribution methods
Gradient-Based Attribution. The use of gradients as a foundational component in attribution methods has been widely explored in the interpretability literature. Simonyan et al. [28] directly use the gradient of the target with respect to inputs as the attribution score. Building on this, GradCAM [Gradient-weighted Class Activation Mapping; 25] weights convolutional layer activations by average pixel-wise gradients to identify important regions in image inputs. Integrated Gradients [30] introduces two essential axioms for attribution methods—sensitivity and implementation invariance, which motivates the application of integrated gradients along paths from a baseline input to the instance under analysis.
Perturbation-Based Attribution. Perturbation approaches involve modifying input features to observe the resulting changes in model predictions. Unlike gradient-based techniques, they eliminate the need for backpropagation, allowing the model to be treated as a complete black box. However, such methods can be computationally intensive, especially for intricate model architectures, given the need to re-evaluate predictions under many perturbation conditions. The LIME [Local Interpretable Model-agnostic Explanations; 22] algorithm leverages perturbation in a local surrogate model. Lundberg and Lee [17] propose SHAP (SHapley Additive exPlanations), which employs a game-theoretic approach to attribution by utilizing perturbations to compute Shapley values for each feature. Furthermore, they propose a kernel-based estimation approach for Shapley values, called KernelSHAP, drawing inspiration from local surrogate models.
Attention-based Attribution. The rise of transformers across multiple domains in machine learning has necessitated the development of specialized attribution methods tailored to these models. Using attention as an attribution method offers a straightforward approach to deciphering and illustrating the significance of various inputs in a transformer's decision-making process. Nevertheless, solely relying on attention is inadequate as it overlooks crucial information from the value matrices and other network layers [5]. To address this issue, Chefer et al. [6] propose a method that learns relevancy maps through a forward pass across the attention layers with contributions from each layer cumulatively forming the aggregated relevance matrices.
Information-Theoretic Attribution. Schulz et al. [24] use information bottleneck attribution (IBA), where an information bottleneck is inserted into a layer of a trained neural network to distill the essential features for prediction. IBA is a model-agnostic method and shows impressive results on vision models including VGG-16 [14] and ResNet-50 [10]. Subsequently, IBA has been applied to language transformers [12] and was shown to also outperform other methods on this task. However, IBA has thus far been focused on only one modality and has only been adopted in supervised learning. To the best of our knowledge, there is no previous work on applying the information bottleneck principle to multi-modal models like VLPMs.
2.2 Evaluation of Attribution Methods
Despite active research on attribution methods, we still lack standardized evaluation metrics for attribution due to the task's inherent complexity. For vision models, attribution maps are frequently juxtaposed with ground-truth bounding boxes for a form of zero-shot detection. However, the attribution of high scores to irrelevant areas might arise from either subpar attribution techniques or flawed models, making it challenging to isolate the root cause of such discrepancies.
To tackle this issue, previous studies have resorted to degradation-based metrics [4, 33]. The underlying principle is that eliminating features with high attribution scores should diminish performance, whereas the removal of low-attributed features, often seen as noise, should potentially enhance performance. Additionally, Hooker et al. [11] introduce ROAR (Remove and Retrain), a methodology that deliberately degrades training and validation datasets in alignment with the attribution map, thereby accounting for potential distribution shifts. Should the attribution be precise, retraining the model using these altered datasets would result in a pronounced performance decline. Rong et al. [23] argue that mere image masking can cause a leak of information via the mask's shape, and thus propose a Noisy Linear Imputation strategy that replaces pixels with the average of their neighbors.
It is important to note that a visually appealing saliency map does not guarantee the efficacy of the underlying attribution method. For example, an edge detector might generate a seemingly plausible saliency map, yet it does not qualify as an attribution method because it is independent of the model under analysis. Adebayo et al. [1] propose a sanity check designed to assess whether the outputs of attribution methods genuinely reflect the properties of the specific model under evaluation. Notably, several prominent models, including Guided Backprop [29] and its variants appear insensitive to model weights and consequently do not pass the sanity check.
3 Attribution via a Multi-Modal Information Bottleneck Principle
In this section, we introduce a simple, multi-modal variant of the information bottleneck principle and explain how to adapt it to feature attribution.
3.1 The Information Bottleneck Principle
The information bottleneck principle [32] provides a framework for finding compressed representations of neural network models. To obtain latent representations that reflect the most relevant information of the input data, the information bottleneck principle seeks to find a stochastic latent representation  $Z$  of the input source  $X$  defined by a parametric encoder  $p_{Z|X}(z|x;\theta)$  that is maximally informative about a target  $Y$  while constraining the mutual information between the latent representation  $Z$  and the input  $X$ . For a representation parameterized by parameters  $\theta$ , this principle can be expressed as the optimization problem
$$
\max  _ {\theta} I (Z, Y; \theta) \quad \text {s . t .} \quad I (Z, X; \theta) \leq \bar {I}, \tag {1}
$$
where  $I(\cdot ,\cdot ;\theta)$  is the mutual information function and  $\bar{I}$  is a compression constraint. We can equivalently express this optimization problem as maximizing the objective
$$
\mathcal {F} (\theta) \doteq I (Z, Y; \theta) - \beta I (Z, X; \theta), \tag {2}
$$
where  $\beta$  is a Lagrange multiplier that trades off learning a latent representation that is maximally informative about the target  $Y$  with learning a representation that is maximally compressive about the input  $X$  [2].
3.2 A Multi-Modal Information Bottleneck Principle
Unfortunately, for VLPMs, the loss function above is not suitable since we wish to learn interpretable latent representations using only text and vision inputs without relying on task-specific targets  $Y$  that may not be available or are expensive to obtain. To formulate a multi-modal information bottleneck principle for VLPMs, we need to develop an optimization objective that is more akin to optimization objectives for self-supervised methods for image-text representation learning that only use (text, image) pairs [21, 18].
This learning problem fundamentally differs from supervised attribution map learning for unimodal tasks. For example, we may have an image of a bear,  $X_{\mathrm{bear}}$ , and a corresponding label,  $Y_{\mathrm{bear}} =$  "bear". For a unimodal classification task, we can simply maximize  $I(Y_{\mathrm{bear}};Z_{\mathrm{bear}};\theta) - \beta I(X_{\mathrm{bear}};Z_{\mathrm{bear}};\theta)$  with respect to  $\theta$ , where  $Z_{\mathrm{bear}}$  is the latent representation of  $X_{\mathrm{bear}}$ .
In contrast, in image-text representation learning, we typically have text descriptions, such as "This is a picture of a bear"  $(L_{\mathrm{bear}}^{\prime})$  instead of labels [21]. In this setting, both  $X_{\mathrm{bear}}$  and  $L_{\mathrm{bear}}^{\prime}$  are "inputs" without a pre-defined corresponding label. To obtain a task-agnostic image-text representation independent from any task-specific ground-truth labels, we would like to use both input modalities and define a multi-modal information bottleneck principle and whereas the outputs are closely dependent on the specific downstream task. This requires defining an alternative to the "fitting term"  $I(Z,Y;\theta)$  of the conventional information bottleneck objective.
Fortunately, there is a natural proxy for the relevance of information in multi-modal data. If image and text inputs are related (e.g., text that describes the image), a good image encoding should contain information about the text and vice versa. Based on this intuition, we can express a multi-modal information bottleneck objective for  $X_{m}$  with  $m \in \mathcal{M} = \{\mathrm{modality1},\mathrm{modality2}\}$ , as
$$
\mathcal {F} _ {m} \left(\theta_ {m}\right) = I \left(Z _ {m}, E _ {m ^ {\prime}}; \theta_ {m}\right) - \beta I \left(Z _ {m}, X _ {m}; \theta_ {m}\right), \tag {3}
$$
where  $m' = \mathcal{M} \backslash m$  is the complement of  $m$ , and  $E_{m'}$  is embedding of modality  $m'$ . Next, we will show how to use this multi-modal variant of the information bottleneck principle for attribution.
3.3 A Multi-Modal Information Bottleneck for Attribution
To compute attribution maps for image and text data without access to task-specific labels, we will define an information bottleneck attribution method for multi-modal data.
To restrict the information flow in the latent representation with a simple parametric encoder, we adapt the masking approach in Schulz et al. [24]. For clarity and brevity, we henceforth represent the  $V_{m} \times W_{m}$ -dimensional latent representation  $Z_{m}$  in its vectorized form in  $\mathbb{R}^{J}$  with  $J \doteq V_{m} \cdot W_{m}$ . Assuming independence across latent representation dimensions, we then define
$$
p _ {Z _ {m} \mid X _ {m}} \left(z _ {m} \mid x _ {m}; \theta_ {m}\right) = \mathcal {N} \left(z _ {m}; h _ {m} \left(x _ {m}; \lambda_ {m}\right) \odot f _ {m} ^ {\ell_ {m}} \left(x _ {m}\right), \sigma_ {m} ^ {2} \operatorname {d i a g} \left[ \left(\mathbf {1} _ {J} - h _ {m} \left(x _ {m}; \lambda_ {m}\right)\right) ^ {2} \right]\right), \tag {4}
$$
where for a pair of modalities  $\mathcal{M} \doteq \{\text{image, text}\}$  with  $m \in \mathcal{M}$ ,  $\theta_{m} \doteq \{\lambda_{m}, \sigma_{m}, \ell_{m}\}$  are parameters,  $h_{m}(x_{m};\lambda_{m}) \in \mathbb{R}^{J}$  is a mapping parameterized by  $\lambda_{m}$ ,  $f_{m}^{\ell}(X_{m}) \in \mathbb{R}^{J}$  is the vectorized output of the  $\ell_{m}$ th layer of modality-specific neural network embedding  $f_{m}$ ,  $\sigma_{m}^{2} \in \mathbb{R}_{>0}$  is a hyperparameter,  $\mathrm{diag}[\cdot]$  represents an operator that transforms a vector into a diagonal matrix by placing the vector's elements along the main diagonal,  $\mathbf{1}_{J} \in \mathbb{R}^{J}$  is an all-ones vector, and  $\odot$  is the Hadamard product. To avoid overloading notation, we will drop the subscript in probability density functions except when needed for clarity. Based on Equation (4), we can express the stochastic latent representations as
$$
Z _ {m} \mid x _ {m}; \theta_ {m} = h _ {m} \left(x _ {m}; \lambda_ {m}\right) \odot f _ {m} ^ {\ell_ {m}} \left(x _ {m}\right) + \sigma_ {m} \left(\mathbf {1} _ {J} - h _ {m} \left(x _ {m}; \lambda_ {m}\right)\right) \odot \varepsilon , \tag {5}
$$
where  $\varepsilon \sim \mathcal{N}(0, I_J)$ . From this reparameterization, we can see that  $[h_m(x_m; \lambda_m)]_i = 1$  for  $i \in \{1, \dots, J\}$  means that no noise is added at index  $i$ , so  $[Z_m]_i$  will be the same as the original  $f_m^\ell(x_m)_i$ , whereas  $[h_m(x_m; \lambda_m)]_i = 0$  means that  $[Z_m]_i$  will be pure noise.
We can now express the multi-modal information bottleneck attribution (M2IB) objectives as
$$
\mathcal {F} _ {\text {i m a g e}} \left(\theta_ {\text {i m a g e}}\right) = I \left(Z _ {\text {i m a g e}}, E _ {\text {t e x t}}; \theta_ {\text {i m a g e}}\right) - \beta_ {\text {i m a g e}} I \left(Z _ {\text {i m a g e}}, X _ {\text {i m a g e}}; \theta_ {\text {i m a g e}}\right) \tag {6}
$$
$$
\mathcal {F} _ {\text {t e x t}} \left(\theta_ {\text {t e x t}}\right) = I \left(Z _ {\text {t e x t}}, E _ {\text {i m a g e}}; \theta_ {\text {t e x t}}\right) - \beta_ {\text {t e x t}} I \left(Z _ {\text {t e x t}}, X _ {\text {t e x t}}; \theta_ {\text {t e x t}}\right), \tag {7}
$$
which we can optimize with respect to the modality-specific sets of parameters  $\lambda_{\mathrm{image}}$  and  $\lambda_{\mathrm{text}}$ , respectively.  $\{\beta_{\mathrm{image}},\sigma_{\mathrm{image}},\ell_{\mathrm{image}}\}$  and  $\{\beta_{\mathrm{text}},\sigma_{\mathrm{text}},\ell_{\mathrm{text}}\}$  are each sets of hyperparameters.
3.4 A Variational Objective for Multi-Modal Information Bottleneck Attribution
To obtain tractable optimization objectives, we use a variational approximation. First, we note that  $I(Z_{m},X_{m};\theta_{m}) = \mathbb{E}_{p_{X_{m}}}\left[\mathbb{D}_{\mathrm{KL}}(p_{Z_{m}}|_{X_{m}}(\cdot |X_{m};\theta_{m})\parallel p_{Z_{m}}(\cdot ;\theta_{m}))\right]$ , where  $Z_{m}\mid X_{m};\theta_{m}$  can be sampled empirically whereas  $p(z_m;\theta_m)$  does not have an analytic expression because the integral  $p(z_m;\theta_m) = \int p(z_m|x_m;\theta_m)p(x_m)\mathrm{d}x_m$  is intractable. To address this intractability, we approximate  $p(z_m)$  by  $q(z_m)\doteq \mathcal{N}(z_m;0,\mathrm{I}_J)$ . This approximation leads to the upper bound
$$
\begin{array}{l} I \left(Z _ {m}, X _ {m}; \theta_ {m}\right) = \mathbb {E} _ {p _ {X _ {m}}} \left[ \mathbb {D} _ {\mathrm {K L}} \left(p _ {Z _ {m} \mid X _ {m}} (\cdot \mid X _ {m}; \theta_ {m}) \| q _ {Z _ {m}} (\cdot)\right) \right] - \mathbb {D} _ {\mathrm {K L}} \left(p _ {Z _ {m}} \| q _ {Z _ {m}}\right) \\ \leq \mathbb {E} _ {p _ {X _ {m}}} \left[ \mathbb {D} _ {\mathrm {K L}} \left(p _ {Z _ {m} \mid X _ {m}} (\cdot \mid X _ {m}; \theta_ {m}) \| q _ {Z _ {m}} (\cdot)\right) \right] \tag {8} \\ \dot {\mathbf {\theta}} = \mathcal {F} _ {m} ^ {\mathrm {c o m p r e s s i o n}} (\theta_ {m}). \\ \end{array}
$$
Next, while the unimodal information bottleneck attribution objective uses ground-truth labels to compute the "fit term" in the objective, the multi-modal information bottleneck attribution objectives require computing the mutual information between the aligned stochastic embeddings,
$$
\begin{array}{l} I \left(Z _ {m}, E _ {m ^ {\prime}}; \theta_ {m}\right) = \int p \left(e _ {m ^ {\prime}}, z _ {m}; \theta_ {m}\right) \log \frac {p \left(e _ {m ^ {\prime}} , z _ {m} ; \theta_ {m}\right)}{p \left(e _ {m ^ {\prime}}\right) p \left(z _ {m}\right)} d e _ {m ^ {\prime}} d z _ {m} (9) \\ = \int p \left(e _ {m ^ {\prime}}, z _ {m}; \theta_ {m}\right) \log \frac {p \left(e _ {m ^ {\prime}} \mid z _ {m}\right)}{p \left(e _ {m ^ {\prime}}\right)} d e _ {m ^ {\prime}} d z _ {m}, (10) \\ \end{array}
$$
which is not in general tractable. To obtain an analytically tractable variational objective, we approximate the intractable  $p(e_{m'} \mid z_m)$  by a variational distribution  $q(e_{m'} \mid z_m) \doteq \mathcal{N}(e_{m'}; g_m(z_m), I_K)$ , where  $g_m$  is a mapping that aligns the latent representation of modality  $m$  with  $E_{m'}$  and  $K$  is the dimension of the embedding, and get
$$
\begin{array}{l} I \left(Z _ {m}, E _ {m ^ {\prime}}; \theta_ {m}\right) \geq \int p \left(e _ {m ^ {\prime}}, z _ {m}; \theta_ {m}\right) \log q \left(e _ {m ^ {\prime}} \mid z _ {m}\right) d e _ {m ^ {\prime}} d z _ {m} \\ = \int p \left(x _ {m}\right) p \left(e _ {m ^ {\prime}} \mid x _ {m}\right) p \left(z _ {m} \mid x _ {m}; \theta_ {m}\right) \log q \left(e _ {m ^ {\prime}} \mid z _ {m}\right) d x _ {m} d e _ {m ^ {\prime}} d z _ {m} \tag {11} \\ \dot {=} \mathcal {F} _ {m} ^ {\mathrm {f i t}} (\theta_ {m}). \\ \end{array}
$$
With this approximation, we can obtain a tractable variational optimization objective by sampling  $X_{m}$  and  $E_{m'}$  from the empirical distribution
$$
\hat {p} \left(x _ {m}, e _ {m ^ {\prime}}\right) \doteq \frac {1}{N} \sum_ {n = 1} ^ {N} \delta_ {0} \left(x _ {m} - x _ {m} ^ {(n)}\right) \delta_ {0} \left(e _ {m ^ {\prime}} - f _ {m ^ {\prime}} \left(x _ {m ^ {\prime}} ^ {(n)}\right)\right), \tag {12}
$$
where  $f_{m'}(x_{m'})$  is the embedding input  $x_{m'}$  under the VLPM. With these approximations, we can now state the full variational objective, which is given by
$$
\mathcal {F} _ {m} ^ {\text {a p p r o x}} \left(\theta_ {m}\right) \doteq \mathcal {F} _ {m} ^ {\text {f i t}} \left(\theta_ {m}\right) - \beta_ {m} \mathcal {F} _ {m} ^ {\text {c o m p r e s s i o n}} \left(\theta_ {m}\right). \tag {13}
$$
The derivation of this objective has closely followed the steps in [2]. In practice, the objective can be computed using the empirical data distribution so that
$$
\begin{array}{l} \mathcal {F} _ {m} ^ {\text {e m p i r i c a l}} \left(\theta_ {m}\right) \doteq \frac {1}{N} \sum_ {n = 1} ^ {N} \int p \left(z _ {m} \mid x _ {m} ^ {(n)}; \theta_ {m}\right) \log q \left(e _ {m ^ {\prime}} \mid z _ {m}\right) d z _ {m} \tag {14} \\ - \beta_ {m} \mathbb {D} _ {\mathrm {K L}} \left(p _ {Z _ {m} \mid X _ {m}} (\cdot ; x _ {m} ^ {(n)}; \theta_ {m}) \parallel q _ {Z _ {m}} (\cdot)\right). \\ \end{array}
$$