SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\beta = 1$
consolidation in
$\beta = 10$
consolidation in
the left lung
funny animals of the week, animal pictures
$\sigma^2 = 0.001$
consolidation the left lung
$\sigma^2 = 0.01$
consolidation in the left lung
$\sigma^2 = 0.1$
consolidation in the left lung
$\sigma^2 = 1$
consolidation in the left lung
$\sigma^2 = 10$
consolidation in the left lung
funny animals of the week, animal pictures
$l = 6$
consolidation in the left lung
$l = 7$
consolidation in the left lung
$l = 8$
consolidation in the left lung
$l = 9$
consolidation in the left lung
$l = 10$
consolidation in the left lung
funny animals of the week, animal pictures
Figure 5: Visualization of the Impact of Different Hyperparameters.  $\beta$  and  $\sigma^2$  that make the fitting and compression terms be at a similar scale and deeper layer  $\ell$  usually give better performance.
Appendix B Attribution Maps for Additional Examples
Figure 6: Attribution maps for randomly picked examples from the MS-CXR chest X-ray dataset [3]. Note that the attribution score is assigned to each token, instead of each word, due to tokenization.
GradCAM
Saliency
Chefer et al.
girl walking over a bridge
Figure 7: Attribution maps for randomly picked examples from the Conceptual Captions dataset [26].
Figure 8: Attribution maps for MRI and CT examples from the ROCO dataset [19]. The highlighted areas increase when  $\beta$  decreases. Due to the lack of segmentation masks in ROCO, we are unable to perform a quantitative evaluation.
Appendix C Extension to Additional Modalities With ImageBind
Our method can be easily extended to representation learning of modalities other than image and text, as long as features of different modalities are projected into a shared embedding space. We use ImageBind [9] as an example to showcase the effectiveness of our method in interpreting audio-image and audio-text representation learning. Since ImageBind aligns other modalities (audio, sensors that record depth (3D), thermal (infrared radiation), and inertial measurement units (IMU)) embedding to image embeddings through contrastive learning, the embeddings of all modalities are in one shared embedding space. Thus, we can insert the information bottleneck into certain layers of the feature encoders of ImageBind respectively and adopt the M2IB objective similar to Equation (6) and Equation (7).
(a) Audio: Dog Barking
(c) Audio: Dog Barking
(b) Audio: Cat Meowing
(d) Audio: Cat Meowing
Figure 9: Attribution map on image and text when the other modality is audio.