ICLR 2025

Published as a conference paper at ICLR 2025
NARROWING INFORMATION BOTTLENECK THEORY
FOR MULTIMODAL IMAGE-TEXT REPRESENTATIONS
INTERPRETABILITY
Zhiyu Zhu1, Zhibo Jin1, Jiayu Zhang2, Nan Yang3, Jiahao Huang3 ,
Jianlong Zhou1 & Fang Chen1 ∗
University of Technology Sydney1, SuZhouYierqi2, University of Sydney3
ABSTRACT
The task of identifying multimodal image-text representations has garnered in-
creasing attention, particularly with models such as CLIP (Contrastive Language-
Image Pretraining), which demonstrate exceptional performance in learning com-
plex associations between images and text. Despite these advancements, ensuring
the interpretability of such models is paramount for their safe deployment in real-
world applications, such as healthcare. While numerous interpretability methods
have been developed for unimodal tasks, these approaches often fail to transfer ef-
fectively to multimodal contexts due to inherent differences in the representation
structures. Bottleneck methods, well-established in information theory, have been
applied to enhance CLIP’s interpretability. However, they are often hindered by
strong assumptions or intrinsic randomness. To overcome these challenges, we
propose the Narrowing Information Bottleneck Theory, a novel framework that
fundamentally redefines the traditional bottleneck approach. This theory is specif-
ically designed to satisfy contemporary attribution axioms, providing a more ro-
bust and reliable solution for improving the interpretability of multimodal models.
In our experiments, compared to state-of-the-art methods, our approach enhances
image interpretability by an average of 9%, text interpretability by an average of
58.83%, and accelerates processing speed by 63.95%. Our code is publicly acces-
sible at https://github.com/LMBTough/NIB.
1
INTRODUCTION
CLIP (Contrastive Language-Image Pretraining) has rapidly become a pivotal model in the field
of multimodal learning, especially excelling in its ability to connect th...