ICML 2025

Aligning Multimodal Representations through an Information Bottleneck
Antonio Almud´evar 1 Jos´e Miguel Hern´andez-Lobato 2 Sameer Khurana 3 Ricard Marxer 4 Alfonso Ortega 1
Abstract
Contrastive losses have been extensively used
as a tool for multimodal representation learning.
However, it has been empirically observed that
their use is not effective to learn an aligned rep-
resentation space. In this paper, we argue that
this phenomenon is caused by the presence of
modality-specific information in the representa-
tion space. Although some of the most widely
used contrastive losses maximize the mutual in-
formation between representations of both modali-
ties, they are not designed to remove the modality-
specific information. We give a theoretical de-
scription of this problem through the lens of the
Information Bottleneck Principle. We also em-
pirically analyze how different hyperparameters
affect the emergence of this phenomenon in a con-
trolled experimental setup. Finally, we propose a
regularization term in the loss function that is de-
rived by means of a variational approximation and
aims to increase the representational alignment.
We analyze in a set of controlled experiments and
real-world applications the advantages of includ-
ing this regularization term.
1. Introduction
Multimodal Learning is an area of AI that is focused on pro-
cessing and integrating information from multiple modali-
ties (e.g., text, image or audio). It is becoming a pivotal topic
in the community because of multiple reasons, including,
but not limited to, (i) it permits to mimic human cognition
processes (Fei et al., 2022; Lee et al., 2023); (ii) it allows to
use a greater amount of training data from different modali-
ties, which tends to improve the performance of the models
1ViVoLab, Arag´on Institute for Engineering Research (I3A),
University of Zaragoza, Zaragoza, Spain 2University of Cam-
bridge, Cambridge, UK 3Mitsubishi Electric Research Labora-
tories (MERL), Cambridge, U...