Aligning Multimodal Representations through an Information Bottleneck
Authors: Antonio Almudévar
José Miguel Hernández-Lobato
Sameer Khurana
Ricard Marxer
Alfonso Ortega
Venue: ICML 2025 poster (2025)
Keywords: multimodal representation learning
representational alignment
modality gap
information bottleneck
Primary area: general_machine_learning->representation_learning
TL;DR: Contrastive losses maximize shared information but retain modality-specific content, causing misalignment. We formalize this with information theory and propose a lightweight regularizer to improve cross-modal representation alignment.

Abstract: Contrastive losses have been extensively used as a tool for multimodal representation learning. 
However, it has been empirically observed that their use is not effective to learn an aligned representation space.
In this paper, we argue that this phenomenon is caused by the presence of modality-specific information in the representation space. 
Although some of the most widely used contrastive losses maximize the mutual information between representations of both modalities, they are not designed to remove the modality-specific information.
We give a theoretical description of this problem through the lens of the Information Bottleneck Principle. 
We also empirically analyze how different hyperparameters affect the emergence of this phenomenon in a controlled experimental setup.
Finally, we propose a regularization term in the loss function that is derived by means of a variational approximation and aims to increase the representational alignment.
We analyze in a set of controlled experiments and real-world applications the advantages of including this regularization term.

Aligning Multimodal Representations through an Information Bottleneck
Antonio Almudévar' José Miguel Hernández-Lobato"
Sameer Khurana Ricard Marxer Alfonso Ortega
Abstract
(Kaplan al., 2020; Cuervo & Marxer, 2024); and (iii) it is essential in real-world applications such autonomous vehicles (Xiao et al., 2020), healthcare (Kline et al., 2022) or human-compute interaction (Sinha et al., 2010).
Contrastive losses have been extensively used as a tool for multimodal representation learning. However, it has been empirically observed that their use is not effective to learn an aligned rep- resentation space. In this paper, we argue that this phenomenon is caused by the presence of modality-speciffe information in the representa- tion space. Although some of the most widely used contrastive losses maximize the mutual in- formation between representations of both modali- ties, they are not designed to remove the modality- specific information. We give a theoretical de- scription of this problem through the lens of the Information Bottleneck Principle. We also em- pirically analyze how different hyperparameters affect the emergence of this phenomenon in a con- trolled experimental setup. Finally, we propose a regularization term in the loss function that is de- rived by means of variational approximation and aims to increase the representational alignment. We analyze in a set of controlled experiments and real-world applications the ad advantages of includ- ing this regularization term.
Similarly to humans, most AI systems work through ob- taining intermediate representations, which are compressed versions of the raw data that preserve useful information to solve different downstream tasks (Bengio et al., 2013; Cadieu et al., 2014). One of the most widely used ways of training multimodal systems is Contrastive Representation Learning (Karpathy & Fei-Fei, 2015; Oord et al., 2018; Tian et al., 2020a). In this paradigm, representations correspond- ing to similar input data are brought closer than dissimilar ones. For example, the caption "a photo of a dog" should become closer to an image of a dog than to that of cat. The most widely used of the contrastive losses is the InfoNCE (Oord et al., 2018 1). Minimizing this loss is equivalent to maximizing lower bound of the mutual information of the representations from both modalities. In other words, when minimizing this loss, representations from each modality should maximize the information that they contain about what is common between them.
However, the above does not imply that representations from both modalities contain the same information. Repre- sentations could contain all the information about what is common to both modalities, but still preserve much of the information that is specific to their own modality (a.k.a. nui- sances from now on). We argue that this can translate into a substantial representational misalignment (Klabunde et al., 2023), especially when the inputs contain high level of nui- sances. In other words, representations from two modalities of a positive pair could be not so similar to each other due to the fact that they are encoding different information. Fig- ure illustrates a trivial example in which two similar, yet different, images have exactly the same caption. Thus, the text representations are exactly the same, while the image representations are different from each other, since they can be encoding information about aspects like the color of the dog, the number of clouds in the sky or the number of blades of grass. This misalignment phenomenon has been already observed and denominated modality gap (Liang et al., 2022). However, to the best of our knowledge, the present the first work in which this phenomenon is explained from an information theory perspective.
1. Introduction
Multimodal Learning is area of AI that is focused on pro- cessing and integrating information from multiple modali- ties (e.g., text, image audio). It is becoming pivotal topic in the community because of multiple reasons, including, but not limited to, (i) it permits to mimic human cognition processes (Fei et al., 2022; Lee et al., 2023); (ii) it allows to use a greater amount of training data from different modali- ties, which tends to improve the performance of the models
'viVoLab, Aragón Institute for Engineering Research (I3A), University of Zaragoza, Zaragoza, Spain UUniversity of Cam- bridge. Cambridge. UK 'Mitsubishi Electric Research Labora- tories (MERL). Cambridge, USA "nniversité de Toulon, Aix Mar- seille Univ, CNRS, LIS, Toulon, France. Correspondence to: An- tonio Almudévar <almudevar@ unizarees>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Aligning Multimodal Representations through an Information Bottleneck
modalities (e.g- image and text) into joint representa- tion space. Some foundation works used rank-based losses (Yager, 1988; Usunier et al., 2009; Schroff et al., 2015) to learn multimodal representation spaces (Weston et al., 2010; Frome et al., 2013; Karpathy & Fei-Fei, 2015) while more modern approaches have used the InfoNCE loss (Tian et al., 2020a; Radford et al., 2021; Jia et al., 2021; Xu et al., 2021; Girdhar et al., 2023) for this purpose. However, has been observed that, when trained in contrastive way, rep- resentations from different modalities tend to be located in different regions of the space, phenomenon called modal- ity gap (Liang et al., 2022; Udandarao, 2022; Ramasinghe et al., 2024; Fahim et al., 2024; Schrodi et al., 2024). This phenomenon can an issue in some applications such as Image Captioning or Visual Question Answering, so sophis- ticated training methods have been proposed to palliate it (Chen al., 2020b; Li et al., 2021; 2022: 2023).
ft
fi
A Labrador Retriever
A Labrador Retriever sitting on grassy field
silting grassy field
(1)
(2)
3
Figure 1: Different modalities usually contain different in- formation. A trivial example this is the case in which two different images have exactly the same caption. As a consequence of this, representations from different modal- ities tend contain different information (thus leading to misalignment) if the opposite is not explicitly imposed.
Measuring Representational Alignment The represen- tational alignment (or similarity) typically measured through kernel alignment metries (Cristianini al., 2001; Cortes et al., 2012). Examples of these include Centered Kernel Alignment (CKA) (Kornblith et al., 2019), SVCCA (Raghu et al., 2017) and nearest- et al., 2023). However, in this work we restrict our attention to the former, since it is the most widely used for this pur- pose. Let 7(a) E TT. ng and Z E Ruxdd be two sets of representations, K = k(2l") 2 (0) and L = [(2, (8) (1)) where k dR Rd. + R and / 18da IRd= E are kemel functions. Then, the Hilbert-Schmite Independence Criterion is defined as:
It is precisely this explanation which allows us to propose a solution to this phenomenon. Concretely, we propose to apply an Information Bottleneck (IB) (Tishby et al., 2000) in the representation space. With this, apart from maximiz- ing the mutual information between the representations of both modalities through the contrastive loss, we reduce the nuisances that can be found in the representations. This IB is applied by means of a regularization term in the loss func- tion that is derived using variational approximation. Thus, is efficient, straightforward to implement and modality- agnostic, which is advantageous over alternative approaches (Li al., 2021).
metrics (Klabunde
1
HSIC L)
Tr (KHLH)
(I)
(n - 1)2
where H = In - -11" T is the centering matrix. Then, the CKA defined as follows:
2. Preliminaries
Contrastive Representation Learning (CRL) This compasses set of techniques that learns a representation space in which representations of similar inputs are closer than those of dissimilar ones. It has emerged as one of the most competitive methods for learning representations with- out labels in a supervieee way (Oord et al., 2018; Hjelm Lee, 2018:
HSIC(K, L)
CKA(K, = VHSIC(K, KYHSIC((L,L)
(2)
This metric ranges between zero and one and we say that a pair of representations is perfectly aligned when CKA 1.
et al., 2018; Wu et al., 2018; Logeswaran Bachman al., 2019 Tian al., 2020a; Chen et al., 2020a; Henaff, 2020). The most widely used among the contrastive losses is the InfoNCE (Oord et al., 2018) and it has been shown that minimizing this is equivalent to maximizing lower bound of the mutual information (MI) between a pair of representations (Bachman et al., 2019; Tian et al., 2020a).
Information Bottleneck It is a framework that aims to find a representation that contains all the information in an input that is necessary to solve a given task, while dis- carding irrelevant information (Tishby et al., 2000; Tishby & Zaslavsky, 2015). Given an input X, a task Y and a representation Z, the goal can be formulated as:
max - (I
Multimodal Contrastive Representation Learning One of the tasks in which contrastive losses have gained popu- larity is Multimodal Representation Learning, which con- sists in designing systems that map inputs from different
where 3 is a Lagrange multiplier that controls the trade-off between compression and preserving the information that is relevant for the task Y.
2

---

Aligning Multimodal Representations through an Information Bottleneck
3. On the Importance of Minimal Sufficient Representations
Xa
Na
Y
N₂
X
To formulate our hypothesis, we assume that the data of modality are composed of an essence, which is all that infor- mation that can be found in both modalities; and nuisances, which refers to all that information that can be found only in one modality. addition, our goal is to obtain representa- tions of the inputs of each modality. Next, we explain more in detail these concepts. All the proofs of the Lemmas and Theorems can be found in Appendix A
Figure Diagram of the inputs, essence and nuisances
Given representation Za of Xa, the following Markov chains are satisfied:
Y Xa t Za
(8)
No ++ Xo Zo
3.1. Input Data: Essence and Nuisances
Definition 1. Let (Xa, be pair of positive inputs from modalities a and B. respectively. We call essence to a variable Y that satisfies the following Markov Chains:
The goal of representation learning is to obtain representa- tions with desirable properties for the problem to be solved. In the case of multimodal learning we consider two prop- erties to be desirable: (i) sufficiency and (ii) minimality (Achille & Soatto, 20;ba; We define these properties and argue their desirability next.
X3 + Y ++ X9 + Xa
(4)
(5)
Definition 4. Given a representation Z, of the input Xa and an essence Y. We call Za sufficient if it satisfies:
i.e., it refers to the common part of a positive pair of data. Although there exists more than one essence (infinite, in fact), all of them are equivalent under a one-to-one transfor- mation, which is formalized next.
I(Z,; Y)
(10)
i.e., representation is sufficient if it preserves the essence in its entirety or, equivalently, if it preserves all the infor- mation that is common to both modalities. Because of Equation (8) we know by the Data Processing Inequality (DPI) that I(Z:: Y) S I(Xa: i.e I(Xa;Y) is an upper bound of I(Z,; Y). Thus, the objective to be optimized to obtain sufficient representation is:
Lemma Let and Y' be essences of the same pair of modalities. Then, there exist a one-to-one transformation Y such that Y = (YY).
Equivalently, the partitions of Xa and X33 created by Y are unique. We note that the essence Y is a variable that we define help us with the formulation, but our goal is not to discover Y itself, but the partition of the input set that it creates. For example, if we have two images with the same caption, we will consider that both images are equivalent in the sense that they belong to common subset of the images set, but we are not interested defining the subset. Thus, from now on we will refer to the essence it were unique variable.
max I I(Za;)) Za
(11)
Informally, sufficiency is connected to the performance of our representations in downstream tasks. Our representa- tions must have all the information in the essence to per- feetly solve all the tasks that can be derived from the essence. Here, we assume that tasks that are not in the essence can- not be solved. For example, if we have set of images of dogs and set of captions describing different aspects of them except the color, we cannot expect the text encoder to be able to understand the word "brown" even if there are brown dogs in our set of images. We formalize this next.
Definition 2. Let X. be an input of modality a, the essence of X, with respect to another modality. We call variable Na that satisfies:
muisance of modality
N.Y; 00
(6)
I(Xa: N.) = (Xx)) HYY) = H(Na)
(7)
Theorem 1. Let Y and Za be the essence and a representa- tion of input Xa respectively, and let T = (T T = f(Y)} be the set of all the deterministic functions of (i.e., all the tasks derived from Y). Then, we have that:
i.., it refers to all information from Xa that cannot be found in the other modality.
Diagram in Figure 2 schematizes the relationships between all the elements described in this section.
p(t|2.) p(t]aa) VT I(Za; I(Xa; Y)
3.2. Representations
Definition Given a representation Za and the nuisances Na of an input X,.. We call Za minimal if it satisfies:
Definition 3. We say that a variable Z, is a representation of an input Xa if Za is a stochastic function of Xa or, equivalently, if Za is fully defined by p(za (ta)
T(Za; No) = 0
(12)
3

---

Aligning Multimodal Representations through an Information Bottleneck
i..,, a representation is minimal if it eliminates all the nui- sances of its modality or, in other words, if all the informa- tion that it contains can also be found in the input of the other modality. Since mutual information is non- non-negatvve, the objective to be optimized in order to obtain a minimal representation is:
and alignment. We describe next a method obtain them, which connects to the Information Bottleneck principle.
4.1. Obtaining Sufficient Representations
Equation (11) shows that I(Z,; must be maximized to find a representation Za that is sufficient. Since the essence Y is variable that we have defined for our formulation whose distribution is unknown, caleulating this term could seem problematic. However, in Appendix A.4 we show that T(Za T(Zai X3). Thus, our objective becomes
min I [Za::
(13)
Zo
Informally, minimality is connected to representational alignment. As explained in section 2, we can have rep- resentations with good performance in a wide variety of downstream tasks but misaligned. Intuitively, even if two representations Za and Za have all the information about the essence (i.e., they are sufficient), if they also contain information about nuisances (i.e., they are not minimal), then the information they are encoding is different and, consequently, they will be misaligned. Revisiting the previ- ous example, the sufficient representations of an image of a yellow Labrador Retriever and of an image of a brown Labrador Retriever could be different from each other, since they can be coding different information. However, the suf- ficient representations corresponding to the captions of each image will be presumably the same, since both images have presumably the same caption. Therefore, the presence of nuisances in the representations could cause misalignment, as stated next.
max I ((a Xz) Za
(14)
Computing exactly 1(2o; X3) is in general intractable, since it involves integrating over the entire space of B- modality inputs. However, we can obtain a lower bound of [(a:: Xg): since Zg is a representation of X3, we have that Za ++ X8 <>> Z3 and, by the DPI, follows that I [Zai Eg) I 1(Zai Xg). That is, (Zz; Zg) is a lower bound of ((z;Y): Analogously, [Zg; Za) I I [ZZs Y), so given the symmetry of the mutual information, must maximize T T(Za; Zz) in order to jointly maximize [(Za; Y) and [(2;; Y). Thus, the objective becomes:
max I(Za; 2z) Z..28
(15)
Theorem 2 (Informal). Let Zn, and Za be the representation of some inputs with nuisances Na and Ng, respectively, such that Za and Za are aligned in the sense of equation (2). Then, [Za: Na) = I(Zz:Np) =
Again, computing exactly I(Za; requires integrating over the representation spaces, which is in general in- tractable. However, as explained in section 2, minimizing InfoNCE loss is equivalent to maximizing a lower bound of I(Za: Zz). Thus, encoders optimized through InfoNCE lend to give sufficient representations. However, the result- ing representations are not necessarily minimal due the fact that this loss imposes no conditions on I [Za: Na). We derive in the next section a term that aims to increase the degree of minimality of the representations.
Summarizing the above, we have that our ideal representa- tion should be a minimal sufficient statistic for Y. Equiva- lently, it should contain only (minimal) all (sufficient) the information that is common to both modalities (a.k.a. the essence). In this scenario, similarly Lemma we know that all the ideal representations create the same partition of the input. This connects to the "Anna Karenina principle" that has been mentioned in different works in representation learning to hypothesize that all the well-perforning mod- els learn roughly the same internal representations (Bansal et al., 2021; Huh et al., 2024). Here, all the minimal suffi- cient ("happy") representations ("families") create the same partition of the input ("are alike"). ).
4.2. Obtaining Minimal Representations
Equation (13) shows that [Z,,; N.) must be minimized to obtain a representation Z. that is minimal. Since No an abstract concept whose distribution is unknown, calculating this term could seem problematic. However, by the DPI and equation (9), we have that I I(Za; Na.) < I(Za; Thus, the objective to obtain a minimal representation becomes:
4. Obtaining Minimal Sufficient Representations
min I(Zo; X.) Zo
(16)
We have discussed in the previous section the importance of minimal sufficient representations for good performance
Again, computing exactly I(.,; X.) requires integrating over the representation and input spaces, which is intractable. However, we demonstrate in Appendix A.5 that:
1 "All happy familics are alike; each unhappy family is unhappy in its own way." (Tolstoy. 1877). This principle was popularized in (Diamond & Ordunio, 1999) to illustrate why only a small number of wild animals have been successfully domesticated over the course of history.
I(2o; Xa) S E p(pu=70)
[DKr. (pe, (=|w.))le Ipp,, (2 [x][s))]
(17)
4

---

Aligning Multimodal Representations through an Information Bottleneck
Therefore, we can minimize the given upper bound to mini- mize T( Xa). That is, the distributions of the represen- tations of positive pair of data from different modalities should be as similar as possible. Intuitively, if Za and 2 are equal, then they can be affected only by the essence but not by the nuisances.
5.1. Does the contrastive loss alone remove nuisances?
To answer this question we propose several scenarios per dataset. In each scenario provide all but one factor to the encoder, i.c,, the nuisances of the image modality Na are that missing factor. Thus, if the contrastive loss were eliminating the nuisances, then the image represen- tations Za should contain no information about Na. i.e., I (Za; N.) = O. We calculate a lower bound of this mutual information i(Z., Na.) by training a linear classifier from Zo to N.. following (Xu al., 2020). We show in Table I the values of 1(2o:No) for each dataset and category of fac- H(Na)
Spherical Gaussian Case The upper bound of equation (17) does not have a closed form in general. However, it is common to assume in practice that the representa- tions distributions given the input are Gaussian, in which case, KL. Divergence becomes tractable. In the case in which pa. (z|xa) = N (zi nie (Fa), o² and pa, (z|æp) = Nf (2 Po, (Fz), 0²T), as shown in Appendix A.6, we reach:
tors². This value encodes a lower bound of the ratio of the uncertainty of N₂ that is reduced by observing Zn and its value ranges from 0 to 1 (we call it uncertainty reduction ra- tio or simply URR), so if the contrastive loss alone removed all the nuisances, its value would be zero. We can extract the following conclusions from this: (i) non-negligible amount of information about the missing factors is present in the image representation for every category; (ii) image encoder preserves more information on some categories than on others; and (iii) some categories are almost equally conserved among the datasets. We hypothesize that the last two points could be due to the inductive biases of the convolutional architecture of the image encoder (Cohen & Shashua, 2016; Mitchell, 2017; Wang & Wu, 2023), but exploring this point is out of the scope of this work.
E p(Zantz)
[DK (po. (z)xxa) ) Ipo,, (z [=]r x
(18)
[1lvo. (xa) - Hog (x [1))) = LM
E p(za.tp)
4.3. Information Bottleneck for two Modalities
Combining equations (14) and (16), the objective to obtain a representation Za that is sufficient and minimal becomes:
max ( (Zo: X3) - (1. Xa) 2n
(19)
This is equivalent to an information bottleneck in which the task is the input of the other modality Xg. That is, Za must retain only all the information that is common between X, and X3- The same applies for Zz.. Combining equations (15) and (18), we have that this is equivalent to minimizing:
Table I: URR (in percentages) for each dataset and category. Some factors not used because they do not fall into any category.
= LInfoNCE + BLas
(20)
DSprites
MP13D
Shapes3I
Location
16.1 3.7
12.4 7.3
8.5 : 0.1
5. Toy Experiment
Shape Size Objects Color
77.1 4.6
10.3 + 1.0
8.7 0.4
66.3 3.1
07.8
7.2 0.5
The objectives of this experiment are to empirically vali- date the different statements made throughout the previous sections and understand the relations between the different elements of our formulation. For this purpose, we use some datasets typically employed in disentanglement related tasks ( (Wang et al., 2024). Concretely, DSprites (Matthey et al., 2017), MPI3D (Gondal et al., 2019) and Shapes3D (Burgess & Kim, 2018) are used. These datasets contain images and labels that represent multiple independent factors of varia- tion. We jointly train an image encoder and factors encoder (i.e., images and factors are the two modalities). The reason to use these datasets is that we can control the amount of factors that we input to the encoder, thus controlling the information imbalance between both modalities. Unless otherwise stated, a ResNet20 (He et al., 2016) is used as image encoder, an MLP as encoder for the factors and temperature in the InfoNCE loss is a trainable parameter initialized 0.07. More details are given in Appendix C.
68.8 + 2.3
54.1 + 2.0
Not all architectures remove nuisances to the same extent It is well established that di different neural architectures intro- duce distinct inductive biases (Raghu et al., 2021). Conse- quently, the extent to which nuisance factors are retained in the learned representations can vary depending on the model architecture. To investigate this, we replicate the previous experiment using a small Vision Transformer (ViT) (Doso- vitskiy et al., 2020) as the image encoder. Table 2 reveals two key observations: (i) local information -aataatioonaattullaaral Lo- cation- is less preserved in ViTs, likely due to their global attention mechanisms favoring long-range dependencies; and (ii) more global features- such as Color-are compara- bly preserved in both convolutional and transformer-bant based models. We emphasize that these trends may also depend
We organize the factors into categories for ease of understand- ing of the conclusions. Information what factors each category is composed of is provided in Appendix C.
We encode the factors using hot-
5

---

Aligning Multimodal Representations through an Information Bottleneck
on other architectural choices, such as the encoder depth, explored in the following paragraph.
DSprites Posx
MPL3D Object Size
Shapes30 Scale
MP13D Object Golor
MPI33D Camera Height
10
Table 2: URR (in percentages) for each dataset and category for ViT-based encoder.
DSprites
MPI3D
Shapes3D
10-1
Location
2.8 0.6
2.8
1.1 + 0.1
Shape
64.9
0.4
8.7 + 0.2
Size
30.7+
20.8
6.9 + 1.5
0.005
0.01
0.02
0.05
0.2
1.0
Objects Color
63.5
53.5 + 1.6
Figure 4: URR (y-axis) for different values of temperature (x-axis).
Deeper neural encoders remove more nuisances It has been argued that the success of Deep Learning can be plained through the fact that deep neural networks implicitly introduce an Information Bottleneck (Tishby& & Zaslavsky, 2015; Shwartz-Zivv & Tishby, 2017). Intuitively, determinis- tic layers tend to remove information from the input because of the DPI. Thus, when the number of layers grows, the out- put of the neural network tends to be more pruned version of the input but that preserves the information that is nec- essary to solve different downstream tasks (Alemi et al., 2016). We hypothesize then that the use of deeper neural encoders will tend to remove more nuisances. Effectively, we can observe in Figure a trend among different factors in which deeper encoders remove more modality specific information. This can serve as an explanation for The Ca- pacity Hypothesis stated in (Huh al., 2024), which says that bigger models more likely converge to a shared representation than smaller models. We hypothesize that this representation is shared because it contains little infor- mation about the nuisances.
5.2. Does the presence of nuisances in the representation negatively correlate with the level of alignment?
As informally demonstrated in section 3.2, the fact that two representations are minimal is necessary condition for them to be aligned. We hypothesize that misalignment is just an effect of an information imbalance in the representation space. To empirically analyze this phenomenon, we design an experiment similar to the previous one. In this case, more than one factor can be removed, i.e., No can be a set of factors. We generate 100 scenarios per dataset in which randomly chosen subset of factors Na is not provided as input to the factors encoder. Similarly to the previous experiment, we calculate i i(Za; Na) and the CK A metric. In Figure 5 it is shown that, for the three datasets, there exists a negative correlation between [(Z,: N,) and the alignment value.
Sprites
MP3D
Shapes3D
10
0.6
10
0.4
10
0.2
10
0.0
2
20
32
44
56
110
7
8
Figure 3: URR (y-axis) for different number of layers of the image neural encoder (x-axis). Same legend as Figure 4.
Figure 5: Alignment (y-axis) vs. i [Zo; N₂) (x-axis).
5.3. Does our regularization term effectively increase the alignment level?
Higher temperatures remove more nuisances Wang & Liu (2021) observed that the value of the temperature in the InfoNCE. loss considerably impacts on the entropy level of the representations. As stated in section 3.2, alignment is closely related to the level of nuisances in the representa- tions and, consequently, to their entropy. We run an experi- ment identical to the previous one for some factors in which the temperature is fixed. Its results are shown in Figure 4 and we observe that (i) higher values of temperature tend to remove more nuisances and (ii) not all the factors are equally affected by the changes in temperature.
To analyze this, we randomly select, for each dataset, 10 of the 100 scenarios above and we train the encoders for differ- ent values of 3. In all of them we set the temperature fixed to 0.01. We show in Figure 6 the value of different mea- sures for different values of 3. We can extract the following conclusions from this: (i) lower values of B retain better the essence (Fig. 6a), since increasing 1 (2.: : Y) prevails over decreasing I(Z,; N.): (ii) lower values of / also tend to retain more nuisances, since more entropic representations are encouraged in this case (Fig. 6b); (iii) higher values of
6

---

Aligning Multimodal Representations through an Information Bottleneck
0.10
0.2
0.10
0.2
0.05
0.00
-0.05
-0.10
0.10
-0.6
-0.1
-0.20
-0.20
0.03
0.3
0.03
1.0 0.01
1.0
3.0
(a) Y)
(b) ((aa;Na))
(c) Accuracy of Y
(d) Alignment
Figure 6: Relative change (with respect to the case in which 3 = 0) of different measures (y-axis) vs. B (x-axis). The
temperature equal 0.01 in all the cases. Same legend Figure 5.
3 remove more nuisances, since they favor the decreasing of I(Zx; N&) (Fig. 6b); (iv) higher values of 33 also tend to discard more information the essence, since they promote less entropic representations (Fig. 6a); (v) representations with lower [(Za: Y) result in lower accuracy (Figs, 6a and 6c), stated in Theorem I; and (vi) representations with lower [ (Zai Na) result in a higher alignment (Figs. 6b and 6d), as stated in Theorem 2.
0.4
0.50
0.25
0.00
0.0
0.25
-0.2
0.75
On the Information Homeostasis of the representations In the previous experiment the temperature has been set fixed. However, as shown in Figure 4, lower temperatures tend to preserve more information of the nuisances. Thus, the next question arises: Will the lemperature be affected by the value of B? To answer it, repeat the previous experiment but setting the temperature as trainable param- eter. We show the results in Figure 7 and we observe that: (i) temperature tends to become lower when higher values of are used (Fig. 7a); and (ii) this translates into the fact that muisances tend not to be eliminated the same extent as in the case in which the temperature is fixed (Fig. 6b 7b). We call this phenomenon Information Homeostasis, since it seems that, when an external stimulus (i.e., increasing 3) affects the encoder, this activates available mechanisms (i.e., decreasing the temperature) order to preserve to the extent possible the entropy of the representations (DelMonte Kim, 2011) 1). This effect becomes more pronounced for the highest values of B. In these cases, the level of nuisances is similar the case in which 3 = 0, which reminds of homeostatic range (Kotas& & Medzhitov, 2015). This is intriguing phenomenon that is out of the scope of this work.
(a) Temperature
(b) i(Za;
Figure 7: Relative change (with respect to the case in which 3 = of different measures (y-axis) vs. B (x-axis). Tem- perature a trainable parameter. Same legend Figure 5.
between image and text representations: and (ii) language model loss (LM), which trains the Q-Former to generate text through the LLM using images as the condition. How- ever, none of these losses explicitly encourages high rep- resentational alignment. Thus, we experiment by adding: (i) an image-text matching loss (TTM), which the binary cross-entropy loss of task in which model must predict if an image-text pair positive or negative (Chen al., 2020b); or (ii) our regularization term in equation (20). We note that, in contrast ITM, our regularization term is modality-agnostie computationally efficient and straightfor- ward to implement. COCO (Lin et al., 2014) is used to train and test our model. More details are given in Appendix C.
6.1. Image Captioning
We argue that, for optimal performance in image caption- ing, the learned image representations should contain as little information as possible about nuisance factors. When nuisance information is retained, representations may en- code fine-grained visual details that the text decoder not equipped to handle, as it has not been trained to exploit such information. Table 3 summarizes the performance of different models. We observe the following trends: (i) loss functions that promote alignment between modalities tend to improve image captioning performance; (ii) there is a trade-off between image captioning and retrieval: caption-
6. World Applications
Several real-world applications benefits from aligned repre- sentations. These include those that consist in generating modality from another. We analyze what are the implica- tions of introducing our regularization term in a real-world scenario. Concretely, we train Q-Former with a frozen decoder-basee based LLM (Li et al., 2023) with different loss functions. all the cases, two terms are present: (i) an image-text contrastive loss (ITC), i.e., the InfoNCE loss

---

Aligning Multimodal Representations through an Information Bottleneck
Table 3: CIDEr (Vedantam et al., 2015), BLEU@ (Papineni et al., 2002) and retrieval accuracy for Q-Formers trained with
different loss functions.
CIDEr
BLEU@4
12T R@1
T21 R@1
ITC+LM ITC+LM+ITM
91.7 + 0.2
28.6 + 0.1
64.2 + 0.2
52.3 + 0.4
91.8 = 0.5
28.8 + 0.2
61.4 + 0.6
49.7 + 0.8
ITC+LM+0.01&M
92.3 + 0.8
29.1 + 0.4
64.0 + 0.3
52.3 0.5
ITC+LM+0.03CM
92.6 = 0.3
29.2 = 0.2
63.9 0.4
52.1 + 0.5
1
93.0 + 0.3
29.4 + 0.3
63.0 0.5
50.4 + 0.5
ITC+LM+0.3CM
90.5 + 0.4
28.5 + 0.2
59.6 1 0.4
47.1 + 0.4
ing benefits from minimal representations, whereas suffi- cient representations favors retrieval (as is a downstream task); (iii) loss slightly improves captioning performance for low values of 3, with minimal degradation in retrieval performance; (iv) for moderate values of 3, our loss sub- stantially enhances captioning performance, though at a higher cost to retrieval accuracy; and (v) at high values of 3, captioning and retrieval performances drop sharply, as representations become overly compressed and fail to retain sufficient task-relevant information morimattoo-mmirrrinng the trend found in Figure 6.
with 3 0.01. Examples including other loss functions are found Appendix E, showing that those not encouraging alignment, result in worse multimodal retrievals.
C
"sitting"
"snowy mountain"
white fire hydrant sitting on the side of road"
We also show in Figure 8 and in Appendix D some of the captions generated by the different models in Table 3. We observe that, in those cases which representations are less aligned, captions tend be more entropic because the representations are as well. This, in some cases, translates into captions that have information that does correspond to the image. From geometric point of view, we believe that in a misaligned space, for example, the representations of "lots of trees" and "a tree" are closer in the image than in the text space and, thus, the text decoder "confuses" them.
Figure 9: Multimodal image retrieval for 3 0.01.
7. Related Work
Information Bottleneck and Contrastive Representation
Learning Other works have previously connected IB to CRL, especially in the context of multi-view learning. In (Tian et al., 2020b), the authors argue that in CRL good views for given task are those that optimize IB w.r.t. that given task. Federici al. (2020), analogously us, propose loss function obtain representations that retain only the information shared by the two views, which is considered to be the relevant for downstream tasks. Tsai et al. (2020) build on the previous one and argue that including a reconstruction loss encourages to preserve the downstream task-relevant information. (Wang et al., 2022) is argued that, in the multi-view setting, imposing a strong IB can be detrimental for downstream tasks, since it could be removing an excess of information in the representation. However, to the best of our knowledge, the use of the IB in the multimodal setting remains understudied and the present the first work that explores this and connects it to the misalignment typically present between representations from different modalities.
(1): a small airplane flying over
(1): cat is laying down on a bed while blanket
forest lots trees
(2): small airplane flying through
(2): black and white cat laying
blue sky above trees
on a red blanket
small plane flying through
black and white laying
blue sky above tree
on red blanket
(5): small plane flying through
(5) black wnd white cat laying on bed
blue sky above tree
Figure 8: Captions generated by some of the trained models. Numbers correspondence is the same as in Table 3.
6.2. Multimodal Representation Space Arithmetic
Figure shows image retrievals obtained from combining image and text representations from the Q-Former trained
8

---

Aligning Multimodal Representations through an Information Bottleneck
Alignment of Multimodal Contrastive Learned Repre-
Acknowledgements
sentations This is well-studied field but still marked by great amount of unanswered questions. The first work that extensively studied the alignment in CRL was (Wang & Isola, 2020) and demonstrated that, under infinite neg- ative samples, the InfoNCE is globally minimized if the representations are perfectly aligned. They also define the representational alignment as in equation (17), which makes our formulation consistent with this work. However, it was observed in (Liang et al., 2022) that there exists in practice great misalignment between representations from different modalities. This misalignment becomes problematic for tasks that need to combine both modalities. For example, Chen et al. (2020b); Li et al. (2021; 2022; 2023) use modifi- cations to the InfoNCE loss to obtain a better performance in tasks in which image representations serve to obtain text, such as Image Captioning or Visual Question Answering. In the the opposite direction, Ramesh et al. (2022) use generative model to transform text representations to image representations to train a text-conditioned image generator. In our view, this generative model serves to increase the entropy of the text representations to balance them with the image representations, which are typically more entropic. To the best of our knowledge, (Schrodi et al., 2024) is the first work in which this phenomenon is explained through the lens of an information imbalance. However, this im- balance is analyzed in the input space rather than in the representation space. Thus, aspects such as the encoder depth and hyperparameters or modifications of the loss func- tion are not explored.
This work has received funding from the European Union's s Horizon 2020 research and innovation programme under the Marie Sktodowska-Curt grant agreement No 101007666, MCIN/AEE/IO. 13039/9/0000000001111010 100011033 under Grant PID2021- 126061 01-C44 and the Government of Aragón (Grant Group T36 23R). We are also grateful to the French National Research Agency for their support through the ANR-20- CE23-0012-01 (MIM) grant.
This work originated during the ISALT 2024 workshop. We gratefully acknowledge the researchers, mentors, and collaborators who took part in the workshop, well the staff at the Center for Language and Speech Processing at Johns Hopkins University, for cultivating a collaborative and intellectually rich environment that was instrumental in shaping this research. The workshop was partially supported by generous contributions from Amazon, Facebook, Google, and Microsoft.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Achille, A. and Soatto, S. Emergence of invariance and disentanglement in deep representations. Journal of Ma- chine Learning Research, 19(50):1334 2018a.
8. Conclusions
Achille, A. and Soatto, S. Information dropout: Learning optimal representations through noisy computation. IEEE Iransactions on pattern analysis and machine intelligence, 40(12):2897-2005 2018b
We give an explanation to the phenomenon of multimodal misalignment that usually emerges in encoders trained to minimize contrastive losses. These are designed to obtain representations that preserve the information about what is common to both modalities, but not to remove modality- specific information. We theoretically and empirically show that the presence of this modality-specifie information in the representations is correlated with the misalignment phe- nomenon. We also examine the impact that different hyper- parameters such as the temperature or encoder depth have is removed. We
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. arXiv preprint arXh::1612.004100 2016.
Bachman, P., Hjelm, R. D., and Buchwalter, W. Learn- ing representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
on how much of this modality-informaan derive a term that can be added the contrastive loss which aims to eliminate this modality-speciff information and, thus, allows to obtain a more aligned representation space. We find a phenomenon that we call Information Homeosta- sis, which consists in the fact that encoders seem to prefer representations with more nuisances and they modify, if possible, some of their internal parameters for this purpose. Finally, we show that our term in the loss function translates into a better performance in image captioning and seems to result in more consistent multimodal image retrievals.
Bansal, Y., Nakkiran, P., and Barak, B. Revisiting model stitching to compare neural representations. Advances in neural information processing systems, 34:225-236, 2021.
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE transac- tions on pattern analysis and machine intelligence, 35(8): 7798-1828. 2013.

---

Aligning Multimodal Representations through an Information Bottleneck
Burgess, C. and Kim, H. 3d shapes dataset. htts://github.com/// http://iww....oo/////////a//aaae
Federici, M., Dutta, A., Forré, P., Kushman, N., and Akata, Z. Learning robust representations via multi-view in- formation bottleneck. arXiv preprint arXiv:2002.07017 2020.
2018.
Cadieu, C. F., Hong, H., Yamins, D. L., Pinto, N., Ardila, D., Solomon, E. A., Majaj, N. J., and DiCarlo, J. Deep neu- ral networks rival the representation of primate it cortex for core visual object recognition. PLoS computational biology, 10032)3 1009633, 2014.
Fei, N., Lu, Z., Gao, Y., Yang. G., Huo, Y., Wen, J., Lu, H., Song, R.,. Gao, X., Xiang, T., et al. Towards artificial general intelligence via a multimodal foundation model. Nature Communications, 13(1):3094, 2022.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Frome, A., Corrado, G. S., Shlens. J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, T. Devise: deep visual- semantic embedding model. Advances in neural informa- tion processing systems, 26, 2013.
Chen, Y.-C., Li, L., Yu, L., EI Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. Uniter: Universal image-text repre- sentation learning. In European conference computer vision, pp. 104-120. Springer, 2020b.
Girdhar, R., El-Nouby, A., Liu, Z.,, Singh, M., Alwala, K. V., Joulin, A., and Misra, I. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Cohen, N. and Shashua, A. Inductive bias of deep convolu- tional networks through pooling geometry. arXiv preprint 2016.
Gondal, M. W., Wuthrich, M., Miladinovic, D., Locatello, F., Breidt, M., Volchkov, V., Akpo, J., Bachem, O., Schölkopf, B., and Bauer, S. On the transfer of inductive bias from simulation to the real world: a new disentangle- ment dataset. Advances in Neural Information Processing Systems, 32, 2019.
Cortes, C., Mohri, M., and Rostamizadeh, A. Algorithms for learning kernels based on centered alignment. The Jour- nal of Machine Learning Research, 13:795-828, 2012.
Cristianini, N., Taylor,,y J., Elisseeff, A., and Kan- dola, J. On tarne-ttaree alignment. Advances neural information processing systems, 14, 2001.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778 770,777770 2016.
Cuervo, S. and Marxer, R. Scaling properties of speech lan- guage models. arXiv preprint arXiv:2404.0068 00685, 2024.
DelMonte, D. W. and Kim, T. Anatomy and physiology of the cornea. Journal of Cataract & Refractive Surgery, 37 (3):588 598, 2011.
Henaff, O. Data-efficient image recognition with contrastive predictive coding. In International conference on ma- chine learning, pp. 4182-4192, PMLR, 2020.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv: 1810,0885, 2018.
Hjelm, R. D., Fedorov, A.. Lavoie-Marchildon S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. arXiv preprint / /808.06670, 2018.
Diamond, J. M. and Ordunio, D. Guns, germs, and steel, volume 521. Books on Tape New York, 1999.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghami, M., Minderer, M., Heigold, G., Gelly, S., al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.119299 2020.
Huh, M., Cheung, B., Wang, T. and Isola, P. The platonic representation hypothesis. arXiv preprint S0Ni::4405009950 2024.
Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, 7., Pham, H., Le, Q., Sung. Y.-H., Li, Z., and Duerig, T. Scaling up visual and vision- laiiuala representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Fahim, A., Murphy, A., and Fyshe, A. Its not modality gap: Characterizing and addressing the contrastive gap. arXiv preprint 2405 18570, 2024.
Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, I.., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19358-19369, 2023.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001. 083611, 2020.
10

---

Aligning Multimodal Representations through an Information Bottleneck
Karpathy, A. and Fei-Fei, L. Deep visual-semantic align- ments for generating image descriptions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3128-3137, 2015.
Logeswaran, L. and Lee, H. An efficient framework for learning sentence representations. arXiv preprint arXiv: (0w::8803.00 803.02893, 2018.
Matthey, L., Higgins, I., Hassabis, D., and Lerchner, A. dsprites: Disentanglement testing sprites dataset. http://giiibb.oom//ccepn///di/ip dattasst,, 2017.
Klabunde, M., Schumacher, T., Strohmaier, M., and Lem- merich, Similarity of neural network models: sur- vey of functional and representational measures. arXiv preprint arXiv:2305.063322 063299 2023.
Mitchell, B. R. The spatial inductive bias of deep learning.
PhD thesis, Johns Hopkins University, 2017.
Kline, A., Wang, H., Li, Y.. Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., and Luo, Y. Multimodal machine learning in precision health: scoping review. npj Digital Medicine, 5(1):171, 2022.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv: 807.03748, 2018.
Korbliihh, S., Norouzi, M., Lee, H.,, and Hinton, G. Sim- ilarity of neural network representations revisited. In International conference machine learning, pp. 3519- 3529, PMLR, 2019.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguisties, pp. 311-318, 2002.
Kotas, M. E. and Medzhitov, R. Homeostasis, inflammation, and disease susceptibility. Cell, 160(5):816-82) 2015.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Lee, G.-G., Shi, L., Latif, E., Gao, Y., Bewersdorff, A., Nyaaba, M., Guo, S., Wu, Z., Liu, Z., Wang, H., et al. Multimodality of ai for education: Towards artificial gen- eral intelligence. arXiv preprint arXiv:2312.060377 2023.
Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Li, J., Selvaraju, R., Gotmare. A., Joty, S., Xiong, C., and Hoi, S. C. H. Align before fuse: Vision and language representation learning with momentum distillation. Ad- vances in neural information processing systems, 34: 9694-9705, 2021.
Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., and Dosovitskiy, A. Do vision transformers see like convolu- tional neural networks? Advances in neural information processing systems, 34:121 161112228-2228 2021.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-ing image pre-training for unified vision- languae understanding and generation. In International confer- ence machine learning. pp. 18980. PMLR, 2022.
Ramasinghe, S., Shevchenko, V., Avraham, G., and Tha- laiyasingam, A. Accept the modality gap: An exploration in the hyperbolic space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27263-27273 2024.
Ji D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping minguage-imagee pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2):3 2022.
Liang, V. W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Y. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Ad- vances in Neural Information Processing Systems, 35: 17612-17625, 2022.
Schrodi, S., Hoffmann, D. T., Argus, M., Fischer, V., and Brox, Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision- vision-language representation learning. arXiv preprint arXiv:2404.07988) 2024.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. I.. Microsoft coco: Common objects in context. In Computer Visi 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815-8233 2015.

---

Aligning Multimodal Representations through an Information Bottleneck
Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint MrXiv:I703.0088 2017.
Wang, X., Chen, H., Wu, Z., Zhu, W., al. Disentangled representation learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
Sinha, Shahi, R., and Shankar, M. Human computer interaction. In 2010 3rd International Conference Emerging Trends in Engineering and Technology, pp. 1- 4. IEEE, 2010.
Wang. Z. and Wu, L. Theoretical analysis of the inductive biases in deep convolutional networks. Advances in Neu- ral Information Processing Systems, 36:74289-74338, 2023.
Tian, Y., Krishnan, D., and Isola. P. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part XI 16. pp. 776-794. Springer, 2020a.
Weston, J., Bengio, S., and Usunier, N. Large scale im- age annotation: learning to rank with joint word-image embeddings. Machine learning, 81:21-35, 2010.
Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non-parametric instance discrimina- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., and Isola, P. What makes for good views for contrastive learning? Advances neural information processing systems, 33:6827-6839, 2020b.
Xiao, Y., Codevilla, F., Gurram, A., Urfalioglu, O., and López, A. M. Multimodal end-to-end autonomous driv- ing. IEEE Transactions on Intelligent Transportation Systems, 23()):537-547, 2020.
Tishby, N. and Zaslavsky, N. Deep learning and the infor- mation bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. IEEE, 2015.
Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method. arXiv preprint physics/0004057, 2000.
Xu, H., Ghosh, G., Huang, P.-Y., Okhonko, D., Aghajanyan, A., Metze, F., Zettlemoyer, L., and Feichtenhofer, C. Videoclip: Contrastive pre-training for zero-shot video- text understanding, arXiv preprint arXiv:2109.1400 2021.
Tsai, Y.-H. H., Wu, Y., Salakhutdinov, R., and Morency, L.- P. Supe-supe learning from a multi-view perspective. arXiv preprint XXXx:2006.05577 2020.
Xu, Y.. Zhao, S., Song. J., Stewart, R., and Ermon, S. theory of usable information under computational con- straints. arXiv preprint arXiv:2002.1068) 2020.
Udandarao, V. Understanding and fixing the modality gap in lision-language models. Master's thesis, University of Cambridge, 2022.
Yager, R. R. On ordered weighted averaging aggregation operators in multicriteria decisionmaking. IEEE Transac- tions on systems, Man, and Cyhernetics, 18(1):183-190 1988.
Usunier, N., Buffoni, D., and Gallinari, P. Ranking with ordered weighted pairwise classification. In Proceedings of the 26th annual international conference on machine learning, pp. 1057-1064, 2009.
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus-based image description evaluation. In Pro- ceedings of the IEEE conference computer vision and pattern recognition, pp.
2015.
Wang, F. and Liu, H. Understanding the behaviour of con- trastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2495- 2504, 2021.
Wang, H., Guo, X., Deng, Z.-H., and Lu, Y. Rethinking min- imal sufficient representation in contrastive learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16041-16050, 2022.
Wang, T. and Isola, P. Understanding contrastive represen- tation learning through alignment and uniformity on the hypersphere. In International conference on machine learning, pp.
PMLR, 2020.
12

---

Aligning Multimodal Representations through an Information Bottleneck
A. Proofs of Sections 3 and 4
A.1. Proof of Lemma! 1
Lemma 1. Let Y and Y' be essences of the same pair of modalities. Then, there exist a one-to-one transformation such that Y = (YY).
Proof. By Definition 1, Y and Y' are minimal sufficient statisties of Xa for X3. Then, by the definition of minimal sufficient statistic, there exist two functions v/ and v such that (YY")) and Y' (YY)). Then, v L/
A.2. Proof of Theorem
Theorem 1. Let Y and Za be the essence and a representation of input Xa respectively, and ler T (T f(Y)) be the set of deterministic functions of (i.e., all the tasks derived Y). Then, we have that:
p(t]za) = p(t)tt)) ET=> = I(Z Y) = I(Xa;Y)
Proof. First, we know from equation (4) that I(Y: E Xa): Second, since T = f(Y), we have the Markov Chain Za ++ Y ++ Xa and, thus, by the DPI, [(T; S ((Y7a)) Third, since p(t]xa)) p(1]=a), we know that T(T; Xx) [T;; Za). Thus, we have that [(T I[T; - :[(Y)Za) X()) Finally, since can be any function of Y, can be the identity function, in which case [(T; (();
A.3. Proof of Theorem 2
Theorem (Informal). Let Za and Zg be two representations of a pair of inputs with nuisances Na and N3 respectively, such that Zo and Zg are aligned in the sense of equation (2). Then, I(Za; N.) [(23; = O.
Proof. First, if I(Z,; N.) + O, then there exists surjective function such that Z/ = [(Za), i.e., more than one repre-
sentation of modality can correspond with one representation of modality B. Let (1(0)
{aa pe.. (2)=²")
(1)
:za p(za))
(1)
and (1) 2 ~ pea
(xlaf) (1)
33 ~ >p(z)) be infinite sets. Then, there exists a pair (1, for which z! # (1)) and
(1)
( ((') and for which K(=a) (1) =00) F K(=a) za =(')), but K(=3) (1) 22 (0)
(1)
k(=)) 211)
2g
We must note that, in practice, we usually work with finite sets, could obtain the maximum value of alignment while having the presence of nuisances in the representation.
A.4. Proof of I(Za; Y) = I(Za: (g)
Proof.
p(za xz) plzalus a a aa alla xg)) dr
(21)
| plza Ia (alaaa dea = p(zaly)
(22)
In line (21), we apply Definition 3 and equation (4). Then, we have the following Markov Chain X3 ++ Y Za and, by the DPI, T(Z;; Y) > T(7a; Xz).
/ x. )l yy, xza) dra
p(Eal1,40))
(23)
f p(zz K p( (1)) dr. = p(zal2z)
(24)
In line (23), we apply Definition 3 and equation (5). Then, we have the following Markov Chain + Xg + Za and, by the DPI, T(Za; Y) & T(Za; X3)-
13

---

Aligning Multimodal Representations through an Information Bottleneck
A.5. Proof of equation (17)
Proof.
/f po, (2, F.) log
Po. (2( de dæ
[ I Xa)
log
(25)
psa (2)
[I] pa. (z/ (elrrap((ac (p( 13)) log
Pea (2(xa) dz dra dxæ
(26)
po (z)
/I] po. (2/ 03) log
Po. (2 (2 ) pop (2/2g)
de ax dx
(27)
p8 (2/2g) po. (2)
D (po, (2| (2|xx))]
[DKr. (po, (2)llpa, (=(x+))
(28)
p(rastp)
p(Ra)=0l2))
$
D (s/
(29)
p(
A.6. Proof of equation (18)
Proof. Let Pe.. (= xa) = (2; Pa: 0²1) and (e mn) =
with Mea = 1ra (Fa) and ugg Ney (x The
KL divergence between these two Gaussians is given by:
1 DRI(Plla) = 2 (r(20g Ar(Ep'Ep) Ep) + (My - Up) E (Mg - 1p)) + log det 2ig det Exp.
(30)
where = (Hp, Ep) and N(Hor Eg). For 2p Dig = =021, Eq. (30) simplifies to:
1
DKL (po_ (z xa) || pag (z xs))) = 20 1| (xt) - nas (2=))l2
(31)
Taking expectation over the joint distribution p(za; we obtain:
Ep(a.. DKL (po.. (= (aa) po, (z x3))
E Ep(amng)
[llee, (xa) - Mo, (13))1
(32)
2o
Hence, the expected KL divergence is proportional to the expected squared &a distance between the mean embeddings:
2p) DKL (po. (z x.a) po, (= xg))] E (p(
[lmo. (xa) no, (13))
(33)
The constant of proportionality is + and independent of the model parameters Oa, 0g.
14

---

Aligning Multimodal Representations through an Information Bottleneck
B. Connection between our loss function and temperature
In the case where the embeddings are unit-norm, our proposed loss takes the form:
cxp(su/T) 2,8(1 st)),
L: log Σn exp($x//T)
(34)
where Sis denotes the cosine similarity between the embeddings 20) and 2(h) The gradient of C; with respect to each similarity tern is given by:
(1 exp($1/TT) 0sh
Oc:
Ex exp(sit/T) 2/3,
(35)
OC, 1 exp(sig/T)
for j + V.
(36)
0sij Σx exp(8ix/7)
We further analyze a modified variant of the InfoNCE loss where the temperature differs between the numerator and denominator:
exp(su/T')
C log
(37)
Σx exp(su/T))
The gradients of this variant are:
05
exp(88//T)
1
(38)
Üss
Σx exp(sik/T)
OC;
1 exp($sg/T)
for j +
(39)
Osij
Σn exp(sax/7)
which matches Eq. (36), confirming that the two losses only differ in their treatment of 8:
By comparing Eq. (35) and Eq. (38), we find that our regularized loss is equivalent to optimizing a variant of InfoNCE with a temperature mismatch between numerator and denominator. Solving for B, we obtain:
exp(su/T) exp(sH/T))
B 2 TT cxp(sit/7)
(40)
This expression reveals that:
The effective temperature gap depends on the similarity between the anchor and all other samples in the batch.
When 8a Es cxp(su/7), i.e., predictions far from the target distribution, the second term in Eq. (40) is negligible, and
1
AT
2
with Ar= = T Thus, larger values of 3 correspond to larger temperature mismatches.
Conversely, when sn En exp(sxx/T), i.e., the model confident in its match, we have:
B ~ Ar -1- exp(Arr)
indicating that the temperature gap required to match fixed 3 is smaller in this regime.
Hence, our regularizer can be interpreted as an adaptive temperature adjustment that decreases the denominator temperature when the model is uncertain, and aligns it closer the numerator temperature when predictions confident.
15

---

Aligning Multimodal Representations through an Information Bottleneck
C. Experimental Details
Below, we provide summary the experimental setup. Full implementation details and code are available at: https:
/ //github com/ ant oni amnioalmddevar/mmliiimae htwpaaarwmmtmmmmmoooo ibb
Table 4: Hyperparameters of Section 5
DSprites
MPI3D
Shapes3D
factors encoder MLP number of epochs batch size optimizer
{16, 16, 8, 8, 16, 16}
{128, 128, 64, 64, 128, 128]
{64, 64, 32, 32, 64, 64)
50
50
50
128
128
128
Adam
Adam
Adam
learning rate scheduler step size (epochs) scheduler ?y
0.001
0.001
0.001
Step
Step
Step
20
20
20
0.3
0.3
0.3
Table 5: Categories of factors in section 5.1. In MPI3D, the factor background collor color actually refers to the color of ring in the images, so we consider it as an object (see https: / / g tt hub. com / rr- prr-eeariingl /wtddiisesttt sentangnttneeene ement_ dataset) There are some factors missing because the do not fall into any category.
DSprites
MP13D
Shapes3D
Location
posX, posY shape
horizontal_ane _axis, vertical orientation
Shape
object.shape
shape
Size
size
object size
scale
Object Color
object_color, background.color
object_hue
Table 6: Hyperparameters of Section 6
vision encoder
VIT-g/14 (Fang et al., 2023)
image size
224
of query tokens
32
cross attention frequency
2
representation dimension
256
text encoder
BERT base (Devlin, 2018)
batch size optimizer
128
Adam
learning rate
0.0001
optimizer 3
(0.9, 0.999)
scheduler
cosine annealing
warm-up steps
1000
training steps
50000
16

---

Aligning Multimodal Representations through an Information Bottleneck
D. More Results of Section 6.1
(1): woman sitting bench with bag food
(1): taking picture of himself mirror
(1): a cat laying on top of
wooden chair in room
(2): a woman sitting on a bench with
(2): a man is taking a picture of himself in the mirror man seen in the reflection of mirror
(2): cat sitting
chair looking the camera
plate of food
(3): woman sitting bench in
(3):
(3): a cat sitting on a chair looking
snow
at the camera
(5): woman sitting on bench in
(5): a man standing front a
(5): a cat laying top of
the snow
bathroom mirror
wooden chair
(1): a couple of benches that are in
(1): toy set of man in a hospital bed with a robot in the middle
(1): man and woman in a white dress are sitting on bed
the grass
(2): white park bench sitting next tree
(2): two dummy heads are that is shaped like boat
bed
(2): man and woman are sitting in bed
(3): bench sitting the middle a
(3): fake bed with a dummy head and legs on it
(3): picture of couple a bedroom with a glass frame
park
(5): metal bench sitting in the
(5): display of demonic joker
(5): photo of a couple in frame on a bed
middle of a park
character in bed
17

---

Aligning Multimodal Representations through an Information Bottleneck
JAM
(1): person holding teddy bear with pair of scissors
(1): a jetliner flying through cloudy blue sky
(1): group horses standing around baby horse
(2): a person holding a teddy bear with woman's pregnant belly
(2): a plane flying in the sky with a half moon in the background
(2): group of horses standing around a pile of hay
(3): person with teddy bear
(3): plane flying in the sky with half moon in the background
(3): group of horses eating hay in field
their lap
(5): pregnant woman holding
(5): plane flying in the sky with
(5): group of horses standing next each other
teddy bear with baby on
half moon in the background
Run for Rights
(1): couple of chairs and an
(1): man standing under banner
(1): cow standing in field with a house in the background standing in field with
umbrella field
with frisbee
(2): couple of chairs sitting next
(2): standing under banner
(2):
table
that says bicycle
house in background
(3): a couple chairs and a table on
(3): a man standing under a banner
(3): a couple of cows standing on top of a lush green field
a field
that says bicycle relief
(5): couple of chairs sitting next to
(5): man standing under sign
(5): cow is standing in field with
fence
that reads for cause
few trees
(1): dog herding sheep in field
(1): wooden bench sitting in the middle of yard
(1): close of pair of scissors in pile
with dog
(2): dag herding sheep field
pile of wood sitting on the side road
(2): group of scissors that hanging up together
with dog running behind them
(3): a dog is herding three sheep in
(3): pile of wood sitting on the side
(3): bunch of pink scissors are chained together in a room
a field
of road
(5): dog is herding some sheep in
(5): bunch of old used wooden poles in yard
(5): bunch of pink scissors are all grouped up
field
Figure 10: Captions generated by some of the trained models. Numbers correspondence is the same in Table 3.
18

---

Aligning Multimodal Representations through an Information Bottleneck
E. More Results of Section 6.2
woman with her face close to mans face"
ITC++M
ITC +TMM
ITC+LM+0.01CM
ITC+LM+0.1Cx
white fire hydrant sitting on the side of road"
ITC+LM
ITC-LM+ITM
ITC I LLM+00000 0.01L
ITC+LM+01LM
19

---

Aligning Multimodal Representations through an Information Bottleneck
"snwwy mountain"
ITC+LM
ITC-IM+ITM
TTC+LMM/ 0.01C
LTC+LM+
"modern building
ITC+LM
ITC+LM+ITM
LTC+LMM0.0110
ITC-LM+0.1LM
20

---

Aligning Multimodal Representations through an Information Bottleneck
"old"
LT
ITC-LM+TTM
LTC+LM+0.01CM
ITC LM10.M
"bike"
ITC+LM
ITC-LM+ITM
I
+0.11MM
Figure 11: Multimodal image retrievals from models train with different loss functions. We believe that text representations from more simple captions (less entropic) are better aligned with image representations from encoders trained with a higher values of B, since they are less entropic too.
21


Reviews:
[{"review_id": "Ejrtmk26gD", "reviewer": "Reviewer_DSZo", "summary": "This paper addresses the challenge of misalignment in multimodal representation learning when using contrastive loss functions. The authors argue that this misalignment stems from modality-specific information present in the representation space that contrastive objectives fail to remove. Leveraging the Information Bottleneck Principle, they provide a theoretical framework to explain this phenomenon and propose a novel regularization term that enhances representational alignment by reducing modality specific information and find the sufficient minimum information. Through empirical validation, the authors demonstrate that their approach not only improves alignment but also enhances performance in real-world tasks such as image captioning, aiming at highlighting that balancing information preservation and compression in multimodal learning is important.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to begin by thanking you for the time dedicated to give us feedback to improve our work. We address your main concerns next:\n\n> (i) additional multimodal benchmarks would enhance generalizability.\n\nMore experiments were not included due to space limitations.\n\n> (ii) task-specific metrics beyond alignment would strengthen the assessment.\n\nIn section 5, we have Fig.6 for this purpose. For section 6, we have calculated retrievals, shown next.\n\n||CIDEr|BLEU@4|I2T R@1|T2I R@1|\n|-|-|-|-|-|\n|ITC+LM|$91.7\\pm 0.2$|$28.6\\pm 0.1$|$64.2\\pm 0.2$|$52.3\\pm 0.4$|\n|ITC+LM+ITM|$91.8\\pm 0.5$|$28.8\\pm 0.2$|$61.4\\pm 0.6$|$49.7\\pm 0.8$|\n|ITC+LM+$0.01\\mathcal{L}_M$|$92.3\\pm 0.8$|$29.1\\pm 0.4$|$64.0\\pm 0.3$|$52.3\\pm 0.5$|\n|ITC+LM+$0.03\\mathcal{L}_M$|$92.6\\pm 0.3$|$29.2\\pm 0.2$|$63.9\\pm 0.4$|$52.1\\pm 0.5$|\n|ITC+LM+$0.1\\mathcal{L}_M$|$93.0\\pm 0.3$|$29.4\\pm 0.3$|$63.0\\pm 0.5$|$50.4\\pm 0.5$|\n|ITC+LM+$0.3\\mathcal{L}_M$|$90.5\\pm 0.4$|$28.5\\pm 0.2$|$59.6\\pm 0.4$|$47.1\\pm 0.4$|\n\nWe have that:\n1. Text generation (TG) and retrieval performances are inversely correlated. This makes sense, since TG gets benefited from minimal representations and retrieval from sufficient representations (retrieval is a specific type of downstream task). This inverse correlation can be observed also in Fig. 6.\n2. Our loss increases TG performance for low or medium values of $\\beta$, since its goal is to increase the minimality of representations.\n3. For $\\beta=0.3$, the performance starts to excessively decrease the part of the essence that representations retain, similarly to in Fig. 6.\n\nThis would be included and better explained in the final version.\n\n> additional experiments to isolate confounding factors would strengthen this argument.\n\nWe agree that the experimental setup that analyzes the Information Homeostasis phenomenon does not serve to demonstrate any causal relationship. However, this analysis would require more space and, additionally, it could confuse the main line of the paper.\n\n> the paper does not compare its method against several state-of-the-art techniques designed for alignment...\n\nWe compare in section 6 our loss function with the ITM, which is used in most of the state-of-the-art methods to obtain alignment between text and image modalities. As explained in lines 379-382, ITM is defined for a very specific architectural choice so it cannot be used in experiments in section 5.\n\n> a formal guarantee on how different encoders may lead to equivalent partitions in practical settings could be added in Lemma 1.\n\nWe are not sure to understand this point very clearly. Lemma 1 is stated in line 134 and representations are defined in line 161, so there is no notion of encoder or representation at the point in which Lemma 1 is stated. Thus, this lemma is completely independent of the encoders.\n\n> Theorem 1 assumes that tasks derived from the essence fully capture all relevant downstream tasks, which may not always hold universally across applications.\n\nThis is never assumed. Theorem 1 simply states that sufficient representations are a necessary condition to solve all the tasks that can be derived from the essence. Sometimes, contrastively trained models are used to solve downstream tasks that are not in the essence. However, there are no theoretical guarantees that this should work no matter if the representation is sufficient or not.\n\n> The paper assumes that modality-specific information always contributes to misalignmentt, but this may depend on the dataset and task.\n\n> Theorem 2 relies on the assumption that perfect alignment can only be achieved if modality-specific information is entirely removed.\n\nThis is not an assumption but a theorem that is demonstrated in Appendix A.3. If the demonstration is correct, then it is universally true. Apart from that, alignment is totally independent of the task, since it is an intrinsic property of the representations. We are open to elaborate on this, but we are not sure to understand your point.\n\n> Essential References Not Discussed\n\nThank you for the given references. Some of them were already mentioned in the paper and the rest of them will be discussed in the related work.\n\n> how this approach is related to PID.\n\nWe believe that the connection is very weak. PID analyzes how two source inputs contribute to the information in a target variable. This is relevant in the cases in which we are using two inputs at the same time, but this is rarely the case in multimodal learning. The main connection that we could make between our work and PID is that, in case that we wanted to use both representations to solve a downstream task, then the redundant information would be equal to zero for minimal representations.\n\nWe hope that your questions have been addressed. If this is the case and you consider that our paper deserves an increase in the score, we would thank you if you made it effective. If this is not the case, we are open to continue with the discussion in the next stage of the rebuttal.", "claims_and_evidence": "The claims in the paper are generally well-supported by both theoretical and empirical studies, with the authors providing meaningful contributions in both areas. The theoretical framework leverages the Information Bottleneck Principle to explain misalignment caused by modality-specific nuisances, while empirical validation includes controlled experiments and real-world applications (e.g., image captioning), demonstrating the effectiveness of their proposed regularization term.\n\nHowever, certain claims require further substantiation. The paper assumes that modality-specific information always contributes to misalignment, but this may depend on the dataset and task. More evidence is needed to establish the universality of this claim. Additionally, the Information Homeostasis phenomenon—where encoders purportedly adjust internal parameters to preserve nuisance information—is an intriguing hypothesis but lacks deeper causal analysis. Conducting additional experiments to isolate confounding factors would strengthen this argument.\n\nMoreover, the paper does not compare its method against several state-of-the-art techniques designed for alignment beyond contrastive learning. Including such comparisons would provide a clearer assessment of its advantages and limitations.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally speaking well-suited to the problem of multimodal representation alignment. The authors leverage the IB principle to design a regularization term that explicitly reduces modality-specific nuisances, effectively addressing misalignment. Their evaluation includes both controlled experiments and real-world applications, providing a solid empirical foundation.\n\nHowever, there is room for improvement:\n(i) Expanding the evaluation to include additional multimodal benchmarks would enhance generalizability.\n(ii) Incorporating additional task-specific metrics beyond alignment (e.g., retrieval accuracy, downstream task performance) would strengthen the assessment.\n(iii) Rather than solely validating the proposed approach, benchmarking against existing alignment techniques (e.g., cross-modal transformers, adversarial methods) would provide a clearer comparison and better contextualize the contribution of this work.", "theoretical_claims": "The paper presents several theoretical claims, primarily leveraging the IB principle to explain modality-specific misalignment. The proofs for key results, including Theorems 1, 2, and Lemma 1, appear correct and logically sound. However, some aspects could be further clarified or refined. For instance, a formal guarantee on how different encoders may lead to equivalent partitions in practical settings could be added in Lemma 1. Theorem 1 assumes that tasks derived from the essence fully capture all relevant downstream tasks, which may not always hold universally across applications. Theorem 2 relies on the assumption that perfect alignment can only be achieved if modality-specific information is entirely removed, which may need fine-tuning, as reasonable alignment can still be attained in practice even if some nuisances remain. While the theoretical foundations are solid, addressing these points would enhance the rigor and practical relevance of the claims.", "experimental_designs_or_analyses": "The experimental design in the paper is generally sound and well-structured, combining controlled experiments on disentanglement datasets with real-world applications. The controlled experiments effectively analyze the impact of different factors on alignment and nuisance removal, while the image captioning task serves as a strong practical validation. The use of CKA as a metric is appropriate, but incorporating additional task-specific evaluations, such as retrieval accuracy, could further strengthen the analysis. A key limitation is the lack of comparisons with state-of-the-art alignment methods beyond contrastive learning, which would provide a clearer perspective on the proposed method’s relative advantages and potential shortcomings. Additionally, the study of the information homeostasis phenomenon, while interesting and relevant, lacks causal analysis, making it difficult to fully substantiate its claims. Conducting more ablation studies to isolate potential confounding factors would improve the robustness of this finding.", "relation_to_broader_scientific_literature": "The key contributions of the paper build upon and extend foundational ideas in multimodal learning, contrastive learning, and information theory. It is closely related to prior work on contrastive representation learning (Oord et al., 2018; Radford et al., 2021), which aims to align representations by maximizing mutual information between different modalities. However, contrastive objectives alone have been shown to be insufficient for achieving true alignment due to the presence of modality-specific nuisances (Liang et al., 2022). This paper addresses this limitation by leveraging the Information Bottleneck Principle (Tishby et al., 2000; Alemi et al., 2016; Achille & Soatto, 2018) to derive a principled regularization term that explicitly reduces nuisances, providing a more structured approach than heuristic modifications proposed in prior works (Li et al., 2021; 2022; 2023). The study also aligns with recent efforts in multi-view learning (Tian et al., 2020) and information-based disentanglement (Wang et al., 2022), which emphasize the role of minimal sufficient representations in improving alignment. Additionally, the paper introduces the Information Homeostasis hypothesis, which suggests that encoders adjust internal parameters to maintain a balance in information entropy—a concept related to implicit regularization in deep networks (Shwartz-Ziv & Tishby, 2017) but not yet extensively studied in the context of multimodal alignment. Nevertheless, incorporating additional literature and direct comparisons with alternative alignment techniques would further strengthen its positioning within the broader research landscape.", "essential_references_not_discussed": "Some key works in the area that could be added: adversarial alignment (Lample et al., NeurIPS 2018), cross-modal transformers (imagebind of Girdhar et al., CVPR 2023), minimum sufficient representation learning (Wang et al., CVPR 2022), platonic Representation Hypothesis (Huh et al., 2024 - https://arxiv.org/abs/2405.07987) for self-supervised, multimodal IB (https://arxiv.org/abs/2210.17444), PID appraoches (https://arxiv.org/pdf/2401.13503, https://arxiv.org/pdf/2409.07402v1 (for alignement), and https://arxiv.org/pdf/2402.06223v1 (Multimodal Contrastive Representation Learning through Latent Partial Causal Models).", "other_strengths_and_weaknesses": "The strengths and the weaknesses have been discussed in the previous sections. Nothing major to add here.", "other_comments_or_suggestions": "Most questions have been raised in the previous sections but a key one is how this approach is related to partial information decomposition (PID). PID seems to be relevant to this problem and can be leveraged to formalize some of the claims.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hpC28Dd1Xm", "reviewer": "Reviewer_ug8B", "summary": "The paper analyzes the problem that the contrastive losses in multimodal representation learning fail to align representations effectively due to their retention of modality-specific information.  To address this, the authors propose a variationally-derived regularization term that reduces modality-specific information, enhancing alignment based on the Information Bottleneck Principle.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to begin by thanking you for the time dedicated to give us feedback to improve our work. We address your main concerns next:\n\n> 1. Quantitative results of image retrievals are missing.\n\n> What will the model perform when having a stronger $L_M$ constraint?\n\nPlease see second answer to Reviewer DSZo.\n\n> 2. providing the quantification of essence and nuisances and real-world dataset will strengthen the soundness of the paper.\n\nQuantifying the essence and nuisances is impossible in general because they are abstract variables that we simply define for our formulation. For this reason (and also because computing mutual information is expensive), it is in general impossible to quantify the essence and nuisances in the representation. In fact, this is the reason why we use variational approximations for the derivation of our loss function. If it were possible to exactly calculate the amount of the essence or nuisances in the representations, then we could simply maximize and minimize these quantities, respectively. The toy datasets are precisely used to set by ourselves the essence and nuisances and having a scenario in which it is straightforward to calculate a reliable estimation of the mutual information between the essence (or the nuisances) and the representations.\n\n> The ideas of essence and nuisances are similar to the unique and shared information concepts in [1].\n\nThank you for your recommendation. This paper will be included in the related work section.\n\n> Is the spherical Gaussian assumption reasonable for general multimodal representation learning?\n\nOn the one hand, the use of Gaussian distributions in the representation space is not an assumption but a design choice that we make mainly for tractability reasons of the KL divergence in equation 17. We argue its reasonability next.\n\n> If the data are non-Gaussian, this assumption might be questionable.\n\nOn the other hand, the fact that the data distribution $p(x)$ is not Gaussian should not be problematic. For example, most of the VAEs (or other SoTA generative models, such as Diffusion Models and Flow Matching) use a encoder $p_\\theta(z|x)$ that is Gaussian (also mainly for tractability reasons of the KL divergence) and they provide impressive performance in real-world applications, in which the data $p(x)$ is far from being Gaussian.\n\n> ...it is difficult to justify that the learned representations strictly follow a simple Gaussian distribution.\n\nFinally and most importantly, we must note that choosing the encoder $p_\\theta(z|x)$  as Gaussian, does not imply at all that the representation space is Gaussian. What is Gaussian is the distribution of an embedding given a single input $p_\\theta(z|x)$, but not the whole representation distribution $p_\\theta(z)$.\n\nMore specifically, we have that $p_\\theta(z)=\\int p_\\theta(z|x)p(x)dx$. If we approximate the data distribution $p(x)$ as its empirical distribution given by $N$ datapoints, i.e., $p(x) \\approx \\sum_{i=1}^N \\delta\\left(x-x^{(i)}\\right)$, then the distributions of the representation space $p_\\theta(z)$ becomes a Gaussian mixture, i.e.,  $p_\\theta(z) \\approx \\frac{1}{N} \\sum_i \\mathcal{N} \\left(z; \\mu_\\theta\\left(x^{(i)}\\right), \\sigma^2 I\\right)$, which \"is a universal approximator of densities, in the sense that any smooth density can be approximated with any speciﬁc nonzero amount of error by a Gaussian mixture model with enough components\" [1].\n\nFurthermore, we note that using a stochastic encoder (even if it is Gaussian) results in a distribution that is richer than in the case of deterministic encoders (i.e., vanilla encoders). In the latter, $p_\\theta(z|x)$ is a delta distribution, i.e., simpler (one parameter) than a gaussian (two parameters) and using deterministic encoders is rarely seen as problematic.\n\n[1] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning., p. 65\n\n> Eq. (20) assumes identical covariances for the two modalities, which is even less realistic given typical differences in modality-specific representation distributions.\n\nFollowing with the previous answer, this should not be problematic. In fact, having the same covariance matrices in the representation spaces of both modalities could be considered as desirable since we would like these spaces to be as similar (or aligned) as possible.\n\n> The notation $t$ is not explained.\n\n$t$ means a realization of the task $T$. In probability theory, random variables are usually denoted by uppercase letters while their realizations are denoted by their corresponding lowercase letters (see https://en.wikipedia.org/wiki/Notation_in_probability_and_statistics). We will add a paragraph in section 2 to clarify this.\n\nWe hope that your questions have been addressed. If this is the case and you consider that our paper deserves an increase in the score, we would thank you if you made it effective. If this is not the case, we are open to continue with the discussion in the next stage of the rebuttal process.", "claims_and_evidence": "The spherical Gaussian assumption for representation and indendical covariance for different modalities' representation are not supported by convincing evidence.  Such assumptions should be critically assessed and justified clearly in the context of general multimodal representation learning.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "The soundness of experimental results on real-world tasks is weak. \n\n1. Quantitative results of image retrievals are missing. It is not clear how well the model performs on this task. \n\n2. The analysis of the toy example is interesting. However, providing the quantification of essence and nuisances and real-world dataset (e.g., image-text) will strengthen the soundness of the paper.", "relation_to_broader_scientific_literature": "This paper is relelated to the theoretical analysis on multimodal representation learning.", "essential_references_not_discussed": "The ideas of essence and nuisances are similar to the unique and shared information concepts in [1]. The discussion between relevant references is missing. \n\n[1]. Liang, Paul Pu, et al. \"Factorized contrastive learning: Going beyond multi-view redundancy.\" Advances in Neural Information Processing Systems 36 (2023): 32971-32998.", "other_strengths_and_weaknesses": "It is interesting to see the analysis on the toy examples. However, I have the following concerns:\n\nIs the spherical Gaussian assumption reasonable for general multimodal representation learning?\nIf the data are non-Gaussian, this assumption might be questionable. Particularly, considering your use of layer-wise normalization (e.g., with per-sample operations like those in the provided setting), it is difficult to justify that the learned representations strictly follow a simple Gaussian distribution.\n\nMoreover, Eq. (20) assumes identical covariances for the two modalities, which is even less realistic given typical differences in modality-specific representation distributions. A more general and rigorous form that relaxes this assumption would strengthen the analysis.", "other_comments_or_suggestions": "See my weaknesses and questions. If they are all addressed, I am willing to raise my score.", "questions_for_authors": "In Theorem 1, could you clarify what exactly $p(t|z_{\\alpha})$ represents? The notation $t$ is not explained.\n\nIn Table 2, 0.1$L_M$ has the best CIDEr and BLEU scores. What will the model perform when having a stronger $L_M$ constraint? Do you have an ablation study on this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4RNKEPHEy6", "reviewer": "Reviewer_Xy8R", "summary": "The manuscript shows that contrastive learning methods for multimodal representations do not remove modality-specific information, which leads to misaligned representations. It uses an Information Bottleneck approach to add a regularization term to the loss function to filter out this extra information while preserving the alignment. The approach demonstrates improved performance in tasks such as image captioning and multimodal retrieval.\n\n## update after rebuttal\nThe score was increased from 3 (Weak Accept) to 4 (Accept). Most of the concerns have been addressed. The authors reran the experiments multiple times and included additional experiments using different architectures to evaluate how architectural choices affect the URR. They also clarified the questions regarding nuisances, which resolved my earlier confusion.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to begin by thanking you for the time dedicated to give us feedback to improve our work. We address your main concerns next:\n\n> could the choice of image encoder introduce bias in the URR measurements?\n\nTo analyze this point, we have performed experiments that are identical to those in Table 1, but using a small ViT as image encoder. We show the results next:\n\n||DSprites|MPI3D|Shapes3D|\n|-|-|-|-|\n|Location|$2.8\\pm 0.6$|$2.8\\pm 0.3$|$1.1\\pm 0.1$|\n|Shape|$64.9\\pm 1.4$|$7.0\\pm 0.4$|$5.7\\pm 0.2$|\n|Size|$30.7\\pm 3.5$|$20.8\\pm 3.5$|$6.9\\pm 1.5$|\n|Objects Color|-|$63.5\\pm 9.8$|$53.5\\pm 1.6$|\n\nBy comparing the latter and Table 1 we can observe that:\n\n1. Local attributes, such as *Location* and *Size*, are better preserved in convolutional encoders, which is consistent with the inductive biases towards local structures of convolutional layers.\n2. Global attributes, such as *Shape* and *Objects Color* are almost equally conserved in both image encoders.\n\nSubsection 5.1 would be extended with these two experiments and conclusions. Thanks for noticing.\n\n> how are invariance and equivariance with respect to different factors accounted for in this analysis?\n\nSorry, we do not understand this question, could you please further develop it?\n\n> I recall an ICLR 2021 paper...\n\nWe think that the paper that you could be referring to is [1]. If this is the case, your question is very interesting and points of [1] and ours perfectly compatible.\n\nOn the one hand, what the mentioned paper states is that, if the task to be solved is correlated with other simpler features, then a model trained with SGD will tend to learn these simpler features rather than the task. If we relate this idea with the concepts used in our work, the point of [1] would be something like: “given a task Y, models trained with SGD tend to learn minimal sufficient statistics of Y instead of Y”. For example, if our task were to classify between images of bananas and strawberries, our model would simply learn to differentiate between yellow and red.\n\nOn the other hand, our paper argues that, even given the previous statement, models tend not to remove all the information that is unnecessary to solve the task (i.e., nuisances). In the previous example, a model could be learning, for instance, the background color of the images. Thus, our model could learn the fruit color and the background color, which makes the point of [1] and ours perfectly compatible.\n\nIf this is the paper you refered to, thanks for bringing it up, its results are very interesting. If this is not the case, we are open to discuss other works.\n\n[1] Ahmed, F., Bengio, Y., Van Seijen, H., & Courville, A.. Systematic generalisation with group invariant predictions.\n\n> For the Information Homeostasis..., could this phenomenon be linked to issues inherent to softmax formulations?\n\nFirst, for higher values of $\\beta$, representations tend not to retain the information that is unique to their input. Then, all the representations are more similar to each other and closer in the space. Thus, maybe predictions are less sharp, so the temperature is decreased when $\\beta$ increases in order to maintain the same level of sharpness in the predictions no matter the value of $\\beta$.\nThus, an hypothesis that could connect to Veličković et al. (2024) is the fact optimization processes \"prefers\" a given level of sharpness. Also, since the level of sharpness changes with the batch size, using Sigmoids instead, as Zhai et al. (2023) suggests, would make the temperature to be invariant to changes in $\\beta$. Intuitively, since the rest of the similarities are ignored for the Sigmoid, the fact that representations of negative pairs are closer in the space, should not affect the predictions.\nThis point is pure speculation and excessive importance should not be attributed to it.\n\n> I think Equation 18 should be derived in the appendix.\n\nThe derivation of this equation will be included in an appendix.\n\n> Table 2 needs to show std by running repeated experiments with different seeds.\n\nPlease see the second answer to Reviewer DSZo.\n\n> I think preliminaries can be cut down to related work without the equations.\n\nWe agree that the expressions of HSIC and CKA are not necessary for the understanding of the paper. They will be removed from the paper. Thanks for noticing.\n\n> Also, it would be great if you annotated the essence and nuisances in the examples.\n\nEssence and nuisances will be annotated in the examples. We suppose that you refer to the examples in section 3. Please let us know if you refer to other examples.\n\nWe hope that your questions have been addressed. If this is the case and you consider that our paper deserves an increase in the score, we would thank you if you made it effective. If this is not the case, we are open to continue with the discussion in the next stage of the rebuttal process. Similarly, if you could elaborate on the question that we were unable to answer, we will try to address it.", "claims_and_evidence": "- While the paper supports its claims about contrastive losses failing to remove nuisance information using the Uncertainty Reduction Ratio (URR), I have a few concerns. First, could the choice of image encoder introduce bias in the URR measurements? Second, how are invariance and equivariance with respect to different factors accounted for in this analysis? Finally, I recall an ICLR 2021 paper suggesting that neural networks tend to focus on the easiest factor to minimize the training objective rather than learning all factors. How does this observation affect the interpretation of the paper’s results on nuisance information removal?\n-  Minimal sufficient representations argue that achieving representational alignment requires that the learned representations be both sufficient (containing all shared “essence”) and minimal (excluding nuisances). Hence, the manuscript introduces a regularization term for alignment. This seems to be supported by the results in Table 2.\n- For the Information Homeostasis phenomenon, the evidence seems preliminary. Considering that InfoNCE employs a softmax-like objective, could this phenomenon be linked to issues inherent to softmax formulations, as discussed by Veličković et al. (2024)? Moreover, Zhai et al. (2023) suggest that using sigmoids instead for sigmoids.\n\nVeličković, Petar, et al. \"softmax is not enough (for sharp out-of-distribution).\" arXiv preprint arXiv:2410.01104 (2024).\nZhai, Xiaohua, et al. \"Sigmoid loss for language image pre-training.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.", "methods_and_evaluation_criteria": "The methods and evaluation criteria seem reasonable.", "theoretical_claims": "I think Equation 18 should be derived in the appendix.", "experimental_designs_or_analyses": "Table 2 needs to show std by running repeated experiments with different seeds.", "relation_to_broader_scientific_literature": "Broadly speaking, this work seems to be InfoNCE with regularization on the latent space to ensure that multimodal representation aligns, where latent space is VAE with Gaussian priors but without reconstruction.", "essential_references_not_discussed": "I do not have concerns on this.", "other_strengths_and_weaknesses": "Strengths:\n- Multiple datasets\n- Real-world experiments\n- Ablations for the hyperparameters", "other_comments_or_suggestions": "- I think preliminaries can be cut down to related work without the equations.\n- Also, it would be great if you annotated the essence and nuisances in the examples.", "questions_for_authors": "See above. I will be open to increasing the score based on more clarification.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "elXeVDlqYk", "reviewer": "Reviewer_N9GW", "summary": "In this paper, the authors study the alignment of representation in multimodal learning through information theory. \nFor a positive pair $X_\\alpha, X_\\beta$ from modalities $\\alpha, \\beta$, they formulate the essence $Y$ and nuisance \n$N_\\alpha, N_\\beta$ of the inputs as the common and modality-specific parts (in the mutual information sense) of the \ninputs, respectively. Then, they define a representation $Z_\\alpha$ of $X_\\alpha$ to be sufficient if it preserves all \ninformation in $Y$ and say $Z_\\alpha$ is minimal if it contains no information about $N_\\alpha$. After that, they \nrelate these notions to the maximization of $I(Z_\\alpha; Y)$ and minimization of $I(Z_\\alpha; N_\\alpha)$, and show that \nminimizing a regularized version of InfoNCE can be used as a surrogate of this optimization task. Finally, they provide \nexperimental evidence supporting the validity of their formulation and the utility of their regularizer.\n\n## update after rebuttal\nI thank the authors for correcting my misunderstandings and the new discussion on the regularizer. I will keep my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to begin by thanking you for the time dedicated to give us feedback to improve our work. We address your main concerns next:\n\n> these datasets seem to be rather old (from 2015 and 2002).\n\nWe believe that you may be refering to the metrics instead of to the datasets. These are still widely used despite of the fact that they are old. See for example [1] and [2], two recent papers with a great impact in the field.\n\n[1] Li, J., Li, D., Savarese, S., & Hoi, S. (2023, July). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.\n\n[2] Team, G., L. (2023). Gemini: a family of highly capable multimodal models.\n\n> the role of Theorem 1 is rather unclear.\n\nTheorem 1 states that sufficiency is a necessary condition to solve all the downstream tasks (that can be derived from the essence). In other words, if a representation is not sufficient, then it cannot solve all the downstream tasks. The opposite is also true, but we do not find it so valuable. Having a representation that can potentially solve all the downstream tasks is not valuable because it does give no information on how “easy-to-use” the information to solve this task is. In the extreme case, the input can be seen as a representation of itself (since it satisfies Definition 3) and it can potentially (through an oracle model) solve any downstream task. Thus, obtaining a representation that can potentially solve all the downstream tasks is not meritorious nor theoretically valuable. However, the importance of this theorem lies in the fact that any representation that is not sufficient cannot be used to perfectly solve any downstream task, no matter how “easy-to-use” the information it contains is, which enhances the value of sufficient representations. This idea connects to that of usable information [3], which allows to formulate this differentiation between having information present in a representation and how “easy-to-use” this information is. This clarification can be made in the paper.\n\n[3] Xu, Y., Zhao, S., Song, J., Stewart, R., & Ermon, S. (2020). A theory of usable information under computational constraints. arXiv preprint arXiv:2002.10689.\n\n> Consider using the mathrm or texttt...\n\nmathrm will be used.\n\n> If we normalize the output representation $\\mu$ ... \n\nIn the case in which the embeddings are unit-norm, our loss becomes $\\mathcal{L}_i = \\log\\frac{\\exp(s\\_{ii}/\\tau)}{\\sum_k \\exp(s\\_{ik}/\\tau)} + 2\\beta(1-s\\_{ii})$, where $s\\_{ik}$ is the cosine similarity between $z^{(i)}$ and $z^{(k)}$. We have the following:\n- $\\frac{\\partial{\\mathcal{L}_i}}{\\partial s\\_{ii}} = -\\frac{1}{\\tau} \\left(1 - \\frac{\\exp(s\\_{ii}/\\tau)}{\\sum_k \\exp(s\\_{ik}/\\tau)} \\right) - 2\\beta$\n- $\\frac{\\partial{\\mathcal{L}_i}}{\\partial s\\_{ij}} = \\frac{1}{\\tau} \\frac{\\exp(s\\_{ij}/\\tau)}{\\sum_k \\exp(s\\_{ik}/\\tau)}$\n\nWe also analyze the gradients of a modification of the $\\mathrm{InfoNCE}$ in which a different temperature is used for the numerator and the denominator, i.e., $\\mathcal{L}'_i=\\log\\frac{\\exp(s\\_{ii}/\\tau')}{\\sum_k \\exp(s\\_{ik}/\\tau)}$. Then, it can be easily checked that:\n- $\\frac{\\partial{\\mathcal{L}'_i}}{\\partial s\\_{ii}} = -\\frac{1}{\\tau'} \\left(1 - \\frac{\\exp(s\\_{ii}/\\tau')}{\\sum_k \\exp(s\\_{ik}/\\tau)} \\right)$\n- $\\frac{\\partial{\\mathcal{L}'_i}}{\\partial s\\_{ij}} = \\frac{\\partial{\\mathcal{L}_i}}{\\partial s\\_{ij}}$\n\nThus, we have that optimizing our loss is equivalent to optimizing a modification of $\\mathrm{InfoNCE}$ with different temperature in numerator and denominator. Rearranging, $\\beta=\\frac{1}{2} \\left[ \\frac{\\tau-\\tau'}{\\tau\\tau'} + \\frac{\\exp(s\\_{ii}/\\tau) - \\exp(s\\_{ii}/\\tau')}{\\sum_k \\exp(s\\_{ik}/\\tau)} \\right]$. Thus:\n\n- The difference in the temperature between numerator and denominator depends on the similarity with respect to all the elements in the batch.\n- If $s\\_{ii} \\ll \\sum_k \\exp(s\\_{ik}/\\tau)$, i.e., the predictions are far from the target distribution, then $\\beta \\approx \\frac{1}{2}  \\Delta\\tau$, where $\\Delta\\tau = \\frac{\\tau-\\tau'}{\\tau\\tau'}$. Thus, the larger the $\\beta$, the larger is the difference between the numerator and denominator temperatures.\n- If $s\\_{ii} \\gg \\sum_k \\exp(s\\_{ik}/\\tau)$, i.e., the predictions are close to the target distribution, then $\\beta \\approx \\frac{1}{2} \\left[ \\Delta\\tau + 1 - \\exp\\Delta\\tau \\right]$. Thus, the temperature difference between numerator and denominator for a given value of $\\beta$ is lower than in the previous case. \n\nThus, our term can be seen as a regularizer the adapts the value of the temperature in the denominator based on the how close to the true distribution the prediction is.\n\nThese comments will be added in an appendix. Thank you for your interest. Please let us know if you have any other insight with respect to this last analysis.\n\nWe hope that your questions have been addressed. If this is the case and you consider that our paper deserves an increase in the score, we would thank you if you made it effective.", "claims_and_evidence": "* This paper proposes an information theoretical interoperation (essence, nuisance, and sufficient/minimal representations) \n  of the misalignment phenomenon in multimodal learning. They provide both theoretical results and toy experiments to \n  support their interpretation. \n* Based on their interpretation, they propose a regularizer to reduce the amount of nuisance contained in the learned \n  representation and verify on real-world datasets and it improves the performance of the model.", "methods_and_evaluation_criteria": "* They verify thier interpretation on various toy datasets (DSprites, MPI3D, Shapes3D), which allow them to control the \n  ratio between essence and nuisance in the data. \n* They approximate the amount of preserved nuisance (which is not computatable in general) by training a linear classifier \n  to predict the nuisance from the learned representation. This makes sense and is also used in previous works. \n* For those more realistic experiments, they use CIDEr and BLEU@4 to test the performance of the regularized/unregularized \n  models and provide some examples for the image retrieval task. I'm not familiar with the more empirical side of this \n  field, but these datasets seem to be rather old (from 2015 and 2002).", "theoretical_claims": "Yes. The theoretical claims are valid. However, the role of Theorem 1 is rather unclear. It says that if one can solve \nall downstream tasks using a representation $Z_\\alpha$, then it is sufficient in the sense of Definition 4. To justify\nthe definition, the reverse direction looks more natural.", "experimental_designs_or_analyses": "Yes. The experiment designs and analyses are sound. For the real-world applications, it would be good to have experiments\non more recent datasets.", "relation_to_broader_scientific_literature": "The key contribution of this paper is to extend the idea of (Tian et al., 2020b) to the multimodal setting, where the \ndefinition of common and modality-specific parts of inputs are less clear, and based on this extension, provide an \nexplanation on the misalignment phenomenon in multimodal learning.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Overall, this is a neat paper, well-written and easy-to-follow.", "other_comments_or_suggestions": "Consider using the mathrm or texttt when writing, say, HSIC and infoNCE in an equation.", "questions_for_authors": "If we normalize the output representation $\\mu$ to have unit norm, then the $l^2$ distance between $\\mu_\\alpha$ and \n$\\mu_\\beta$ is equivalent to the (negative) cosine similarity of them. Meanwhile, (at least when the temperature is high,) \nwe can linearize/Taylor expand the softmax in InfoNCE to get the cosine similarity out. Is it possible to combine these \nto explain the model's behavior under the proposed regularization?\n\nI'm asking this purely out of curiosity, and the response is unlikely to affect the score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper has overall positive scores, except some concerns from reviewer ug8B. These concerns seem to have been addressed in a adequate manner by the authors and the concerns were not shared by the other reviewers. The other reviewers (in particular reviewers DSZo and N9GW) highlight that the theory and experimental design are overall sound. The theoretical part and the methodology bring a useful contribution to the field.
Decision: Accept (poster)