On Information Maximisation in Multi-View Self-Supervised Learning
Authors: Borja Rodríguez Gálvez
Arno Blaas
Xavier Suau
Jason Ramapuram
Dan Busbridge
Luca Zappella
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-view Self-supervised Learning
Information Theory

Abstract: The strong performance of multi-view self-supervised learning (SSL) prompted the development of many different approaches (e.g. SimCLR, BYOL, and DINO). A unified understanding of how each of these methods achieves its performance has been limited by apparent differences across objectives and algorithmic details. Through the lens of information theory, we show that many of these approaches are maximising an approximate lower bound on the mutual information between the representations of multiple views of the same datum. Further, we show that this bound decomposes into a ``reconstruction" term, treated identically by all SSL methods, and an ``entropy" term, where existing SSL methods differ in their treatment. We prove that an exact optimisation of both terms of this lower bound encompasses and unifies current theoretical properties such as recovering the true latent variables of the underlying generative process (Zimmermann et al., 2021) or or isolating content from style in such true latent variables (Von Kügelgen et al., 2021). This theoretical analysis motivates a naive but principled objective (EntRec), that exactly optimises both the reconstruction and entropy terms, thus benefiting from said theoretical properties unlike other SSL frameworks. Finally, we show EntRec achieves a downstream performance on-par with existing SSL methods on ImageNet (69.7% after 400 epochs) and on an array of transfer tasks when pre-trained on ImageNet. Furthermore, EntRec is more robust to modifying the batch size, a sensitive hyperparameter in other SSL methods.

Under review as a conference paper at ICLR 2023
ON INFORMATION MAXIMISATION IN MULTI-VIEW SELF-SUPERVISED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The strong performance of multi-view self-supervised learning (SSL) prompted the development of many different approaches (e.g. SimCLR, BYOL, and DINO). A unified understanding of how each of these methods achieves its performance has been limited by apparent differences across objectives and algorithmic details. Through the lens of information theory, we show that many of these approaches maximise an approximate lower bound on the mutual information between the representations of multiple views of the same datum. Further, observe that this bound decomposes into a "reconstruction" term, treated identically by all SSL methods, and "entropy" term, where existing SSL methods differ in their treat- ment. We prove that an exact optimisation of both terms of this lower bound encompasses and unifies current theoretical properties such recovering the true latent variables of the underlying generative process (Zimmermann et al., 2021) or or isolating content from style such true latent variables (Von Kügelgen et al., 2021). This theoretical analysis motivates naive but principled objective (En- tRec), that directly optimises both the reconstruction and entropy terms, thus ben- efiting from said theoretical properties unlike other SSL frameworks. Finally, we show EntRec achieves a downstream performance on-par with existing SSL meth- ods on ImageNet (69.7% 7% after 400 epochs) and on an array of transfer tasks when pre-trained on ImageNet. Furthermore, EntRec is more robust to modifying the batch size, sensitive hyperparameter in other SSL methods.
INTRODUCTION
Representation learning commonly tackles the problem of learning compressed representations of data which capture their semantic information. A necessary, but not sufficient, property of a good representation is thus that it is highly informative of said data. For this reason, many represen- tation learning methods aim to maximise the mutual information between the input data and the representations, while including some biases in the model that steer that information to be semantic, e.g. (Agakov, 2004; Alemi et al., 2017; Hjelm et al., 2018 8; Oord et al., 2018; Velickovic et al., 2019)- Moreover, mutual information has been the central object to understand the performance of many of these algorithms (Saxe et al., 2019; Rodríguez Gálvez et al., 2020; Goldfeld & Polyanskiy, 2020).
subfield of representation learning is self-supervised learning (SSL), which consists of algorithms that learn representations by means of solving an artificial task with self-generater labels. A par- ticularly successful approach to SSL is multi-view SSL, where different views of the input data generated and the self- generated task is to make sure that representations of one view are predictive of the representations of the other views c.f. (Jing& & Tian, 2020: Liu et al., 2022).
Multi-view SSL algorithms based on the InfoNCE (Oord et al., 2018) like (Bachman et al., 2019; Federici et al., 2020; Tian et al., 2020a) focus on maximising the mutual information between the representations and the input data by maximising the mutual information between the representa- tions of different views (Poole et al., 2019). Similarly, Shwartz-Ziv et al. showed that (Bardes et al., 2022, VICReg) also maximises this information, even though it was not designed for this purpose. Moreover, Tian et al. (2020b); Tsai et al. (2020) provide perspectives on why maximising this mu- tual information is attractive and discuss some of its properties. However, Tschannen et al. (2019); McAllester & Stratos (2020) warn about the caveats of this maximisation (e.g. that it is not sufficient for good representations). Here, complement these efforts from multiple fronts and contribute:

---

Under review as a conference paper at ICLR 2023
Showing that maximising the lower bound (1) on the mutual information between repre- sentations of different views has desirable properties in good representations (Section 2). More preciscly, we show that this maximisation unifies current theories on learning the true explanatory factors of the input (Zimmermann et al., 2021) and separating semantic from irrelevant information (Von Kügelgen al., 2021).
Showing how many existing multi-view SSL algorithms also maximise this mutual infor- mation, although not exactly maximising the lower bound (1). This completes the picture of contrastive methods with an analysis of (Chen et al., 2020a, S SimCIR), where such a result was only known for (Tian et al., 2020a, CMC)-like methods (Poole et al., 2019; Wu et al., 2020) under the InfoNCE (Oord et al., 2018) assumptions requiring i..d. nega- tive samples. It also provides unifying framework with other projections' reconstruction methods such as (Chen & He, 2021, SimS i am), (Grill et al., 2020, BYOL), (Caron et al., 2018; 2020, DeepCluster and SwAV), and (Caron et al., 2021, DINO).
Demonstrating how a proposed naive method that directly maximises the aforementioned bound (1) on this mutual information (EntRec) has comparable performance to current of-te-of-the-aae art methods and is more robust changes in training hyperparameters such as the batch size (Section 4 and Section 5).
This paper is recognition of the importance of maximising the mutual information between the rep- resentations of different views of the imput data, as doing so by maximising (1) has desirable prop- erties (Section 2), and many methods that maximise it (Section 3), including naive ones (Section 4), have good empirical performance (Section 5). However, since maximising mutual information is not sufficient for good representations (Tschannen et al., 2019), this paper also a call to include more biases in the model and the optimisation enforcing the representations to learn semantic information. Appendix A completes the positioning of the paper with respect to related work.
Notation Upper-case letters X represent random objects, lower-case letters x their realisations, calligraphic letters X' their outcome space, and Fx their distribution. Random objects X are as- sumed to have a density px with respect to some measure μ. and the expectation of a function of X is written as [(((X) == Ex-px [=)]. When two random objects X, Y are considered, the conditional density of X given Y is written as PxlY, and for each realisation y of Y it describes the density Px Sometimes, the notation is abused to write a "variational" density 4x of X given Y. Formally, this amounts considering different random object X such that PXir = 9xlY.
The mutual information between two random objects X and Y is written as I(X; Y), and their con- ditional mutual information given the random object 2 as I(X: YZ).. The Shannon entropy and differential entropy of a random object X are both written as H(X), and are clear from the context. The Jensen- Shannon Jensen-Shannon divergence between two distributions P and Q is written DJs (IP) [Q). A set of k elements x(11, , x(()) is denoted as 2 (1:6), (possibly unordered) subsequence x(()) ,...,, (0)) of those elements is denoted as (a:b)) and all the elements in (1:6k) except of z(() is denoted as (--1)
2 MULTI-VIEW SSL AND MUTUAL INFORMATION
In multi-view SSL, two (or more) views (potentially generated using augmentations) of the same data sample X are generated (Bachman et al., 2019; Tian et al., 2020a;b; Chen et al., 2020a; Caron et al., 2020; Zbontar et al., 2021). Views Vi, 1/2 are engineered such that most of the semantic information S of the data is preserved (Tian et al., 2020b). This process generates two branches where the views are processed to generate representations R1, R2 which are later projected into a lower dimensional space Z1, Zz. Finally, the model's parameters 6 are optimised so that the projected representations (projections) from one branch, say Z1, are predictive of the representations of the other branch Zz (see Figure 1). In particular, as shown Section 3, many mutli-view SSI. methods aim to maximise the mutual information between the projections I(21; 72)
Consider the following decomposition of the mutual information (Agakov, 2004; Rodríguez Gálvez et al., 2020)
Entropy
Reconstmaction
I [1; Zz) H(2)) H H(Zz/Z)) 2 H H(Z))
E
log 9z, z (Zz)
(1)
'eere, this measure will either he the I Lebesgue measure and Px will denote the standard probability density function (pdf) the counting measure and px will denote the standard prohahility mass function (pmf).
2

---

Under review as a conference paper at ICLR 2023
The role of both terms from (1) in SSL is distinet: the entropy term determines how much infor- mation from one projection can be learnt, while the reconstruction term determines how much of this available information is learnt. For instance, imagine the projections lay the sphere: the more spread out (higher entropy) the projections of different images are, the more revealing (higher mutual information) it is if projections from different views of the same image close (lower construction error). On the other hand, if all images of one branch are projected to the same point (lowest entropy, also known as collapse), the projections from the other branch can't 't reveal any information about them, because their location is always the same.
Although large mutual information does not necessarily imply good downstream perfor- mance (Tschannen et al., 2019), maximising this lower bound is sensitive objective since it pro- motes learning the semantic information and discarding irrelevant information.
To gain intuition, assume the data can be separated into some semantic S and some irrelevant vari- ables such that X ( U) and S U. Further assume that the views can be written as Vi (S, U1) and 1/2 = ( U2). where U1 and U2 independent. Since the mutual information between the views contains only semantic information, maximising 12,, E2) encourages the projec- tions to learn only semantic information. Indeed, imagine the projections contain integrally the irrel- evant variables, i.c. Z1 = (1/2 (S), U1). Zg = (1/2(5) 9, U2), then their mutual information would be the same if they did not contain them at all: 1(<) (S), U1: the (S), more, assume the projections lay in compact set C Rd and the reconstruction density is defined with a semi-metric P such that Q2,12, =2, (x2) x e - Pa), Then, maximising the reconstruction term minimises Bip(Z,, Z2)]. thus pulling together the non linear mappings Ti a fo o p(S, U1) and Te o Se (sS, U2) (see also Figure 1). Therefore, if the reconstruction is maximised, on average, the irrelevant variables U,, U2 are not contributing to Z1 and Z2, therefore promoting the discarding of irrelevant information.
1( (S); e))) Further-
Remark 1. SSL promotes learning semantic information and discarding irrelevant information.
This highlights the importance of the selection of the views of the input data. This is not the object of this paper, and it has been previously studied by Tian et al. (2020b). A similar insight can be obtained with Tsai et al. (2020)'s 's framework.
Importantly, such benefits can formalised when directly maximising the lower bound (1), as this maximisation unifies the theory from Zimmermann et al. (2021) and Von Kügelgenn et al. (2021).
Theorem 1 (Informal. Details in Appendix B). Assume there is true data generating process X o (), where g is invertible and V₁ and V22 can be understood gi ((71) and g (2)) Then,
1. Under Zimmermann et al. (2021)'s conditions, maximising (1) ensures the projections Z equate the true latent variables Z up to affine transformations.
2. Assume also the true latent variables are separated into semantic (or content) S and irrel- evant (or style) U variables, such that 2 = [S, U}, where []] is the concatenation operator. Then, under Von Kügelgen et al. (2021)'s conditions, maximising (1) ensures that the pro- jections isolate semantic information, in the sense that there is a bijection from Z to S.
MUTLI-VIEW SSL METHODS MAXIMISE MUTUAL INFORMATION
In this section, we demonstrate how many different multi-view SSL methods aim to maximise the mutual information 1(Z,, Z2). ). In Figure 1, schematically highlight the four prototypes that all main multi-view SSL methods that we are aware of can be partitioned into.
In the following, exhibit how under the lens of the decomposition (1), the different methods employ different reconstruction densities, 42, Zi or 4w>|2i> and different ways to maximise or control the entropy, H(Z)) or H(W2), which is empirically shown to be controlled in Appendix H. Importantly, none of them directly maximises the lower bound (1) of I 1(Z,, Zz), preventing them from the theoretical benefits highlighted in Theorem
First, in Section 3.1, we study contrastive methods (Tian et al., 2020a;b; Bachman al., 2019; Chen et al., 2020a) and later, in Section 3.2, pure latent variables' (or projections') reconstruction methods (Grill et al., 2020; Chen & He, 2021; Caron et al., 2018; 2020; 2021).
3

---

Under review as a conference paper at ICLR 2023
(a)
(b)
fo
Vi
R
Zi
Vi
R1
Z1
Wi
predict
X
predict
12
V2
R2
Z2
Eg
V/2
R2
Z2
W2
Jv
Jo
(0)
(d)
fo
fo
m
Vi
122
R1
predict
.cll
predict
1/2
R2
Za
1/2
R₂
Z2
W2
ft
F
f
m
Figure I: Graphical representation of main multi-view SSL prototypes. Solid lines describe the main variable flow: an image X is transformed with augmentations 4, la to generate two views V1, V22 that are encoded into representations R1, Rg and projected into Z1, Zz (and potentially further processed into W1, W2). Dashed lines describe method objectives, and dotted lines indicate optional relationships between variables. Top row: the parameters of the encoder f and projector æ are shared for the processing of both views and the projections are manipulated so projections of one view are predictive of the other, and vice-versa. Bottom row: the parameters of the processing of V/2 are distinct and the projections are manipulated so that projections of V1 are predictive of projections of V/2 Left column: the projections are not further processed. Right column: the projections are further processed into a surrogate variable W1, W2 (potentially using another variable C), and then are manipulated so that projections of one view are predictive of the surrogate variable of the other. For example, (a) is followed by SimCLR and EntContnt, (b) is followed by SwAV and EntRecDi sc, (c) is followed by BYOL, and (d) is followed by DINO.
3.1 CONTRASTIVE LEARNING METHODS
Contrastive learning methods (Wu et al., 2018; Bachman et al., 2019; Tian et al., 2020a;b; Chen al., 2020a;b; He et al., 2020; Ramapuram et al., 2021) have the InfoNCE loss (Oord et al., 2018) at their core and usually have a symmetric structure (Figure la).
Consider a batch of data samples X(1kk), For each projection of each view of a sample ( say Z these methods consider the projection of the other view of that image Z its positive pair and try to identify such a pair among set of other projections (or negative pairs) by minimising cross- entropy loss based on a similarity score. This similarity score is usually defined as the temperature normalised cosine similarity sim(:.-)/T Then, the different methods are essentially distinguished by the projections they consider negative pairs.
In what follows, we introduce two representatives of these methods and their relationship with max- imising 1, Z2). In Appendix C, we give the details and caveats of the analyses and discuss further contrastive methods, such as (He et al., 2020, MoCo), that inherit the analyses below.
Contrastive Multiview Coding In contrastive multiview coding (Tian al., 2020a;b, CMC), the negative pairs of a projection from one branch, say z(0), are all the other projections from the opposite branch z That is, for batch of size is the optimised loss is
k
exp (sim( 2. 2(00 )/T)
(0)
log
(2)
E 1= exp (sim( ZZ , 2 2(5))
b=1
where ō is the opposite branch of b. This loss benefits from all the properties of the InfoNCE (Oord et al., 2018; Poole et al., 2019). For example, if both sets of projections z(") as well as z are i..d., then minimising Leme maximises lower bound on the mutual information; more precisely i(Z; Zz) 2 log Leve(()) (0) However, minimising (2) does not directly maximise the lower bound (1). Looking at (10) in Appendix C, we see how the numerator of the logarithm in (2) is
4

---

Under review as a conference paper at ICLR 2023
recovered by considering the reconstruction density 92a/2;=z; to be von Mises Fisher density with mean direction 21 and parameter 1/T and symmetrising with the reconstruction of the other branch. However, the denominator of the logarithm is not a Joe (1989)'s kernel density estimator (KDE) approximation of the entropy H(Z2) since the average of the logarithm of the kernel is taken over samples of Pz and not 1zz.. Hence, it only maximises an estimation of the entropy if Z₁ and Ze have the same (or approximately the same) marginals, i.e. Pz, ~ Pzz.
SimCLR In SimCLR (Chen et al., 2020a). the negative pairs of a projection from one branch, say 2 all the other projections z/-1), _g(1h) That is, for a batch of size the optimised loss is
2
k
exp (sim 2,00 //)
CSICLLR(0)
log
EB= Σ j=1 (((i b) * (j.b)) exp (sim - , 1/7))/)))
(3)
2k i=1
This loss does not directly inherit the InfoNCE properties as the CMC loss (see Appendix C.2.1). However, (3) can be approximately rewritten according to the decomposition (1), as is demonstrated in (13) in Appendix C.2.2. The numerator of the logarithm in (3) is then recovered using a von Mises-Fisher reconstruction density, but the denominator is not an approximation of the entropy since the average of the logarithm of the kernel is taken, this time, over samples of both Izz, and Fz.- Nonetheless, this difference allows us to consider that the samples come from the mixture Pz 0.5P2, 0.5P 22. and thereby recover the KDE estimator from Joe (1989) of H(Z) with a von Mises-Fisher density kernel. Therefore, taking into account the relationship between the Shanno''nn'''s 's divergence and the entropies of two random variables results in the approximate inequality
1(1 Z2) 2 log CSimLLR (0) DDs (F,,Fz,,1 (Pzz,
which reveals that minimising (3) approximately maximises 1(71; 72) when Pz, and IP,, are equal.
3.2 PROJECTIONS' RECONSTRUCTION METHODS
The projections' reconstruction methods (Grill et al., 2020; Chen & He, 2021; Caron et al., 2020; 2021) focus on making sure that predictions from one branch are informative of those from the other branch. To achieve this goal, their loss functions consist of a term that can be understood as the reconstruction term in (1) with the appropriate density.
To avoid collapse in the absence of negative pairs, they have to employ different engineering tech- niques that as we show can help to maintain a high entropy term in (1) in different ways. Below analyse the self-distillation methods (Grill al., 2020, BYOL) and (Caron al., 2021, DINO) from our (non seff-distillation) projections' reconstruction methods such as (Chen & He, 2021, simSi am) and (Caron et al., 2018; 2020, DeepCluster er and SwAV). While DeepCluster always keeps certain level of entropy and SwAV maximises it, the other methods rely on parameters to maintain it.
cetpmaatioonooorettt viewpoint based on (1). In Appendix D, we further analyse other
In Bootstrap Your Own Latent (Grill et al., 2020, BYOL), they consider asymmetric structure (Figure 1c) and try to predict the projections from the bottom branch Zz using the predic- tions of the top branch Z, and small predictor network ga. For this purpose, they try to minimise the &2 normalised mean squared error,
BYOL
k
LBYOL (8) =
go (z()
=
2
1
sim (oo (z (2("),2()))
using gradient descent, where 7 = a/llall aa|a||]]]] Note that this is equivalent, up to constants that do not affect the optimisation, to maximising the reconstruction term in the decomposition (1) with a von Mises-Fisher reconstruction density with mean direction go (o()) and concentration parameter 1, i... 9z, nı= (22) x exp (sim(ga (gn(gg(()))))) 22)) Note that the parameters of the branch that outputs the predicted projections Ze are parameterised with different parameters Hence, if these parameters were fixed or modified so that H(Z)) is increasing or maintained constant, then minimising LBYOL
In fact, the density is Gaussian, hut it essentially turns into an unnormalised von Miscs-Fisher after noting all samples employed in the estimation have norm
5

---

Under review as a conference paper at ICLR 2023
would indeed maximise the mutual information 1(Z1; Finding way to fix or modify them in such a way is however challenging. For example, fixing ξ to random values ensures constant entropy H ((Z)), but also means that the then random projection Z2 contains very little information about X. Thus, in this case, while minimising LBYOL would maximise 1(1;; Z2), the information learned is still little as by the data processing inequality 1( 72) S 1(X; 72)) ). On the other hand, if È depends on 0, there is the risk of collapse: for example, in the extreme case of & = 0, minimising LBYOL will maximise -H( Z1) ) and an optimal solution 0* could be highly concentrated or degenerate Zi and Z2 around one point 2, under which H Zz) + -o, which clearly does not maximise (Z1; Z2).
In BYOL, they circumvent these issues by updating the parameters & during the optimisation with the moving average & + X + (1 - ^]0 for some À E (0, 1) close to 1. The idea (hypothesis) is two-fold: on the one hand, while { does depend on 0, the dependence is weak enough so that H(Zz) is not degrading to values yielding trivial bounds; and on the other hand, the dependence of È on 8, while weak, still makes sure that the representations Zg capture information about the data. This hypothesis is backed up by the results sweeping the parameter À in (Grill et al., 2020). In fact, it has later been seen (Caron et al., 2021) 1) that this dependence resembles a exponential decay (Polyak & Juditsky, 1992; Ruppert, 1988), which is standard practice to improve the performance of the model, e.g. (Jean et al., 2014).
averaging with
DINO
Caron et al. (2021, DINO) also consider an asymmetric structure (Figure 1d) and, simi- larly to DeepCluster and SwAV, generate a discrete surrogate variable W₂ = ((Z)) and try to minimise a cross entropy term. More precisely, they minimise
(0)) = A: Σ (((()) - C)/r2) logs (ogs(<(0/m)) /T),
where C' is some centring variable, TI. AZ are temperature hyperparameters, and s the softmax oper- ator. Letting Pw, = = (((22 (2)=2(() C)/TT) and qw = s(21/T)) shows that minimising LDIOO directly maximises the reconstruction term in the decomposition (1) of 1(Z1, W2) S 1(Z1, Zz).
DINO does not directly maximise the entropy H(W2) to avoid collapse. However, they promote a high conditional entropy H(W2) Z2) through the centring before the softmax operation defining Pwalz,. To be precise, the centre C is updated with a moving average of the previous projections,
that is, C + pC+ 0 Σ -i=1& Ze for some μ (0, 1). Then, the right balance between the moving average and temperature parameters and T2 adjusts how uniform the conditional density Pwa1Z, should be. Hence, since H(W,) Z2) < H(2)), controlling the conditional entropy controls HWW)).
Finally, similarly to BYOL, DINO faces the potential problem of obtaining useless representations due to uninformative targets if the parameters & do not ensure that the projections Zz capture enough information about the data X. They solve this issue in BYOL updating them with a moving average & < Xg + (1 - 2)0 for some À E (0, 1). As previously mentioned, with the appropriate selection of X, this resembles Ruppert, 1988) and makes sure that Zn captures information about the data X (Caron et al., 2021).
Polyak-Ruppert averaging with exponential decay (Polyak & Juditsky, 1992;
4 THE ENTREC METHOD
Previously, we established how many multi-view SSL methods aim maximise the mutual infor- mation between the projections on both branches I 1( 72), and that they can be understood by decomposing the mutual information into an entropy and a reconstruction term as in (1). However, none of these methods takes such a decomposition and tries to maximise these two terms directly.
In this section, we present the Ret method which does exactly that, and naively maximises both entropy and reconstruction terms. The method comes in two variants: (i) EntRecCont, direct maximisation of I(Z,, Z2), where the entropy is estimated with a KDE, that follows Figure la's structure; or (ii) EntRecDisc, generation of a discrete surrogate random variable Wi = o(Z)) and posterior maximisation of (I(Z;; W2) + I(W1; Z2)) /2 < I(Z;; Z2), where the entropy of W. can be easily estimated with a plug-in estimator, that follows Figure Ic's structure. Ent unlike all of the methods described previously, directly maximises the lower bound (1) on the mu- tual information 1(Z₁: Zz), and therefore enjoys the theoretical properties from Theorem 1, such recovering true latent variables and separating semantic from irrelevant information. However, the KDE requires a large number of samples to properly estimate the entropy. Ent sc ad- dresses this potential drawback by estimating the entropy of a discrete (surrogate) random variable

---

Under review as a conference paper at ICLR 2023
instead, although the price of maximising lower bound on I(W1; Z2) only. Hence, it maximises a looser bound on 1( Z2) and does not benefit from the theoretical properties of Theorem 1. The pseudocode of the algorithm is given in Appendix E.
4.1 ENTRECCONT
EntRecCont considers the mutual information decomposition from (1) and directly maximises an estimate of the lower bound by maximising the loss function
A(Zz, 2 (1kk)
log (9% 2,-2 (9% (2 )
LemReeCoom (0) =
(4)
b=1
where HZZ;; Zf =(( is an estimate of the entropy and 4z,l7, is parameterised reconstruction den- sity. Multiple potential estimates of the entropy H H(.)) exist, but this paper considers those generated with Joe (1989)'s KDE, which comes with the same caveats mentioned for the analysis contrastive methods in Appendix C.3. More precisely, the estimator takes the form
- b ))
H(z,, ((akk))
Σ kha
log
h
j=1
where q is some kernel density and IR+ is its bandwidth.
The loss (4) recovers Wang & Isola (2020)'s alignment-uniformmis loss up to constants independent of the parameters 8 when the reconstruction density is qz, 775- (7,=2,((2) x exp (- -112 =l and
the kernel density is Gaussian. Applying the log-sum inequality to the entropy estimation term fully recovers that loss, revealing is an estimation of looser bound of the mutual information 1(2,, Z2)-
Moreover, Ent RecCont enjoys the following theoretical benefits on its asymptotic behaviour.
Theorem 2 (The EntRec loss (4) tends proper bound 1( Z2)). If X; are i.i.d. for all i e [k] and and π do not use batch normalisation, then, for constant bandwidth h > 0
lim LEmtRecConn -CDD-KD k)) - Σ k+000
H H(Z)) + log (92.12, (2a)) ((2)1)(())))] -1(2;; 72), (5)
where OKDE E O(h²) and the convergence rate is detailed in Appendix F.I. Moreover if h O(k - for some small and 4, then (5) still holds and CKDE (h, k)
This theorem reveals that as the batch size increases, RecCont maximises exactly the lower bound (1) and therefore it enjoys the theoretical properties highlighted in Section 2 (details in Ap- pendix B): (i) if there exists a true generative process that generates the data X from some true latent variables, maximising (4) may recover these latent variables; and (ii) these latent variables are separated into some semantic (or content) variables and some irrelevant (or style) variables, maximising (4) may isolate and recover the content latent variables.
4.2 ENTRECDISC
The KDE of the entropy converges slowly for high dimensions (c.f. Theorem 2), which requires large batch sizes. is also computationally costly (requires Okk? operations like other con- strastive methods such CMC SimCLR, as seen comparing (10) and (13) with (4)) for large batch sizes. Therefore, it is suitable to have an alternative to (4) that does not involve KDEs, for example considering discrete random variables instead of continuous ones.
EntRecDi sc considers a discrete surrogate random variable W,, = (b) and maximises a lower bound of (I(Z,; W2) + 1(Z2: W1)) /2. This way, (i) considering the lower bound from (1) to each of the terms allows to deal with the entropy of a discrete random variable and avoid KDEs, and (ii) by the data processing inequality, we are still maximising a lower bound on 1(Z1. Z2). To be precise, this version of Ent minimises the loss function
2
1 2 Σ LimtReeDiee (0) = 1
(H(WG:W,(Ls)) (1(1k))
log (9 20 (11( (6)
k
h=1

---

Under review as a conference paper at ICLR 2023
At first sight, both (4) and (6) seem indistinguishable except from the fact that now Wi is replacing Zo and that the loss is looser bound on Z2): However, this extra processing of the projections allows to estimate the entropy better and to calculate the reconstruction term (cross-entrope) exactly. For instance, let 2,=2s = s(2b), where s is the softmax operator. Then, W& is a discrete random variable in [d] and (W)) may be estimated with the plug-in estimator using the empirical estimate of the marginal, i.e. Pw, = Σx= Pw.1z,=2, Finally, letting the reconstruction density be Z = s(z;) results in the following loss function
1 2
s(zz)) T log C.. h: -. + )(( log (s(z (7)
DiiiReeDisc (0)
As opposed to the KDE of the entropy, the plug-in estimate only requires (kk) operations and converges faster than the KDE (Appendix F.2). Thus equation (7) has a better asymptotic behaviour than (4). This is formalised in the following theorem.
Theorem 3 (The EntRec loss (7) tends to proper bound on 1(21; If X, are iid. for all i E [k:] and f and at do not use batch normalisation, then
1 Σ
(H(W))+B(( + E [1o8 (ams1z, (Wa))) 2 -1(11; 22),
lim Limee (0) k-000
(8)
b=1
where the convergence rate is detailed in Appendix F.2.
5 EXPERIMENTS
While Ent Rec has a principled derivation that allows for direct maximisation of the lower bound in (I) and as result is equipped with desirable theoretic properties, its practical use has yet to be investigated. We do this in a series of experiments, in all of which we use ImageNet (Deng al., 2009) for pre-training. We compare both variants of EntRec with all methods analysed in Section 3. All experimental details can be found in Appendix G.
In the main part of the paper, we analyse how Rnt compares the other methods in terms of top-1 classification accuracy on the ImageNet test set under the linear evaluation protocol, and further study how changes in individual training hyperparameters affect this metric for each method if no other training or model hyperparameter is changed. In Appendix H, we further include comparison in terms of transfer learning performance (fine-tuned top-1 classification accuracy on 10 natural image data sets that differ from ImageNet) and a qualitative analysis of the behaviour of the entropy term during training for Ent RecDiec ss and DINO (for which the entropy can be estimated accurately as they discrete surrogate variables).
As can be seen from the first column of Table 1 and Table 2. EntRee's performance on the Im- ageNet test set is comparable to that of the contrastive methods. The same can be observed with respect to its transfer learning performance in the additional experiments that are included in Ap- pendix H.
The projections' reconstruction methods (DINO, BYOL) perform better on their optimised settings, but can struggle under changes in hyperparameters (without further adjustments). Here, we see that Ent compares overall favourably in terms of robustness to changes in training hyperparemeters:
Batch size. It known that lowering the batch size can adversely affect the performance of SSL methods (Chen et al., 2020a). In Table I we see this is also true for Ent however to a lesser extent than for all other methods. Importantly, the projections' reconstruction methods (DINO, BYOL) which rely engineering techniques to maintain high entropy can be very severely affected by lower batch sizes, which potentially call for further hyperpa- rameter adjustments to recover performance."
3 All results obtained with our implementation of the algorithms. The hyperparameters have been op- timised by us to obtain the best performance each of the SSL methods. For KntRec, we just used the parameters from S SimCIe without any further optimisation. *For example, Grill et al. (2020) use gradient accumulation so BYO can cope with lower hatch sizes.
8

---

Under review as a conference paper at ICLR 2023
Epochs. Furthermore, SSL methods typically need very high number of total training epochs to achieve their strongest performances (Grill et al., 2020). In Table we see this is also true for EntRec. However, this time another method, DINO, seems to be the least affected by a lower number of total training epochs. Still, EntRec performs comparably the remaining SSL methods and is notably more stable than BYOL again.
Altogether, these experimental results showcase that EntRec can indeed have practical use good objective for multi-view SSL beyond its theoretical benefits highlighted in the previous sec- tions.
Table Performance of the different methods across batch sizes after 400 epochs of training.
Accuracy
AAceuracy wrt. 4096
4096
2048
1024
512
SiMCIR
69.5
-3.1
-2.5
-4.5
CMC
69.1
-1.1
-2.9
-4.8
BYOL
73.8
-2.1
-22.0
-42.5
DINO
71.6
+0.5
-15.0
-67.0
EntRecCont
69.7
-0.7
-2.3
-4.2
EntRec
66.9
-1.0
-2.2
-3.9
Table Performance of the different methods across epochs with batch size of 4096.
Accuracy
Acceurcy wrt. 400
400
300
200
100
SimCLR CMC
69.5
-1.0
-2.2
-5.4
69.1
-0.6
-1.8
-5.6
BYOL
73.8
-1.8
-5.0
-13.4
DTNO
71.6
+0.2
+0.3
-1.2
EntRecCont
69.7
-0.8
-2.1
-5.9
EntRecDisc
66.9
-0.2
-1.1
-4.1
CONCLUSION
We provided unifying information theoretic analysis of common SSL methods, showing how they partially or approximately maximise lower bound to the mutual information between the learned representations of distinct views of the same datum. Based this analysis, we introduced EntRec: a simple SSL. method that directly maximises this lower bound, We showed that it possesses a range of desirable theoretical properties, such recovering the true latent variables of the underlying gen- erative process or isolating content from style in such true latent variables. Furthermore, we demon- strated empirically that its classification performance is comparable the existing SSL methods and most notably robust to changes in individual training hyperparameters such as the batch size or the number of training epochs.
Limitations and future directions Maximising mutual information is not enough to learn good representations, and strong inductive biases are important. For instance, the usage certain recon- struction densities or projection spaces when maximising the lower bound (1) grants the theoretical properties highlighted in Theorem and deepened Appendix B. Now that we know these meth- ods maximise the mutual information between the representations of different views, the next step is to (i) understand which inductive biases they possess to justify their difference performance (e.g. why DINO performs better when properly tuned), and (ii) design methods from first principles that both maximise this mutual information and have these inductive biases.

---

Under review as a conference paper at ICLR 2023
Reproducibility statement Regarding theoretical results, made effort to give clear explanations of any assumptions and complete proofs of all claims in the main part of the paper in Appendices B, C, D, and F. Regarding Appendix G that clearly states the experimental protocol used to obtain these results. Furthermore, are working release the code used in this paper as possible.
experimental results, included section in the
REFERENCES
David Barber Felix Agakov. The im algorithm: variational approach to information maximization. Advances in neural information processing systems, 16(320):201, 2004.
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information
bottleneck. In International Conference on Learning Representations, 2017. URL https: //openreview. net/ wtt.:/oorrnnnninimaaer
András Antos and loannis Kontoyiannis. Convergence properties of functional estimates for discrete distributions. Random Structures & Algorithms, 196344):133-193, 2001.
YM Asano, C Rupprecht, and A Vedaldi. Self-labelling via simultancous clustering and representa- tion learning, In International Conference Learning Representations, 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximiz- ing mutual information across views. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https httt:p::/prroceeeee /ttp://eooceeeeeeceeeeecpcc..sse neurips. cc/ pdf.
Adrien Bardes, Jean Ponce, and Yann Lecun. Vicreg: Variance-ivvarincc-covvaaae regularization for supervieed learning. In ICLR 2022-10th International Conference on Learning Represen- tations, 2022.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-99244 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 2tti::200000022999 2020b.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- crarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Marco Federici. Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.00017 2020.
10

---

Under review as a conference paper at ICLR 2023
Igor' Vladimirovich Girsanov. On property of non-degenerate diffusion processes. Theory of Probability & Its Applications, 4(3):329 333, 1959.
Ziv Goldfeld and Yury Polyanskiy. The information bottleneck problem and its applications in machine learning. IEEE Journal on Selected Areas in Information Theory, 1(1):19-38, 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 1tdi::700.00000 02677, 2017.
Robert M Gray. Entropy and information theory. Springer Science & Business Media, 2011.
Nean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 1111222-222820 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Ciled on, 14(8):2, 2012.
Devon Hjelm, Alex Fedorov, Samuel Maroii-Marc,ildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: (etiv:8888.000 066700 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-454-45 PMLR, 2015.
Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. arXiv preprint arXiv: 141..207 2014.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 43(11):4037-4058, 2020.
Harry Joc. Estimation of entropy and other functionals of multivariate density. Annals of the Institute of Statistical Mathematics, 41(4)6683-697 1989.
Akshay Krishnamurthy and Yipei Wang. Lectures on information processing and learning. UC Berkeley lecture notes, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Commmnccatioss the ACM, 60(6):84-99) 2017.
Shuo Liu, Adria Mallol-Ragolde Emilia Parada-Cabelede Kun Qian, Xin Jing, Alexander Kathan, Bin Hu, and Bjoemn W Schuller. Audio self-supervised learning: A survey. arXiv preprint arXiv:2203.01205. 2022.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 160iv: 03983, 2016.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information.
In International Conference on Artificial Intelligence and Statistics, pp. 875-884. PMLR, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint / 2018

---

Under review as a conference paper at ICLR 2023
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838 -55188888555,,, 1992.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Muchine Learning, pp. 5171- 5180. PMLR, 2019.
Jason Ramapuram, Dan Busbridge, Xavier Suau, and Russ Webb. Stochastic contrastive learning. arXiv preprint arXiv:2110.00552 2021.
Borja Rodriguez Gálvez, Ragnar Thobaben, and Mikael Skoglund. The convex information bottle- neck lagrangian. Entropy, 22(1):98, 2020.
David Ruppert. Efficient estimators from slowly convergent robbins-monro procedure. School of Oper. Res. and Ind. Eng., Cornell Univ, Ithaca, NY, Tech. Rep, 781, 1988.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124020, 2019.
Ravid Shwartz-Ziv, Randall Balestriero, and Yann LeCun. What do maximize in self-supervised learning? In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022.
Richard Sinkhorn. Diagonal equivalence to matrices with prescribed row and column sums. ii. Proceedings of the American Mathematical Society, 4552):195-198, 1974.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pp. 776-794. Springer, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-6839, 2020b.
Naftali Tishby, Fernando Percira, and William Bialek. The information bottleneck method. arXiv preprint physicc/00004057.000 2000.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised learning from a multi-view perspective. In International Conference on Learning Representations, 2020.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.1362. 2019.
Petar Velickovic, William Fedus, William Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster), 2(3):4, 2019.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:1 14451- 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929- PMIR, 2020.
Mike Wu, Milan Mosse, Chengxu Zhuang, Daniel Yamins, and Noah Goodman. Conditional neg- ative sampling for contrastive learning of visual representations. In International Conference on Learning Representations, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
12

---

Under review as a conference paper at ICLR 2023
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv 2017.
preprint 1708.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervieed learning via redundancy reduction. In International Conference Machine Learning, pp. 12310- 12320. PMLR, 2021.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pp. 12979-12990. PMLR, 2021.
13

---

Under review as a conference paper at ICLR 2023
Appendix
A RELATED WORK
This section of the Appendix puts into perspective this paper with respect to the relevant work closest to ours. First, it does so with other works that study (multi-view) SSL around mutual information, namely (Tian et al., 2020a; Tsai et al., 2020; Federici et al., 2020). Then, it continues with other works (Poole et al., 2019; Shwartz-Ziv et al.) that analysed the mutual information that particular SSL algorithms, InfoNCE-basee describing how the relationship between Zimmermann et al. (2021) and Von Kügelgen et al. (2021) is expanded in this work.
and (Bardes et al., 2022, VICReg), is maximising, It finalises
A.1 MUTUAL INFORMATION- BAAED FRAMEWORKS FOR MULTI-VIEN SSL
Tian et al. (2020h) focus their study on how to generate the views in multi-view SSL so that the learned representations have good downstream performance. They focus their analysis assuming contrastive learning algorithm that maximises the InfoNCE (Oord et al., 2018) lower bound on the mutual information between the views (and more precisely on the projections of said views). They conclude that the views should share the information of a desired downstream task and nothing else, and provide with an algorithm to enforce that. Then, Tsai et al. (2020) consider multi-view SSL algorithms where one view is always the unperturbed input and show that, under certain conditions, these methods can keep rask-relevant and discard task-irrelevant information for some downstream task. This is similar to (Federici et al., 2020), where they show similar conclusion for multi-view version of the information bottleneck (Tishby et al., 2000).
Instead, in Section 2 of this paper, we assume perfect view selection to gain intuition on how maximising the mutual information between the representations of the different views of the input data encourages the learning of semantic information and the discarding of irrelevant information. After that, we show that under the conditions of Zimmermann et al. (2021) and Von Kügelggnn et al. (2021), their theory is unified when maximising the lower bound (1) on this mutual information. More precisely, saying that the maximisation of the mutual information between the representations of different views of the same input datum can recover the true explanatory factors of the input data and separate semantic from irrelevant information.
A.2 STUDY OF SPECIFIC ALGORIITMM
Poole et al. (2019) showed that InfoNCE maximised a lower-bound on the mutual information, which includes PnfoNCEE-bsse methods such as the CMC. This analysis was complemented by Wu al. (2020) that showed that as long as the negative samples are i.i.d., they may come from diffrent distribution than the positive samples. Shwartz-Ziv et al. show that (Bardes et al., 2022, VICReg) maximises the mutual information between the representations of one view and the other views. Similarly, show how many multi-view SSL methods maximise the mutual information between the representations of different views of the input data. More precisely, we lift the i.i.d. assumption for contrastive methods and include an analysis of (Chen al., 2020a, S MCLRR) completing the picture of contrastive learning methods, as well as analysis for other and latent variables' reconstruc- tion methods like (Chen & He, 2021, SimSi i am), (Grill al., 2020, BYOL), (Caron et al., 2018; 2020, DeepCl uster and SwAV), and (Caron et al., 2021, DINO).
A.3 RELATIONSHIP BETWEEN RECOVERING TRUE LATENT VARIABLES AND SEPARATING CONTENT FROM STYLE
Von Küggelgen et al. (2021) already discussed some connections of their work and (Zimmermann ct 2021). In particular, they mentioned that their equation (1) could be interpreted as an alignment (numerator) and uniformity (denominator) terms, and that the latter constituted nonparametric estimator of the entropy as the batch size grew to infinity. Therefore, they related their work with (Wang& & Isola, 2020) and by proxy to (Zimmermann et al., 2021).
However, this relationship is not direct. The practical implementation of the uniformity term in (Zimmermann et al., 2021) is equivalent to KDE estimation of the entropy, and tends to the
14

---

Under review as a conference paper at ICLR 2023
entropy at the rate stated in our Theorem 2. In Von Kügelgen et al. (2021), they consider the real entropy for their theorem, and not an approximation. So the connection in (Von Kügelgen et al,, 2021) with (Zimmermann et al., 2021) only holds in the asymptotic regime.
Conversely, we prove that both results hold when maximising the lower bound (1) on the mutual information, without using the uniformity loss but the entropy directly. That is, both results unified through mutual information maximisation. Therefore, the connection in our analysis holds without the need of employing neither the uniformity loss nor KDE estimation of the entropy.
B MAXIMISING MUTUAL INFORMATION PROPERTIES
This section of the Appendix formalises and contextualises Theorem I's statement.
B.I RECOVERING THE TRUE LATENT VARIABLES
Let us consider the standard assumption independent components analysis (ICA), namely that the data X is generated by a nonlinear, invertible generative process = g()) from some original latent variables Ż. Assume further that the different views from the image can be understood as V1 = 1 and 1/2 = g(Z)), where there is some joint density of the latent variables P2..2.' The next theorem shows how Zimmermann et al. (2021) theory can be adapted to prove that mueli-view SSL methods that maximise the mutual information between their projections 1(21; Z2) can obtain projections equivalent to the true latent variables up to affine transformations.
Theorem 4. Assume that the latent variables and the network's projections lay on convex body Z Rd. Further assume that the latent variables' marginal distribution is uniform and that the conditional density described by a cemi-metric p as P P (32) = C(&)ee ((5) Now let the ((EE.))))))))))
reconstruction density match the conditional density up constant scaling factor AZa|Zi=ai (22) = - If the generative process 9 and the parameterised network functions ι f are invertible and differentiable, and the parameters O maximise the lower bound (1) of the mutual information 1(Z1; Z2), then the projections are equivalent to the true latent variables up to affine transformations.
Proof. As in (Zimmermann al., 2021), let h = fo g be a helper function that brings the true latent variables to the projections that Z₁ = h(Zz) and Zg (Z2).
Disregard for a moment the entropy term HZZ)). From (Zimmermann et al., 2021, Proposition 4) we know that if the reconstruction term is maximised (the cross entropy is minimised) then p(ă ε2) ap(h(2)), ), ((22)) and C(2)) = C2 Moreover, from (Zimmermann et al., 2021, Theorem 4) we further know that h is an invertible affine transformation; i.e. Z2 = AZ2 + for some A E Rddd and some b E Rd
Now note that
H(Z)) -E log EE
C2,, C, ( (0(M(21)
-E log IE (C(Z))e-P(2..
H(Zz).
Then, since the latent variables' are uniformly distributed, their entropy is maximal H(Ž) log |Z]. Therefore, the unique family of maximisers of the reconstruction term recover the latent variables up to affine transformations are maximisers of the entropy, and hence the unique family of maximisers of the mutual information. Indeed, take some other maximiser of the entropy, it is from this family, it is not a maximiser of the reconstruction and therefore the resulting mutual information is lower.
Remark 2. Following the same reasoning and supporting Zimmermann al. (2021)'s theory, we may note that in the particular case that the semi-metric p is described by an LP norm, then the projections are equivalent the true latent variables up to generalised permulations; that is, 7, AZ for some A E &daxd such that (Az); o:Bizo where a E R, B E (1, (1,-1), and is
15

---

Under review as a conference paper at ICLR 2023
permutation. Similarly, in the more restrictive case that the projections are in the sphere Z = = and the conditional densities are von Fisher densities, then the projections are equivalent to the true latent variables up to linear transformations; that is, Z AZ for some E gaxdd such that ATA = al for some I.
B.2 ISOLATING SEMANTIC FROM IRRELEVANT INFORMATION
Similarly to Appendix B.1, 1et us consider that the data X is generated by nonlinear, invertible generative process X = g Z) from some original latent variables Z and that the different views can be understood as V1 (Z)) and 2/2 = 2(2)) where there is some joint density of the latent variables ,2
Assume that the latent variables can be written as 2 1S. UJ. where S F is some semantic (or content) variable, U E Rdd is some irrelevant (or style) variable, and [-] denotes the concatenta- tion operation. Furthermore, let adopt the assumptions from Von Kügelgen et al. (2021) for the content-preserving conditional density P/212--
Assumption 1 (Content-invariance) The conditional density P2,12, of the latent variables of dif- ferent views has the form
1/ (32) 8(82 81 P1) (w2).
for all & [[1,, U1] and bg [82, U2) in Z and where Pt,, Urmuy is continuous for all u1 E R. Assumption 2 (Style changes). Let A be the set of subsets of irrelevant variables A C (l,...,duu and let PA be a density on A. Then, the conditional density PUa|U1 is obtained via
a ~ PA PU. (0llU.A==a (= = 8(u2,a a1 Milaa=)PUu,uluu.m JU (u2,a)
where PU2.aUU1. =11a is a continuous density for all U1,a Ielol, and where U2,a is an abuse of notation to refer to the elements of u2 indexed by and analogously for un and for aº.
Then, the next theorem shows how Von Kügelgen et al. (2021) can be adapted to prove that multi- view SSL methods that maximise the mutual information between the projections 1(Z,; Z2) can obtain projections that capture and isolate the semantic information of the true latent variables.
Theorem 5, Consider Assumption I and Assumption and further assume that
the generative process g smooth, invertible and with a smooth inverse (iee a diffeomor- phism);
2. Pa is smooth, continuous density Z with Pı 0 Dae.,, and
3. for any {l...., Ryy, there an a S. {l...., Thy such that j a. PA(a) > O. PUs..W0,..u1,. (u2,a) is smooth with respect to both u1aa and W2 and any U1,a it holds that lU,.ali., (u2,a) > 0 for all u2,a in some open, non-empty subset containing W1.a-
the parameterised network retworkfuncttin o f smooth, the projections space (0, 1)d gd, and the parameters are found to maximise the mutual information 1(21; Z2) lower bound (1) with the reconstruction density 4zz|2==z1 (22) Cganss (1)e 1| =al1,, then there is a bijection between the projections Z and the true semantic variables
-1)a--1=>--
Proof. The proof follows directly by Von Kügeegeen al. (2021, Theorem 4.4) by noting that the minimising the mutual information lower bound (1) with the reconstruction density 42,|2,=2, (#2)
Ogauss (1)e =1 1 coincides with the minimisation of their theorem.
C CONTRASTIVE LEARNING METHODS AND MUTUAL INFORMATION
As mentioned in the main text, contrastive learning methods are inspired by the contrastive loss in the InfoNCE Oord et al. (2018). The objective in Oord et al. (2018) is maximise the mutual
16

---

Under review as a conference paper at ICLR 2023
information between the representations R and the data X. To do so, they consider batch of ls i..d. samples (1:kk)) and try to ensure that the representation R( of sample X can correctly predict which sample generated among all other samples from the batch. More precisely, they minimise
s(X() R
(0)
E log
2j=1 E)=1 ((X(), RG))
using gradient descent, where & is some score or similarity function. Then, it is proven that minimis- ing (9) maximises lower bound on the mutual information between the data and the representations, namely ((;;R) 2 log k LNCE (Poole et al., 2019).
C.I
CONTRASTIVE MULTIVIEW CODING
Consider the InfoNCE loss and substitute X by Z2 and R by Zi maximise 1(21; E2). This results in
E log
Σj=1 s (2)") 2
and further considering the symmetric loss (i.e., the InfoNCE loss substituting X by Z and R by Z2) recovers LCMe. Hence, the CMC loss is exactly an InfoNCE loss with the temperature nor- malised cosine similarity the score function, s(,,)) :) exp (sim(:-)/T)) and therefore minimising
it maximises lower bound on 21(Z1; Z2).
For the analysis based on the mutual information decomposition (1), consider the re-writing of LCMC loss as follows
LCMC (0)
sim 2 2z5 ///
log
exp (sim( 2 2z 1/1)) //",)())))))
(10)
Consider the term when 2 and focus inside the parenthesis, since the analysis is analogous when
b = 1. In the reconstruction term from (I), letting the reconstruction density 42,|2,=2, be a von Mises- density with mean direction ž1 and parameter 1/T, that is
42,/2==11 (22) (T))( sim(22,>1))
recovers the first term in the parenthesis up to additive constants that do not affect the optimisation.
For the second term, consider Joe (1989)'s KDE estimator with standard Gaussian kernel density Here, we may note that the restriction of the Gaussian density to elements of norm indeed a von Mises-Fisher density and that therefore have that for all z, gd-1
Cganss (1)e
12-8
gauss (1)e 1/5 sim,)))2,,))//
when choosing bandwidth VT.
Then, if z and zZ() were come from the same distribution, Joe (1989) estimator with the above kernel density would recover the second term up to additive constants that do not affect the optimi- sation. This would the case if 2k augmentations' transformations were sampled randomly from a set T and applied independently to each data sample and 2 of them were applied independently to
each data sample (*) to generate the views (i()) (((i) However, this is often not the case and only 2 augmentations are sampled and applied to the whole batch of samples to generate the views (Chen
al., 2020a) or one branch is dedicated to generate only global views, while the other also generates
local views (Caron et al., 2020).
C.2 SIMCLR
C.2.1 SIMCLR AND THE INFONCE
Contrary to CMC, the SimCLR loss is not directly an InÉoNCE loss, since there are terms in the denominator of the logarithm that are neither independent nor identically distributed. Indeed, z(0)
17

---

Under review as a conference paper at ICLR 2023
and 2 generally not identically distributed due to differences in the transformations from each branch, and they are not independent since they both obtained from X
In fact, Poole et al. (2019)'s proof techniques cannot be directly modified to ensure that it is, even under the assumptions that the samples each batch (1=ak) are i.id. and both the encoder fo and projector Te networks are deterministic and do not employ batch normalisation, which effectively ensures that _ ((=:k) and 2 =1(kk)) are i.i.d. and that X T1 2 and X TT 2 for all i + J. To see this, note that ((2 2 _1 1( - since - only depends on 2 via X1 and is independent of all other projections. Then, similarly to (Poole et al., 2019), one may employ Donsker- Varadhan's 's variational bound the relative entropy (Gray, 2011, Theorem 2.2.1) and see that
(((100) ; =11 N E[((Z( _2 log E exp (c(Z(1.)) 22 211
(1I)
where is critic function such that the above expectations are defined and Z is identically distributed to 2 but independent of all other random variables. Then, choosing the critic
5(22
(lik) cz 222 (1:k))
log
107
Ex= Σ}= ((11 + (ib))s
s 2/8
ensures that the lower bound on ((Z(1:h) (1), _ (() is tight (an equality) if
log (Pzcne) (00 22 =2 (2) (1(1 22 ((ih))) = c(z1 ((=(1:A) (1:h)) + { =2
(1:k)
(1:k)
(1:0))
for any function Φ). Then, after averaging over all pairs (2 z ). the first term (11) becomes I.CRR loss for a score function s(,,)) :) = exp (sim( //Tr). However, the second term becomes
$(2 (1) /
EE= Eø= I( b) (i.b))s(Z,'' (2,7) ) [
log
E
2k
where again 2, is identically distributed to ()) but independent of all other random objects. Let us focus the term bi = (1, in the beginning of this note, that is
s(Z (1) =(
log
IF
E b Σド 1((1,)) + G1 (, ,2,71))
To eliminate this term, one could use the fact that x00² are i.i.d. and take the average of these terms as in (Poole al., 2019) in order to end up with the desired log term. However, this is not possible since in the denominator there are additional terms associated with z+41 for j * 2. Moreover, it is also not possible include the terms z(6) this fictional average and end up with a term log (2k 1)
since 2[z and z(z) are not i.i.d. as previously discussed.
However, consider again the assumption that the samples of each batch X (I:k) are i.i.d. and both the encoder So and projector xy networks are deterministic and do not employ batch normalisation. Further consider the score function
exp (sim(TT fo o t2()),
if no fo (x) 21
s(z, 21)
b exp (sim(T fo to(x), 21)/T)
otherwise
where t and to are the known transformations of each branch. Then, as in the InfoNCE we observe that
((X; 21 = (t1, 2
k
exp (sim (2 (((,))))))))))))))))
2(())
log k +
Σ
E
log
[ = (t1,
EB=1 En= 1((1,)) # (j. b() exp (sim (sim(z())) (z 2(9)
1=1
18

---

Under review as a conference paper at ICLR 2023
where A denotes the augmentation random transformations. Taking the expectation in both sides with respect to the augmentations leads to the same inequality on 1(X [1) Then,
(|)) = I(x; 21, A) - ((x
= -I(X; 21) + I(X; Z1) = 1(X; Z1)
where the second equality holds since the augmentations are drawn independently from the data, and the last one holds from the Markov chain X - - A.
Note that here the assumptions are heavily employed: first, they are needed for the usual requirement in the InfoNCE that I( IA (41, = ((x 2 A (41, (2)), and second, they are needed to perform the comparison Ty fa ( = 21. Taking the expectation with respect to the augmentations A and averaging with the same observation on ((X; A2) reveals that
(I(X; Z1) 1xx Z2)) log k LSimCLu(0)
(12)
From (12), between the projections 1(Zz; Z2), but also the information that each projection has about the data that is not shared by the other projection. Namely,
how minimising the SimCLR loss does not only minimises the information
1
I 1( 22) + 2 (IX: 21) + (X: Z2 2 log k: LSiNCLR (f).
where we used the equality ((X; 21) Z2; 21) 1(1; Z2) + 1(X; Z1 [2) and its analogous for 1(X; Z2). Therefore, the above bound gives no guarantee that SimCLR learns projections that
capture the semantic information, since 1(X; 21222 could be made arbitrarily large capturing all the information not shared by the views. As shown in the approximate analysis with KDEs, this bound crude.
C.2.2 SIMCLR AND KDE
As with CMC, the SimCLR loss may be re-written for an analysis based on the mutual information decomposition from (1). Namely, LSINCIR (0) is
2
2
& (((ib))
sim ((0) 200 /
log
Σ V=1j=1
+ (i,6)) exp (sim z (()))
K
i=1
(13)
Similarly before, considering von Hlises-Fisher reconstruction density with parameter 1/T re- covers the first term of the parenthesis for both b = 1 and b = 2. Also, considering the second term introduces similar problems to those in CMC since Z(a) and 2 are not identically distributed and, even they were, they are not independent. Consider the second term with the whole sum, i.e.
the parenthesis individually for and
2
2
2k log [ΣΣN I(i,b) (j,0') exp (sim (Z( 22(2z 2 )+)) 0=1i=1
1
and model the samples as coming from the distribution Pz Pz, + Pz, Then, this term is approximately Joe (1989)'s estimator of H(Z), with the exemption that some pairs of the elements in the average are not independent of each other, which may harm its performance. However, the bias of this dependence is to balance the number of samples from each distribution the mixture, and thus is not severe. Considering this estimator, disregarding the constants that do not affect the optimisation, and using the we obtain that
I (Zz Zz) 2 log LSiNCLR (f) DIs (P2, PP2)).
This equation also indicates that when both projections have similar marginals minimising the SiCLLR loss approximately maximises the mutual information 1( 72).
19

---

Under review as a conference paper at ICLR 2023
C.3 CAVEATS OF THE ANALYSIS AND RELATED WORK.
Caveats of the analysis. Entropy estimation in high dimensions performs poorly: the rate of con- vergence decreases exponentially with the increasing dimensions (Joe, 1989). However, the highest entropy distribution on a finite, convex set 2 is precisely the uniform distribution on that volume. Hence, these KDE estimators are good proxy's 's for the objective of maximising the entropy of the projections, since they force the samples to be maximally separated in Z, and therefore are good proxy for entropy maximisation and therefore mutual information maximisation. vation is also done in (Wang & Isola, 2020; Zimmermann et al., 2021), where they assess that maximising the uniformity of the samples on the projections' space results in good performance. Moreover, these analyses require that the projections of different images each branch are i.i.d., which is usually not the case due to the use of batch normalisation. The breaking of the i.i.d. as- sumption can be important in the INEE lower bound on the mutual information, nonetheless, does not discredit that the result of the KDE is good proxy to maximise the entropy.
similar obser-
Related work. The following methods are variants of CMC or SimCLR and they inherit the anal- ysis above either fully or partially. A whole new analysis is not written in the interest of space and repetition. In (Bachman al., 2019). they adapt the Deep InfoMax (Hjelm al., 2018, DIM) method to CMC and include the maximisation of local features and several level (layers of the net- work) features to the standard contrastive multi-view coding. In (Tian al., 2020a), they consider CMC with multiple views instead of only two, and in (Tian et al., 2020b), they intend to learn the augmentations that best suit the information maximisation. Then, (Wu et al., 2018) and MoCo (He et al., 2020; Chen et al., 2020b) adopt the CMC setting without the symmetrisation (Figure 1c) and use additional negative pairs from the batches in previous iterations of training. Finally, in (Rama- puram et al., 2021), they adopt the SimCLR setting, where one of the two branches is modified to generate stochastic latent variables. More precisely, the representation R = (X) ) is stochasti- cally transformed to latent variable U ~ Puj z which is later decoded into the representation space R2 Pu) The contrastive learning proceeds as usual with this reconstructed representation.
D FURTHER PROJECTIONS' RECONSTRUCTION METHODS
In this section of the Appendix, we continue the analysis started in Section 3.2 on how different projections' reconstruction methods maximise the mutual information between the projections of different views.
SimSiam Chen & He (2021, SimS i am) consider a symmetric structure (Figure la) and, as BYOL, try to predict the projections from one branch using the predictions from the other branch and a small predictor network go. With this purpose, they try to minimise the negative cosine similarity
2 ΣΣ sim (Zz), (Z)),
SiinSium (0)
k
which, as for BYOL, is equivalent to maximise the reconstruction term in (1) with von Mises-Fisher reconstruction density after symmetrising with the reconstruction from the other branch.
Basically, S simSi am is BYOL in the extreme case where = 0. The realisation from Chen et al. (2020b) is that collapse can be avoided by making sure that in the optimisation of LsinSiam (0) the parameters & are only updated that ga (Z)) gets closer to Zg and not so that Zg also gets closer to (Z)). They ensure that using a sttppggaadeeeee ent operator to Zz. However, this is sensitive to the employed parameters
in the experiments in BYOL when È 0 (Grill et al., 2020) and even in (Chen & He, 2021), where they see that the method collapses or is unstable without a predictor network and depending on where in the network batch normalisation is employed.
Similarly, SimSi am also considers a variant with a cross-entropy loss. Practically, they try to make sure that the predictions from one branch generate a similar distribution on [d] the projections from the other branch. That is, they try to minimise the cross entropy
2 k
Simsiiam (0) = k b= 1=1
(Zo)) (log((2) ((Z))
20

---

Under review as a conference paper at ICLR 2023
where s is the softmax function. Theoretically, this amounts to generating surrogate variable Wi = o(Z,) and trying to maximise 1(Z1: W2) +1(W1; Z2), thus maximising 21( Z2) due to the data processing inequality (c.f. Figure 1b). Focus on b = 2 and the decomposition of ((Zz; W2) using (1). Then, letting the reconstruction density be qwa|2,=#1
((ge(21) recovers LsimSiam (0). Finally, the same analysis as before holds for the study entropy terms 2 and H (Zz) (or collapse prevention).
DeepCluster and SWAV Both (Caron et al., 2018; Asano et al., 2019, DeepCluster) and (Caron et al., 2020, SwAV) generate discrete surrogate variables W,- = ((2) based on a cluster- ing approach and try to minimise a cross-entropy term similar to that in Simsiaam Then, they ensure that the entropy is large by engineering the clustering so that it is balanced (i.e., there are enough projections assigned to each cluster). For the rest of the section let 2 C Rd and W W= = [m].
DeepCluster has an asymmetric setting (Figure Id) but with & = 0. First, the cluster assignments w o0 ) of all the data points are obtained solving the problem
C = arginff inf 1 Ce n2
1
(1) 1lx =I, 122 S.t. l|p_"
=1,p?' E (0.1)",
where C' represent the m centroids of the clusters in 2 and p is the p.m.f. of w/( given z Then, the parameters 0 are optimised by minimising the cross entropy
DeepCluseer (0)
1 k
22 (6)
log (90()
where 9o 2 + RT is a small predictor network and clearly 9wal = = (0 This optimisation amounts to maximising the reconstruction term in (1) for 1(21:W:) S 1(2;; Zz). However, without accounting for the entropy term H(W2) the loss DispClustrr is trivially min- imised by assigning all projections to a single cluster and always predicting such a cluster (i.e.
H(W2) = 1(Zz; W2) 0). Caron et al. (2020) circumvent this issue by randomly reas- signing points from full clusters to empty clusters and sampling the images of each batch based on a uniform distribution on the clusters' labels. Hence, effectively keeping a high entropy H(W2).
SwAV has a symmetric setting (Figure 1b). Let us focus on one branch (b = 2). since the analysis of the method then follows by symmetrisation. Here, the cluster assignments w((") oi o(2))) are obtained solving the following optimisation problem
P2 = arg max Tr 1 (1ik) (((NOTPPT + te((P) PEP
using the Sinkhom-Knowe algorithm (Sinkhorn, 1974; Cuturi, 2013), where Z E Rexd C' R.xdd are the centroids (or prototypes) in Rdd P {P kx PT1, 1m/m / and P1,,
1g/k] } is the transportation polytope, and 1g is the all ones vector in R*_ Let ((0)) and P(()) denote the i-th row of C' and P2, respectively. In SwAV, both the projections and the prototypes lay in the unit hypersphere, i.e. z(d), C( E gd-1, and as seen in BYOL maximising the dot product is equivalent to minimising the squared & norm distance. Hence, the optimisation problem above is equivalent
to the otpimal transport problem of moving the l samples (11=²) IRd to the positions of the m prototypes C' with the minimum &, distance cost. Moreover, to aid the optimisation calculations, an entropic regularisation is included and solved using the Sinkhorn Sinkhorn-Knopp algorithm (Sinkhorn,
1974; Cuturi, 2013) where H(P2) D (p!(2) log 1(())
Note that the j-th element of p!()) can be understood as the probability of assigning 2 to the cluster Wx()) = j. The optimisation aims to have P2 E P and therefore P.J 1, R 1m/m, which by this interpretation would mean that pw, 1m/m. thus maximising the desired entropy H(W2)
'In (Asano et al., 2019), the clusters are obtained solving an optimal transport problem similar to SwAV. 6 "Actually, SwAV only approximately solves the problem, since the algorithm is run for only three steps.
21

---

Under review as a conference paper at ICLR 2023
in (1). Then similarly to DeepCluster, the reconstruction term in (1) for 1(21; W2) is maximised minimising the SwAV loss
k
P2 ) log (C2
LSWAV (0)
k:
b=1
for b = 2, where p = p / ) and qwa|Zı= = s(C21), hence maximising the mutual reveals that minimising Swwv
information 1( W2). An analogous analysis for the branch with the entropic regularisation assignment maximises the mutual information ((Zz; W1). In SWAV (and hence in this analysis), the prototypes are treated as parameters the network (i.e., C E 0) and updated using stochastic gradient descent minimise LswaV.
E ENTREC ALGORITHM
Below, we sketch EntRec's c' main algorithm.
Algorithm EntRec's main learning algorithm.
Input: Dataset 'D = [= batch size k, reconstruction density GRec. kernel density ĄKDE, encoder and projector networks fo and na, augmentation set T, and number of iterations iterations.
1: Set iteration =1
2: while iteration < itteraiions do
3: Draw batch x (11:k) uniformly at random from the dataset D.
4: Draw two augmentation functions t and uniformly at random from T. for all i {1....,k} k) do
5:
6:
Calculate 2| no fo to (x and 22 = no o fo o t2(x
7:
end for
8:
if EntRecDi Dsc then
9:
Calculate pı = Σ s(2
and p2 & Et= s(zb")
10: end
for all kk do
12:
if EntRecCont then
for all j E (1...., kt] do
14:
Calculate Fmn. 1 (0) = 9KDE
210 -10-)60) h
and ((..)) (0) = 9KDE
1--2(1) (5)
b
15:
end for Calculate ( (0) 1 llog
Σj= (ann) log
1
x j=1
((i.j)
khd
Rhd
Fin,2
Calculate (0 (0) 1 log (qRec( (2 (=())(()))) + log (qRea (2(1) (=())1(()) (6) )))
18:
else if EntRecDi
Calculate Em (0) 1 (pi log log (p1) p5 log (pz)
19:
Calculate Rec (0) 1 (s(z(") log (s(=(")) s(22 )0) log (s(=(
20:
21:
end if
22:
end for
23:
Calculate LEn=Roc (0)
Em Update encoder fe and projector Ta to minimise LEnEReO (0).
Σ
(0) + Rec (0))
24:
end while
26:
return encoder network fe. and throw away
F ENTREC CONVERGENCE
In Theorem and Theorem 3 state the bias and variance properties of the EntRec estima- tors, which in turn describe their convergence mean squared error (MSE). In this section of the Appendix, formalise and complement these statements.
22

---

Under review as a conference paper at ICLR 2023
F.1 EXTRECCONT CONVERGENCE
F.1.1 ENTROPY ESTIMATION AND SELECTION OF THE BANDWIDTH PARAMETER
The bias and variance of Joe (1989)'s KDE estimator HKDE of the entropy H(Zg) are (Joe, 1989, Section 4, page 695)
[[HKDE] E 0( 1/4 2h 22-22h2-00 2d) + +(4+) and
V[HKDE] - 0( O(k- + 0(k----224 2n--2d) O(1)).
Hence, as long as O(k-1/d++)) for some small e > both the bias and the variance vanish, and the estimator convergences in MSE, even if it does so at slow rate. Then, sensible choice of the bandwidth is h ~ since k- --1/(++)) + as d increases.
Under the further assumption that the distribution of Z5 is 3-smooth (i.e., belongs to the Hölder or Sobolev classes) then the bias and variance of the estimator are (Krishnamurthy & Wang, 2015)
B [[KDE]] =(hh) and V[HDEE] E c((k-1
As previously, the bias and the variance of the estimator only vanish if h E O(k-1/(d+s) for some small 8 >> 0, with the optimal choice h = k=1/(425)) Nonetheless, having bias term independent of the parameter of the optimisation not harmful itself. Hence, when the KDE estimator employed only for optimisation purposes both h OOk ((k-1/d+t)) and h E (L) may work. For instance, for the experiments using the von Mises-Fisher distribution we set h = 0.1 to match the temperature employed by (Tian et al., 2020a, CMC) and (Chen et al., 2020a, SiMCLR).
F.1.2 CROSS ENTROPY ESTIMATION
independent and identically distributed random variables with ex- pectation Ellog 17,17 (Z))]. Hence, the empirical estimator is unbiased. Similarly, the variance of the estimator is V Σ & i= log 9 12 (Z,") = o kk, where of (z.2g9 (Z6)].
Note that log 9z22 12 (Z(3))
Consider now that reconstruction density is of the form Q2,12,=,, (2x) = (((6:)))))) and that the projections lay in a convex body Z E Rd. Then, we know that log 92,12;=2z (zx) E [log C' p(2), log C). where p( 2) is the diameter of with respect to p. Therefore, by the Popoviciu's inequality variances have that og < 2) /4, which implies that for p(2) < the estimator converges in MSE. This holds for the two cases considered in this paper:
Von Fisher distribution in Z 1d-1, Here the diameter with respect to p(21; 22) ksim(&1, .22) is p(2) x² and hence the estimator converges in MSE at x² /4k) rate.
Gaussian distribution in Z = 1-1, 1] Here the diameter with respect to p(21, 22) = ||
22|l² /(o²)) is p(2) 2d/o² and hence the estimator converges in MSE at a d/(2ko² rate.
F.2
EXTRECDTSC CONVERGENCE
F.2.1 ENTROPY ESTIMATION
The plug-in in estimator Hplug-in of the entropy H(W)) is known to have the following bias and variance terms (see e.g. (Girsanov, 1959, Equations (3) and (4)) or (Antos & Kontoyiannis, 2001, Introduc- tion)):
B il E 0
(dd-))
+0(1
+
and
2k
1:2
2
1
V[Agpnge E E
0
k
R
23

---

Under review as a conference paper at ICLR 2023
where o3 V log Pw, (W&)]- The bias and the variance vanish long d is fixed and of < oc, meaning that the estimator converges in MSE.
Note that pw;
([s(2))), where is the softmax operator. Hence, we have that
V log Pw; (W)) - Ellog" Pw, pw, (Wi)l
2
s E Zo.ws log Σ Le,
- IE (log + Zb.amxx Zo. )²].
where the first inequality follows from the fact that V[X] I inequality and the formula of the softmax; and the last one from the log-sum-exp trick. Here, Zb, denotes the i-th element of the random vector Zo-
E[ the second from Jensen's 's
In the particular case where the projections lie in the sphere gd-1 we have that of s (log d 1) Similarly, if they lay in the cube [_1,1]% we have that of S (log d + 2)². Therefore, under these standard conditions the variance vanishes at rate in O(log" (d)/k) + 0(1/k²).
F.2.2 CROSS ENTROPY ESTIMATION
As in Appendix F.1.2, note that log 12. (W-)) are independent and identically distributed random variables with expectation Ellog qw, 24 (W)) Hence, the empirical estimator is unbi- ased. Similarly, the variance of the estimator is V Σ log 9w, 20 1 (W?)) = o where of = V[log 4w.iz, (Wo)]. Hence, the variance vanishes as long as og < co, meaning that the estimator converges in MSE.
As for the entropy estimation, note that qw,lz, (W)) Z())). Hence, repeating the analysis above in Appendix F.2.1 obtain that o² < [B[[log d + Zj,max Zm. and therefore for projections the sphere gd-1 the cube [-1, the variance vanishes at rate in O(log (dlok(()/kk) + (11)2))
G EXPERIMENT PROTOCOL DETAILS
G.1 PRE-TRAINING
We do unsupervised pre-training the 1000-class ImageNet training set (Deng et al., 2009) without using labels. Unless specified, our explorations use the following settings for unsupervised pre- training:?
Optimizer: We use LARS for pre-training (You et al., 2017) and apply a scaling of Ir x Batchsize / 256 (linear scaling (Goyal et al., 2017)). The base Ir 0.1, except for BYOL and DINO (Ir = 0.3). The learning rate has cosine decay schedule (Loshchilov & Hutter, 2016; Chen et al., 2020a) with an initial linear warm-up for 10 epochs. The weight decay is le-4, except for BYOL and DINO (Ie-6). We use batch normalization (BN) (loffe & Szegedy, 2015) synchronized across devices, following Chen et al. (2020a).
Backbone. We ResNet-50 (He et al., 2016) as the default backbone.
Projection MLP. The projection MLP has 3 layers. The hidden fc is 2048-d, except for BYOL when it is 4096-d. The output size 128-d, except for BYOL (256-d) and DINO (60000-d).
Temperature. The temperature 7 for SiMCLR and CMC is 0.1. For DINO, the student temperature is 0.1, while the teacher temperature is 0.07.
Augmentations. We use DINO augmentations (Caron al., 2021) for all methods with global crops (cropping scale [0.4, 1.0]) and 8 local crops (cropping scale [0.05, 0.4]).
'These hyperparameters have been optimised by us obtain the best performance of each of the SSL methods with our implementation. For EntRec, we just used the parameters from SimCLR without any further optimisation.
24

---

Under review as a conference paper at ICLR 2023
EntRecCont reconstruction density. In Section 4.1 we intentionally left the reconstruc- tion density 92,12;=2, of EntRecCont unspecified to highlight the generality of our theoretical results. In the experiments, we choose this to be a von with mean direction 2 and parameter 10, so that they match the one used by SimCLR and CMC. Nonetheless, in Appendix H.3 we give the results of Ent RecCont with Gaus- sian distribution with mean 7/2 and variance 0.99, 89 where we can see they are of the same order than those obtained with the von Mises-Fisher distribution.
density
Float precision. We use half precision for all methods and experiments, except for DINO where full precision was found to be needed.
G.2 EVALUATION
On ImageNet, the evaluation of the learned representations is done by training a supervised linear classifier on frozen representations for one epoch in the training set, and then testing it in the test set. For transfer learning, the evaluation of the learned representations is done by supervised fine- tuning of the learned representation together with a linear classifier on the respective training set for 200000 steps with batch size of 256, following Chen et al. (2020a). We fine-tune using the RM- SProp (Hinton et al., 2012) with a learning rate 0.0003 and a cosine decay schedule with initial standard ImageNet aug- mentations (random cropping followed by random horizontal flips and normalisation, Krizhevsky al. (2017). Testing is done the respective test sets.
linear warm-up for 5 epochs. As augmentations during fine-tuning,
H ADDITIONAL EXPERIMENTS
H.1 TRANSFER LEARNING
Below analyse the transfer learning performance on a series of transfer tasks. Results can be found in Table 3. Again, we see performance of EntRec that is comparable with the performance of other multi-view SSL methods. Note that BYOL and DINO results were not finished at the time of submission, but will be added later.
Table 3: Fine-tuning transfer learning results. Pre-training on ImageNet for 400 epochs with batch size 4096. Numbers are top-1 accuracy.
Accuracy %
Flowers
CIFARIO
CIFARI00
Caltehh- 101
Aircraft
Cars
Food
SUN397
DTD
Mean
SimCLa
85.2
96.1
82.1
78.7
67.3
S7.1
87.1
$4.1
57.6
77.2
CMC
82.2
96.0
81.9
79.5
68.5
88.4
86.6
$4.0
57.6
77.2
BatRecCont BatRecDisc
81,8
96.4
80.7
77.1
68.3
58.2
86.3
54.4
57.1
76.7
81,5
95.0
82.0
79.5
66.4
$8.2
86.9
54.7
57.7
76.9
H.2 THE EFFECT OF DIRECT ENTROPY MAXIMISATION ENTROPY
For the two methods that use discrete surrogate random variables (DINO, Ent EncResc),)) we can get accurate finite samples estimations of the entropy term in (1). This allows us to analyse how including this term into the training objective with Ent RecDicc affects the behaviour of this term during training in comparison to what happens when engineering techniques are used to maintain high entropy. Figure 2 confirms that this indeed helps to maintain higher entropy: Ent Recte is maintains a non-decreasing level of entropy, while DINO slowly lowers the entropy term, however not totally collapsing.
For the methods that do not employ discrete surrogate variables, we cannot quantify how much of the potential entropy they capture, but we can observe qualitatively how this entropy is controlled during training through a KDE proxy (see Figure 3). We see that for BYOL, it is dropping at some point, but then stabilising and not collapsing either. We think it would be interesting to explore the
*This variance is selected to comply with the kernel bandwidth requirements of Theorem 2. "To fulfil the theoretical requirements of Theorem 4, restrict the projections the box 2 [1,
25

---

Under review as a conference paper at ICLR 2023
1.00
0.95
% MMx Entropy [nats]
0.90
0.85
DINO
EntRec Discrete
0.75
0.70
0.65
200
Epochs
Figure 2: Entropy dynamics during training for DINO and EntRecDisc, which allow for accurate entropy estimation through their discrete probability distributions. The entropy is shown as per- centage the total potential entropy log()1) (oW).
SINCLA
BYOL
EntRecCont
100 150
200
250
300
350
400
Spacits
Figure 3: Entropy dynamics during training for SimCLR, CMC, BYOL, and EntRecCont. The y- axis a proxy of the entropy obtained with KDE, and has been normalised for each method based the range obtained during the trajectory in order to have them all in one single plot. Note that this means that the curves can only be interpreted qualitatively and we are interested in the trend. For example, BYOL does not collapse 0 entropy but simply stabilises at level of entropy that is lower than where started from.
performance levels BYOL could reach more direct way of maintaining or maximising the entropy term can be found for this method.
H.3 RESULTS ON OTHER ENTREC RECONSTRUCTION DENSITIES
As mentioned above, we used a von Mises-Fisher density for the reconstruction density of EntRecCont in the experiments in the main part of this paper. However, EntRecCont and its theoretical properties are more general than this special case, and thus below we also include the results obtained for varying batch sizes and epochs on ImageNet with a Gaussian reconstruction density below in Table and Table 5. We see that the results are still comparable to the remaining methods and to Ent.Rttonnt with von Mises-Fisher reconstruction density (compare to Table and Table 2).
Table 4: Performance on the ImageNet test set of Ent RecCont with Gaussian reconstruction density across batch sizes after 400 epochs of training.
Accuracy
A Accuracy wrt. 4096
4096
2048
1024
512
Ent:ReCCont (Gauss)
69.4
-1.2
-2.5
-4.7
26

---

Under review as a conference paper at ICLR 2023
Table 5: Performance on the ImageNet test set of Ent with Gaussian reconstruction density across epochs with batch size 4096.
Accuracy
A Accuracy wrt. 400
400
300
200
100
EntCott (Gauss)
69.4
-0.8
-2.5
-5.6
27


Reviews:
[{"review_id": "wGJH_Zgpb0x", "reviewer": "Reviewer_r5LV", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to unveil some relationships between mutual information maximization and multi-view self-supervised learning.\nIt claims to show that many SSL approaches can be framed as maximizing mutual information, and provide yet another approach that\nconsiders maximizing mutual information as its objective (in an arguably 'more general' way). It then provides some empirical\nresults using two variants of this method. \n", "summary_of_the_review": "The paper at hand provides a new take on the relationship between multi-view self-supervised learning and mutual information maximization. While the topic is interesting, the paper falls short in many directions. The main theoretical contributions of the paper are either already known (contrastive methods optimize a lower bound on the mutual information) or hardly made precise enough to be useful (non-contrastive methods e.g. BYOL and DINO optimize a lower bound on the mutual information).\n\nI am more than doubtful of the empirical results: the results of at least one of the baselines provided are drastically below the results provided in the corresponding paper (BYOL @300 epochs is mentionned to achieve 65.8%, the original paper mentions 72.5%), and the batch size analysis displays the same trend (the paper claims that BYOL loses 34.2% accuracy when trained with batch size 512, while the original paper mentions that the loss in accuracy is about 1%).\n\nGiven both points, I strongly recommend rejecting this paper.", "strength_and_weaknesses": "**Strength:**\n- The paper tackles a potentially interesting (though quite thoroughly studied) topic.\n- The quality of the writing is above average. \n\n**Weaknesses:**\n\n__Theory:__\n- The theoretical contributions of the paper consists in showing that different self-supervised learning approaches all aim at maximizing the mutual information. This contribution falls short for several reasons:\n  - Contrastive approaches are already known to approximate a lower bound on the mutual information even though this lower bound mostly holds when both views come from the same distribution. The paper provides slightly more precise, but rather straightforward analysis for when views come from different distributions.\n  - The analysis for both BYOL and DINO is extremely handwavy: there is no clear explanation for why and how either of these algorithms would maximise the entropy term. In their current form, these paragraphs provide no precise statement relating BYOL's or DINO's objective to mutual information maximization, and I would argue in favor of removing them.\n- I fail to understand what makes EntRecCont significantly different from CMC/SimCLR; in the case where $P_{Z_1} = P_{Z_2}$. When we add back the missing - in the definition of the entropy and remove the constant terms inside of the logarithm, we fall back to a standard NTXEnt loss (potentially up to normalization of the projections). Given how close the two are, it seems necessary to at least mention the equivalence/similarity. \n- I fail to understand the usefulness of mentionning Theorem 1, given both that the KDE approximation has clear limitations, probably making\n it a very bad approximation of Equation (1) (typically the provided lower bound cannot approximate Mutual Information higher than O(ln(k)), where k is the number of 'negatives' used, as mentionned in (McAlester et al. 2018)), and there are no experimental validation that any of the methods mentionned either verifies the hypothesis of the theorem, or matches their implications.\n  \n__Experiments:__\n\nThe experimental part requires a significant rework, as it currently gives a biased picture of the performance of the algorithms. More precisely:\n  - The performance of BYOL in Table 1 is **significantly** lower than the performance reported in (Grill et al. 2020), namely the performance at 300 epochs is claimed to be 65.5% top-1, while the original paper mentions 72.5% **using the same architecture**. This changes significantly the story told by Table 1. While the difference may be explained by a change of hyperparameters, or image transformations, it seems disingenuous to place the algorithm in a setting that does not allow it to provide its best performance. I would suggest both putting back the original results in the table, and revising the claim of near SOTA performance.\n  - Similarly, the results presented in Table 2 strongly contradicts results presented in (Grill et al. 2020), where BYOL loses less than 1% top-1 accuracy when going from batch size 4096 to 512, as opposed to 34.2%(??) in this paper. Again this significantly alters the conclusion provided by the paper.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "nUXTyMEds-_", "reviewer": "Reviewer_D75g", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript provides an information theoretical perspective for multiview self-supervised learning (SSL). In specific, the authors show that a lower bound of mutual information is a useful objective function to learn informative representations. Additionally, optimizing such an objective function could be regarded as a generalization of both learning true latent variables (as in Zimmermann et al. (2021)) and separating content information from the irrelevant (as in Von Kugelgen et al. (2021)), from a generative model perspective.\n\nBesides, the work also analyzes the loss functions of different SSL methods, e.g., contrastive learning based and projection reconstruction based ones, and demonstrates that these approaches do not maximize the mutual information lower bound exactly.\n", "summary_of_the_review": "This paper gives an interesting point of view for analyzing SSL using mutual information lower bound and proposes a method based on the criterion. However, both the theoretical and empirical results are not sufficiently clear.", "strength_and_weaknesses": "**Strength**: \n\nIt is an interesting idea to generalize previous model identification results of different generative models by using the mutual information lower bound. The authors also provide detailed analysis showing that many existing multiview SSL algorithms approximately maximize such a lower bound, which could help deepen the understanding of different SSL paradigms. However, the theoretical aspects as well as the empirical ones are not sufficiently clear in its current version.\n\n**Weaknesses**:\n1. The novelty seems not significant since a similar loss has been proposed and analyzed in Wang and Isola (2020) as uniformity and alignment terms. The uniformity is the entropy term here while the alignment corresponds to the reconstruction term (or cross entropy term). Could the authors spell out the differences and the advantages?\n2. It is unclear that how Eq. (1) unifies both model identification theorems of Zimmermann et al. (2021) and Von Kugelgen et al. (2021). Specifically, contrastive loss (i.e., InfoNCE loss) is analyzed in Zimmermann’s work while $l_2$ matching loss is used in Von Kugelgen’s work. How is the equivalence between such a loss and (1) established? It is hard to tell how the proof techniques here differ from those of previous works.\n3. The convergence rate results of the proposed methods are interesting but not very clear. Could the authors give more details on how the results are derived and perhaps more explanation?\n4. It would help readers better understand if the detailed algorithm is listed.\n5. It unclear how useful the proposed method is in practice since kernel density estimation is required for each iteration. Is there a wall time or complexity comparison?\n6. In terms of experiment results, it might be more straightforward using synthetic experiments to showcase the desired theoretical properties. For example, how well the mutual information is maximized? How does that help identify the true latent variables and/or separate the content from the style, compared to the baselines? Is it necessary to learn good representations only if a tight bound is maximized?\n\nSome minor comments and questions:\n1. Both Zimmermann et al. (2021) and Von Kugelgen et al. (2021) assume that the generative function is the same for each view. As a comparison, in [1], it is shown that the generative function $g(\\cdot)$'s do not need to be identical (in a more natural multivew setting) for each view in order to recover the true latent variables. Does the proposed criterion lead to similar model identification results under such a setting? \n2. The interpretation in the first paragraph on page 3 is not very clear to the reviewer. Could the authors be more specific about what information can be and is learned is? It seems the entropy term is simply a constrain (could be other alternatives) to avoid degenerated solutions.\n3. Why is the proposed method more robust to changes of hyper-parameters, compared to other baselines?\n4. How is the parameter of kernel band width selected? Or how the density is estimated?\n5. In the Conclusion section, typo ‘or or isolating’.\n\n[1] Qi Lyu, Xiao Fu, Weiran Wang, and Songtao Lu. \"Understanding latent correlation-based multiview learning and self-supervision: An identifiability perspective.\" In International Conference on Learning Representations. 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jgZmIbePoBp", "reviewer": "Reviewer_Nfqh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a general framework to understand multi-view self-supervised learning (SSL) methods from a mutual information maximization perspective. Authors then use a naive lower bound of MI, i.e., entropy + reconstruction, as the objective function. The proposed methods, namely EntRecCont and EntRecDisc, achieve comparable performances to baselines.", "summary_of_the_review": "This paper contributes some interesting ideas and also provides a good summary to previous literature. However, the theoretical contribution seems marginal. The experiment is also a bit weak.", "strength_and_weaknesses": "Strength:\n1. The manuscript provides a good summary to previous multi-view SSL methods. \n2. It is interesting to see that a straightforward approximation of entropy (with traditional kernel density estimator or discrete entropy) and reconstruction can achieve comparable performances. \n\nWeaknesses:\n1. The main result that multi-view (augmentation-based) SSL methods maximize MI has already been published in previous work, e.g., \nhttps://arxiv.org/abs/2010.02037. For acceptance, I would expect the authors to provide convincing arguments on why their work improve over previous work in this regard.\n2. A large part of the paper reads as a report covering previous approaches to SSL, making it difficult to see the actual contribution of this work over previous work.\n3. There are very few experiments included in the paper. Moreover, current result is very weak and does not demonstrate the advantage of EntRec:\n3.1. The results show that the proposed methods perform worse than DINO (ICCV’21) which is already a year old. If the proposed methods are principled and direct approaches to the same objective which all others minimize, why do they perform worse?\n3.2. The authors claim that the low bound in Eq. (1) helps obtain good representation and separate semantics from irrelevant information. \nHowever, it is not verified in detail.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6Ww9kuzOHW", "reviewer": "Reviewer_KfpZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper deals with the topic of mutual information (MI) maximization in multi-view self-supervised learning (SSL). The paper takes an information-theoretic perspective and shows that many current self-supervised learning methods maximize a lower bound on the MI between the representations of different views---a result that is not entirely novel. Based on a well-known decomposition of the MI into a cross-entropy and an entropy term, the paper argues that it can be beneficial to estimate these two terms individually. On the theoretical side, the authors draw a connection to two recent identifiability results related to nonlinear ICA. Empirically, the paper shows that models that estimate the individual (cross-)entropy terms perform on par with existing self-supervised learning methods.", "summary_of_the_review": "The decomposition of the MI and estimation of the two individual terms (cross-entropy and entropy) is an interesting research direction, which can lead to new insights into SSL. However, in its current form, the paper provides insufficient contribution compared to the results from previous work. In particular, the authors should state their contribution more clearly and verify the benefits of the proposed objective more thoroughly.", "strength_and_weaknesses": "**Strengths:**\n- The authors test two different estimators for the entropy terms, a KDE estimator and a plug-in estimator. It's also nice that the authors add information about the asymptotic behavior of these estimators, even if these results are not entirely novel.\n- The authors reference critiques of the information-theoretic perspective on SSL objectives.\n \n\n\n**Weaknesses**\n- It is well known that many existing SSL methods can be connected via MI maximization and the contribution of the paper in this regard is not clear enough. For which methods is the MI maximization perspective actually new? This should be clearly stated in the abstract and/or introduction.\n- Analogously, the decomposition of the MI into cross-entropy and entropy terms is well known and should not be stated as a contribution (\"we show ...\") in the abstract.\n- The claim that the proposed objective \"exactly optimises both the reconstruction and entropy terms\" is misleading, because the model uses estimators.\n- The \"unification of current theoretical properties\" is overstated. The two referenced papers (Zimmermann et al.; Kügelgen et al.) are quite similar in that they show identifiability results for multi-view nonlinear ICA and related settings. Specifically, Kügelgen et al. already discuss the connection of their work to the work from Zimmermann et al.\n- The claimed \"improved robustness\" of one of the proposed models is not sufficiently clear from the experiments. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject