Poly-View Contrastive Learning
Authors: Amitis Shidani
R Devon Hjelm
Jason Ramapuram
Russell Webb
Eeshan Gunesh Dhekane
Dan Busbridge
Venue: ICLR 2024 poster (2024)
Keywords: Contrastive learning
Self-Supervised Learning
SimCLR
Multi-View
Augmentations
Multiplicity
InfoMax
Sufficient Statistics
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We look at contrastive learning with more than two related views per sample and find that for some objectives, performance can be improved compared to two views (e.g. SimCLR) at no computational cost.

Abstract: Contrastive learning typically matches pairs of related views among a number of unrelated negative views. Views can be generated (e.g. by augmentations) or be observed. We investigate matching when there are more than two related views which we call poly-view tasks,
and derive new representation learning objectives using information maximization and sufficient statistics. We show that with unlimited computation, one should maximize the number of related views, and with a fixed compute budget, it is beneficial to decrease the number of unique samples whilst increasing the number of views of those samples. In particular, poly-view contrastive models trained for 128 epochs with batch size 256 outperform SimCLR trained for 1024 epochs at batch size 4096 on ImageNet1k, challenging the belief that contrastive models require large batch sizes and many training epochs.

Published as a conference paper at ICLR 2024
POLY-VIEW CONTRASTIVE LEARNING
Amitis Shidnin
Devon Hjelm, Jason Ramapuram, Russ Webb, Eeshan Gunesh Dhekane, and Dan Bushridge Apple
Department of Statistics University of Oxford, UK
shidani@state ox ac. uk
dbusbridge@apple.coom
ABSTRACT
Contrastive learning typically matches pairs of related views among a number of unrelated negative views. Views can be generated (e.g. by augmentations) or be observed. We investigate matching when there are more than two related views which we call poly-view tasks, and derive new representation learning objectives using information maximization and sufficient statistics. We show that with un- limited computation, one should maximize the number of related views, and with a fixed compute budget, it is beneficial to decrease the number of unique samples whilst increasing the number of views of those samples. In particular, poly-view contrastive models trained for 128 epochs with batch size 256 outperform Sim- CLR trained for 1024 epochs at batch size 4096 on ImageNetlk, challenging the belief that contrastive models require large batch sizes and many training epochs.
INTRODUCTION
Self-Supervised Learning (SSL) trains models to solve tasks designed take advantage of the structure and relationships within unlabeled data (Bengio al., 2013; Balestriero et al., 2023; Logeswaran & Lee, 2018; Baevski et al., 2020; Grill et al., 2020). Contrastive learning is one form of SSL that learns representations by maximizing the similarity between conditionally sampled views of single data instance (positives) and minimizing the similarity between independently sampled views of other data instances (negatives) (Qi & Su, 2017; van den Oord al., 2018 8;; Bachman al., 2019; Hénaff et al., 2019; He et al., 2019; Tian et al., 2020a;b; Chen et al., 2020a).
One principle behind contrastive learning is Mutual Information (MI) maximization (van den Oord et al., 2018; Hjelm et al., 2019). Many works have elucidated the relationship between contrastive learning and information theory (Poole al., 2019; Tschannen et al., 2020; Lee et al., 2023; Gálvez et al., 2023). However, MI maximization is only part of the story (Tschannen et al., 2020); successful contrastive algorithms rely on negative sampling (Wang & Isola, 2020; Robinson et al., 2021; Song et al., 2016; Sohn, 2016) and data augmentation (Bachman et al., 2019; Tian et al., 2020b; Chen et al., 2020a; Fort et al., 2021; Balestricro et al., 2022b;a) to achieve strong performance.
While it is possible to design tasks that draw any number of views, contrastive works typically solve pairwise tasks, i.c they maximize the similarity of exactly two views, or positive pairs (Balestriero et al., 2023; Tian et al., 2020a). The effect of more views, or increased view multiplicity (Bachman et al., 2019), was investigated in SSL (van den Oord et al., 2018; Hjelm et al., 2019; Tian et al., 2020a; Caron et al., 2020). However, these works optimize a linear combination of pairwise tasks; increasing view multiplicity mainly improves the gradient signal to noise ratio of an equivalent lower view multiplicity task, as was observed in supervised learning (Hoffer et al., 2019; Fort al., 2021).
In this work, we investigate increasing view multiplicity in contrastive learning and the design of SSL tasks that use many views. We call these tasks poly-view to distinguish them from multi-view, as multi usually means exactly two (Tian et al., 2020a; Balestriero et al., 2023). In addition to improved signal to noise (Hoffer et al., 2019; Fort et al., 2021), poly-view tasks allow a model to access many related views at once, increasing the total information about the problem. We show theoretically and empirically that this has a positive impact on learning. We make the following contributions:
We generalize the information coormation-thhoreticc foundation of existing contrastive tasks to poly-view (Section 2.3), resulting in new family of representation learning algorithms.
'Work done during an internship Apple. For detailed breakdown of author contributions see Appendix T.

---

Published as a conference paper at ICLR 2024
2. We use the framework of sufficient statisties provide additional perspective on contrastive representation learning in the presence of multiple views, and show that in the case of two views, this reduces to the well-known SimCLR loss, providing new interpretation of con- trastive learning (Section 2.4) and another new family of representation learning objectives.
Finally, we demonstrate poly-view contrastive learning is useful for image representation learn- ing. We show that higher view multiplicity enables new compute Pareto front for contrastive learning, where it is beneficial to reduce the batch size and increase multiplicity (Section 3.2). This front shows that voly-view contrastive models trained for 128 epochs with batch size 256 outperforms SimCLR trained for 1024 epochs at batch size 4096 on ImageNetlk.
2
VIEW MULTIPLICITY IN CONTRASTIVE LEARNING
We seek to understand the role of view multiplicity contrastive learning (Definition 2.1).
Definition 2.1 (View Multiplicity). The view multiplicity M is the number of views per sample. In batched sampling, drawing K samples results in V -M x K views per batch. (Hoffer et al., 2019).
Multiple data views may occur naturally as in CLIP (Radford et al., 2021) or, as is our primary interest, be samples from an augmentation policy is common in SSL. Our goal is to develop tasks
Multiview
Poly-viow
M SimCLR/InfoNCE 7(x:y) 2 Lannee
M 2
M 2 2 Sufficient Staristics Section 2.4 I(x; Y) 2 Lsamtaa
Id
h
M -o
X
M 2
1/2
IIM
M 2 2
Lower bounds
M
Crop InfoNCE v) [(x:y) 2 EN (x.
Generalized MI Section 2.3 Z(x; Y) 2 Canewo
X2
XMI
Views
(a) View multiplicity in contrastive learning.
(b) View multiplicity generative process.
Figure I: (a) The role of multiplicity in contrastive learning. T(x: y) present the MI between two random variables x and y, while [(x; Y) is the MI between x and the set Random Variable (RV)s Y. LMetoa denotes the contrastive lower-bound achieved by each method, ignoring the constants. In the multi-crop box, La (x) is the contrastive lower-bound produced by the a-th crop/view. (b) The multiple view sample generation with generative factor c, where the main sample is generated through the generative process p. and views are generated through different view-genert generatiernn processes Π/ for [M], c.g. augmentations, The goal to find the map h² such that the reconstructed generative factor è recovers c, hence the identity map.
that use multiplicity M. We start by presenting the generative process underlying multiplicity (Section 1.. We then consider optimizing many pairwise tasks (Section 2.2). known as Multi-Crop. and show that Multi-Crop reduces the variance of the corresponding paired objective but cannot improve bounds on quantities like MI. Next, we revisit the information theoretic origin of InfoNCE, and derive new objectives that solve tasks across all views and do not decompose into pairwise tasks (Section 2.3). Finally, as the framework of sufficient statistics is natural at high multiplicity, we use it derive new objectives which solve tasks across all views (Section 2.4). All of these objectives are related, as is shown in Figure la. Before proceeding, we introduce our notation.
Notation We denote vector and set of random variables (RVs) as x and X, with corresponding densities Px and px. and realizations æ and X. Vector realizations live in spaces denoted by X. The conditional distribution of y given a realization æ is denoted Pylx=z= The expectation of scalar function f X' + IR is Exf(x)] = Exppy [ For S c S b, X asth = (xa, Xa+1, Xx. represents a set of RVs, and x (#c)
= X..b \ {xx). The density of X is the joint of its constituent RVs. MI between x and y is denoted I(x;y) and is defined over RV sets as I(X: Y). We denote the Shannon and differential entropy of x H(x), and the Kullback-Leible Divergence (KLD) between densities p and q by DKI. (p q). Finally, we write the integer set {1...., K} as [K]. and use Latin and Greek alphabet to index samples and views respectively.
os
2

---

Published as a conference paper at ICLR 2024
2.1 GENERATIVE PROCESS AND INFOMAX FOR VIEW MULTIPLICITY
We present the causal graph underlying M view XI:ar = (xa= a E [M]} generation in Figure lb.
The InfoMax principle (Linsker, 1988) proposes reconstruct unknown by optimizing h arg NEH T(x, /x)))- To avoid trivial solutions, two-view contrastive methods (van den Oord al., 2018: Hjelm al., 2019; Hénaff et al., 2019; Tian al., 2020a) perform InfoMax through a proxy task that instead maximizes a lower bound on the MI between two views [(h(x)); h(x2)), These methods rely on information about being in the information shared between each pair of views. A natural extension to two-view contrastive learning is to consider many views, where the total amount of information about с is potentially larger. In Sections 2.2 to 2.4, we investigate different approaches to solving this generalized InfoMax, beginning with Multi-Crop (Section 2.2) before considering more general MI approaches (Section 2.3) and sufficient statistics (Section 2.4).
2.2
LINEAR COMBINATIONS OF PAIR-WIS WISE TASKS
The first approach combines objectives on pairs x,, X from the set of M views X
M
Mullii Crop (1:M)= M(M
L (Xa x33).
The objective Equation is the all-pairs formulation of Tian et al. (2020a), and corresponds to Multi-Crop (Caron al., 2020; 2021) in the presence of M global views For convenience, we will refer to the objective Equation as Multi-Crop. Multi-Crop has been used numerous times in SSL, here will show how it achieves improved model performance through its connection InfoMax.
Proposition 2.1. For K independent sumples and multiplicity M denoted XI.KI..M. the Multi-Crop of any Purr in Equation has the same MI lower bound as the corresponding P
x2) 2 log(K)- [CMuiti-Crop X1.KI.MM = loo(K') EE [Caair (XIK (.K,1:2);
(2)
where the expectation is over K independent samples (see Appendix C.I for the proof).
Proposition 2.1 shows that increasing view multiplicity Multi-Crop does not improve the lower-bound compared to vanilla InfoNCE with two views. However, Multi-Crop does improve the variance the MI estimate (Proposition 2.2).
Proposition 2.2. For K independent samples and multiplicity M. M 2 3. denoted 1.KKI.MM. the Multi-Crup Crop of any Paair in Equation has a lower sample variance than the corresponding P
2(2M- - 1) Var (L, x2)] < Var [L Xz)l:
Var [Loutti-Crop (X:M))
(3)
3M 1)
where the variance is over K independent samples (see Appendix C.2 for the proof)-
Propositions 2.1 and 2.2 show that better Multi-Crop performance follows from improved gradient signal noise ratio as the supervised case (Fort et al., 2021) and supports the observations of Balestriero al. (2022b). See Appendix D for further discussion about Multi-Crop.
2.3
GENERALIZED INFORMATION MAXIMIZATION AS CONTRASTIVE LEARNING
In this subsection, we develop our first objectives that use M views at once and do not decompose into objectives over pairs of views in Section 2.2.
2.3.1 1 GENERALIZED MUTUAL INFORMATION BETWEEN M VIEWS
As InfoNCE optimizes lower bound on of the MI between two views (van den Oord et al., 2018; Poole et al., 2019), consider the One-vs-Rest Rest MI (Definition 2.2).
'The original Multi Multi-Crop also takes a mixture of smaller views and compares them to larger views, resulting in more complicated augmentation policy. As our work is focused on studying the effect of multiplicity, we do not investigate the extra benefits obtainable by also changing the augmentation policy. For investigations into augmentations, see Tian al. (2020h).
3

---

Published as a conference paper at ICLR 2024
Definition 2.2 (One-vs-Ree Rest MI). The One-vs-Rest MI for any [M] given set of M 2 Random Variables (RVs) XI:M = {xa: a E [M]] is
Xai X = DKL (PXLI
Px. Px. Px/xx
(4)
One-vs-Rest MI (Definition 2.2) aligns with generalized InfoMax (Section 2)1); can contain more information about the generative factor c. Note that due to the data processing in- equality T(x,: t & I(xa; c), estimating One-vs-Rest MI gives us a lower-bound on InfoMax.
1); the larger set X X M:M
Estimating One-vs-Rest MI Contrastive learning estimates lower-bound the MI using sample-based estimator, for example InfoNCE (van den Oord et al., 2018; Poole et al., 2019) and INwr (Hjelm et al., 2019; Nguyen al., 2008). Theorem 2.1 generalizes the INwJ lower-bound for the One-vs-Rest MI (see Appendix C.3 for the proof).
Theorem 2.1 (Generalized IxwJ). For any M 2 2, a [M], set of M random variables XI:M, and for any positive function F(M) X x X + +
I(xai X (TM) 2 Epx,,
(E (x.
X xTM)
Epg P Xx
FL (za x x + 1 IIonNWw. (5)
We can use the IcienNWJ lower bound (Theorem 2.1) for any function F(M) X' x X + R+ In order to efficiently maximize the MI, want the bound in Equation 5 to be tight as possible, which we can measure using the MI Gap (Definition 2.3).
Definition 2.3 (MI Gap). For any M 2 2, [M], set of M random variables X and map
(M) X X m IR+t of the form
9a
Pr extox
1:M (ts)
X 1:M)
(6)
PX
the MI Gap GMi (XIMMiga ga is
GMI (XiMi (aM) = I (xxixxx2 XTMM IGeaNWI Epx
9a
log g
(M)
1]. (7)
1.M
where we have written gaad) instead of go2h) (I (xF: X X 1 when the arguments are clear.
The map go (M) in Equation aggregates over M views and is called the finnctontioniunctionn
2.3.2 PROPERTIES OF THE AGGREGATION FUNCTION
(M) The choice of ga is important as it determines the MI Gap (Definition 2.3) at any multiplicity M. As we wish to employ ga to obtain a lower bound on One-vs-Rest MI, it should be
(M)
1. Interchangeable: I(x;: # l:M = I
E. x = ga (x, x a = ga
(M)
ta I:M ax IIX 1.M),
Da);
2. Reorderable: I(xa; X Ixx::
II
X (M)
= grz (M) (M) (za. X
x ( (M))
where [T( .....,,NN))
(TI1, Jn,, is permutation operator, and
3. Expandable: ga can accommodate different sized rest-sets 1 i.e. can expand to any M.
We seek non-trivial lower bounds for the One-vs-Rest MI (Fquation 5), and to minimize the MI Gap (Equation 7). The Data Processing Inequality (DPI) gives Z(xa; X 2 I (xa:Xe) for all
1:Md
X 1:M- So. I (x, X x ) 2 (M - 1) E, xx)?, provides baseline for the lower- bound for One-vs-Ress MI, leading us introduce the following requirement:
/a
(M1 should give gap that is most the gap given by the 4. Valid: The aggregation function ga
mean of pairwise comparisons with g (2)
GMI (1.MMigh ((M))
Σ GMI (1xaxX9):
M
Baa
We note that the objective introduced by Tian et al. (2020n) for the multi-view setting is indeed the average lower-bound present here.
4

---

Published as a conference paper at ICLR 2024
2.3.3 POLY-VIEW INFOMAX CONTRASTIVE OBJECTIVES
We now present the first poly-view objectives, corresponding choices of (FMM) and its aggregation
function gå (M) with the properties outlined in Section 2.3.2. For any function F(2), define F(M) and
their aggregation functions correspondingly by Equation 6 as following:
Arithmetic average:
F(A
Ba. X 1:M
log
Σ 8fa
(9)
M
Geometric average:
F(M)
Ba
X
th L:M
F(2) (F Lz).
(10)
M
By0
Both functions satisfy the properties in Section 2.3.2 (see Appendix C.4 for proof).
To establish connection to contrastive losses, we introduce notation for sampling the causal graph in Figure 1b. From the joint distribution PXI... we draw K independent samples denoted by:
(X. 1L:M): K 1==1 1(x.1.......... X.,M
{{ {(x
1M
1 i=1 XX.KKI.M i.e. X X (11)
K
Evaluating the functions in Equations and in Theorem 2.1 reveals the lower bound on One-vs- Rest MI and the Poly-view Contrastive Losses (Theorem 2.2, see Appendix C.5 for the proof).
Theorem 2.2 (Arithmetic and Geometric PVC lower bound One-vs-Rest MI). For any K, M 2 2, B K.M, [M], any scalar function f C C ~ R2 and map h X - C, we have
K
I Xai X
(B
log
1
Σ
lia
c (CB M)
Arithmmeic PVC: PVC: (12)
K
Byl
K
I Xni 1:M) cBB
log
=cBB,M) CGeomerrie PVC, (13)
K
M
BFa
where c(B, M) = log( M + 1), the expectation is over K independent samples X XI.KIIMM, and
(RRL.181.)))
L (XI:K.:.))= (R )n + Ejji En (k3,)
X,,a = h((x,a)
(14)
We have written Li instead of E, (XI:,I1.M) where the meaning is clear
Maximizing lower-bound means maximizing map h, leading to h* in Figure 1b. In Appendix C.5, we show (22) X i,0, R3, = c(B, M) + log li,0.s, where X, (X,,)) $41,9 U (xi.).
Tightness of MI Gap Valid property (Equation 8) ensures that the lower-bound for fixed M has a smaller MI Gap than the average MI Gap those views. Without loss generality, taking a = 1, valid solution guarantees that the MI Gap for M is smaller than the MI Gap for M 2. The
One would expect the lower-bound to be also increasing, which indeed is the case. In fact, we can prove more; consider that the MI Gap is monotonically non non-increasing with respect to M i.e. the MI Gap would either become tighter or stay the same as grows. We show that the aggregation functions by Equations and 10 have this property (Theorem 2.3, see Appendix C.6 for the proof).
DPI implies that for N 2 M and fixed a. I x.: X I:M
< I Xx, X t0 1:N)
Theorem 2.3. For fixed a, the MI Gap of Arithmetic and Geometric PVC are monotonically non- increasing with M:
GMi(XiMM,; ((M)) S (MIMXi i (M1)
M1 S M2.
(15)
Recovering existing methods Arithmetic and Geometric PVC optimize One-vs-Rest MI. M gives the two-view MI that SimCLR maximizes and the corresponding loss (see Appendix E.2). Additionally, for a choice of (22), we recover SigLIP (Zhai et al., 2023b), providing an information- theoretic perspective for that class of methods (see Appendix E.3).
Note that this guarantces that the lower-bound is increasing with respect M.
5

---

Published as a conference paper at ICLR 2024
2.4 FINDING GENERALIZED SUFFICIENT STATISTICS AS CONTRASTIVE LEARNING
Now we develop our second objectives that use M views at once. Using probabilistic perspective of the causal graph (Figure lb), we show how recover the generative factors with sufficient statistics (Section 2,4.1). We then explain how sufficient statistics connects to InfoMax, and derive further poly-view contrastive losses (Section 2.4.2). Finally, we will see that the approaches of MI lower- bound maximization of Section 2.3, and sufficient statistics are connected.
2.4.1 REPRESENTATIONS ARE POLY- VIEW SUFFICIENT STATISTICS
To develop an intuition for the utility of sufficient statistics for representation learning, we begin in the simplified setting of an invertible generative process, h p and lossless view generation procedure 7o I(c; To (x)) I(c; x). the function space H is large enough, then 3 E H such that č = hxx) Using the DPI for invertible functions,
have
max [(x; (xx)) = I(x; c) = max maxx)()());c)
(16)
KCH
kEH
If we let h" arg m&X T(x; h(x)), then h (x) is sufficient statistic of with respect to (see e.g. Cover & Thomas (2006)), and the information maximization here related to InfoMax.
If we knew the conditional distribution Pxle. finding the sufficient statistics T(x) of x with respect to c gives T = h² In general, we do not know Px|e: and generative processes are typically lossy.
Therefore, to make progress and find h arg maX I(x; h(x)) with sufficient statistics, we need to estimate Px/c- For this purpose, we use view multiplicity; we know from DPI that a larger set of views XI:M may contain more information about c, i.e. I(Xi:M 2 I(Xi:M c) for M2 2 M1. Our assumptions for finding the sufficient statistics Ty (x) of with respect to y are
1. The poly-view conditional Px,, X is a better estimate for Px. lc for larger M,
2. All views have the same generative factor: Tx(x.) Tx(x)).
The representations are given by a neural network and are therefore finite-dimensional. It means that the generative factor is assumed to be mnite-dimensional. theorem (Daum, 1986) proves that the conditional distributions Ix and Px,, are exponential
families, i.e. for some functions 72, T and reorderable function (Section 2.3.2) Q:
1 = (x 72 1.M) exp (Tx/a (xa) Q(-QQ(FFF))
to
#a
I.M
(17)
Px.le = rx (c) exp (To(xa) (²))..
(18)
The first assumption says that for any M, it is enough to find the sufficient statistics of Xa with X respect to J:M estimate for To (x,x)). Since the estimation of the true conditional distribution
becomes more accurate as M grows,
lim sup (Tx(xa) T Tx (xx)] + 0, lim [[Q - Q( l:M) + 0.
(19)
M-+00
M-+0
We see that sufficient statistics gives us new perspective on InfoMax for representation learning: representations for sufficient statistics of x with respect to the generative factor c, which can
be approximated by sufficient statistics of one view x., with respect to the other views 1.M"
2.4.2 POLY-VIEW SUFFICIENT CONTRASTIVE OBJECTIVES
As in Section 2.3.3, we begin by outlining our notation for samples from the empirical distribution. Let us assume that we have the following dataset of K independent M-wpless
= (xi.,
( > Ul(x;,. t {x,M))
(20)
Following Section 1, the goal is to distinguish between conditionals Px... Ix for any j A i and i.e. classify x; correctly V i E [K], giving the following procedure for finding the sufficient statisties T and Q*
and Px.. JX I
T Q* arg
arg max lra0 T.Q
(21)
TQQ
Px.a (X
E EN Px. Px.aix

---

Published as a conference paper at ICLR 2024
leading to the the sufficient statistics contrastive loss (Equation 22),
ex Qr,6
LsfffStats
-E
log Ls
L,a et Puo EE= 2x=
(22)
K
M
P.. Rs.4
==1
a=1
where x= denotes vector transposition, Ti = a( and Qí,a= = =Q(X (11MM)
Designing Q As Q parameterizes the conditional (Equation 17), it is reorderable. Choices for Q include DeepSets (Zaheer et al., 2017) and Transformers (Vaswani al., 2017). Requiring M to recover SimCLR (Chen et al., 2020a) implies Q(x) T(x), so for simplicity, we restrict ourselves to pooling operators over T. Finally, we want the representation space to have no special direction, which translates orthogonal invariance of the product of T and Q
[OT(x.)) ((0T(x)) : a}) Txxa) TQ ((T(x;) 3 aa))
(23)
i.e. Q is equivariant Q ((OT(xx)) : = OQ (xT(x)) * a)) which is satisfied by
M
Q(x = Q (1T(2)) a M
T(x3) T(X 1.M
Ta.
(24)
Bya
With the choice Q - Tá. when M = 2, LSuffSrats (Equation 22) recovers SimCLR (see Appendix E.2 for the detailed connection), and therefore lower bounds two-view MI. For general M, SuutiSiats lower bounds sac-vs-Rest MI (Theorem 2.4).
Theorem 2.4 (Sufficient Statistics lower bound One-vs-Rest MI). For any K, M 2 2, B = KM, [M]. and the choice of Q in Equation 24, we have (see Appendix C.7 for the proof)
Xai X 2 (B +
log l
(25)
K
where c(B, log(B- M + 1), the expectation is over K independent samples X1:K1:M.
Theorem 2.4 completes the connection between Sufficient Statistics and InfoMax (Section 2.1). We note that contrary to Average and Geometric PVC (Equations 9 and 10), the Sufficient Statistics objective for M > 2 (Fquation 25) cannot be written using F(2) function basis.
EXPERIMENTS
3.1 SYNTHETIC ID GAUSSIAN
Our first interests are to check intuition and to validate how well each objective bounds the
One-vs-Rest MI as described in Thcorems 2.2 and 2.4. We begin with ID Gaussian setting, which for the generative graph (Figure 1b) corresponds to Independent and Identically Distributed (i.i.d.)
samples C; ~ N (0, on) for é [K], p is identity map, and views X1,a ~ N(c,, d²) for each a E [M] and i. One can compute One-vs-Rest MI in closed form (see Appendix E.6 for the proof):
1
log (1
(
oo G²
0² (1-
(1
o
I(xiar 1IM
(26)
2
Mo.
which, as anticipated (Section 2.1), is an increasing function of M. Using the closed form for Gaussian differential entropy,
lim I(x:: X #a = H(x,) - H(xx/c) = I(x.:c), M
(27)
i.c. One-vs-Rest MI becomes better proxy for InfoMax M increases. Finally, can evaluate the conditional distribution for large M and see (see Appendix E.6 for the proof):
lim Ix...a M-+o0
ax.
(28)
validating first assumption for Sufficient Statistics (Section 2.4.1).

---

Published as a conference paper at ICLR 2024
To empirically validate claims we train Multi-Layer Perceptron (MLP) with the architecture (1->32, GeLU, 32->32) using the objectives presented in Sections 2.2, 2.3.3 and 2.4 on the synthetic Gaussian setup. We use AdamW (Loshchilov & Hutter, 2019) with learning rate 5 x 10-4 and weight decay 5 x 10-3, generate = 1024 ID samples in each batch, M views of each sample, and train each method for 200 epochs.
We compare One-vs-Rest lower bounds of these different objectives to the true value (Equation 26). In Figure 2, we see that increasing multiplicity M decreases the MI Gap for Geometric, Arith- metic and Sufficient, with Geometric having the lowest gap, whereas for Multi-Crop, the MI Gap increases, validating Theorem 2.3 and Proposition 2.1. The Multi-Crop loss expectation is also Miinvariant, whereas its variance reduces, as was proven in Section 2.2.
One-vs-Rest MI
1.0-
Relative MI
7.5
1.5
MI Gap
0.2
Loss
Geometric
0.9
7.0
Sullicient
0.1
Arithmetic
Multi-Crp
6.5
1.3
0.8
10
View Multiplicity M
View Multiplicity M
View Multiplicity M
View Multiplicity M
Figure 2: Comparing MI bounds with MI in the Gaussian setting. Each method is trained for 200 with multiplicities M E (2, 4, 8, 10}. Left to right: 1) True One-vs-Ress MI (Equation 26); 2) MI Gaps decrease as M grows for all methods except Multi-Crop due to the log(K) factor; 3) Relative MI True MI Lower Bound MI: and 4) losses for each objective. Bands indicate the mean and standard deviation across 16 runs. Points indicate final model performance of corresponding hyperparameters.
3.2 REAL-WORLLD IMAGE REPRESENTATION LEARNING
We investigate image representation learning on ImageNetlk (Russakovsky et al., 2014) following SimCLR (Chen et al., 2020a). Full experimental details are in Appendix F.1, and pseudo-code for loss calculations are in Appendix F.3.2. We consider two settings as in Fort et al. (2021):
Growing Batch, where we draw views V = x M with multiplicity M whilst preserving the number of unique samples K in batch.
2. Fixed Batch, where we hold the total number of views V K A fixed by reducing the number of unique samples as we increase the multiplicity M.
We investigate these scenarios at multiplicity M 8 for different training epochs in Figure 3a. We observe that, given a number of training epochs or model updates, one should maximize view multiplicity in both Fixed and Growing Batch settings, validating the claims of Sections 2.3 and 2.4.
To understand any practical benefits, we introduce Relative Compute" (Equation 29), which is the total amount of compute used for the run compared SimCLR run 128 epochs.
MM Relative Compute /M, Epochs) = 2 x 128
Epochs
(29)
In the Growing Batch case, there are only minor gains with respect to the batch size 4096 SimCLR baseline when measuring relative compute. the Fixed Batch case, we observe a new Pareto front in Relative Compute. Better performance can be obtained by reducing the number of unique samples while increasing view multiplicity when using Geometric PVC or Sufficient Statisties. Notably, batch size 256 Geometric PVC trained for 128 epochs outperforms a batch size 4096 SimCLR trained for 1024 epochs. We also note that better performance not achievable with Multi-Crop. which is compute-equivalent SimCLR.
To further understand the role of multiplicity, we hold Epochs 128 and vary multiplicity M in Figure 3b. Increasing multiplicity is never harmful, with Geometric PVC performing the strongest overall. We note that Multi-Crop outperforms Sufficient Statistics in the Growing Batch setting.
*Note that there is dependence the number of unique samples per batch K, increasing K both increases the compute required per update step and decreases the number of steps per epach.
8

---

Published as a conference paper at ICLR 2024
Growing Batch
Growing Batch
Test Top (%)
Test Top (%)
70-
70
6.5
64
128
255
512
1024
0.0
0.1
0.2
0.3
1/2
32
Fixed Batch
Fixed Batch
72-
Test Top (5)
Test Top (%)
70
70-
SimCLR Geomnttric
SimCLR
68
Gloomnnic
68
Sufficioni Multi-Crop
Sufficient
Malti-Crop
66-
64
128
256 Epochs
512
1024
0.0
0.5
1.0
1/2
4
16
32
10
15
Updates
1
Relative Compute
View Multiplicity M
(a) Training multiplicity M 8 varying training epochs.
(b) Varying multiplicity.
Figure 3: Contrastive ResNer 50 trained on ImageNetlk for different epochs or with different view mulriplicities. Blue, red, orange and black dashed lines represent Geometrie, Multi-Crop Sufficient Statistics, and SimCLR respectively. Bands indicate the mean and standard deviation across three runs. Points indicate final model per- formance of corresponding typrraarameters. We use K 1096 for Growing Batch and K (2/M) 4096 for Fixed Batch. (a) Each method trained with a multiplicity
8 except the M SimCLR baseline. We compare models in terms of performance against training epochs (left), total updates (middle) which is af- fected by batch size K, and relative compute (right) which is defined in Equation 29, See Appendix F.3. for FLOPs comparison. Each method is trained for 128 epochs for each multiplicity M 12, 6, 8, 12, 16).
4 RELATED WORK
We present work related to view multiplicity here and additional related work in Appendix G.
View multiplicity Hoffer et al. (2019) showed that multiplicity improves both generalization and convergence of neural networks, helping the performance scaling. Balestriero et al. (2022b) showed two-view contrastive learning helps the estimation of the MI lower- bound to have smaller variance and better convergence. Similarly, Tian et al. (2020a) studied multi- ple positive views in contrastive learning, however, their work enhances the loss variance by averag- ing over multiple fwo-view losses. While similar to the extension we present in Section 2.3.3, Tian al. (2020a) do not consider the multiplicity effect in negatives, and the log ()) factor, resulting to just more accurate lower-bound. Song & Ermon (2020), however, increases the log( K') factor by including positives to solve a multi-label classification problem. In the supervised setting, Fort et al. (2021) studied the effect of augmentation multiplicity in both growing and fixed batch size, showing that the signal to noise ratio increases in both cases, resulting to a better performance overall.
that more augmentations
CONCLUSION
In self-super learning. the multi in multi-view representation learning typically refers two views per unique sample. Given the influence of positives, and the number of negatives in contrastive learning, we investigated the role of the number of positives.
We showed that Multi-Crop, popular self-supervised approach, which optimizes combination of pair-wise tasks, reduces the variance of estimators, but cannot change expectations or, equivalently, bounds. To go beyond Multi-Crop, used information theory and sufficient statistics to derive new families of representation learning methods which we call poly-view contrastive.
We studied the properties of these poly-view contrastive methods algorithms, and find that it is beneficial to decrease the number of unique samples whilst increasing the number of views those samples. In particular, pily-view contrastive models trained for 128 epochs with batch size 256 outperform SimCLR trained for 1024 epochs at batch size 4096 on ImageNetlk, challenging the belief that contrastive models require large batch sizes and many training epochs.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENTS
We thank Arno Blaas, Adam Goliński, Xavier Suau, Tatiana Likhomanenko, Skyler Seto, Barry Theobald, Floris Weers, and Luca Zappella for their helpful feedback and critical discussions throughout the process of writing this paper; Okan Akalin, Hassan Babaie, Brian Gamp, Denise Hui, Mubarak Seyed Ibrahim, Li Li, Cindy Liu, Rajat Phull, Evan Samanas, Guillaume Seguin, and the wider Apple infrastructure team for assistance with developing scalable, fault tolerant code. Names are in alphabetical order by last name within group.
REFERENCES
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by max- imizing mutual information across views. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'AlcééBBuc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference
Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 15509-15519, 2019. URL https:: /ptpceed///oooeeinggs wweerip..c/pppperreeeee pttpccpppppp../////// hash http://f5522aacc71444eeeeeeeeee.
htmm..
Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning speech representations. Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: http://///ccccecc.s.cc.. neurips. wwteerrps.ccpppppeeeeeeeee wttcc///ppppe wttpc/pppe///////aa.e... 92aleleblcc6I9fba222 (111888000667777-------
Hugo
htmll.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In NeurIPS, 2022. URL http://papers wwwpaaprrs.ips..ooo cc/paperpppppppeceeeccccceeeeeee http:/ppperrrllleeppprr//// / www.ee-btttttttotee-eeee-ee
html.
Randall Balestriero, Léon Bottou, and Yann LeCun. larization and data augmentation are class dependent. wwwp://pppeeeeeeeeeeeeeeeeeteeee
The effects of regu-
In NeurIPS, 2022a.
URL
http://paperr .nps httppps.ccppppe...ee htt.aaaaceddda000222222------aaa
html.
Randall Balestriero, Ishan Misra, and Yann LeCun. A data augmentation is worth A thousand sam- ples: Exact quantification from analytical augmented sample moments. CoRR, abs/2202.08325, 20226. URL htttts://///iii orgg/ abs/ 2202. 08325.
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Grégoire Mialon, Yuandong Tian, Avi Schwarzschild, An- drew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pir- siavash, Yann LeCun, and Micah Goldblum. A cookbook of self-supervised learning. CoRR, abs/2304. 12210, 2023. doi: 10.44550Maxxi..2344.2222 48550/arXXa 2304. 12210.
URL https:: httpp::///oi..o///1//.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variaaceiivvriaccccovvriinccn regulariztionn for self-supervised learning. CoRR, abs/2105.04996, 2021. URL https://arxiv
abs/2108 04906.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 3588):1798-1822 2013. doi: 10. 1109/TPAMI.20135000 URL https: http//ooooooooo/./ooooooooooooot 10.
2013.50 50.
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Big vision. httpps://gt wttb. com/ 2022.
google www.eeereeeecceeeeeeeeeeeeeeeeee
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne
10

---

Published as a conference paper at ICLR 2024
Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part VI, volume 8694 of Lecture Notes Com- 29. URL
puter Science, pp. 446-461. Springer, 2014. doi:
https: /tops::/doi... .allo..orgo orgg / 0000/000---11-1000000000
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. CoRR, fbs/1807.055500 2018. URL http://arxiv
org/aba abs 1107. 05520.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: http:://poocodiioeo neur 70feb62b6 f1
newriperiee pesrrpp... scc/paperr2220/haait Abstract html.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Ar- mand Joulin. Emerging properties in sueerssee vision transformers. CoRR, abs/2104.14294, 2021. URL https: aaaiiviiiiv.....ii...aaaaaiiiiia wwwrriii.iorg/ooooooo abs
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020a. URL http: / http:///ccceee....cc... mol..prss// v1 19/ chen20j html.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Computer Vision and Pattern Recognition, CVPR 2021, vir- tual, June 19-25, 2021, pp. 15750-15758. Computer Vision Foundation / IEEE, 2021. doi: 10.1009/CVV2444422220000555 thecvf.
IEEE Conference
URL https://openaccest
hwtp://wnteeee////////////////aa Representatioluteeming..cc.c
1ttmpmmmeeemeeeeeee____eeee_ html.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momen- tum contrastive learning. CoRR, abs/2003.04297,, 2020b. URL https: http:s:///axvvvvo org. abs/ 2003.00 04297.
Xinlci Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vi- sion transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 1ww.00.1110//1C.V22..2.. 1199/CCC44922.2211..00 00950.
IEEE, 2021. doi: 10.
URL https: /ttp///oooooo//ooo/ooo/ dow..ooooooodoo//oo wttpg//...11/////1.
Yanzhi Chen, Dinghuai Zhang, Michael Gutmann, Aaron Courville, and Zhanxing Zhu. Neural approximate sufficient statistics for implicit models. arXiv preprint arXiv:2010. 10079, 2020c.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In 2014 IEEE Conference Computer Vision and Pattern Recog- nition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pp. 3606-3613, IEEE Computer Society, 2014. doi: 10. 2..109/CVPR.22114.460 URL https: doi org/ 0 1109/CVPR. 2014. 44.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory 2nd Edition (Wiley Series in Telecommunications and Signal Processing). Wiley- Inlessenners,,,nee, July 2006. ISBN 0471241954.
Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, wwtpp/pppp//////eeeeeeeep virtual, 2020. URL httpss://proceeiigss..eom html.
In Hugo

---

Published as a conference paper at ICLR 2024
Frederick E. Daum. The fisher-darmois-ers hate--daaoi---opmaaa---- pitmann theorem for random processes. In 1986 25th IEEE Conference on Decision and Control, pp. 1043-1044, 1986. doi: 10.1109/CDC.1986. 267536.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. id-YichF dNTTy.
net/ forum?
Li Fei-Fei, Robert Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Comput. Vis. Image Underst., 106(1):59-70, 2007. doi: 10.10160.CVII..000000000 20..0000.000000001...... URL https: //ts:////ooo/// org/
10. 10.1016/3.... 200. 09. 012.
Stanislav Fort, Andrew Brock, Razvan Pascanu, Soham De, and Samuel L. Smith. Drawing mul- tiple augmentation samples per image during training efficiently decreases test error. CoRR, abs/2105.13343, 2021. URL https:/ arxiv. org/ abs/aaaslloo 05.13333..
Borja Rodríguez Gálvez, Arno Blaas, Pau Rodríguez, Adam Golinski, Xavier Suau, Jason Ramapu- ram, Dan Busbridge, and Luca Zappella. The role of entropy and reconstruction in multi-view self-supervised learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barhara Engel- hardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 29143-29160. PMLR, 2023. URL https : /ttt://prroeeee /ttpp//rcceegs.s../////////...se mmr. press/v22 rodri- guez sere-gaalezzzaee html.
Jean- Man-BBatien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach sulf--sue learning. In Hugo Larochelle, Mare' Aurelio Ranzato. Raia Had- sell,
Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: :ttppp:://pee http///oooediiccc//c/cc/oc ngs. neurips cc/ paper/2020 http///////aaaaaaaaaeaeeeeeeeeee htt.aacceeee.0022222777712222222 Abst html.
Paul R. Halmos and Leonard J. Savage. Application of the radon-nikodym theorem to the theory of sufficient statistics. Annaly of Mathematical Statistics, 20:225-241, 1949. URL https:
/ //api. www.eeeisnnoooleeee...oeee
wrgg/////ooooo /Ct//CorpussD : 119959959.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In 2015 IEEE International Confer- ence Computer Vision, ICCV 2015, Santiago. Chile, December 7-13, 2015, pp. 1026-1034. IEEE Computer Society, 2015. doi: 10.1 w..1100/C...2015..00 URL https::////// ///o:///oiio////oo orc 1109/ 2015.1222 123.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016..0 URL https: http:://:: /tto:://dooioooo wtt../.........////......... 2016. 90.
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. CORR, abs/1911.05720 2019. URL http:
//arxiv. www.vorrggoooo awtpaass...1// 05722.....
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw. Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. CoRR, 0hss/1955009211 2019. URL http: /wtp///rxvvvvvvv/////// www.wwwww.....oooo.o......oooore ab//////////// 09272.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information
12

---

Published as a conference paper at ICLR 2024
estimation and maximization. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: // aepenzeview.net.eo net/
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: better training with larger batches. CoRR, hts/1901.09335, 2019. URL http:
//arxivvorg// abs 1901.00000001 09335.
Jin Young Kim, Soonwoo Kwon, Hyojun Go, Yunsung Lee, and Seungtaek Choi. Scorecl: Augmentation-adape contrassive learning via mcore-matching function. CoRR, abs/2306.04175, 2023. doi: 10-48550/a2XX.225366.077777 URL https://dos org /10. 48550/ 2306.04 04175.
Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine-grained cars. 2013. URL https://aat https://aa..semnt.cccccoooo semanticscholar: Crg/CorppsID:
16632981.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced search). 2014. URL http://t//// hwww//wccccmm cs
www.:ccc..roottooooooo wwronto..oooooo.o..ooo. setrontto.eddokk//////eeaae html.
Kyungeun Lee, Jaeill Kim, Suhyun Kang, and Wonjong Rhee. Towards a rigorous analysis of mutual information in contrastive learning. CoRR, abs/2308. 15704, 2023. doi: 10.48550/arXiv. 1508. URL https: /ttps:///oiii arg/1c... 48550 wtt.0///r.vv.... wwt............0.....
Ralph Linsker. An application of the principle of maximum information preservation to lin- ear systems. David Touretzky (ed.), Advances in Neural Information Procexsing Sys- tems 1, INIPS Conference, Denver, Colorado, USA, 1988), pp. 186-194. Morgan Kauf- mann, 1988. URL https: http:::://pppprssssp cc/papppppcpppreepepppppp..ppp.. saapperrfflle/eeeeeeeeeeeeeeeese www.fffllleee///aeeaaa http//paar/198/hheeeee/ cc336666..97777111111aaa/7/eeee www.cce7acccccceeeeeeeeee Abstract html.
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representa- tions. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https:: htttp:://opereviiii .eeteeerreeeeeeeeeeeeeeeee wtw.rrewwwwweeeeeeotmme.mc.. 2orum2id==JvJxXbb0
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations. ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. CORR, abs/1 306.511,,,,, 2013. URL httt:://ta/iiiii abs/1306 5151.
XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. CoRR, abs/0809.085: 0853,, 2008 URL http: //arxiv. wor.//r//aa... abs 08000000... ...0000000 0853.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, pp. 3498-3505. IEEE Computer Society, 2012. doi: 00.100//CVPR.222222240000 https: http//doi.ooo .org/i.rrg// 1111///PP.... 2012. 62 48092..
URL
Ben Poole, Sherjil Ozair, Aäron van den Oord, Alex Alemi, and George Tucker. On variational bounds mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Pro- ceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp- 5171-5180. PMLR, 2019. URL http: htppp/ooooeenggss//// ml www.ss.ppe////oe.....oo..oe. / v ppoppeollllll html.
Ce Qi and Fei Su. Contrastive-center loss for deep neural networks. In 2017 IEEE International Conference image Processing, ICIP 2017, Beijing. China, September 17-20, 2017, pp. 2851- 2855. IEEE, 2017. doi: 10.1109/ICIP200178896600 URL htttps:///c. 10101.1110.g/0011199// ICIP. 2017.07200000. 77.8288888..
13

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Ma- rina Meila and Tong 7hang (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, ICMI. 2021, 18-24 July 202 I, Virtual Event, volume 139 of Proceedings of Ma- chine Learning Research, pp. 8748-8763. PMLR, 2021. URL http: pttpp/eecceeens..e/ee///// mlr.. press/vil /1/3///13/// rttt///dadddddaaea html.
Joshua David Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learn- ing with hard negative samples. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL. https: // //ooeneeview wwwp:rewwwwee.eeeeeeeeeeeee wtw.ormmmmnccc.0OoT0TTTTT
Alex Rogozhnikov. Einops: Clear and reliable tensor manipulations with einstein-like notation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. https:/::/ /tt.httss////peneee// /wwp///eeeeeeeeeeeeeeeeeeeeeeeee ew. net forum? i= oapKsVM2bcj.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, and Li Fei-Fei. Imagenet large scale visual recognition challenge. CoRR, abs/1409.0575, 2014. URL http: / http///xxvvvorrr/ abs 2009 0575.
Ravid Shwartz-Ziv, Randall Balestriero, Kenji Kawaguchi, Tim G. J. Rudner, and Yann LeCun. An information-theorrtic perspective on aariance-invarinn--cvaiiia regularization. CoRR, abs/2303.00633, 2023. doi: 10.48550arXi22233.00333 URL https: /ttp///ooo../....///o//// 48550/arxiv.2300 220.0.0033...3.......
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Gar- nett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 2016. URL https: pttp///pceeeenneeeeeeeopoo neuri ips. cc/ pttpppappp/////aaaeeeppppp 6b1 E111803762 http:333aabbbbbbe99999ee faw.aaaaaa22aaaaaaaaaaaaaaaaaaa Abstrract.battttt......... html.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In 2016 IEFF Conference on Computer Vision and Pattern Recog- nition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 4004 IEEE Computer Society, 2016. doi: 10. 10.1109/CVP..200.4455 URL https: 2016.1 434.
1109/CVPR.
Jiaming Song and Stefano Ermon. Multi-label contrastive predictive coding. In Hugo Larochelle. Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: wpwooeedingg..eeuseee wtt.eripss...pppe//////0... ccc/paper/222//hast/ Abstract hhmll .hmll.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision - ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI, volume 12356 of Lecture Notes in Computer Science, pp. 794. Springer, 2020a. doi: 10. 378-3-030-58621-8 45. URL https:: 45.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Hugo Larochelle, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NeurIPS 2020. December 6-12. 2020, vir- tual, 2020b. URL https::/prroeddd/// newrppsss.......n.spppp..e...... http:ss.c...p.ppp.peepeeeee 2c2e5eaae9152000 1rsr5aaaee2277768998855733-----3
html.
14

---

Published as a conference paper at ICLR 2024
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenRe- view.net, 2020. URL https::/p/// oww.oepeevvieee ew net. wtt.eeeeweeeeeeee.eeeeeee
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/ 807.800...003 607.03748, 2018. URL http:///axiy org/ ata/abbs11800 03748.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gameett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL http:: 3. 7eee aa Abstttaattaaatttt...aa.... html.
httttttt////pceeeeeeeee /ttp://///ceeeeeeeeeooocooc ngs. neuri ips wwtppppr////cce//pe/pe 2211 //aahh/
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmen- tations provably isolates content from style. In Marc' Aurelio Ranzato, Alina Beygelz- imer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Informa- tion Processing Systems 2021, NewrlPS 2021, December 6-14, 2021, virtual, pp. 16451- 16467, 2021. URL https:/ https://pproeeddingg newrips........... cc/ pttpp//ppp///aae///e 1922670faG110412288884442.... -bstteccc-bbbattttcceee html.
Haoqing Wang. Xun Guo, Zhi-Hong Deng, and Yan Lu. Rethinking minimal sufficient represen- tation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16041-16050, 2022.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 9929-9939, PMLR, 2020. URL http: http::/ttt//ooooo /ptp:///ccoceenss.. mlr. wwwpppessssvv.. //l//vll1/// wang20k. html.
Xiao Wang and Guo-Jun Qi. Contrastive learning with stronger augmentations. IEEE transactions on paltern analysis and machine intelligence, 45(5):5549-5560, 2022.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pp. 3485-3492. IEEE Computer Society, 2010. doi: 10.110//VPR.20005533970 URL https: //tp:///ooio///oo org/10 110111199//0 /119/CVPR.... www................. 2010.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size 32k for imagenet training. CoRR, 0ts//7008.38888 2017. URL //attp:://axxv..../ org/ abs / 11111100111 03888.
Manzil Zaheer, Satwik Kottur. Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov. and Alexander J. Smola. Deep sets. CoRR, abs/1703.06114, 2017. URL abs/1703. 06114.
Shuangfei Zhai, Tatiana Likhomanenko., Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M. Susskind. Stabilizing transformer training by preventing atten- tion entropy collapse. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 40770-40808 PMLR, 2023a. URL ht /ttpc////.cceeeceooooooooooococc wiggiiinin....... ngs mmlr, press/ v202/zhai
html.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training CoRR, abs/2303. 15343, 2023b. doi: 104485550/rXXi.2333..55 15343. URL
https:///oi..rr//22 wwt.//550//......//m
15

---

Published as a conference paper at ICLR 2024
Appendices
A Broader impact
17
Limitations
17
Proofs of Theorems
18
C.I MI lower-bound with Multi-Crop.
18
C.2 Lower variance of Aulti-Crop MI bound
18
C.3 Generalized INwJ
19
C.4
Validity Property
20
C,5 Arithmetic and Geometric PVC
C.6 Behavior of MI Gap
23
Connection between Sufficient Statistics and MI bounds
25
D Notes Multi-Crop
26
D.1 Distribution factorization choice of Multi-Crop
26
D.2
Aggregation Function for Multi-Crop
27
E Additional theoretical results and discussions
28
E.1 Generalizing one-vs-rest MI ................................
28
E.2
Recovering SimCLR
28
E.3
SigLIP connection MI bound
................................
29
E,4
Sufficient Statistics extension
29
E.5 Optimal multiplicity in the Fixed Batch setting
.tt.............................
30
E,6
Synthetic ID Gaussian
32
Real-world image representation learning
34
F.1
Experimental details
34
F.2
Fine-tuning results on transfer tasks
................................
35
F3
The role of augmentation strength at high multiplicity
36
F.3.1
Comparison of total floating operations
37
F3.22
Implementation of loss functions
38
Expanded related work
40
H Extensions to distillation
41
Contributions
41
16

---

Published as a conference paper at ICLR 2024
A BROADER IMPACT
This work shows different ways that view multiplicity can be incorporated into the design of repre- sentation learning tasks. There are number of benefits:
1. The improved compute Pareto front shown in Section 3.2, provides way for practitioners to achieve the desired level of model performance at reduced computational cost.
Increasing view multiplicity has a higher potential of fully capturing the aspects of a sample, as is reinforced by the limiting behavior of the synthetic setting (Section 3.1). This has the potential to learn more accurate representations for underrepresented samples.
We also note the potential undesirable consequences of our proposed methods:
1. We found that for fixed number of updates, the best results are achieved by maximizing the multiplicity M. If a user is not compute limited, they may choose a high value of M, leading greater energy consumption.
2. In the case wants to maximize views that naturally occur in data in CLIP (Radford et al., 2021), the intentional collection of additional views may be encouraged. This presents number of challenges: 1) the collection of extensive data about a single subject increases the effort needed to collect data responsibly: 2) the collection of more than one type of data can be resource intensive; and 3) not all data collection processes are equal, and larger number of collected views increases the chance that at least one of the views is not good representation of the subject, which may negatively influence model training.
The environmental impact of each of these two points may be significant.
B LIMITATIONS
The work presented attempts present fair analysis of the different methods diseussed. Despite this, we acknowledge that the work has the following limitations, which are mainly related to the real-world analysis on ImageNetIk (Section 3.2):
1. Our ImageNetlk analysis is restricted to variations of SimCLR contrastive learning method. However, there are other variations of contrastive learning, for example van den Oord et al. (2018); Chen et al. (2020b; 2021); Caron al. (2020). There are also other types of Self- Supervised Learning (SSL) methods that train models to solve tasks involving multiple views of data, for example Grill al. (2020); Caron al. (2021). While we expect our results to transfer to these methods, we cannot say this conclusively.
2. Our ImageNetlk analysis also restricted to the performance of the ResNet 50 architecture. It is possible to train SimCLR with Vision Transformer (ViT) backbone (Chen et al., 2021; Zhai et al., 2023a), and anticipate the effect of increasing view multiplicity to be stronger in this case, as ViTs has less strong prior on image structure, and augmentation plays larger role in the training (Dosovitskiy et al., 2021). However, we cannot make any conclusive statements.
3. The largest number of views we consider 16. It would be interesting to see the model behavior in for e.g. two unique samples per batch, and 2048 views per sample, or increasing the number of views beyond 16 for a larger setting. However, these settings are not practical for to investigate, limiting the conerete statements we make for real world applications to views M < 16.
4. Although we presented some sensitivity analysis regarding augmentation policy choice in Ap- pendix F.3, all of the augmentations consider for ImageNetlk variations on the SimCLR augmentation policy.
5. Our method is less applicable in the case of naturally occurring (multi-modal) data, as here M is limited by the data available and cannot be arbitrarily increased.
6. Our empirical analysis is limited to synthetic data and the computer vision dataset ImageNet While we don't anticipate significantly different conclusions for other domains, we are unable make any conclusive empirical statements.
17

---

Published as a conference paper at ICLR 2024
7. There alternatives to One-vs-Rest Mutual Information (MI) when considering M variables. We introduce an alternative partitioning in Appendix E.1, but do not investigate as is less simple to work with.
8. In all of our experiments, hyperparameters are fixed to be those of the reference SimCLR model. principle is possible that different conclusion could drawn hyperparameter search was done per multiplicity configuration, and then the best performing hyperparameters for each point were compared to each other.
C PROOFS OF THEOREMS
C.I MI LOWER-BOUND WITH MULTI-CROP
Proposition 2.1. For K independent samples and multiplicity M denoted I.M. the Multi-Crop of any Liair in Equation has the same MI lower bound as the corresponding P
X2) 2 log(K) E [Cmulsi (X K.KK, [:M] (og(K) IE [Crair (X1x,12)) (30)
where the expectation is over K independent samples.
Proof. Note that for the pair objective Luirr we have the following lower-bound for the pair using INwi (Hjelm et al., 2019; Nguyen et al., 2008) sample estimator:
[(x, x3) 2 log(K) E [Cair (I.K,(o,)))
(31)
the views are uniformly and independently generated, i.e. na Uniform(T), where I' is the set of view-generating processes, then
I(xa; X3) x()
Va * B
(32)
Following Equations 31 and 32, we have
x(x1;X2) M(M 1)
I(xx; X3))
(33)
Aza
M
2 log ())
E
Pair (X (1.K.000))
(34)
M(M a=
8pa
loog () E [CMulti-rrop [I.
(35)
Moreover, rewrite the Multi-Crop objective follows in expectation:
M
E (ruuti-Crop [ (IIK,MM) = M(M
Saiir (airr(Xiiae (X.::.
(36)
a 3g0
= E [Rr [Cpair( (X::K,:)))]]
(37)
where the second equality is due to the fact that all the views are uniformly and independently sampled from the set Now, getting expectation over all the randomness lead us to
E [EMutri.Cmp (I.K,::a)] E [Er [Cpair X X1:K,12)]] E [Ciir (X (X1:K112))--
(38)
This completes the proof.
O
C.2 LOWER VARIANCE OF MULTI-CROR MI BOUND
Proposition 2.2. For independent samples and multiplicity M, M 2 3, denoted X X.K.I.M the Multi-Crop of any Lpair in Equation has a lower sample variance than the corresponding L
Var [LMulti- Crop (X:as) 2(2M -
Var (X1, x2)] < Var [Lpuir (X1, x2)]
(39)
3M -
where the variance is K. independent samples.
18

---

Published as a conference paper at ICLR 2024
Proof. We start with computing the variance of both side of Equation 1. Note that for any two pairs of (Xa, Xg) and (Xy; x,) such that {a, n (r, // = 0, we have
Cov pu (Xa, xg), puii (xy, x,,] = 0,
(40)
where Cov denotes the covariance operator. This is due to the fact that view generation processes are conditionally independent (condition x). Thus, for any realization of the conditional covari- ance would be zero, which leads to the expectation of the conditional covariance, and consequently Equation 40 be zero. We can also rewrite Equation as follows:
M Σ
Muuti-Crop (1IM)
EPai x3)
(41)
M(M 1)
a=1 Bya
2
Σ
( (Xa, x3).
(42)
( - 1)
Having the pairwise loss to be symmetric, we can now compute the variance of both sides as follows:
Var [Coulti-Crop (X1:M)] M 1)2
Var (Xa, Xx)1
(43)
ΣΣΣ Cov [Cpair(Xa Xy); Cpair (P x3)]
One way to count the number of elements in the covariance term note that we can sample a, 3, and from [M] but only one of the ordered sequence of these three is acceptable due to the ordering
condition in Equation 42, which results in M M choices, where M 2 3.
Another main point here is that due to the identically distributed wew-generative processes.
Var LLii x3)] Var [Laur x,]] Va + 8, E [M].
(44)
Thus, using the variance-covariance inequality, we can write that
Cov (Xa, x,), Lrair (X,,, xx)] Var [Laair (Xo. x,)] = Var [Laair (Xy,, x8)]
(45)
Substituting Equations 44 and 45 in Equation 43, we have the following:
4
M(M - 1) Var [X1
Var (CMuti-Crrop (1:M)s M2(M 1)2²
2
(46)
M 1)(M 2)
Var [L (x1, (X1, (X1,
6
Simplifying the right hand side, we get
2(2M 1) 3M -
Var [LMult-Crop (X1.a))s
Var (x1, Var [X1, X2)]
(47)
for any M > 3. M = 2, the claim is trivial both sides are equal. Thus, the proof is complete
and Multi- Crop objective has strictly lower variance compared to the pair objective in the presence of view multiplicity.
C.3 GENERALIZED INWI
Theorem 2.1. For any M 2 2. a E [M). M random variables X1MM- and for any positive function F(M) X M- 1 H +
I(xx; X *ta
Exxiax
(p(M) X k(nr)
Epen Pxtta
FCM (m. xxto
+1 IGeNNWI. (48)
19

---

Published as a conference paper at ICLR 2024
Proof. We start by the definition of MI:
log p( X 1:M
I Xa: xa:
=R Epx
(49)
1:M5
pl (a )pl X l:M
p X 1:M
F(ND
X (oo.
1ibe)
log
(50)
p( 3p( ( 1:M le
mean
(a Iiid
p p(
to X
1.M )e me (a
X
1.31
-Mxx p(M) X (M))
Bpx
log
(51)
p( X L:M
Now, we note that the argument the second term of right hand side in Equation is always positive. For any 2 O, we have that log(z) - Thus, have:
p 2 )pl
ta I.M ) ( (
I Xai Xxixx 1:M
-
(MM)
(F
X
Bpx
log
(52)
1:M
plo 1:M)
2 E
(M))
Xta
-Epx
plaa) )p( (X
1.M
(r(M) (aa
+1. (53)
1:M
p(a 1:M
Now, we can use the change of measure for the second term on the right hand side and the proof is complete:
I
Xai
1:1s
>
Ex
FM)
(Fa, X x:i0 1
E Epx
p( )p(
p p X X#a
t FlM) (c0
(
+1
(54)
1:dd
= -xxx
F(M)) xxax
1:1d
Epp Proza
e FLM To X s0
+1
(55)
-IGenNWI.
(56)
C.4 VALIDITY PROPERTY
Theorem C.1. Both aggregation functions introduced by Equation and Equation 10 satisfy the Validity property, i.e. Equation 8.
Proof. Let us define zB exp( (2) (xa, for given &a and 3
Thus, we can rewrite
Equations 9 and 10 as follows:
Arithmetic:
F(M)
Tei x
llog
Σ 2B
(57)
1:M
M
3
Geometric:
F(M)
Boy
X ta l:M
log(z.).
(58)
M
Px. P = LiM,
Following the definition of the aggregation function, and denoting C, the aggregation functions as following:
we can rewrite
PX
))
Arithmetic:
ga
(M)
ca exp
log
Σ Z/
Co
(59)
M
M -1
B#a
Bza
M-
Geometric:
Ja ca exp
log(z (2,8)
II CB
(60)
M
Bza
20

---

Published as a conference paper at ICLR 2024
Now, to prove the Validity for these aggregation functions, it is enough to show the following:
Arithmetic:
GMI
Σ
2B
Σ
SMu (cazz)
(61)
M
M
Bza
D&a
Geometric:
Gmi
II B
Ca
Σ By0
Gmu (Cazz)
(62)
M
We start by proving Equation 61. Following the definition of MI Gap in Equation 7, we note that the MI Gap is a convex function since Ia loo(ga)) is convex. Now, using the Jensen's 's inequality, have:
GMT (Ez [c, Ez, (SMI (c.=)],
(63)
which is another expression of Equation 61 and completes the proof for Arithmetic mean. For the Geometric mean, by expanding on the definition of MI Gap in Equation 62, and removing the constant from both sides, we get the following inequality:
I
I
E
log
II C 8fa
Σ
ca - log (ca [g)
(64)
M
1
3fa
1
(cazs log
log (cazz))
(65)
M
BBa
So, proving Equation 62 is equivalent to prove the following:
M-1
M-1
E
II Ca
- loog
II B$a
ca
Σ (cazs - log (ca*))) I 0. (66)
M -
We show for any realization of zg, the inequality is true, then the same applies to the expectation
and the proof is complete. Note that log (IIæ Ca23 N =
M Es log (Ca moreover,
1
using arithmetic-geomerre inequality for any non-negative values of 2n and ca. we have:
II Ag
Ca
23
Σ Ca
(67)
M -
which proves Equation 66, and completes the proof.
C.5 ARITHMETIC AND GEOMETRIC PVC
Theorem 2.2. For any K, M 2. K.M [M], any scalar function f C R. and map h : X + C, we have
K
Arithmetic PVC: I IxxxiXtta #a
2c
1
log
Σ liaas
(68)
K
M
1=1
Báa
K
Geometric PVC:
Xxx (xo;Xita 1:M0 (B,
Σ
Σ log li
(69)
K
M
(=1
B4a
where e(B, = -log(B M +1, 1), the expectation is over K independent samples X XI.KIMM. and (xc....
li (EIKEM) = (R X.,,) + Efar EMI (R,,,,
X =
(70)
We have written li, instead of f (X1: where the meaning is clear.
21

---

Published as a conference paper at ICLR 2024
Proof. Let sample K independent sets of 1,1: where i denotes the sample number for E [K]- By independent here, mean vi * j. VB, 7
2,,0 ,0
jjn Now, let us define Xi,a following:
X a
[X 3.8 3 U(xsa))
(71)
Since the samples are ii.d and the views of different samples are also independent, then X has no more information than X about X a
1,1:M Thus,
a(x ( I(X,,a (iM) I(X,, X
(72)
Moreover, since the samples are identically distributed, have:
I(x.:
#2 X
I(xi,a
X
1 K
I(X oos X
(73)
L:M
K
Now, following the proof of Theorem 2.1, need define F(M) (Xi.s xx11
Following the
Arithmetic and Geometric mean Equations and 10, we only need define FF(2)
(Xina Bi,8
for 3 f the basis. Defining li B (XI.KI.M) follows:
L (XI-,1:a)== = a Ejfi EM x(
K = h(xia))
(74)
can now define ((2) for both Arithmetic and Geometric
F(2) i, 21,8) log ((B MM + 1) (X1:,1:M)) ((B,M) + log lia
(75)
Now, substituting (M)) (Xiar X/ k.ii (denoted by F(M) for simplicity) in Theorem 2.1, we have the following:
Arithmetic mean:
K K
#0 I(xa; X
(76)
E Exx PI
1
((M)
Ellyas Px pr Px 1.1M
pinr)
+
(77)
K
K
K
B M +
E Epx
log
Liaa
K
M
B7a
K
B
M M+1
Σ 3ya
Ens Pai
+1.
(78)
Pxtt
K
M-
LIIM
Noting that the expectation in Equation 78 is taking over variables independently, and noting that samples are identically distributed, and different views generated independently, we can replace x1,9 by a fixed e.g. without loss of generality, 1. Now, we can easily see that this term becomes equal one. Thus.
K
I(x:; X (ta) > E
Epx,
log
M +1
Σ lia
(79)
K
M
i=1
Baa
K
c(B +E
log
Σ Laa
(80)
K
M -
Béa
which is the claim of the theorem, and the proof is complete for Arithmetic mean.
22

---

Published as a conference paper at ICLR 2024
Geometric mean:
K
I(xo; XI:M)
I I(Xi,ar i X
(,1:M)
(81)
K
K
K
Flar
2 EXXLKKIM
F(M)
P Pai,a Px
+1
(82)
K
K
Am1
im
K
c c(B,M)
+
log bia
+1
K
M
1=1
K
Euy Pal P Pxti
exp(c B, M)+ + - -
Σ
log (ia
(83)
K
i=1
Byo
Since exp(2) is convex function, we can use the Jensen's 's inequality for Equation 83:
B Par boss.extte
exp(c(B,, M) + M
1
Σ
log (y)
(84)
K
1 K
I ElgaaP Pai Px X
K M 1 Σ (B-MM
Σ :=1
1 M 1)
(85)
340
=1.
Where the last equality is resulted with the same reasoning behind Equation 78. Thus, we have:
I(xai X/
EpxLK,
E
cB,
Σ BFa
log
lia
+ +1-1
(86)
M
c) +
Σ log Li
(87)
M
340
and the proof is complete.
C.6 BEHAVTOR OF MI GAP
To investigate the behavior of MI Gap and to provide the proof of Theorem 2.3, we first provide the following lemma, which is resulted only by the definition of expectation in probability theory:
Lemma C.2. Let C (1,...,k) k) with 7| = m, m S k; be uniformly distributed subset of distinct indices from (1,...,k) k)). Then, the following holds for any sequence of numbers a1.-..y Ok.
aig + 0i.
+
E, $1=(11....u
(88)
m
k
Now, for Theorem 2.3, we have the following:
Theorem 2.3. For fixed a, the MI Gap of Arithmetic and Geometric PVC are monotonically non- increasing with M:
(X.M, (Ma)))))) < Gral (Xa,M, (.1))) A M1 S M2.
(89)
23

---

Published as a conference paper at ICLR 2024
Proof. Let use the new form of aggregation functions' definition with 23 in Equations 59 and 60.
For M1 S M2, and for Arithmetic mean, i.e. 9o M- Ezea Ca we have:
(M) 1
(M1(XMM,2::0 (M2))) = Epx
Σ
CB
M2 - 1
Byaee
log
1
Σ
CoEB
(90)
M2 -
Bsa
M,-1
M2
E, e==(0 (1-.n-..70001111
Ca 2nys
M 1
j=l
M
1
Eyx
log
Ca
-1 (91)
M1
E Epx
Σ Cazy
M1 1
Bya
M: -1
E I= m (M -13
log
1
Σ
Ca 2ng
-1 (92)
Mi -
j=1
Epx
Σ
ca
M1 1
Bya
-Exx1M
log
1
Σ BBa
cazB
-1
(93)
Mi
GMI (X1: ((
(Mi)),
(94)
where the first equality is due to the Lemma C.2, and the inequality is resulted from Jensen's 's in- equality. Therefore, for Arithmetic mean, the MI Gap is decreasing with respect to M.
For the Geometric mean, and following the definition of MI Gap, we have:
(Nuu,igg (Ma)))
Epxcinss
II C Bitta
Epxx kig
Σ
log (Ca
-
(95)
M2
Báa
Epx
II Ca
Bytl
log (caza)
-1,
(96)
Mr
Dza
24

---

Published as a conference paper at ICLR 2024
where the equality is followed by Lemma C.2, similarly to the corresponding proof for the Arith- metic mean. Now, mainly focusing on the first term of the MI Gap, we have:
Ma 11
M2
Gui(XinMaiga MaMa)) = E
exp log
II Co
B
Epx
Σ Bya
log (ca
(97)
M1
M2
Epxi
exp
log CB
M2
1
8za
-Exx, Ms
1
Σ
log (cozg)
-1
(98)
M1
Báa
Eyxx
exp
BI=(m 7M1
log Ca Znys
M1
Epx
1
Σ
log (cazg) -1
(99)
M1
Mi
< Epxcinda
E
exp
log Ca 2ny
M
j=1
1
- -Epx
log (CazB)
-1
(100)
M1
Bta
NI
II
Epxcass
Ca
Epx
1
Σ Báa
log (Cazg)
-1
(101)
M1 -
= GM (Xi ga (b11))
(102)
Here, Equation 100 is resulted using Lemma C.2 by replacing 0g log zg. and the inequality is due to the Jensen's 's inequality. Thus, the proof complete.
C.7 CONNECTION BETWEEN SUFFICIENT STATISTICS AND MI BOUNDS
Theorem 2.4. For any K, M 2 B KM, a [M], and the choice of Q in Equation 24, we have (see Appendix C.7 for the proof)
1 K
Xxx
X (I:M
2 cB
+
log &
(103)
where c) = log(B M + 1), the expectation is over K independent samples X1:K1:M.
Proof. The proof consists of two parts:
1. Show that there is (MM) corresponding the choice of Q in Equation 24.
25

---

Published as a conference paper at ICLR 2024
2. Achieving the lower-bound using the given F(M) for I Xa. X(i)
We prove both points together by studying the lower-bound for one-vs-rest MI given the aforemen- tioned ((AD) The proof is very similar to the proof of Theorem 2.2. We use the definition of X; as Equation 71. We also note that since the samples are i.i.d, and the view generation is indepen- dent, can also use Equations 72 and 73. Consequently, we only need to define the sumple-based F(M)
X X ).
Note that here, in contrast with Arithmetic and Geometric, we do not have
((2) our basis for F(M), We define the (MM) X X t 3,1:M5)
as follows:
K
F(M) (XIK,EM;a) = c(B +
1
log E,,0,
(104)
K
We also note that the introduced F(M) and its corresponding aggregation function, follows all the main properties, i.e interchangeable arguments, poly-view order invariance, and expandability. Thus,
which is the sample-based generalization of F(M) Xa. X t
T(xa)
2820 T(xo) M
1:M)
the first point is correct. Now, we continue with the lower-bound. Substituting Equation 104 in Theorem 2.1, get the following:
I(xai (xxaiXT. x
I(X;s
X (LMM)
(105)
K
1 K
2 E Epx
F(M)
Ell Pai,a Px 1:MM
1
+
1
(106)
K
K
i=1
x=
E Epx
cB
+
1
M Σ log lia
K
340
ETIyiPP Per P Xxii
(B- M 1)era.
+1
(107)
K
c(B.M) (MM))+Exxx... EXXERAM
1
1 Σ log Li
(108)
K
where the last inequality is resulted with the same reasoning as having identically distributed pairs of (X,, X ta ( due to the sample generation process, and the fact that expectation is taken over random variables independently. Note that here, maximizing the lower-bound corresponds to maxi- mizing 3,,, which provides the same optimization problem as Equation 21 with Q in Equation 24. Thus, the proof is complete and sufficient statistics is also an MI lower-bound.
D NOTES ON MULTI-CROP
D.1 DISTRIBUTION FACTORIZATION CHOICE OF MULTI-CROP
As explained in more detail in the main text, Tian et al. (2020b) and Caron et al. (2020) studied the idea of view multiplicity. While their technical approach is different, they both took a similar approach of multiplicity; getting average of pairwise (two-view) objectives we show that this choice of combining objectives inherently applies a specific choice of factorization to the estimation of true conditional distribution of p(x(c) in Figure lb using multiple views, i.c. applies an inductive bias in the choice of distribution factorization.
in Equation 1. Here,
Following the InfoMax objective, we try to estimate I(x; c) using the pairwise proxy I(Xa: Thus, the idea of Multi-Crop can be written as following inequalities:
M
M M
I(x;c) 2
I(x; xx) 2 M(M - 1)
I(XaiX8).
(109)
M
8xa
26

---

Published as a conference paper at ICLR 2024
Assuming that these two lower-bound terms estimations for the left hand side applies distribu- tional assumption. To see this, we start with expanding on the MI definition in each term:
p( p(x(c)
I(x: c)
log
(110)
p(x)
M
M
p(x(xx))
[ xx)
log
(111)
M a=1
M
p(x)
a=1
M
M
M M
(xx [x,)
X xg)
E
log
(112)
M - 1)
MM
1)
pl p(xx:)
a=
Now, assuming that the view-generative processes do not change the marginal distributions, i.c. for any a, pxx) = p(Xx), and considering Equation 109, we have:
pl c() p(x)
M
p(x xa)
M
M pxx (xg) p(x)
XI(ad 17
E
log
log
II La=1
log
(113)
x(x)
a=1 Sygho
Therefore, the distributional assumption or the choice of factorization is estimating p(x(c) by the following distributions:
at II p(x(xe))
p(x|c) =
(114)
\0=1
p(x|c)= _
II
II p(xa ((xx)(x))) 8
(115)
which translates to estimating the distribution using its geometric mean. Note that the symbol = reads as "estimates", and is not equality.
D.2 AGGREGATION FUNCTION FOR MULTI-CROP
Following the result of Proposition 2.1, Multi-Crop an average of pairwise objectives, which means that if know the aggregation function and F(2 for the pairwise objective, then we can write the aggregation function for multi-crop as following:
I(xai
X 1:M
I(xa xx8,,
(116)
M
p( (p(xx))())))))))())(())))) (f x())
I(xa:X3)) T.. (a m3)
(117)
p( x3z)
where the second line is from Theorem 2.1 by setting M 2. Now, by getting average over 3 from both sides, we can see:
Egea I(Xxix8) x3) M
p( (( (( (a)))
> EB (F
xx),)))))) - E
+1. (118)
M
pl 2gg)
Bya
Following the proof of Theorem 2.1 and the definition of aggregation function in Equation 6, we achieve the following aggregation function for Multi-Crop:
g0
(M)
X
fa 1:M
g0 (2)
(X (x)
(119)
M
3
27

---

Published as a conference paper at ICLR 2024
E ADDITIONAL THEORETICAL RESULTS AND DISCUSSIONS
E GENIRALIZING VS--VT-REST MI TO SETS
A generalization of the one-vs-rest MI is to consider the MI between two sets. Let assume that the set of [M]] is partitioned into two sets å and B, i.. XA Xg = XI:Ms and A n B = 0, where @ denotes the empty set. Defining the density of X and X5 the joint distribution of their corresponding random variables, we can define the generalized version of one-vs-rest MI:
Definition E.1 (Two-Set MI). For any two purtition set of A and B over [M], define the two-sel MI as following:
I(X,, XB) - Dxr. (Px1:ss Px.Pxx)
(120)
We can now also generalize the INwJ to the two-set case. The main change here is the definition of F(M) as it needs to be defined over two sets as inputs. We have the following:
Theorem E.1. For any M 2 2, and partition sets A and IB over [M], such that A o @ and IB of 0,
and for any positive function F(M)), : x1Al X8 + IR,, we have:
aX; XR) 2 Exx
[P(M) (X.: Xn)
Exx,px
(xx.))))
+ (121)
Proof. The proof follows the exact proof of Theorem 2.1 by replacing x., and X f by X and X3, respectively.
Thus, as long as one can define such a function F(M), the other results this paper follows.
E.2 RECOVERING SIMCLR
Geometric and Arithmetic PVC Here, we show that in case of M = 2 and for specific choices of function F(2), we can recover the existing loss objective for SimCLR, i.e. InfoNCE. Setting M = 2, we make the following observations:
In the case of M = 2. the arithmetic and geometric aggregation functions result in the same lower-bound.
Recovering SimCLR Chen et al. (2020a): Substituting M in Equations 12 and 13, we recover the following contrastive loss, which is equivalent to InfoNCE, i.c. SimCLR objective:
el (xx.1011.)))
LVVC
-E
log
LIaOONCE (122)
K
((x.11111))) + Ejjs Enm el (x(.).))))))
Letting f(x,y)
X-Y x
lead us to the exact SimCLR loss.
Sufficient Statistics We can also show that in the sufficient statistics loss (Equation 22), the case of M = 2 and the choice of Q = Ta (Equation 24) recovers the SimCLR loss. To prove this, note the following observations:
When M = 2, X I:M X3-ai i.e. if = 1, X 1:01 = x2 and if a = 2, X 1:M = X1.
By Equation 24, Q(x M-
Epta T(x;) T(x3-a) when M 2. Therefore, in
Equation 22, we have the following:
6 TTa
SsantSaas = -E
log
K
Tiz-a + E/ >2
T. Tya
1=1
a=1
K
Ti,
1
=-B
log
(123)
K
2
TJ ThiTi
2)= E
e T.
+ +lo
c
ThaTa
Σ/
2j=1
2
e
TlaTi
28

---

Published as a conference paper at ICLR 2024
which is the symmetric InfoNCE. Choosing T(x)
recovers the SimCLR objective.
E.3 SIGLIP CONNECTION TO MI BOUND
We show that the objective introduced in Zhai et al. (2023b) is MI bound. As of our best under- standing, this not present in the existing literature.
SigLIP is MI bound: As shown in the proof of Theorem 2.2, to achieve the lower-bounds define F(M) to have boftmax-based form (see Appendix C.5 for more details). However, could choose other forms of functions. we replace Softmax with a Sigmoid-bar based form, we can recover the SigLIP loss, i.e:
F(2) (0,11 log
+ eft 2.) 2t.j = 1 (i =) else 1.
(124)
Following the same procedure as the proof of Theorem 2.2, and defining the F(2) over positives and negatives as (x1,112))) = * D. Σラ log 19, This shows that SigLIP is also MI bound. Zhai et al. (2023h) has discussion the importance of having the bias term (b) in the practical setting to alleviate the imbalance effect of negatives in the initial optimization steps. However, it would be of future interest whether the generalization of SigLIP by either arithmetic or geometric aggregation function to poly-view setting would help to remove the bias term.
E.4 SUFFICIENT STATISTICS EXTENSION
So far, we have assumed that there is generative factor с affecting the samples. However, in a more general case, we have multiple factors affecting the sample generation. Let us consider the causal graph presented in Figure 4. Here, we assume that the main factors important for the down-stream task denoted by c, called content, while the non-related factors are shown by Sa. called styles. The styles can be different among views while the task-related factor is common among them all.
Figure 4: Content-Style causal graph A more general poly-view sample generation with task-related generative factor c, called content. For each a E [M]. s,, shows the view-dependent and task non-related fac- tors, called styles. The views are shown as before by X.- In the most general case, con- tent and styles are not independent, while in some settings they might be independent. In the independent scenario, the arrows from can be ignored.
S1
S2
SM
Styles
X1
X2
XM
Views
The goal of this section is to generalize the approach of sufficient statistics introduced in Section 2.4 the case of content-style causal graph. We start with assuming that content and style are indepen- dent and then move to the general case of dependent factors.
Independent content and style In this scenario, the arrows Figure from e to 8a will be ignored as there is no dependency between these two factors. We show that for any a E [M], the sufficient statistics of xa with respect to {c, s.) has tight relations with sufficient statistics of x, to and S, separately.
Theorem E.2. In the causal generative graph of Figure if Sa, then we have:
T(c.s. (Te(x), Ts. (x))
(125)
Proof. Having independent factors means p(xa c, s.) p[xx/c)p(xx Isa). Now, using the Soyman-Fisshe factorization (Halmos Savage, 1949) for sufficient statistics, alongside assum- ing that have exponential distribution families, similar Section 2.4, have the following
29

---

Published as a conference paper at ICLR 2024
factorization:
p(Xa/e, c, ((x)C((,, s.) exp (c..sa)
(x) (c,
(126)
p(xalc) = (x)ex
((x))c((e) (c) exp (x) Qc(c))
(127)
p(xa (2( exp (Ten (x) (sa)).
(128)
Substituting these factorizations in the definition of independent generative factors results in:
p(xa S.) (. p(xa c)p((a
(129)
(ri(x)e (c) exp (Te(x) Qc(c))) (r2(x) exp (Te, (x) Qsa (sa)))
(130)
=Rx))((c, sa) exp ((Tc(x), Ts, (x)) (Qc(c), Qs (sa)))
(131)
which completes the proof by showing:
T = (To(x), Te (x)) (cs..) (c, Sa) (Qe(c), Qx, (8a))
(132)
(133)
Note that Theorem E.2 also shows that if the space of generative factor e in Figure 1b a disentan- gled space of two more spaces, i.e. C = Ci @ C2, then the sufficient statistics of with respect to c equal to concatenation sufficient statistics of x with respect to с₁ and Cz2 ie, sufficient statistics keeps the disentanglement.
Dependent content and style When the factors are dependent, the sufficient statistics becomes an entangled measure of e and sa. However, if we assume x., f(c, sa) for any E [M] and specific function f C x S + X, we have the following theorem:
Theorem E.3. In the causal generative graph of Figure 4, assume for any E [M], x. f(c, for an unknown invertible function f. such that
f
Sa = f
(134)
where ne and ng show the elements of (-((xx)) that is corresponded e and s., respectively. Then, Tx(xa) Te (xx)
Ya, B [M].
(135)
Proof. Assume that S. and S3 are sampled i.i.d from Ps/c- Then, have:
p(x = z. c) = p(f(c, s) = xa(c)
(136)
de p(s = 8.. / (I (mga),
(137)
8e p(s 83|c)
(138)
8g p(s 83 f (xxg) (c)
(139)
=p(f(c, s) = xz/c)
(140)
p(x x x3/c)
(141)
Thus, p(x,/c) /c) p(x=/c), which using the Neyman-Fisher factorization and exponential family distribution, results in Te (x.) = Te(x3). This completes the proof.
Note that the result of Theorem E.3 recovers the result of von Kügelgen et al. (2021) using the idea of sufficient statistics.
E.5 OPTIMAL MULTIPLICITY THE FIXED BATCH SETTING
In the previous results of Arithmetic and Geometric PVC (Theorem 2.2), we assumed that M can be any number, and accordingly the total number views B increases. However, it is interesting to investigate the Fixed Batch scenario outlined in Section 3.2 which corresponds to holding B fixed by reducing when M increased. What the optimal multiplicity A for the bottom results of Figure 3? We first note that due the complexity of
x M for a fixed K increases if
30

---

Published as a conference paper at ICLR 2024
MI dower-bounds in Equations 12 and 13, it is not trivial answer this question it will depend on the behavior of function of and the map h.
Here, attempt to provide simplified version of Geometric PVC by adding some assumptions on the behavior of f and h. Although this result is for a simplified setting, we believe provides an interesting insight that for fixed batch size B, depending the functions f and h, there is an optimal number of multiplicity M which maximizes the lower-bound. While even in this simplified version, is computationally challenging to compute M* exactly, it possible prove its existence.
In the case that B is fixed, we can rewrite the Geometric PVC in Equation 13 as following by replacing K B.
I (xx X1:M) 2 c(, +
log li
(142)
K
M
1
840
8
M
MT
c c(B, M) +
Σ 8y0
log
(143)
B
M
Geometric
(144)
To prove that there is an optimal M, we need to show that there is M such that &T point M M Since a,,8 depends on functions f and h (see Equation 14), and these functions are in practice trained, we assume that for a long enough training of their corresponding neural net- works, ai, converges to optimum value denoted by (i..g. Moreover, we assume that negative samples are uniformly distributed on the hypersphere (following the Wang & Isola (2020) benefits of uniformity criteria). Also, we assume that the convergence of &a,0, 3 to its desired value of 1s, happens when M - 00 linear way. Note that the first part of this assumption is not limiting since we know as M grows, the lower bound becomes tighter. Due to the fact that one point is M - oc, there will many mappings that follow the desired behavior of converging to one as M grows. Here, we introduce two choices for the mapping fi,,, with the correct limiting behavior in M to investigate the importance of mapping choice on the optimum value of M:
3M = 0 at
- M-2
1. 0 p" +
M -p
(145)
2. f (M)) = 11 - p'
M 1 = (M),
(146)
where in both equations, p= = a (M- = 2) and lim ( = 1. Note that the uni- formity assumption helps to have the same value of p* for any a and f3. Now, we can rewrite the Ieeemetric as following:
M B
Geooetriic = M) +
Σ Báa
log (M)
(147)
M
i=1
c(B.M) log (*(M).
(148)
Now, we can compute
8T 8M
0. We have,
OL Geometri
dc(B,M)
OE (M)
(149)
2M
(M) Òe
1
de
(150)
B M +1
(M)
5M
0.
(151)
'The desired value of f..a, is it is likelihood.
31

---

Published as a conference paper at ICLR 2024
Therefore, finding the solution of at"(M) (M) would provide us with M For each of
8M B-M+1
choices of e* (M) in Equations 145 and 146, have:
M (1 p²) M 2(1 p²)
1.
B - M + 0
(152)
2. 1- p (M* - (1 p') =0
(153)
M 11-11
B -- M++ +1
Solving these equations results in the following optimum value for each:
1. M = V 1)(1 p*)
(154)
2. = v B(1-p")
(155)
It is interesting to see that p" plays a role in the optimum value of M, which shows the importance of the design of scalar function f and the map h. The differences between the values of M in the two scenarios also emphasize the importance of the behavior of the contrastive loss as M increases.
E.6 SYNTHETIC 1D GAUSSIAN
Following the synthetic setting in Section 3.1, here we show the following results:
Provide the proof for the closed form of one-vs-rest MI.
Evaluate the convergence of the conditional distribution for large M to the true distribution.
Closed form of one-vs-rest MI We start with finding the closed form for joint distributions, p(X1,a) and p(x,) (X f:M) Since all the samples and their views are Gaussian, the joint dis- tribution will also be Gaussian. Thus, it is enough to find the covariance matrix of each density function; note that the mean set to zero for simplicity.
E [xx]= = / p(c = C)E xalc cl dc / cp(c c) = EEcc] 0,
(156)
Var [xa] -E [x]]]] [x+] = / p(c= c)B [x c] de = o² cpplc== = de 0²+0%
(157)
Cov (Xa, x3) [xxxs] = / p(c = c)E x = dc de
(158)
= - plc p(c= c c]B X
-
E cj de = / c²p(c= og.
(159)
Thus, we present the covariance matrices of (( and p(xx p(xx)p(XXtt (x #0 tively, we have the following:
1:M1 by Es and En respec-
T o
Σm
2
EM
[ o
0
(160)
0
ΣM-
Consequently, we can write the density functions as follows:
p(X.:N)) (2T)* v det (EM) exp
T End
x
(161)
1 exp
p(x,)p(X ta (iM) (2Tr) - det (IM)
En x
(162)
where I following lemma:
..). Now, we can compute the closed form for one-vs-rest MI using the
Lemma E.4. Assume x and y are two multivariate Gaussian random variables size n with laws
N₂ and Nop covariance matrices Exp and Eyy, and mean vectors pas Pay respectively. Then,
DKL (Nr || N1) = (Ir(E,, (x)) + (My - -22TT Eny (My - -Pz) - n log
det (Ey) det (E,)
(163)
32

---

Published as a conference paper at ICLR 2024
Therefore, the one-vs-rest MI will be follows:
I(xa: (.M) DKL (p (p(XI:M) |l p(xx))(XXft
(164)
det ESM
= 2 tr (S M + +og
(165)
det (2iM)
Define A= = B= = of1 and C A+ + B. One can easily see that and B commute; therefore, they are simultaneously diagonizable. Thus, there exists matrix P such that the following holds:
PD
== PDBP-1
P + DB)
(166)
where DA and DB show the diagonalized matrices. We also know that det(C") = II AD + DB), where X; denotes the i-th eigen value of matrix DA + DB. Due to the structure of the matrices A and B, we can show D I, and DB is as follows:
M
DB = 0
(167)
0
As result we have the following:
det (EM) = (o²) M-1 1
(o²+
+ Moo)
(168)
det (EM) = (o² + +z)) det (EM = (o² + di))
(02) M-2 (02 + +M - 1)o
(169)
Also, EM has block matrix multiplication form:
DMI E
0 + o
0
0
v
(170)
ΣM
ΣM-
(02
+ 0 -1
o02
v
(171)
Σ-1
ΣM-1
(o² + 0
(172)
DM110
I
Where v (og...., ,g) a (M 1) matrix. Therefore, (STT M. Thus, for the one-vs-rest MI, we have:
of I(xai (M)) 2 log (1++2) ch (1- + Mop
(173)
Convergence of conditional distribution Using the covariance matrices in Equation 160, and expanding the distributions p(X,,1:0)) and p(X(rm) we can write the conditional distribution as follows, which helps us to evaluate Equation 19:
Px; X 11:
a 02 +
(aiaa - )²)²
Mx? 2( + MoJ)
La=1
Px Kii
V
Mo 2
exp
20²
(174)
Σ 3at (xi,8
(M - x
20²
2( + (M 1)0
where e; and /. #a
Ma? are the average of X and X ( respectively. As M increases, 2 Maaj
and (M- 1)
c?
200*(MM1166) converge to which results in these terms cancelling each other. Therefore,
203
we have:
1
(iaa x2)²
lim Px X
lim 000
exp
M M
2
202
(175)
Σ Eota (24,00 - x.. 00 2 (x..0 - 22,2
+
2
33

---

Published as a conference paper at ICLR 2024
Table Hyperparameters for all ImageNetIk experiments in Section 3.2
ResNet 50
Weight initialization
kaiming_unie uniffrm (He et al., 2015)
Backbone normalization
BatchNorm
Head normalization
BatchNorm
Synchronized BatchNorm over replicas
Yes
Learning rate schedule
Single Cycle Cosine
Learning rate warmup (epochs)
10 409G 256
Learning rate base value Learning rate minimum value
0.2
3.2
Optimizer
LARS (You al., 2017)
Optimizer scaling rule
None 0.9
Optimizer momentum
Gradient clipping
None x 10
Weight decay
Weight decay scaling rule
None Yes
Weight decay skip bias
Numerical precision
bf16
Augmentation stack
SimCLR (Chen et al., 2020a)
Using the central limit theorem, ã; converges to C₁ and the second term converges to zero, yielding
"x.alXP 1,1: M-
lim
alo.
(176)
F REAL-WORLD IMAGE REPRESENTATION LEARNING
F.I EXPERIMENTAL DETAILS
Hyperparameeers We present the base for training SimCLR (Chen al., 2020a) and other multi- view methods with ResNet 50 (He et al., 2016) in Table 1.
Augmentations We use SimCLR augmentations throughout (Chen et al., 2020a), with petoo__jitterssteeggeeee Bttterrssteenthhhe 1.0 and image size override of 224 224. For complete- ness, we provide our training augmentation here, our testing augmentation is the standard resize, center crop and normalize, and general multiplicity M corresponds sampling M independent transformations.
trans www.onkccizzeccrrgg.... setee__aie__vveeeee scale=orop_scall
laterp BICUBIC BrCUBCC
transforms
www......ddooeriiee.......... Sanddomlorizottall""igg.. cransforms.. Randomapply?
transtorms ColorTitter
brightnese= color Sito__jjtter_atreeeee
contrast=0. celor_jicted stitterssttength
aaturation=3. 00 po.or_ or_tttterr_____oeeeeoeeeeeee_ee_ atrenghh, hue-0.2 coior_jitter_ateest
p-0.8,
transforms. Stannsoorm..Rnndomrrre Raw.omrrayscale (p=0.2),
trans transtorma RandamAppdd [M. [M. www.saannnuurruuuu.. 2.0]) p-0.5),
transforms, ToTensor (),
IMAGENET NORKALI2E,
Data All experiments in Section 3.2 are performed on ImageNetlk (Russakovsky et al., 2014). This dataset is commonly used in computer vision and contains 1.28M training, 50K validation and 100K test images of varying resolutions, each with label from one of 1000 object classes.
34

---

Published as a conference paper at ICLR 2024
Table Hyperparameters for fine tuning experiments
ResNet 50
Hcad weight initialization
(Beyer ct al., 2022)
Head bias initialization
In classe (Beyer et al., 2022)
Synchronized BatchNorm over replicas
No
Learning rate schedule
Single Cycle Cosine
Learning rate warmup (epochs)
5
Learning rate base value
{0.0001, 0.001, 0.0013 0
Learning rate minimum value
Batch size
{384, 1024)
Training epochs
{300, 1000, 2000, 4000) SGD
Optimizer
Optimizer scaling rule
SGD
Optimizer base batch size
256
Optimizer momentum
0.9
Gradient clipping
None
Weight decay
0.0
Numerical precision
fp32
Augmentation stack
RandAug (Cubuk al., 2020)
Repeated Aug.
2
RandAug
9/0.5
Mixup prob.
0.8
Cutmix prob.
1.0
Random Erasing prob.
0.25
F.2 FINE-TUNING RESULTS ON TRANSFER TASKS
To investigate whether poly-view methods improve transfer learning performance, we evaluated the ImageNetlk pre-trained models of Section 3.2 by fine-tuning all model weights for a new task.
Datasets Following SimCLR (Chen et al., 2020a) we investigated transfer learning performance on the Food-10 101 dataset (Bossard et al., 2014), CIFAR- 10 and CIFAR- 100 (Krizhevsky et al., 201 4), SUN397 (Xiao et al., 2010), Stanford Cars (Krause et al., 2013), Aircraft (Maji et al., 2013), the De- scribable Textures Dataset (DTD) (Cimpoi al., 2014), Oxford-IIIT Pets (Parkhi et al., 2012) 2), and Caltech-101 101 (Fei-Fei et al., 2007). We report top-1 I accuracy for all datasets, and use the predefined train, validation and test splits introduced by the dataset creators.
Hyperparameters Fine-tuning hyperparameters are summarized in Table 2. Hyperparameters are optimized for the SimCLR model only and are then re-used for the poly-view methods, following the experimental protocol outlined in Section 3.2. All fine-tuning is performed using SGD using momentum. Fine-tuning on smaller datasets is done for a larger number of (e.g. 4k epochs for Aireraft) and with lower learning rate (e.g. 10-4 for Caltech-101). Fine-tuning head weights initialized to zero, with head bias initialized to n ceaasess where Ficlases is the number of classes in the corresponding downstream dataset (Beyer al., 2022).
Results In Table 3 we report the test top-1 accuracy after fine-tuning. We do this for the Geo- metric PVC models with both batch strategies introduced Section 3.2: Growing Batch and Fixed Batch, as well as the SimCLR model, for small (256 epochs) and large (1024 epochs) amounts of pretraining. In all cases, Poly-view methods match or outperform the SimCLR baseline for the same set of hyperparamcters. We also observe that the Geometric PVC method trained for 256 epochs outperforms the SimCLR method trained for 1024 epochs on transfer to Food, SUN297, Cars, Pets. and Caltech- 101, reinforeing the computational efficiency findings of Section 3.2.
35

---

Published as conference paper at ICLR 2024
Model
Food
CIFARIO
CIFARICO
SUN397
Cars
Aircraft
DTD
Pets
Caltech- 101
Pre-training Epochs: 256
Geometric PVC (growing)
89.56
98.33
87.38
65.91
93.47
81.13
72.41
91.31
88.27
(shrinking)
90.08
73.59
SimCLR
88.58
97.76
86.55
65.10
92.96
78.54
71.44
90.51
86.49
Pre-training Epocks: 1024
Geometric PVC (growing)
90.00
98.31
87.66
66.59
93.58
80.58
73.24
90.78
89,80
Geometric PVC (shrinking) SimCLR
90.27
65.68
73.30
88.57
89.42
98.17
86.70
65.24
93.48
80.72
72.60
90.36
89.66
Table 3: Comparison of transfer learning performance of poly-view Geometric PVC against a baseline SimCLR for the same hyperparameter set across nine natural image datasets. Geometric (growing) and Geometric (shrinking) correspond Geometric PVC (M 8) using Growing Batch and Shrinking Batch strategies respectively (see Section 3.2). Top: ImageNetlk models pre-trained for 256 epochs; bottom: ImageNetIk models pre-trained for 1024 epochs. Results not significantly worse than the best (bootstrap confidence interval of 90%) are shown in bold.
F.3 THE ROLE OF AUGMENTATION STRENGTH HIGH MULTIPLICITY
We present the role of augmentation strength at high multiplicity in Figure 5, investigating the effect of different color jittering (Figure 5a) and cropping (Figure 5b). We do not observe significantly different qualitative behavior between the SimCLR baseline and the poly-view methods.
Growing Batch
Growing Batch (K constant)
70.0
69
Test Top (%)
Top (96)
65.0
62.5
60.0
66
61
52.5
Fixed Batch
Fixed Batch (M K constant)
69
67.5
65.0
Test Top (96)
68
62.5
SimCI
67
SimCLR
Test Top
Geometii Arithmetic
Geometric
Arthmetti
57.5
55.0
65
$2.5
(0.08.0.20)
(0.08, 0.40)
1008 0.65)
duue 0.80)
mont 1.00)
(0.1,, 1.0h
(0.25, am))
(0.40, 1.00
(0.65 I 1.00
1.75
Color Jitter Strength
Crop Scale
(a) Varying color strength.
(b) Varying cropping strategy.
Figure 5: ResNet 50 trained for 128 epochs with different objectives for different strengths of color augmenta- tion (a) and cropping strategy (b), Gcometric and Arithmetic methods presented use multiplicity M 4.
36

---

Published as a conference paper at ICLR 2024
F.3.1 COMPARISON OF TOTAL FLOATING OPERATIONS
In Section 3.2 and Figure 3a, Relative Compute (Equation 29), which measures the total number of encoded views during training, was used to quantify the practical benefits of using Poly-Vies methods.
To quantify the overall training budget, in Figure 6 we report the total number of FLOPs (FLoat- ing OPerations) performed during training. This is the total number of FLOPs in the forward and backward passes for every training step of the model as measured by the Py PyTorch profiler".
The conclusion of Section 3.2 are unchanged when considering total FLOPs instead of Relative Compute. This happens because for sufficiently large models like the ResNet50 we use here, the FLOPs computation dominated by the model encoder and not the loss computation. This results in Relative Compute being proxy for total FLOPs.
Growing Batch
Test Top 1 (%)
65
64
128
256
512
1024
0.0
0.1
0.2
0.3
10
10
10
20
Fixed Batch
Test Top (%)
70
SimCLR Geometric
65
Sufficient
Multi-Crop
128
256
512
1024
0.0
0.5
1018
10
10
Epochs
Updates
x00
Total FLOPs
Figure Training at multiplicity M 8 varying training epochs.
Figure 7: Contrastive ResNet 50 trained ImageNetlk for different epochs or with different view multiplicitiiss Blue, red, orange and black dashed lines represent Geometrie, Multi-Crop, Sufficient Statisties, and SimCLR respectively. Bands indicate the mean and standard deviation across three runs. Points indicate final model performance of corresponding hyperparameters. We use K = 1096 for Growing Batch and K = (2/M) x 4096 for Fixed Batch. Each method is trained with multiplicity M = 8 except the M = 2 SimCLR baseline. We compare models in terms of performance against training epochs (left), total updates (middle) which affected by batch size K, total FLOPs.
hem.htm1
html
37

---

Published as a conference paper at ICLR 2024
F.3.2 IMPLEMENTATION LOSS FUNCTIONS
The pseudocodes for poly-view contrastive loss and sufficient statisties contrastive loss are presented in Algorithms and 2 respectively. Both algorithms have the same structure, with the definition of score matrix as the primary difference. The rearrange and repeat functions are those of einops (Rogozhnikov, 2022).
Algorithm Poly-View View Contrastive Loss pseudocode.
net sug
encoder projector network
augmentatin icn policy
X[k, 3, tau
minibatcch of images
temperature
get_ (beta: int) Tensor:
Th* self self-supervisnd target is j=i, beteralpha. Produce a mask that removea the contribut buutioon ot 1-1, betal=alpha, 1.e, return [kymy K]
tensor zeros with ones on:
The self-sample index The batas not equa to alpha .n*
mask the sample
mask_sample rearrange (diagona (k), "ka kb -> ka 1 kbt)
beta view
mask_beta rearrange (enes (m), "m -> 1*)
mask mask_beta(;, beta] =
return mask_beta mask_sample
[k,
d generate i views for each sample
x (XX X_2, .... X_ml, dim=1) aug
Di, by
extract cormalized features for each view 12_nermalize (net 4X_a), dim=-1)
Di, d]
bd score matrix scores einaum (" jmd, knd knd->jnnkk) 2) tau
Dry k1
$ track the lossea for each phpha losses_a_pha (1st()
iterste over alpha and bata for alpha in zange(m) (m)
losses_alpha_bet:
()st()
for beta in range(m):
skip on-dingonal terms if
logits acoresl;, alpha]
[k, my K]
labels arange (k) beta v
[K]
get_mask (bata)
[k,
logits flagten mask LARGE NUM)
[K, m * X]
loss_elpha_beta cross eapropyrropyyyoooe flogits, labels)
[k]
lasses_alpha__ (posn bsta)
losses_alpha stack (losa_a_pha) dim=-1)
[k, m-13
aggregate over the betas for each alpha
method "srithmetie":
Loss_alpha = logoumexp (Conses_alpha, dim=-1) - log (k)
[k)
el1: method =- "geometrie"
:oss_alpha mean (losses_aaphhe dim--1)
[k]
losses_alpha aop.enddaapddddccccc (Loss_aaaaa)
build final less matrix
losses = stack (losses_alpha, dins-1)
[k)m)
take expecsations
mean flosses, dins-1} lass (samglo_losses)
[k]
scalar
38

---

Published as a conference paper at ICLR 2024
Algorithm Sufficient Statisties Contrastive Loss pseudocode.
net
encoder projector network augmentation policy
aug
XIkr 3, el
minibatch of Lmages temperature
net_ (beta:
The self supervised target is j-1, beta-alpha. Produce mask that renoves the contributicn of jri, betalmalpha, i.e. return k]] tensor of
with The self sample index
The betas not equal to alpha
mask the sample
mask sample zesrrange (diagonal (X), "ka kb -> ka 1 kb*)
mank the the beta-th view mask_beta rearrange (cnes (n), mask_beta[1 beta]
1*)
return masa_beta mask_sample
4. Dk, k]
generate m views for each comple x cXL, X_2, ... XJM,, dim-1) (X)
4 [k, by el
extract normalized features for each view = 12_normalize (net iX_al, dim=-1)
4 [k, d]
build the average of the rest-set
a step 1: repeat the teaturea M times 2tiide repeat
Dky d]
a step 2: zeplace the effect of alpha-th, th view by zero
and correct the biss confficient. m2 1 ml 1") diagonal_one rearrange leye (m), diegonal_zero - ones i [k, m, m, d]) diagonsl_ene &_iid
[k, my d] [k, d]
1) -_tilde diagoral_zext
J step 3: getting the avezage of reat set and nomalize Ltiide mean(a_tilde) dimel)
[k, d] [k, m,
A_tilda 12_normalize (2_tiild, dimm-1)
build scere matrix
scores cinsum ("jmd, knd->-jnkk,, &_tilde) tau
[k, m, m. k]
3 track the losses for each alpha losses_aiphs ()st(()
i itorate over alpha and beta slpaa (m))e(m)
josses_algha_bets list
for beta in range (m)
non-dlagonal
if alpha beta:
lagits scores(;, alpha]
[e, k]
labels
[K] beta
[k]
mask get_mask(bbta)
[k, my K]
lagits = (lagten mask MARGE_NUM)
[k, m 36
loss_elpha_beta cross_entropy flogits, labels)
[k]
losses_alpha stack (losa_a_pha, dim=-1)
[k, m-1]
aggregate over loas_4 appsaaappaa-- = mean (lanssss__ppppa, dims-1]
each alpha
[k)
Losses_alphar dosse__elgha.eppendiii.. aope
4 build rinal loss matrix
stack (assckcoosss_eppaa. dim--1)
[kym]
take expeccations
sample_losses flosses, dim--1) losa mean (eampleeeee__
[K] Boalax
39

---

Published as a conference paper at ICLR 2024
G EXPANDED RELATED WORK
SSL methods and contrastive learning Contrastive learning appears in many SSL methods. Sim- CLR (Chen & He, 2021) leverages the InfoNCE objective to train the encoders to find good repre- sentations. MoCo (He et al., 2019; Chen et al., 2020b; 2021) uses momentum encoder create a moving average of the model's parameters, enabling it to learn powerful image representations without the need for labeled data. CLIP (Radford et al., 2021) is novel multi-modal approach that leverages contrastive learning to bridge the gap between text and images. VICReg (Bardes et al,, 2021) is another SSL method that uses contrastive learning but also address the collapse problem in which the encoders produce non-informative vectors using regularization terms. There are some works (Shwartz-Z4) ZZi et al., 2023; Balestriero & LeCun, 2022) providing theoretical understanding of VICReg's ' performance and comparing it to the other methods like SimCLR. Tian et al. (2020a) is the closest work we know of that has investigated a simple form of multiplicity view in contrastive learning. Their approach is to get the average of pairwise contrastive learning, which translates to the lower-bound of Validity property that we have, for which we have shown theoretically in Equa- tion 8 that our aggregation function outperforms this lower-bound. We also note that the authors did not provide any thcoretical explanation regarding multiplicity.
information-theoredl perspective in SSL One of the main approaches to understand SSL and contrastive learning study the dependencies between pairs of variables or views. MI provides an insightful metric for quantifying dependencies resulting to the point that estimating and optimizing the MI becomes important. van den Oord et al. (2018) introduces InfoNCE loss for the first time. combines predicting future observations with probabilistic contrastive loss, hence the name Con- trastive Predictive Coding. The intuition behind this work is that different parts of the signal share same information. Poole et al. (2019) provides framework to estimate MI by showing connections between different MI lower-bounds, and investigating the bias and variance of their sample-based estimators. Tschannen al. (2020) leverages this framework and builds connection between MI maximization in representation learning and metric learning by also pinpointing that under which dependency conditions MI approaches perform well. Lee et al. (2023) provides more insights on MI maximization in contrastive learning like the effect of same-class-sampling for augmentations by upper-bounding the MI. Gálvez et al. (2023) shows that not only contrastive SSL methods, but also clustering methods (Caron et al., 2020; 2018) 8), and (partially) distillation methods (Grill et al., 2020; Caron al., 2021) implicitly maximize MI.
The role of augmentation in SSL Augmentation is a critical part of SSL methods in computer vision to keep the task-relevant information (Tian et al., 2020b). Trivial augmentations result in non-informative representations, preventing the model to find the main features to distinguish be- tween positives and negatives, while hard augmentations make it difficult for the model to classify the positives from negatives. Balestriero et al. (2022b) tackles this problem by quantifying how many augmentations are required for good sample-based estimation of MI have low variance and better convergence. Kim et al. (2023) addresses this challenge in contrastive learning with a different approach and by adding weights that implies the goodness of the augmentation. On the importance of augmentation, von Kügelgen et al. (2021) shows that augmentation helps to disen- tangle the content from the style in the images. From another perspective, some works explore the effect of different augmentation strategy like multi-crop in contrastive learning and SSL methods (Caron et al., 2020; 1). Fort et al. (2021) has a similar setting to ours and shows that increasing the number of augmentations, i.c increasing the signal to noise ratio, helps the supervised learning classifier in both growing batch and fixed batch scenarios. Wang & Qi (2022) studied the effect of strong augmentations in contrastive learning, proposing new framework transfer knowledge from weak augmentations to stronger ones in order to address the loss of information due to harsh transformations; improving the performance.
Sufficient Statistics Sufficient statistics provide a summary of the data, from which one can make inferences on the parameters of the model without referencing samples. Sufficient statistics can be readily connected to the Infomax principle and have been used to re-formulate contrastive learn- ing Chen et al. (2020c). One key observation is that two-view contrastive learning may not yield representations that are sufficient w.r.t the information they contain to solve downstream tasks (Tian et al., 2020a; Wang al., 2022). Poly-view contrastive learning tasks have increased amount of
40

---

Published as a conference paper at ICLR 2024
available information shared between views, which we believe improves the resulting represen- tations' sufficiency w.r.t. any possible downstream task that would have been possible from the unknown generative factors.
H EXTENSIONS TO DISTILLATION
Our primary contributions use the frameworks of information theory and sufficient statistics in- vestigate what is possible in the presence of a view multiplicity M > 2 and derive the different Poly-View objectives from first principles.
It is possible to incorporate multiplicity M 2 into distillation setup like BYOL (Grill et al., 2020). For example, DINOv1 (Caron et al., 2021), which shares many algorithmic parts of BYOL, benefits lot from using the pair-wise Multi-Crop task that we described in Section 2 and Ap- pendix D (although DINOvI, there is more than one augmentation policy).
One option for extending distillation methods like BYOL and DINOv1 from Multi-Crop to poly- view tasks One-vs-Rest sense is to have the EMA teacher produce M logits, which are aggregated into single logit (similar to the sufficient statistics choice for Q in Equation 24) for producing the target pseudo-label distribution. The gradient-based student could then be updated based its predictions from the held-out view, and this procedure aggregated over all the view hold-outs.
The core difference between the distillation procedure above and the poly-view contrastive meth- ods is that the large-view limit of poly-view contrastive methods is provably a proxy for InfoMax (Section 2 and Equation 26). There may be way to obtain theoretical guarantees for the large- view distillation methods (using for example tools from Gálvez et al. (2023)), and could prove an interesting future direction for investigation.
CONTRIBUTIONS
All authors contributed to writing this paper, designing the experiments, and discussing results at every stage of the project. All contributions are in alphabetical order by last name.
Writing and framing Majority of writing done by Dan Busbridge, Devon Hjlem and Amitis Shidani. Research framing done by Dan Busbridge and Devon Hjlem.
Theoretical results Proofs of MI lower-bound with Multi-Crop (Appendix C.1), lower variance of Multi-Crop MI bound (Appendix C.2), Generalized Ixws (Appendix C.3), Validity of Arithmetic and Geometric PVC (Appendix C.4), the connection between sufficient statistics and MI bounds (Appendix C.7), the generalization one-vs-rest MI to arbitrary set partitions (Appendix E.I,, and the linking of poly-view methods SimCLR (Appendix E.2) and SigLIP (Appendix E.3) done by Amitis Shidani. Derivation of optimal multiplicity (Appendix E.5) done by Amitis Shidani, Dan Busbridge and Devon Hjelm. Derivation of Arithmetic and Geometrie PVC loss functions (Appendix C.5) done by Dan Busbridge and Amitis Shidani. Proof of Behavior MI Gap (Ap- pendix C.6) done by Eeshan Gunesh Dhekane and Amitis Shidani.
Sufficient statistics Extension to sufficient statistics framework (Appendix F.4) proposed by De- von Hjlem. Derivation of sufficient statistics loss function (Fquation 22) done by Amitis Shidani and Russ Webb. Derivation of Q (Equation 24) done by Dan Busbridge.
Synthetic ID Gaussian Synthetic setting proposed by Dan Busbridge and Amitis Shidani based on discussions with Devon Hjelm. One-vs-rest MI (Equations 26 and 173) derived by Dan Bus- bridge and Amitis Shidani. Proofs of convergence to InfoMax (Equation 27) and to the sufficient statistic conditional distribution (Equations 28 and 176) derived by Amitis Shidani. Code to produce empirical results (Figure 2) and related analysis by Amitis Shidani.
Real world representation learning Experimental protocol for training duration experiments (Section 3.2 and Appendix 1..1) designed by Dan Busbridge. Experiments conducted by Dan
41

---

Published as a conference paper at ICLR 2024
Busbridge, Eeshan Gunesh Dhekane, Jason Ramapuram, Amitis Shidani, and Russ Webb, Fine- tuning transfer experiments (Appendix F.2) done by Jason Ramapuram. Investigation into the role of augmentation strength (Appendix F.3) done by Amitis Shidani.
Implementation details ImageNerlk investigations carried out in PyTorch distributed frame- works developed by Dan Busbridge, Eeshan Gunesh Dhekane and Jason Ramapuram. Design and implementation of fast poly-view contrastive losses (Algorithm 1) by Dan Busbridge. Design and implementation of fast sufficient statistics loss (Algorithm 2) by Amitis Shidani. Baseline imple- mentation of SimCLR, by Jason Ramapuram.
42


Reviews:
[{"review_id": "zVPU448ixd", "reviewer": "Reviewer_ymcZ", "summary": "The paper proposed a multi-view (against the previous 2-view) contrastive learning, they provide theoretical and empirical evidence that their derived multi-view loss is better than previous multi-crop loss, as it provides a tighter lower bound on the generalized mutual information. They also provide real data evidence showing that their multi-view loss allows more efficient learning compared with previous two-view contrastive learning like SimCLR.", "questions": "I have no questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper investigated an interesting angle of contrastive learning: instead of increasing batch size, they increase the number of views. They provide a solid theoretical framework for their proposal, linking their proposed multi-view loss with previous SimCLR loss and the InfoMax framework. They also provide detailed analysis for comparing these two losses both theoretically and empirically. Overall it is clearly written and easy to follow, and the theoretical analysis aligns with the empirical findings is another big plus. Overall, they provide a new angle to improve the contrastive learning idea, which I believe might unleash further power of self-supervised learning.", "weaknesses": "I just have one suggestion: maybe you can comment (or leave for future work) about how other self-supervised learning can fit in your framework, or how your idea can be extended to other SSL methods like BYOL etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pconQFN5RO", "reviewer": "Reviewer_AhEA", "summary": "Although it is possible to design tasks that drawn arbitrary number of views, contrastive works typically focus on pairwise tasks. So, this paper investigates how to match more than two related views in contrastive learning, and derive new learning objectives by information maximization and sufficient statistics. They show that multi-crop reduces the variance of corresponding paired objective but fail to improve bounds on MI; Then they derive new objectives which solve tasks across all views through information theory, and show that the MI Gap is monotonically non-increasing with respect to the number of views. Also, the poly-view contrastive method is beneficial to reduce the batch size and increase multiplicity.", "questions": "1. The experiments shown in section 3 display the relative compute of algorithms and show One-vs-Rest objectives could beat simCLR with the same relative compute, how about the real training complexity. And how does it perform on real downstream tasks.\n\n2. I cannot fully understand why the One-vs-Rest loss could effectively reduce the training epoch and batch size.\n\n3. The Geometric loss is actually also an extension of simCLR loss, just like Multi-Crop, but Geometric loss could be a tighter bound of MI while Multi-Crop cannot. It seems to be theoretically correct, but how can we understand it empirically.\n\n4. An extension of simCLR loss outperforms the carefully designed SUFFICIENT STATISTICS loss in section 3, does it mean that the poly-view contrastive learning works mainly because the superiority of simCLR loss?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Generalizing the information-theoretic foundations to poly-view is an interesting idea, and the One-vs-Rest MI seems to be quite reasonable.\n\n2. Those theoretical results are clear, the derivation process of the One-vs-Rest objective is convincing.\n\n3. The paper is well written and easy to follow.", "weaknesses": "1. The assumption 3 in section 2.4.1 is kind of strong.\n\n2. The experiments do not show the superiority of poly-view contrastive learning. The computation time is not evaluated by real time, and the downstream performance is not displayed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rlUOvp9vhI", "reviewer": "Reviewer_P4Q6", "summary": "The paper investigates the effect when introducing view multiplicity in contrastive learning. Specifically, the paper gives a generic information-theoretic derivation of such multi-view framework and shows that SimCLR loss is a special case of the derived `poly-view' contrastive learning. The paper concluded from the theoretical foundation that higher view multiplicity enables a new contrastive learning where, surprisingly, it is beneficial to reduce the batchsize and increase multiplicity. The paper also associate their  theoretical findings with experiments.", "questions": "Please see the 4 weakness above for questions to be addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper delves into the impact of incorporating multiple perspectives in contrastive learning. The strength include: \n\nS1: it presents a comprehensive information-theoretic analysis of this multi-view approach and establishes that the SimCLR loss can be considered as a special case of the resulting 'poly-view' contrastive learning. \n\nS2: Based on the theoretical framework, the research concludes that greater view multiplicity facilitates a novel form of contrastive learning, wherein it proves unexpectedly advantageous to decrease the batch size while augmenting the multiplicity. \n\nS3: Furthermore, the paper justifies these theoretical discoveries with empirical experiments.", "weaknesses": "However, the paper has several weakness that worths further discussion. \n\nW1: What is the exact loss function the paper used to define the poly-view contrastive learning? It seems the Eq. (22) is the poly-view contrative loss, whereas it is in a very high level abstract and implicit form, making it hard to interprete how to compute the empirical loss for the terms, and why M=2 links to SimCLR. I recommend to make the loss in a more explicit form of empirical losses and interpretation (e.g., what is the used sufficient statistics for M=2 for SimCLR? ) \n\nW2: Empirical evidence lacks suitable interpretation and linkage to the significance of the theorems. For significance, I mean how we can use the theorem takeaways to practically improve the SSL algorithms? I expect to see the evidence on larger dataset with mainstream architecture such as ResNet and ViT/transformer with longer epochs. \n\nW3: It is unclear to me if the multiplicity of views simply benefits from more equivalent of epochs (in the experiments) or whether the exposure to the number of data has been constrained to be exactly same between the poly-view contrastive learning and other baselines. \n\nW4: There is no comparison between the proposed method and SOTA method, in terms of how the method contributes to and improves the SOTA methods under the theoretical foundations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dnOEn2ndwM", "reviewer": "Reviewer_PqKH", "summary": "This paper studies contrastive learning by matching more than two related views, which is called poly-view contrastive learning. Unlike traditional contrastive learning methods that take pairs of tasks, it increases the view multiplicity and investigates the design of SSL tasks that use many views. Experiments show that it is beneficial to decrease the number of unique samples while increasing the number of views of each sample.", "questions": "I wonder how strong the theory on multiplicity can be. Is it possible to specify how exactly the number of views on each example improves the algorithmic performance? Would it be essential on the average number of views, or maximal number? Does there exist a threshold on the number of views, such that once it exceeds the threshold, more views do not help?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of designing contrastive learning methods using poly-view seems novel. It utilizes an observation from the prior works that using multiple positive views improves the performance. The paper is well-written.", "weaknesses": "Although there are prior works showing that multiplicity improves generalization and convergence of neural networks, it lacks rigorous theory on the relation between contrastive learnability and the number of views on each sample.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)