RényiCL: Contrastive Representation Learning with Skew Rényi Divergence
Authors: Kyungmin Lee
Jinwoo Shin
Venue: NeurIPS 2022 Accept (2022)
Keywords: Contrastive learning
Self-supervised learning
Representation learning
Data augmentation

Abstract: Contrastive representation learning seeks to acquire useful representations by estimating the shared information between multiple views of data. Here, the choice of data augmentation is sensitive to the quality of learned representations: as harder the data augmentations are applied, the views share more task-relevant information, but also task-irrelevant one that can hinder the generalization capability of representation. Motivated by this, we present a new robust contrastive learning scheme, coined RényiCL, which can effectively manage harder augmentations by utilizing Rényi divergence. Our method is built upon the variational lower bound of a Rényi divergence, but a naive usage of a variational method exhibits unstable training due to the large variance. To tackle this challenge, we propose a novel contrastive objective that conducts variational estimation of a skew Renyi divergence and provides a theoretical guarantee on how variational estimation of skew divergence leads to stable training.  We show that Rényi contrastive learning objectives perform innate hard negative sampling and easy positive sampling simultaneously so that it can selectively learn useful features and ignore nuisance features. Through experiments on ImageNet, we show that Rényi contrastive learning with stronger augmentations outperforms other self-supervised methods without extra regularization or computational overhead. Also, we validate our method on various domains such as graph and tabular datasets, showing empirical gain over original contrastive methods. 

RényiCL: Contrastive Representation Learning with Skew Rényi Divergence
Jinwoo Shin Korea Advanced Institute of Science and Technology (KAIST) (kyungmnlee, jinwoos)ekan @kaist. ac kr
Kyungmin Lee
Abstract
Contrastive representation learning seeks to acquire useful representations by esti- mating the shared information between multiple views of data. Here, the choice of data augmentation is sensitive to the quality of learned representations: as harder the data augmentations are applied, the views share more task-relevant information, but also task irreeeateleant one that can hinder the generalization capability of repre- sentation. Motivated by this, we present new robust contrastive learning scheme, coined RényiCI, which can effectively manage harder augmentations by utilizing Rényi divergence. Our method built upon the variational lower bound of Rényi divergence, but a naïve usage of variational method is impractical due to the large variance. To tackle this challenge, we propose novel contrastive objective that conducts variational estimation of a skew Rényi divergence and provide a theoretical guarantee on how variational estimation of skew divergence leads stable training. We show that Rényi contrastive learning objectives perform innate hard negative sampling and easy positive sampling simultancously so that can selectively learn useful features and ignore nuisance features. Through experiments on ImageNet, we show that Rényi contrastive learning with stronger augmentations outperforms other self-supervised methods without tational overhead. Moreover, we also validate our method on other domains such graph and tabular, showing empirical gain other contrastive methods. The implementation and pre-trained models are available at
regularization compu-
Introduction
Recently, many AI studies are enamored by the power of contrastive methods in learning useful representations without any human supervision, e.g., image [1, 2, 4, 5, 6], text [7, 8], audio [9, 10] and multimodal video [11, 12, 13]. The key components of contrastive learning can be summarized into two-fold: data augmentation and contrastive objectives. The data augmentation generates different views of data where the views share information that relevant to the downstream tasks. The contrastive objective then enforces the representation capture the shared information between the views by bringing the views from the same data together and pushing the views from different data away in the representation space. Therefore, the choice of data augmentation is critical in contrastive learning, cf., see [3, 14, 15]. If the views share insufficient information, the representations cannot learn sufficient features for downstream tasks. On the other hand, if there is too much information between the views, the views might share nuisance features that hamper the generalization.
contrastive learning objective using Rényi divergence [16], coined RényiCI. which can effectively handle the case when the views share excessive information. The Rényi divergence is generalization of KI. divergence that is defined with additional parameter
In this paper, propose
'nttps://github http:://ggghuuu.oooo.o.. wwwweo/wwwwl////e/eeeooooe
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

called an order. As the order becomes higher, the Rényi divergence penalizes more when two distributions mismatch. Therefore, since the goal of contrastive representation learning train an encoder to discriminate between positives and negatives, we hypothesize that maximizing the Rényi divergence between positives and negatives could lead to more discriminative representations when the data augmentations are aggressive. Furthermore, our theoretical analysis shows that RényiCL. weighs importance on easy positives and hard negatives by using a higher order.
To implement RényiCL, one can consider variational lower bound of Rényi divergence [17]. However, we show that a variational lower bound of Rényi divergence does not suffice for contrastive learning due to their large variance. To tackle this challenge, we first made a key observation that the existing contrastive objectives are a variational form of skew KL divergence, and show that variational estimation of skew divergence induces low variance property. Inspired by this, we consider a variational lower bound of a skew Rényi divergence and use to implement RényiCI.
Through experiments under ImageNet [18], RényiCI. achieves 76.2% linear evaluation accuracy, outperforming other self-supervised learning methods even using significantly less training epochs. Moreover, we show that RényiCL representations transfer to various datasets and tasks such as fine-grained object classification and few-shot classification, outperforming other self-supervised baselines. Finally, we validate the effectiveness of RényiCL on various domains such as vision, graph, and tabular datasets by showing empirical gain over conventional contrastive learning methods.
2 Related Works
Since Chen et al. [3] emphasized that the usage of data augmentation is crucial for contrastive learning, a series of works proposed various contrastive learning frameworks that showed great empirical success on visual representation learning [2, 3, 4, 5, 6, 19]. Inspired by the empirical success, recent works focused on optimal view generation for both positive and negatives. Tian et al. [14] proposed the InfoMin principle for view generation, which states that the views should share minimal information that is relevant to the downstream tasks. Another line of research focuses on hard negative samples [20, 21, 22] to enhance the performance of contrastive learning. Especially, Robinson et al. [20] proposed to use importance sampling to learn with hard negative samples.
Meanwhile, others are interested in searching for different contrastive objectives by modeling with various probabilistic divergences between positives and negatives. Oord et al. [23] proposed Contrastive Predictive Coding (CPC) (as known as InfoNCE) that is widely used in various contrastive learning scheme such as [2, 3, 4]. They theoretically prove that CPC objective is variational lower bound to the mutual information, later [24] proposed Multi-Label CPC (MLCPC) which is a tighter lower bound to the mutual information. Hjelm et al. [25] proposed DecpInfoMax which used Jensen- Shannon divergence for variational estimation and maximization of mutual information. Zbontar et al. [26] proposed Barlow-Twine which is shown to be HillbertSSchiie Independence Criterion that approximates a Maximum Mean discrepancy measure between positives and negatives [27]. Ozair et al. [28] proposed Wasserstein Predictive Coding, which is both lower bound to the Wasserstein distance and mutual information. Similar to our approach, Tsai et al. [29] proposed relative predictive coding that uses skew x² >dvdieeree for contrastive learning, and empirically and theoretically show that variational estimation of skew ddvveergee leads to stable contrastive representation learning. Lastly, our work is similar [30], which proposed robust contrastive learning objective that uses symmetric binary classification loss for contrastive learning so that it can deal with noisy views.
3 Preliminaries
3.1 Rényi divergence
We first formally introduce Rényi divergence, which is a family of probability divergences including the popular Kullback-Leibler (KL) divergence as special case. Let P. Q be two probability distributions such that P is absolutely continuous with respect to Q denoted by P & Q (i.e., the Radon-Nikodym derivative dPP/Q exists). Then, the Rényi divergence [16, 31] of order y E (0, 1) U (1, oo) defined by
dP
7-1
= 7(7 1)
log Er
dQ
2

---

It is intertwined with various f f-divergences, e.g., Rényi divergence with y and becomes KL divergence Dx. (I Q) and monotomic transformation of x2 divergence, respectively. Remark that the Rényi divergence with higher order y, penalizes more when the probability mass of P does not overlap with Q [32]. Rényi divergence has been studied various machine learning tasks such as variational inference [33, 34] and training [35, 36] or evaluation of generative models [37].
3.2 Variational lower bounds of mutual information
The estimation and optimization of mutual information is important topic in various machine learning tasks including representation learning [38]. The mutual information is defined by the KL divergence between the joint distribution and the product of marginal distributions. Formally, given two random variables X ~ Px and Y ~ Py, let Pxy be the joint distribution of X x Y. Then, we call the pair of samples (x, y) Pxy as a positive pair, and (Ir y) PxPy negative pair. Then, the mutual information is defined by KL. divergence between positive pairs and negative pairs:
I(X;Y) (Px Px
In general, the mutual information is intractable compute unless the densities of X and Y explicitly known, thus many works resort to optimize its variational lower bounds [38] associated with neural networks [39]. The Donsker-Varadhan (DV) [40] objective a variational form of KL
divergence defined as follows:
(f) = Er[s] - log Eqle") where (x = sup
=sup Iov(s); fEF
(1)
where F is a set of bounded measurable functions on the support of and Q, and the optimum f E F satisfies f x log dP aQ' Then, Belghazi et al. [39] proposed MINE that uses (I) follows:
THINE (J) = Exxx [f(,, v)] - log Epx By (ef(ma))).
However, Song al. [41] showed that estimation with MINE objective might occur large variance unless one uses large number of samples, and Tsai et al. [29] also empirically evidenced that contrastive learning with the DV objective suffers from training instability.
To address the issue, the contrastive predictive coding (CPC) objective (also as known as In- foNCE) [23] is popular choice for various practices including contrastive representation learn- ing [2, 4, 3]. Given batch of samples {x; [ from X, assume we have a single positive y," and K negatives (MG) j=1 for each z;. Then, the CPC objective is defined as follows:
K
(K + 1) (ur.y")) e((xyt)) +EK MaN)))
Icpe(f) E Jx 1 log
where it known that (f) & I(X;Y) Y) for any bounded measurable function f [23]. However, as Topa(f)) < log(K' + 1) for any f, the CPC objective becomes a high-bias estimator if the true value I(X; YY) is larger than log( (K( + +1) 1). To address this, Poole et al. [38] proposed a-CPC which controls the bias by inserting E [0, into CPC as follows:
(
(f)
18, (2 (x 1 ~P log
(ef(at 1-a K Σ e (2.27)
The a-CPC objective also admits a variational lower bound the mutual information I(X; Y) for any a E [0, 1] (recovers the original CPC when a x ) and it achieves smaller bias when using smaller value of a. Furthermore, Song et al. [24] proposed a-Multi Label CPC (MLCPC) which provides even tighter lower bound to the mutual information. While CPC can be considered as (K- + 1) 1)-way 1-shot classification problem, given B batch of anchors { to B(K + 1)- B-shot multi-label classification problem as follows:2
mmm MLCPC is equivalent
7(a) T( E ((aastt)) P Pxy
1 B
Σ
log
e((
(l( +
1-e B
EK ((nn)1)))
B
BK
RRemark that slightly different form as in [24] scale be reside in [0, 1).
3

---

4 Variational Estimation of Skew Rényi Divergence
Similar to variational estimators of Dxa (P Py in the previous section, we consider a varia- tional estimator of Rényi divergence R, (Pxy P contrastive representation learning. To that end, we first introduce the following known lemma which states variational representation of Rényi divergence similar to the DV objective in (1).
). in particular, considering its application to
Lemma 4.1 ([17)), For distributions P. Q such that P & Q, let F be set of bounded measurable functions. Then Rényi divergence of order 7 E (0, 1) U (1, 00) admits admits following variational form:
R (P = -sup Inany: for IRamgs (f) = 7
I Ronys
log Toggppe(--111
log Eqle""], (4)
fEF
where the optimum f E satisfess f x log dP aq
Also, as y - 1, (4) becomes DV objective in (1).
4.1 Challenges in variational Rényi divergence estimation
To perform contrastive representation learning, one can use the variational estimator (4) for R, (Pxx)l similarly did for KL divergence. However, the contrastive learning with (4) is impractical due to the large variance. In Appendix C.2, we show that even for simple synthetic Gaussian dataset, (4) not available due exploding variance when estimating Rényi divergence
between two highly-correlated Gaussian distributions. This exploding-varian variance issue of (4) resembles
that of the variational mutual information estimator well observed in the literature [29, 38, 41]. In the following theorem, we provide an analogous result for the Rényi variational objective (4).
Theorem 4.1. Assume P < Q and Varq /dQ] < co. Let Pm and Qn be the empirical distributions of m iidd samples from P and n. i.d samples from Q respectively. Define
12) Ranys (f) =
7-1 log E.,, (el 1)5 log Eq, 167,,
1
and assume we have f x log(dP/dQ). Then Vy 1, Vm N, we have
lim Varr.q (f)) e² Da -72 n-220
1211
The proof of Theorem 4.1 is in Appendix A.I. Theorem 4.1 implies that even though one achieves the optimal function for variational estimation, the variance of (4) could explode exponentially with respect to the ground-truth mutual information. This result is coherent with [42], which identified the problems in variational estimation of mutual information.
On the other hand, we recall that CPC and MLCPC are empirically shown to be low variance estimators [38, 24]. Then, the natural question arises: what makes CPC and MLCPC fundamentally different from the DV objective? In the following section, we answer to this question by showing that the CPC and MLCPC objectives are variational lower bounds of a skew KL divergence, and provide a theoretical evidence that variational estimators of skew divergence can have low variance. This insight will be used later for designing low-variance estimator for the desired Rényi divergence.
4.2 Contrastive learning objectives are variational skew-divergence estimators
We first introduce the definition of a-skew KL divergence. For distributions P, Q with P << Q and for any a E [0, 1], the a-skew KL divergence between P' and Q is defined by the KL divergence between P and the mixture aP + (1-a)) a)
(a) (PII)) = Dzi a
One can see that the DV objective (1) for æ-skew KL divergence can be written as following:
D (a) (P Q) = sup Er(s] log (aEEle + a))eee
(5)
fer
Then following theorem reveals that a- CPCCC and a-MLCPC are variational lower bounds of a-skew divergence between Pxy and PxPy, D(a) (Pxy || PxPr)

---

Theorem 4.2. For any a E (0.1/2). the a- a-CPC and a-MLCPC are variational lower bound of a-skew KL divergence between Pxy and PxPy. Py, i.e., the following holds:
( = sup I (f) = D (Pxv PxPr)
sup FEF
feF
Here we provide a simple proof sketch and the full proof is in Appendix A.2. From the Jensen's inequality, one can see that the variational form in (5) is a lower bound to the a-MLCPC. Thus, we have D (Pxy PxPr) I sup, I On the other hand, also show that (f) I Dle) (Pxy Py) for all f E F. Therefore, show that a-ML.CPC is variational lower bound of D (Pxv Px)r) Note that the same argument holds for a-CPC. Remark that Theorem 4.2 strictly loose bounds of mutual information. In particular, from the convexity of KL divergence, the following holds for any > 0:
implies that a-CPC and a-MLCPC
DKo (PxY Px (1 ai) Px Drx Px
Also, Theorem 4.2 reveals that a a-CPC and a-MLCPC can be written as following population form:
Ioed (/) Exxy (sx v) Pxx [log (alBpyj (e (ar))) + (1-a)) Br ( [e/(c.)))]]
I (f) = Ex (z. y)] log (aEpxy (e + aa)EE Pv (x.))).
Variational lower bounds of skew-divergence have low variance Further, formally show that the variational estimation of empirical skew divergence has low variance, explaining why CPC and MI.PPC objectives become low-variance estimators of the mutual information. The following theorem shows that the variance of the variational estimator can be adjusted with the right choice of a.
Theorem 4.3. Assume P & Q and Varq dd//d) < oc. Let Pm. and Qu be empirical distributions of m ii.d samples from and n i.i.d samples from Q. Then define
1F==p.... -
log (alE,,, [e'] + -x)BB,.(e
and assume that there is ffe E Fhat [Z(E (f) - 1 (Pm 1[Qr] < E for some E > O. Then for Va 1/8, the variance of estimator satisfies
C2(a)
logg)
(og)
Varr.q [Tla) C1ef
min(n,m) m]
m
a²n
for some constants C1, C3. C4.
> 0 that are independent of n, and DEL DzL(P((), and c2(a)
satisfies (2 = min (PII(( ), where xP(11) Eo[(dP/dQ)])
1-a
The proof is in Appendix A.3. Since we have x²(PIIQ) > (u(P)( 1, if is too small, the bound in Theorem 4.3 is loose as in Theorem 4.1. Therefore, Theorem 4.3 implics that one should use sufficiently large a achieve low variance. Theorem. 4.3 demonstrates that if one can find a sufficiently close critic for empirical skew KI. divergence, the variance of the estimator is asymptotically bounded unless we choose large enough a = a, = w(n -1/2).
4.3 Variational estimation skew Rényi divergence
Inspired by the theoretical guarantee that skew divergence can achieve bounded variance, we now present a variational estimator of skew Rényi divergence. Remark that the analogous version of Theorem 4.3 can be achieved for Rényi divergence (see Appendix A.4), For any E [0. 1] and y E (0, 1) U (1, oo), the a-skew Rényi divergence of order ? defined as:
R = R2 aP + Q)
Then, we define (a, R)-Rén Multi-Label CPC (RMLCPC) objective by using the variational lower bound (4) for a-skew Rényi divergence between Pxy and Px Py:
1
log Exy ( (1)())(( log(aiErx ( (e7f(x.0)) + -a))EExx By (erf(as)))
1 ?
7
5

---

which satisfies supy I(a0] [ R (Pxy PxPy) for any E [0, and 7 E (0, 1) (1).
Note that the RMLCPC objective can be used for contrastive learning with multiple positive views. For example, when using multi-crops data augmentation [43], instead of computing contrastive objective for each crop, we gather all positive pairs and negative pairs and compute only once for the final loss. The pseudo-code for the RMLCPC objective is in Appendix Algorithm 1.
5 Rényi Contrastive Representation Learning
In this section, we present Rényi contrastive learning (RényiCL) where we use the RMLCPC objective for contrastive representation learning. As discussed earlier, the Rényi divergence penalizes more when two distributions differ. Thus, when using harder data augmentations in contrastive learning, one can expect that RényiCL can learn more discriminative representation.
5.1 Rényi contrastive representation learning
We begin with backgrounds for contrastive representation learning. Given dataset X, the goal of representation learning is to train an encoder X' - Rd that is a useful feature of X. Especially, contrastive representation learning enforces to discriminate between positive pairs and negative pairs [2], where positive pairs generated by applying data augmentation on the same data and negative pairs are augmented data from different source data. Formally, let V, V' random variables for augmented views from dataset X. Then denote Pvv be the joint distribution of a positive pair and Pv be the distribution of negative pairs. For the sake of brevity, we denote (z, ~ Pvv be a positive pair, and (2,27) ~ Pv Pv: be negative pair. Also, let g(v), 2' g(x") be features of the encoder and Z = gV). Z' = g(V') ) be corresponding random variables of feature distributions. Then we define (2, Pzz be features of positive views, and (z, -) Pz Pz be features of negative views.
The InfoMax principle [25] for representation learning aims to find g that preserves the maximal mutual information between V and V by following:
sup -2
PPz(2z) PzP2))
sup I(Z s I(V;V'), 9=x
ie., finds a neural encoder g which discriminates between positive and negative pairs as much as possible. Here, a-CPC a-MLCPC are used for plug-in variational estimators of mutual information, for example contrastive learning with a-MLCPC satisfies following:
sup g g
sup fEF
I(a) (f. sup g:X-
D/a
(a) (Pzz:
|| Pz Pz.),
since (f (/,g) )) is variational estimator of D(a (Pzz) P2.).
Now we present Rényi contrastive representation learning, which considers the following optimization problem by using skew Rényi divergence:
Ilan) g: fer (F,9)=
R( P2 P2:),
sup
sup
sup
g:X
since T g) is a variational estimator of R2 (Pzz: PzPP:)
5.2 Gradient analysis for RényiCL
In this section, provide an analysis the effect of in RényiCL based on the gradient of contrastive objectives. For simplicity, we consider the case when a = 0, and we provide general analysis for 0 in Appendix B.1. Let fo be neural network parameterized by 6. By using the reparametrization trick, the gradient of (xaco((fe) becomes
VoIncee([oo) = E2,, (VoJo(z, Eg,s-letoee lefo(z, Vofo(z,2
E.,: [efo(s,->)]
= =E
[Vofo(z,2+)) Eagioo [VVof((zz2

---

Table 1: Linear evaluation the ImageNet validation set. We report pre-training epochs Top-1 classification accuracies (%).
Table 2: Semi-supervised learning results on ImageNet. We report Top-1 and Top-5 classification accuracies (%) by fine-tuning pre-trained ResNet-50 with 1% and 10% ImageNet datasets.
Method
Epuchs
Top-1
1% ImageNet
10% ImageNet
SimCLR [3]
800
70.4
Method
Top-1
Top-5
Top-1
Top-5
Barlow Twins [26]
800
73.2
Supervised [3]
25.4
48.4
56.4
80.4
BYOL [47]
800
74.3
SimCLR [3]
48.3
75.5
65.6
87.8
MoCo v3 [6]
800
74.6
BYOL [47]
53.2
78.4
68.8
89.0
SwAV [43]
800
75.3
SwAV [43]
53.9
78.5
70.2
89.9
DINO [43]
800
75.3
Barlow Twins [26]
55.0
79.2
69.7
89.3
NNCLR [19]
1000
75.6
NNCLR [19]
56.4
80.7
69.8
89.3
C-BYOL [48]
1000
75.6
C-BYOL [48]
60.6
83.4
70.5
90.0
RényiCL
200
75.3
RényiCI.
56.4
80.6
71,2
90.3
RényiCL
300
76.2
where ga(a, == efo(z,=") is self-normalized importance weights [44], and sg is stop-gradient operator. Thus, the MLCPC objective is equivalent to following in terms of gradient [45]:
INLCPc(fo) E [f, 2+)] (eggio(s..-> [fo(,, 27)
(6)
This shows that the original contrastive objective performs innate hard negative sampling with importance weight qo [20], as the gradient of negative pairs becomes larger as the value of fo becomes larger. On the other hand, for RMLCPC objective, by letting a 0, and taking gradient with respect to 8, we have
IF 12 (e(t- (ol VoJo Re,. (er-lerto Vofo(2,2
V VeIlt) INLccce((fo)
E (elt (e(7-1)550(22
E (e)56(2.2
Begloo Vefe (e - Bagino(a. Vefo(2,2
where sg(go(2, 22 importance weights for each positive and negative term. Hence, the RMLCPC objective is equivalent the following in terms of gradient:
1)) clr-1)fo(a,=") and sg(go(2, =)))) effo(s.. are self self-normalizing
7(7) (
-) [fo2=, (eegl
Remark that (6) and (7) have common form that fe is maximized for positive pairs and minimized for negative pairs, which is similar to contrastive divergence [46]. On the other hand, (6) and (7) have two differences:
Hard negative sampling: the gradient weighs more on harder negatives, i.c., (2, 2 with high value of fo(2,=7), as increases [20].
Easy positive sampling: the gradient weighs more on easier positives, i.e., (z, =+) with high value of fe +) as E oc) increases.
Therefore, when there are positive views that share task-irrelevant information, the RMLCPC objective resists updating. and it regularizes the model to ignore issk-ireleevatt information. The hard negative sampling with importance weight was proposed in [20], but our analysis shows that CPC and MLCPC conduct intrinsic hard negative sampling. Also, the Rényi contrastive learning can control the level of hard negative sampling by choosing the appropriate y Lastly, remark that analysis given for a = O, but in practice we use nonzero value of a. In Appendix B.1, we show that requires sufficiently small values of have the effect of easy positive sampling for harder data augmentations.
Experiments
6.1 RényiCL for visual representation learning ImageNet
Setup. For ImageNet [18] experiments, we use ResNet-50 [49] for encoder g. We use MLP projection head with momentum encoder [4], which is updated by EMA, and we predictor, following the
7

---

Table 3: Transfer learning performance object classification datasets. For Aircraft and Flowers, report mean per-class accuracy (%), and for VOC2007, report I-point mAP. Otherwise, report Top-1 classification
accuracies (%).
Method
CIFARIO
CIFAR100
Food101
Flowers
Cars
Aireraft
DTD
SUN397
VOC2007
Supervised [3]
93.6
78.3
72.3
94.7
66.7
61.0
74.9
61.9
82.8
SimCLR [3]
90.6
71.6
68.4
91.2
50.3
50.3
74.5
58.8
81.4
BYOL [47]
91.3
78.4
75.3
96.1
67.8
60.6
75.5
62.2
82.5
NNCLR [19]
93.7
79.0
76.7
95.1
67.1
64.1
75.5
62.5
83.0
RényiCL
94.4
79.0
78.0
96.5
71.5
61.8
77.3
66.1
88.2
Table 4: Comparison on contrastive self-supervised methods with harder augmentations. We compare Top-1 linear evaluation accuracy (%) on ImageNet, Pets, Caltech101 datasets and few-shot classification accuracy (%) over 2000 episodes FC100, CUB200, and Plant disease datasets. denotes trained without multi-crops. Here. we consider RényiCL without multi-crops as other basclines do, i.c., for fair comparison.
Linear evaluation
5-way 1-shot
5-way 5-shot
Method
Epochs
ImageNet
Pets
Caltech101
FC100
CUB200
Plant
FC100
CUB200
Plant
InfoMin [14]
800
73.0
86.37
88.48
31.80
53.81
66.11 11
45.09
72.20
84.12
CLSA [59]
800
72.2
85.18
91.21
34.46
50.83
67.39
49.16
67.93
86.15
RényiCL
200
72.6
88.43
94.04
36.31
59.73
80.68
53.39
82.12
94.29
practice of [47, 6]. Then the critic f is implemented by the cosine similarity between the output of momentum encoder and base encoder with predictor, divided by temperature T = 0.5. We maximize RMLCPC objective with 7 2.0.
Data augmentation. Given the popular data augmentation baseline [47, 31, we further use RandAug- ment [50] and multi-crop augmentation [43] implement harder data augmentation. When using multi-crop, we gather all positives and negatives, then directly compute the RMLCPC objective, which differs from the original method in [43] (see in Appendix B.1).
Linear evaluation. We follow the linear evaluation protocol, where we report the Top-1 ImageNet validation accuracy (%) of a linear classifier trained on the top of frozen features. Table compares the performance of different self-supervised methods using linear evaluation protocol. First, RényiCL outperforms other self-supervised methods with large margin by only training for 300 epochs. Meanwhile, RényiCL enjoys better computational efficiency in that it does not require a large batch size [3, 47, 43, or extra memory queue [19].
Semi-supervised learning. We evaluate the usefulness of the learned feature in a semi-supervised setting with % and 10% subsets of ImageNet dataset [3, 43]. The results in Table 2. We observe that RényiCL achieves the best performance on fine-tuning with 10% subset, and runner-up on fine-tuning with 1% subset, showing the generalization capacity in low-shot learning scenarios.
Transfer learning. We evaluate the learned representation through the performance of transfer learn- ing fine-grained object classification datasets by using linear evaluation. We CIFAR 10/100 [51], Food101 [52], Flowers [53], Cars [54]. Aircraft [55], DTD [56], SUN397 [57], and VOC2007 [58]. In Table 3, we compare with other supervuper methods. Note that RényiCL achieves the best performance in 8 out of 9 datasets, especially showing superior performance on SUN397 (3.6%) and
VOC2007 (5.2%) datasets.
Comparison among self-supervised methods using hard augmentations. To demonstrate the effectiveness of RényiCL on learning with harder augmentations, we compare with InfoMin [14] and CLSA [59] where they also use harder augmentations for contrastive representation learning. Table 4 demonstrates the performance of representations learned with different self-supervised methods by linear evaulation on ImageNet [18], Pets [60], Caltech 101 [61], and few-shot classification on FC100 [62], Caltech-UCSD Birds (CUB200) [50], and Plant Disease [63] datasets. While RényiCL achieves slightly lower performance than InfoMin ImageNet validation accuracy, it outperforms InfoMin and CLSA on other transfer learning tasks with large margin, which indicates the superiority of RényiCL on learning better representations under the hard data augmentations.
8

---

Table S: Ablation on contrastive objectives for unsupervised representation learning on image (ImageNet- 100, CIFAR-I00, CIFAR-IO) and tabular (CovType, Higgs-100K) datasets. We report Top-1 accuracy (% with linear evaluation. Average over runs.
ImageNet-100
CIFAR-100
CIFAR-10
CovType
Hliggs-100K
Method
Base
Hard
Base
Hard
Base
Hard
Base
Hard
Base
Hard
CPC
79.7
81.1(+1.4)
65.4 67.1(+1.7)
91.7 91.9(+0.2)
71.6 74.3(+2.7)
64.7 71.3(+6.6)
MLCPC 79.5
81.2(+1.7)
65.6 66.6(+1.0)
91.9 92.1(+0.2)
71.7 74.1(+2.4)
64,9 71.5(+6.6)
RMLCPC
78.9
81.6(+2.7)
64.5 68.5(+4.0)
90.7 92.5(+1.8)
72.1 74.9(+2.8)
64.5 72.4(+7.9)
Table Comparison of different contrastive learning methods on graph TUDataset. Average over runs.
Method
NCT1
PROTEINS
DD
MUTAG
COLLAB
RDT-B
RDT-M5K
IMDB-B
InfoGraph [72]
76.20
74.44
72.85
89.01
70.65
82.50
53,46
73.03
GraphCL [15]
77.87
74.39
78.62
86.80
71.36
89.53
55.99
71.14
JOAO [73]
78.07
74.55
77.32
87.35
69.50
85.29
55.74
70.21
JOAOv2 [73]
78.36
74.07
77.40
87.67
69.33
86.42
56.03
70.83
RényiCL
78.60
75.11
78.98
90.22
71.88
90.92
56.18
72.38
6.2 RényiCL for representation learning various domains
Setup. We perform ablation studies of RényiCL on various domains such as images, tabular, and graph datasets. For images, we consider CIFAR-10/I00 [51] and ImageNet- 100 [64], which is 100-
class subset of ImageNet [18]. For the base data augmentation, we use a popular benchmark from [3], and for the harder augmentation, we further apply RandAugment [50] and RandomErasing [65]. For ImageNet- 100, we use the default settings of [6] with ResNet-50 backbone with only change in learning objective. For CIFAR-10 and CIFAR- 100, we follow the settings in [3] with ResNet- 18 backbone adjusted for the CIFAR dataset, We use same for all CPC, MLCPC, RMLCPC, and use y = 2.0 for RMLCPC
For tabular experiments, we use Forest Cover Type (Cov'Type) and Higgs Boson (Higgs) [66] dataset from UCI repository [67]. Due its massive size, we consider subset of 100K for the Higgs experiments. Since the tabular dataset has limited domain knowledge, we do not use any data augmentation for the baseline and when using data augmentation, we use random masking noise [68] for the Higgs dataset and random feature corruption [69] for the CovType dataset. We follow the settings of MoCo v3 [6] except that a 5-layer MLP used for the backbone. We use y 1.1 for Cov and y 1.2 for Higgs. For evaluation, we use linear evaluation for CovType and we compute the linear regression layer by pseudo-inverse between the feature matrix and label matrix.
Lastly, we examine RényiCL on graph TUDataset [70], which contains numerous graph machine learning benchmarks of different domains such as bioinformatics, molecules, and social network. We follow the experimental setup of [15]: we use graph isomorphism network [71] backbone and use node dropout, edge perturbation, attribute masking, and subgraph sampling for data augmentations. For the graph experiments, we do not change the strength of data augmentation nor add new data augmentation. The detailed hyper- paprrmparaeeter for training and evaluation for the whole dataset can be found in Appendix B.3.
Results. Table 5 compares the Top-1 linear evaluation accuracy (%) of representations with different contrastive learning objectives. We observe that RMLCPC obtains the best performance when using harder data augmentations on all images and tabular datasets. Also, RMLCPC has the largest gain on using harder augmentation, showing that the effectiveness of RényiCL is amplified when we use stronger augmentation. Table compare the Top-1 linear evaluation accuracy (%) of graph RényiCL. with other graph contrastive learning methods. Even without harder data augmentation, RényiCL. achieves the best performance 7 out of 8 benchmarks. Especially, RényiCI. outperforms JOAO [73], where the views are learned to boost the performance of graph contrastive learning.

---

6.3 Mutual information estimation and view selection
Remark that CPC and MLCPC are intrinsically high-bias mutual information estimator as a-skew KL divergence is strictly smaller than original KL divergence for 0 < a < 1, i.e.
D (PIIQ) S (1 - ax < (P(())))).
from the convexity of KI. divergence. Similarly, RMI.PC is also a strictly-biased estimator of Rényi divergence. However, one recover the original mutual information by using the optimal condition of function f from all CPC, MLCPC, and RMLCPC objectives [29]. Here, we describe the sketch and present detailed method in Appendix C.I. From the optimality condition of (1) and (4), the optimal critic f" of a-CPC, a-MLCPC and (a, 7) Rt)-MMCCCC of any y satisfies f xc dPxy
log od (I-o)dxxxx then we use Monte-Catte method [44] to approximate log- nognoommaization constant, and use it to approximate the log-density ratio log dPxy aPxdPy Then we estimate the mutual information by ((X;Y) = Σこ log f(z, yt), where (z,, Pxy are positive pairs.
Mutual information estimation of synthetic Gaussian. Following [38, 24], we conduct experi- ments estimating the mutual information of multivariate Gaussian distributions. In Appendix C.2, we show that while a-CPC, a-MLCPC, and (a, 7))RMCC objectives are high biased objectives, one can achieve low bias, low variance estimator by using the approximation method above.
7 Conclusion and Discussion
We propose Rényi Contrastive
Learning (RényiCL) which utilizes Rényi divergence to deal with stronger data augmentations. Since the variational lower bound of Rényi divergence is insufficient for contrastive learning due to large variance, we introduce a variational lower bound skew Rényi divergence, namely Rényi-MLCPC. Indeed, we show that CPC and MLCPC variational forms of skew KL divergence, and provide theoretical analysis on how they achieve low variance. Through experiments, validate the effectiveness of RényiCL by using harder data augmentations.
I Limitations and future works. While RényiCI. beneficial when using harder data augmentations, did not identified the optimal data augmentation strategy. One use policy search method on data augmentation [74] with our RényiCL similar to that of [75]. Also, we think that the proposed scheme based Rényi divergence could be useful for other tasks, e.g., information bottleneck [76]. Lastly, many works focused on the diverse perspectives of contrastive learning [45, 77, 78], where those approaches could be complementary to work. We leave them for future works.
Negative societal impacts. Since contrastive learning often requires long epochs of training, raises environmental concerns, e.g. carbon generation. Nevertheless, RényiCL is shown to be effective even under smaller number of epochs, compared existing schemes.
Acknowledgement
This work was partly supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea governmen((MSIT) Intelligence Graduate School Program (KAIST)) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea (MVII (No.2022. 0-00959, Few-shot Learning of Causal Inference in Vision and Language for Decision Making).
0990000055, Artificial
References
[1] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. European conference on computer vision, pages 776-794. Springer, 2020.
[2] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via pan-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-3742, 2018.
[3] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXivv2003.04297, 2020.
10

---

[4] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie. and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[5] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprini arXiv:2003.0429) 2020.
[6] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Pruceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9640 2021.
[7] Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence repre- sentations. arXiv preprint DoXiv:8803.02899 2018.
[8] John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.0365: 2020.
[9] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33: 33:12449-12460 2020.
[10] Aaqib Saced, David Grangier, and Neil Zeghidour. Contrastive learning of general-purpor audio representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3875-3879, IEEE, 2021.
[11] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang. Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[12] Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Active contrastive learning of audio-visual video representations. arXiv preprint arXiv:2009.09805, 2020.
[13] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12475-12486, 2021.
[14] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-6839) 2020.
[15] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812-5823,, 2020.
[16] Alfréd Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium Mathematical Statistics and Probability, Volume I: Contributions the Theory of Statistics, volume 4, pages 547-562. University of California Press, 1961.
[17] Jeremiah Birrell, Paul Dupuis, Markos A Katsoulakis, Lue Rey-Bellet, and Jie Wang. Varia- tional representations and neural network estimation of rényi divergences. SIAM Journal on Mathematics of Data Science, 1(4):1093-111) 2021.
[18] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
[19] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representa- tions. In Proceedings of the JEEF/CVF International Conference on Computer Vision, pages 9588 9597, 2021.
[20] Joshua Robinson, Ching- Mao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples, arXiv preprint 4rXiv:2010.04592, 2020.
11

---

[21] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel. and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33:21798-21809, 2020.
[22] Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. Boosting contrastive self-supervised learning with false negative cancellation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 2785-27955 2022.
[23] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint
2018.
[24] Jiaming Song and Stefano Ermon. Multi-label contrastive predictive coding, arXiv preprint ::Xi:20077.09852. 2020.
[25] Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon. Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 06670, 2018.
[26] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing, pages 22310-12320 PMLR, 2021.
[27] Yao-Hung Hubert Tsai, Shaojie Bai, Louis-Philippe Morency, and Ruslan Salakhutdinov. A note on connecting barlow twins with Segaatie-sample--e contrastive learning. arXiv preprint arXiv:2104113713 2021.
[28] Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. Advances in Neural Information Processing Systems, 32, 2019.
[29] Yao-Hung Hubert Tsai, Martin Q Ma, Muqiao Yang. Han Zhao, Louis-Philippe Morency, and Ruslan Salakhutdinov. Sepe--uue representation learning with relative predictive coding. arXiv preprint arXiv:2 1275, 2021.
[30] Ching- Ching-Yao Chuang, R Devon Hjelm, Xin Wang, Vibhav Vineet, Neel Joshi, Antonio Torralba, Stefanie Jegelka, and Yale Song. Robust contrastive learning against noisy views. arXiv preprint arXiv:2201.00 043009 2022.
[31] Tim Van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(7):3797-3820, 2014.
[32] Tom Minka et al. Divergence measures and message passing. Technical report, Citeseer, 2005.
[33] Yingzhen Li and Richard E Turner. Rényi divergence variational inference. Advances in neural information processing systems, 29, 2016.
[34] Liqun Chen, Chenyang Tao, Ruiyi Zhang, Ricardo Henao, and Lawrence Carin Duke. Vari- ational inference and model selection with generalized evidence bounds. In International conference on machine learning, pages 893-902 PMLR, 2018.
[35] Chenyang Tao, Liqun Chen, Ricardo Henao, Jianfeng Feng, and Lawrence Carin Duke. Chi- square generative adversarial network. In International conference on machine learning, pages 4887-4896. PMLR, 2018
[36] Lantao Yu, Jiaming Song, Yang Song, and Stefano Ermon. Pseudo-spherical contrastive divergence. Advances in Neural Information Processing Systems, 04-22348-22366 2021.
[37] Josip Djolonga, Mario Lucic, Marco Cuturi, Olivier Bachem, Olivier Bousquet, and Sylvain Gelly. Precision-recall curves using information divergence frontiers. In International Confer- ence on Artificial Intelligence and Statistics, pages 2550-2559, PMLR, 2020.
[38] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. International Conference on Machine Learning, pages 5171-5180. PMLR, 2019.
12

---

[39] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio. Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pages 531-540. PMLR, 2018.
[40] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, Communications on Pure and Applied Mathematics, 28 (1):1-47, 1975.
[41] Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual informa- tion estimators. arXiv preprint vrXiv:1910.06220 2019.
[42] David McAllester and Karl Stratos. Formal limitations on the measurement of mutual infor- mation. In International Conference on Artificial Intelligence and Statistics, pages 875-884. PMLR, 2020.
[43] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882, 2020.
[44] Art B Owen. Monte carlo theory, methods and examples. 2013.
[45] Qing Guo, Junya Chen, Dong Wang, Yuewei Yang, Xinwei Deng. Lawrence Carin, Fan Li, and Chenyang Tao. Tight mutual information estimation with contrastive fenchel-legendre optimization. arXiv preprint wxxv:2107.00131, 2021.
[46] Yang Song and Diederik P Kingma. How to train your energy -baged models. arXiv preprint arXiv:2101.0 032888, 2021.
[47] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 2020.
[48] Kuang-Huei Lee, Anurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. Advances in Neural Information Processing Systems, 34, 2021.
[49] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016
[50] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[51] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[52] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- moniming discriminative components with random forests. In European conference on computer vision, pages 446-461. Springer, 2014.
[53] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
[54] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[55] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Finc- grained visual classification of aircraft. arXiv preprint arXiv: 2013.
[56] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
13

---

[57] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3499 IEEE, 2010.
[58] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.paccal htmll.
[59] Xiao Wang and Guo-Jun Qi. Contrastive learning with stronger augmentations. arXiv preprint arXiv:2104.07713, 2021.
[60] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages -408-3505. IEEE, 2012.
[61] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
[62] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems, 31, 2018.
[63] Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers in plant science, 7:1419, 2016.
[64] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning. pages 9929 992999993. PMLR, 2020.
[65] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAl conference on artificial intelligence, volume 34, pages 13001-13008, 2020.
[66] Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles high-energy physics with deep learning. Nature communications, 5(1):1-9, 2014.
[67] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL ics. uci. edu/ml.
[68] Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. i-mix: A domain-agnostic strategy for contrastive representation learning. In ICLR, 2021.
[69] Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. arXiv preprint arXiv:2106.15144 2021.
[70] Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv preprint arXiv:2007.08663 2020.
[71] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00850 2018.
[72] Fan- Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv: 1908.100000 01000, 2019.
graph-
[73] Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. International Conference on Machine Learning, pages 12121-12132. PMIR, 2021.
[74] Ekin Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXi: 001.800,,9950,,0111
14

---

[75] Colorado J Reed, Sean Metzger, Aravind Srinivas. Trevor Darrell, and Kurt Keutzer. Selfaug- ment: Automatic augmentation policies for self sule-suuer learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 2674-2683, June 2021.
[76] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1-5. IEEE, 2015.
[77] Junya Chen, Zhe Gan, Xuan Li, Qing Guo, Liqun Chen, Shuyang Gao, Tagyoung Chung, Yi Xu, Belinda Zeng. Wenlian Lu, et al. Simpler, faster, stronger: Breaking the log-k curse on contrastive learners with flatnce. arXiv preprini arXiv:2107.01155 2021.
[78] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 14:5000-5011, 2021.
[79] XuanLong Nguyen, Martin Wainwright, and Michael Jordan. Estimating divergence func- tionals and the likelihood ratio by convex risk minimization, IEEE Transactions on Information Theory, 56(11):5847-5861, 2010.
[80] Lang Liu, Krishna Pillutla, Sean Welleck, Sewoong Oh, Yejin Choi, and Zaid Harchaoui. Divergence frontiers for generative models: Sample complexity, quantization effects, and frontier integrals. Advances in Neural Information Processing Systems, 34, 2021.
[81] Paul Rubenstein, Olivier Bousquet, Josip Djolonga, Carlos Riquelme, and lya O Tolstikhin. Practical and consistent estimation of f-divergences. Advances Neural Information Processing Systems, 32, 2019.
[82] Mathilde Caron, Hugo Touvron, Ishan Mísra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin, Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2 14294, 2021.
[83] Ross
Wightman. Pytorch image models. pytorch- image- models, 2019.
https://gtthab..co///uggguuttt
[84] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 7008 03888, 2017.
[85] Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE conference on computer vision and pattern recognition, pages 413-420. IEEE, 2009.
[86] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances Neural Information Processing Systems, 33: 13:1866
2020.
15


Reviews:
[{"review_id": "fPkxpzXNKkD", "reviewer": "Reviewer_dj8j", "summary": "This paper investigates a novel alternative formulation of contrastive learning that leverages the Renyi divergence. Motivated by the sensitivity of augmentation schemes for the learned representations, the author(s) argues using Renyi divergence can better manage hard augmentations. To overcome the variance issue in the naive formulation, a more stable version based on skewed statistics is advocated, with theoretical justifications. The author(s) showed that using the proposed RenyiRL can spared the need for additional hacks or extra computations to get decent performance. Strong empirical evidence supports the claims made in the paper.", "questions": "See my first comment in the Weakness section.", "limitations": "I don't have any concerns on limitations and potential negative societal impact.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n* *(Quality, Originality)* The application of Renyi contrastive learning for self-supervised learning is novel. Although I have seen other recent works also attempting to Renyi for contrastive learning, I believe this manuscript is the only one that well balances theoretical and practical aspects.\n*  *(Clarity, Significance)* This paper is written with exceptional clarity and I enjoyed reading it. It provides an extensive coverage on related work, and I am genuinely surprised to see some research threads not well known to the general contrastive learning community. Nice work. I added some additional thread which should make the literature review more comprehensive, and will hopefully inspire future research ideas.\n* *(Quality)* Strong experimental results, well curated experimental code. \n\n### Weakness\n\n* It is not clear to me how the variance in Theorem 3.2 turns into the variance in Theorem 3.1 in the limiting case $\\alpha=0$, could the author(s) elaborate on that?\n\n* Variance is not the only reason causing InfoNCE to perform sub-optimally. Discussions from [5-7] have presented diverse perspectives on how contrastive learning can be improved based numerical or optimization views, also backed up strong mathematical arguments. These works are complementary to the approach adopted here and should be discussed. \n\n### Misc\n\n* Theorem 3.1 is related to the result from [1] on the exponential variance issue for MI estimation, which should be  discussed.\n* The objective function for skewed $\\chi^2$ divergence presented in the [Tsai et al. 2021] (which this work is based on) is related to the spectrum contrastive learning objective [5], which is simpler and also demonstrated strong performance gains over the vanilla InfoNCE objective. The Renyi divergence and $\\chi^2$ divergence have also been successfully applied in the context of generative modeling [2-4], where the goal of minimizing $D_{\\alpha}(P_{XY}\\parallel P_XP_Y)$ or $D_{\\alpha}(P_X\\parallel Q_X)$ is shared. \n* Potential typos: $\\alpha$-CPC is actually proposed by [B Poole et al. ICML 2019], not [J Song, et al. NeurIPS 2020].\n\n[1] McAllester, D. and Stratos, K. Formal limitations on the measurement of mutual information. AISTATS 2020\n[2] Y Li, et al. Rényi Divergence Variational Inference. NeurIPS 2016\n[3] L Chen, et al. Variational inference and model selection with generalized evidence bounds. ICML 2018\n[4] C Tao. Chi-square Generative Adversarial Network. ICML 2018\n[5] J HaoChen, et al. Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss. NeurIPS 2021\n[6] Q Guo, et al. Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization. 2021\n[7] J Chen, et al. Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE. 2021\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "dS-UEKvmZ9", "reviewer": "Reviewer_6o5u", "summary": "This paper revisits the derivations of the InfoNCE objective from the original *Contrastive Predictive Coding* paper, replacing the KL divergence in the mutual mutual information with the Renyi divergence. As with the KL divergence, the Renyi divergence is known to enjoy a variational representation, which in both cases has a naive estimator that suffers from high variance. They then note a variance reduction idea, which I do not recognize from the contrastive literature, that shows that for D \\in \\{KL, Renyi\\}, replacing D(P|Q) with D(P | alpha*P + (1-alpha)*Q) for some small alpha yields an estimator with much improved variance. The paper proposes to use this variance reduction with the Renyi divergence as a plug-in replacement for InfoNCE as a training objective. The final theoretical component is to show that the resulting objective has gradients that naturally place high importance weights on hard negative and easy positives. The authors suggest that the easy positive in particular lend themselves to using more aggressive data augmentations in practice. Experiments cover visual, tabular, and graph structured data. ", "questions": "See previous box, particularly regarding experimental comparisons. ", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Congratulations to the authors on a well written, tidy piece of work. There are certainly some interesting ideas in this work, and I enjoyed reading this paper. First I outline some of the positive aspects I saw in this work, before turning to some weaker points and a few concerns I have, particularly with some of the experimental comparisons. \n\nIt is good to see that the thought process behind the mutual information —> InfoNCE derivation can be replicated for other divergences. Perhaps the same could be done with all sorts of other divergences, leading to a variety of different behaviors in objective functions. I also liked the presentation of the skew-divergence connection. The skew-divergence seems like it has useful low-variance, high-bias properties that could be useful in other contrastive learning contexts. \n\nOne of the premier benefits of RenyiCL is touted to be able to gracefully handle, and make use of, aggressive positive sampling thanks to its weighting biassing towards easy positives. As such the authors use what is said to be “harder data augmentations” for their method.  One of the ingredients constituting “harder augmentations” is multi-crop sampling à la SwAV. This raises a major concern of a lack of apples to apples vision comparisons, since the multi-crop method has, in all cases I have ever seen, always boosted performance, and increase the memory and runtime requirements. So the results reported in Tables {1,2,3} all compare a method using multi-crop to methods that do not (except for SwAV, which does and performs similarly to RenyiCL). It would be much fairer to remove the multi-crop sampling from RenyiCL. Table 4 has this number, but it is only for a 200 epoch run so again it is not really fair to compare. Finally, Table 5 compares   RenyiCL to other methods using the hard data augmentations. An important bright-spot here is that RenyiCL does see the biggest boost of the three methods, which is great to see. However RenyiCL is only compared to CPC and a variant of CPC; it remains unclear, for instance, how MoCo-v3 would fare in comparison using multi-crop. In all, it remains hard to say exactly how RenyiCL compares to other methods, besides to say that it is likely broadly comparable.\n\nThe inclusion of experiments on tabular and graph data is welcome, and I am glad to see promising results on other domains besides vision. \n\nI end with a much more philosophical point. The motivation for “why Renyi” divergence is mostly post hoc—instead of any intrinsic explanation for why Renyi may be a better estimator than KL, the authors first go through the process of deriving their low-variance estimator, and analyzing its gradients. They then find a number of natural, potentially desirable, attributes of the final objective such as easy positive/hard negative sample importance weights.  This is, again, very similar to process of producing the InfoNCE objective, starting from mutual information. It was convincingly argued by Tschannen & Djolonga et al. (2020), that it is the form of the InfoNCE loss itself, rather than any relation to mutual information, that explains its success. I have to say that I very much expect the same to hold for the-based Renyi objective presented in this work. From this point of view, it is worth putting to one side for a moment all of the scaffolding built on top of the idea of the Renyi divergence, and simply focus on the form of the final RenyiCL objective. As noted by the authors, this objective is of similar form to prior idea, but with an importance sampling re-weighting to emphasize hard negatives and easy positives. The main distinction here from prior works, which have considered hard negatives and hard positives using similar importance weightings, is to keep using hard negatives, but instead to reverse the focus to be on *easy positives*, and note that actually this may buy some benefits in terms of being able to use more aggressive data augmentations. This is an interesting proposal, and may well be at the core of the effectiveness of the proposed method; the Renyi divergence may mostly serve to obfuscate this.  \n\n\nTo conclude, while I have some gripes with the vision experimental setup, and suspect that the main benefit of RenyiCL of easy positive sampling could similarly be implemented in any other contrastive framework using importance weights, I am still broadly supportive of this work. To me this core recipe of easy positive weights + aggressive data augmentations (and its justification in Table 5) is already an interesting contribution, since how to produce good positive samples is perhaps the single most important question in contrastive learning. On a different note, the paper also offers other ideas, such as the skew-divergences, and their applicability to both KL and Renyi divergences seem interesting and may be of independent interest for producing low-variance contrastive learning objectives. \n\n---- \n\nPost rebuttal: increased score by one.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UXMX1mHF84", "reviewer": "Reviewer_UaDs", "summary": "This paper proposed the new robust contrastive learning scheme called RényiCL. By using the Rényi divergence instead of KL divergence as the objective function, the proposed method becomes robust against harder data augmentations. In order to keep the variance of the objective function small, this paper proposed the use of a variational lower bound for skewed Rényi divergence, which allows for training stabilization. The use of such skewed divergence is based on the observation that the original contrastive learning corresponds to the variational lower bound of the skewed KL divergence.", "questions": "## Implications of Theorem 3.2 for this study\nThe claim that motivates the technical contribution of this paper is Theorem 3.1, and the claim is roughly as follows: The estimated variance around the true solution of the Rényi divergence can be bounded from below by an exponential function on the difference between the KL divergence and the Rényi divergence. This observation indicates the variance of the naive objective could explode exponentially. Against this, Theorem 3.2 and additional theorem in the appendix argue that by introducing the skewed divergence instead of naive divergence, the estimated variance can be bounded from above by a constant. Although it attempts to show that exponential explosion of variance can be prevented because the constant appearing in the upper bound is independent of the number of samples, I believe this argument is inadequate for the following reasons\n\nTo begin with, the exponential function itself, which appears in the lower bound of the estimated variance, is also not a sample size dependent issue. The lower bound on the estimated variance in Theorem 3.1 depends essentially on $\\gamma$ and the magnitude of divergence between distributions $P$ and $Q$, not on the number of samples. In other words, even if the constants appearing in the upper bound are independent of sample size, if they have exponential dependence on these variables, the upper bound is loose and meaningless in current context. The authors should explicitly mention in the text the order these constants have.\n\n## Logical developments in the discussion of the gradient analysis for RényiCL\nI have several concerns on the logical developments in the discussion on the gradient analysis. This paper discussed the effect of using $\\alpha$-skewed Rényi divergence by investigating the gradient of the corresponding gradient. The brief logical flow is as follows:\n1. Consider the case of $\\alpha = 0$ for simplicity (the paper states it can be done without loss of generality)\n2. Take the gradients for both baseline objective (MLCPC) and proposed Rényi objective (RMLCPC)\n3. Construct the alternative objective functions of which gradient is equivalent to the original one\n4. Interpret the implication of each term appears in the alternative objective function\n\nI have two concerns about the above logical flow. The first one is a major and the second one is a minor concern.\n\n### The simplification of $\\alpha=0$ could mislead the discussion\nBy interpreting the positive first term for positive pairs and the negative second term for negative pairs, respectively, the authors state that the proposed method with $\\gamma>1$ achieves easy positive sampling & hard negative sampling simultaneously compared to existing methods. However, it is only in the special case where $\\alpha=0$ that terms for positive/negative pairs appear corresponding to exactly positive first/negative second terms, respectively. If we set α > 0 (which is done in practical experiment), the negative term which include positive samples will appears. For this reason, I believe that the claim that easy positive sampling occurs by using Rényi divergence with $\\gamma>1$ somewhat ignores the real mechanism.\n\n### The gradient of the alternative objective functions presented is not equivalent with original one\nThe paper states that the original objective function \"is equivalent\" to the alternative objective function in terms of gradient (e.g., p.6, l.184). However, this statement is not true **unless we introduce the stop-gradient operator for $q_{\\theta}$**. This logical development itself is not a problem as long as the stop-gradient is mentioned, so the authors should clearly state how each equations related.\n\n## Minor comments\n- p.5, l.174: \"$\\alpha$-skew $\\alpha$-skew Rényi divergence between ...\" -> \"$\\alpha$-skew Rényi divergence between ...\"", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work in conclusion section. In particular, the paper properly mentioned that it is difficult to identify what kind of data augmentation would be effective as a limitation of the proposed method. Yet, since the introduction of the paper motivates that the choice of data extension is the key in contrastive learning, there seems to be a slight discrepancy between the research objective and the actual proposed method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n- The flow of the text is clear, and the purpose, background, and proposed method of this study can be easily understood.\n- The proposed method is simple to implement and can be easily used as a plug-in to existing SSL methods.\n- Numerical experiments clearly demonstrate the effectiveness of the proposed method. The proposed method achieves comparable or slightly better performance than existing SSL methods when using normal data augmentation. In the situation with strong data augmentation, there was significant performance improvement over the existing methods, InfoMin and CLSA, for the two conditions of CUB200 and the Plant dataset.\n\n## Weakness\n- The significance of the theorem 3.2 is not adequately presented. This paper argues that skewed divergence allows the variance of the objective function to be bounded from above using some constants. However, this argument is not sufficiently convincing because there is not enough reference to the actual magnitude of the constant. More details are given in the Questions section\n- Discussion about the gradient analysis of RényiCL is not fully convincing. The interpretation of the effect of the change in divergence is discussed by comparing the gradient of the objective function of RényiCL with the baseline gradient, but the conclusion is not convincing due to its logical development. I will discuss in detail in the Question section.\n- Lack of ablation studies on robustness. Since the main concern of this paper is robustness to various data augmentation, the paper should directly demonstrate the effectiveness of the proposed method by comparing the performance by ablation of the contrastive objective under strong data augmentation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "COBMt-z0zPp", "reviewer": "Reviewer_B1D9", "summary": "The paper proposes to use the variational estimation of the (lower bound of the) skew renyi divergence for contrastive representation learning. The authors compare the reynyi divergence against traditional divergence such as KL divergence and skew KL divergence, and argue the proposed skew renyi divergence is beneficial for contrastive representation learning empirically.", "questions": "1. equation below line 86: g is not defined\n2. issues with using Donsker style variational estimation: The sup is over all possible functions. However in practice one has to assume some function class which is more restrictive. In the end the estimation is only a lower bound.\n3. line 212: can we have some details on how to estimate the constant Z using Monte-Carlo methods? how complex is that? usually this is a headache for a lot of problems\n\n", "limitations": "NA", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "While it is nice to have a new divergence and understand some properties of it, I am not quite convinced why the proposed skew Renyi divergence is better than the traditional ones, for example, skew KL divergence. Besides the empirical results, is there any explanation why skew Renyi divergence is preferred? \n\nAnother thing that bothers me is the paper entangles the analysis of of the skew Renyi divergence with data augmentation. I am curious if it is possible to decouple the two. One can do contrastive learning using the estimator of the skew Renyi divergence without data augmentation. Coupling the two makes the analysis harder to read. In particular, I don't feel the transition probability A(v|x) is well explained. Why do authors need to use the transition probability to define the positive and negative pairs? Can we define them just using the data distribution without the transition probability?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept