$f$-Mutual Information Contrastive Learning
Authors: Guojun Zhang
Yiwei Lu
Sun Sun
Hongyu Guo
Yaoliang Yu
Venue: ICLR 2022 Submitted (2022)
Keywords: contrastive learning
f-divergence
mutual information

Abstract: Self-supervised contrastive learning is an emerging field due to its power in providing good data representations. Such learning paradigm widely adopts the InfoNCE loss, which is closely connected with maximizing the mutual information. In this work, we propose the $f$-Mutual Information Contrastive Learning framework ($f$-MICL) , which directly maximizes the $f$-divergence-based generalization of mutual information. We theoretically prove that, under mild assumptions, our $f$-MICL naturally attains the alignment for positive pairs and the uniformity for data representations, the two main factors for the success of contrastive learning.  We further provide theoretical guidance on designing the similarity function and choosing the effective $f$-divergences for $f$-MICL. Using several benchmark tasks from both vision and natural text, we empirically verify that our novel method outperforms or performs on par with state-of-the-art strategies.

Under review as a conference paper at ICLR 2022
f -MUTUAL INFORMATION CONTRASTIVE LEARNING
Anonymous authors Paper under double-bling blind review
ABSTRACT
Self-supervised contrastive learning is an emerging field due to its power in pro- viding good data representations. Such learning paradigm widely adopts the In- foNCE loss, which is closely connected with maximizing the mutual information. In this work, we propose the f-Mutual Information Contrastive Learning frame- work (f-MICL), which directly maximizes the sriverrenncebbseee generalization of mutual information. We theoretically prove that, with mild assumptions, our f-MICL naturally attains the alignment for positive pairs and the uniformity for data representations, the two main factors for the success of contrastive learn- ing. We further provide theoretical guidance on designing the similarity function and choosing the effective f f-divergences for f-MICL. Using several benchmark tasks from both vision and natural language, empirically verify that our novel method outperforms or performs on par with statc-of-the strategies.
INTRODUCTION
Contrastive learning has attracted surge of attention recently due to its success in learning in- formative representation for image recognition, natural language understanding, and reinforcement learning (Chen et al.,, 2020) He et al. 2020) Logeswaran & Leel 2018: Sriiivas al., (020). Such learning paradigm is fully unsupervised by encouraging the contrastiveness between similar and dissimilar sample pairs. Specifically, the feature embeddings of similar sample pairs are expected to be close while those of dissimilar sample pairs are expected to be far apart. To attain this goal, a softmax cross- cross-entropyy loss, a.k.a. InfoNCE, has been widely used (Wu et al., 2018 van den Oord let al. 2018) Chen et al.) 2020: Hénaff et al.) 2020, He et al. (020), which aims to maximize the probability of picking a similar sample pair among batch of sample pairs.
InfoNCE can be interpreted as a lower bound of the mutual information (MI) between two views of data samples (van den Oord et al. 2018 Bachmaa et al. 2019 Tian et al. 2020a Tschannen et al. 2020). This explanation is consistent with the well-known "InfoMax principle [Linsker] 1988). Nevertheless, it has been shown that maximizing a tighter bound on the MI can result in worse representations (Tschannen et al.) 2020); and reducing the MI between views while only keeping task-relevant information can improve the downstream performance (Tian et al.] 2020b). These observations suggest that maximizing the MI may be insufficient in contrastive learning and thus better objective design is required.
To attain the aforementioned goal, we propose novel contrastive learning framework, coined as f-MICL. In a nutshell, leveraging the fact that MI can be formulated as the Kullback-Leibler (KL) divergence between the joint distribution and the product of the marginal distributions, we replace the KL divergence with the general f f-divergence family (Ali & Silvey, 1966; Csiszár 1967). Doing so, we obtain generalization of mutual information. called f f-mutual information (MMI. Csiszár 1967). Notably, by maximizing a lower bound of f-mutual information we naturally decompose the objective function into two terms, which correspond to the properties of the alignment and the uniformity. Such characterization has been revealed in WWng & Isola 20200. Theorem 1) for the InfoNCE loss. Compared with Wang & Isola| 2020). our result applies to a wide range of the f- divergence family such as KL, Jensen- Censen-Shannon (JS), Pearson x² and Vineze Le Cam. and it does not rely on the limit of an infinite number of dissimilar samples. This allows us to explore the space of fMI and improve the performance of InfoNCE-basee contrastive learning.
The similarity function is crucial for the evaluation of the contrastiveness of similar and dissimilar sample pairs. Commonly used similarity functions include the cosine similarity (Chen et al. 2020

---

Under review as a conference paper at ICLR 2022
Smage
by
evg over
posdltm pair
Gaussian
(At, sR)
signment
st
F-mutual Information
Dux
-"
aug
nogative pair
Gouzzian kerne
(#t kj)
BAY
uniformit
imago
NnT
m(ay)
Figure Network architecture of our proposed f-MICL. image,; the 5th image in the current batch; f: the function used in the f-mutual information (:2): feature embedding: I, 4,, l2: augmentation functions drawn from the same family T of augmentations; I": the derivative; f": the Fenchel conjugate. The symbol denotes function composition. The sum of the two terms gives the variational lower bound of f-mutual information. See equation I0 more details.
He al. 2020). the bilinear functions (van den Oord et al. 2020), and the neural network based scores (Hjelm et al. 2018). While most aforementioned sim- ilarity functions for contrastive learning are heuristic and pre-designed, in this work, we provide principled way to design the similarity function. By assuming that the joint feature distribution of two similar samples proportional to Gaussian kernel, we derive an optimal similarity function for practical use, which resembles the well-known radial basis functions (Powell 1987).
2018 Tian et al., 2020a Hénaff al.
Figure | gives a high-level summary of our f-MICL framework. Given a batch of samples (e.g., images) generate positive pairs via data augmentation and negative pairs using other augmented samples in the current batch. With the optimization of our f-mutual information objective the pos- itive pairs aligned with each other and the data representations uniformly distributed. Our contributions can be summarized as follows:
We propose a novel framework for contrastive learning (called f-MICL) by encouraging the contrastiveness of positive and negative pairs with general f-divergence family. With an assumption on the joint feature distribution we provide an optimal design for the similarity function with Gaussian kernels, which shows interesting connection between contrastive learning and kernel methods.
We characterize the properties of alignment and uniformiity for f-MICL and provide guid- ance on choosing proper f-divergences. Our f-MICL objective can be estimated with finite samples and we give the corresponding error bound.
Experimentally, our framework is better or par with popular baselines, by simply replac- ing, e.g., the InfoNCE loss function with our theoretically grounded f-MICL objectives.
Notations We assume dominating measure À (e.g. Lebesgue) is given and all other probability measures are represented as some density w.r.t. X. We denote (p(lg) as the f-divergence between two densities functions p and q. Given the joint density p(x, y). we denote p(x) f p(x, yy)d(yy) and p(y) = I ((x, ()) as the marginals. We use supp(-) denote the support distribution, and f to denote the Fenchel conjugate of convex function f. Every norm presented is Euclidean.
2 PRELIMINARIES
We provide some preliminaries for our framework the context of contrastive learning. Contrastive learning is popular unsupervised method for learning data representations. In contrastive learning.
2

---

Under review as a conference paper at ICLR 2022
Table 1: summary of common f f-divergences. KL: Kullback-Leibler JS: Jensen-Shannon; and SH: Squared Hellinger. For JS, we defnn g(u) = - +u+ log 2
1++ + w u. For the Pearson x². we take (t) = 1 - -2. The T divergence defined in Tsallis (1888) and we have a> 1 for f-divergences. The Vincze-Le Cam (VLC) divergence can be found in Le Cam (2012 p.47), which closely related to the Pearson x" and Hellinger divergences. For the Vineze-Le Cam divergence we require -3 < < and (1) -1 if -3.
Divergence
f(u)
(1)
f'(
f o f(()
KL
explt - 1) u llgu p(u)
log u +1 log 2 + log 1
IS
1og(2 ee")
log 1+m
Pearson
X²
(u - 1)2
f/
2(u 1)
1
SH
(Vu 1)²
1/2
u 1/2
1-t
Tsallis-a
a² /²/(a- 1)
((a 1)1/a)a/(a-1)
a
Vincze-Le Cam
(u-1)²
1 1-1-4v1-1
3 u+1
u+1
[u41122
we expect that similar sample pairs to be close to each other in the embedding space while uncorre- lated pairs to be far away. Denote Ppos as the distribution of positive pairs, i.e., two samples that have similar representations. Assume that this distribution is symmetric w.r.t. the two random variables, then the resultant two marginals both follow the data distribution Pdata (Wang & Isola 2020).
In this work we propose the f-mutual information framework for contrastive learning. First recall the f-mttual information (f-MI) between a pair of random variables X and Y:
Definition (f-mutual information, Csiszár 1967). Consider a pair of random variables (X,Y) with density function p(x, v). The f-mutual information T between X and Y is defined as
p(a,y) (p(z)pt)) ((((()))))) p(z)p(z) dd(x, v),
I,(X;Y) == Dy (p(z, v) v)ll(()))()) =
(1)
where f IR+ - R is (closed) corvex with [(1)= O, and recall that p(x)) and ply) are the marginal densities p(r,y) whereas X is a dominating measure (e.g. Lebesgue).
Common choices of ʃ can be found in Table and Table 4 (Appendix which we will discuss in more details }3 It well-known that f-mutual information non-negative and symmetric, and provided that ff is strictly convex, L (X; Y) 0 iff X and Y are independent (Ali & Silvey| 1966). When X and are of high dimension, is quite challenging to estimate the f-divergence directly. Instead, Nguyen al. (210)) derived variational method by maximizing the dual problem:
Ly(X; 2 sup YY) = (x [xx v)] (. (f((xx,))) (2)
TET
where (1) = SuP=ER, (xl - (xx)) is the (monotone) Fenchel conjugate of S. and is always monotonically increasing. Here T is a class of functions T supp(Paata) x supp(Panca) + dom f* Nguyen al. (010)) showed that the bound in eq. (2) is tight if there exists T E T such that
T (x, f (p(x,y) y)/ (p(x)p(())) for any (a, y) E supp(Paata) x supp(Paata), and in particular, if T comprises all (measurable) functions.
3 DESIGN OF f-MICL
Based on the f-MI introduced in 2 we propose a novel framework for contrastive learning. Fur- thermore, we characterize the properties of alignment and uniformity theoretically for general f- divergences. Following Chen et al. (2020), we design the streture of function T as follows:
T(x v) = (()(z))...)))))))))(()) g())), where [[]())]| = 1 for any sample x.
(4)
The function g produces a dddiimnnsional normalized feature encoding on the hypersphere gd-1 and
k is similarity function that measures the similarity between two embeddings g(x: and g(v). With the above interpretation, we can rewrite our objective of f-mutual information, equation 2 as:
sup YX;)) == ((x.))) ppa [k(g(z), (u)))
E( (( Peeas Pers (((K((()))
(5)
GEG
3

---

Under review as a conference paper at ICLR 2022
where 9 and IC are the function classes of the feature encoder g and the similarity function k. We can treat the first term as the similarity score between positive pairs in the feature space, and the second term the similarity score between two random samples, a.k.a. negative pairs, in the feature space. As I is increasing, maximizing f-II is equivalent to simultaneously maximizing the similarity between positive pairs and minimizing the similarity between negative pairs.
3.1 OPTIMIZED SIMILARITY FUNCTION AND IMPLEMENTATION
Let us now study how to search for the optimal similarity function k. To our best knowledge, there has been no theoretical study on the choice of similarity functions. Most existing contrastive learning methods (e.g. Chen et al. 2020 sai et al.] 2020; He et al. 2020) adopt pre-designed similarity function, such as the cosine similarity. For the case of notation, from now on we define x ;= g(x) and yd g(y). Suppose (x, y) ~ Ppoz. then we denote Poos as the distribution of (x yy), and Pasta as the marginal feature distribution of xa or y The corresponding density functions are written as
Py Py(x Pg (3)) and Py(x9, y²). We remind the reader of the following result:
Lemma 2 (..g..ll Nguyen et al. 2010 Lemma 1). Suppose f is differentiable, and the encoder function g is fixed. The similarity function
ka (x9.1)) = l (_Py(a Pg 3
(6)
Pg (2 p (19))
maximizes (X;) in eq. (5) long as it is contained in the function class K.
Equation 6 provides optimal similarity function, which nevertheless depends on the density func-
tions. Comparing equation 6 with equation [S] we realize that the optimal k:* in fact gives the f-MI
on the feature space, y (g(() gYY)) which is low bound of the original f-MI, (X; To use k* practically we make the following assumption on the joint density:
Assumption 3. The joint feature distribution is proportional to a radial basis function (RBF), i.e.,
Pg Pa(29, y/)
<0[1>> ) for real-valued function functioon sp
Murphy 2012). and the Gaussian kernel is perhaps the most well-known RBF. Throughout this work we mainly consider yp as a Gaussian kernel:
Radial basis functions are widely used in kernel methods (Powell 1987
"0|1>> - 0 Go - y0113) = exep || - y
2n
with μ constant left to be determined. Fixing yd then (. (0)y)) ) is known as the von Mises-Fisher Fisher distribution (von Mises| 1918 Fisher 1953] Bingham Mardia 1975). since 2 and y are unit vectors. With Assumption 3 on the joint density, the resultant marginal feature distribution playa uniform on the hypersphere gd-1, where d the dimension of the feature space (see Prop. 8 in App. BB). Additionally. for positive pairs the distance in the feature space, ||x:9 yal|, is more likely to be small. If the variance o² + 0, then the Gaussian kernel becomes the Dirac delta distribution, 5y9=yo- This requires that the two features xº and y" to be the same, which is desirable. In general, the radial basis function &o should be decreasing since a positive pair should be more likely to be adjacent in the feature space. For example, with ptt)= = 1 1/2 we obtain the cosine similarity.
Based on Assumption 3 we propose the following similarity function between pairs of features:
Theorem (Gaussian similarity). Under A Assumption wiith Gaussian kernels and the same sel- lings as Lemma 2 the optimal similarity function k* satisfies that for any x9 y E $d-1;
k" (2",, y8) ) J'(CGa(|) - - 12)))
(8)
where d is the feature dimension and C is an absolute constant.
For simplicity we will rewrite k=" .y) = I' G. 2
212) by absorbing the constant C
into Go, since we have left some flexibility in equation 7 Although Assumption 3 with Gaussian kernels may not always reflect the real feature distribution, in equation g even if it might not be optimal. In our experiments in {5 the Gaussian similarity equation equation&consiiseee consistently outperforms the default cosine similarity in contrastive learning.
still use the similarity function
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1: /mmutall information contrastive learning (f-MICL)
Input:
batch size N, function f, weighting parameter a, constant je (in Go), variance o², optimizer
for each batch {z; }/ do
forall [1, N
randomly sample two augmentation functions &, 12
V/ < (=x),, Zs < (2(x)
compute iy N Σ [f' Ga(I= - -f
MN-I Exss M S' Go(laa -231
update by taking a step to maximizing i, using the optimizer
Bringing the optimal k- in equation 8 into our objective equation 5 we have the following objective:
[f (1 va [[]] (x Spans [f f Go (1|=9 y01l
sup (ma))=ppa gEG
7 With similar sampling method of positive and negative pairs in Chen et al. (2020). given a batch N samples we can estimate the objective in equation | as:
where G. is defined in equation
(x;
1 N E fo Go v11
1 N(N-1)
E
Go 2²913)) (10)
where z; and y/ are two different kinds of data augmentation of the i-th sample, and x; and æ are different samples of the same kind of data augmentation.
With the objective in equation 10 we propose our algorithm for contrastive learning in Algorithm T Note that we treat and o² in our Gaussian kernel equation as hyperparameters. To balance the two terms our objective, we additionally include weighting parameter front of the second term. We can prove that rescaling the second term with the factor a is equivalent to changing the function f to another convex function fa (see Prop. in Appendix (A).
3.2 ALIGNMENT AND UNIFORMITY
Notably, if we choose the f-divergence to be the KL divergence, the objective in equation becomes:
||x y
sup SEG
202 ((
|||2 3 1| ((( (x) -aparaa Spene
exp
(1I)
2x2
which retrieves the objective the alignment and uniformity in Wang & Isola (2020). Specifically, in equation 11 the first expectation is the same Lalágn: and the second expectation is the same as Luniform In Wang & Isola (2020) (up to the logarithmic transformation). Therefore, under our Assumption we naturally decompose the objective for the KL into two: the first term leads the alignment of positive pairs while the second term results in the uniformity of data representations.
Compared to Theorem 1 of Wang & Isola 22200) our conclusion
based on the lower bound of f-MI instead of the InfoNCE loss, and does not rely on the limit of infinite negative samples. More importantly, besides the KL divergence we can extend our conclusion to various f-divergences.
Aligument Consider the objective in eq. (9) For the first term, since f is convex, the derivative f' is monotonic increasing. Therefore, in the ideal case, maximizing the first term would yield x0 y for all (x. y) ~ Ppos, i.e., similar sample pairs should have aligned representations.
Uniformity In Wang & Isola (2200) it is proved that for the KL divergence minimizing the second expectation in equation 11 gives the uniform distribution, if we have infinite samples. Here we discuss the general f-divergences with finite samples by considering equation 10, i.e., our empirical estimation of the objective. We list some common choices of f-divergences in Table and more detailed version can found in Table 4 Appendix A.I We have the following theorem:
Theorem 5 (uniformity). Suppose that the batch size N satisfies 2 < N d + with the dimension of the feature space. If the real function
h(t)= = Go(t) is strictly convex on [0,4].
(12)
5

---

Under review as a conference paper at ICLR 2022
then all minimizers of the second term of equation the following condition: the feature representations of all samples are distributed uniformly on the unit hypersphere $d All f-divergences in Table ] satisfy equation 12
10 i.e. Eiz, f o f Go(la - satisfy
Here by "distributed uniformly" we mean that the feature vectors form regular simplex (see Figure 2 and thus the distances be- tween all sample pairs are the same. It reflects our intuition that the feature embeddings are evenly distributed. Although minimizing the negative term gives uniformity, the positive term is also needed for aligning similar pairs, as we observe in our experiments $ Therefore, there is tradeoff between alignment and uniformity. In Theorem 3 we assumed N S d+1, with as the feature dimension. This assumption is always satisfied in our experiments in $5 For instance, for CIFAR-10 10 experiments we choose N = d = 512.
Figure 2: A regular simplex on a hypersphere.
Not all f-divergences lead to the property of uniformity. In Ap- pendix A.I we also listed some f-divergences that do not satisfy equation 12 and Theorem S such as the Reversed Kullback-Leibler (RKL) and the Neynman x² divergences. Experimentally, we found that these divergences generally result in feature collapse (i.e., all feature vectors are the same) and thus poor performance in downstream applications. Compared to the existing literature (Nowozin et al. 2016; AAcuna et al. 202I) where no suggestion is given for the choice of f-divergences theo- retically, with Theorem we provide guidance for choosing desirable f-divergences.
ESTIMATION OF fMMUTALL INFORMATION
In | we have provided the empirical estimation of our objective in equation 10 However, it remains a question whether our estimation of f-mutual information is consistent. In this part we give an upper bound for the estimation error. Recall that our f-MICI objective is ix(X; (X:Y) (equation and its empirical estimation is i (X;Y) (equation I0p, with T(x, y) = f' Go (1|23 -yy0 1)9))))
Theorem 6 (estimation error). Suppose that the function T is taken from function class T and define Tn as the function class of Txx,)) given some x E supp(Peata). Denote RK to be the Rademacher complexity w.E.t. the distribution P with N i...d. drawn samples. Then for any T E T, the estimation error (X;)) - -x(XxYY) is upper bounded with probability at least 1 - 6:
1
log 6/8
2D
(T) + 2p
E
Exnqusna Ai (Tx) +
Rann (Ta,)) + (rt (0r+2rg))
(13)
N
2(N - 1)'
with the constants rT f'(n) f've 22/0" 1.r== f'())= - fr l'mee- 2/0²).
Here the constant z is from our Gaussian kernel in equation 7 Rademacher complexity evaluates the richness of class of real-valued functions regarding probability distribution, and its formal definition can be found in Koltchinski (00T)) (see also Definition II in Appendix B
Note that the function class T depends on the class of the feature encoder and the f-divergence. The estimation error eq. (13) is composed of three parts: (1) the Rademacher complexity of the function class T. In general, if T is richer then its Rademacher complexity is also larger. (2) the expected Rademacher complexity of the one-side function class Tz and its empirical estimation; and (3) error term that decreases with the number of samples. Since the feature embeddings are usu- ally neural networks, we can use existing theory (Bartlett et al. 2019) to give more detailed bounds for the Rademacher complexities of T. Specifically, if the Vapnik Chervonenkis
(VC) dimension of T is finite, then our estimation error in eq. 13 goes to zero as N + co (Mohri et al. 2018).
Approximation and estimation tradeoff' In order to minimize the estimation error in equation 13 we should choose a simpler function class T to reduce the Rademacher complexities. However, T should also be rich enough so that equation can be satisfied, since our objective (X;Y) should approximate the f-mutual information I (X, Y') if we choose the optimal T. Therefore, there is a natural tradeoff between approximation and estimation errors when we change the complexity of T.
It is worth mentioning that our conclusion in Theorem ili theoretically non-trivial since our sample pairs are non-i.idd. although the individual samples are assumed to be i.i.d., the negative pairs are not independently drawn (e.g., (21, and (I1, which makes the derivation challenging.

---

Under review as a conference paper at ICLR 2022
5 EXPERIMENTS
We compare our framework with various frameworks on several popular vision and language datasets, and show the wide applicability of our method. In particular, our f-MICL gives state- of-the-art performance compared to popular baseline algorithms, such as SimCLR (van den Oord al. 2018: Chen et al., 2020), MoCo (He al., 2020), Uniformity (Wang & Isola 2020), and RPC (Tsai et al. 2020). Additional experimental results can be seen in Appendix
Specifically, our results confirm the following:
Our f-MICL encourages alignment between positive pairs, and encourages dissimilar sam- ple pairs to be equally far apart and thus leads to uniformity.
By replacing the cosine similarity with the Gaussian kernels, the performance is consis- tently better across a variety of f-divergences in our f-MICL framework.
5.1 EXPERIMENTAL SETTINGS
For vision tasks, we use SGD with momentum as our optimizer, and apply the cosine learning rate schedule (Loshchilov & Hutter 2017). Regarding the neural network architecture, we employ ResNets (He al. (020) the feature encoders. For the language dataset, we use the BERT models (Devlin et al. 2019) and Adam with weight decay as the optimizer (Loshchilov & Hutter 2018). To achieve fair comparison, keep the network architecture and optimization the same, while only changing the objective accordingly in our comparison. See Appendix C for more detailed settings.
5.2 COMPARISON WITH BENCHMARKS
We compare with several oftte-of-theart benchmarks in Table 2 Note that SimCLR and RPC use the cosine similarity while we use the proposed Gaussian similarity.
Vision task Our vision datasets include CIFAR-10, 10, CIFAR-100 100 (Krizhevsky et al. 2009). STL-10 10 (Coates et 201 TinylmageNet (Chrabaszez al,, 2017) and ImageNet (Deng al. 2009) for image classification. After learning a feature embedding, we evaluate the quality of representation using the test classification accuracies via a linear classifier. We observe from Table 2 that our proposed f-MICL consistently outperforms the benchmarks across all datasets. Specifically, we find that the JS divergence is superior in general, especially on the large datasets.
Language task To show the efficacy of our furff-Ma f-MICL framework, also conduct experiments natural language dataset: English Wikipedia (Gao et al. (021). We follow the experimental setting Gao et al.] (021), which applies SimCLR with BERT models (Devlin al. 2019 Liu et al., 2019). The application task is semantic textual similarity (STS, Aggire et al. 2013) and we report the averaged the Spearman's correlation in Table 2 for comparison. We can that our f-MICL performs better or on par with the benchmarks, especially for the KL divergence.
5.3 ABLATION STUDY
To develop thorough understanding of f-MICL we also perform the ablation study regarding the choice of the similarity function, the design of the f-divergences, and the batch size. More ablation study can be found in Appendix C
Cosine similarity vs. Gaussian In Table We compare the cosine and Gaussian similarities for different f-divergences on CIFAR-10. It can be seen that under our fMIIC framework the Gaus- sian similarity consistently outperforms the cosine similarity for various f-divergences. This agrees with our Theorem and equation g and also implies the validity of Assumption
Non-satisfying f-divergencess We have shown analytically that not all f-divergences are proper in our f-MICL framework (see the proof of Theorem 5 Appendix B}. For example, in our experiments on CIFAR-10 and 100, the RKL and Neyman x" do not perform well: the test classification accuracies on these two datasets are 10.00% and 1. 1.00%, respectively, which means that the classifier simply outputs the random guess. Moreover, f f-divergences such as the RKL, the features collapse to a constant. These observations emphasize the importance of choosing proper /diieergences and confirms our conclusion in Theorem 5
observe from Figure 3 that for the non-sftisfyig
7

---

Under review as a conference paper at ICLR 2022
Table 2: Test classification aceuracy (%) on the vision datasets. For the Wikipedia dataset we evaluate the semantic textual similarity (STS) via the Spearman's correlation. For ImageNet we train for 100 epochs with batch size 256 due to computation limit. See Appendix C for detailed experimental settings.
Baselines
f f-MICL
Dataset
MoCo
SimCLR
Uniformity
RPC
KI.
JS
Pearson
VLC
CIFAR-10
90.30
89.71
90.41 #0.26
90.39 20.25
90.61 10.47
89.66
4.028
89.35 +0.52
89.13 =0,33
CIFAR-100
62.77
62.75
62.31
62.66
63.00
20.41
63.11 0..33
61.69
20.57
61.1910.29
STL-10
83.6940.22
82.9740.32
84.44.0.19
82.41 10.14
85.3310.39
85.9420.17
82.6420.37
83,2710.72
TinyImageNet
35.72+0.17
30.5640.25
41.20:0.19
34.95+0.25
39.4640.20
42.98
38.45=0.54
38.65
ImageNet
58.59
57.66
59.12
56.11
58.91
61.11
55.33
54.26
Wikipedia
77.88 77.40 10.12 77.95
68.32 10.23
78.0210.18 13 76.76=0.09
77.59 10.12 55.07
Table 3: Comparison between the cosine similarity and our Gaussian similarity on CIFAR-I0 using the test classification accuracy (%).
Similarity
KL
JS
Pearson
SH
Tsallis
VLC
Cosine 88.95
87.06
87.79
87.06
88.55
10.00
Gaussian 89.34
89.12
89.44
88.13 13 89.18
89.15
Sensitivity to batch size We study the sensitivity to the batch size of our f-MICL framework on CIFAR-10. On the right panel of Figure we evaluate the classification accuracy by varying the batch size for different f-divergences and SimCLR. We can see that for all different batch sizes and with proper choice of f-divergences, our performance is always better than SimCLR. In other words, we require less negative samples to achieve the same performance.
5.4 UNIFORMITY TEST
To check the uniformity of feature vectors (Theorem 5) we plot the pairwise distance || - x of the feature representations within the same batch on CIFAR-10 and CIFAR-100. We compute the distances between the normalized features of every pair from random batch, and then sort the pairs with the increasing order. From Figure we can see that f f-MICL gives nearly uniform distances for dissimilar pairs (orange regions) on both datasets with various proper f /-divergences. In contrast, random initialized model gives less uniform distribution for dissimilar pairs. Besides, for f-MICI. observe small pairwise distances for similar pairs (green regions). On CIFAR-100 there are less similar pairs compared to CIFAR-10 as there are more classes
CIFAR-10
CIFAR-10
1.25
best accuracy (96)
1.00
distance
82
0.75
0.50
Fearzon
0.25
CLR
0.00
random
pairs
pairs
100
200
300
40D
503
batch
Figure 3: (left and middle) Distances between pairs of normalized features within batch. Green region: similar pairs. Orange region: dissimilar pairs. f-MICI. gives nearly uniform distances for dissimilar pairs for the f-divergences in Table For non-satisfying f-divergences such the RKL, the features collapse to a constant and thus the distances are zero. (right) The test classification accuracy v.s. the batch size after training 200 epochs for all algorithms.
8

---

Under review as a conference paper at ICLR 2022
6 RELATED WORKS
Contrastive learning Self-supervised contrastive learning learns representations by encouraging the contrastiveness between positive pairs and negative pairs. Recently it has been shown analyt- ically that improving the contrastiveness can benefit the downstream applications (Saunshi et al.) 2019 Tosh et al.) 202T). For popular contrastive learning methods such as Contrastive Predictive Coding (CPC) (van den Oord et al.) 2018), SimCLR (Chen et al.] 2020). and MoCo (He et al.) 2020), their loss functions can be interpreted as a lower bound of mutual information, which essentially the KI. divergence between the joint distribution Ppes and the product of margin distribu- tions Pasta X Pasts- Besides the KI. divergence, other statistical divergences or distances have been individually studied under the context of contrastive learning, e.g., the Wasserstein distance (Ozair
2019). Pearson x2 divergence (Tsai et al. 2021), and Jensen-Shannon divergence (Hjelm ctt al.
et al.
2018). Our paper differs from Hjelm al. 2018); sai al. (022T) in the following ways:
Instead of treating f-divergences estimators of mutual information, we directly consider f f-mutual information, which builds solid foundation for our objective.
Instead of using the default cosine similarity, we challenge this routine and construct the optimal design with the Gaussian kernel, which consistently performs better (Table
We show that optimizing the f-MICL objective of the contrastive learning yields alignment and uniformity for proper f -ddiveggences with finite samples.
Uniformity Our work is inspired by Wang & Isola (2020), which shows the alignment and uni- formity in contrastive learning. However, Wang & Isola (2020) focuses on the InfoNCE loss and the loss function Cuiifora they consider is somewhat ad hoc. In contrast, our f-MICI. objective is directly built from the f-mutual information theory. Another difference is that Wang & Isola (2020) proves uniformity in the limit of infinite samples, while our Theorem 5 Jonly needs finite samples.
f-divergences have been widely used in, e.g., generative models (Nowozin et al. 2016), variational inference (Wan et al. 22200), and domain adaptation (Acuna et al. (021). for measuring the discrep- ancy of two distributions. Compared to the existing work such as f-GAN (Nowozin et al., (016) which evaluates the similarity of two marginal distributions (i.e., the real and the modeled data dis- tributions), our f-MICI, objective measures the similarity between the joint distribution and the product of data distributions. Furthermore, while f-GAN aims to learn generative model and thus minimizes the L f-divergence, our f-MICL intends to encourage the contrastiveness and maximizes the f f-divergence. Finally, provide theoretical criterion for choosing proper f f-divergences.
Metric learning Our work is closely related to metric learning (Kaya & Bilge 2019 Suarez- Diaz et al. 2018), which aims learn a distance metric bringing similar objects closer and distancing dissimilar objects further. Distance metrics that are designed heuristically or independently of the problem may not lead to satisfactory performance in applications. In contrastive learning, pre- defined similarity metric, e.g., the cosine similarity (Chen et al., 2020: He et al. (020) or a bilinear function (van den Oord et al. [2018] Tian et al. [2020a] Hénaff al.J (020) is commonly employed to measure the similarity of sample pairs. Comparably, we have derived an optimal similarity function assuming that the joint positive distribution is proportional to a Gaussian kernel.
CONCLUSION AND FUTURE WORK
In this work we proposed f-MICL, a novel contrastive learning framework with a generalization of mutual information, called f-mutual information (f-M)). Our objective corresponds to the vari- ational bound of f-divergences. By making the assumption that the joint feature distribution proportional to a Gaussian kernel, we naturally characterized the properties of the alignment and uniformity. Our f-MI based objective can be well estimated with finite samples. Experimentally. showed the efficacy of f-MICL across wide array of datasets and the advantage of using the Gaussian similarity. Our results imply the following: (1) even though mutual information is widely used in contrastive learning, generalization to f-MI can bring better performance; (2) the cosine similarity, the de facto option in contrastive learning, can be replaced with more effective similar- ity functions such as the Gaussian similarity. As our work shows interesting connection between contrastive learning and kernel methods, it would be promising to explore the usage of more RBF kernels in contrastive learning.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Our novel fMMICL objectives are implemented in anonymous downloadable source code as in the supplementary. For theoretical results, clear explanations of any assumptions and complete proofs are included in Appendix| A and Appendix [ For detailed experimental settings and complete ex- perimental results, the complete descriptions can found Appendix C
REFERENCES
David Acuna, Guojun Zhang, Mare Law, and Sanja Fidler. f addomain-avversaarrr learning: Theory and algorithms. In International Conference on Machine Learning. 2021.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez Gonzalez-Agirre, and Weiwei Guo. *SEM 2013 shared task: Semantic textual similarity. In Second joint conference on lexical and computational seman- tics (*SEM), volume 1: proceedings of the Main conference and the shared task: semantic textual similarity, pp. 32 43, 2013.
Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficients of divergence of one distri- bution from another. Journal of the Royal Statistical Society: Series (Methodological), 28(1) (1)): 131-142, 1966.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximiz- ing mutual information across views. Advances in Neural Information Processing Systems, 32: 15535-15545, 2019.
Peter L. Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research, 20( 20(1):2255-201,, 2019.
MS Bingham and KV Mardia. Maximum likelihood characterization of the von Mises distribution. In A Modern Course Statistical Distributions in Scientific Work, pp. 387-398. Springer, 1975.
Sergiy V Borodachov, Douglas P Hardin, and Edward Saff. Discrete energy on rectifiable sets. Springer, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, pp. 1597-1607, 2020. URL httpp://groceedingggg www.:/eeee...in..............ee.
1t19/chen20..00 html
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of ImageNet as an alternative to the CIFAR datasets. arXiv preprint arXiv: 170.v01777.088/9, 2017.
Adam Coates, Andrew Ng. and Honglak Lee, An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Imre Csiszár. Information-type measures of difference of probability distributions and indirect ob- servation. studia scientiarum Mathematicarum Hungarica, 2:229-33 1:229-318, 1967.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 148-255..00 IEEE, 2009.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components
estimation. arXiv preprint arXiv: 14 10.85 16, 2014.
10

---

Under review as a conference paper at ICLR 2022
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference on Learning Representations, 2017.
Ronald Aylmer Fisher. Dispersion on sphere. Proceedings of the Royal Society of London, Series A. Mathematical and Physical Sciences, 217(1130):295 305, 1953.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. In International Conference Machine Learning, pp. 41 4182-4192. PMLR, 2020.
R Devon Hjelm, Alex Fedorov, Samuel
Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference Learning Representations, 2018.
Mahmut Kaya and Hasan Şakir Bilge. Deep metric learning: A survey. Symmetry, 11(9):1066, 2019.
Diederik P Kingma and Prafulla Dhariwal. Glow: generative flow with invertible 1x convolutions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 10236-10245, 2018
Vladimir Koltchinskii. Rademacher penalties and structural risk minimization. IEEE Transactions on Information Theory, 17(5):192--1114 2001.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009. Technical report.
Lucien Le Cam. Asymptotic methods in statistical decision theory. Springer Science & Business Media, 2012.
Ralph Linsker. Self-organization in perceptual network. Computer, 21(3):105-117, 1988.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pre- training approach. arXiv preprint 4rXiv:190..11692 2019.
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representa- tions. In International Conference on Learning Representations. 2018.
llya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. Interna-
tional Conference on Learning Representations, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2018.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning- MIT press, 2018
Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):54475581. 2010.

---

Under review as a conference paper at ICLR 2022
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: plers using variational divergence minimization. Proceedings of the 30th International Con- ference Neural Information Processing Systems, pp.
Training generative neural sam-
2016.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron van den Oord, Sergey Levine, and Pierre Ser- manet. Wasserstein dependency measure for representation learning. In Advances in Neural Information Proceeding Systems, pp. 15604-15614, 2019.
M. J. D. Powell. Radial Basis Functions for Multivariable Interpolation: A Review, pp. 143-167.
Clarendon Press, USA, 1987. ISBN 0198536127.
Ralph Rockafellar. Characterization of the subdifferentials of convex functions. Pacific Journal of Mathematics, 17(3):97--100 1966.
Igal Sason and Sergio Verdú. f-divergence inequalities. IEEE Transactions on Information Theory, 62(11):5973-6006 2016.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In Proceedings of the 36th International Conference Machine Learning, pp.
2019. URL URLhttp:
//proceedings mlr. www.ppres/vvv. http//aunssshiiie html
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. CURL: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, 2020.
Juan Luis Suárez-Díaz Salvador García, and Francisco Herrera. A tutorial on distance metric learn- ing: Mathematical foundations, algorithms, experimental analysis, prospects and challenges (with appendices on mathematical background and detailed algorithms explanation). arXiv preprint arXiv:1812.0550 0544444. 2018
Yonglong Tian, Dilip Krishman, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16. pp. 776-794. Springer, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. Advances in Neural Information Processing Sys- tems 33 (NeurIPS 2020), 2020b. URL /
/pwp://iinoceeeeeggecoe.c.cs wwwrineuus.... httpiippp.cpppppppeeeeeee -ab. Abstract
html
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory, pp. 1179-120 1206. PMLR, 2021.
Yao-Hung Hubert Tsai, Han Zhao, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Neural methods for point-wise dependency estimation. arXiv preprint arXiv:2006.05555 055553,, 2020.
Yao- Hung Hubert Tsai, Martin Q Ma, Muqiao Yang, Han Zhao, Louis-Philippe Morency, and Rus- lan Salakhutdinov. Self-supervised representation learning with relative predictive coding. In International Conference on Learning Representations, 2021.
Constantino Tsallis. Possible generalization of Boltzmann- Gibbs statistics. Journal of statistical physics, 52(1):479-487, 1988.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference Learning Representations, 2020.
Jean- Ban-tBttitte Hiriart Urruty and Claude Lemaréchal. Convex analysis and minimization algorithmss. Springer-Veriie Verlag, 1993.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint
2018.
12

---

Under review as a conference paper at ICLR 2022
Richard von Mises. Uber die "ganzzahligkeit" der atomgewicht und verwandte fragen. Phys. Z., 19: 490-500, 1918.
Neng Wan, Dapeng Li, and Naira Hovakimyan. f-divergence variational inference. Advances in Neural Information Processing Systems, 33, 2020.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929 9939. PMLR, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3733-3742, 2018.
Guojun Zhang, Han Zhao, Yaoliang Yu, and Pascal Poupart. Quantifying and improving transfer- ability in domain generalization, arXiv preprint vrXiv2206600332, 2021.
13

---

Under review as a conference paper at ICLR 2022
Table 4: summary of common f-divergences. KL: Kullback-eeibler, JS: Jensen-Shannon; SH: Squared Hellinger. For JS, we define s(u) = -" + 1) log 1 + log u. For Pearson x², we take (t) = -1 iff S -2. For Jeffrey, W = W + W and (-) the Lambert-W product log for f-divergences. We ignore constant addition -1/(a - because it does not change the optimization problem, The Vincze-Le Le Cam divergence can be found in (p.47, [e Cam 2012) which is closely related to x² and Hellinger divergences. For the Vincze-Le Cam divergence we require -3 < t < and (*) = -1
function. The Tsallis-a divergence is defined in Tsallis (988) and we have
if -3.
Divergence
(()
(()
f'(u)
f" o f
KL
u log
exp(t - 1)
log + -1/u
Reverse KL
log
-1 -1-log( -t)
log u
JS
((u)
1o ee)
log 2 log Ifu
log 2
log(1 + u)
Pearson x²
(u 1)²
2/4+1
2(u 1)
u²
SH
(Vu - 1)²
-
u 1/2
1/2
Neyman x²
(1-w)²
2 2V1
u 2
2 2u
Jeffrey
(u 1) log
W(c'-t) +
M + log
W (c /u) log
Tsallis
v 1(a
((a 1)t/a)o/lo_1)
""
Vineze-Le Cam
(u-1)²
1 1-1- 1V1
1u++3)
u++1
(++1)
#+1
A ADDITIONAL THEORETICAL RESULTS
In this appendix provide additional theoretical results, including additional f-divergences and the theory for weighting parameters.
A.1 ADDITIONAL DDIVEREESSSS
We expand Table I and give more examples of f-divergences in Table As we will see the proof of Theorem 5 Table gives special class of f-divergences that guarantees uniformity. A detailed description of f-divergences can be found in e.g. Sason & Verdú (2016).
A.2 WEIGHTING PARAMETERS
In Algorithm we added a weighting parameter a to balance the alignment and uniformity. We prove that even after adding this parameter we still maximizing the f-mutual information, al- though with respect to a different /.
Proposition 7 (weighting parameter). Given a > 0 and a closed convex function f R. + R such that f(1) = O, define Sa a dom f TR. with fa(x) a f (x/a) af(1/a) for any x dom f. Then T,., is still a valid f-matual information (see Definition| Besides, by replacing f with fo in equation Twe have the following optimization problem:
sup (e [5' -,1"(=9"11111//)) (x. Peaes [s (G=[(119~''113)/))
gEG
where G.(ll2 - 1)) -
exp
||z -|=1-1"11"* 20t
is the Gaussian kernel.
Note that dom f means the scalar multiplication of set which is applied element-wisely. Ac- cording to Definition [] fa is also a valid f-divergence. This proposition tells us that rescaling the another convex function fo-
equivalent to changing the function The transformation from / to o: (x/a) is also known as right scalar multiplication (e.g. Chapter X. Urruty & Lemaréchal 1993). Let us now move on our proof:
second term with factor
Proof. By definition we know that fa. is convex and closed with fa (1) = 0. and thus Iy, is valid Moreover, we have S/ (x) f'(x/() for any
f-mutual information according to Definition [
14

---

Under review as a conference paper at ICLR 2022
dom f and
(1)
sup xt fa(x)
zédom fo
sup xt +
#Eadom f
sup
(x/a) (al) - af(2///) +a/(1/a))
s/o/dom
a sup (t +af(0/
s/aSdom
(1) + a1 /a);
(A.1)
where in the last line we used the definition of (t). Plugging fa and fa into equation 9 yields the desired result.
A.3 UNIFORM DISTRIBUTIONS
In the following we prove that under Assumption the marginal feature distribution plata uniform. Proposition Under Assumption 3 the marginal feature distribution Paata is uniform on the hy- persphere Sd-1, with d the dimension of the feature space.
Proof. Under Assumption 3 we have:
(-1 ||20 1012
Pa( yo) Co exp
(A.2)
20²
where Co is a normalizing constant. Integrating 1 we have the marginal distribution for xº:
Pg(x {
||
y0
Co exp
dy
(A.3)
|g
20²
It suffices to show that Pg = Py (x²) for any x9. x2 Sd-1, such that:
Suppose Q is the orthogonal matrix
Q- x
(A.4)
Such a matrix Q always exists and constructing Q is not difficult. For example, assume that {0f, xg]) span plane with orthonormal basis c1, 62, and
x = cos 01 E1 + sin 01 -2.
x. cos 02 61 + sin 02 62
(A.5)
then Q can take the following form:
cos((), 01)
sin (in -13 cos(02 - 01)
[e]
(ei e2]
(A.6)
sin 01)
C2
such that Q is orthogonal and satisfies
Qu? .... Hence we have:
||
y
Pg (r(x =
On exp
dy
8
20
QQ x/
1
Co Coexp
dy
J88-1
2
|QQ
x2
Q
29112
Co Coexp
d(Q
184-1
20
|1
29
Cu
d
84-1
20
Py (x2),
(A.7)
where the second line we used equation A.4 in the third line we made the transformation y Q 29 with 2 a unit vector; in the fourth line used the fact that applying orthogonal matrix does not change the norm and that the corresponding Jacobian determinant is one.
In fact, the proof above can be generalized from the Gaussian kernel to any radial basis functions, by replacing the Gaussian kernel with (([[ 1 and repeating the same proof. Here şp can be any function such that the integral fga-1 ( y ydd is finite.
15

---

Under review as a conference paper at ICLR 2022
B PROOFS
Lemma (e.g. Nguyen et al. 2010 Lemma 1). Suppose f differentiable, and the encoder function g is fixed. The similarity function
(2 Pg (2:9) (p (1)))
k- (2) f
(6)
maximizes YY) in eq. 5 long as it is contained in the function class K.
Proof. From Definition
I
we are computing the following supremum:
/
Po 1²) Pg( p () k(x (2x))))) - ff o k((" .x) dpáata @ Paata
sup g.k
(B.1)
Suppose k is unconstrained and we fix g. The optimal solution should satisfy:
(r (x9 yo) Pg (x Py (19)
E (8'")
(B.2)
almost surely for (x, v) Paata Paata From (3.11) Rockafellar 19666) this is equivalent to:
Po (0(x9199)
k² (= v9) E 0f
(B.3)
Po (2(19)p1 (19)
f is differentiable, then for any u E dom f. 0f(u) = (f' is singleton.
Theorem 4 (Gaussian similarity). Under Assumption 3 with Gaussian kernels and the same set- tings as Lemma 2 the optimal similarity function K- satisfies that for any x9 y E sd-1;
ke (a9, y8) = f'(CG=(1)2 y8 12)).
(8)
where d the feature dimension and C is an absolute constant.
Proof. Simply combine Proposition with Lemma 2
Theorem 5 (uniformity). Suppose that the batch size N satisfies 2 < N S d + 1, with the dimension of the feature space. If the real function
h(t) = o f Go (t) is strictly convex on [0, 4),
(12)
then all minimizers of the second term of equation the following condition: the feature representations of all samples are distributed uniformly on the unit hypersphere , All f f-divergences in Table 1 ssatisfy equation 12
10 i.e., Eit, f o f' Go(lz 29913), satisfy
Proof. From the definition of it clear that h is decreasing since f and I' are both monotonically increasing white G is decreasing. Using we rewrite the second term of equation 10 as
min
Σ h(11 - x311))
(B.4)
When E d + 1]. there exists a neat characterization of the minimizers, see e.g. Borodachov et all. 2019 Theorem 2.4.1). We include the proof below for completeness.
16

---

Under review as a conference paper at ICLR 2022
Apply Jensen's ' inequality, we have:
1
h(1k;;
=x,13]] h
20 xy11
N2
N2
1
h
21 -2,1?
N
h
Σ
2014 + 235)
N2
N
= h
2
1
N
2 h (2),
(B.5)
where in the first line used Jensen's inequality; in the third line used ||æ;||| ||= 1 for any j E [A]] in the last line we note that E æ; 2 and h is decreasing function. When h is strictly convex and decreasing, it is in fact strictly decreasing, and hence the two inequalities above can be attained
T
N Σ 33 = 0, and 1 x, x,,1² c for all + j.
(B.6)
namely that (x1,..., INj form a regular simplex with its center at the origin. We remark that when h is merely convex, points forming a centered regular simplex may form strict subset of the minimizers.
To the necessity of N d+ let us note that
1,
i=
1. Tj
(B.7)
i
N-I*
since
2N
2
Σ 1x. 2, 2 2N N(N - 1)c = c N =2
(B.8)
N 1
Performing simple Gaussian climination note that the matrix T X has rank N where
X (i....,,,NN E _dx Therefore, we must have N 1
Lastly, need to show when h is (strictly) convex function, which may not always be true depending on the f-divergences. We give the following characterization (we ignore the constants and 2 in equation 7 as they do not affect convexity):
h strictly convex: 7KL (f) = e- hts((t) = loog(1 + c-t) 1 llog 2, hpearson (t) = c 2t 1,
hsn (l) = -t/2 1, /otsaliaa (1) e- lIVI.C = I+e
h convex but strictly convex: HRKL( - (RKL stands for Reversed Kullback- Leibler, see Appendix A.1);
h concave: NNyemman (t) = 2e (Neyman stands for Neyman x², see Appendix A.1
Only for the last case we do not have the guarantee that the minimizing configurations could form regular simplex. For RKL,, in fact, any configuration that centers at the origin suffices since h is linear function.
Li
Theorem (estimation error). Suppose that the function is taken from a function class T and define Te the function class T T(x,)) given some E supp(Paata). Denote Be be the
17

---

Under review as a conference paper at ICLR 2022
Rademacher complexity W.E. the distribution P with N i.i.d drawn samples. Then for any T E T, the estimation error j - is upper bounded with probability at least - ô:
1 N
log 6/6
N (T) + 2p
E Euwpente Ratt (Tz) +
W (Ts.) + +rr++ + +2r
(13)
2(N - 1)
with the constants rT = f'(p) f' (ue-22/0))
2/0 ), ),1
f()) f'me 2
Proof. Our proof uses the following notion of Rademacher complexity:
Definition 11 (Rademacher complexity. Koltchinskii (2001)). Let F be family of functions from subset of Euclidean space W to [a, b) and (WI..... Wm) a fixed sample of size m with elements in W. The empirical Rademacher complexity of F w.r.t. the sample set S is defined as:
Rs(F) = Eog
Σ orf(we)) (EF m
sup
1
(B.9)
where o; are independent uniform random variables taking values 1-1,+1), Let P be the un- derlying distribution of samples. The Rademacher complexity Rmm (F) of function class F is the expectation over sample sels:
R Em [F(FF).
(B.10)
With the definition of Rademacher complexity we have the following lemma:
Lemma 12. Let F be a family of functions from subset of Euclidean space W to (a, b) with b a. Then for any < < 1 with probability at least 1 - 8, the following holds for any f F:
1 f(:)) (F) + (b adv
1og(2/8)
E [f(w)]
(B.11)
m
2m
with w; i.i.d examples any distribution Pa-
Proof. Our proof follows from Lemma 11 of Zhang et (2021). For any function F we can construct f (f ab/(b- - a) such that f W [0, 1]. The rest follows from the linearity of Rademacher complexity.
Let us first recall our definition equation T(x = k(g(z). From the definitions of ix(;; Y) and (X;;)) we write the following:
(x)(); -xxx)))
B(2,)) [(x, 2)] Eucpanc Eyr (T(xx,)))
T(ai, 23)
Σ
(T((,,)))))
N(N - 1)
($)
N
((z,e) [(xx2)]
T(x, 23)
+
N
+ Exrpaaa Eysp [f
If (T(x,
Σ
(T
N(N 1) itj
(B.12)
where we used the triangle inequality. With Lemma equation B. as:
12
we can upper bound the first term of
log(2/0)
29
Pres (T) + (br ar)
w.p. 1 - 8,
(B.13)
2.V
18

---

Under review as a conference paper at ICLR 2022
where (ar, by] is the range of T. Now upper bound the second term of equation B.12 Note that we cannot apply Lemma 12 to equation B. B. directly since (2,, Ty)'s are not i..d. samples. For instance, the pairs (I1, Ø2) and (21, F2) are not independent since they share the same 2 However, all ay 's are i.i.d. sampled. We can upper bound the second term of equation
B. B.12 3.122ase as:
E Eerpese Pasts of (T(2.)) N(N - 1)
Σ ((( itj
Exne Byy-reace 5 (T(,, v)) Eurepess
(((2)
j=1
E Pgat
(T(((I,3)) N(N 11
(T(())))))) 5
E Paaaee E S (T((x,)) - Rarapaan N j=1
(T(((,,,)))) +
N
1 N(N - 1n1
E Eggupara
(T((2,z)))
Σ
(T(EE))))
(B.14)
N
The first term of equation B. can be upper bounded as:
E Ey--pmn J (T((x,))) Σ (T((x,))))
< 2T 2uz Altnr (f* Tz) + (by -ay) 1og(2/8)
2N
log(2/5)
< 2L 2 Beuopen R Ma (Tx) + (bg 1ay-ajp anv
(B.
2N
with probability at least 1 1-6, where used Lemma 12 in the first line and Talagrand's 's lemma (e.g., Lemma 5.7 of Mohri et al. (2018) in the second line. We denote Ly as the Lipschitz constant of f* oT. Similarly, and [ay, by] as the range of fo..
upper bound the second term of eq. (B.14)
L
log(2/0) - 1)'
N (Tay) + (by anyy
(B.16)
2(N-1)
j=1
with probability at least
Absolute constants Let compute the constants ay, by: or, br, Ly. First note that our feature embeddings are normalized. For any 19 3d gd-1 we have:
0 < ||ا" - 1 < 21 + | (12) = 4.
(B.17)
From the expression of Gaussian similarity equation and the monotonicity of I' and f", we have:
by f'()), f'uee-2/07 2/00 ), f f' ('))) f (pe (/-=)
(B.18)
For simplicity we will write 7T = by - aT and r by - aj. We can also compute the Lipschitz constant of f*, which is bounded by the norm of the gradient:
L < sup
(f*Y a f Go(l - ) = sup 29 E84411
Go(1 - 1- = μt, (B.19)
x² y0
where we used the equality (f*) o f' = id Rockafellar 1966). with id the identity function.
Combining equation B.13 the proof.
equation B.15 and equation B.16 we can use the union bound to finish
19

---

Under review as a conference paper at ICLR 2022
C ADDITIONAL EXPERIMENTAL RESULTS
We present additional experiment details in this appendix, to further support our experiments the main paper.
C.1 IMPLEMENTATION DETAILSS
In this paper, we follow the implementations in SimCLR (https:/
com/sthalles/
SimCLR). For vision tasks, we use ResNet (He et al. 2016) as the feature encoder, and we adopt the similar procedure of SimCLR for sampling. For the language dataset, we follow the exact experimental setting of Gao al. (2021) and only change the objective. Our experimental settings are detailed below:
Hardware and package: We train GPU cluster with NVIDIA T4 and P100. The plat- form we use pytorch. Specifically, the pairwise summation can be easily implemented from pytorch.
using torch.nn. www.tnfnnnnnnnn......ooo
Datasets: the datasets we consider include CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009) STL-10 (Coates al.] (011), TinyImageNet (Chrabaszez et al. 2017). ImageNet (eng et al. 2009) and English Wikipedia (Gao et al. 2021).
Augmentation method: For each sample in a dataset we create a sample pair, a.k.a. posi- tive pair, using two different augmentation functions. For image samples, choose the augmentation functions to be the standard ones in contrastive learning, e.g., in Chen et al.) (2020) and He et al. (2020). The augmentation
composition of random flipping, crop- ping, color jittering and gray scaling. For text samples, following the augmentation method Gao et al. 2021) we use dropout masks.
Neural architecture: For CIFAR-10 and CIFAR-100 use ResNet-18 (He et al. 2016); for STL-10, TinyImageNet and ImageNet we use ResNet-50 (He et al. 2016); for the Wikipedia dataset we use BERT base (Devlin et 2019).
Batch size and embedding dimension: for experiments in CIFAR-10 and CIFAR-100 choose batch size 512, and for STL-I0 we choose batch size 64 to accommodate one GPU training. Finally, for TinylmageNet and ImageNet, choose batch size 256. For all the vision datasets, we choose the embedding dimension to be 512. Regarding the language dataset, the batch size is 64 with the feature dimension 768. In all of these cases, assumption N S d d in Theorem 5 satisfied.
Hyperparameters: in all our experiments we fix the constant factor Jr in practice the weight parameter often needs to be large (e.g. in the Wikipedia dataset), which requires moderate tuning. Note that we also implement RPC (Tsai et al., (021) in our paper. For all the datasets, we follow Tsai et al. (2021) and choose the relative parameters a 1.0,3 3 = 0.005 and = 1.0 for all datasets.
We find that
Optimizer and learning rate scheduler: For vision tasks, we use SGD with momentum for optimization and the cosine learning rate scheduler (Loshchilov & Hutter 2017). For the natural language task, we use Adam with weight decay (Loshchilov & Hutter 2018 and the linear decay scheduler.
Evaluation metric:
for vision tasks, we use --nearest-neiihhor (kNN) and linear evaluation to evaluate the performance, based on the learned embeddings. For the NLP task, we use the Spearman's 's correlation to evaluate the averaged semantic textual similarity score (Gao et al. 202T).
Baseline methods: for the four baseline methods, we follow the implementations in:
MoCo: http://s//gghhaa.ccoo//aaaaaaaaa
SimCLR: ht
Uniformity:
RPC: https://gthhaacco//arrtaaaa////
coding
For fair comparison we use the experimental settings in Table for all the baseline methods, which might differ from the original settings.
20

---

Under review as a conference paper at ICLR 2022
Table 5: Detailed experimental settings. arch: the neural network architecture used. N: batch
size; d: the dimension of the feature representation; 1r: learning rate; the constant factor in μr 1/(20²) and follow from Algorithm| 1
epoch: the number epochs we run; ks the number of nearest neighbors in kNN evaluation.
Dataset
arch
N
d
1r (2a")
epoch
A
CIFAR-10
ResNet-18 18
512
512
0.1
800
200
CIFAR-100
ResNct-18
18
512
512
0.1
40
1000
200
STL-10
ResNet-50
64
512
0.1
1
40
800
200
TinyImageNet
ResNet-50
256
512
0.1
40
800
200
ImageNet
ResNet-50
256
512
n/a
Wikipedia
BERT
64
768
3e-5
20
409600
n/a
Table 5 gives common choices of hyperparameterss for different datasets. Note that may need to further finetune a and for different f-divergences. See our supplementary code for more details.
Table 6: Ablation study weighting parameter for KL and JS divergences CIFAR-1O. We compare test classification accuracies (%) for different choices of using kNN evaluation.
0.1
10
20
30
40
50
KL 13.16
77.60 83.53
83.77 81.39
84.19
82.77
JS 8.84 73.31 81.39
83.21
83.49
84.06
82.61
Table 7: Ablation study on weighting parameter a for KL and Pearson x² divergences on Wikipedia. We compare the semantic textual similarity (STS) via the Spearman's correlation for different choices of weighting parameter a.
10
10a
10a
10a
10
409600
10
KL
67.52 70.47 72.43
75.12
76.90
77.78
78.02
77.78
Pearson 64.58 67.78 71.58
74.03
74.95
74.40
77.59
76.47
C.2 ADDITIONAL ABLATION STUDY WEIGHTING PARAMETER
We provide additional ablation study the weighting parameter We perform experiments using a vision dataset (CIFAR- 10) and a language dataset (Wikipedia). For CIFAR-I0, we vary a from 0.1 to 50 for KL. and JS divergences and run for 200 epochs. Table justifies choice of in T where the downstream test classification accuracy indicates the optimal performance when choosing a 40. For the Wikipedia dataset, we observe that much bigger is desirable for maximum performance. We vary a from to 106 for KL and Pearson x² divergences and run for epoch, there is large number of samples (109) in the language dataset. Table 7 justifies choice of a in Table 5] where the best performance is reached at a 409600. Such an a is found by starting from a 100 and doubling iteratively.
C.3 ADDITIONAL EXPERIMENTS
Our final experiments show that f-MICL is stable in terms of training and the variation of perfor- mance is well controlled.
Training stability We depict the training loss curves of different divergences on CIFAR-10 in Figure This figure shows that our methods exhibit stable training dynamics with fast convergence.
kNN evaluation and additional ffaioonnnlf--iiie f-divergences We show more detailed results of Table in Table 8 including experiments using k-nearest neighbour (kNN) evaluation. Additionally, we have added experiments other f-divergences such Squared Hellinger and Tsallis-a divergences.
21

---

Under review as a conference paper at ICLR 2022
CIFAR-10
KL
Pearson
VLC
10
SH
Tsallis
105s
25
50
75
100
125
150
175
200
epoch
Figure 4: The training loss curves various f-divergences on CIFAR-10 with 200 epochs.
SimCLR
KL-divergence
6700
6700
6650
6600
log density
6600
6500
6550
6400
6450
0.000
0.002
0,004
0.006
0.008
0.000
0.001
0.002
0.003
0.004
0.005
squared distance
squared distance
Figure 5: Experiment for verifying Assumption 3 We draw the relation between the squared dis- tances || - 1 and the averaged log,2, with RealNVP. The features are learned by different algorithms trained on CIFAR-IO. (left) SimCLR; (right) f-MICL with the KL divergence.
22

---

Under review as a conference paper at ICLR 2022
Table 8: Test classification accuracy (%) on the vision datasets. For the Wikipedia dataset evaluate the semantic textual similarity (STS) via the Spearman's correlation. For each method, we take three separate runs, and show the mean and stand derivation.
Baselines
f-MICL
Evaluation
Dataset
MoCo SimCLR Uniformity
RPC
KL
JS
Pearson
SH
Tsallis
VLC
90.30
89.71
90.41
90.39
90.61
89.66
89.35
89.52
89.15
89.13
CIFAR-10
10.19
10.37
10.26
10.25
10.47
10.28
10.52
10.25
10.42
10.33
62.77
62.75
62.51
62.66
63.00
63.11
61.69
61.47
60.55
61.19
Linear
CIFAR-I00
+0.17
10.44
100.6
10.39
10.44
10.33
1..57
10.23
10.44
1.0.9
83.69
82.97
84.44
82.41
85.33
85.94
82.64
82.80
84.79
83.27
STL-10
+.0.2
+.0.32
+0.19
#.1.4
+0.39
:00.77
+0.37
+00.27
4-0.31
+-0.22
35.72
30.56
41.20
34.95
39.46
42.98
38.45
40.83
32.99
38.65
TinylmageNet
+0.17
+-0.8
+0.19
+0.25
+0.20
+0.18
+0.54
+0067
+-0.9
+0.45
ImageNet
58.59
57.66
59.12
56.11
58.91
61.11
55.33
52.37
53.11
54.26
88.70
84.92
89.42
84.21
89.34
89.12
89.44
88.13
89.18
89.15
CIFAR-I0
+0.22
+0.39
+0.18
0.224
+0.57
+0.38
+0.60
+0.18
+0.62
+0.23
59.21
53.47
61.07
50.02
59.12
61.36
58.68
57.66
58.12
59.17
KNN
CIFAR-100
10.15
10.47
10.38
10.34
10.37
10.35
10.57
10.33
10.45
10.28
78.77
74.34
79.57
73.27
79.99
80.45
76.64
78.31
76.11
79.34
STL-10
+0.25
+0.14
40.52
+0.40
10.47
+0.19
#0.26
+0.33
40.24
40.62
36.22
29.60
37.44
24.25
36.17
38.20
35.14
35.56
33.11
35.21
TinylmageNet
+0.20
+0.39
+0.27
+0.35
+0.29
+0.26
+0.63
+0077
+0.52
+0.33
77.88
77.40
77.95
68.32
78.02
76.76
77.59
73.60
72.68
55.07
STS
Wikipedia
+0.15
+0.12
+0.08
+0.23
+0.13
+0.09
#0.12
+0.10
+0.09
+0.13
Verification of Assumption 3 that the joint feature distribution is a Gaussian kernel. However, is a valid assumption? In this experiment try show empirical evidence that this assumption approximately holds in practice. Recall that Assumption 3 says that the joint feature distribution of positive pairs is:
Throughout paper made assumption (Assumption 3
Pg (x9, yd) x exp (-1ee² ||2
y
(C.1)
20
the RBF kernel Gaussian. In order to estimate the joint density of positive pairs, we use nor- malizing flows (Dinh al., 2017), which is popular method for the density estimation. Popular normalizing flow models include RealNVP (Dinh et al. 2017), NICE (Dinh et al. 2014) and Glow (Kingma & Dhariwal 2018 Equation C.I equivalent to the following:
||20
10
log (x yy)
const,
(C.2)
202
and thus it suffices to show that the log likelihood is linear w.r.t the distances between each positive pair. In Figure 5 we plot the relation between log estimated by RealNVFT and the squared distances 2
112 The representations are learned by SimCLR, and f-MICL with the KL divergence on the CIFAR-I0 dataset. To alleviate the estimation error in the flow model, we divide the distances into small intervals and compute the average log likelihood within each interval. From this figure can see that the log likelihood roughly linear w.r.t. the squared distance, and thus verifying our Assumption
'Code available wtt.thhhhhhaaa...t.oo..../o//ooo
23


Reviews:
[{"review_id": "gnQ2RHJ97wD", "reviewer": "Reviewer_Q5hx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n\nThe theoretical results show that the generalization from mutual information to f-mutual information is not merely a mathematical game but also has its own merits. Specifically:\n\n1. The natural connection with alignment and uniformity is a neat observation. \n\n2. The finite-sample result on uniformity provides a counterpart of the existing results in the infinite-sample case, which is interesting because it demonstrates that not all f-divergences are equal.\n\n3. The result of the estimation error is able to handle the dependent negative pairs, using a non-trivial application of Rademacher complexity theory.\n\nWeakness:\n\n1. The results in Table 9 in the Appendix seem not to demonstrate a clear advantage of the new framework compared with existing baselines. As such, I am worried if the results in the main body, such as  Table 2, are convincing enough to illustrate the advantage of the new framework.\n\n2. For f-divergences that satisfy the condition (12) in Theorem 5, can you provide some guidelines on how to choose between them?\n\n\n", "summary_of_the_paper": "This paper proposes the f-mutual information objective for contrastive learning, which generalizes the existing mutual information framework. Theoretically, it makes a connection with alignment and uniformity in Wang & Isola (2020) when the joint feature distribution can be represented by a radial basis function, and derives its estimation error. Numerically, it demonstrates the efficacy of f-mutual information with Gaussian similarity.", "summary_of_the_review": "The paper is well-motivated and clearly written. I enjoy reading the paper, and I believe the results are technically sound as I do not find major flaws in the proof. Although mathematically the new framework simply replaces mutual information with f-mutual information, I think it produces several interesting observations and draws connections with the literature. Due to computational efficiency, the authors are not able to provide confidence intervals of the objective in their numerical experiments, which raises the concerns whether the new framework is indeed better in some applications.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QyjrjL2UjaL", "reviewer": "Reviewer_qEh5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses an interesting problem, and demonstrates useful empirical results. While the paper has promise, I cannot recommend acceptance of the current state of the paper due to sometimes incoherent results, unclear statement of contributions, and some unjustified assumptions. Of course, I am happy to change my score if I mis-understood any parts of the paper or if the issues are improved in the rebuttal period. \n\nTheoretical: I find the assumptions quite odd. If the paper already assumes that the true distribution over features is a Gaussian, then naturally Gaussian kernels are the right choice. Therefore, I do not really see why the theorem provides interesting insight. Such a strong assumption might require experimental / theoretical justification, which is not provided in the paper. \n\nIn section 4 the paper bounds the estimation error of the mutual information. However, I do not see how this is relevant to the goal of representation learning. If accurate estimation of the lower bound matters for representation learning, I think the paper needs to demonstrate this either theoretically or empirically. \n\nThese are some claims in the intro that in my opinion are not adequately supported in the paper, and hence might benefit from being downplayed. One seeming contradiction is that the intro first claims that a tighter bound on the MI does not improve the representations, which is one of the motivations for this work. However, much of the theoretical results focus on how to obtain tighter bounds on the f-MI. This can be confusing as the paper did not justify why tighter bound to Shannon MI leads to worse representation learning performance, but it is desirable to obtain tighter bounds to the f-MI (which after all includes Shannon MI as a special case)? Also, the introduction claims that the newly proposed similarity function is ‘optimal for practical use’. However, I do not think such claims are adequately supported by theoretical or empirical results. \n\nEmpirical: Given that the theoretical results are not fully convincing, the experiments become crucial. The paper would be a great contribution if it can convincingly demonstrate the empirical advantage of Gaussian similarity compared to cosine similarity, and the benefit of using another f-MI instead of Shannon MI. The experiments provide some evidence that some choices of f-MI (JS) is better on CIFAR, STL and TinyImagnet compared to baselines such as SimCLR. In addition, the experiments provide some evidence that using Gaussian similarity is better than using cosine similarity. \n\nThe experiments have some short-comings. First, there is no experiment on the original setup of e.g. SimCLR which was applied to Imagenet. Second, on the STS dataset, compared to the cited paper, the reported baseline performance seems worse? While this is not a fatal issue, maybe the original paper used additional techniques that were difficult to reproduce, it is certainly something that needs to be discussed. Finally, there is not insufficient statistical rigor such as error bars, as some of the improvements are small enough to be possible to come from random fluctuations. \n\nThat being said, I do think that the paper can become much stronger with a clearer statement of the actual contribution. In my opinion, the most interesting contribution would be proposing that f-MI lead to a new class of new objectives, that while not necessarily better than Shannon-MI, at least provides more options. In addition, experiments can be used to study which options are the best empirically. \n\n--------------\n\nPost rebuttal: Thank you for the detailed rebuttal. After the rebuttal, I feel better about the experiments. There is a lot of subtlety with representation learning evaluation, so I believe the paper should include all the details of evaluation, including justification on why e.g. SBERT-base is chosen as a baseline (which is fine as the baseline does not necessarily have to be state-of-the-art with all the bells and whistles, but it should be clearly discussed and disclosed since the original paper contains methods with significantly better performance). The same goes for the newly added Imagenet experiments.\n\nIn addition, while the authors understandably had insufficient time to revise the paper during the rebuttal period to address the writing issues, I believe the claims of the paper should be more precise and well-supported by the actual experiments. A mediocre contribution that is well supported is better science and probably contributes to research more than amazing contributions that are weakly supported or questionable. \n\nGiven the detailed rebuttal and the new experiments, I will raise my score. ", "summary_of_the_paper": "This paper makes two contributions to contrastive representation learning: 1. Using the more general f-mutual information rather than using Shannon mutual information for contrastive learning 2. Experimental results to compare the possible options given the new design freedom. ", "summary_of_the_review": "The paper proposes an interesting family of new contrastive learning objectives, and some empirical comparison of the objectives. However, I think the paper needs a more clear and accurate statement of the contribution, better justified theoretical results, and better empirical evaluations. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nwa9sWdAyd-", "reviewer": "Reviewer_fZBZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1) The paper is well written and easy to understand.\n2) The experiment is quite comprehensive, and the results look positive.\n\nCons:\n1) The paper has limited novelty compared to the InfoNCE framework.\n2) The advantages/disadvantages of the proposed method compared to InfoNCE are not well analyzed in the paper.\n3) The standard deviation of results is not provided, results of the baseline do not match those in the original paper.", "summary_of_the_paper": "The paper proposes another variant of Contrastive Learning that uses a different lower bound of the mutual information based on f-divergence instead of InfoNCE.", "summary_of_the_review": "1) Novelty:\n- The main limitation of the paper is that the proposed idea is not novel. The variational lower bound of mutual information (MI) based on f-divergence was well discussed in previous works (e.g., [1, 2, 3]) so it is straightforward if we replace the InfoNCE bound with the new bound.\n\n2) About the method:\n- The main advantage of InfoNCE compared to other lower bounds of MI is its low variance which allows stable learning to achieve good representations. The low variance comes from the fact that InfoNCE uses multiple negative samples per positive sample and normalizes over them. On the downside, InfoNCE is biased and requires a large batch size. SimCLR has this drawback but it was overcome by MoCo. Therefore, I expect the authors to theoretically and empirically analyze the advantages and disadvantages of their proposed method compared to InfoNCE in terms of bias-variance trade-off. They should also compare their method with MoCo, which has been shown to perform better than SimCLR.\n\n3) About experiments:\n- The experiment results do not show the clear advantage of the new method compared to the existing baselines based on InfoNCE when using a linear classifier. The difference in performance in Table 3 is small which can be attributed to randomness in running. I think the authors should at least provide the standard deviation of their results to highlight the significance.\n\n- The results in Table 3 in the paper are still very far from the results reported by SimCLR (Table 8 in the SimCLR paper). For example, SimCLR reports 95.3% accuracy on CIFAR10 with a linear classifier but in Table 3, the authors only show 89.7% accuracy. I think the authors should check their experimental settings again to make their results more comparable.\n\n[1] On Variational Bounds of Mutual Information, Poole et al., ICML 2019\n\n[2] Mutual Information Neural Estimation, Beghazi et al., ICML 2018\n\n[3] Learning Deep Representations by Mutual Information Estimation and Maximization, Hjelm et al., ICLR 2019\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IUUlDbmhJgp", "reviewer": "Reviewer_HSE4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is novel to substitute the cosine similarity with $f$-divergence in contrastive learning, and the experiments validate that this is a very effective modification.\n\nTheorem 5 is an important condition to help us choose the correct function which can avoid feature collapse.\n\nThe notation in Figure 1 is not easy to understand. I suggest the authors think of another way to express $-f^*f'$.\n\nThe description of the algorithm is not well written, according to Figure1, there is aug1, aug2, and aug. I assume aug is equal to either aug1 or aug2. And how to randomly sample augmentation functions t1 and t2?", "summary_of_the_paper": "This paper proposed to combine contrastive learning with $f$-divergence, which naturally attains the alignment for positive pairs and the uniformity for data representations. The authors provided some theoretical results on choosing the correct function f and the upper bound of the estimation error. Finally, the authors carried out extensive experimental results to validate the effectiveness of the method.", "summary_of_the_review": "Theory 5 provides a way to tell whether an f divergence could be a good candidate, which attracts me a lot. But at the same time, the description of the main algorithm is not complete, some details are missing and the notations in the figure is not well-explained.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most of the reviewers have concerns that the experimental results don’t show stronger enough improvements over baselines and that the theoretical contribution of the paper is not completely clear. These concerns make the paper a borderline paper for NeurIPS. Some reviewers have pointed out problematic or unsupported claims in the paper. With these in mind, I encourage the authors to revise the paper with more clarity and address the reviewers' comments on the exposition of the paper.
Decision: Reject