$f$-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning
Authors: Yiwei Lu
Guojun Zhang
Sun Sun
Hongyu Guo
Yaoliang Yu
Venue: Accepted by TMLR ()

Abstract: In self-supervised contrastive learning, a widely-adopted objective function is InfoNCE, which uses the heuristic cosine similarity for the representation comparison, and is closely related to maximizing the Kullback-Leibler (KL)-based mutual information. In this paper, we aim at answering two intriguing questions: (1) Can we go beyond the KL-based objective? (2) Besides the popular cosine similarity, can we design a better similarity function? We provide answers to both questions by generalizing the KL-based mutual information to the $f$-Mutual Information in Contrastive Learning ($f$-MICL) using the $f$-divergences. To answer the first question, we provide a wide range of $f$-MICL objectives which share the nice properties of InfoNCE (e.g., alignment and uniformity), and meanwhile result in similar or even superior performance. For the second question, assuming that the joint feature distribution is proportional to the Gaussian kernel, we derive an $f$-Gaussian similarity with better interpretability and empirical performance. Finally, we identify close relationships between the  $f$-MICL objective and several popular InfoNCE-based objectives. Using benchmark tasks from both vision and natural language, we empirically evaluate $f$-MICL with different $f$-divergences on various architectures (SimCLR, MoCo, and MoCo v3) and datasets. We observe that $f$-MICL generally outperforms the benchmarks and the best-performing $f$-divergence is task and dataset dependent.

Published in Transactions on Machine Learning Research (10/2023)
f-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning
Yiwei Lu School of Computer Science University of Waterloo Vector Institute
giwei.lu@uwaterloo..co
Guojun Zhang
muojjun.zhanng@huuuess.ccoo
Huowei Noah ss Ark Lab
Sun Sun
sun.sun
School of Computer Science University of Waterloo
National Research Council Canada
Hongyu Guo
hongyu. Suottowa.ca
National Research Council Canada Universily of Otlawa
Yaoliang Yu
yroliang-yuhhuuateen
School of Computer Science University of Waterloo Vector Institute
Reviewed OpenReview: https: // openreview. net/ forum? id= ZDO3VUZmRo
Abstract
In self-supervised contrastive learning, widely-adopted objective function is InfoNCE, which uses the heuristic cosine similarity for the representation comparison, and is closely related to maximizing the Kullback Leibler (KL)-based mutual information. In this paper, we aim al answering two intriguing quest.ions: (1) Can we go beyond the KT-bbasd objec- tive? (2) Besides the popular cosine similarity, can we design better similarity function? We provide answers to both questions by generalizing the KL-based mutual information to the f-Mutual Information in Contrastive Learning (J-MICL) using the f-divergences. To answer the first question, we provide wide range of f-MICL objectives which share the nice properties of InfoNCE (e.g., aligmment and uniformity), and meanwhile result in similar or even superior performance. For the second question, assuming that the joint feature distribution is proportional to the Gaussian kernel, we derive an f-Gaussian sim- ilarity with better interpretabllity and empirical performance. Finally, identify close relationships between the f-MICL objective and several popular InfoNCE-based objectives. Using benchmark tasks from both vision and natural language, empirically evaluate f- MICL with different f-divergences on various architectures (SimCLR, MoCo, and MoCo v3) and datasets. We observe that f-MICL generally outperforms the benchmarks and the best- pest-performing f-divergence task and dataset dependent.
"Equa contribution

---

Published in Transactions on Machine Learning Research (10/2023)
1 Introduction
Recent advances in self-super learning aim at learning similar representations from different augmented views of the same data sample. However, naively implementing this idea would easily make representations converge to some trivial constant (i.c., feature collapse) in practice. To address this problem, researchers propose new algorithms either from the model architecture perspective or the training objeclive perspective The former method (e.g., Grill et al. (2020); Chen & Нe (2021); Zhang et al. (2021)) applies techniques such stop gradient or predictor module to create asymmetry in networks, while the latter method encourages the contrastiveness between similar (positive) and dissimilar (negative) sample pairs through the objective design.
In this paper, we intend to deepen our understanding of contrastive learning by generalizing the current objective design. To achieve supervipe contrastive learning, existing objectives are proposed from dif- ferent. perspectives such as the mutual information (e.g., InfoNCE. (Wu et al., 2018; van den Oord et al., 2018; Chen et al., 2020; Hénaff et al., 2020; He al., 2020) ), the information redundancy (e.g., Barlow Twins (Zbontar et al., 2021)), and the regularization (e.g., VICReg (Bardes et al., 2021)). In particular, the InfoNCE objective is widely used, which aims to maximize the probability of picking a similar sample pair among a batch of sample pairs, and can be interpreted as a lower bound of the mutual information (MI) between two views of samples (van den Oord el al., 2018; Bachman et al., 2019; Tian et al., 2020a; Tschannen et al., 2020). This is consistent with the well-known "InfoMax principle' (Linsker, 1988). To measure the similarity between sample pairs, cosine similarity is usually adopted.
To attain the aforementioned goals of better understanding and generalizing cantrastive learning, we here focus on the addely- InfoNCE objective, and aim at two questions regarding it:
(1) MI is essentially the Bulback--Leiblr (KL) divergence between the joint distribution and the product of the marginal distributions. Is this KL-based objeclive oplimal! If not, can we go beyond the KL-based objective!
(2) Besides the commonly used cosine similarity for measuring the distance between samples, can we provide a better similority function with a theoretical basis?
To answer the above two questions, we generalize the KL-based mutual information to the broader f- divergence family (Ali & Silvey, 1966; Csiszár, 1967), and propose the benchmark of fa information in contrastive learning (f-MICL). By searching through wide range of f-divergences, we observe that the KL divergonce is not always the best, and several other f-divergenees in fact show similar or even superior performance in practice.
For the second question, while it is challenging to provide an answer based on the InfoNCE objective, it is possible to derive proper similarity function under the f-MICL framework. By assuming that the joint feature distribution is proportional to the popularly-adopted Gaussian kernel, we propose novel f G-Gussiian similarity function that enjoys better empirical performance.
Finally, we show the generalization of f-MICL by drawing connections between the f-MICL objective and several popular InfoNCE basedd objectives (e.g., SimCLR et al., 2020), MoCo(He et al., 2020), and Alignment and Uniformity (AU) (Wang Isola, 2020)). We identify that those objectives are closely related to f-MICL: Aligument and Uniformity (AU) (Wang & Isola, 2020) can be treated as a special case, and SimCLR (Chen et al., 2020) and MoCo (He et al., 2020) are upper bounds for transformed f-MICL. These results provide different angle to better understand InfoNCE. Moreover, show both theoretically and empirically that nice propertics of InfoNCE (o.g., alignment and uniformity (Wang & Isola, 2020)) can be naturally extended to J-MICL.
We summarize our main contributions as follows:
Motivated by InfoNCE, we propose a general framework for contrastive learning by extending the MT to the general f-MI, which provides a wide range of objective choices.
2

---

Published in Transactions on Machine Learning Research (10/2023)
Instead of using heuristic similarity functions, provide novel similarity function, called f- Gaussian simalarily, based on the convex conjugate and an assumption on the joint feature distri- bution.
We identify close relationships between our f-MICL objective and several InfoNCE-based contrastive learning objectives.
Empirically, we show that f-MICL achieves notable improvement over benchmarks on various datasets, and the best-performing f-divergence depends on the specific task and dataset. In ad- dition, our proposed f- f-Guussian similarity consistently outperforms the cosine similarity.
2 MMMutal Information
To provide answers to the above two questions regarding the InfoNCE objective, we first extend the KL-based mutual information to the more general f-mutual information. The definition of the f-mutual information (f-MI) as follows:
Definition 1 (f-mutual information, Csiszár 1967). Consider a pair of random variables (X, Y) with joint density function p(a, y) and marginal densities p(z) and p(y). The f f-mutual information I between X and Y is defined
1xx;Y) = f plo p(z)p(v) dx(x,
p(a)p(t)
(1)
where f :.R - R is (closed) convex with f(1) O, and is a dominating measure (e.g., Lebesgue).
Note that the f-MI is essentially the f-divergence between the joint distribution and the product of marginal distributions. It is well-known that the f-MI is non-negative and symmetric. Moreover, provided that f is strictly convex, (Y; = 0 iff X and Y are independent. (Ali & Silvey, 1966).
Since is challenging provide an accurate estimation of the f-divergences in high dimensions, Nguyen et al. (2010) used the convex conjugate lower bound for the f-divergencos. With this result we lower bound IY(X;Y) as follows:
(x;Y)) 2 sup E s(X,Y) oEF (X.)))((..))
f o (X)
(2)
(X.Y)~((s)p())
where p(r, y) denotes the joint density, p(z)p(y) stands for the product of marginal densities, and the symbol o denotes function composition. The function (t) == supaeR, (at (xx)) is known as the convex conjugate* of f and is monotonically increasing, and s(.) belongs to F, a class of functions on (x, v) that we can parameterize. Using results in Nguyen et al. (2010), one can show that eq. (2) is equal to (X; Y) if there exists s, F such that for any (r,y) E supp(p(z)) x supp(p(y)), where supp(-) denotes the support of distribution, we have:
s. (x = 8' p(c,,))
p(x)p(y))
In other words, plugging the optimal s. (z. y) into (2) we obtain equality. In Table we list common choices of f-divergences, their conjugates, and the derivatives. We also include the composition f" f' for later purposes (see Theorem below).
3 f-MICL
With the introduction of the more general f-MI we now proceed to the design of the objective and similarity function. Then we will analyze the property of the proposed framework and compare it with some existing InfoNCE-based benchmarks.
1 Mooe precisely, this is the convex conjugate since restrict domain of f R4-
3

---

Published in Transactions on Machine Learning Research (10/2023)
Table Common choices f-divergences. KL: Kullback Leibler; JS: Jensen Shannon: SH: Squared
Hellinger; VLC: Vineze-Le Cam (Le Cam, 2012). For JS, we define g(u) (u + 1) 14 2
+ u logu. u.
The Tsallis-a divergence is defined in Tsallis (1988). See Appendix A.1 for more details.
Divergence
f(u)
(*)
f'(u)
f f'(u)
KL
u alogu ((u)
exp(t - 1)
log
JS
-log(2 et)
log 2 + log v Ita
log 1+
Pearson
X²
(u - 1)2
2/4+t
2(u
2
SH
(Vū 1)²
1/2
1/2
1-1
Tsaliss-o
(0-1) a-1
a-1 11
4
a-I
VLC
1-t 4 -
(++
3.1 /-MICL objective
Contrastive learning is a popular self-supervised method for representation learning. In contrastive learning, we expect similar sample pairs to be close to each other in the embedding space, while dissimilar pairs to be far apart. Based on the f-MI introduced in $ 2, we propose a general framework for contrastive learning, coined as f-MICL.
We denote 9 X' + gd-1 as the feature encoder (usually constructed by neural network) from the input space X' to the hypersphere, and use the shorthands 2² == g(x) and y g to represent the feature embeddings. The notation pa stands for the data distribution, and px Pa @ pa means its self product (product of marginals, c.g., pairs of images). We denote p+ as the distribution of positive pairs, i.e., two samples with similar feature embeddings (joint distribution, e.g,, the same image with different data augmentation). Using the lower bound of the f-MI in eq. (2), we have the general f-MICL objective as follows:
sup (aww),, s(29, E( S" s(x9, oEF
(4)
where s(z9, yy) can be understood as the similarity measurement between two feature embeddinngs in the context of contrastive learning. Essentially, we are studying the variational lower bound eq. (2) in the feature space, with the feature embeddings learnable. We can treat the first term as the similarity score between positive pairs with similar feature embeddings, and the second term as the similarity score between two random samples, a.k.a. negative pairs. As f* is an increasing function, maximizing the f-MI equivalent simultaneously maximizing the similarity between positive pairs and minimizing the similarity between negative pairs.
With eq. (4) we have answered the first question: there are a spectrum of f-MICT, objectives that can be applied in contrastive learning by using different f functions. We will discuss how to choose the best f empirically in $4.
3.2 Gauusssi similarity
Previous works in constrastive learning usually adopt some heuristic similarity function such as the cosine similarity function s(x9, y²) = x y Although shows promising performance practice, our second question is that, can we provide better similarity function than the popular cosine similarity?
Note that eq. (3) is natural choice of s(-) from the perspective of deriving the f-MI. In the context of contrastive learning, by denoting the density functions of the marginal feature distributions as Py (,(x)) and (y²), and the density of the joint feature distribution as Py (x y²), from eq. (3) we have an optimal similarity function as follows:

---

Published in Transactions on Machine Learning Research (10/2023)
Gaussian
Uniform
6700
6700
6600
log density
log demsity
6600
6500
6500
6400
6400
6300
0.000
0.002
0.004
0.006
0.000
0.002
0.004
0.006
0.008
squared distance
squared
Figure 1: Experiment for verifying Assumption 3. Here we draw the relation between the squared distances ||~09 1 and the averaged log likelihood log with log Pg estimated by the flow model RealNVP (Dinh et al., 2017). (left) Gaussian prior; (right) Uniform prior. The features are learned by SimCLR trained on CIFAR-10. See more details in Appendix D.3.
Lemma 2 (e-g-, Nguyen al. 2010, Lemma 1). Suppose is differentiable, and the embedding function g is fized. The following similarity function s, marimizes eq. (4):
Po(a9. (( (x( = 8 (1 (g))))
(5)
Obviously, the optimal 8, in fact induces the f-MI on the feature space, which is a low bound of the original f-MI. Although eq. (5) provides an optimal similarity function, it nevertheless depends on the unknown density functions. How can we implement eq. (5) in practice? Among various known density functions, it is natural to choose typical kernel function for structured data for validation (Balcan et al., 2008; Powell, 1987; Murphy, 2012), e.g., the Gaussian kernel.
Assumption (Gaussian kernel). The joint feature density (wrt the uniform distribution over the hy- persphere ) is proportional to Gaussian kernel, namely
Po(x0, y8) x Go([ y01 = exp (- 1100-y01²)
20
where μ exp( is a constant that we determine below. Since r9, x9,0 sd have unit (Euclidean) norm, have
Pg (29,, u exp( (a
(6)
which belongs to the von Miscs-Fisher bivariate distribution (Mardia, 1975, Eq. 2.11). It clear that the marginals of Pg are uniform. Indeed, for any orthogonal matrix Q, with J = E(go yo yo) exp( "0 we have
Pg (Qx0) Eyenge- C' exp( Qua-y²)
v2 Eyanss O exp(
x Q v"
(z =:
(7)
where we have used the fact that the only invariant distribution on gd-1 wrt the orthogonal group is the uniform distribution. Similarly, we have (v()) = (where e is the reciprocal of the surface area of the hypersphere gd-1). The distribution (6) has nice interpretation? terms maximum entropy (Mardia, 1975), and admits the factorization
(,(29 " - (x.) Py (265\1>)) = ps( Pg (2)1)9))
(8)
2 As suggested by the action editor, may also interpret the distribution (6) "copula," i.., joint density x with uniform marginals. Note that the conventional notion of copula replaces the hypersphere gd-1 with the unit interval [o. More generally, we could consider the "copula" Pg (2.% y2) h(x² yd) increasing function h, capture other types of correlation between the two views 2 and y
5

---

Published in Transactions on Machine Learning Research (10/2023)
where the marginals Py (29) and (3() uniform while the conditionals (5()) and Py Pg(x918) again belong to the von Mises-Fisher distribution.
Combining eq. (5) and Assumption 3, we can write the similarity function with the Gaussian kernel as follows:
85(x9 Go Gol y/11²).
As noted above, the product of marginals (p is constant, which has been absorbed into our definition of Ga, see in Assumption 3. We observe that s-(+) depends on the choice of f as well, thus we call it f-Gaussian similarity. As a result, we have provided a new way to design the similarity function, again from the f-MI perspective.
Verifying Assumption 3: One may question that Assumption can be too strong for practical usage. For example, replacing the Gaussian kernel G, with any other decreasing function would also provide valid assumption. However, we found that among several popular choices only the Gaussian kernel works well in practice. Also, we can empirically verify that Assumption 3 approximately holds. To this end, it is sufficient to check whether the log density, i.e., log (x (1),,))) is linear with the distance between each positive pair, i.e., ||29 - y0 In Figure 1, we use the Kow-based model RealNVP (Dinh et al., 2017) to estimate the log density with a Gaussian prior and a uniform prior, and learn the feature encoder g from SimCLR (Chen et al., 2020). We observe that the lincar relationship approximately holds for CIFAR-10
We will empirically compare our f-Gaussian similarity with the cosine similarity in $4.
3.3 Implementation
With our designed f-Gaussian similarity sy we now have an implementable f-MICL objective in eq. (4). Bringing the f-Gaussian Similarity 8y in eq. (9) into objective eq. (4) we have a specific objoctivvo
E sy(x", yd)
E
o (osy("",)))
(10)
(2/))-p4
(m.x))>x
Given a batch of N samples, its empirical estimation as follows:
Σ 2s((%.11) N
1
N(N 1) Σr o (T{)).
(11)
where and are two types of data augmentation of the i-th sample, and a; and xy are different samples with independently sampled data augmentations.
With the f-MICL objective in eq. (11) we propose our algorithm for contrastive learning in Algorithm 1. To balance the two terms our objective, we additionally include a weighting parameter a front of the second term (which also absorbs the parameter μ in G.). This change can still be incorporated within our f-MICT framework, as we show in Appendix A.2. Figure 2 gives high-level summary of our f-MICT, framework. Given batch of samples (e.g., images), we generate positive paárs via data augmentation and negative pairs using other augmented samples in the same batch. This sampling method follows SimCLR (Chen el al., 2020).
3.4 MMICL family
In this section, we will deepen the understanding of f-MICL by drawing connections with some popular constrastive learning methods.
Connection with InfoNCE: Firstly, we show that InfoNCE is an upper bound of f-IICL Recall our f-MICL objective in eq. (4), and the popular InfoNCE objective LIIONCE as follows (here we take the
RRalNNV applics real-valued non-volume preserving transformation for log-hioel computation.
4The linear relationship in Figure might also depend the data, Le,, the CIFAR-10 dataset here. In practice, other eustomized datasets might require additional verification.

---

Published in Transactions on Machine Learning Research (10/2023)
Algorithm 1: f-MICI
Input:
batch size N, function f, weighting parameter a, constant (in Go), variance x²
for each sampled mini-batch {2ili= do
for in 1,...,I do randomly sample augmentation functions & and t2 yk b Fy t2(2x)
define sy(r9, y0) = f' G. (1 y0ll²)
compute -L as
N
8 (+ v
a
Σ
r o 08
N
N(N 1) (#)
update by minimizing
imager
T
g
a(+t)
aug over (it 0c)
16
positive
Gaussian kernel
aignment
la T
sun
a(3u)
E-mulual information
o(x,)
ff
over
nogative pair
Gaussian kerne
[z,,
unformity
inaga
s(xyy)
Figure 2: Network architecture of f-MICL. image;: the ith image in the current batch; f: the function used in the f-mutual information ($2); g: feature embedding; f, 41, l2: augmentation functions drawn from the same family T of augmentations; f": the derivative; f*: the Fenchel conjugate. The symbol denotes the function composition. The sum of the two lerms gives the variational lower bound of f-mutual information. 2; and are two types of data augmentation of the i-th sample, and 2; and z, are different samples with independently sampled data angmentations. max stands for maximization. Seo eq. (11) for more details.
maximization) (van den Oord et al., 2018):
Re 8(x9, 10) Exyop log Eny-s exp(s(x9 .)))
(12)
Consider that we perform a Donsker-Vaute Varadhan (DV) shift transformation v (Donsker & Varadhan, 1975; Tsai et al., 2021) from eq. (4) such that by taking the maximum over the transformation we have:
sup
(Err))) s(29,19) HER
E(z,))~p I (s(x" v)).
(13)
In practice, such shift transformation can be approximated by a scaling factor (a in Algorithm 1) such that eq. (4) and eq. (13) are equivalent. Given that f is the KL divergence, thus f(u) ulog and

---

Published in Transactions on Machine Learning Research (10/2023)
FMICL
KL-divergence
DV-Shift LB of InfoNCE
f.Gaussian Similarity: AU
Figure 3: f-MICL generalizes InfoNCE-based objectives.
('(t) = exp(t- 1 1) from Table 1, the maximizer of in eq. (13) occurs at u, = s(29, ,3))
1).
With v.. eq. (13) can be written as follows:
(( s(x9, v8) log ((n)) exp(s(x".y")
(14)
According Jensen's inequality we have
Ex-p(-) log Ey-p(-) exp(s(x", s log (a,))>xx exp(s(x", yy)).
(15)
Therefore,
(a.)) s(x", v2) log (ur exp(s(x", v²)) S LInONNE
(16)
The above transformation shows that the InfoNCE loss is an upper bound of the f-MICL objective. In other words, maximizing f-MICL can potentially increase the InfoNCE objective.
Connection with Alignment and Uniformity (AU) (Wang & Isola, 2020): We further show that the Alignment and Uniformity (AU) loss is special cases of f-MICL. Wang & Isola (2020) shows that InfoNCE approximately aligns positive feature embeddings while encouraging uniformly distributed negative ones. Wang & Isola (2020) further proposes new objective which quantifies such properties. Here we show that this new objective is essentially a subclass of the InfoNCE loss under the f-MICL framework. Concretely, applying the f-Gaussian similarity function for the KL divergence, we have f'(u) log u+1 from Table and thus sy (x", 9²) -|129 1l Using sy("9, y²) eq. (14) we can recover the AU objective:
E (. ||ا 10122 log E Ex [cxp(-1|2* 1|220 y9||3]]
(17)
Note that for KL, this is equivalent the cosine similarity with a scaling factor: - 112 2 = 2n 8
Connection with the Spectral Contrastive Loss: Here we show that f-MICI. objective is closely related to the Spectral Contrastive Loss (HaoChen et al., 2021). Recall our objective:
B(za)>pi s(x ((x","") y²) E(za)~p s(x". .")),
(18)
where s (2 3 f'oGo (1|299 8a (P). If we choose the Pearson X² divergence, where f(u) (u- 1),, f'
2(u f" fu) u² 1, we have our 2 objective:
21E (ax Go(129 113 E(a.)>ps Go - -3
(19)
8

---

Published in Transactions on Machine Learning Research (10/2023)
This recovers the spectral contrastive loss exactly if choose the proper hyperparameter and apply the cosine similarity instead. Thus we generalize the spectral contrastive loss as special case of X2-MICT.
More on AU: Finally, based on our objective in eq. (10) we will show that the alignment and uniformity (AU) property of InfoNCE also extends to the general f-MICL family: (1) Alignment: In the ideal case, maximizing the first term of eq. (10) would yield x = y for all (x,y) ~ p-: f.e., similar sample pairs should have aligned representations. Note that the derivative f' is increasing since f is convex. (2) Uniformity: We demonstrate the uniformity property by minimizing the second term of eq. (10), more rigorously and realistically, its empirical version in eq. (11).
Theorem (Uniformity). Suppose that the batch size N satisfies 2 N S dd+1, 1, with d the dimension of the feature space. If the real function
h(t) = f (t) is strictly conver on [0,
(20)
then all minimizers of the second term of eq. (11), Le.,, Ex, o 8y (2), salisfy the following condition: the feature represenlations of all samples are distributed wniformly on the unil hypersphere Sd-1,
In Theorem 4, the assumption N < d + is always satisfied in our experiments in $4. For instance, CIFAR-10 we chose N = d = 512. Also, we claim that the samples are "distributed uniformly" if the feature vectors form a regular simplex, and thus the distances between all sample pairs are the same. Although minimizing the negative term gives uniformity, the positive term is also needed for aligning similar pairs, as we observe in $4. This implies the tradeoff between alignment and uniformity.
In fact, eq. (20) provides us guidance select proper f-divergences for uniformity. In Table 1, we list some common choices of f-divergences. By inspecting the last column and using the definition of Go, we easily verily that they all satisfy eq- (20). However, this is not true for all f-divergences. In Appendix A.1 we also provide some counterexamples that violate eq. (20) and thus Theorem 4, such as the Reversed Kullback Leibler (RKL) and the Neynman x" divergences. Experimentally, we found that these divergences generally result in feature collapse (i.e., all feature vectors are the same) and thus poor performance in downstream applications.
4 Experiments
In this section, empirically evaluate the analysis on provided answers: (L) Can go beyond the KL-based objective (84.2): we apply various f-MICL objectives to popular vision and language datasets. In particular, we show that under the same network architecture design, f-MICL can always provide a better choice of objective. We observe that the best-performing f-divergence is largely dataset dependent. (2) Can we design a better similarity function ($4.3): we show that the proposed f-Gaussian similarity is more powerful than the heuristic cosine similarity, regardless of the choice of f. Morcover, we confirm empirically that f-MICL extends the nice property of alignment and uniformity in $4.4.
4.1 Experimental settings
Our detailed settings can be found in Appendix D. In all our experiments, we change only the objective of different methods for fair comparison. We use the f-Gaussian similarity in f-MICL by default.
Vision task. Our vision datasets include CIFAR-10 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), TinyImageNer (Chrabaszcz et al., 2017), and ImageNet (Deng et al., 2009) for image classification. After learning the feature embeddings, we evaluate the quality of representations using the test accuracy via linear classifier. Note that we = 40 across all vision experiments.
ResNet-18 (He et al., 2016) for CIFAR-10; ResNet-50 (He et al., 2016) for the rest. Our implementation is based on SimCLR (Chen et al., 2020), where we used the same 3-layer projection head during training. All models are trained for 800 epochs.
(1) Smaller datasets: For feature encoders,
(2) ImageNet: We choose Vision Transformer (VIT-S) (Dosovitskiy et al., 2020) as our feature encoder. We choose the smaller VIT-S model with blocks because larger ViT models are extremely expensive to train on GPUs Our implementations are based on MoCo V3 (Chen et al., 2021), where models are trained for 1000 epochs.
9

---

Published in Transactions on Machine Learning Research (10/2023)
Table 2: We compare the test accuracy (%) obtained with the linear evaluation on the vision datasets. On the Wikipedia dataset, we compare the semantic textual similarity (STS) via the Spearman's correlation. For each dataset and each method we take three different runs get the mean and the standard derivation.
Baselines
J-MICL
Dataset
MoCo
SimCLR
AU
KL
JS
Pearson
VLC
CIFAR-10
90.41 80.37 20.10 10.26
90.6110.47
89.66
89.35 20.52
89.13
10.33
STL-10
23.69_0.22
10.32
84.4440.19
85.33
85.94
82.64
85.9440.72
TinylmageNett
35.72 10.17
30.5610.20
11.20
39.26
42.98 10.18
43.45 10.54
38.65 10.45
Wikipedia 77.88
77.40
77.95 77.9510.08
78.02
76.7610.09
77.59
55.07 10.13
Table 3: We compare the lest accuracy (%) with SOTA methods on ImageNet. We take three different runs to get the mean, where the standard derivations are less than 0.1% for f-MICL.
Baselines
f-MICL
Datasct
SwAV
BYOL
Barlow Twins
VICReg
RényiCL
MoCo v3
KL
JS
Pearson
ImageNet
75.3
74.3
73.2
73.2
76.2
73.2
73.9
76.5
74.6
Language task. To show the wide applicability of our f-MICL framework, we also conduct experiments on a natural language dataset, English Wikipedia (Gao et al., 2021). We follow the experimental setting in (Gao et al., 2021), which applies BERT-based models to SimCLR (Devlin et al., 2019; Liu al., 2019). Specifically, we choose the BERT model due to limited computing resources. For f-MICL objectives, we choose = 409600. The application task is called semantic textual similarity (STS (Agirre el al., 2013)) and we report the averaged Spearman's correlation in Table 2 for comparison.
4.2 f-MICL objectives
Smaller datasets and language task. We first compare f-MICL with several InfoNCE- contrastive learning algorithms (i.e., SimCLR (Chen et al., 2020), MoCo (He et al., 2020), and AU (Wang & Tsola, 2020)) smaller datasets and the language task in Table 2. Here choose four f-divergences with the best overall performance. See Appendix D for results other f-divergences.
From Table 2 we observe that: (1) As we have shown in $3.4 that f-MICL generalizes InfoNCE-based objectives, empirically KL-MICL achieves similar performance to the basclincs. In practice, we can tunc the hyperparameter a such that KL-MICL outperforms the Info-CCE- objectives. (2) KL-MICL is not always the optimal choice. We can see that the best-performing f-MICL objectives refer to four different f-divergences four datascts.
The above results indicate that f-MICI. can provide wide range of objective choices for the downstream tasks. Although how to derive an optimal f-divergence deserves more study in theory, in practice we can select the best f among several common f-divergences on a validation set.
Besides the f-divergences Table 1, in Theorem we have identified non-satisfying f-divergences. Tn our experiments, we found that applying these f-divergences such as the RKL and Neyman x² divergences would result in feature collapse. For example, in Figure show that with RKL the features all collapse to a constant.
To further demonstrate the efficacy of f-MICL we then compare with several popular self-supervised learning methods, including both contrastive-based and non contrastive-based ones. These methods can be categorized into the ResNet-basee (SwAV (Caron et al., 2020), BYOL (Huynh et al., 2020). Barlow Twins (Zbontar et al., 2021), VICReg (Bardes et al., 2021)) and RényiCL (Lee Shin, 2022), and the ViT-based (Dosovitskly et al., 2020). For the ResNet-based methods, we directly retrieve results from (Bardes et al., 2021), which are obtained by training ResNet-50 models for 1000 epochs. Chen et al. (Chen
ImageNet results:
10

---

Published in Transactions on Machine Learning Research (10/2023)
Table 4: Comparison between the cosine and G-Gaussian similarities on CIFAR-10 with the test accuracy (%). For the Tsallis-a divergence we take a 3.
Similarity
KL
JS
Pearson
SH
Tsallis-o
VLC
89.95
88.06
87.79
87.06
88.55
10.00
Cosine
+0.26
+0.33
#0.42
+0.55
=:0.28
+:0.0
90.61
89.66
89.35
89.52
89.15
89.13
Gaussian
+0.447
+:0.8
+-0.52
+00.55
+00.42
+.033
CIFAR-10
CIFAR-100
CIFAR-10
1.25
test test accuracy (%)
84
1.00
distance
82
0.75
80
0.50
Pearson
-0- KL
Pearson
0.25
RKI
SimCLR
random
0.00
100
200
400
500
pairs
pairs
batch
Figure 4: (left and middle) Distances between pairs of normalized features within a batch. Green region: similar pairs. Orange region: dissimilar pairs. f-MICL gives nearly uniform distances for dissimilar pairs for the /-divergences constant and thus the distances are zero. (right) The test accuracy v.s. the batch size after training 200 epochs for all algorithms.
Table 1. For non-salislying f-divergences such as the RKL, the features collapse
et al., 2021) show that these two types of methods are directly comparable in verms of the model size and supervised learning performance. For the ViT-based method and our f-MICL, we apply ViT-S for 1000 epochs. Specifically, our f-MICL follows MoCo v3 with only the objective changed for different choices of the f-divergence.
Results in Table 3 show that: (1) By only changing the objective function, method improves MoCo v3 by 2.1% using JS-MICL. (2) f-MICL objectives are comparable with the Nes-et-basod methods (c.g., SwAV and RényiCI...
Overall, our experiments confirm that f-MICL can provide a better choice of objective than InfoNCE on a variety of datasets, tasks, and encoder architectures.
4.3
f- Gaussiian similarity
Next, we want to examine the effect of our similarity function while fixing the f-divergences. Table 4 we compare the cosine and f-Gaussian similarities for different /-divergences on CIFAR-10. can be seen that under our f-MICL framework the f-Gaussian similarity consistently outperforms the cosine similarity for various f-divergences". This agrees with our Thcorem 2 and oq. (9), and also implies the validity of Assumption 3. In particular, we identily that without the theoretical guarantee, the heuristic cosine similarity would fail for certain f-MICL objectives (e.g., VT.C).
4.4 Alignment and uniformity test
We empirically check the properties of alignment and uniformity for f-MICL by plotting the pairwise distance
| - 2 of the feature representations within the same batch on CIFAR-10. We compute the distances
5 As have shown the equivalence between cosine and Gaussian similarity KL, the difference results KL. just. show the choice of different scaling factors.
11

---

Published in Transactions on Machine Learning Research (10/2023)
between the normalized features of every pair from random batch, and then sort the pairs in increasing order. From Figure 4 we can see that f-MICT. gives nearly uniform distances for dissimilar pairs (orange regions) on both datasets with various proper f-divergences. In contrast, a random initialized model gives a less uniform distribution for dissimilar pairs. Besides, we observe small pairwise distances for similar pairs (green regions).
4.5 Sensitivity to batch size
Finally, we study the sensitivity to the batch size of our f-MICL framework on CIFAR-10. On the right panel of Figure 4, we evaluate the classification accuracy by varying the batch size for different f-divergences and SimCLR. We can see that for all different batch sizes and with the proper choice of f-divergences, our performance always better than SimCLR. In other words, we require fewer negative samples to achieve the same performance.
5 Related Work
Contrastive learning. Self-supervised contrastive learning learns representalions by contrasting sample pairs. Recently it has been shown analytically that improving the contrastiveness can benefit the downstream applications (Saunshi et al., 2019: Tosh et al., 2021). For popular contrastive learning methods such as Contrastive Predictive Coding (CPC) (van den Oord et al., 2018), SimCLR (Chen al., 2020), and MoCo (He al., 2020), their loss functions can be interpreted as lower bound of mutual information, which is essentially the KL divergence between the joint distribution and the product of margin distributions. Besides the KL divergence, other statistical divergences or distances have been individually studied under the context of contrastive learning, e-g-, the Wasserstein distance (Ozair et al., 2019), Pearson x² divergence (Tsai et al., 2021), and Sensen-Shannon divergence (Hjelm et al., 2018).
f-divergences have been widely used in generative models (Nowozin et al., 2016) and domain adaptation (Acuna et al., 2021), for measuring the discrepancy of two distributions, where the variational lower bound is often employed for estimation. Compared to f-GAN (Nowozin et al., 2016) and f-DAL (Acuna et al., 2021) which minimize the f-divergence between two different distributions, our f-MICL objective is to maximize the f-divergence between the joint distribution and the product of marginal distributions. This agrees with our purpose of contrasting sample pairs. Moreover, we provide a theoretical criterion for choosing proper f-divergenccs.
Mutual Information also plays an important role in the context. of deep representation learning (Tian et al., 2020a; Bachman et al., 2019; Hjelm et al., 2018; Tian et al., 2020b; Poole et al., 2019; Belghazi al., 2018). Loss function wise, our losses partially cover the losses in the literature and generalizes them: c.g., Poole et al. (2019) considers several variational lower bounds of mutual information, where we generalize the DV objective; (b) application-wwsee none of them considers contrastive learning: e.g., Poole et al. (2019) considers mutual information estimation, Belghazi et al. (2018) improves adversarial generative models, Hjelm et al. (2018) considers representation learning that maximizes local and global information.
Metric learning. Our work closcly related to metric learning (Kaya & Bilge, 2019; Suárcz-Diaz et al., 2018), which aims to learn distance metric bringing similar objects closer and distancing dissimilar objects further. In contrastive learning, a pre-defined similarity metric, e.g., the cosine similarity (Chen et al., 2020; He et al., 2020) or bilinear function (van den Oord et al., 2018; Tian et al., 2020a; Hénaff et al., 2020) is commonly used to measure the sample similarity. These pre-designed metrics may not necessarily lead to satisfactory performance in practice. Comparably, the design of our similarity function is empirically tailored for contrastive learning.
Finally, we summarize existing representation learning methods that utilize f-divergences and compare with M-MICL in Table 5 for clear view of the literature.
12

---

Published in Transactions on Machine Learning Research (10/2023)
Table Comparison between different representation learning methods that apply f-divergences.
Method
Objective
f-divergence
Similarity
Task
CPC (van den Oord et al., 2018)
InfoNCE RPC
KL
log- bilinear cosine
predictive coding predictive coding
RPC (Tsai et al., 2021)
Pearson X²
MINE (Belghazi al., 2018)
DV bound of MI
KL
neural net
GAN generation
DIM (Hjelm et al., 2018)
JSD/ NCE
JS & KL
neural net joint/separable
representation learning
Poole et al. (2019)
MI bounds
KL
representation learning
SimCLR (Chen al., 2020)
InfoNCE
KL
cosine contrastive learning
MoCo (He et al., 2020)
InfoNCE
KL
cosine contrastive learning
AU (Wang & Isola, 2020)
InfoNCE
KT.
Gaussian
contrastive learning
f-MICL (Ours)
f-MI
general
f-Gaussian
contrastive learning
6 Conclusion
We developed f-MICL for contrastive learning, which generalizes the KL-based mutual information to the f-mutual information. With f-MICL provided broad spectrum of objective choices with better down- stream performance. We also proposed novel f-Gaussian similarity function, which shows superior perfor- mance to the commonly used cosine similarity. In addition, we confirmed the generalization of f-MICI, by comparing with popular InfoNCE-based objectives. Empirically, we exhibited the efficacy of f-MICL across a wide range of datasets from both vision and natural language.
Limitations and future work. While f-MICL provides a variety of objective functions, it is yet unclear how choose an optimal based on task and dataset in theory, such that we usually rely on a validation set in practice for selection. An interesting future work is to learn optimal f-divergence using parametrized neural network. Moreover, Lee & Shin (2022) applied Skew Rényi divergence for contrastive learning. However, we observe that applying Rényi-MICI. naively leads to a large variance (similar to Section 4.1 in Lee & Shin 2022), and we leave the discussion on skew divergences for future works. Additionally, McAllester & Stratos (2020) showed that there exist some inherent statistical limitations on accurately estimating the mutual information with various lower bounds. In future work it would be interesting examine such limitations extend to f-MI, and if limited estimation of f-MI necessarily affects f-MICL whose goal is to compare and learn representations through (lower bounds of) f-MI.
13

---

Theory and Published in Transactions on Machine Learning Research (10/2023)
Acknowledgement
We thank the reviewers and the action editor for their constructive comments. Part of this work performed during YL's internship at NRC. YY thanks NSERC and CIFAR for funding support. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companics sponsoring the Vector Institute.
References
David Acuna, Guojun Zhang, Mare T Law, and Sanja Fidler. f-domain-adversarrial learning: algorithms. In JCML, 2021.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Monzalez-Agirea and Weiwei Guo. *SEM 2013 shared task: Semantic textual similarity. In Second joint conference on lerical and computational semanties (*SEM), volume proceedings of the Main conference and the shared task: semantic textual similarity, 2013.
Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficienss of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological)) 1966.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. NewrIPS, 2019.
Maria-Florina Balcan, Avrim Blum, and Nathan Srebro. A theory of learning with similarity functions. Machine Learning, 72(1):89 112, 2008.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variauce-invarianeccccoaaaaa regularization for self-supervised learning. arXiv preprint arXiv:2105.04: 04906, 2021.
Peter Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks. JMLR, 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. MINE: mutual information neural estimation. arXiv preprint #rXiv:1801044022 2018.
Sergiy V Borodachov, Douglas P Hardin, and Edward B Saff. Discrete energy on rectifiable sets. Springer, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 13:9912-9924, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In TCML, 2020. URL URI http://prooeeedings mlr presss/v119/chen20f html.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. Tn Proceedings of the TEEE/CV International Conference on Computer Vision, pp. 9640 9640-9649, 9649, 2021.
Patryk Chrabaszez, Tya Loshchilov, and Frank Hutter. A downsampled variant of ImageNet alternative to the CIFAR datasets. arXie preprint
2017.
Adam Coates, Andrew Ng, and Honglak Lec. An analysis of singlo-layer networks in unsupervisod feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
14

---

Published in Transactions on Machine Learning Research (10/2023)
Imre Csiszár. Information-typo measures of difference of probability distributions and indirect observation.
studia scienliarum. Malhemalicarum Hungarica, pp. 229-318, 1967.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database, In CVPR. IEEE, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapler of the Associalion for Compulational Linguisties: Human Language Technologies, 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. arXiv preprinl or Xiv: 1 100.8566, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. ICLR, 2017.
Monroe D Dousker and SR Srinivasa. Varadhan. Asymplotic evaluation of certain markov process expecta- tions for large time, i. Communications on Pure and Applied Mathematics, 28(1):1-47, 1975.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXic:2010. 11929, 2020.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271 21284, 2020.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. NeurIPS, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw Ali Razavi, Carl Doersch, SM Eslami, and Aaron den Oord. Data-efficient image recognition with contrastive predictive coding, In ICML. PMLR, 2020.
R Devon Hjelm, Alex Fodorov, Samuel Lavcic-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2018.
Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. Boosting contrastive self-supervised learning with false negative cancellation. arXiv preprint arXi::2011.1199 11765, 2020.
Mahmut Kaya and Hasan Şakir Bilge. Deep metric learning: A survey. Symmetry, 2019.
Diederik P Kingma and Prafulla Dhariwal. Glow: generative flow with invertible 1x convolutions. In NeruIPS, 2018.
Vladimir Koltchinskii. Rademacher penalties and structural risk minimization. IEEE Transactions on Information Theory, 1995)):102-1114, 2001.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009. Technical report.
15

---

Published in Transactions on Machine Learning Research (10/2023)
Lucien Le Cam. Asymptotic methods in statistical decision theory. Springer Science & Business Media, 2012.
Kyungmin Lee and Jinwoo Shin. Rényicl: Contrastive representation learning with skew rényi divergence. Advances in Neural Information Processing Systems, 35:6463 6477, 2022.
Ralph Linsker. Self-organization in a perceptual network. Computer, 1988.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint Xiv: 1907. 11692, 2019.
Nya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent. with warm restarts. International Conference on Learning Representations, 2017.
Πya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018.
K. V. Mardia. Statistics of directional data. Journal of the Royal Statistical Sociely: Series B, 37(3):
349 -371, 1975. URL https://doi.org: htrg/10.1111/// 2517-61161 1975 tb01550.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, pp. 875-884. PMLR, 2020.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
XuanLong Nguyen, Martin Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural samplers using
variational divergence minimization. In NewrIPS, 2016.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. In NewrIPS, 2019.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171 5180. PMLR, 2019.
M. J. D. Powell. Radial Basis Functions for Multivariable Interpolation: A Review. Clarendon Press, USA, 1987. ISBN 0198536127.
Ralph Rockafellar. Characterization of the subdifferentials of convex functions. Pacific Journal of Mothe- matics, 1966.
Tgal Sason and Sergio Verdú. f-divergence incqualitics. IEEE Transactions on Information Theory, 2016.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A thcorotical analysis of contrastive unsupervised representation learning. In ICML, 2019. URL http:
//proceedings
mll. mlr. ppess//99//sumshhi1...
Juan Luis Suárez-Díaz, Salvador García, and Francisco Herrera. A tutorial distance metric learning: Mathematical foundations, algorithms, experimental analysis, prospects and challenges (with appendices on mathematical background and detailed algorithms explanation). arXiv preprint 1rXi::1812.0594 2018.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, In ECCV. Springer, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. In NewrIPS, 2020b. URL https: htttp:://ppooedings.noooeo cc/ Abstracct html.
16

---

Published in Transactions on Machine Learning Research (10/2023)
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory. PMTR, 2021.
Yao-Hung Hubert Tsai, Martin Ma, Muqiao Yang, Han Zhao, Louis-Philippe Morency, and Ruslan Salakhutdinov. Self-supervised representation learning with relative predictive coding. In TCI.R, 2021.
Constantino Tsallis. Possible generalization of Boltzmann-Gibls statistios. Journal of statistionl physics, 1988.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual infor- mation maximization for representation learning. In ICLR, 2020.
Jean-Baptiste Hiriart Urruty and Claude Lemaréchal. Convex analysis and ménimization algorithms. Verlag, 1993.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprinl 1817.00077 2018.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In TCML. PMLR, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310 12320. PMLR, 2021.
Shaofeng Zhang, Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. Zero-cl: Instance and feature decorrelation for negative-free symmetric contrastive learning. In International Conference on Learning Representations, 2021.
17

---

Published in Transactions on Machine Learning Research (10/2023)
Table 6: summary common f-divergences. KL: Kullback Leibler; JS: Jensen Shannon; SH: Squared Hellinger. For JS, we define g(u) -(u + 1) log 1 + ulog u. For Pearson x²: we take (t) -1 if 2
< -2. For Jeffrey, W W + W-1 and W--) is the Lambert-W product log function. The Tsallis-a divergence is defined in Tsallis (1988) and we have > for f-divergences. We ignore constant addition -1/(a- - 1) because it does not change the optimization problem. The Vincze- Le Cam divergence can be found in Le Cam (2012) which is closcly related to x² and Hellinger divergences. For the Vincze-Le Cam divergence we require -3 t < and (1) = -1 if -3.
Divergence
f(u)
(*)
f'(u)
f f'())
KL Reverse KL
u ulog u loggu
exp(t - 1)
log u + -1/u
u
-1 -1-log(
log -
JS (uu)
1oog(2 et)
log 2 log I+u
log + log(1 + u)
Pearson x²
(u 1)2
2//4tt
2(u 1)
21
SH
(Vū- - 1)2
I-
u 11/ 1/2
w//3 -
Neyman x²
(1-u)²
2/1 1
1
-2u" 22-1
" (u 1) log
Jeffrey
((el-1) + 1 2
1 u
+ log
W (e*/u (e/////) + log 1+
Tsallis-o
""
(a-lt)a-1
a u aa1
aaa
Vincze Le Cam
[u-1)²
4-t 11-t-4414 4vI-t
1)++33
3
4
(2+1))
u+1
A Additional theoretical results
In this appendix, we provide additional theoretical results, including additional f-divergences and the theory for weighting paramcters.
Notations. We assume that dominating measure X (e.g. Lebesgue) is given and all other probability measures are represented as some density w.r.t. X. Given the joint density p(x, J p(z, v))(y) and p(y) f(z y)dA(z) as the marginals. We use supp(-) to denote the support of a distribution, and f to denote the conjugate of function f. Every norm presented is Euclidean. We use xº == g(r) as the shorthand notation for the feature embedding, with x a raw sample. The notation pa stands for the data distribution, and px := Pa @ pa means its self product. We denote p+ as the distribution of positive pairs, i.e., two samples with similar feature embeddings. The symbol denotes function composition.
denote p(x)
A.1 Additional /-divergences
We expand Table 1 and give more f Theorem 4, Table gives a special class of S-divergences that guarantees uniformily. A detailed description of f-divergences can be found in e.g. Sason & Verdú (2016).
of f-divergences in Table 6. As we will see in the proof of
A.2 Weighting parameters
In Algorithm 1 we added a weighting parameter to balance the alignment and uniformity. We prove that even after adding this parameter are still maximizing the f-mutual information, although with respect to different f.
Proposition 5 (weighting parameter). Given > 0 and closed convex function f R. R such that f11 = =0, define fa : dom f + R with
fa(x) af - af 1
18

---

Published in Transactions on Machine Learning Research (10/2023)
for any a E dom f. Then fa is still valid f-matuul information (see Definition 1). Besides, by replacing I with Sa in eq. (10) we have the following oplimization problem:
sup B(ew)rp+ (5" gEg
[5r pr
Go (0o(1la--99199))
- )
O. (z
a
where Go (1|29 y9113) = cxx
1 - is the Gaussian kernel.
20
Note that a dom f means the scalar multiplication of a set which is applied element-wisely. According to Definition 1, fo is also a valid f-divergence. This proposition tells us that rescaling the second term with factor a is equivalent to changing the function f to another convex function fa- The transformation from f to af (=) is also known right scalar multiplication (Urruty & Lemaréchal, 1993). Let now move to our proof:
Proof. By definition, we know that fa is convex and closed with fo (1) 0, and thus I5, is valid f-mutual information according to Definition 1. Moreover, have (x) (²) for any dom f and
(t) = sup xt fa()
#Edom fo
sup zt af ( + +af 21)
.Eadom
(al) - as (=) + as
sup Edom
0
T 1 (²) af !
sup
Edom
a
= af* +af
(21)
where the last line we used the definition (t). Plugging f. and fa into eq. (10) yiclds the desired result.
B Proofs
Lemma 2 (e.g., Nguyen al. 2010, Lemma 1). Suppose f is differentiable, and the embedding funclion g is fized. The following similarity function 8, marimizes eq. (4):
Po(29. s.(x9,9) -f ( (1g
(
(5)
Proof. From Definition 1, we are computing the following supremum:
/ sup 9.a
Po 3 s(x9, y8) f s(x9, yo) drl va-
(22)
Suppose s is unconstrained and we fix The optimal solution should satisfy:
(29 2d) Pg (1 Py (19)
(O ,)),
(23)
almost surely for (x,z) ~ Pd @ Pa. From (3.11) of Rockafellar (1966) this is equivalent to:
Pg
sy (. , E 0s
(24)
If f is differentiable, then for any dom f, af(w) (f') is singleton.
7
19

---

Published in Transactions on Machine Learning Research (10/2023)
Theorem 4 (Uniformity). Suppose that the batch size N satisfies 2 N N d with d the dimension of the feature space. If the real function
h(t) f o Gt is strictly conver on [0. 4].
(20)
then all minimizers of the second lerm of eq. (11), i.e,, ( æÿ), satisfy the following condition: the feature representations of all samples distributed wniformly the unit hypersphere $6-1
Note that say the samples are "distributed uniformly" if the feature vectors form regular simplex (see Figure 5), and thus the distances between all sample pairs are the same.
Proof. From the definition of h it is clear that h is decreasing since f and f are both monotonically increasing white G. is decreasing. Using h we rewrile the second term of eq. (11) as
min
- J
(25)
EB4
When E [2, d 1], there exists a neat characterization of the minimizers, see e.g. Borodachov et al. (2019). We include the proof below for completeness.
Apply Jensen's ' inequality, we have:
Σ
h(1=; - xx,11) 2h
1
20 - 1311
N2
N2
1
h
1
Σ
20 =zz1?
N2
1
h
2
204
23))
N2
N Σ
h
2
1
N
2 h (2),
(26)
where in the first line we used Jensen's inequality; in the third line used ||=;;| ||= for any i, [N]: decreasing function. When is strictly convex and
in the last line we note that | æzll 2 0 and decreasing, it in fact strictly decreasing, and hence the two inequalities above can be attained iff
Σ x 0, and ||. - xz 1| = c for all i j
(27)
namely that ( xN} form a regular simplex with its center at the origin. We remark that when is merely convex, points forming a centered regular simplex may form a strict subset of the minimizers.
To the necessity of N let us note that
i=j
xo x
(28)
i
N-1+
since
Σ ||: 2;112 = N(N 1)e =
(29)
2N
c=
2+ N -
N W-1
20

---

Published in Transactions on Machine Learning Research (10/2023)
Figure 5: A regular simplex hypersphere.
Performing simple Gaussian elimination we note that the matrix X x has rank N 1 where X [r1....,IN IN] ROXN Therefore, we must have N -
Lastly, we need to show when h is (strictly) convex function, which may not always be true depending on the f-divergences. We give the following characterization (we ignore the constants and 20 in Assumption as they do not affect convexity):
strictly convex: hKI. = e (1) log(1 +t- -log 2, hpearson (1) e- 21 hsn(() = e12/2 hTsallii (t) -at lVLC
Ita-ti
h convex but not strictly convex: (tKL(t) -t (RKL stands for Reversed Hallback-Leiblee see Appendix A.1);
h concave: NNeeyman (t) 2 - 2e' (Neyman stands for Neyman x², see Appendix A.1).
Only for the last case we do not have the guarantee that the minimizing configurations could form a regular simplex. For RKL, in fact, any configuration that centers at the origin suffices since h is linear function.
C Estimation of f-MICL Objective
In 3.3 we have provided the empirical estimation of our objective in eq. (11). However, it remains a question whether estimation of f-mutual information is consistent. In this part we derive an upper bound for the estimation error from statistical learning theory.
We denote our f-MICL objective as ix; Y) (eq. (10)) and its empirical estimation as ix; Y) (eq. (11)).
After fixing the similarity function, we have T(x, y) = = sy(x9, "a2,,y' y = f Go((aa )2)
Theorem (estimation error). Suppose that the function T taken from a function class T and define as the function class of T(x,-) given some x E supp(pa). Denote Ry to be the Rademacher complerity w.r.t. the distribution P with N i..d. drawn samples. Then for any T T, the estimation ervor (X;
7 (X;Y)) is upper bounded with probability least 1:
20 (T) + 24
E amups
Di (Ta)
Σ Rid (Ta)) + (rr + +22))
log 6/8
(30)
2(N 1)"
with the constants rT f'()) lue 2/0² -l/a"
and
=-= f'r) fr f'e 2/02).
Here the constant is from our Gaussian kernel in Assumption 3. Rademacher complexity evaluates the richness of a class of real-valued functions regarding probability distribution, and its formal definition can be found Koltchinskii (2001).
Non-i.i.d. proof. Our conclusion is theoretically non-trivial since our sample pairs are non-i.i.d. although the individual samples are assumed be i.i.d., the negative pairs are not independently drawn (e.g., (x1, and (z)) which makes the derivation challenging.
21

---

Published in Transactions on Machine Learning Research (10/2023)
Note that the function class T depends on the class of the feature encoder g and the f-divergence. Our est estimation error eq. (30) is composed of three parts:
the Rademacher complexity of the function class T. In general, if T is richer then its Rademacher complexity is also larger.
the expected Rademacher complexity of the one-side function class To and its empirical estimation; an error term that decrcases with more samples.
Since the encoders are usually built, with neural networks, we can use the existing theory (Bartlett et al., 2019) to give more detailed bounds for the Rademacher complexities of T. Specifically, if the Vapnik- Chervonenkis (VC) dimension of T is finite, then our estimation error in eq. (30) goes to zero as N + 00 (Mohri et al., 2018).
Approximation and estimation tradeoff. In order to minimize the estimation error in eq. (30), we should choose a simpler function class T reduce the Rademacher complexities. However, T should also be rich enough so that eq. (3) can be satisfied, since our objective (y(X; should approximate the f- mutual information YX; if we choose the optimal T. Therefore, there natural tradeoff between approximation and estimation errors when change the complexity of T.
D Additional experimental results
We present additional experiment details in this appendix, to further support our experiments in the main paper.
D.1 Implementation details
In this paper, we follow the implementations in SimCLR (bttps://githe //gtthbb./ com/sthalless//iiCC and MoCo v3
For vision lasks, we use ResNet (He el al., 2016) and ViT-S (Dosovitskiy et al., 2020) as the feature encoder, and we adopt the similar procedure of SimCLR/MoCo for sampling. For the language dataset, we follow the exact experimental setting of Gao et al. (2021) and only change the objective. Our experimental scttings are detailed below:
Hardware and package: We train
GPU cluster with NVIDIA T4 and P100. The plat- form use is pytorch. Specifically, the pairwise summation can be casily implemented using forch.nn.functien www.n.ffuuniiiaal..diii..i. odast from pytorch.
Datasets: the datasets we consider include 10, STL-10 (Coates et al., 2011), IinylmageNet (Chrabaszez et al., 2017), ImageNet (Deng et al., 2009) and English Wikipedia (Gao et al., 2021).
Augmentation method: For each sample in dataset we create a sample pair, a.k.a. positive pair, using two different augmentation functions. For image samples, we choose the augmentation func- tions to be the standard ones in contrastive learning, e.g., in Chen et al. (2020) and He al. (2020). The augmentation is a composition of random flipping, cropping, color jittering and gray scaling, For text samples, following the augmentation method of Gao et al. (2021) we use dropout masks. Neural architecture; For 1IFAR-10 we use ResNet-11 (He et al., 2016); for STL-10, TinylmageNe we use ResNet-50 (He et al., 2016); for ImageNet we use ViT-S (Dosovitskiy et al., 2020); for the Wikipedia dataset we use BERT (Devlin et al., 2019).
Batch size and embedding dimension: for experiments CIFAR-10 we choose batch size 512; for STL-10 we choose batch size 64 to accommodate one GPU training; for TinylmageNet, we choose choose
choose batch size 1024. For all the vision datasets, the embedding dimension to be 512. Regarding the language dataset, the batch size is 64 with the feature dimension 768. In all of these cases, our assumption N < d 1 in Theorem 4 is satisfied. Hyperparameters: in all our experiments we fix the constant factor μ = 1. We find that in practice the weight parameter a often needs to be large (e.g., in the Wikipedia dataset), which requires moderate tuning.
batch size 256; for ImageNet,
Optimizer and learning rate scheduler: For smaller vision tasks, we use SGD with momentum for optimization and the cosine learning rate scheduler (Toshchilov & Hutter, 2017). For the ImageNet
22

---

Published in Transactions on Machine Learning Research (10/2023)
Table 7: Detailed experimental settings. arch: the neural network architecture used. N: batch size; the dimension of the feature representation; 1r: learning rate; μi: the constant factor in μ4; 1/(20²) and a follow from Algorithm epoch: the number of epochs we run; the number of nearest neighbors in K-NN evaluation.
Dataset
arch
N
d
1r
(202)-1
Cx
epoch
k:
CIFAR-10
ResNet-18
512
512
0.1
40
800
200
STL-10
ResNet-50
64
512
0.1
40
800
200
TinyImageNet
ResNet-50
256
512
0.1
40
800
200
Tmt
ViT-S
1024
512
0.1
40
1000
n/a
Wikipedia
BERT
64
768
3e-5
1
20
409600
1
n/a
task and natural language task, use Adam with weight decay (Loshchilov & Hutter, 2018) and the linear decay scheduler.
Evaluation metric: for vision tasks, we use k-nearest-neighbor (k-NN) (only small datasets) and linear evaluation to evaluate the performance, based on the learned embeddings. For the NLP task, we use the Spearman's correlation evaluate the averaged semantic textual similarity score (Gao al., 2021).
Baseline methods: for the four baseline methods, follow the implementations in:
MoCo: attpee//ggtthu..coo//sssaalll///
SimCLR:
Uniformity:
MoCo v3: https://githhh htgpt//..co/tteeoooorrrrrroooaoo
For fair comparison we use the experimental settings in Table 7 for all the baseline methods, which might differ from the original scttings.
Table gives common choices hyperparameters for different datasets. Note that we may need to further finetune a and for different f-divergences. See our supplementary code for more details.
D.2 Additional ablation study weighting parameter
We provide additional ablation study the weighting parameter We perform experiments using vision dataset (CIFAR-10) and language dataset (Wikipedia). For CIFAR-10, vary from 0.1 to 50 for KL and JS divergences and run for 200 epochs. We perform the same experiments on KL when = 1 for 800 epochs and observed an accuracy of 83.58% (lower than SimCLR). This observation further provides empirical evidence that KL-MICL is different from InfoNCE and needs special tuning on a to perform well
Table 8 justifies our choice of performance when choosing a 40. For the Wikipedia dataset, we observe that a much bigger is desirable for maximum performance. We vary from to 10 for KL and Pearson x2 divergences and run for 1 epoch, as there is large number of samples (10%) in the language dataset. Table 9 justifies our choice of in Table 7, where the best performance is reached at a = 409600. Such an is found by starting from 100 and doubling iteratively.
Table 7, where the downstream test accuracy indicates the optimal
Table 8: Ablation study on weighting parameter a for KL and JS divergences on CIFAR- 10, We compare test accuracies (%) for different choices of a using k-NN evaluation.
0.1
1
10
20
30
40
50
KL
13.16
77.60
83.53
83.77
81.39
84.19
82.77
JS
8.84
73.31
81.39
83.21
83.49
84.06
82.61
23

---

Published in Transactions on Machine Learning Research (10/2023)
Table 9: Ablation study on weighting parameter a for KL and Pearson x² divergences on Wikipedia. We compare the semantic textual similarity (STS) via the Spearman's correlation for different choices of weighting parameter a.
a
10
102
10a
104
109
409600
10%
KT. 67.52
70.47
72.43
75.12
76.90
77.78
78.02
77.78
Pearson
64.58
67.78
71.58
74.03
74.95
74.40
77.59
76.47
CIFAR
KL
17
Pearson
VLC
Tsallis
1055
100
125
epach
Figure 6: The training loss curves of varions f-divergences on CIFAR-10 with 200 epochs.
D.3 Additional experiments
Our final experiments show that f-MICL stable terms of training and the variation of performance is well controlled.
Training stability We depict the training loss curves of different divergences on CIFAR-10 in Figure 6. This figure shows that our methods exhibit stable training dynamics with fast convergence.
k-NN evaluation and additional f-divergences We show more detailed results of Table 2 Table 10, including experiments using k-nearest neighbour (A-NN) evaluation. Additionally, we have added exper- iments on other f f-divergences such as Squared Hellinger and Tsallis-a divergences.
Verification of Assumption 3 Throughout paper made assumption (Assumption 3) that the joint feature distribution is Gaussian kernel. However, is it valid assumption? In this experiment, we try to show some empirical evidence that this assumption approximately holds in practice. Recall that Assumption 3 says that the joint feature distribution of positive pairs is:
11) x exp (-1ley y
|| 20
Pg (x
(31)
if the RBF kernel is Gaussian. In order to estimate the joint density of positive pairs, we use normalizing flows, which a popular method for density estimation. Popular normalizing flow models include NICE (Dinh al., 2014), RealNVP (Dinh al., 2017) and Glow (Kingma & Dhariwal, 2018). Equation (31) is equivalent the following:
|2 1
log (29, (L.,,y9 y2)
const,
(32)
20
and thus it suffices to show that the log likelihood is linear w.r.t. the distances between each positive pair. Figure 7, we plot the relation between log Py, estimated by RealNVP with Gaussian prior, and the squared
Codd available wttp:ttthb..cc.ootttteeee http:/thhhh...o/tttttttrr//ttrra
24

---

Published in Transactions on Machine Learning Research (10/2023)
SimCLR
KL-divergence
6700
6700
6600
6600
log density
log density
6500
6500
6400
6400
6300
0.000
0.002
0.004
0.006
0.008
0.000
0.002
0.004
0.006
0.008
squared distance
squared distance
Figure 7: Experiment for verifying Assumption 3. We draw the relation between the squared distances | 1 and the averaged lo with RealNVP. The features are learned by different algorithms trained CIFAR-10. (left) SimCLR; (right) f-MICL with the KL divergence.
Test accuracy (%) on the smaller vision datasets. For the Wikipedia dataset we evaluate the semantic textual similarity (STS) via the Spearman's correlation. For each method, we take three separate runs, and show the mean and stand derivation.
Table 10:
Baselines
f-MICL
Evaluation
Dataset
MoCo SimCLR Uniformity
KI
JS
Pearson
SH
Tsallis
VLC
90.30
89.71
90.41
90.61
89.66
89.35
89.52
89.15
89.13
CIFAR-10
+0.19
+0037
+-0.66
+0.47
+0.28
+0.52
+0.25
++0.42
+0.33
83.69
82.97
84.44
85.33
85.94
82.64
82.80
84.79
85.94
Linear
STL-10
#0.22
+0.32
+0.19
+0.39
+0.17
+0.37
10.27
+0.34
10.72
35.72
30.56
41.20
34.95
42.98
43.45
40.83
32.99
38.65
TinylmageNet
#0.17
+0.28
#0.19
+0.20
10.18
+0.54
10.67
+0.49
10.45
88.70
84.92
89.42
89.34
89.12
89.44
88.13
89.18
89.15
CIFAR-10
#0.22
+033
+0.18
+0.57
#0.38
#0.60
#0.18
#0.62
+0.23
78.77
74.34
79.57
79.99
80.45
76.64
78.31
76.11
79.34
K-NN
STL-10
+0.25
#0.14
+0.52
+0.47
+0.19
+0.26
+0.33
#0.24
+0.62
36.22
29.60
37.44
36.17
38.20
38.14
35.56
33.11
35.21
Tiny ImageNet
+0020
+0.39
+0.27
+0.29
+0.26
+0.63
+0.77
+.0.2
+0.33
77.88
77.40
77.95
78.02
76.76
77.59
73.60
72.68
55.07
STS
Wikipedia
10.15
#0.12
+0.08
10.13
+0.09
+0.12
10.10
#0.09
10.13
distances ||2-9 1 The representations are learned by SimCLR, and f-MICL with the KL divergence on the CIFAR-10 dataset. To alleviate the estimation error in the flow model, we divide the distances into small intervals and compute the average lig-liieihood within each interval. We can see that the log- likelihoooo is roughly linear w.r.t. the squared distance, and thus verifying our Assumption 3. Moreover, in Figure also show the estimation by training RealNVP with uniform prior. Over 5 different combinations of random data augmentations, observe that the linear relationship generally holds and the estimations by Gaussian prior and uniform prior are very similar.
25

---

Published in Transactions on Machine Learning Research (10/2023)
Gaussian
Uniform
6700
6700
1og density
6600
log density
6600
6500
6500
6400
6400
6300
0.002
0.004
0.006
0.008
0.004
0.006
squared distance
squared distance
6700
doncity
6600
log dencity
6600
6500
6500
6400
0.000
0.002
0.004
0,006
0.008
0.000
0.002
0.004
0,006
0.008
squared distance
squared distance
6700
6700
log density
6600
6600
6500
log
6500
6400
6400
0.000
0.002
0.004
0.006
0.008
0.000
0.002
0.004
0.006
0.008
squared
distance
6700
6700
density
log density
8600
8500
6500
6400
0.000
0.002
0.004
0.006
0.008
0.000
0.002
0.004
0.006
0.008
squared distance
squared distance
6700
6700
log density
log density
6600
6500
6500
6400
0.000
0.002
0. 0.004
0.006
0.008
0.000
0.002
0.004
0.006
0.008
squared distance
squared
Figure 8: Additional exporiment for verifying Assumption 3. Here take different combinations of random data augmentations and draw the relation between the squared distances || y/ and the averaged log Pa with Re..NVP. (left column) Gaussian prior; (right column) Uniform prior. The features are learned by SimCLR trained on CIFAR-10.
26


Reviews:
[{"review_id": "ERLpngL65D", "reviewer": "Reviewer_zeWK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. f-MICL is a generalization of Shannon mutual information. According to Table 2, the KL-based mutual information does not always perform the best and may underperform JS/Pearson/VLC-based mutual information. Thus, f-MICL offers more options to practitioners and people can search for the most appropriate loss function for their downstream application.\n2. Experiment results show that f-Gaussian similarity is beneficial.\n\nWeaknesses\n1. The paper is highly relevant to Deep InfoMax (https://arxiv.org/pdf/1808.06670.pdf). For example, Deep InfoMax (DIM) proposed to control the characteristics of the representation by matching the push-forward marginal with a prior distribution (DIM used the uniform distribution in the experiment and mentioned that Gaussian distribution under-performs uniform as a prior). The f-Gaussian idea proposed by the author is conceptually similar to the idea in DIM. By adopting the f-Gaussian similarity function, the author intrinsically assumes that the marginal distribution is a Gaussian. This also encourages the compactness of the learned representation like the regularization term in DIM. Moreover, DIM generalizes KL-based MI to JS-based MI via the technique in the f-GAN paper. The idea of extending MI to f-MICL is very incremental since prior work has already tried to extend MI via f-divergence.\n2. Although the KL-based MI does not always perform the best, no f-divergence clearly wins over the others according to Table 2 and Table 3. The author mentioned that \"We will discuss how to choose the best f empirically in §4.\" but I haven't seen the algorithm of choosing the best f for a given dataset. In real-world applications, trying out all variants can be computationally impractical.\n3. Adopting RealNVP for verifying the Gaussian assumption in Figure 1 is questionable since RealNVP usually assumes the latent distribution follow a Gaussian.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studied a new class of contrastive representation learning objective called f-MICL. f-MICL replaces the KL divergence in Shannon mutual information with the f-divergence. The representation learning framework adopted in the paper is the same as InfoNCE except that mutual information is generalized to f-MICL. In addition, the author proposed the f-Gaussian similarity function by assuming that the joint feature distribution is proportional to a Gaussian. Experiments on vision and text datasets demonstrated the benefit of f-Gaussian similarity over the cosine similarity.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The author needs to emphasize its novelty over Deep InfoMax, and compare f-Gaussian with regularizing the marginal distribution with a compact prior distribution.\n2. I think the author needs to revise Figure 1 or find other ways to justify the validity of assuming the marginal feature distribution be a Gaussian.\n3. The author needs to discuss how to choose the best f empirically.", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XpaoydKCyJ", "reviewer": "Reviewer_NShA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n    - Experiments are quite extensive, covering both vision and language datasets and several popular baselines. Results are averaged over 3 seeds.\n    - The paper also compares different choices of the $f$-divergence and similarity function.\n    - The paper provides empirical support for its assumption (Fig 1)\n    - The paper is generally well written.\n- Weaknesses\n    - I thought there should be $\\text{sup}_{s \\in \\mathcal{F}}$ in Eq 4. To approximate this $\\text{sup}$, we use $s = s_f$ defined in Eq 6.\n    - Fig 1 only shows that Assumption 3 approximately holds for a specific feature encoder $g$ (that is trained by SimCLR loss). However, the paper seems to suggest that Assumption 3 should hold for arbitrary $g$.\n    - I don't see how Eq 6 is derived from Eq 5 and Assumption 3.\n    - I'm wondering why the negative samples should use the same data augmentation (Eq 8). Can they use different data augmentations?\n    - What is \"max\" in Fig 2?\n    - The $\\alpha$ values used in vision ($\\alpha = 40$) and language ($\\alpha = 409600$) tasks are drastically different. Can you provide some guidelines on how to choose $\\alpha$?\n    - Some text are colored and I don't see why. Please fix that.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes $f$-MICL, a framework for contrastive learning. It generalizes InfoNCE by using $f$-divergence instead of KL divergence. This provides a wide choice of objectives as we can use different $f$-divergences, and it is shown that several existing contrastive losses can be interpreted as a specialization of the $f$-MICL by choosing a specific $f$-divergence and proper hyperparameters. The framework also provides theoretical guidance to choosing the optimal similarity function. Experiments show that $f$-MICL achieves comparable or better linear evaluation accuracy across multiple datasets compared to several popular self-supervised learning methods. It is also shown that the proposed $f$-Gaussian similarity works better than cosine similarity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall I think the proposed framework makes sense. There are some clarity issues that I have listed in Weaknesses, and I hope the authors could address them, especially the second and third point.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X7eL4qFDfE", "reviewer": "Reviewer_DHPy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n\n* The paper suggests a rich family of SSL loss functions, naturally generalizing InfoNCE.\n* Comprehensive benchmark on several different datasets.\n* Based on the existing theory of f-divergences.\n\nWEAKNESSES:\n\n* Some parts are unclear -- see below.\n\nOverall I think the paper can be accepted to TMLR after some straightforward revisions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In their paper \"f-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning\" the authors suggest a generalization of the InfoNCE loss function commonly used in self-supervised learning (SSL) methods such as SimCLR. They show that they can set up meaningful loss functions using many different f-divergences, obtaining a whole family of SSL losses. Empirically, they find that different losses perform differently well on different datasets (even though the differences are mostly moderate).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "MAJOR ISSUES:\n\n* I was confused by the relationship between p+ (distribution of positive pairs) and p(x,y) (joint distribution). Are they the same thing? For example, in Assumption 3 you say that it's reasonable to assume that p(x,y) decreases with distance between x and y. This only makes sense if x and y are a positive pair. But you don't explicitly say that, and in general you use p+ notation for the distribution of positive pairs. Can you elaborate on the relationship between p+ and p(x,y) somewhere early on in Section 3?\n\n* In Section 4, the quality of representation is assessed in which space? In SimCLR there is usually a projection head after the ResNet; the loss operates on the output of the projector, but the evaluation (linear classification accuracy) is assessed on the output of the ResNet before the projector. From your description it is unclear how you did it. Please clarify.\n\n* Section 4.1 says \"All models are trained for 200 epochs\". However, Table 6 in the appendix says most models were trained for 800 epochs. Also App E2 says Table 7 used 200 epochs, and Table 7 reports lower accuracy for KL loss with alpha=40 than Table 2. So how many epochs were used? 200 is very short training for SimCLR.\n\n* Section 4.2: Please report in the main text the alpha that you used (currently you just say that you \"can tune\" alpha with KL-MICL to outperform InfoNCE). It's interesting that the value is 40 because it's so large, which makes the loss very different from InfoNCE (with alpha=1).\n\n* In Table 7 you show the results using KL loss and alpha=1 which should be close to SimCLR. You get 77.6% accuracy on CIFAR-10, which is much lower than what one get with SimCLR on CIFAR-10. But that's only using 200 epochs. If you run KL loss with alpha=1 for longer, e.g. for 800 epochs, what performance would you get? Would it be similar to what you report for SimCLR in Table 1 (89.71%)?\n\n\nMINOR ISSUES:\n\n* Eq 2: is this is a bound of I(X,Y) then maybe it would be clearer if you write it as an inequality, e.g. \"I(X,Y) >= ...\".\n\n* The very end of section 2: At this point I was wondering if the inequality in Eq (2) becomes an equality if you plug s* from Eq (3) into Eq (2). I think the answer is yes, and I think it can be stated more directly.\n\n* Figure 1: The caption should say that x^g and y^g here refer to positive pairs, i.e. distinct augmentations of the same image. This wasn't immediately clear to me.\n\n* Please add some brief description of how RealNVP works and what it does.\n\n* Table 4: Maybe change \"Gaussian\" into \"f-Gaussian\" for clarity?\n\n* Section 4.3: here you change similarity metrics, but it's unclear what loss functions were used? Do you always use the corresponding loss function? So e.g. f-Gaussian similarity for JS loss was used with JS loss, f-Gaussian similarity for Pearson loss was used with Pearson loss etc.? Please clarify.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considered a generalized view of contrastive learning by extending the KL divergence and dot product kernel to a generalized setting (f-divergence and a general kernel view). Most reviewers appreciated the insights and results and recommended acceptance. However, there are some concerns raised by reviewers. After reviewing the paper and the concerns, I think the current version can be accepted if some very minor points are clarified. Below are detailed thoughts on the concerns:

- [About novelty]. *Generalizing MI to f-MI is incremental since it has been widely adopted in previous papers...* I would think this is fine since the main scope of TMLR is to accept papers with certain interesting insights. I would agree that the extension is interesting because it provides a general view for understanding contrastive learning.

- [About f-Gaussian similarity function] I agree that the f-Gaussian similarity seems relatively strong, while it really depends on the *data assumption*. Based on Fig. 1, the current approximation is empirically validated on the benchmark dataset. I would suggest that  authors discuss this limitation.

Other suggestions. There is no need to fix them in the current version, but it would be great to have a short discussion as a limitation/future work.

- On the inherent limitation of variational terms of f-divergence. [1] pointed out the statistical limitation of using variational forms to measure mutual information. I would think that there might be similar limitations when using the duality term of f-divergence. It might be an open problem and interesting to explore.

- Copula in joint distribution modeling. Concerns about f-Gaussian similarity function are quite related to Copula [2] by forming different joint distributions by giving marginal distributions. I would think that the current approximation could be considered as a specific setting under Gaussian Copula. Moreover the positive and negative pair have specific dependence relations, it might be quite interesting to explore a general form.

Based on these, I would recommend an acceptance. 

References 

[1] Formal Limitations on the Measurement of Mutual Information. AISTAS 2019

[2] Introduction to Copula https://bochang.me/blog/posts/copula/
Decision: Accept (poster)