Self-supervised learning with rotation-invariant kernels
Authors: Léon Zheng
Gilles Puy
Elisa Riccietti
Patrick Perez
Rémi Gribonval
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Self-supervised learning
maximum mean discrepancy
rotation-invariant kernel
hypersphere

Abstract: We introduce a regularization loss based on kernel mean embeddings with rotation-invariant kernels on the hypersphere (also known as dot-product kernels) for self-supervised learning of image representations. Besides being fully competitive with the state of the art, our method significantly reduces time and memory complexity for self-supervised training, making it implementable for very large embedding dimensions on existing devices and more easily adjustable than previous methods to settings with limited resources. Our work follows the major paradigm where the model learns to be invariant to some predefined image transformations (cropping, blurring, color jittering, etc.), while avoiding a degenerate solution by regularizing the embedding distribution. Our particular contribution is to propose a loss family promoting the embedding distribution to be close to the uniform distribution on the hypersphere, with respect to the maximum mean discrepancy pseudometric. We demonstrate that this family encompasses several regularizers of former methods, including uniformity-based and information-maximization methods, which are variants of our flexible regularization loss with different kernels. Beyond its practical consequences for state of the art self-supervised learning with limited resources, the proposed generic regularization approach opens perspectives to leverage more widely the literature on kernel methods in order to improve self-supervised learning methods.

Published as a conference paper at ICLR 2023
SELF-SUPERVISED LEARNING INVARIANT KERNELS
WITH
ROTATION-
Léon Zheng
Gilles Puy
Elisa R
Patrick Pérez!
Rémi Gribonval
1 valco.ai, Paris, France
2 Univ Lyon, EnsL, UCBL, CNRS, Inria, LIP, F-69342, LYON Cedex 07, France.
ABSTRACT
We introduce a regularization loss based on kernel mean embeddings with rotation-invariant kernels the hypersphere (also known dot-product kernels) for self-supervised learning of image representations, Besides being fully compet- itive with the state of the art, our method significantly reduces time and memory complexity for self-supervised training, making it implementable for very large embedding dimensions on existing devices and more easily adjustable than pre- vious methods to settings with limited resources. Our work follows the major paradigm where the model learns be invariant some predefined image trans- formations (cropping, blurring, color jittering, etc.), while avoiding a degener- ate solution by regularizing the embedding distribution. Our particular contri- bution is to propose loss family promoting the embedding distribution to be close to the uniform distribution on the hypersphere, with respect to the maxi- mum mean discrepancy pseudometrie. We demonstrate that this family encom- passes several regularizers of former methods, including uniformity -baeed and information-maxinn maximizaiiiiiiiiiiniiiio methods, which are variants of flexible regulariza- tion loss with different kernels. Beyond its practical consequences for state-of- the-art self-supervised learning with limited resources, the proposed generic reg- ularization approach opens perspectives to leverage more widely the literature on kernel methods in order improve superviee learning methods.
INTRODUCTION
Self-supervised learning a promising approach for learning visual representations: recent methods (He et al., 2020; Grill et al., 2020; Caron et al., 2020; Gidaris et al., 2021) reach the performance of supervised pretraining in terms of quality for transfer learning in many downstream tasks, like classi- fication, object detection, semantic segmentation, etc. These methods rely on some prior knowledge on images: the semantic of an image is invariant (Misra Maaten, 2020) to some small transforma- tions of the image, such as cropping, blurring, color jittering, etc. One way to design an objective function that encodes such an invariance property is enforce two different augmentations of the same image to have similar representation (or embedding) when they encoded by the neural network. However, the main issue with this kind of objective function is to avoid an undesirable loss of information (Jing al., 2022) where, e.g., the network learns represent all images by the same constant representation. Hence, one of the main challenges in self-suuervised learning is to propose an efficient way to regularize the embedding distribution in order avoid such collapse.
Our contribution is to propose a generic regularization loss promoting the embedding distribution to be close the uniform distribution on the hypersphere, with respect to the maximum mean dis- crepancy (MMD), a distance on the space of probability measures based on the notion of embedding probabilities reproducing kernel Hilbert space (RKHS), using the so-called kernel mean embed- ding mapping. Inspired by high-dimensional statistical tests for uniformity that are rotation-invarian (Garefa-Poee Portugués & Verdebout, 2018), we choose to embed probability distributions using rotation- invariant kemelss on the hypersphere (dot (dot-product kernels). i.c., kernels for which the evaluation for two vectors depends only on their inner product (Smola et al., 2000). This paper shows that such an approach leads to important theoretical and practical consequences for self-supervised learning.
Code: https: / http//gittttt wetittu..ccm.. Teoai fririk

---

Published as a conference paper at ICLR 2023
Uniformity via MMD minizahation
K)(u.v) o(u'v]
(o(u), #(v1l
(0)
[x
fo
fw
Hypanshate
Minimize Egelistinno
norza
(0)
gro
{el
MMD metintation
Uniformity MMD minizaioation
E (
T7
T milaam distributiios S
Figure I: Self-supervised learning with intation-invarian kernels. The invariance criterion min- imizes the f2 f2-distance between two normalized embeddings {a, (x,("))) J==122 of two views of the same image encoded by the backbone fo and the projection head Ow- To avoid collapse, the embedding distribution is regularized to be close to the uniform distribution on the hypersphere, in the sense of the MMD associated to a rotation-invarian kernel KC(u, v) = yp(" v) defined on the hypersphere.
We demonstrate that our regularization loss family parame- terized by such rotation-invariant kernels encompasses several regularizers of former methods. As illustrated in Table I, they are variants of our generic loss with different kernels: the quadratic kernel yields the general sample-contrastinn criterion of Garrido et al. (2023) that englobes many contrastive learning methods like (HaoChen et al., 2021) (ef. Appendix A.2); the radial basis function (RBF) kernel yields the uniformity loss of Aligmment& & Uniformity on the Hypersphere (AUH) (Wang & Isola, 2020); the generalized distance kernel (cf. Example 2) yields one of the regularization used in PointContrast (Xie et al.,
Correspondence be- tween kernel choices (C(:,)) in our generic regularization loss and regularizers of former methods.
Table 1:
K(u,v) [u
Method
Contrastive
du vi
AUH PoiniContrast
C- [u - vila
b.uvT + byy
Analog VICReg (cf. Section 3.3)
2020); and linear combination of the linear kernel and the quadratic kernel yields regularizer that promotes the covariance matrix of the embedding distribution to be proportional to the identity ma- trix, similarly to information-maxinn words, these former methods turn out to be particular ways of minimizing the MMD between the embedding distribution and the uniform distribution on the hypersphere during training, with var- ious specific kernel choices. The proposed generic regularization approach opens perspectives to leverage more widely the literature on kernel methods in order to improve seff-supervised learning.
methods like VICReg (Bardes et al., 2022). In other
Numerically, we show in a rigorous experimental setting with separate validation set for hyperpa- rameter tuning that method yields fully competitive results compared to the state of the art, when
choosing truncated kernels of the form K(u,, v) Ei=o byPe(q: u v), with L E {2,3). be 2 0 for l E (O.... I.). where (e(g:) :) denotes the Legendre polynomial of order f, dimension q. To our knowledge, this kernel choice has not been considered in previous seff-supervision methods. There- fore, we introduce SFRIK (SelE-supervised learning with Rotation-Invarial Kernels, pronounced like "spheric"), which regularizes the embedding distribution to be close to the uniform distribution with respect to the MMD associated to such truncated kernel, as summarized in Figure 1.
Importantly, our method significantly reduces time and memory complexity for self-supervised training compared to information-maxinn maximization methods. Due to the kernel trick, the complexity of SFRIK's loss is quadratic in the batch size and linear in the embedding dimension, instead of being quadratic in VICReg. In practice, SFRIK's pretraining time is up to 19% faster than VICReg for an embedding dimension 16384, and can scale at dimension 32768, as opposed to VICReg whose memory requirement is too large at this dimension for machine with 8 GPUs and 32GB of memory per GPU, Hence our work opens perspectives in self-supervised learning on embedded devices with limited memory like in (Xiao et al., 2022). We summarize our contributions as follows:
We introduce a generic regularization loss based on kernel mean embeddings with rotation- invariant kernels on the hypersphere for self-supervised learning of image representations. We show that our loss family encompasses several previous self-supervised learning methods, and information-maxinne maximization methods.
like uniformity-basee
We numerically show that SFRIK significantly reduces time and memory complexity for self- supervised training, while remaining fully competitive with the state of the art.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Instance discrimination methods typically rely on a contrastive loss that behaves asymptotically like an alignment and uniformity loss the hypersphere in the limit of infinite samples. Our contribu- tion is to formalize and generalize existing uniformity- -easee methods by using kernel mean embed- dings. To the best of our knowledge, the proposed kernel framework establishes the first connection between uniformity-based methods and Coormation-maiiizzant methods like VICReg.
Instance discrimination One way of learning image representations that are invariant to prede- fined image transformations (Misra Maaten, 2020) is to rely on an instance classification approach (Wu et al., 2018). Typically, contrastive learning (Oord et al., 2018: Hjelm et al., 2019; Chen et al., 2020a;b; He et al., 2020; Henaff, 2020) discriminates instances within a batch of sampled images using the noise contrastive estimator (Gutmann & Hyvärinen, 2010), by attracting embeddings of transformed images coming from the same image instance, and repulsing embeddings coming from different image instances. In practice, this estimator needs a large number of image representations in order to achieve good results, which requires a large batch size like SimCLR (Chen et al., 2020a) or memory bank (Wu al., 2018; He et al., 2020). In the limit of infinite samples, the contrastive loss is shown to behave asymptotically like the alignment and uniformity loss of AUH.
Uniformity on the hypersphere Existing uniformity- based methods avoid collapse by regulariz- ing the embedding distribution to be somehow close to the uniform distribution on the hypersphere, which has a high entropy. Bojanowski & Joulin (2017) perform this kind of regularization by align- ing the learned representations on fixed number of vectors sampled uniformly at random on the hypersphere. AUH maximizes the average pairwise distance between embeddings using an RBF kernel, in the spirit of energy minimization methods that address the problem of scattering points evenly on the hypersphere (Hardin & Saff, 2005; Liu et al., 2018; Borodachov et al., 2019). Al- though alternative high-entropy prior distributions (e.g.. the uniform distribution on the hypercube) can be used for regularization (Chen et al., 2021), encoding images into fz-normalized representa- tions helps to stabilize training (Schroff et al., 2015; Parkhi al., 2015; Liu et al., 2017).
Kernel mean embedding As contribution, our generic loss formalizes and generalizes these previous uniformity losses. by relying on kernel mean embeddings (cf. Appendix A.1) to mea- sure the distance between probability distributions on high- nigh-dimensional spaces, using the MMD pseudometric (Gretton et al., 2012; Li et al., 2015; Dziugaite et al., 2015; Briol et al., 2019) with rotation-invariant kernels on the hypersphere (Smola et al., 2000; Pennington et al., 2015; Lyu, 2017; Dutordoir et al., 2020). These tools are adapted for high-dimensional problems on the hypersphere whose geometry is different from the one in small dimension, as illustrated by García-Portt Porttu-Po & Verdebout (2018): many statistical tests for uniformity on the hypersphere, i.e., tests for rejecting the null hypothesis where a batch of normalized vectors is sampled from the uniform distribution on the hypersphere, are in fact precisely estimators of the MMD between the embedding distribution and the uniform distribution, for different kernels. Our kernel method for self-supervision is comple- mentary to (Li et al., 2021). in which the dependency between image instances and their embedding is maximized with respect to the Hilbert-Schmidi independence criterion (cf. Appendix A.3).
Information maximization Our generic regularization approach has the benefit of connecting uniformity-based and ixformttio-mmaximizatii 2021: Bardes et al., 2022). The latter are alternatives distillation methods (Grill et al., 2020; Gidaris et al., 2020; 2021; Chen & He, 2021; Caron et al., 2021) where a student network learns to predict the representations of a teacher network. In such methods, using various architecture tricks (like prediction head, stop-gradient, momentum encoder, batch normalization or centering) is shown empirically to be sufficient avoid collapse without instance discrimination, even though it is not fully understood how these multiple factors induce regularization during training (Richemond et al., 2020; Tian et al., 2021). Instead of using these tricks, information -amaaatiiiimmiiizzaitiio methods use a Siamese architecture and avoid collapse by maximizing the statistical information of a batch of embeddings, using a whitening operation (Ermolov et al., 2021), or an explicit regularization term making the covariance (Bardes et al., 2022) or the cross-correlation (Zbontar et al., 2021) matrix close to a scaled identity matrix. This paper shows that our generic regularization loss with an appro- priate kernel also promotes the covariance matrix of the embedding distribution to be proportional to the identity matrix. But in contrast to VICReg which explicitly computes the covariance matrix, method uses the kernel trick to significantly reduce complexity large embedding dimensions.
methods (Zbontar et al., 2021; Ermolov et al.,
3

---

Published as a conference paper at ICLR 2023
3 METHOD DESCRIPTION
Given an unlabeled dataset of images X; ~ P.i := (1.... N).
sampled independently from data distribution the goal is to learn backbone network fo parameterized by (e.g.. convolutional neural network) such that any new image x ~ P is encoded by good representation fo(x) whose quality evaluated in several downstream tasks (see Section 4).
3.1 INVARIANCE AND UNIFORMITY FOR SUPERIIPERIIIION
Our self-supervised learning method (see Figure 1) follows the principle of the recent methods like SimCLR or VICReg. During self--supr training, each image x; is augmented using two different random transformations ((1) and ((2) sampled from a distribution T, which yields two views x, (1) = (x;; and x, := (x;)(;;) of the image x;. Two representations z (1) (v 1,2)
are obtained by encoding each x, (*) with the backbone fo and éz-normalizing the resulting feature
vector. For given subset of indices / S [N], we write Z, (v) = (z. (")eer The backbone so
trained by minimizing the total objective function:
E4 P BIGIN] (Z, (1) Z, (2)),
where batches are drawn at random with a prescribed batch size, and the loss é is a weighted sum involving an alignment term la and uniformity term Lu, in the spirit of AUH:
(Z Z (2) ta [ (1) Z (2) 0.5 (Z, ()) (Z, (2))):
> 0 is hyperparameter that tunes the balance between the two terms. The loss la enforces the invariance property of the model, and defined for a batch C [N] of cardinality as:
(Z) Z (2))
Σ 1 SET
z, (2)
(3)
|
Our main contribution is in the choice of the uniformity term Lu detailed in the rest of the section. Note that instead of applying the loss (2) to the output of fe (called image representation), we add projection head 9w (a multi-layer perceptron) parameterized by the output of fo and apply (2) at the output of Iw (called image embedding). This common practice (Caron et al., 2020; Grill et al., 2020) improves the performance in the downstream tasks. Therefore, denoting S9-1 1 the unit hyper-
sphere in IR*, the image embedding actually reads z, (o) := (gw (x, (*)/ (gw fe) ()) S4-1, Both In and fe are jointly trained without supervision by minimizing (1) using a stochastic mini-batch algorithm. After training, Iw is discarded and only fo is kept for the downstream tasks.
3.2
UNIFORMITY LOSS VIA MMD MINIMIZATION
We continue by explaining our generic kernel formulation of b, using the MMD pseudometric and Itation-invarian kernels. Then we provide examples of such kernels and describe our kernel choice.
3.2.1 MMD PSEUDOMETRIC AND ROTATION-INVARIA KERNELS
Our uniformity loss relies divergence in the space of probability distributions based on a positive definite kernel K. defined on some space X. Denoting H the corresponding RKHS with norm ||. 1l the MMD between two probability distributions Q1, Q2 on X' can be expressed as the distance in || I between their kernel mean embeddings (Borgwardt et al., 2006; Muandet et al., 2017):
MMD(Q, Q2) = | K(u,, (dQ
/ (dQzz))
(4)
We propose to use this pseudometric to measure the distance between the probability distribution of the embeddings z, (v) (u = 1, 2) and the uniform probability distribution on the hypersphere S defined by U = 0--1 | 1 where a denotes the normalized Hausdorff surface measure on 19-1 and S := S do = 2x 9 /T(3) )) is the surface area of ST-1, with I denoting the Gamma function. Intuitively, good choice of kemel for measuring the distance (4) should
4

---

Published as a conference paper at ICLR 2023
distinguish any probability distribution from the uniform distribution. Inspired by statistical tests for uniformity that are aozation-ivvarian (García- Porttiuée & Verdebout, 2018), we propose to use rotation-invariant kernels on X' := S of the form K(u, v) sou v) with yp continuous function defined on [-1,]] 1] (Smola et al., 2000). The following theorem characterizes the form of function yp that ensures positive definiteness of K. and thus that (4) is a valid pseudometric.
Theorem (Schoenberg (1942, Theorem 1)) The kernel K(u, v) == qp v) on X := S4-1 with (p continuous is positive definite if, and onty if. the function ip admits an expansion:
(t))
bePe(g;t).
with be 2 0.
(5)
where Pelg;t) = A (-4) K (1-8)8-2 (((--211T(--2 is the Legendre (or Gegenbauer) polynomial of degree f in dimension q (Müller, 2012, (2.32)).
As we are interested in measuring the distance between the embedding distribution and the uniform distribution on the hypersphere U, we compute the kernel mean embedding of TJ for a kernel satisfy- ing the condition of Theorem I using the following known result used, e.g., implicitly in (Brauchart et al., 2014). As we could not locate a formal proof, we provide one in Appendix B.1.
Lemma 2 Let K(u, v) == vv(Tvv be enaatin-invariinn kernel X S where ip ad- mits the expansion (5). The kernel mean embedding of the uniform distribution U on S is constant: Jsa-1 K(u, v) du(u) = bn E R for all v 8 The kernel mean embedding of any probability distribution Q defined on the hypersphere satisfies: Jsv-1 K(u..) dQ(u) by + Scr-K((u, .) dQ((),, where K(u, v) = p(u v) for any u, E S551 with ; = ΣtS by be(r
Using Lemma 2 in (4) yields MMD(Q, U) 11 II R(u, ))QQ()))x for any probability distribu- tion on 89-1 Then, by the reproducing property in the RKHS H. the squared MMD satisfies, for any rotation-invarnee invariant kernel IC verifying the condition of Theorem 1:
MMD U) = EExx'~ [xc (z, z')
with z, z' i.i.d.
(6)
3.2.2 ESTIMATOR OF THE SQUARED MMD AND KERNEL CHOICES
The proposed uniformity loss lu for self-supervision is a biased estimator (Gretton et al., 2012) of MMD?(Q. (Q, U) in (6). Given a batch Z, := (zi sampled from Q our uniformity loss is:
Ey (Z,) MMD (Q. U: (z))
ΣΣ SEI VEl
K(2,, 2.) TT
1
ΣΣ
(z, n.). SET YET
In our framework, any intaaiiannn kernel satisfying the condition of Theorem can be used to compute (7) and train a self-supervised model by minimizing (1). The uniformity term (7) can be interpreted as an energy functional (Brauchart et al., 2014): minimizing the average pairwise energy quantified by K tends seatter evenly the embeddings on the hypersphere. We now give examples of kernels that can be used for this uniformity term. This illustrates that our framework offers unification of several strategies for selffsupervision.
Example 1: RBF kernel. Using IC(u, v) = e -'lluvvilé (with t > 0) in the uniformity term Ey (7) yields the regularization term from AUH, with the only difference that AUH uses the logarithm of the energy functional as their uniformity loss.
Example 2: Generalized distance kernel. It is defined as KCu,, v) := C - ||u v 20-9+1 with g-1
g+1 and C > 0 sufficiently large (Brauchart et al., 2014). A variation of this kernel choice is, e.g., used in the hard-contrastive loss of PointContrast for self-supervision on point clouds.
2 < < -
Example 3: Truncations of the Laplace-Fourier series. A truncated kernel up to order T (Brauchart et al., 2014) is kernel KC(u, v) = El bePe u Tv). with by 2 0 for & = O..... L. admits a closed-form expression given by the definition of Legendre polynomials Pı -) in The- orem 1, e.g., t) t, Pz(g,t) af-1 kernel choice in Section 4, since it has never been considered in previous self-supervision methods.
9-1 t) = ( We explore numerically this g-1
5

---

Published as a conference paper at ICLR 2023
The expansion of o in Legendre polynomials (5) for the RBF (Example 1) and the generalized distance kernel (Example 2) verifies by > 0 for each integer 8 (see Appendix B.2). By (Micchelli et al., 2006, Theorem 10), this is necessary and sufficient condition for rotation-invariann kernel to be universal, and universality is a sufficient condition for injectivity of the corresponding kernel mean embedding mapping, i.e., the kernel is characteristic (Fukumizu al., 2004). The benefit of this property is to guarantee that the uniform distribution U is the unique solution to the minimization problem: min (Q, D) Q a probability distribution on 14-1). In contrast, the truncated rotation-inve invtrrnnn kernel up to an order L (Example 3) is not universal. Yet, our experiments in Section 4 show that truncated kernels up to order L 2, provide better results than, e.g., AUH whose uniformity loss is based on the RBF kernel.
In summary, the uniformity loss in our method, called SFRIK, corresponds to (7) with a truncated kernel up to order T, 3 and satisfies:
1
ΣΣ
biz z. + by ((z,2v) -
by (g + (2, ze )3 32, 2,'
(8)
1T8
q
REl PEI
where by 2 20, 1,2, are hyperparameters, and q is the dimension of the image embedding z;.
3.3 CONNECTION WITH INFORMATION-MAN COORMMTTON-MAXIIIZATOON METHODS
We now show that choosing an appropriate kernel in the proposed uniformity term (7) leads to a reg- ularizer that maximizes statistical measure of information analog to the one used in VICReg. To the best of our knowledge, this is the first connection made between uniformity-based and information- maximization methods. The regularization loss of VICReg is weighted sum between two terms:
max (0,7 )ar(())+))) c(Zr) --)==- Σ [C(Z))]
(ZZ) =
1SF3''S
for a batch of image embeddings Z == fii,, where 2 denotes the j-th coordinate of a (random) vector and E is a fixed small scalar. The variance term uZZr enforces the empirical variance Var(=/) == / Ever( - د² in each coordinate j = 1,..., ,q to be above a certain threshold y² > 0 (here 7 is the empirical mean of Z,). The covariance term cl (Z,) enforces the non-diagonal entries of the empirical covariance matrix C(Z,) = 1 (z z) z) to be zero
In order to connect VICReg and SFRIK, let us consider for simplicity a truncated kernel K(u, v) El_, belg u v) of order L = 2 (the reasoning would be the same if the kernel was not trun- cated), and assume bj. by > 0. By the addition theorem (Mäller, 2012, Theorem 2, $1), there exists a feature map & S4-1 - T. involving an orthonormal basis of spherical harmonics (homogeneous harmonic polynomials restricted to the hypersphere) of order and 2 such that (u)) Tvvv)= = K(u, v). Hence, the kernel mean embedding of a distribution in the associ- ated RKHS contains its first and second-order moments (see Appendix B.3). Therefore, denoting Níq. E) the dimension of the space of spherical harmonics of order l, dimension g, and defining N(q.9)
de : S9-1 + Ra,0)) z (Ye, (2)) (4.00 for l E (1.2} with
Na. {u- n² 1 1555
(10)
N9. = { u ud w² |1s < ) u (w 1
it is possible show (see Appendix BB3) that the squared MMD (6) can be written as
MMD = a1 ||M R2- (2)]1 [ where de : by| by E) for E (1, and M,, M2, are two lower triangular matrices with nonzero diagonal entries. Hence, when Q plays the role of the embedding distribution during self- supervised training, minimizing MMD2 U) in (11) as we propose for regularizing the embedding distribution promotes its expectation Ez~o z)] and its autocorrelation matrix Ez~q [zz | to be close
+ 02 (2
(11)
to 0 and 9 I,, respectively, where I, is the identity matrix. When MMD2 (Q) U) O, the covari- ance matrix is equal to Ez Ezz)) E| E(zz" E/z) and the two terms on the right-hand side of (11) are null.
q I because b,, by 0

---

Published as a conference paper at ICLR 2023
In conclusion, the regularization both in VICReg and SFRIK induces the embedding distribution to have covariance matrix with zero non-diagonal entries. The diagonal entries of the covariance matrix are encouraged to be equal to 1/q in SFRIK, and greater than y2 in VICReg (we recall that the image embeddings heer are not fz-normalized VICReg). However, one difference in terms of regularization behavior is that SFRIK encourages the expectation of the embedding distribution to be zero, as shown in the first term of (11). This is not the case for VICReg, as we can see in (9).
Finally, the memory and computational complexities for computing the uniformity term (8) in SFRIK are 0 and ((g/41), as opposed to (q²²) and ((12 for the variance and covariance terms (9) in VICReg. In the setting where SFRIK and VICReg work best, i.e., larger dimension q
and smaller batch size (T) SFRIK has the lowest memory and computational complexities. This computational advantage due to the kernel trick and it illustrated in Section 4.
EXPERIMENTS
We first demonstrate numerically that the regularization loss (8) SFRIK outperforms existing alternatives, in a rigorous experimental setting with subset of ImageNet- 1000 (Deng et al., 2009) for pretraining and a separate validation set for hyperparameter tuning. Then, we pretrain ResNet- 50 backbone (He et al., 2016) with SFRIK the full ImageNet dataset and show competitive results compared to the state of the art, with significant computational benefits during pretraining.
4.1 EXPERIMENTAL SETTING
The backbone fe is either ResNet-18 or ResNet-50, depending on the experiment. Following Zbon- tar et al. (2021), the projection head ghe is three-layer MLP made of two hidden layers with ReLU activation and batch normalization (loffe & Szegedy, 2015), and linear output layer. Unless other- wise specified, the size (number of neurons) of the two hidden layers is the same as the one, denoted g. of the output layer and the default value is q = 8192. The augmentations used for transforming images into views are the same as the ones used in VICReg. The backbone and the projection head are trained with a LARS optimizer (You et al., 2017). The weight decay fixed at 10-6, The learning rate scheduling starts with 10 warm-up epochs (Goyal et al., 2017) with a linear increase from O to Initial_r = base_r bs/256, where base_rr is called the base learning rate (Goyal et al., 2017) and bs is the batch size, followed by cosine decay (Loshchilov & Hutter, 2017) with a final learning rate 1000 times smaller than initial.r. For pretraining, we consider a 20% subset of ImageNet- 1000 (denoted by IN20%), like in (Gidaris et al., 2021), and 100% of ImageNet- 1000 (denoted by IN100%). In IN20%, we keep all the 1000 classes but only 260 images per class.
4.2 SFRIK'S S REGULARIZER OUTPERFORMS EXISTING ALTERNATIVE ON IN20%
Many existing self superviievn methods are based on the Siamese architecture and have the same form of training loss Ae. (Z, (1) Z, (2)) pe, (Z/ (1), Z (2)). This is the case of SimCLR, AUH and VICReg, for which Appendix B.4 gives the expression of the regularization loss f,. For SFRIK, following (2), we have μ 0.5 and (Z, (1) Z, (2)) = l, (Z, (1) C, (Z, with 6, given by (8).
Protocol.
To isolate the impact of l, on the quality of the learned representations, we (re)implement all these four methods in the setting of Section 4. to get rid of the influence of other design choices, like image augmentations or projection head architecture. We fix the batch size at 2048, and tune the base learning rate and hyperparameters specific to each method's loss. We also compare different embedding dimension E { [1024, 2048, 4096, 8192]. In order to perform an extensive hyperparam- eter tuning by grid search of each method for fair comparisons,
choose smaller backbone and a reduced dataset for pretraining, i.e., we pretrain a ResNet- 18 on IN20% for 100 epochs with all methods. Pretrained backbones are then evaluated by linear probing trained on IN20% with labels.
Number of hyperparameters. Note that in total SFRIK with L = 2 has as many hyperparameters to tune AUH VICReg, and SFRIK with L 3 has single additional hyperparameter.
Rigorous hy hyperparameter tuning. In contrast to the common practice in the literature where hy- perparameters are directly selected the evaluation dataset, we choose tune hyperparameters on

---

Published as a conference paper at ICLR 2023
Table 2: Linear probing on IN20% (top-1 accuracy) at different embedding dimensions q. All methods were pretrained on IN20% with a ResNet-18 18 for 100 epochs. We tuned all hyperparameters specific to each method and the learning rate. Symbol f indicates models that we retrained ourselves.
SimCLR
AUH
VICReg!
SPRIK
L
L 1-2
L
1024
45,2
45.3
40.6
45.2
q
45.8
45.9
44.0
45.9
4096
46.0
46.7
44.9
46.9
9 8192
46.1
46.8
46.0
27.7
47.0
47.5
a separate validation set that consists of another 20% subset of the ImageNet train set. We select the hyperparameters that yield the highest top-1 accuracy obtained by weighted cls-caassiiinttion (k = 20) (Wu et al., 2018) on this validation set, and we finally report the evaluation results by linear probing on the usual ImageNet validation set, which never seen during hyperparameter tuning.
Results. Table 2 shows that SFRIK optimal truncation order I. = 3 outperforms SimCLR, AUH and VICReg by at least 0.7 points at 8192. The gain in top-1 accuracy by linear probing between and L = is important, but is smaller between = 2 and = 3. This suggests that L > 3 is likely to marginally improve performance, while requiring more hyperparameter tuning, which is why we did not explore L > 3. We also remark that all methods benefit from an increase in embedding dimension g, including SimCLR which was originally introduced with smaller dimension. Appendix D.2 provides extra results for linear classification on Places205 (Zhou al., 2014) and VOC2007 (Everingham al., 2010) that further support our findings: SFRIK outperforms AUH while having the same pretraining complexity, and is fully competitive compared to VICReg with reduced pretraining complexity.
SFRIK at L
Ablation. Table 3 confirms empirically that truncated kernel is better than the RBF or the generalized distance kernel for the uniformity term (7). During tuning we observed that the trun- cated kernel performs well when the weights 82, by in (8) are larger than b,. e.g., (b,, b2 by) = (1, 40, 40) for 1 = 8192. This contrasts with the RBF and the generalized distance kernel for which the weights by decay polynomially with respect to l (see Appendix B.2). This suggests that it is important to focus more on order 2, 3 than on order in the Legendre expansion (5).
Table 3: Impact of kernel
choice in the uniformity term (7). Linear probing on IN20% of ResNet-18 pretrained on IN20% for 100 epochs, at q 8192.
Kemel
Top-1 1 acc.
RBF
41.3
Generalized distance
27.8
Truncated, L 3, cf. (8) 47.5
4.3 RESULTS FOR RESNET-50 PRETRAINED ON IN100%
Protocol. We pretrain a ResNet-50 with SFRIK on IN100% under the setting of Section 4.1, with a batch size of 2048. We study the impact of larger embedding dimension in SFRIK by considering projection head with two hidden layers of size 8192, and an output layer of size q {8192, 16384, 32768]. Truncation order is either L or L For comparison, we also pretrain a ResNet-50 with VICReg under the same setting with 8192. Similarly to the original paper (Bardes al., 2022), the alignment, variance and covariance weights are respectively 25, 25, 1, and the base learning rate is 0.2 for VICReg. All pretrained backbones are evaluated by: lin- ear probing on IN100%; linear classification on Places205 and VOC2007 in order to measure how the learned representations generalize to an unseen dataset; and semi-supervised learning with few labels of IN100% (backbones are fine-tuned for classification using 1% or 10% of labeled images).
Computational complexity. We show under this protocol that SFRIK's time and memory com- plexity during pretraining is significantly smaller than the one of VICReg for large dimensions. This allows us to scale SFRIK at dimension 16384 and even to 32768 for better results on downstream tasks.? 2 We measure the peak memory per GPU during pretraining 2048 and the pretraining wall time of both methods on a 8x AMD Radeon Instinct MI50 32GB:
IN100% with batch size of
I 'The performance gap hetween AUH and the RBF kernel only due to the presence the logarithm in AUH (cf. Example 1). Future work could clarify the role of this logarithm for regularization in self-supervision. We recall that the time and memory complexity identical for all methods downstream tasks.
8

---

Published as a conference paper at ICLR 2023
Table 4: Linear classification on IN Places205, VOC2007, and semi-supervised learning with few labels of IN100% (accuracy or mean average precision). Methods are pretrained on IN100% with ResNet-50. We only include methods relying on Siamese architecture with image augmentations limited to two views. The scores of methods marked with are from Chen & He (2021). The score of VICReg! was obtained by retraining the model ourselves. For each downstream task, we highlight in bold the best score among all backbones pretrained on 200 epochs.
Linear classification
Semi-supervisee
Method
Epochs
IN
Places205
VO007
1% labels
10% labels
Top-1
Top-5
Top-1
Top-5
mAP
Top-1
Top-5
Top-1
Top-5
SimCLR (Chen ct al., 2020a)
200
68.3
SwAV" (Caron
2020) (mo multi crop)
200
SimSiam (Chen & Hc, 2021)
200
70.0
VICReg! (Bardes et al., 2022) (g 8192)
200
70.0
89.3
54.1
83.4
84.9
49.4
75.1
65.9
87.2
SFRIK (/. 2. 8192)
200
70.1
89.3
53.8
83.0
85.1
46.6
73.3
65.7
87.3
SFRIK (L- 3.9 8192)
200
70.2
89.6
54.5
83.9
84.6
46,9
73.6
66.0
87.7
SFRIK (/. 2. 16384)
200
70.3
89.6
54.3
83.4
85.2
46.0
73.0
65.3
87.2
SFRIK (L- 2. 32768)
200
70.3
89.6
54.1
83.0
85.0
46.1
73.0
65.4
87.3
SFRIK 3.9 32768)
200
70.3
89.7
54.4
83.2
46.6
87.5
SFRIK (I. 2.9 8192)
400
70.8
89.9
54.4
83.5
85.7
47.8
74.3
66.4
88.0
at 8192, SFRIK is 8% faster than VICReg and needs 3% less memory per GPU;
Table 5: Peak memory per
GPU during pretraining of ResNet-50 on IN100% q 32768.
at q = 16384, SFRIK is 19% faster than VICReg and needs 8% less memory per GPU;
at q 32768, SFRIK is still 2% faster than VICReg run in the lower dimension 16384. It only requires 30.9GB per GPU while VICReg at q = 32768 needs more than the available memory. Ta- ble 5 emphasizes this memory advantage at reduced batch sizes.
Batch size
VICReg
SFRIK
(ratio)
22,5GB
10.3GB
512 1024
25.4GB
13.IGB
(1.9)
31.1GB
18.8GB
(1.7)
Results. Table 4 compares methods that have the same Siamese architecture and use the same image augmentations described in our protocol. For completeness, this table is completed in Appendix D.3 by evaluation results of other existing methods such as BYOL (Grill et al., 2020), OBoW (Gidaris al., 2020) and SWAV with multi-crop (Caron et al., 2020), which are not comparable to the meth- ods of Table 4 they use teacher-student architecture with momentum encoder and/or image augmentations with multi-scale cropping, and are beyond the setting of a Siamese architecture with only two views. Incorporating such designs in SFRIK is possible, and is left future work.
Table 4 demonstrates the competitiveness of SFRIK: it has the best accuracy for linear probing on IN100% among SimCLR, SwAV with no multi-crop, SimSiam and VICReg, and it performs better than VICReg for linear classification on Places205, VOC2007, and semi-supervised-leamiii with 10% of labels. We observe that SFRIK and VICReg offer different trade-off between performance on linear probing on IN100% and performance on semi-supervised learning with 1% of labels. But as shown in Appendix D.3, other methods like BYOL and SwAV with multi-crop similarly have a performance drop compared to VICReg on semi-supervised learning with 1 1% of labels, even though they perform better on linear probing. Future work will therefore involve understanding what specific ingredients of VICReg make it more robust for cemi-supervised learning with few labels. Ideally we could combine these ingredients with our generic kernel framework to design an improved version of SFRIK that can still benefit from its computational advantages over VICReg.
5 CONCLUSION
We proposed regularization loss family based on the MMD and -vation-invariant kernels. We demonstrated that several regularizers of former methods are indeed variants of our flexible loss with different kernels. This generic regularization approach allowed us to leverage degrees of freedom in rotation-invariant kernel design to improve self-supervision methods. In practice, using a truncated kernel, we derived from the proposed framework a fully competitive self-supervised pretraining method, SFRIK, which significantly reduces time and memory complexity during pretraining com- pared to information maxiiiizaiaiiiiiioonnn methods. Combining the approach with kernel approximation techniques such as quadrature rules or random feature expansions offers promising perspectives to further enhance the ability to perform self-supervised training with limited computational resources.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
The authors are concerned by the carbon footprint of deep learning research. To raise awareness of this issue in the community, we report the energy used for the computational resources of this project, which is approximately 12500 kWh.
REPRODUCIBILITY STATEMENT
In the interest of reproducible research, we provide our code and our pretrained ResNet-50 back- bones with SFRIK on IN100% at https: / / lighthubee t.ub. com/ vva 1 eooai/ nom/valle 2022). All details about our experimental setting can be found in either Section 4 or Appendix C. For the experiments IN20%, detailed in Appendix C.4, hyperparameters are tuned validation set different from the one used for evaluation, in contrast to the common practice in the literature where hyperparameter directly selected the evaluation dataset. All hyperparameters that yield the evaluation results reported in the paper are given in Appendix C.
frik (Zheng et al.,
separate
ACKNOWLEDGMENTS
This project was funded by the CIFRE fellowship Nº2020/1643 and supported in part by the Alle- groAssai ANR project ANR-I9-CHIA-O0ON This work was granted access to the HPC resources of IDRIS under the allocation 2021 1012940 made by GENCI. Experiments presented in this paper were carried out using the Grid' 5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https: /www///wwww//..////////// http://wwww.ddd000 fr).
REFERENCES
Sheldon Axler, Paul Bourdon, and Ramey Wade. Harmonic function theory, volume 137. Springer Science & Business Media, 2013. 19
Adrien Bardes, Jean Ponce, and Yann LeCun, VICReg: tion for self-supervised learning. In ICLR, 2022. 2, 3, 8, 9, 15, 20, 21, 25, 26, 27
regulariza-
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. ICML, 2017.
3
Karsten M. Borgwardt, Arthur Gretton, Malte Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J. Smola. Integrating structured biological data by kemnel maximum mean discrepancy. Bioinformatics, 2214)):49---7 2006.
Sergiy V. Borodachov, Douglas P. Hardin, and Edward B. Saff. Discrete energy on rectifiable sets.
Springer, 2019.
Johann Brauchart, Edward B. Saff, Ian H. Sloan, and Robert S. Womersley. QMC designs: Optimal order quasi Monte Carlo integration schemes on the sphere. Mathematics of computation, 83 (290))2821-2851, 2014. 5,
Francois-Xavier Briol, Alessandro Barp, Andrew Duncan, and Mark Girolami. Statistical infer- ence for generative models with maximum mean discrepancy. arXiv preprint arXiv:1906.05500 2019. 3
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
1, 4, 9, 25, 27
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 3, 22, 25
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICMI., 2020a. 3, 9, 25, 26, 27
10

---

Published as a conference paper at ICLR 2023
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. In NeurIPS, 2020b.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. In NeurlPS, 2021.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021. 3,9
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 2011. 24
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 7, 25
Vincent Dutordoir, Nicolas Durrande, and James Hensman. Sparse Gaussian processes with spheri- cal harmonic features. In ICMI, 2020. 3
Gintare K. Dziugaite, Daniel M. Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. In UAI, 2015. 3
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-
supervised representation learning. In ICML, 2021.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2):303-338, 2010. 8, 25, 26
Rong-En En Fan, Kai-Wei Chang, Cho-Jui Hsich, Xiang-Rui Rui Wang, and Chih-Jen Lin. LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research, 9:1871-1874, 2008.
Kenji Fukumizu, Francis R. Bach, and Michael I. Jordan. Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces. Journal of Machine Learning Research, (Jan): 73-99, 2004. 6
Eduardo Partua-Portuggés and Thomas Verdebout. An overview of uniformity tests on the hyper- sphere. arXiv preprint arXiv:1804.00286. 2018. 3,
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive self-supervised learning. In ICLR, 2023. 14
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations by predicting bags of visual words. In CVPR, 2020. 3,9
Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Online bag-of-visuullwords generation for self-supervised learning. In CVPR, 2021. 1, 7, 20, 21, 25, 27
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate large minibatch SGD: Training ImageNet in hour. arXiv preprint arXiv: /ee../006677777 2017.
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. VISSL. https: //tp:/ggtttaaa/ com/ hhtc////ccboooeeeaaccccceceoceee ciisss,, 2021. 21, 25
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical de- pendence with Hilbert Schmidt norms. In ALT, 2005. 14
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-773, 2012. 3,5
11

---

Published as a conference paper at ICLR 2023
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena
Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In NeurIPS, 2020. 1, 4, 9, 27
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. In NeurIPS, 2021. 2, 14
Doug P. Hardin and Edward B. Saff. Minimal Riesz energy point configurations for rectifiable d-dimensional manifolds. Advances in Mathematics, 193()):174-004, 2005. 3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 1,3
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding, In ICML, 2020.
R. Devon Hjelm, Alex Fedorov, Samuel Marchhannnn Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019. 3
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.
Yazhe Li, Roman Pogodin, Danica J. Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. In NeurlPS, 2021. 3, 14, 15
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In ICML, 2015.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In CVPR, 2017.
Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. In NeurIPS, 2018. 3
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017. 7
Yueming Lyu. Spherical structured feature maps for kernel approximation. In ICML, 2017. 3
Charles A. Micchelli, Yuesheng Xu, and Haizhang Zhang. Universal kernels. Journal of Machine Learning Research, 7(12), 2006.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In CVPR, 2020. 1,3
Claus Müller. Analysis of spherical symmetries in Euclidean spaces, volume 129. Springer Science & Business Media, 2012. 5, 6, 15, 16, 17, 18
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Bernhard Schölkopf. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends@ in Machine Learning, 1011- 2017. 4,
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 100.v:1807.03748. 2018.
12

---

Published as a conference paper at ICLR 2023
Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In BMVC, 2015.
Jeffrey Pennington, Felix Xinnan X. Yu, and Sanjiv Kumar. Spherical random features for polyno- mial kernels. In NewrIPS, 2015.
Pierre H. Richemond, Baan-eattien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, and Michal Valko. BYOL works even without batch statisties. arXiv preprint arXiv:2010. 10241, 2020.
Isaac J. Schoenberg. Positive definite functions on spheres. Duke Mathematical Journal, 9(1): 96-108, 1942. 5
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015. 3
Alex Smola, Zoltán Ovári, and Robert C. Williamson. Regularization with dot- dot-product kernels. In NeurIPS, 2000. 1, 3,5
Murray R. Spiegel, Seymour Lipschutz, and John Liu. Mathematical Handbook of Formulas and Tables. Schaum's Outlines, fourth edition, 2013. 17
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICML, 2021. 3
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. ICMI., 2020. 2
Hermann Weyl. On the volume tubes. American Journal of Mathematics, 61(2):661-472, 1939. 19
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-
parametric instance discrimination. In CVPR, 2018. 3, 8,
Xiao Xiao, Changjian Li, and Yinjie Lei. A lightweight self-supervised representation learning algorithm for scene classification spaceborne SAR and optical images. Remote Sensing, 14 (13):2956, 2022. 2
Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas Guibas, and Or Litany. PointContrast: Unsupervised pre-training for 3D point cloud understanding. In ECCV, 2020.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: /ttp:/00...88880 2017. 7
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021. 3, 7, 15, 27
Léon Zheng, Gilles Puy, Elisa Riccietti, Patrick Pérez, and Rémi Gribonval. Code for reproducible research Self-supervised learning with rotation-invariant kernels, July 2022. Code repository available at https: //hal inrri.ini. ft/ali0033//11131777/73 updates at https://github. com/valecai/sfrikk 10
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using Places database. In NeurIPS, 2014. 8, 25, 26
13

---

Published as a conference paper at ICLR 2023
A
EXTENDED RELATED WORK
We further discuss some related works referenced the main paper.
A. REMINDERS ON KERNEL. MEAN EMBEDDINGS
In this appendix we provide high-level introduction the notion of kernel mean embedding. We refer the reader to (Muandet et al., 2017) for a complete survey.
The idea of kernel mean embedding is encode probability distribution in an RKHS H. Denoting K. : X' x X' + I the reproducing kernel of H defined on some space X', the kernel mean embedding of probability distribution Q defined on A' is
PQ = // KCu,, (dQ(u) E H.
(12)
In other words, the kernel mean embedding mapping Q μQ transforms probability distribution into an element H. As an application, this allows one to quantify the divergence between proba- bilities using the norm || |/w associated to H. Given two probability distributions Q1, Q2 defined on X, one can indeed quantify their divergence by
1/ ((uu))QQ((u) (u) KC,, (QQu)))Q
| MQa 11
(13)
which is precisely the MMD between Q and Q2 defined in (4).
A.2 SAMPLE-CONTRASTVV CRITERION
Given a batch of embeddings Z, == {zilee (that are not necessarily normalized), the general conpte-constrative criterion of Garrido et al. (2023) is defined by:
(Z()
Σ
(z. zz
(14)
Garrido al. (2023) show that this criterion minimized in many contrastive learning methods like (HaoChen et al., 2021). 1). In the case where the embeddings normalized, this sample-contrastive criterion can be derived from the proposed generic uniformity loss fy defined by (7) with the quadratic kernel KCuu, v) (u v)² where u, S as claimed in Section 1. Indeed, since
||z]|| = 1 for all E
(o(Z)) Σ (2,20 1
Σ iE!
12 2,) 2 (Z,) 14]. -4)
(15)
where //| is the batch size. Therefore, the sample-contrastive criterion the normalized is an estimator of the MMD associated to the quadratic kernel between the embedding distribution and the uniform distribution the hypersphere.
A.3 KERNEL DEPENDENCE MAXIMIZATION
We further explain the positioning of our paper with respect to (Li et al., 2021), which proposes a self-supervised learning method based on kernel dependence maximization, using the Hilbert- Schmidt Independence Criterion (HSIC) (Gretton et al., 2005). The HSIC measures the dependence between two random variables X E X and Y E y using two RKHS F on X' with kernel & and on y with kernel in order capture nonlinear correlations. It is defined the squared MMD) associated to the reproducing kernel of the tensor product space F & & between the joint probability distribution P x and the product IP x Py of marginal probability distributions:
HSIC(X, Y)
|| MrxTY 13-
(16)
where Q + μf is the kernel mean embedding mapping defined by (12). Then, the self-supervised learning loss in (Li et al., 2021) is defined as:
LSsL HSIC(Z,Y) TV HSIC 2),
(17)
14

---

Published as a conference paper at ICLR 2023
where Z encodes embeddings of transformed images, and Y encodes image identity as the index of the original image (before transformation) in the training dataset. By maximizing H Y), the backbone learns image representations that are invariant to image transformations. To avoid collapse, high-variance representations are penalized by minimizing HSIC(Z, 7). This is similar previous iiformationmmaxiiizaaiiii methods (Bardes al., 2022; Zbontar et al., 2021), with the difference that they take into account nonlinear correlations using kernels.
Although both our approach and the one in (Li et al., 2021) view self-supervised learning as a kernel method, we highlight here a main distinction between the two works. Both approaches use the MMD. but they do not use it to measure the same quantity. As explained above, Li et al. (2021) use the MMD to measure dependency between random variables (like Z and Y), while the regularization loss we propose uses the MMD to measure the divergence between the embedding distribution and the uniform distribution on the hypersphere. As explained in Sections I and 2, this kernel approach for eeffssuprviiee learning new in the literature and allows for the unification of several previous self-supervised learning methods as illustrated in Table 1.
Note that when the image identity Y a one-hot encoding, (Li et al., 2021) shows that -HSICZZ C (-Br2,,,,)aa 2n) [K,,, Z2)] Ez, (Kaa, Z4)l),
(18) where C is a constant, (Z1.2) is a positive pair of embeddings, i.e., they are embed- dings of two transformations of the same original image, and (Z3, Z4) is a pair of indepen- dent embeddings. In other words, (SIC Y) proportional to the sum of an alignment term -E(z, (zzz.z))~poull (22))>po(k(ZZ Z')] and energy term Ez, [k( Zµ)]. similarly
(3) combined with (7), which yields the proposed loss (2). Our paper shows that, if k(:;.) is rotation-invariant kernel on the hypersphere, then the energy term Ez, (Zz. Z4)] precisely the MMD between the embedding distribution and the uniform distribution on the hypersphere, cf. (6). However there are two differences between the maximization of HSIC Z, Y) and the minimization of the proposed loss (2). First, the alignment term and the energy term in (18) quantified with the same kernel ((.-)), -). which is not the case in (2) where the alignment term is quantified by the Ey-distance between the linear kernel when the embeddings are normalized), and the unifor- mity term (7) is quantified by another rotation-invariant kernel. Second, the loss (2) is a weighted sum between the alignment loss (3) and the uniformity loss (7) controlled by the hyperparameter that tunes the balance between the two terms, which is not the case of (18).
embeddings (equivalent
B THEORETICAL RESULTS
We provide proofs and more details about the theoretical results in the main text.
B.1 PROOF OF LEMMA 2
Consider rotation-invarian kernel KCl,, v) defined on the hypersphere SV-1 such that:
+:00
K(u, v) Σ bePe(o:uTTv) v),, Vu,v E 89-1 E=
(19)
with weights by 2 0 and Pe(g:) -) the Legendre polynomial of order l in dimension g. The proof of Lemma 2 relies on an orthonormal system of spherical harmonics. Let (f. == Ssv 1 Igda be the inner product in the space of continuous functions defined on S and, for each & e N, consider (Ye, 1 = I..... (g, } an orthonormal basis of spherical harmonics of order & in dimension q (homogeneous harmonic polynomials in q variables restricted S4-1, see e.g. (Müller, 2012) for more details), where N(g, E) denotes the dimension of this space, which is by (Müller, 2012, Exercise $3):
for l 1,
N f) = (2 (2 2) 3)! ((g-2)
(20)
for l 2 2.
By the addition theorem (Müller, 2012, Theorem 2, $1):
6(g.6)
N(g, E) Yeu(u)Veu(v)=
|8a-1| Pe(y,Tvv, s
(21)
15

---

Published as a conference paper at ICLR 2023
Hence, the kernel Ku,,, v) can be rewritten
t00 ((
b
K(u, v) = Σ
Yea Ye (u) ()a.e)
(22)
Since (Yex == +o0, = l...., (g, {)} is an orthonormal system for the inner product (s (2) and since 1 is constant on S4-1, we have for any integer & and E (l,..., (g, {)} that:
I....,,
/ JS Yendo--1 1o,1
Yo,1 1 00
if 0, =1
(Ye,a: Vo,1)(0)
=10
(23)
otherwise
Moreover Yo.1 1/VIS-I1 because 1 = (Yo,1, Yo.1 = Ss- Yo, do = Yo, 18-11
Therefore, the kernel mean embedding of the uniform distribution on the hypersphere 0g 1g-1/SS-1 associated to the kernel K. is:
/ 85-1
do 1 1ST-1
Vv E 54-1
K(u, v(duu)dU(u))
Σ be bePelg
v)
84-1
toc
N(g.9)
be
Yen Y=x(u)Yxs(v)doo.-11
N(g.
t=0
S0-1
k=
(24)
t:c
be
N(a.()
Σ
Yeu(u)do,- Yev(v)
N(g.
t=
bo Y
Vi,0
bo.
Yi,0
where we inverted series and integral in the second equation using the dominated conver- gence theorem: the series Ex bePe(q: u vv) converges for every u, v, and for any L, El bele u Tv)l I Ee-obbeP u v) < Σte be = E (g::(g;)) 1), because Pe(g: < 1 for all l by (Müller, 2012, Lemma 2, $8), 1) for all by (Müller, 2012, 89), and EE bePe((; 1) < too is integrable on S This yields the first claim of Lemma 2.
Consider now any probability distribution Q defined on the hypersphere. The kernel mean embed- ding of Q is simply rewritten as:
/ 8T-1
Vv E S9-1
K(u, v)dQ(u) = (V
v)dQ(u)
S4-1 t=
Polç; v((q:TTv)d((u)) + f ST-1
bu
Σ be Pe vV)((()
S4-1
/ Ā(u, v)dQ(u), S
bo +
(25)
because the Legendre polynomial of order is the constant function equal to (see the closed form expression of Po(g: in Theorem 1 in the main text) and J dQ 1. This ends the proof of Lemma 2.
B.2 LEGENDRE EXPANSION OF ROTATION-INVARIANT KERNELS
We show that the kernel weights by in the Legendre expansion (5) of the RBF kernel and the gener- alized distance kernel decay with rate at least polynomial with respect to f.
RBF kernel
The RBF kernel is defined as:
KREF(U, v) a llluvvlll - 20(11u for u, E S9,
(26)
16

---

Published as a conference paper at ICLR 2023
where 0 is the scale of the RBF kernel. Denote (tt) e 2-(((1=t) for t E [-1, 1]. Since the RBF kernel is positive definite and rotation-invarian, by Theorem 1, there exist weights by 2 0, l = ..... too, such that:
((t) e -2(11-)))
Σ
bele(g;t) for t E [-1,1]. t=
(27)
The Legendre polynomials P)(g::) are orthogonal on the interval [-1, 1] with respect to the weight function (1 - ) 2 see e.g. (Müller, 2012):
[ (m) - =3 dt =0 0 for m + n.
(28)
Moreover, by (Müller, 2012, Exercise 3, $2):
[ (Pa()t))((1 dt= [St- 2| N(g.n)
|89-1|
for any n.
(29)
We multiply (27) by Pe(y: () - -3) and integrate the equation on [-1,
S (() 2) dt
+oo
b ((1 - -12) -2)= dt
n=0
+
bn
((()0))((((()11-)))))))) di (30)
n=0
b S5--1 1
ST N)a
where the inversion between series and integral is justified by the dominated convergence the-
orem: the series Ete bn (g (g) 2) 2 converges for every t, and for any N,
E P( - 2² I Σtz 61 +²"²" == g(t) since (P -) S for any n by (Mäller, 2012, Lemma 2, $8), and is integrable on [-1,1] 1] because Σn bn < +o0.
Hence:
S9-2
["
b. N(q, 6)
(()) t) (t) dt.
(31)
St-1
By the Rodrigues rule (Müller, 2012, Exercise 1, $2), since has continuous derivatives of all orders on [-1, 1], we have:
S
by ((g,))
dt, LEN,
(32)
S 2'T(E + 1
where (00) is the f-th derivative of ip, which is (t) 2x (20) We now show that the weights by decay very fast with respect to f. We bound the integral:
p 19 (0) di = [-
di S [ dt 2(20)*. (33)
20 2
Hence:
S r
be < 2N
--11 (2
(34)
[S--1
2TT
Denote (a).. == aaTaa) the Pochhammer symbol defined for any integer n and any sealar a. By the Stirling approximation of the Gamma function (Spiegel et al., 2013, (25.15)), the asymptotic behavior of (a), when n goes to infinity is:
2 e n t
(a)n ~
1/2
as n + 00.
(35)
[(a)
17

---

Published as a conference paper at ICLR 2023
Moreover, for fixed dimension 9, the asymptotic behavior of (g, E) defined by (20) when & goes to infinity is:
N(a () ~ 2 19 as l + oc.
(36)
(q 2)
Therefore, the asymptotic behavior of by goes to infinity is:
be = o (66
as l + +00.
(37)
Generalized distance kernel sphere SV-1 defined in (Brauchart ct al., 2014, Section 5) as:
For 9-1 < < g+1 the generalized distance kernel on the hyper-
2
2
K (u, v) 22
||u v1 for u, V E S
(38)
where
vl 22-4-111 ( = 220-1 T(q/2)[(s)
V Vq-1-20(8"-1) gr-1 S
VTI((g 1)/2 s)
(39)
Following Brauchart et al. (2014, Section 5), the Legendre expansion of the generalized distance kernel K (c(s) ed
is:
+o0
k(() uu v) = (8-1 Σ of t=1
(6)
(40)
((g 1)/2 s)t ((g 1)/2 + s)e
(0) = --
121.
(41)
The kernel weights indeed decay polynomially with respect to E, because according to Brauchart al. (2014, Section 5), the asymptotic behavior the (f.)) is:
r(g/2)[(ss) 220-1 VTT((1- 1)/2
2s
as + +00.
(42)
B.3 CONNECTION BETWEEN SFRIK AND VICREG
Consider a rotation-invariant kernel Ć(u v) := Σts by uvv defined on S such that be 2 0 for & {1...., +00), with bj > 0. To show the connection between SFRIK and VICReg, we construct a high-dimensional feature map - - &g (N), where (2(N) denotes the space of square-summable sequences with its canonical inner product ) such that (o(u), +(v)),, É(,, v) for any u, v SV-1
One way to construct such a feature map is to consider an orthonormal system of spherical harmon- ics. For any integer L, denote {Yr,) J N(q.F) orthonormal basis of spherical harmonics of order & dimension q. By the addition theorem (Müller, 2012, Theorem 2, $1) recalled in (21), the kernel C(u, v) admits the decomposition:
to0
too N(g.t)
be S4-1 N(q. (e.(u)Ye.uv))
k(u,v)= = Σ
bePelq
Σ
(43)
(T) (vV))
where
+00
belsa-11 e)
5 (g.0)
with t
- (Ye, (u)) N(g for l E foo..., (44)
Then, the MMD in (6) between any probability distribution Q defined on the hypersphere and the uniform distribution U the hypersphere can be written the norm in &g (N) of the generalized
18

---

Published as a conference paper at ICLR 2023
moment of Q with the mapping 4::
MMD?(Q. U) 2 (K(,
Ex c (T(()) ( )v2)
(Bzzo(((22) (f(z)) Exinn G ( Dik |Bz [(()]]]
(45)
b
((g E) B (0.(2)]
We now explain how to construct explicitly an orthonormal basis spherical harmonics (Ye, |
I,.... too; = = ,,.....,N( (qq, {)}. based on the following theorem.
Theorem (Axler et al. (2013, Theorem 5.25)) For any order & E and any dimension g 2 3, the famity
(Y M(. == {" 1 02 0 [|ul|
2-9 + + +... og
and on <1}
(46)
a (non-orthonormal) basis of the space of spherical harmonics of order in dimension q, where aj (j =,,... q) are nonnegative integers, and 07" denotes the ay-th partial derivative with respect to the j-th coordinate.
Typically, construct the orthonormal basis ) by orthonormalizing the basis (Y (g.e) of Theorem 3 using, e.g., the Sraam-Shhmidt procedure. For & ,,...,toc. denote:
Mgg.e)
Vy= s 1 + N u (Yi,, (u)) N(n.))
(47)
Then, for each 8= 1, 000, there exists lower triangular matrix M2 such that:
(u) (u), for all u E S
(48)
Remark that is possible compute explicitly the entries of the matrices Me, e 1,.... to0, because there exists a closed-form expression for the inner product (Y Yi,))( for any E, k. ki indeed, the function Yú, for any f, is polynomial defined on the hypersphere, and the integral of any monomial with respect to the measure a--1 the hypersphere S admits closed-form expression given by Weyl (1939, Section 3).
By injecting (48) in (45), obtain:
Se(Sa-1 E)
Σ t=1
|M (x)]1
(49)
This yields the claim Section 3.3 by remarking with Theorem that the families
(f N(g.1) = {u {u>> w²
1 و 5
{ 1 N(g.2) = {u u' 1 | < s uojufu- (w))2 1
1
(50)
9
are bases of the space of spherical harmonics of order and 2 in dimension q.
B.4 REGULARIZATION LOSS OF SIMCLR, AUH AND VICREG
In SimCLR, AUH, VICReg and SFRIK, each image X; in batch (xther is augmented into two different views (1) and x; (2), which are encoded into two embeddings z and (2) These embeddings are normalized in SimCLR, AUH and SFRIK, but not in VICReg. This yields two
batches of embeddings Z, (w) = (z, )ier (v = 1, 2). The four methods share the same form of loss function:
E(Z, (1) Z, (=) X 7 (1) 2, (2)) 1 (Z, (1) .Z, (2)).
(51)
for some scalars X, p > 0, where f, is the alignment loss defined by (3) (which is the same for all the four methods), and f, is the regularization loss specific to each method.
19

---

Published as a conference paper at ICLR 2023
SimCLR
The regularization loss in SimCLR is:
2
exp exp (z,"0) (v) z 20100/1)) /+)
(Z (1) Z (2)
ΣΣ log Σ
E 10.00
(52)
2 v=l iET
w'=lFEr
where T > is hyperparameter of the method called the temperature, and [(v.))) is equal if (v, 1) # (v',8), f') and 0 otherwise. The scalars X, 1u fixed at À 1 and μ
Alignment & Uniformity
The regularization loss in AUH is:
(Z, 1
Z
(2)
log
ΣΣ
z 10 (2)
(53)
2
JEET YEr
where t > 0 is a hyperparameter called the scale of the RBF kernel. The scalar is tuned as a hyperparameter and j is fixed at μ 1.
VICReg introduced in Section 3.3, the regularization loss VICReg is:
1 c( + ())
(Z (1) Z, (2)
uZ )
(54)
2p
where j is the scalar from (51). Here, v(.) and c(.) are respectively the variance and covariance terms defined by (9). Both À and f are tuned as hyperparameters.
C EXPERIMENTAL SETTING
In the interest of reproducible research, give more details about the setting of experiments presented in Section 4.
C.I IN20% DATASET DESCRIPTION
The datasets used in our experiments include a subset of 20% of ImageNet-1000 as in (Gidaris al., 2021). ). This reduced dataset, denoted IN20%, contains all the 1000 classes of ImageNet, but we keep only 260 images per class. The 260 images extracted are the same as those extracted the official implementation of OBoW (https: : //iitt tritii thub. com/ valeoai/obow use another 20% subset the ImageNet train set as separate validation set for hyperparameter tuning (see Appendix C.4 below). The construction of this validation set is based on the code of OBoW, and will be exactly detailed in our code that will be released at publication.
In Section 4.2, also
C.2 IMAGE AUGMENTATIONS
We follow the same image augmentation pipeline as in (Bardes al., 2022). Our experiments include the following image augmentations implemented by PyTorch (t (torchvision.c ttorchvision..taasffon....
RandomResizedCron (224, scale- 08, 1.0) : crop a random area of the image between 8% and 100% of the total area, and resize it to an image of size 224 x 224;
carrommorrizotalFFiie (): flip horizontally image;
ColorJitter (brightness=O. 4,, contrast=0.4
intturation=0...
hue=0.1), randomly change brightness, contrast, saturation and hue of an image by factor randomly sampled in respectively [0.6, 1.4], [0.6, 1.4], [0.8, 1.2] and [-0.1,0.1].
RandomGraysca () convert an image into grayscale.
We also use image augmentations implemented by PIL, as in VICReg's code available at https: /github. com/ www.bookeeearcceeee.....
Gaussi ansBrri (): blur an image using Gaussian kernel with standard deviation uniformly
sampled in [0.1,2.0]:
Solarization on () randomly invert all pixel values above threshold, which is 130.
20

---

Published as a conference paper at ICLR 2023
In our experiments, the first image view is obtained by composing the following random augmenta- tions: random cropping resized to 224 x 224, random horizontal flip applied with probability 0.5, random color jittering applied with probability 0.8, random grayscale conversion applied with prob- ability 0.2, random Gaussian blur applied with probability 0.1, and random solarization applied with probability 0.2. The second view is obtained by composing the same random augmentations as the first view, except that Gaussian blur is applied every time (probability 1), and solarization is never applied (probability 0).
C.3 EVALUATION PROTOCOL
We describe the downstream tasks on which self-supervision methods are evaluated in our experi- ments of Section 4.
Linear probing on IN20% and IN100% Following, e.g., (Bardes et al., 2022), the weights of the backbone (ResNet-18 18 ResNet-50) are frozen and linear layer followed by softmax on top of the backbone is trained in a supervised setting on a training set. Then the model is evaluated on a test set. The training set is either IN20% or IN100%, but with labels. The test set is the validation set of ImageNet. The linear layer is trained using an SGD optimizer with momentum parameter equal to 0.9 during 100 epochs. We apply weight decay of 10-6, The learning rate follows cosine decay scheduling. The batch size is fixed at 256. Training images are augmented by composing a random cropping of an area between 8% and 100% of the total area resized to 224 x 224, and random horizontal flip of probability 0.5. Images at test time are resized to 256 x 256, and cropped at the center with a size 224 x 224. The initial learning rate is tuned as a hyperparameter, and we report the top-1 accuracy on the validation set of ImageNet obtained after the last training epoch, along with the corresponding top-5 accuracy. The code that we use for linear probing IN20% IN100% is adapted from (Bardes et al., 2022) available at https: //github. com/ facebookresearcch...oooo /vicrgg..
Linear probing on Places205 We use the code of Gidaris et al. (2021), available at https:// // for the evaluation by linear probing on Places205, The weights
github . ccm/valeoai////ooooccoooo/ of the backbone (ResNet-50) pretrained on IN100% are frozen and linear prediction layer is trained for the classification task on Places205. We note that a batch normalization layer with noa-meaaable scale and bias parameters is added the output of the backbone in (Gidaris al., 2021). The linear prediction layer is trained with an SGD optimizer with a 0.9 momentum parameter during 28 epochs. The weight decay is 10-1 The batch size is 256. The learning rate decreases by factor of 10 at epoch 10 and epoch 20. We use the same image augmentation pipeline for training and testing as in linear probing on IN IN100%. The initial learning rate tuned as hyperparameter, and we report the top-1 accuracy on the validation set of Places205 obtained after the last training epoch, along with the corresponding top-5 accuracy.
Linear classification on VOC2007 After pretraining a ResNet backbone, we use the VISSL library (Goyal et al., 2021) extract features of VOC2007 images resized 224 224 by taking the output of the last average pooling layer of the pretrained ResNet backbone. We then learn a linear SVM with LIBLINEAR (Fan et al., 2008) on top of these features to predict the presence or the absence of a given class in the test images. An average precision score is then computed for each class after a 3-fold cross-validation, and we report the mean score over all classes as the mean average precision (mAP).
Semi-supervised learning After pretraining a ResNet backbone by self-supervision, we fine-tune this backbone and the linear classifier on the ImageNet classification task with only 1 1% or 10% of the labeled data. The labeled images that are considered in these subsets are the ones used in the official code of SimCLR available at https: ht////ttttts com/ poooglereeeezee
simolr. We use an SGD optimizer with momentum parameter equal to 0.9 during 20 epochs, without weight decay. The batch size is fixed at 256. The learning rates of the backbone and the linear classifier follow a cosine decay scheduling with different initial learning rates. These initial learning rates are tuned as hyperparameters. We report the top-1 accuracy on the validation set of ImageNet obtained after the last training epoch, along with the corresponding top-5 accuracy. We use the same image IN100%. The code that use
augmentation pipeline for training and testing as in linear probing for superviie learning with few labels of IN100% is the one of Bardes et al. (2022) available at https: / / thub. com/ Facebookreseen
/ creg.
21

---

Published as a conference paper at ICLR 2023
IN20%
IN20%
ImageNet train set
ImageNet validation set
Train
Validation
Test
Figure 2: ImageNet dataset split for hyperparameter tuning experiments of Section 4.2 on IN20%, as explained in Appendix C.4
Weighted KNN classification We follow the usual protocol of Wu et al. (2018); Caron et al. (2021). We compute the normalized representations fx;) of the images x, i e [N], in the training set. The label of an image Xist in the test set is predicted by weighted vote of its k nearest neighbors N₁ in the representation space: the class gets a score of we DiEN, cxp(fo(x,) fa (0s0)))) (c:=c) where (x(a))) is normalized, c; is the class of x,, and [e,=c] equal to if c, c, and otherwise. We report the kNN classification top-1 accu- racy for k 20. The image augmentation pipeline for both training and testing is the following one: images are resized to 256 x 256, and cropped at the center with a size 224 x 224. The code that use for kNN classification is the one of Caron et al. (2021) available https: //g1 thub. com/ www.aaakkeeeeeeeeeooooooooooooee www.eeeeerreeeeeeeeeeeee ch/dino.
C.4 HYPERPARAMETER FOR EXPERIMENTS ON IN20%
We describe in detail our hyperparameter tuning protocol for the experiments Section 4.2 on IN20%. For a rigorous tuning. it is important that the dataset used for the final evaluation remains unseen during pretraining and hyperparameter tuming. For each pretraining method, we pretrain on the IN20% training set (blue subset Figure 2) a backbone for each choice of hyperparameters. These backbones are then evaluated by weighted kNN classification on separate validation set, which another 20% subset of the ImageNet train set (purple subset in Figure 2), and we select the hyperparameters yielding the highest top-1 accuracy on this kNN evaluation. Then, we tune the learning rate the linear probing evaluation, again on our separate validation set (purple subset in Figure 2). Finally,
the model trained with the best learning rate discovered for linear probing evaluation on the usual ImageNet validation set (red subset Figure 2), which has never been seen during hyperparameter tuning.
We report the values of the optimal hyperparameters found after our hyperparameter tuning on a separate validation set, for each pretraining experiment IN20% with a ResNet-18 presented in Section 4.2. These hyperparameters yield the evaluation results reported in Section 4.2 for linear probing on the usual ImageNet validation set. We recall that the hyperparameters specific each self-supervision method was introduced in Appendix B.4.
SimCLR For each embedding dimension, we fix the batch size at 2048, and tune the temperature and the base learning rate bose._r for pretraining with SimCLR. Then, we tune the initial learning rate head for linear probing on IN20%. The optimal hyperparameters are shown Table 6.
AUH. For each embedding dimension, fix the batch size 2048, and tune the alignment weight X, the scale of the RBF kernel t, and the base learning rate base Ir for pretraining with AUH. Then tune the initial learning rate bred for linear probing on IN20%. The optimal hyperparameters are shown in Table 7.
Table 6: Hyperparameter choice for SimCLR pretrained on IN20% with ResNet-18 during 100 epochs, evaluated by linear probing on IN20%.
Dimension
Temperature
base
by head
q 1024
0.15
1.0
0.2
q
2048
0.15
1.0
0.2
q 4096
0.15
1.0
0.2
q 8192
0.15
0.8
0.2
22

---

Published as a conference paper at ICLR 2023
Table Hyperparameter choice for AUH pretrained on IN20% with ResNet- during 100 epochs, evaluated by linear probing on IN20%.
Dimension
Alignment weight
Scale
baseJr
Ir_head
g 1024
400
2.5
1.0
10.0
ạ = 2048
1000
2.5
1.0
4.0
ạ 4096
2000
2.5
1.0
1.0
Q 8192
3000
2.5
1.0
2.0
Table 8: Hyperparameter choice for VICReg pretrained on IN20% with ResNet-18 during 100
epochs, evaluated by linear probing on IN20%.
Dimension
Alignment weight
Variance weight μ
base Ir
Ir head
1024
0.4
0.2
2048
4
4
0.7
1.0
4096
0.6
0.2
8192
10
10
0.7
0.2
Table 9: Hyperparameter choice for SFRIK pretrained on IN20% with a ResNet-18 18 during 100 epochs, evaluated by linear probing on IN20%.
Order
Dimension
Alignment weight À
Kernel weights (b,. b2 by)
baseJr
Ir_head
L L=1 1 q 8192
10000
(1, 0, 0)
0.4
10.0
L=2 2
4 == 1024
400
(1, 40, 0)
1.0
4 2048
400
(1, 40, 0)
1.0
4.0
4096
1000
(1, 40, 0)
1.0
4.0
q 8192
2000
(1, 20, 0)
1.0
10.0
= 3
8192
4000
(1, 40, 40)
1.2
1.0
VICReg, For each embedding dimension, we fix the batch size at 2048, and we tune the alignment weight the variance weight μu, and the base learning rate base_rr for pretraining with VICReg. Then, we tune the initial learning rate Ir head for linear probing on IN20%. The optimal hyperpa- rameters are shown Table
SFRIK, batch size 2048. For each embedding dimensionn we fix the batch size at 2048, and tune the alignment weight À in (2), the kernel weights be (e (2,3}) in (8), and the base learning rate base Ir for pretraining with SFRIK. Without loss of generality, the first kernel weight b in (8) is fixed at b, 1. Then, tune the initial learning rate Ir head for linear probing IN20%. The optimal hyperparameters are shown Table
SFRIK, batch size 4096. We fix the dimension at q = 8192, and the batch size at 4096, and tune the aligmment weight À in (2), the kernel weights be (f E (2,3)) in (8), and the base learning rate baseJr for pretraining with SFRIK. Without loss of generality, the first kernel weight by in (8) is fixed at b, = 1. Then, we tune the initial learning rate Ir head for linear probing on IN20%. The optimal hyperparameters are: X 4000, (b,, b2, b3) (1, 20, 0), base 0.8, ly hr_h = 1.0. This yields top-1 accuracy 46.3 for linear probing on IN20%, meaning that it is not necessary to use batch size larger than 2048 in SFRIK to obtain better performance.
C.5 HYPERPARAMETERS FOR THE ABLATION ON THE KERNEL CHOICE
We detail the experimental setting of our ablation study the kernel choice for the generic unifor- mity term (7) in Table 3 of Section 4.2. We follow the protocol of Section 4.2, with an embedding dimension of q = 8192. The training loss is Xe, (Z, Z, (2) 0.5((e, (Z, (1) + +,, (Z, (2))). where t, is the loss (7) with RBF or general distance kernel.
23

---

Published as a conference paper at ICLR 2023
The kernel is KC(uvv v) e- We fix the batch size at 2048, and tune the alignment weight A, the scale of the RBF kernel t, and the base learning rate base_lr for pretrain- ing. Then, we tune the initial learning rate Ir head for linear probing IN20%. The optimal hyperparameters are: = 100, 1 = 2, base Ir 1.0, Ir head = 10.
RBF kernel.
Generalized distance kernel. The kernel is K[, v) C- ||u vIβ, where we fixed C because the value of this constant does not change the gradients of the optimization problem, and p = 2 because we verified empirically that choosing p < yields poor results. We the batch size at 2048, and tune the alignment weight X, and the base learning rate base_lr for pretraining. Then, we tune the initial learning rate _hed for linear probing on IN20%. The optimal hyperparameters are: À 10000, base Ir = 0.6, Ir head 10.
As shown in Table 3, a truncated kernel is empirically a better choice than the RBF or the generalized distance kernel for the uniformity term (7).
C.6 HYPERPARAMETERS FOR EXPERIMENTS ON IN 100%
Table 10 reports the selected hyperparameters for pretraining ResNet-50 on IN100% with SFRIK in Section 4.3. The tuming protocol is as follows. Since hyperparameter tuning is costly on IN100%, we pretrain several ResNet-50 for different values of kernel weights by, alignment weight À and base learning rate JaseJrr and pause the pretraining after 50 epochs. We evaluate the obtained backbones on kNN classification (top-1 accuracy, continue pretraining these selected backbones until reaching epoch 200 400. Finally we select the hyperparameters that yield the highest top-1 accuracy for linear probing on IN100% after 200 or 400 epochs of pretraining. Because of the conclusions of Appendix D.I, our hyperparameter tuning follows the common practice in the self-supervised learning literature where hyperparameters are selected by measuring the performance on the validation set of ImageNet. Note that verified experimentally a posteriori that the hyperparameters obtained by our tuning protocol are similar to the ones obtained from a tuning on smaller dataset like STL-10 (Coates al., 2011). This means that an alternative is to tune the hyperparameters on STL-10, and generalize these hyperparameters to pretrain SFRIK on IN100%.
20), and select the best performing backbones. Then we
Table 10: Hyperparameter choice for SFRIK pretrained on IN100% with a ResNet-50 during 200 or 400 epochs.
Dimension
Order
Epoch
Alignment weight À
Kernel weights (b), b2 ba)
buse
q 8192
L = 2
200
4000
(1, 20, 0)
0.4
I = 3
200
4000
(1, 40, 40)
0.4
g 16384
L =
200
20000
(1. 40, 0)
0.4
9 32768
L
200
40000
(1, 40, 0)
0.4
L = 3
200
40000
(1. 40, 40)
0.4
9 == 8192 L 400
4000
(1. 20, 0)
0.4
Tables 11 and 12 give the optimal hyperparameters found for linear probing IN100%. linear probing on Places205, and semi-supervised learning with limited labels of IN100% when evaluating pretrained ResNet-50 backbones with SFRIK and VICReg. The hyperparameters that tuned for evaluation are: the initial learning rate -rhhead of the linear layer in linear probing; and the initial learning rate Ir bbccockoone and (r_head for respectively the backbone and the linear layer in semi- supervised learning. The reported hyperparameters in the two tables yield the evaluation results reported in Section 4.3.
C.7 COMPUTATIONAL RESOURCES
Pretrainings of ResNet- 18 on IN20% with a batch size of 2048 (respectively 4096) are performed with 4 (respectively 8) NVIDIA Tesla V100 GPUs with 32GB of memory each. Pretrainings of ResNet-50 on IN100% are performed with 8 NVIDIA Tesla V100 GPUs with 32GB of memory each. The total amount of compute used for this work around 50000 GPU hours.
24

---

Published as a conference paper at ICLR 2023
Table 1I:
Hyperparameter tuning for linear probing on IN100% and Places205 for SFRIK and VICReg pretrained on IN100% with ResNet-50.
head
Method
Epochs
IN100%
Places205
VICReg (q = 8192)
200
0.02
0,01
SFRIK (I. = 2, 8192)
200
1.0
0.01
SFRIK (L = =3. 9 =8192)
200
2.0
0.01
SERIK (T. =2, -16384)
200
0.3
0.01
SFRIK -2.4 32768)
200
0.01
SFRIK (Z. = 3, 32768)
200
0.4
0.01
SPRIK (L -22. 8192)
400
2.0
0.01
Table 12: Hyperparameter tuning for semi-supervised learning for SFRIK and VICReg pretrained on IN100% with ResNet-50.
Semi-supervised, labels
Semi-supervised,
10% labels
Method
Epochs
backkooe
br Ir_head
Ir .._bockbone
Irahead
VICReg (q = 8192)
200
0.02
0.2
0.04
SFRIK (L 2. 8192)
200
0.004
1.6
0.02
0.4
SFRIK (I. 3. 8192)
200)
0.002
1.0
0.01
0.2
SFRIK (L = =2. 16384)
200
0.004
1.4
0.04
0,2
SFRIK (L. = =2. 32768)
200)
0.004
1.0
0.04
0.1
SFRIK (L 3. 32768)
200
0.004
1.0
0.02
0.1
SFRIK (1. 2, 8192)
400)
0.004
1.4
0.02
0.2
C.8 PUBLIC RESOURCES
We acknowledge the of the following public resources, during the course of the experimental work of this paper:
VICReg official code (Bardes et al., 2022)
MIT License
DINO official code (Caron al., 2021)
Apache License 2.0
OBoW official code (Gidaris et al., 2021)
Apache License 2.0
AwAV official code (Caron et al., 2020)
CC BY-NC 4.0
SimCLR official code (Chen et al., 2020a)
Apache License 2.0
VISSL code (Goyal et al., 2021)
MIT License
ImageNet dataset (Deng et al., 2009)
Places 205 dataset (Zhou al., 2014)
Attribution CC BY
VOC2007 dataset (Everingham al., 2010)
25

---

Published as a conference paper at ICLR 2023
D ADDITIONAL EXPERIMENTAL RESULTS
We provide in this appendix other experimental results to complement the main paper.
D.1 HYPERPARAMETER TUNING WITHOUT A SEPARATE VALIDATION SET
A common practice in the self- learning literature, c.g., (Bardes et al., 2022; Chen et al., 2020a), is select the hyperparameters by measuring the performance on the validation of Im- ageNet (red dataset in Figure 2) instead of separate validation dataset (purple dataset in Figure 2). In this paragraph, we verify whether this less rigorous practice changes the conclusion of the exper- iments in Section 4.2. In Table 13, report the evaluation of the different backbones pretrained on IN20% after tuning each method directly on the validation set of ImageNet, which is the same dataset used for evaluation in linear probing. By comparison with Table 2, which follows the more rigorous hyperparameter tuning protocol described in Appendix C.4, we observe that although the absolute figures of merit slightly vary if we use the less rigorous protocol instead of the more rigor- ous one, the conclusion of the experiments in Section 4.2 does not change. This gives an empirical justification to this common practice.
Table 13: Linear probing on IN20% (top-1 accuracy) at different embedding dimensions q. All methods were pretrained on IN20% with a ResNet-18 for 100 epochs. Hyperparameters specific to each method and the learning rate are tuned on the same dataset as the one used for evaluation in linear probing, which less rigorous than tuning the hyperparameters on a separate validation set described in Appendix C.4. Symbol indicates models that we retrained ourselves.
SimCLR
AUH
VICRegi
SFRIK
L=1
L=
L
1024
45.2
45.2
40.8
44.2
2048
45.8
45.6
44.1
45.5
4096
46.3
46.8
44.9
47.0
8192
46.2
46.8
46.0
27.5
47.0
47.6
D.2 ADDITIONAL EVALUATION OF SFRIK PRETRAINED ON IN20%
In complement to the results on linear probing on IN20% presented in Table 2, we evaluate SFRIK by linear probing on Places205 (Zhou al., 2014) and linear SVM VOC2007 (Everingham al., 2010), following the protocol described Appendix C.3. The learning rate for linear probing on Places205 is fixed 0.01. Table 14 compares SFRIK, AUH and VICReg pretrained with ResNet-18 on IN20% under the setting of Section 4.2.
Table 14: Linear classification on Places205 and VOC2007 (aceuracy and mean average pre- cision). All methods pretrained IN20% with ResNet- 18 for 100 epochs. We tuned all hyperparameters specific to each pretraining method and the learning rate. The symbol indicates models that retrained ourselves.
Linear classification
Method
Places205
VOC07
Top-1
Top-5
mAP
VICReg (g 8192)
41.6
71.7
73.3
AUH (g 8192)
42.3
72.8
73.6
SFRIK (L. = 3. 8192)
42.7
72.9
74.1
We conclude that, under the rigorous protocol of Section 4.2 and Appendix C.4, SFRIK performs better than AUH on various downstream tasks, while having the same computational saving offered by the kernel trick. Compared to VICReg, SFRIK performs better on these tasks with a reduced complexity.
26

---

Published as a conference paper at ICLR 2023
D.3 RESULTS OF OTHER PRETRAINING METHODS ON IN100% WITH RESNET-50
In complement to Table 4, Table 15 reports evaluation results for linear probing on IN100% and semi-supervised learning with few labels of IN100% of different ResNet-50 pretrained on IN100% with other state-of-the-art methods than the ones presented in Table 4. As mentioned in Section 4.3, we observe that, similarly SFRIK, both BYOL and SwAV with multi-crop have performance drop compared to VICReg on semi-supervised learning with I % of labels, even though they perform better on linear probing on IN100%.
Table 15: Linear probing on IN100%, semi-supervised learning with few labels of IN100 % (top-1 and top-5 accuracy). All methods have been pretrained on IN100% with a ResNet-50 during reported number of epochs.
Linear probing
Semi-supervised
Method
Epochs
IN
IN100%
1% labels
10% labels
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
SimCLR (Chen al., 2020a)
1000
68.3
89.0
48.3
75.5
65.6
87.8
OBoW (Gidaris et al., 2021)
82.9
90.7
BYOL (Grill et al., 2020)
1000
74.3
91.6
53.2
78.4
68.8
89.0
SwAV (with multi-crop) (Caron et al., 2020)
800
75.3
53.9
78.5
70.2
89.9
Barlow Twins (Zbontar et al., 2021)
1000
73.2
91.0
55.0
79.2
69.7
89.3
VICReg (Bardes et al., 2022)
1000
73.2
91.1
54.8
79.4
69.5
89.5
27


Reviews:
[{"review_id": "ou2Dog1axdD", "reviewer": "Reviewer_uDSt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a generic regularization loss on self-supervision learning of image representations. The regularization is proposed to enforce the uniformity of transformed representations hypersphere. The author show that the proposed loss family accommodates several previous self-supervision learning methods. Moreover, the author emphasized that the proposed regularization with truncated kernel can significantly save memory and runtime. Experiment results show the advantages of applying SFRIK in self supervision learning.", "summary_of_the_review": "Overall, the paper needs some clarification of the motivation and novelty. ", "strength_and_weaknesses": "Pros:\nThe author proposed SFRIK for avoiding representation collapse in invariant representation learning. \nThe author did thorough study on the truncated kernel and drew connection between the proposed method and information maximization.\nThe author conducted experiments for both representation quality and runtime complexity measurement.\n\n\n\nCons:\n1. The author proposed a family of losses which are powered by different choice of kernels. However, the choice of RBF kernel and generalized distance kernel have already been studied in existing works. as the author mentioned in table 1. Hence the novelty of the proposed method shrunk to the adaptation of truncated kernel.\n2. The author mentioned that the proposed SFRIK can help saving computation and memory complexity comparing with VICReg due to the kernel trick. However, applying RBF kernel should provide the same saving. As the author mentioned adopting RBF kernel is nearly equivalent to the AUH method. The saving on computational complexity shouldn’t be considered as the novelty of SFRIK.\n3. In the experiment section. Table 2 shows AUH method performs quite competitive to the proposed method. However, in Table 3, the author show the performance of applying RBF kernel is much worse than the proposed truncated kernel. The author may add some explanation of the discrepancy if AUH works equivalent as applying RBF kernel.\n4. The motivation of applying the truncated kernel was not clear enough. Though this kernel function was not investigated in self-supervision learning. The author may explain more on why the adoption is necessary.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ua5CceGyYSR", "reviewer": "Reviewer_3r4M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary:\nThis paper proposes a regularization loss by resorting to rotation-invariant kernels. The method promotes embedding distribution to be close to uniform distribution in order to avoid collapse in self-supervised learning.\n\n", "summary_of_the_review": "Given the novelty, technical and theoretical contribution of the paper, I recommend acceptance of the paper. ", "strength_and_weaknesses": "Strength:\n\n- This paper introduces a generic regularization loss based on kernel trick and might be of interest of the community to better understand self-supervised method from a kernel perspective.\n\n- The introduced loss family uniforms several mainstream self-supervised methods. This includes uniformity-based and information-maximization methods as a special case of the proposed framework under different kernel choices. \n\n- The method is able to reduce computation time and memory efficiently with the help of kernel trick while it remains competitive on downstream tasks\n\n\nWeakness:\n- W1:  Why L is only constrained to be 2 and 3? What happens when L becomes bigger? Is there any relevant ablation studies on this? \n\n- W2: Is it possible to explain the relationship between different kernel choices and the eventual feature quality? How does different kernel choices affect downstream task performance? I asked this because it seems they all have the same goal of pull the embedding distribution to be close to uniform distribution. How can we know that which choice of kernel is the best among all other kernels? Also, it seems you don't achieve SOTA performance on all settings.\n\n- W3: It might be helpful to give a brief introduction about kernel mean embeddings as preliminaries to improve readability for those who are not familiar with this topic.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GbVI71zQ-pO", "reviewer": "Reviewer_w91x", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper deals with self-supervised learning and proposes a method named SFRIK that complements a Siamese architecture with a regularization term that consists in the MMD discrepancy between the embedding distribution and the uniform distribution on the hypersphere. In essence, the idea of this term is to prevent representation collapse by making sure that the learned embeddings are uniformly spread out across the hypersphere.\nThe methods interestingly unifies and subsumes several previously proposed uniformity-based self-supervised learning procedures, including recent ones such as VICReg, and connects to information-maximization methods.\nThe paper exhaustively examines the effect of the various hyperparameters (such as expander dimensionality, kernel, and kernel truncation order) of SFRIK and compares its performance in terms of quality of the learned representations to those learned by the main competing algorithms like SimCLR, SwAV and VICReg on multiple downstream on ImageNet1k and Places205. The conclusions are that SFRIK is competitive with these algorithms both in terms of quality of the learned embeddings for downstream classification tasks, as well as in training speed and memory efficiency.", "summary_of_the_review": "Very interesting and relevant work proposing a new class of self-supervised learning regularizations, based on solid theoretical foundation with possibly wide-reaching implications, even beyond the convincing empirical results already presented in the paper.", "strength_and_weaknesses": "This paper has no obvious major weaknesses and several strengths:\n- The construction of the method is based on solid theoretical foundation connecting to kernel methods, integral probability metrics (MMD) and spherical harmonics.\n- The paper established interesting and wide-reaching connections to other methods, unifying uniformity-based self-supervised learning and information-maximization methods.\n- SFRIK enjoys different complexity scalings compared to for instance VICReg thanks to its use of the kernel trick, making it clearly advantageous in specific regimes like large expander dimension and small batch size.\n- The paper delves into rigorous empirical comparisons with the major competing methods (SimCLR, SwAV, SimSiam, VICReg), fixing architecture and probing different embedding dimensions on multiple downstream tasks of interest (linear probing, kNN classification, semi-supervised learning), and even comparisons with teacher-student methods like BYOL.\n- The paper includes a rigorous study on the effect of the main hyperparameters, such as embedding dimension, kernel type and kernel truncation order.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "17aSVs1Fsz9", "reviewer": "Reviewer_7yz3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of self-supervised learning without negative pairs (such as that used in InfoNCE) and defines a so-called “loss family” which expresses existing methods such as VICReg as a specific case of the family. The paper also proposes the use of a truncated kernel which results in a linear scaling of the loss complexity in relation to the SSL embedding dimension (as opposed to quadratic for VICReg). The proposed method shows respectable performances in comparison to the state-of-the-art, while requiring less memory and training time compared to VICReg in particular.", "summary_of_the_review": "The paper was a pleasure to read through, and the formalization of uniformity in instance-wise SSL was done well. The experimental results are reasonably convincing, but fall short when compared to similar works in literature. I would like to read the comments from the other reviewers, as well as the rebuttal from the authors before making a more certain recommendation.\n\n\n**Edit after rebuttal period**\n\nI have carefully gone through the other reviews as well as the responses provided by the authors. I believe that the authors' responses are rigorous, earnest and sufficient. I agree in spirit with the authors that pre-training on IN100% with ResNet-50 is certainly excessively expensive when considering a fair and sufficiently hyperparameter-tuned experimental setting. Thus, I raise my rating to 8.", "strength_and_weaknesses": "The proposed method is quite elegant, and can be understood reasonably well in an intuitive manner. The related works are positioned and explained well. \n\nThe method-wise contribution of this paper can be summarized as the use of a truncated kernel for the uniformity loss (called SFRIK). Though the mathematical formalization of the loss formulation and the rigorous discussion of related work is very much appreciated, one must wonder about the simple question: are the proposed changes demonstrated in a convincing manner. While the improved RAM and training time (shown in Tab. 5) are great, the main results shown in Tab. 4 does not paint a convincing picture as the proposed method mostly reaches performance parity in linear probing and does not perform well in the semi-supervised 1% case.\n\nThe authors perform their main experiments on a benchmark they dub “IN20%”. This is not a standard benchmark, so the reviewer would like to ask about the rationale for selecting this setting. Furthermore, despite the heavy comparisons to VICReg, many experiments are missing that were conducted in the VICReg paper (such as VOC07, iNat18, COCO det+seg, MS-COCO retrieval). While not all experiments should be required, one wonders if there are sufficient tasks being considered to argue for the general applicability of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%