Self-Supervised Learning with Kernel Dependence Maximization
Authors: Yazhe Li
Roman Pogodin
Danica J. Sutherland
Arthur Gretton
Venue: NeurIPS 2021 Poster (2021)
Keywords: self-supervised learning
representation learning
dependence maximization
HSIC
deep kernel methods

Abstract: We approach self-supervised learning of image representations from a statistical dependence perspective, proposing Self-Supervised Learning with the Hilbert-Schmidt Independence Criterion (SSL-HSIC). SSL-HSIC maximizes dependence between representations of transformations of an image and the image identity, while minimizing the kernelized variance of those representations. This framework yields a new understanding of InfoNCE, a variational lower bound on the mutual information (MI) between different transformations. While the MI itself is known to have pathologies which can result in learning meaningless representations, its bound is much better behaved: we show that it implicitly approximates SSL-HSIC (with a slightly different regularizer).
Our approach also gives us insight into BYOL, a negative-free SSL method, since SSL-HSIC similarly learns local neighborhoods of samples. SSL-HSIC allows us to directly optimize statistical dependence in time linear in the batch size, without restrictive data assumptions or indirect mutual information estimators. Trained with or without a target network, SSL-HSIC matches the current state-of-the-art for standard linear evaluation on ImageNet, semi-supervised learning and transfer to other classification and vision tasks such as semantic segmentation, depth estimation and object recognition. Code is available at https://github.com/deepmind/ssl_hsic.

Self-Supervised Learning with Kernel Dependence Maximization
Yazhe Li DeepMind and Gatsby Unit, UCL e yazhe@google.co com
Roman Pogodin'
Gatsby Unit. UCL
roman pogodin. 170uc1..... ac uk
Danica J. Sutherland UBC and Amii! dsuth@cs. ubc.u.c...
Arthur Gretton
Gatsby Unit, UCL
arthur aethur.grettooeee www.ergggeettttlccccommmmmmmm com
Abstract
We approach self-supervised learning of image representations from a statistical dependence perspective, proposing Self-Supervised Learning with the Hilbert- Schmidt Independence Criterion (SSL-HSIC). SSL-HSIC maximizes dependence between representations of transformations an image and the image identity, while minimizing the kernelized variance of those representations. This framework yields a new understanding of InfoNCE, a variational lower bound on the mutual information (MI) between different transformations. While the MI itself is known to have pathologies which can result in learning meaningless representations, its show that it implicitly approximates SSL- HSIC (with a slightly different regularizer). Our approach also gives us insight into BOOL, negative-free SSL. method, since SSL-HSIC similarly learns local neighborhoods of samples. SSL-HSIC allows us to directly optimize statistical dependence in time linear in the batch size, without restrictive data assumptions indirect mutual information estimators. Trained with or without a target network, SSL-HSIC matches the current state-of-the-art for standard linear evaluation on ImageNet [1], semi-supervised learning and transfer to other classification and vision tasks such as semantic segmentation, depth estimation and object recognition. Code is available at https: / /et:::/ggttttt github. com detp/ind////d////////e/////dd ssl hsic.
bound much better behaved:
Introduction
Learning general-purposs visual representations without human supervision is a long-standing goal of machine learning. Specifically, we wish find a feature extractor that captures the image semanties of a large unlabeled collection of images, so that e.g. various image understanding tasks can be achieved with simple linear models. One approach takes the latent representation a likelihood based generative model [2-8]; such models, though, solve a harder problem than necessary since semantic features need not capture low-level details of the input. Another option model for "pretext task," such as predicting the position of image patches, identifying rotations, or image inpainting [9-14]. Designing good pretext tasks, however, a subtle art, with little theoretical guidance available. Recently, a class of models based on contrastive learning [15-22] has seen substantial success: dataset images are cropped, rotated, color shifted, etc. into several views, and features are then trained pull together representations of the "positive" pairs of views of the same
train self-supervised
'These authors contributed equally.
Work done in part while at the Toyota Technological Institute at Chicago.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Supervised
ImageNot top-] accuracy (96
80
SSL-HSIC (ours]
78
ranan...........................
BYL SWAV
SinCLR (4x)
76
74
SimCLR (2x)
Barlow Twins
72
v2
70
SimCLR
24M
94M
250M
375M (ReeNie-50-5o)
Figure 2: Statistical dependence view of con- trastive learning: representations transformed images should highly depend image iden- tity. Measuring dependence with HSIC, this pushes different images' representation distribu- tions apart (black arrows) and pulls representa- tions of the same image together (colored shapes).
Number of parameters
Figure 1: Top-1 accuracies with linear evaluation for different ResNet architecture and methods: supervised (as in [25]), SSL--HIIC (with a target network; ours), BYOL [25]. SWAV [18], SimCLR [19], MoCo v2 [20] and Barlow Twins [22].
source image, and push apart those of "negative" pairs (from different images). These methods are either understood from an information theoretic perspective as estimating the mutual information between the "positives" [15], or explained as aligning features subject to a uniformity constraint [23]. Another line of research [24, 25] attempts to learn representation without the "negative" pairs, but requires either target network or stop-gradient operation to avoid collapsing.
We examine the contrastive framework from statistical dependence point of view: feature represen- tations for a given transformed image should be highly dependent on the image identity (Figure 2). To measure dependence, we propose new loss for self-supervised learning which we call SSL-HSIC. Our loss is inspired by HSIC Bottleneck [27, 28], an alternative to Information Bottleneck [29]. where we use the image identity as the label, but change the regularization term.
the Hilbert-Schmidt Independence Criterion (HSIC) [26], and
Through the dependence maximization perspective,
present unified view of various self- supervised losses. Previous work [30] has shown that the success of InfoNCE cannot be solely attributed to properties of mutual information, in particular because mutual information (unlike kernel measures of dependence) has no notion of geometry in feature space: for instance, all invertible encoders achieve maximal mutual information, but they can output dramatically different representa- tions with very different downstream performance [30]. Variational bounds on mutual information do impart notions of locality that allow them to succeed in practice, departing from the mutual information quantity that they try to estimate. We prove that InfoNCE, a popular such bound, in fact approximates SSL-HSIC with variance- regularization. Thus, InfoNCE can be thought of as working because it implicitly estimates a kernel-based notion of dependence. We additionally show SSL-HSIC is related to metric learning, where the features learn to align the structure induced by the self seff-supervieed labels. This perspective is closely related to the objective of BYOL [25], and can explain properties such as alignment and uniformity [23] observed in contrastive learning.
Our perspective brings additional advantages, in computation and in simplicity of the algorithm, compared with existing approaches. Unlike the indirect variational bounds mutual information [15, 31,311 32]. SSL.-HSIC can be directly estimated from mini-batches of data. Unlike "negative-free" methods, the SSL-HSIC loss itself penalizes trivial solutions, so techniques such as target networks are not needed for reasonable outcomes. Using a target network does improve the performance of our method, however, suggesting target networks have other advantages that are not yet well understood. Finally, we employ random Fourier features [33] in our implementation, resulting in cost linear in batch size.
Our main contributions are as follows:
We introduce SSL-HSIC, a principled self-supervised loss using kernel dependence maximization. We present unified view of contrastive learning through dependence maximization, by establishing relationships between SSL-HSIC, InfoNCE, and metric learning.

---

Our method achieves top-1 1 accuracy of 74.8% and top-5 accuracy of 92.2% with linear evaluations (see Figure for a comparison with other methods), top-1 accuracy of 80.2% and Top-5 accuracy of 94.7% with fine-tuning, and competitive performance on a diverse set of downstream tasks.
2 Background
2.1 Self-supervised learning
Recent developments in self-supervised learning, such as contrastive learning, try to ensure that features of two random views of an image are more associated with each other than with random views of other images. Typically, this is done through some variant of classification loss, with one "positive" pair and many "negatives." " Other methods can learn solely from "positive" pairs, however. There have been many variations of this general framework in the past few years.
Van den Oord et al. [15] first formulated the InfoNCE loss, which estimates lower bound of the mutual information between the feature and the context. SimCLR [19, 34] carefully investigates the contribution of different data augmentations, and scales up the training batch size to include more negative examples. MoCo [17] increases the number of negative examples by using a memory bank. BYOL [25] learns solely on positive image pairs, training so that representations of one view match that of the other under a moving average of the featurizer. Instead of the moving average, SimSiam [24] suggests a stop-gradient on one of the encoders is enough to prevent BYOL from finding trivial solutions. SwAV [18] clusters the representation online, and uses distance from the cluster centers rather than computing pairwise distances of the data. Barlow Twins [22] uses an objective related to the cross- corseloorel matrix of the two views, motivated by redundancy reduction. It is perhaps the most related to our work in the literature (and their covariance matrix can be connected to HSIC [35]), but our method measures dependency more directly. While Barlow Twins decorrelates components of final representations, we maximize the dependence between the image's abstract identity and its transformations.
On the theory side, InfoNCE proposed tion between the representation of two views of the same image [15, 32]. Tschannen et al. [30] observe that InfoNCE performance cannot be explained solely by the proper- ties of the mutual information, however, but is influenced more by other factors, such as the formulation of the estimator and the architecture of the feature extractor. Essentially, representations with the same MI can have drastically different representational qualities. To see this, consider a problem with two inputs, A and (Figure 3. green and purple). and a one- dinenimmnsi featurizer, parameterized by the integer M,, which maps to Uniform( (0, 2,.... ..M)) and B to Uniform( (1,, 3.... ...., 2M + 1)). When M 0, the inputs are encoded into linearly separable features A = and B = (Figure 3, bottom). Otherwise when M > 0. they are interspersed like ABAB ABAB a representation which much harder to work with for downstream learners. Nevertheless. the mutual information between the features of any two augmentations of the same input (a positive pair) is independent of M, that is H - H Z Z2] = log 2 for any M. Note that InfoNCE would strongly prefer M O, indeed behaving very differently from MI.
variational bound on Mutual Informa-
MI(2),a") log 2 for all cases
better for SSL
Figure 3: Three distributions of positive examples for two classes (green and purple) that have the same mutual informa- tion, but drastically different quality for downstream learners.
Later theories suggest that contrastive losses balance alignment of individual features and uniformity of the feature distribution [23], or in general alignment and some loss-defined distribution [36]. We propose to interpret the contrastive loss through the lens of statistical dependence, and relate it to metric learning, which naturally leads to alignment and uniformity.
2.2 Hilbert-Schmidt Independence Criterion (HSIC)
The Tillbert-Schmide Independence Criterion (HSIC) [26] is a kernel-based measure of depen- dence between probability distributions. Like mutual information, for a wide range of kernels HSIC(X, Y)- = if and only if X and are independent [37], and large values of the measure correspond to "more dependence." Unlike mutual information, HSIC incorporates notion of ge-
3

---

ometry (via the kernel choice), and is both statistically and computationally easy to estimate. It has been used in a variety of applications, particularly for independence testing [38], but has also been maximized in applications such as feature selection [39], clustering [40, 41], active learning [42], and as classification loss called HSIC Bottleneck [27, 28] (similar ideas were expressed in [43, 44]).
HSIC measures the dependence between two random variables by first taking nonlinear feature transformation of each, say á X' + F and e y - & (with F and reproducing kernel Hilbert spaces, RKHSes'), ), and then evaluating the norm of the cross-covariance between those features:
HSIC(X, Y) = [ (()) B[o(X)] (()) Iss
Here I|- the Hilleet-Schhiie norm, which in finite dimensions the usual Frobenius norm. HSIC measures the scale of the correlation in these nonlinear features, which allows it to identify nonlinear dependencies between and with appropriate features @ and v.
Inner products in an RKHS are by definition kernel functions: k(r, x') = (6(x), o(x')) and 1(y. 5/) = (vb(t) (b())) g)g. Let (X', Y") (x", Y") be independent copies of (X, Y): this gives
HSIC(X,Y) E [+ XY, Y*] - 2 [M(X, Y")) + E [K X') E ((Y, Y") (2)
HSIC is also straightforward to estimate: given i..d. samples {(<I1,1))..... (xN,, UN)} drawn i.i.d. from the joint distribution of (X, Y). Gretton et al. [26] propose an estimator
vi).....
1 HITC(X, (N 1)2 Tr(KHLH),
where K,y = k( xy) and L, Uy,, f1) are the kernel matrices, and H = = 1 is called the centering matrix. This estimator has an (1/N) bias, which is not a concern for our uses; however, an unbiased estimator with the same computational cost is available [39].
3 Self-supervised learning with Kernel Dependence Maximization
Our method builds on the self-supervised learning framework used by most of the recent self- supervised learning approaches [16-19, 22, 24, 25]. For dataset with N points T; each point goes through a random transformation t (x;; (e.g. random crop), and then forms a feature representation x! = (tP(()))) with an encoder network fg. We associate each image z; with its identity yı, which works one-hot encoded label: y/ E RR- and (ıı)a = iff d = (and zero otherwise). To match the transformations and image identities, we maximize the dependence between z; and yi such that zi is predictive of its original image. To build representations suitable for downstream tasks, also need to penalize high-variance representations. These ideas come together in our HSIC-based objective for self-supervised learning, which we term SSL-HSIC:
HSIC (Z, Y) + 7 HSIC 22,
Unlike contrastive losses, which make the 2 from the same closer and those from different Tū more distant, we propose an alternative way to match different transformations of the same image with its abstract identity (e.g. position in the dataset).
Our objective also resembles the HSIC bottleneck for supervised learning [27] (in particular, the version of [, but ours uses a square root for HSIC 2). The square root makes the two terms on the same scale: HSIC(Z, Y' is effectively dot product, and /HSIC Z) a norm, so that e.g. scaling the kernel by a constant does not change the relative amount of regularization," this also gives better performance in practice.
Due to the one-hot encoded labels, we can re-write HSTC (2, as (see Appendix A)
[ISIC(ZY) x E Exn.sgrpps [k(21, Egg Egg [k(21, 22)]
'I slight abuse of notation, we use (V) for the tensor product ((x) & ('v) EFOG.
HS or equiv- alently [46] the distance correlation [47]; the kemne- target alignment [48, 49] is also closely related. Here, the overall scale of either kernel does not change the objective. Our HSIC((Y Y) is constant (hence absorbed in ?), and we found additive penalty to be more stable in optimization than dividing the estimators.
Z) / "Other prior work on maximizing HSIC [41, 45] used HSIC(Z, Y)/ /HSICZZ

---

where the first expectation is over the distribution of "positive" pairs (those from the same source image), and the second one is sum over all image pairs, including their transformations. The first term in (5) pushes representations belonging to the same image identity together, while the second term keeps mean representations for each identity apart (as Figure 2). The scaling of (STC(Z, Y) depends on the choice of the kernel over Y, and is irrelevant to the optimization.
This form also reveals three key theoretical results. Section 3.1 shows that InfoNCE is better understood as an HSIC-based loss than a mutual information between views. Section 3.2 reveals that the dependence maximization in HSIC Y) can also be viewed as form of distance metric learning, where the cluster structure is defined by the labels. Finally, HSIC(Z, Y) is proportional to the average kernel-based distance between the distribution of views for each source image (the maximum mean discrepancy, MMD; see Appendix B.2).
3.1 Connection InfoNCE
In this section we show the connection between InfoNCE and our loss; see Appendix B for full details. We first write the latter its infinite sample size limit (see [23] for a derivation) as
LI (0) = - Pa [k(111 22]] + Es, log Ezg lexp (k(21, =2))]
(6)
where the last two expectations are taken over all points, and the first over the distribution of positive pairs. The kernel k(21,2)
originally formulated
scoring function in form of dot
product [15], and then a scaled cosine similarity [19]. Both functions are valid kernels.
Now assume that k(=1, 22) doesn't i deviate much from Bag [k(21, #2)]. Taylor-expand the exponent in (6) around Ezg [k(z1, 22)]. then expand log(1 + EZg (...)) P (....) We obtain an HSIC Y)- based objective:
LintoNCE(O) = E Ea (K(21, 22)] + E,, Es, (1= 22) ) + E., Warz, (M22, +2)]
(7)
a- HSSI(Z,Y)
variance penalty
Since the scaling of HSIC Y) 2) irrelevant to the optimization, we assume scaling replace & with =. In the small variance regime, we can show that for the right y,
-HSIC (2 + HSIC (Z, Z) S LIntONCE(O) + ol (variance).
(8)
For HSTC
also have that
-HSIC (2 + HSIC (Z, S LSSL-HSIC(O))
due the square root. InfoNCE and SSL-HSIC in general don't quite bound each other due to discrepancy in the variance terms, but in practice the difference is small.
Why should we prefer the HSIC interpretation of InfoNCE? Initially, InfoNCE was suggested
variational approximation to the mutual information between two views [15]. It has been observed, however, that using tighter estimators of mutual information leads worse performance [30]. is also simple to construct examples where InfoNCE finds different representations while the underlying MI remains constant [30]. Alternative theories suggest that InfoNCE balances alignment of "positive" examples and uniformity of the overall feature representation [23], or that (under strong assumptions) it can identify the latent structure hypothesized data- gettraainn process, akin to nonlinear ICA [50]. Our view is consistent with these theories, but doesn't put restrictive assumptions on the input data or learned representations. Section (summarized Table 8a), we show that our interpretation gives rise to better objective in practice.
3.2 Connection metric learning
Our SSL-HSIC objective is closely related to kernel alignment [48], especially centered kernel alignment [49]. As a kernel method for distance metric learning, kernel alignment measures the agreement between a kernel function and target function. Intuitively, the self-supervised labels Y imply cluster structure, and HSIC Y) estimates the degree of agreement between the learned features and this cluster structure in the kernel space. This relationship with clustering is also
5

---

established in [40, 41, 45], where labels are learned rather than features. The clustering perspective is more evident when we assume linear kernels over both Z and Y, and Z is unit length and centered;"
1 Tr(y
HSTC(2,,) &
zJ ZY) + +T((Z 2) - NM
M
N
M
N
M 1 NM =
N M
1 1 -NM, (10)
M
with . the number of augmentations per image and 2 E /MM the average feature vector of the augmented views of z4, We emphasize, though, that (10) assumes centered, normalized data with linear kernels; the right-hand side of (10) could be optimized by setting all z. to the same vector for each i. but this does not actually optimize HSIC(Z,Y).
Equation (10) shows that we recover the spectral formulation [51] and sum sum-of-squares loss used in the k-means clustering algorithm from the kernel objective. Moreover, the self-supervised label imposes that the features from transformations of the same image are gathered in the same cluster. Equation (10) also allows us to connect SSL-HSIC to con-contrastive objectives such as BYOL, although the connection is subtle because of its use of predictor and target networks. If each image augmented with two views, we can compute (10) using 2; (z + x so the clustering loss becomes x >; ||= =-113- This is exactly the BYOL objective, only that 2 in BYOL comes from a target network. The assumption of centered and normalized features for (10) is important in the case of B BOL: without it, BYOL can find trivial solutions where all the features are collapsed to the same feature vector far away from the origin. The target network is used to prevent the collapse. SSL-HSIC, on the other hand, rules out such a solution by building the centering into the loss function, and therefore can be trained successfully without a target network or stop gradient operation.
3.3 Estimator of SSL-HSIC
To use SSL-HSIC, we need to correctly and efficiently estimate (4). Both points are non-trivial: the self-super framework implies non-i..d. batches (due to positive examples), while the estimator in (3) assumes i..d. data; moreover, the time to compute (3) is quadratic in the batch size.
First, for HSIC(Z, 2) we use the biased estimator in (3). Although the i.i.d. estimator (3) results in an (1)BB) bias for B original images in the batch size (see Appendix A), the batch size B is large in our case and therefore the bias is negligible. For HSIC(Z, Z, Y) the situation is more delicate: the i..dd estimator needs re-scaling, and its bias depends on the number of positive examples M, which is typically very small (usually 2). We propose the following estimator:
AI HSIĆ(Z,Y) = N BM - 1)
1
Σ M(2P.22 spl
Σ k(2 kjpl
(1I)
B2M
M
where and j index original images, and p and their random transformations; k is the kernel used for latent Z, ( is the kernel used for the labels, and Al = [(i. i) ((i,j) (1 for same labels minus ( for different labels). Note that due to the one-hot structure of self-supervised labels Y. the standard (i.i.d.-based estimator would miss the 1/N scaling and the M - 1 correction (the latter is important in practice, as we usually have M- = 2). See Appendix A for the derivations.
For convenience, we assume Al- = N (any scaling of can be subsumed by 7), and optimize
HSIL- ( HSIČ(Z,Y) + HSIČ(Z, Z).
(12)
The computational complexity of the proposed estimators is O(²2M²) for each mini-batch of size B with M augmentations. We can reduce the complexity to O(BM) by using random Fourier features (RFF) [33], which approximate the kernel k((1, approximation R(21) R(22) for R(z) : D= > RD,, such that k(21, 22) = E [R(2)) R(22)] Fourier frequencies are sampled independently for the two kernels on Z in HSIC(Z, Z) at each training step. We leave the details on how to construct R(z) for the kernels we use to Appendix C.
222 with a carefully chosen random D-dimensional
*Centered Z is a valid assumption for BYOL, as the target network keeps representations of views with different image identities away from each other. For nigh-dimensional unit vectors, this can easily lead to orthogonal representations. We also observe centered representations empirically: see Appendix B.3.

---

Experiments
In this section, we present our experimental setup, where we assess the performance of the repre- sentation learned with SSL-HSIC both with and without a target network. First, train model with a standard ResNet-50 backbone using SSL-HSIC as objective on the training set of ImageNet ILSVRC-2012 [1]. For evaluation, we retain the backbone as feature extractor for downstream tasks. We evaluate the representation on various downstream tasks including classification, object segmentation, object detection and depth estimation.
(x),
I, y
j0
K
(x),
SSL-HSIC
2
Target
I(y, v)
Figure 4: Architecture and SSL-HSIC objective. A self-supervised label y indicator of the image identity - is associated with an image x. Image transformation functions are sampled and applied to the original image, resulting in views e (a:) and (z)) Features =² and 2² are obtained after passing the augmented views through encoder (f), projector (g). and possibly predictor (g) networks, while label y is retained. Kernel matrices, K for the latents and L for the labels, are computed on the mini-batchh of data; SSL-HSIC is estimated with K and L as in (12). The blue boxes reflect two potential options: when using target network, is moving average of 0, and a predictor network q added; without the target network, g is removed and & is simply equal to 0.
4.1 Implementation
Architecture Figure 4 illustrates the architecture used for SSL-HSIC in this section. To facilitate comparison between different methods, our encoder fo uses the standard ResNet-50 backbone without the final classification layer. The output of the encoder is 2048- dimension embedding vector, which is the representation used for downstream tasks. As in B YOL [25], our projector g and predictor networks are 2-layer MLPs with 4096 hidden dimensions and 256 output dimensions. The outputs of the networks are batch-normalized and rescaled to unit norm before computing the loss. We use an inverse multiquadric kernel (IMQ) for the latent representation (approximated with 512 random Fourier features that are resampled at each step; see Appendix C for details) and a linear kernel for 3. When training without target network, unlike SimSiam [24], we do not stop gradients for either branch. If the target network is used, its weights are an exponential moving average of the online network weights. We employ the same schedule as BYOL [25], 1- (cos(at/T) + 1)/2 with t the current step and T the total training steps.
labels. y (4)
Image augmentation Our method uses the same data augmentation scheme as BYOL (see Ap- pendix D. Briefly, we first draw a random patch from the original image and resize it to 224 x 224. Then, we apply random horizontal flip, followed by color jittering, consisting of a random sequence of brightness, contrast, saturation, hue adjustments, and an optional grayscale conversion. Finally Gaussian blur and solarization are applied, and the view normalized with ImageNet statistics.
Optimization We train the model with batch size of 4096 on 128 Cloud TPU v4 cores. Again, following [19, 25], we use the LARS optimizer [52] with cosine decay learning rate schedule over 1000 epochs. The base learning rate to all of our experiments is 0.4 and it is scaled linearly [53] with the batch size ir 0.4 batch_sizee seersssiie//555 256. All experiments
weight decay of 10-6.
Learning kernel parameters We use a linear kernel for labels, since the type of kernel only scales (12). Our inverse multiquadric kernel for the latent Z has additional kernel scale parameter. We op- timize this along with all other parameters, but regularize to maximize the entropy of the distribution ko (s), where sty = |||, 2 this amounts to maximizing log kgg (s) (Appendix D.1.2).
7

---

4.2 Evaluation Results
Linear evaluation on ImageNet Learned features are evaluated with the standard linear evaluation protocol commonly used in evaluating self-supervised learning methods [15-22, 25]. Table 1 reports the top-1 and top-5 accuracies obtained with SSL-HSIC on ImageNet validation set, and compares previous self-supervised learning methods. Without target network, our method reaches 72.2% top-1 and 90.7% top-5 accuracies. Unlike BYOL, the SSL-HSIC objective prevents the network from finding trivial solutions as explained in Section 3.2. Adding the target network, our method outperforms most previous methods, achieving top- accuracy of 74.8% and top-S accuracy of 92.2%. The fact that we see performance gains from adopting target network suggests that its effect not yet well understood, although note discussion in [25] which points to its stabilizing effect.
Table Linear evaluation on the Ima- geNet validation set.
Table Fine-tuning on 1%, 10% and 100% of the Ima- geNet training set and evaluating on the validation set.
1(%)
Top-5((8)
Top-1(8)
Top-5(98)
Supervised [54]
75.9
92.8
1%
10%
100%
10%
100%
SimCLR [19]
69.3
89.0
Supervised [54]
56.4
75.9
92.8
McCo v2 [20)
71.1
90.1
BYOL. [25]
74.3
91.6
SimCLR [19]
48.3
65.6
76.0
75.5
87.8
93.1
SwAV [18]
75.3
BYOL [25]
77.7
93,9
Barlow Twins [22]
73.2
91.0
SwAV [18]
53.9
70.2
78.5
89.9
SSL-HSIC (wlo target)
72.2
90.7
Barlow Twins [22]
SSL-HSIC (w/ target)
74.8
92.2
SSL-HISIC (wio larget)
45.3
65.5
76.4
72.7
87.5
93.2
SSI. (w/ target)
52.1
67.9
77.2
77.7
88.6
93.6
Semi-supervised learning on ImageNet We fine-tune the network pretrained with SSL-HSIC on 1%, 10% and 100% of ImageNet, using the same ImageNet splits as SimCLR [19]. Table 2 summarizes the semi-supervised learning performance. Our method, with or without a target network, has competitive performance both data regimes. The target network has the most impact on the small-data regime, with % labels.
Table 3: Comparison of transfer learning performance on 12 image datasets. Supervised-IN is trained on ImageNet with supervised pretrainining. Random init trains individual dataset with randomly initialized weights. MPCA refers to mean pla-class accuracy; AP50 is average precision at IoU=0.5.
Dataset
Birdsnap Ca Top-1
Cifar10
Cifar100 DTD
Aireratt Food Flowers
Pets
Cars
SUN397
VOC2007
Metric
MPCA Top-1
Top-1
Top-1
MPCA Top-1
MPCA
MPCA Top-1
Top-1
AP50
Tinear:
Supervised-IN [19]
53.7
94.5
93.6
78.3
74.9
61.0
72.3
94.7
91.5
67.8
61.9
82.8
SimCLR [19]
37.4
90.3
90.6
71.6
74.5
50.3
68.4
90.3
83.6
50.3
58.8
80.5
BYOL [25]
57.2
94.2
91.3
78.4
75.5
60.6
75.3
96.1
90.4
66.7
62.2
82.5
HSI. (wlo target)
50.6
92.3
91.5
75.9
75.3
57.9
73.6
95.0
88.2
59.3
61.0
81.4
SSL-HSIC target)
57.8
95.4
83.3
Fine-tined:
Supervised-IN [19]
75.8
43.3
97.5
86.4
74.6
86.0
88.3
97.6
92.1
92.1
94.3
85.0
Random init [19]
72.6
95.9
80.2
85.9
SimCLR R [19]
75.9
92.1
97.7
85.9
73.2
88.
88.2
97.0
89.2
91.3
63.5
84.1
BYOL[25]
76.3
93.8
97.8
86.1
76.2
88.5
97.0
91.7
91.6
63.7
85.4
SSL-HSIC (wlo target)
73.1
91.5
97.4
85.3
75.3
87.1
87.5
96.4
90.6
91.6
62.2
84.1
SSL- (w/ Lurget)
74.9
93.8
97.8
84.7
75.4
88.9
87.7
97.3
91.7
91.8
61.7
84.1
Transfer other classification tasks investigate the generality of the representation learned with SSL-HSIC, we evaluate the transfer performance for classification on 12 natural image datasets [55-64] using the same procedure as [19, 25, 65]. Table shows the top-1 accuracy of the lincar evaluation and fine-tuning performance on the test set. SSL-HSIC gets state-of-the-ar performance on 3 of the classification tasks and reaches strong performance on others for this benchmark, indicating the learned representations are robust for transfer learning.
Transfer other vision tasks To test the ability of transferring to tasks other than classification, we fine-tune the network on semantic segmentation, depth estimation and object detection tasks. We use Pascal VOC2012 dataset [58] for semantic segmentation, NYU v2 dataset [66] for depth
estimation and COCO [67] for object detection. Object detection outputs either bounding box or object segmentation (instance segmentation). Details of the evaluations setup is Appendix D.2.
Table 4 and Table 5 shows that SSL-HSIC achieves competitive performance on all three vision tasks.

---

Table Fine-tuning performance on semantic segmentation and depth estimation. Mean Intersection over Union (mloU) is re- ported for semantic segmentation. Relative error (rel), root mean squared error (rms), and the percent of pixels (pet) where the error below 1.25 thresholds are reported for depth estimation.
Table 5: Fine-tume Fine-tuning performance on COCO object detection tasks. Precision, averaged over 10 IoU (Intersection over Union) thresh- olds, reported for both bound-
VOC2012
NYU
ing box and object segmentation.
Method
mlol;
peL-<
1.25
pet.< 1.25?
pet.<. 1.25
mms
rel
Method
Aph
Apmk
SimCLR
Supervised-IN
74,4
75.2
83.3
81.1
95.3
96.5
98.8
99.1
0.573
0.557
0.134
0.127
Supervised
39.6
35.6
BYOL
76.3
84.6
96.7
99.1
0.541
0.129
SimCLR
39.7
35.8
SSL- turget)
74.9
84.1
96.7
99.2
0.539
0.130
MoCo v2
40.1
36.3
SSL-HSICCW/ target)
76.0
96.8
99.1
0.548
0.130
BYOL
41.6
37.2
SWAV
41.6
37.8
SSL-IISIC(W/ target)
40.5
36.3
SSL-HSIC(W/ target)
41.3
36.8
5
Ablation Studies
We present ablation studies to gain more intuition on SSL-HSIC. Here, we use ResNet-50 backbone trained for 100 epochs on ImageNet, and evaluate with the linear protocol unless specified.
ResNet architectures In this ablation. investigate the performance of SSL-HSIC with wider and deeper ResNet architecture. Figure and Table 6 show our main results. The performance of SSL--HSI gets better with larger networks. We used the supervised baseline from [25] which training framework is based on ([19) reports lower performance). The performance gap between SSL--HSI and the supervised baseline diminishes with larger architectures. In addition, Table 7 presents the semi-supervise learning results with subsets I 1%, 10% and 100% of the ImageNet data.
Table 6: Top-1 and top-5 accuracies for different ResNet architectures using linear evaluation proto- coL.
Table 7: Top-1 1 and top-5 accuracies for different ResNet architectures using semi- supervised fine-tuning
SSL-IISIC
BYOL[25]
Sup-[25]
Top1
Top5
ResNet
Top1
Top5
Topl
Top5
Topl
Top5
ResNet
1%
10%
100%
1%
10%
100%
50 (1x)
74.8
92.2
74.3
91.6
76.4
92.9
(0x)
52.1
67.9
77.2
77.7
88.6
93.6
(2x)
77.9
77.4
79.9
(2x)
61.2
72.6
79.3
83.8
91.2
50 (4x)
79.1
94.5
78.6
94.2
80.7
95.3
50 (4x)
67.0
75.4
79.7
87.4
92.5
94.8
200 (2x)
79.6
94.8
79.6
94.9
80.1
95.2
200X2x)
69.0
76.3
80.5
88.3
92.9
95.2
Regularization term We compare performance of InfoNCE with SSL-HSIC in Table 8a since they be seen approximating the same HSIC(Z, Y) objective but with different forms of regularization. We reproduce the InfoNCE result in our codebase, using the same architecture and data augmentiation as for SSL-HSIC. Trained for 100 epochs (without target network), InfoNCE achieves 66.0% top-1 and 86.9% top-5 accuracies, which is better than the result reported in [19]. For comparison, SSL-HSIC reaches 66.7% top-1 and 87.6% top-5 accuracies. This suggests that the regularization employed by SSL- is more effective.
Kernel type We investigate the effect of using different kernel latents Z. Training without target network or random Fourier feature approximation, the top-1 accuracies for linear, Gaussian, and inverse multiquadric (IMQ) kernels are 63.27%, 66.67% and 66.72% respectively. Non-linear kernels indeed improve the performance; Gaussian and IMQ kernels reach very similar performance for 100 epochs. We choose IMQ kernel for longer runs, because its heavy-tail property can capture more signal when points are far apart.
Number of RFF Features Table 8b shows the performance of SSL-HSIC with different numbers of Fourier features. The RFF approximation has a minor impact on the overall performance, as long as resample them; fixed sets of features performed poorly. Our main result picked 512 features, for substantial computational savings with minor loss in accuracy.
Batch size Similar to most of the self-supervised learning methods [19, 25), SSL-HSIC benefits from using a larger batch size during training. However, the drop of performance from using smaller batch size is not pronounced as it is in SimCLR[19] as shown in Table 8c.

---

Table 8: Linear evaluation results when varying different hy hyperparameters.
(a) Regularization
(b) # Fourier features
(c) Batch size
(d) Projectar/predictor size
Top-1
Top
RFFS
Top-I(S)
Top I(5)
Ouiput Dim
Top 1(8)
SSL-HSIC InfNCEE
64
66.0
Size
SSL-HSIC
SimCLR
65.4
66.0
86.9
128
66.2
128
66.0
256
66.2
63.7
57.5
256
66.4
512
66.4
512
65.6
60.7
512
66.6
1024
66.5
1024
66.7
62.8
1024
66.6
2048
66.5
2048
67.1
64.0
No Approx,
66.7
4096
66.7
64.6
Projector and predictor output size Table 8d shows the performance when using different output dimension for the projector/predictor networks. The performance saturates at 512 dimensions.
6 Conclusions
We introduced SSL-HSIC, loss function for self-supervised representation learning based on kernel dependence maximization. We provided a unified view on various self-supervised learning losses: we proved that InfoNCE, a lower bound of mutual information, actually approximates SSL.-HSIC with variance- based regularization, and we can also interpret SSL-HSIC as metric learning where the cluster structure is imposed by the self-supervised label, of which the BYOL objective a special case. We showed that training with SSL-HSIC achieves performance par with the state-of-the-an on the standard self-supervised benchmarks.
Although using the image identity as self-supervised label provides good inductive bias, it might not be wholly satisfactory; we expect that some images pairs are in fact more similar than others. based e.g. on their ImageNet class label. It will be interesting to explore methods that combine label structure discovery with representation learning (as in SwAV [18]). this paper, we only explored learning image representations, but in future work SSL-HSIC can be extended to learning structure for Y as well, building on existing work 45].
Broader impact
Our work concentrates providing more theoretically grounded and interpretable loss function for self-supervised learning. A better understanding of self-supervised learning, especially through more interpretable learning dynamics, is likely to lead to better and more explicit control over socictal biases of these algorithms. SSL-HSIC yields an alternative, clearer understanding of cxisting self- supervised methods. As such, it is unlikely that our method introduces further biases than those already present for self-supervised learning.
The broader impacts of the self superviee learning framework is an area that has not been studied by the AI ethics community, but we think it calls for closer inspection. An important concern for fairness of ML algorithms is dataset bias. ImageNet is known for number of problems such as offensive annotations, non-visual concepts and lack of diversity, in particular for underrepresented groups. Existing works and remedies typically focus on label bias. Since SSL doesn't use labels, however, the type and degree of bias could be very different from that of supervised learning. To mitigate the risk of dataset bias, one could employ dataset re-balancing to correct sampling bias [68] or completely exclude human images from the dataset while achieving the same performance [69].
A new topic to investigate for self self-supeeeieed learning is how the bias/unbiased representation could be transferred to downstream tasks. We are not aware any work in this direction. Another area of concern is security and robustness. Compared to supervised learning, self-supervised learning typically involves more intensive data augmentation such as color jittering. brightness adjustment, etc. There is some initial evidence suggesting self-supervised learning improves model robustness [70]. However, since data augmentation can either be beneficial [71] or detrimental [72] depending on the type of adversarial attacks, more studies are needed to assess its role for self-supervised learning.
10

---

Acknowledgments and Disclosure of Funding
The authors would like thank Olivier J. Hénaff for valuable feedback on the manuscript and the help for evaluating object detection task. We thank Aaron Van den Oord and Oriol Vinyals for providing valuable feedback on the manuscript. We are grateful to Yonglong Tian, Ting Chen and the BYOL authors for the help with reproducing baselines and evaluating downstream tasks.
This work supported by DeepMind, the Gatsby Charitable Foundation, the Wellcome Trust, NSERC, and the Canada CIFAR Al Chairs program.
References
[1] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. "ImageNet large scale visual recognition challenge." International Journal of Computer Vision 115.3 (2015), pp. 211-252.
[2] G. E. Hinton and R. R. Salakhutdinov. "Reducing the dimensionality of data with neural networks." Science 313.5786 (2006), pp. 504-507.
[3] P. Vincent, H. Larochelle, Lajoie, Y. Bengio, P.-A. Manzagol, and L. Bottou. "Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion." " Journal of Machine Learning Research 11.12 (2010).
[4] A. Coates and A. Y. Ng. "Learning feature representations with k-means." Neural networks: Tricks of the trade. Springer, 2012, pp. 561-580.
[5] I. Higgins, L. Matthey, Pal, C. Burgess, Glorot, M. Botvinick, Mohamed, and A. Ler- chner. "beta- Learning basic visual concepts with a constrained variational framework."
ICLR. 2016.
[6] van den Oord, O. Vinyals, and K. Kavukcuoglu. "Neural Discrete Representation Learning." NeurlPS. 2017. arXiv: 1711 00937.
[7] W. Grathwohl, R. Q. Chen, J. Bettencourt, and D. Duvenaud. "Scalable Reversible Genera- tive Models with Free-form Continuous Dynamics." ICLR. 2019. arXiv: 1810. 01367.
[8] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P.
Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. "Language Models are Few-Shot Learners." NeurIPS. 2020. arXiv: 2005. 14165.
[9] C. Doersch, A. Gupta, and A. A. Efros. "Unsupervised Visual Representation Learning by Context Prediction." ICCV. 2015. arXiv: 1505. 05192.
[10] G. Larsson, M. Maire, and G. Shakhnarovich. "Learning representations for automatic col-
orization." " ECCV. Springer. 2016, pp. 577-593.
[11] M. Noroozi and P. Favaro. "Unsupervised learning of visual representations by solving jigsaw puzzles." ECCV. Springer. 2016, pp. 69 84.
[12] Pathak. P. Krahenbuhl, Donahue, Darrell, and A. A. Efros. "Context encoders: Feature learning by inpainting." CVPR. 2016, pp. 2536-2544.
[13] R. Zhang, P. Isola, and A. A. Efros. "Colorful image colorization." ECCV. Springer. 2016,
pp. 649-666.
[14] S. Gidaris, P. Singh, and N. Komodakis. "Unsupervised representation learning by predicting image rotations." " ICLR. 2018. arXiv: 1803. 1803.07728.
[15] van den Oord, Li, and O. Vinyals. "Representation learning with contrastive predictive coding." 201 8. aivi: 1807 03748.
[16] P. Bachman, R. D. Hjelm, and W. Buchwalter. "Learning representations by maximizing mutual information across views." NeurlPS. 2019. arXiv: 1906. 00910.
[17] K. He. H. Fan, Y. Wu, S. Xic, and R. B. Girshick. "Momentum Contrast for Unsupervised Visual Representation Learning." CVPR. 2019. arXiv: 1911.05722
[18] M. Caron, Misra, Mairal, Goyal, P. Bojanowski, and Joulin. "Unsupervised learning of visual features by contrasting cluster assignments." NewrIPS. 2020. arXiv: 2006. 09882.
[19] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. "A Simple Framework for Contrastive Leamingg of Visual Representations." ICML. 2020. arXiv: 2002.05709.
11

---

[20] X. Chen, H. Fan, R. B. Girshick, and K. He. "Improved Baselines with Momentum Contrastive Learning." " 2020. arXiv: 2003. 2003.04297.
[21] O. Henaff. "Data-efficient image recognition with contrastive predictive coding." ICML. 2020. arXiv: 1905. 09272.
[22] J. Zhontar, I.. Jing, Misra, Y. LeCun, and S. Deny. "Barlow Twins: Self-Supervised Learning
via Redundancy Reduction." " 2021. arXiv: 2103.03000
[23] T. Wang and P. Isola. "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere." " 2020. arXiv: 2005. 10242.
[24] X. Chen and K. He. "Exploring Simple Siamese Representation Learning." " 2020. arXiv:
10566.
[25] J.-B. Grill, Strub, F. Altché, C. Tallec, H. Richemond, E. Buchatskaya, Doersch, A.
Pires. Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. "Bootstrap
Your Own Latent: A New Approach to Self-Supervised Learning." NeurIPS. 2020, arXiv:
2006. 07733.
[26] A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf. "Measuring statistical dependence with Hilbert-Schmidt norms." " ALT. Springer. 2005, pp. 63-77.
[27] K. W.-D. Ma, J. P. Lewis, and W. B. Kleijn. "The HSIC Bottleneck: Deep Learning without Back-Propagation." AAAI. 2020.
[28] R. Pogodin and P. Latham. "Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks." NeurlPS (2020).
[29] N. Tishby and N. Zaslavsky. "Deep Learning and the Information Bottleneck Principle." IEEE Information Theory Workshop. 2015. arXiv: 1503. 1503.02406.
[30] M. Tschamnen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic. "On mutual information maximization for representation learning." ICLR. 2020. arXiv: 1907 13625.
[31] I. Belghazi, S. Rajeswar, A. Baratin, R. D. Hjelm, and A. C. Courville. "MINE: Mutual
Information Neural Estimation." " ICML. 2018. arXiv: 1801. 04062.
[32] B. Poole, S. Ozair, A. van den Oord, A. A. Alemi, and G. Tucker. "On Variational Bounds of Mutual Information." ICML. 2019. arXiv: 1905. 06922.
[33] A. Rahimi and B. Recht. "Random Features for Large-Scale Kernel Machines." NeurIPS. 2007.
[34] T. Chen, S. Korbblith, K. Swersky, M. Norouzi, and G. Hinton. "Big Self- Seff-Supervise Models are Strong Semi- Superviee Learners." NeurIPS. 2020. arXiv: 2006. 10029.
[35] Y.-H. H. Tsai, S. Bai, L.-P. Morency, and R. Salakhutdinov. "A note on connecting barlow twins with negative-sample- frre contrastive learning." arXiv preprint arXiv:2104.13711 (2021).
[36] T. Chen and L. Li. "Intriguing Properties of Contrastive Losses." 2020. arXiv: 2011 028003.
[37] 7. Szabó and B. K. Sriperumbudur: "Characteristic and Universal Tensor Product Kernels." Journal of Machine Learning Research 1 18.233 (2018), pp. 1-29.
[38] A. Gretton, K. Fukumizu, C. H. Teo, L. Song. B. Schölkopf, and A. J. Smola. "A kernel statistical test of independence." NeurIPS. 2007.
[39] L. Song, A. Smola, A. Gretton, J. Bedo, and K. Borgwardt. "Feature selection via dependence maximization." Journal of Machine Learning Research 13.5 (2012).
[40] L. Song, A. Smola, A. Gretton, and K. M. Borgwardt. "A Dependence Maximization View of Clustering." ICML. 2007.
[41] M. B. Blaschko and A. Gretton. "Learning Taxonomies by Dependence Maximization." "
NeurIPS. 2009.
[42] S. Jain, G. Liu, and D. Gifford, "Information Condensing Active Learning." " 2020. aiviv
2002. 07916.
[43] D. Wu, Y. Zhao, Y.-H. H. Tsai, M. Yamada, and R. Salakhutdinov. "Dependency Bottleneck'
in aato-enccodig Architectures: an Empirical Study." 2018. arXiv: 1802. 05408.
[44] A. Nøkland and L. H. Eidnes. "Training neural networks with local error signals." ICML. 2019.
arXiv: 1901666656.
[45] M. B. Blaschko, W. Zaremba, and A. Gretton. "Taxonomic Prediction with Tree-Structured Covariances." Machine Learning and Knowledge Discovery in Databases. Springer Berlin Heidelberg, 2013, pp. 304-319.
12

---

[46] D. Sejdinovic, B. Sriperumbudur, A. Gretton, and K. Fukumizu. "Equivalence of distance- based and RKHS-based statistics in hypothesis testing." " The Annals of Statistics 41.5 (2013).
pp. 2263-2291.
[47] G. J. Székely, M. L. Rizzo, and N. K. Bakirov. "Measuring and testing dependence by correlation of distances." " The Annals of Statistics 35.6 (2007), pp. 2169-776999999
[48] N. Cristianini, J. Shawe-Taylor, A. Elisseeff, and J. Kandola. "On Kernel- Target Alignment." NeurIPS. Vol. 14. 2002.
[49] C. Cortes, M. Mohri, and A. Rostamizadeh. "Algorithms for learning kernels based on centered alignment." The Journal of Machine Learning Reseurch 13.1 (2012), pp. 795-828.
[50] R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. "Contrastive
Learning Inverts the Data Generating Process." 2021. arXiv: 2102 08850.
[51] H. Zha, X. He. C. Ding. M. Gu, and H. D. Simon. "Spectral relaxation for k-means clustering." NeurIPS. 2001, pp. 1057-1064.
[52] Y. You, I. Gitman, and B. Ginsburg. "Scaling SGD Batch Size to 32K for ImageNet Training."
2017. arXiv: 1708. 03888.
[53] P. Goyal, P. Dollár, R. B. Girshick, P. Noordhuis, L.. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. "Accurate, Large Minibatch SGD: Training ImageNet in I Hour." 2017. arXiv:
1706. 02677.
[54] X. Zhai, A. Oliver, A. Kolesnikov, and L. Beyer. "S*L: Self-Supervised Seni-Supervised Learning"" ICCV. 2019. arXiv: 1905. 03670.
[55] L. Fei-Fei, R. Fergus, and P. Perona. "Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories." " CVPR
Workshop. 2004.
[56] M.-E. Nilsback and A. Zisserman. "Automated flower classification over a large number of classes." 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. IEEE 2008, pp. 722-729.
[57] A. Krizhevsky. "Learning multiple layers of features from tiny images." " 2009. URL: https:
/ //www. . cs toronto. aatu/"kriz/eearning wwwaarnnnnnaauurreeeeeeeeeeeeeee fe -ase--209--T- pdf.
[58] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. "The Pascal visual object classes (VOC) challenge." IJCV 88.2 (2010), pp. 303-338.
[59] J. Xiao, J. Hays, K. A. Fhinger, A. Oliva, and A. Torralba. "SUN database: Large-scale scene recognition from abbey to zoo." CVPR. 2010.
[60] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. "Cats and Dogs." " CVPR. 2012.
[61] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. "Describing Textures in the Wild." CVPR. 2013. arxiv: 1311.3611
[62] J. Krause, J. Deng, M. Stark, and L. Fei-fei, "Collecting a large-scale dataset of fine-grained Fine-Grained Visual Categorization." 2013.
cars. The Second Workshop
[63] L. Bossard, M. Guillaumin, and L. Van Gool. "Food-101-minis discriminative components with random forests." " ECCV, Springer. 2014, pp. 446-461.
[64] S. Maji, F. Rahtu, J. Kannala, M. B. Blaschko, and A. Vedaldi. "Fine-Grained Visual Classifi-
cation of Aircraft." " arXiv: 1306.51 .5511
[65] S. Kornblith, J. Shlens, and Q. Le. "Do Better ImageNet Models Transfer Better?" CVPR. 2019. arXiv: 1805. 08974.
[66] N. Silberman, D. Hoiem, Kohli, and Fergus. "Indoor Segmentation and Support Inference from RGBD Images." ECCV. 2012.
[67] T.-Y. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, Dollár, and C. L. Zitnick. "Microsoft COCO: Common Objects in Context."
ECCV. 2014. arXiv: 1405. 0312.
[68] K. Yang, K. Qinami, L. Fei-Fei, J. Deng, and O. Russakovsky. "Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy." Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 2020, pp. 547-558.
[69] Y. Asano, C. Rupprecht, A. Zisserman, and A. Vedaldi. "PASS: An ImageNet replacement for self-supervised pretraining without humans" (2021).
13

---

[70] D. Hendrycks, M. Mazeika, S. Kadavath, and D. Song. "Using self-supervised learning can improve model robustness and uncertainty." arXiv preprint arXiv: 12340 (2019).
[71] A. Sablayrolles, M. Douze, C. Schmid, Y. Ollivier, and H. Jégou. "White-box vs black-box: Bayes optimal strategies for membership inference." International Conference Machine Learning. PMLR. 2019, pp. 5558-5567.
[72] D. Yu, H. Zhang, W. Chen, J. Yin, and T.-Y. Liu. "How Does Data Augmentation Affect Privacy Machine Learning?" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. 12. 2021, pp. 10746-10753,
[73] L. Grafakos. Classical Fourier Analysis. Vol. 2. Springer, 2008.
[74] R. Piessens. "The Hankel transform." The Transforms and Applications Handbook 2.9 (2000).
[75] F. Johansson al. mpmath: a Python library for eebitrary-precisionn point arithmetic (version 0. 08) http::/mpmaatt
org/ 2013.
[76] J. Long, E. Shelhamer, and T. Darrell. "Fully Convolutional Networks for Semantic Segmenta-
tion." CVPR. 2015. arXiv: 1411 4038.
[77] O. Hénaff, S. Koppula, J.-B. Alayrac, A. van den Oord, O. Vinyals, and Carreira, "Efficient
Visual Pretraining with Contrastive Detection." 2021. arXiv: 2103. 10957.
[78] He, G. Gkioxari, Dollár, and
B. Girshick. "Mask R-CNN." ICCV. 2017. arXiv:
1703. 06870.
[79] T.-Y. Lin, P. Dollár, R. B. Girshick, K. He. B. Hariharan, and S. J. Belongie. "Feature Pyramid Networks for Object Detection." CVPR. 2017. arXiv: 1612. 03144.
14


Reviews:
[{"review_id": "zzk98cpOzB5", "reviewer": "Reviewer_7VXB", "summary": "This paper proposed a new Self-Supervised Learning framework with the Hilbert Schmidt Independence Criterion(SSL-HSIC). SSL-HSIC yields a new understanding of InfoNCE, while can optimize statistical dependence in time linear in the batch size. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well written with detailed proofs and theoretical results.\nThis paper gives a unified view of contrastive learning through dependence maximization, by establishing relationships between SSL-HSIC, InfoNCE, and metric learning with detailed analysis. \n\n**Post Rebuttal Response**\nI have read the response and would remain the current score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Although the paper gives detailed theoretical proof, the experiments are somewhat weak. I still have some concerns:\n1）The most related works SwaV and Barlow Twins outperform the proposed method in some experimental results, as shown in Table 1,2,5. What are the main advantages of this method compared with SwaV and Barlow Twins?\n2) HSIC(Z, Y) can be seen as a distance metric in the kernel space, where the cluster structure is defined by the identity. Although this paper maps identity labels into the kernel space, the information of one-hot label is somewhat limited compared with views embeddings in Barlow Twins. \n3)Since the cluster structure is defined by the identity. How does the number of images impact the model performance? Do more training images make the performance worse or better ?\n\n4) BYOL in the abstract should be explained for its first appearance.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "drTgvAwbXdJ", "reviewer": "Reviewer_j3wZ", "summary": " \nThis paper proposed a new loss function for self-supervised learning using the Hilbert-Schmidt Independence Criterion (HSIC) to measure the kernel space dependence between the learned latent representation of images and their augmented versions. The main contribution is that authors demonstrated the connections between the proposed framework and InfoNCE, metric learning and Maximum Mean Discrepancy, respectively. An algorithm with random fourier feature kernel estimation is proposed and the method is shown to have comparable performance with state-of-the-art self-supervised learning (SSL) paradigms. The sample complexity and bias analysis for kernel estimation are provided.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This paper offers an interesting viewpoint and an alternative formulation for maximal mutual information-based self-supervised learning (SSL). It makes much intuitive sense that one could maximize the minus HSIC to maximize the dependence between two random variables. The proposed method has some advantages in terms of measuring the gap between the population formulation and its sample based ones, reminiscent of HSIC and kernel estimation results. The combination of HSIC and random feature-based kernel computation is natural. The sample complexity analysis supporting this part makes sense. The regularization also makes sense, which is essentially a norm regularization in the kernel space.\n- The paper is relatively well written, although the introduction of Y (image identify) may be confusing.\n\n\nWeaknesses and questions:\n- The novelty seems to be limited and impact in terms of changing the SSL paradigms could be marginal.  Essentially, the proposed method follows [35] but changes to another mutual information (MI) approximator. The effectiveness of this approximator is not rigorously shown but argued from a bit of a loose angle. In particular, the authors argued that exact MI maximization may not be desired due to invertible ambiguities. But it is hard to see why using a looser lower bound is a better choice. In other words, the motivation of using HSIC to replace MI is not crystal clear.  It indeed makes sense that maximizing dependence of different views may not give the best result of SSL. But it feels not easy to grasp the idea of intentionally using a loose lower bound for MI. Why is a loose lower bound better? How loose should one aim for? This is quite unclear to the reviewer. One suspicion is that maybe a good regularization (like the HSIC(Z,Z) regularization) can fix the ambiguity brought by the dependence criterion. How much performance gain is brought upon by the –HSIC part? Ablation study on $\\gamma$ could perhaps provide some insights. \n- The connections between the proposed regularized HSIC and InfoNCE criteria are somewhat interesting, but not identical or equivalent. It does not feel that such a loose connection merits a new publication.\n- Using the kernel method may make the computation a bit more challenging. Indeed, random features could alleviate the memory burden, but it introduces another level of approximation. This also goes back to the previous comment - there seems to be no strong motivation for using HSIC.\n\n- Following the above comment, what is the unique or salient feature of this entire SSL framework is unclear. As a benefit of the proposed method mentioned by the authors, large negative sampling is not needed. However, to the reviewer’s understanding, [23] also has this advantage, which interprets contrastive loss as alignment and uniformity. How is the performance of the proposed method compared with [23]?\n- The proposed formulation seems confusing. In Eq. (4) Y is used, but the actual estimation for HSIC(Z,Y) does not require the information of Y (otherwise it is supervised rather than self-supervised). This was quite confusing when first reading the paper, since SSL does not need the label information Y. By looking into the supplementary material, it was then clever that essentially the kernel dependence maximization is computed between the sample and its augmented counterpart. The Y information was never used in computation. Perhaps it would be better and clearer to elaborate on this gap between the conceptual formulation and the practical implementation. Is it needed to introduce the “label” or image identity term y when formulating the problem? Removing it may make the story smoother, in the reviewer’s opinion.\n-  The analyses mostly are concerned with sample complexity and kernel estimation accuracy. These are of course not uninteresting, but are not exactly relevant to the main question, i.e., how the HSIC formulation affects SSL?\n- The experiment results do not really show improvement compared with state-of-the-art in most cases. For example, in transfer learning tasks (Table 3), most results are not improved compared to other approaches, even the better ones are only with very small margins. Similar results are also observed in segmentation (Table 4) and object detection (Table 5) tasks. Besides, as mentioned above, study on gamma could be interesting and helpful to evaluate the contribution of the regularization term, i.e., HSIC(Z,Z). Overall, it seems that the proposed method does not make a big difference in terms of empirical performances. \n- On the computational side, it seems to require more computations when using kernel functions even though random Fourier features are employed. Is there a time consumption comparison with existing methods to illustrate the computation cost?   \n\n- There are some discussions on scaling the HSIC estimator to reduce bias, and some analyses to support the design. How much does the scaling impact the final result? Could this be validated or supported by some simulations? This part also reads a bit confusing. From the paragraph, it seems that the goal is to handle non-iid samples. But the proofs in Supp Sec. A seem to still use iid assumptions. Can the authors clarify a bit?\n- Overall, the HSIC based dependence measuring seems to be reasonable, but its motivation and impacts in practice are not crystal clear.\n \n \n \n \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "none noted", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "O1OlO75L_dI", "reviewer": "Reviewer_dxjH", "summary": "Authors proposes a novel HSIC based self-supervised learning loss for image representation learning. It relates to InfoNCE and Metric Learning and provides representations that perform well on multiple tasks. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper introduces HSIC based SSL loss that learns generic image representations that perform well across multiple image related tasks. Strong empirical results on ImageNet classification and other tasks such as semantic segmentation, depth estimation and object detection.  It makes a good case where generic learned representations valuables as task specific representations will do better in terms of performance.\n\nFramework uses unique identity of the image as part of loss function (y) which is a strong assumption but authors do describe it as an assumption which can be addressed.\n\nPaper is well written with strong theoretical support of the intuition behind SSL-HSIC loss with ablation studies and choice of parameters.\n\nA t-SNE visualizations of SSL-HSIC with class label would provide a qualitative analysis of the learn representations. It would be interesting to know if it helps improve  Top-1% accuracy of imageNet where SwaV does better (Table 1).\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the technique are addressed but bias and explainability aspects of learned representations are not, even qualitatively.  ", "review_text": "", "extra_scores": {"ethical_concerns": "No specific ethical concerns.", "needs_ethics_review": "Yes", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "AtG-phnmeMY", "reviewer": "Reviewer_ArQY", "summary": "The paper proposes to use the HSIC metric in order to impose view feature dependency on the image identity during self-supervised learning (SSL). The authors made non-trivial efforts in both deriving the SSL-HSIC approach and theoretically linking SSL-HSIC with the existing mainstream SSL approaches. Empirical results show that the method offers competitive results. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: The paper is well written, and the paper is fairly easy to follow. The intuition behind the method is clear and convincing: HSIC regularizer explicitly penalizes the dependency between different images while it encourages views from the same images to be close. The theoretical results are strong. \n\nWeakness:\n1. The eventual loss boils down to Eq. (11). Particualrly, if one employs Gaussian Kernel on the eventual loss on Eq. (11), it is immediate that the first term in Eq.(5)/Eq. (11) is proportional to BYOL up to a scaler, whereas the second term further plays additional contrastive role in the form of inter-image $\\ell_2$ norm on top of the first term; In this sense, this extra contrastive term perhaps explains why SSL-HSIC always remains comparable to BYOL and behaves potentially superior over BYOL. However, my main concern is that, given that SSL-HSIC is closely related to BYOL, it seems the proposed approach SSL-HSIC is still inferior to the BYOL in most of the experiments shown in both Table 2 and 3. Is there any insight behind this phenomenon?  Intuitivelly, SSL_HSIC poses tighter constraints based on this extra contrastive term. Also, for the same reason, the empirical results failed to advacate on the SSL-HSIC side in comparison to BYOL, making the empirical justification much weaker for SSL-HSIC.  Would you like any discussion on this? \n\n2. The notation of $\\Delta l$ term is confused, why this term is out of the sumation over index i and j, since it is an entrywise calculation given specific i and j?  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the HSIC loss for self-supervised learning. While there was significant enthusiasm about this paper from the reviewers, there were also major concerns surrounding the conclusions of the experiments and questions about what to take away from the results. The authors do provide some good clarifications in their general response surrounding the fact that they don’t expect that HSIC should beat other methods and that this is not the point of the paper. However, the exact conclusions are still a bit muddy in the text. 

An important point: there is a high degree of similarity between the objective underlying HSIC and BarlowTwins. This must be clarified in the paper. Now, the description is rather vague and there are no explicit ablations to compare or understand the differences between the two. 

Decision: Accept (Poster)