The hidden uniform cluster prior in self-supervised learning
Authors: Mido Assran
Randall Balestriero
Quentin Duval
Florian Bordes
Ishan Misra
Piotr Bojanowski
Pascal Vincent
Michael Rabbat
Nicolas Ballas
Venue: ICLR 2023 poster (2023)
Keywords: self-supervised learning
unsupervised learning
representation learning
transfer learning

Abstract: A successful paradigm in representation learning is to perform self-supervised pretraining using tasks based on mini-batch statistics; (e.g., SimCLR, VICReg, SwAV, MSN). We show that in the formulation of all these methods is an overlooked prior to learn features that enable uniform clustering of the data. While this prior has led to remarkably semantic representations when pretraining on class-balanced data, such as ImageNet, we demonstrate that it can hamper performance when pretraining on class-imbalanced data. By moving away from conventional uniformity priors and instead preferring power-law distributed feature clusters, we show that one can improve the quality of the learned representations on real-world class-imbalanced datasets. To demonstrate this, we develop an extension of the Masked Siamese Networks (MSN) method to support the use of arbitrary features priors.

Published as a conference paper at ICLR 2023
THE HIDDEN UNIFORM CLUSTER PRIOR IN SELF-SUPERVISED LEARNING
Mahmoud Assran 1,2.3, 1,2.3, Randall Balestriero', Quentin Duval', Florian Bordes Ishan Misra', Piotr Bojanowski', Pascal Vincent Michael Rabbat Nicolas Ballas
Meta AI (FAIR) ?MeGill University, ECE
Mila, Quebec Al Institute *Universite de Montreal, DIRO
ABSTRACT
A successful paradigm in representation learning is to perform self- self-supervised pretraining using tasks based on mini-batch statisties (e.g.. SimCLR, VICReg, SwAV, MSN). We show that in the formulation of all these methods is an overlooked prior to learn features that enable uniform clustcring of the data. While this prior has led to remarkably semantic representations when pretraining on class-balanced data, such as ImageNet, we demonstrate that it can hamper performance when pretraining class caassimmallnced data. By moving away from conventional uniformity priors and instead preferring power-law distributed feature clusters, we show that one can improve the quality of the learned representations on real-world class-imbalanced datasets. To demonstrate this, we develop an extension of the Masked Siamese Networks (MSN) method to support the use of arbitrary features priors.
INTRODUCTION
Self-supervised pretraining has emerged as a highly effective strategy for unsupervised representation learning, with remarkable advances demonstrated by joint-embedding methods (Chen et al., 2020b; Caron et al., 2021; Bardes et al., 2021; Assran et al., 2022). In the context of visual data, these approaches typically learn representations by training a neural network encoder to produce similar embeddings for two more views of the same image. However, since outputting constant vector regardless of the input would satisfy this objective, one of the main challenges with jobbd-emb methods is to prevent such pathological solutions. A common remedy is employ regularizer that maximizes the volume of space occupied by the representations. This is sometimes referred to as the volume maximization principle. In practice, the volume maximization principle is implemented in variety of ways, for example, by contrasting negative samples (Bromley et al., 1993; He et al., 2019; Chen et al., 2020b), by removing correlations in the feature space (Bardes et al., 2021; Zbontar et al., 2021), or by finding high entropy clusterings of the data (Asano et al., 2019; Caron et al., 2020; Assran et al., 2021: 2022). When pretrained on the ImageNet dataset (Russakovsky al., 2015), these methods have been shown to produce representations that encode highly semantic features (Caron et al., 2020; 2021; Assran et al., 2022).
However, the commonly used ImageNet-1 1K dataset is relatively class-balanced, which is in contrast to most rorldworld settings, where data is often dlas--mbalanced and semantic concepts follow a long-tailed power-law distribution (Newman, 2005; Mahajan et al., 2018; Van Horn et al., 2018). Indeed, it has been shown that pretraining the same joint- cobndddmbedding methods on can lead to significant drops in performance (Tian et al., 2021a). Such observation is problematic in that it significantly hinders the applicability of modern research advances with joint-embedding methods weal-world settings.
datasets
In this work, we explore the use of joint-embedding methods for class-imbalanced datasets. First, theoretically show that current methods with volume maximization regularizers such as VI- CReg (Bardes et al., 2021), SWAV (Caron et al., 2020), MSN (Assran et al., 2022) and SimCLR (Chen al., 2020b) (with limited assumptions), have uniform feature prior; i.e., bias learn features that enable grouping the data into clusters of roughly equal size. Consequently, these joint-embedding
s wan@meetet com

---

Published as a conference paper at ICLR 2023
methods will penalize features that do not uniformly cluster the data, even if such features correlate well with class information; see Figure
Second, empirically validate that joint-embedding methods employing volume maximization regularizers are sensitive to the mini-batch class distributions. These approaches fail to learn class- discriminative features when the samples within mini-batch do not follow uniform class dis- tribution. This observation partially explains why performance degrades when pretraining with real-world data, where sampled mini-batches often contain highly imbalanced class distributions.
Finally, based this observation, we propose conventional uniformity priors and instead reformulate self-supervise criteria prefer long-tailed feature priors that are more aligned with the distribution of semantic concepts in real-world datasets. In par- ticular, we extend Masked Siamese Networks (MSN) of Assran al. (2022) to support the use of arbitrary features priors, and refer to this extension as Prior Matching for Siamese Networks (PMSN). When pretraining the iNaturalist 2018 dataset (Van Horn et al., 2018), which is naturally long-taiie longetailed, we demonstrate that moving away from uniform priors leads to more semantic representations and improved transfer on downstream tasks.
away from
(a) K dlas-balanced
BACKGROUND
Given the recent success of joint-embedding methods, there is a growing literature that aims build better understanding of their behaviour. Several works have sought to develop generalization bounds for join-embeddding methods with volume maximization penal- ties (Arora et al., 2019; Balestriero & LeCun, 2022). Other works have sought to better understand the differences between various vol- ume maximization penalties and connect them under limited assump- tions (Garrido al., 2022). general, has been shown that lz- normalized contrastive losses can be decomposed into an "alignment" plus volume maximization component that scatters the representations uniformly on the unit hypersphere (Wang & Isola, 2020). Following this observation, other works (Chen et al., 2021) have sought to refor- mulate contrastive losses to scatter representations either (a) uniformly on the unit hypercube, or (b) onto Gaussian distributions (which have the highest entropy amongst all distributions with given variance). There is also theoretical work (Tian et al., 2021b) which aims to un- derstand why certain joint-embedding methods, such BYOL (Grill et al., 2020), can avoid representation collapse without explicit use of a volume maximization penalty.
K-means class-imbsllanced
Figure Impact of uniform cluster prior in K K-means when class distribution of data is im- balanced. K-means cluster- ing depicted in color (green red). Ground-truth cluster sep- aration depicted with dotted black line. When uniform fea- prior is not satisfied, K- means identify undesirable features for discriminating be- tween data points.
While these works have helped build our understanding on the training dynamics of coint-embedding methods, they do not directly explain why empirical use of these methods with real-world class- imbalanced data has often led to degradation in downstream task performance (Tian et al., 202 la; Goyal al., 2022) (see Appendix for broader discussion of related work). In this work, we explore the use of joint-embedding methods with class-imbalanced data. In particular, we theoretically show that a broad range of methods (beyond contrastive) prescribe a uniform feature prior, and that this prior is detrimental when pretraining with class-imbalanced data.
3 UNIFORM PRIORS IN MODERN SELF-SUPERVISED LEARNING
In this section, we theoretically show that common SSL methods such VICReg (Bardes et al., 2021), SwAV (Caron et al., 2020), MSN (Assran et al., 2022), and (with limited assumptions) SimCLR (Chen et al., 2020b), correspond to variants of K-means, and thereby impose uniform cluster prior; i.e., a bias to learn features that enable uniform clustering of the data. The governing assumption in K-means is the presence of isotropic data clusters, with roughly equal number of
2

---

Published as a conference paper at ICLR 2023
samples per cluster (Wu al., 2009; Liang al., 2012). When this assumption is not satisfied in practice, K K-means may learn undesirable features for discriminating between samples (cf. Figure 1).
3.1 BACKGROUND: K-MEANS FORMULATIONS AND THE UNIFORM PRIOR
Explicit (Centroid) K-means. Recall that K-means proposes centroid based clustering of the data. In particular, given a set of N data points {æn Ji K-means partitions the elements into K disjoint groups Xi...., KK, such that > Zi=l
K
N. The K-means objective can be written
Iiin
1 12
x
where the optimization problem is to identify the members of the disjoint sets X1...., Xx. and pa (the hthh cluster centroid) is precisely the average of the members of Xg.
Implicit (Centroid) K-means. One important note that we will carry through our study that K-means does not require explicit computation of the cluster centroids μı to evaluate the objective. To see this standard result, one can relate the sum of pairwise distances to the sum radial distances for any data partition,
K
K
Σ x
P
Σ Je,at' EX2
||=
2'112
2
This result suggests that the K-means loss can be minimized by either learning cluster centroids (explicit K- K-means), or by learning cluster memberships (implicit K-means). We make this relation precise in Proposition 1, which proven in Appendix C.1 and was already noted in Zha al. (2001); Awasthi et al. (2015).
Proposition The explicit K-means problem, defined by learning set of centroids p1,-... ,.....//KK, J.K. admits the same global optimum the implicit K-means problem, defined by learning cluster membership matrix P E {0, 13 NXK xK such that P1k = 1n.
The fundamental assumptions governing the success of K-means lie in having clusters with roughly the same number of samples and intra-cluster data covariance that isotropic with the form al, where > 0 and I is the identity matrix (Wu et al., 2009; Liang et al., 2012). In the sequel, will show that various SSL. methods can be seen as employing either explicit or implicit K-means.
3.2 How K-MEANS NATURALLY EMERGES FROM SELF-SUPERVISED LOSSES
In this section, we demonstrate how standard SSL methods naturally employ K-means at their core. Although this result might seem intuitive for methods that explicitly compute a clustering of the representations, we surprisingly found that this is also the case for some methods (e.g., VICReg) that do not explicitly involve centroid parameters or a clustering step.
Implicit K-means: VICReg and SimCLR. The VICReg loss (Bardes et al., 2021) consists of three terms, one measuring the & distance between the positive view pairs, one encouraging the off-diagonal entries of the embedding covariance matrix to go to 0, and one encouraging its diagonal entries to be greater than 1. One standard simplification done in practice study VICReg is to switch the variance and covariance terms to a single term so that the new VICReg loss is given by
C [or - 11
11
Y N
Σ
(G)..l
1,j=1
where Cov(2 (Z) is the covariance matrix of the vectors 2 2....., Zn. the constants 7 > O are hyper- parameters, and G.,; E {0, 1} equal to when the representations z; and zj correspond positive views of the same image. From this formulation, we relate VICReg to the implicit K-means algorithm in Proposition 2, which is proven in Appendix C.2. In particular, note how the summation term on the right hand side of equation 3 recovers the right hand side of equation 2.
Proposition 2. VICReg with the embeddings, with additional regularizer enforcing orthogonality of the centroids.
"i > a recovers the K-means loss from equation on
3

---

Published as a conference paper at ICLR 2023
The arper-parameter requirement 7 a is commonly employed with VICReg in practice to ensure invariance to the pre-defined set of data-auggentttions used to construct the positive image views. In fact, in the opposite case, where > y, degenerate whitening information about the input samples due to the non-linearity of the deep neural network mapping, as shown in Balestriero & LeCun (2022). Note that under mild conditions, the VICReg loss has been shown to be equivalent to the SimCLR loss (Garrido et al., 2022). In such scenarios, Proposition 2 would also directly apply to the SimCLR method.
be learned without keeping any
Explicit K-means with soft constraints: MSN. A common relaxation of constrained K-means is to remove the hard cluster assignment constraint, such that the condition P., E {0, generalized to P., E [0, 1] (Wang et al., 2010). This relaxes constrained K-means to the more general Gaussian Mixture Model (GMM) formulation, where each data point can partially belong to all clusters with some probability mass. Since, the derivation of the GMM loss (i.e., the ELBO and log-likeliine (egllikeihhodd) is beyond the scope of this section, we simply state the objective as
K [p(x)]
K
min
x - -
N
log det(2xx + (KL((( || T),
(4)
1 MEX
2
mEX
where π is the cluster prior and [p(x))| is the posterior probability of input belonging to cluster k obtained from
p(x: softmax W x: log( -
|
diag (TW)
where W [,n,.... Mk] concatenates the centroids, and diag(-) extracts the diagonal of its matrix argument into a column vector.
Of particular interest to us the case in which the prior set to the uniform distribution, [T] 1/K for all & E [ the covariance matrix is isotropic, Σn = of for 2 0, and the centroids and data vectors are Eg-normalized, in which case equation 4 simplifies to
K
softmax W x)) x
min
1
2
Σ H (softmax(Wi (W x)) ZEX
(5)
2
REX k=1
One can also express the simplified GMM objective in equation 5 in terms of the marginal entropy (with a summation inside the entropy term) by following the ELBO derivations of Hoffman & Johnson (2016). The Masked Siamese Network (MSN) loss (Assran et al., 2022) with positive pairs wn, x, and posteriors pa softmax W xn/o)) /)) P = softmax W 2,/ /o), with temperature o, is
N
1
Hlpt. -
(6)
N
where À > O, and p p= En Pn.. We show that the MSN objective can be seen as variant of K-means variant employing an explicit ctuster-membership penalty (cf. Appendix C.3). Proposition 3. MSN recovers the GMM loss from equation with the variation that the lg distance replaced with the cross-entropy distance of the posterior.
Explicit K-means with hard constraints: SWAV Among popular K-means variants, one particular version requires specification of the cardinality of the clusters (Kleindessner et al., 2019; Bradley et al., 2000; Rujeerapaiboon et al., 2019). Specifically, the number of items in each cluster [X&] is strictly enforced take specific value N₁. The resulting K-means formulation becomes
K
min
Σ
1 - 1
(x ( k= IXX
When N₂. = N
K. we obtain strict enforcement of the uniform cluster prior, which is otherwise implicit in the formulation, but not strictly enforced. This variant of K-means was previously used in Wang et al. (2010). The SwAV (Caron et al., 2020) loss with positive posterior pairs Pa, P (defined similarly to those in equation 6) is
N
H Hppt (p) subject to Pl R 1ks
P11== 1N,
(8)
N
R=
4

---

Published as a conference paper at ICLR 2023
where P = ipi,.... ..... p concatenates the predictions. In Caron al. (2020), this constraint is enforced in practice by projecting the matrix P onto the constraint set in each iteration using the Barkhorn-Knopp algorithm (Cuturi, 2013). SwAV thus recovers strictly constrained variant of K-means (cf. Appendix C.4).
Proposition 4, SwAV recovers the constrained K-means loss from equation 5 with constraint enforce- ment through Sinkhorn- Knoop normalization.
Note that for all presented SSL methods and their corresponding K-means variants, enforcing normalization of the centroids and features does not reduce the generality of the results; in this case, simply obtain a correspondence to similar variants of spherical K-means (Hornik et al., 2012).
NEGATIVE EFFECT OF UNIFORM PRIORS FOR
DATA
As proven in Section 3, joint-embedding methods employing volume maximization regularizers have an uniform feature prior. Following this observation, in this section we empirically demonstrate that such methods are sensitive to non-uniform mini-batch class distributions.
Experimental setup. We explore three joint-embedding methods employing diverse collapse prevention strategies: SimCLR (Chen et al., 2020b), VICReg (Bardes et al., 2021), and MSN (Assran et al., 2022). We also compare the performance of those models to ietaace--bsse methods such as MAE (He al., 2021) and data2vec (Baevski et al., 2022), which do not employ volume maximization regularizers. In this evaluation, all models are pretrained on the ImageNet-1 I K dataset without access to the class labels. To explore sensitivity to mini-batci class distributions, we explore two dichotomous sampling strategies (which do require semantic knowledge of the image classes).
One strategy, termed class-balanced sampling, constructs the mini-batches in each iteration by first randomly selecting 960 classes out of the 1000 ImageNet-1K K classes, and then sampling an equal number of images from each class.! Given that ImageNet is relatively class-balanced, this strategy produces mini-batches with similar statistics traditional uniform sampling. Another strategy, termed class-imbalanced sampling, constructs the mini-batches in each iteration by first randomly selecting K < 960 classes of the 1000 ImageNet-11 1 classes, and then sampling an equal number of images from each class, such that the total mini-batch size is the same as under the class- balanced sampling strategy. We pick K as the smallest value needed reach the default batch size for each respective method: 8 for SimCLR (which has the a default batch size of 4096) and for VICReg and MSN (which have default batch sizes of 1024). Note that the class-imbalanced strategy maintains the same marginal probability of sampling individual data points as compared to the bassncal sampling strategy (see Appendix for a derivation of this equivalence).
After pretraining all models using the various sampling strategies, we evaluate performance on a wide range of downstream tasks requiring different levels of abstraction, i.e., classification with CI- FAR 100 (Krizhevsky et al., 2009), Places205 (Zhou et al., 2014), and iNat18 (Van Horn et al., 2018); object counting with Clevr/Count (Johnson al., 2017); and depth prediction with Clevr/Dist (John- son et al., 2017) and KITTI (Geiger et al., 2013). We also evaluate in-distribution performance of ImageNet classification (Russakovsky et al., 2015; Chen et al., 2020b). Additional pretraining and evaluation details can be found in Appendix D; the full set of results can be found in Appendix L.
Empirical observations. As can be seen in Table 1, the performance of joint-embedding methods employing volume maximization regularizers degrades significantly on all the semantic downstream tasks (INIK, CIFAR100, Places205, Clevr/Count) when the mini- mini-batches sampled during pretraining are not class-balanced (e.g., drops by as much as 17.7 top-1 on INIK), but remain relatively stable (and even marginally improve) on low-level depth prediction tasks Clevr/Dist and KITTI, suggesting that class-imbalanced pretraining leads the model to capture lower-level (less semantic) features.
By contrast, evaluations with instanced- based methods data2vec and MAE in Table show different trends. The MAE method employs a simple pixel-reconstructiin loss for representation learning, and thus does not explicitly compute mini-batch statistics during pretraining. The data2vee method is more
We choose 960 classes instead of 1000 so that the overall batch size is divisible by the number of GPUs utilized for distributed training.
5

---

Published as a conference paper at ICLR 2023
Table 1: Transfer: Evaluation of the pretraining mini-batch sampling distribution various downstream tasks. Each cell reports the task performance when pretraining with class-balanced sampling minus the task performance when pretraining with class bass-imbalanced sampling. Sampling imbalanced mini- bati-b during pretraining Icads to significant drop in image classification tasks for joint-embedding methods with volume maximization priors (SimCLR, MSN, vICReg), whereas instance-based methods, which do not employ such regularization (MAE, data2voc), relatively unaffected.
ImageNet
INat18
CIFARI00
Places205
Clevr/Count
Clevr/Dist
KITTI
SimCLR
-11.2
-8.0
-10.2
-5.2
-4.3
+0,9
+1.2
MSN
-17.7
-15.1
-13.2
-4.6
-6.4
+1.9
-1.6
VICReg
-17.7
-17.3
-12.0
-6.0
-3.0
+0.7
-1.1
data2vec
-0.8
+0.3
-1.6
-2.1
-1.5
-0.1
MAE
-0.1
+1.4
+2.5
+0.1
-0.8
+0.0
+0.0
similar to MSN, SimCLR, and VICReg in that it utilizes a joint-embedding architecture; however, in contrast to those methods, data2vec does not explicitly compute mini-batch statistics during pretraining, and instead relies on architectural heuristics and careful hyperparameter choices to prevent collapse. When evaluating these methods with class-balanced pretraining versus class-imbalanced pretraining, we observe virtually no change in downstream task performance. Only methods with explicit volume maximization terms exhibit sensitivity the mini-batch class distribution.
Visualizing learned prototype vectors. In Figure 2, we use RCDM (Bordes et al., 2022b) to visualize the prototypes learned by an MSN model pretrained on IN1 with either balsned or class-imbalanced mini-batch distributions. A prototype here refers to row in the weight matrix of the final linear layer in the encoder. Each row in Figure corresponds to samples generated by conditioning on single prototype using various random seeds. Characteristics that remain constant across row in Figure reflect information contained the prototype, whereas characteristics that vary reflect information that is not contained (i.e., to which the representations are invariant). When pretraining with class-balanced mini-batches, the emergent features tend to be associated with high-level concepts, such as specific ImageNet classes (Figure 2a). In contrast, when pretraining with imsalancb mini- batches, the learned features tend to be associated with low-level concepts, such shape, pose, or texture (Figure 2b).
Pretraining with class-balanes balanced mini-batehs sampling
Pretraining with class- inssliimbllncee mini-batch sampling
Figure Visualization of prototypes learned by MSN model pretrained ImageNet-IK with either class-balanced or class-imbalanced mini-batch distributions. use RCDM (Bordes et al., 2022b) to en- able visualization of the prototypes (details in Appendix H). Each row corresponds to samples generated by conditioning on prototype with various random seeds. Features that remain constant across the row depict information contained in the prototypes, whereas features that vary depict information that is not contained. class-balanced
emergent features tend to be associated with high level concepts, such as specific ImageNet classes. b) By contrast, when pretraining with class-imbalanced mini-batches, the learned features tend to be associated with low-level concepts, such as shape, pose, or texture.
a) When pretraining with class balanced batche,

---

Published as a conference paper at ICLR 2023
5 PRIOR MATCHING FOR SIAMESE NETWORKS
Section 4 validates that in settings where the samples within a mini-batch do not follow a uniform class distribution, jointeembedding methods with volume maximization penalties encode less class- oriented features, and perform worse on downstream semantic classification tasks. In this section, we demonstrate that using alternative (long-tailed) feature priors can lead to representations of higher semantic level when pretraining on real-world iaaaliiebleeceed data.
Siamese Networks with Arbitrary Prior. In Assran al. (2022), the MSN prior is explicitly set to the uniform distribution. As discussed in [Section 3, equation 4], the KL penalty in MSN explicitly encourages learning representations that enable uniform clustering of the data. However, when pretraining with class-imbalanced data, the semantic concepts of interest no longer satisfy the assumptions of the uniformity prior. In particular, natural observations "in the wild" tend to follow long-tailed (often power-law) distributions (Newman, 2005; Mahajan et al., 2018; Van Horn et al., 2018). Based on this observation, we introduce Prior Matching for Siamese Networks (PMSN), which extends MSN to support the use of arbitrary feature priors. Specifically, we modify MSN by replacing the negative entropy term in equation 6 with the KL-divergence user-specified distribution." For instance, we can instantiate PMSN as
H(P PP) + ADRI(P Pel(r)),
1=1
where PPL(r) is power-law distribution with exponent 7 > 0.
5.1 TOY SETTING
We claim that the uniform prior in coint-embedding methods significantly impacts the features captured in their learned representations. To illustrate this point to the research community using well-known data, we construct a toy setting to study how changes in the prior affect the suppression and selection of features by the encoder. We take grayscaled CIFARIO (Krizhevsky et al., 2009) images and overlay one of ten MNIST digits (LeCun & Cortes, 2010) in the top left corner, such that the overall distribution of MNIST digits in the dataset follows a power-law distribution with exponent 0.5.² Next we perform self-supervised pretraining on this dataset using PMSN. We compare pretraining using a uniform feature prior, to pretraining using a power-law prior with exponent equal to 0.5, which corresponds to the true power-law distribution of MNIST digits in the toy dataset. The first column in Figure 3 shows reference images from this toy dataset; the images column visualize the corresponding nearest neighbours in the embedding space of the pretrained models. When pretraining using a power-law prior, the MNIST digit is encoded by the model, and the nearest neighbours all have the same digit class (Figure 3b). However, when pretraining with a uniform prior, MNIST digit information is discarded by the model, and therefore the nearest neighbours have different digit classes (Figure 3c). In particular, since the MNIST digit "feature" does not admit uniform clustering of the data, it is discarded when pretraining with the conventional volume maximization penalty. This toy experiment provides insight into how semantic features in class-imbalancee datasets can be suppressed by the encoder and how alternative (non-uniform) priors can be used recover such features. We further show that PMSN with mixture of priors can capture features with different distributions on this task in Appendix I.
subsequent
5.2 NATURAL
SETTING
In this section, we examine the downstream task performance obtained by pretraining PMSN in more realistic settings. We examine pretraining with both the INIK. which is relatively class-balanced, and with the iNaturalist18 dataset (Van Horn et al., 2018), which is relatively llas-e.mbllanced. The iNat18 8 dataset contains approximately 430K images from over 8142 different species of plants and animals; since some species are more abundant and easier to photograph than others, the class distribution of images in this dataset naturally follows long-tailed distribution. After pretraining with either 1N 8 or INIK, we examine downstream task performance on the same set of tasks used in Section 4. Additional details about the experimental setup can be found in Appendix E.
"The negative entropy in cquation 6 is simply the KL-divergence to the uniform distribution plus a constant.
Note the image distribution is still uniform over the CIFAR10 classes.
7

---

Published as a conference paper at ICLR 2023
2
Ref.
(h) Neighbours when pretraining power-law prior (Matching MNIST digit distribution)
(c) Neighbours when pretzaining uniform peior (Nol matching MNIST digit distribution)
Figure 3: Each visualizes the nearest neighbours of the references images (first column), in the embedding space of PMSN model pretrained on grayscaled images with an MNIST digit the top left corner. The distrihution MNIST digits in the dataset is constructed follow a long-tailed power-law distribution. b) When pretraining using a power-law prior, the MNIST digit is encoded by the model, and the nearest neighhours all have the same digit class. c) When pretraining with uniform prior, the MNIST digit information is discarded by the model, and therefore the nearest neighhours have different digit classes.
Results with true class distribution. Table 2 compares iNat18 pretraining with PMSN using uniform distribution or the true class distribution of iNat18. In particular, if have images in the dataset and Da images in the hthh class category. then we define the class-prior, Pelass such that [Pelns) > Ds/D. Of course this information not usually assumed to known self-supervised pretraining: we consider it here to illustrate the effect of the uniform prior representation learning. As shown in Table 2, simply replacing the uniform prior with the long-tailes class- class-prior prior improves performance all downstream tasks: object classification with CIFAR 100 (+0.9%), iNat18 (+3.0%), Places205 (+1 (+1.4%), object counting with Clevr/Count (+2.1%) 1)), and depth prediction with Clevr/Dist (+2.3%) and KITTI (+2.7%). These results clearly support the intuition that moving away from the uniformity prior when pretraining with class-imbalanced data can improve the quality of the representations that learned. However, note that we have used the class distribution in these experiments. While this distribution may be estimated using weak supervision sources often available for internet-scale data (e.g.. image captions or hashtags), we would prefer general methods that do not require precise a priori knowledge of the class distribution.
Table 2: True Class Prior: Comparing the uniform prior with the class prior computed from the true class distribution leads to significant gains on downstream tasks.
Prior
iNat18
CIFAR 100
Place205
Clevr/Count
Clevr/Dist
KITTI
Pretrained iNaturalist88 (VIT-S/16)
uniform class-prior A
29.1
59.4
36.9
69.4
56.8
68.2
32.1
60.3
38.3
71.5
59.1
70.9
+3.0
+0.9
+1.4
+2.1
+2.3
+2.7
Results with general power-law distributions. Table 3 evaluates models pretrained using PMSN on the ImageNet-IK IK and iNat18 datasets when we do not use precise knowledge of the class distribution. In particular. we explore a power-law prior, PPI.(r). such that [PPL.-)] xc 1/2" with power-law exponent 0.25. As expected, use of power-law prior improves downstream task performance when pretraining on the class-imbalanced iNat18 dataset (top half of Table 3), but degrades performance when pretraining on the class-bale baasnbaaanced ImageNet- 1K dataset (bottom half of Table 3). These results indicate that it is preferable to match the prior distribution in self-supervieed algorithms to the distribution of semantic concepts in the pretraining dataset. In the case of ImageNet pretraining, the uniform prior more closely matches the distribution of classes in the dataset, and thus expect it to achieve strong downstream task performance in that setting.
While these results support the observation that one can still improve SSL pretraining on class- imbalanced datasets without having priori knowledge of the class distribution, comparing to Table
8

---

Published as a conference paper at ICLR 2023
Table 3: Power-Law Prior: PMSN with power-law prior achieves better downstream performance than uniform prior. when the pretraining dataset long-tale sailel classes distribution. Power-law prior hurts performance for class-balance pretraining dataset. Feature Prior should therefore matches the class-distrbuttin.
Prior
iNat18
CIFAR100
Place205
Clevr/Count
Clevr/Dist
KITTI
Preirained on iNaturalist18 (VII-8/16)
uniform power-law A
29.1
59.4
36.9
69.4
56.8
68.2
30.1
60.1
37.7
71.1
58.9
68.2
+1.0
+0.7
+0.8
+1.7
+2.1
+0.0
Pretrained ImageNNtt (VTT-S/16)
uniform
40.5
77.6
52.1
78.4
63.2
71.1
power-law
33.3
74.8
50.0
77.4
65.3
69.1
-7.2
-2.8
-2.1
-1.0
2.1
-2.0
shows that would be preferable to use the true class distribution when such information is available.
In short, we consider the results this section demonstration of the effect of non-uniform feature priors and do not claim to have completely solved the issue of class-imbalanced pretraining. Similar observations hold for other SSL approach such as SwAV (Caron et al., 2020) as shown in App. J.
Visualizing learned prototype vectors. In Figure 4, we RCDM to visualize the prototypes of PMSN model pretrained with either power-law or uniform priors on the iNat18 dataset. The features that emerge when pretraining with power-law prior are such as specific image classes. For example, one can recognize specific types of birds and plants in Figure 4a, which not the case with the samples generated using the uniform prior prototypes in Figure 4b. These qualitative results further highlight the effect of the overlooked uniform prior in supersupereieed learning with class- Glas--mballnnee data.
associated with high level concepts
Pretmining on power prior (Prior matching iNat18 class distribution)
(b) Pretraining (Prior not aligned with iNat18 class distribution)
uniform prior.
Figure 4: Visualization of prototypes learned by PMSN model pretrained with either uniform power-law priors the 18 dataset. We use RCDM (Bordes al., 2022b) enable visualization the prototypes. Each row corresponds samples generated by conditioning on a prototype with various random seeds. Features that remain constant across the row depicts information contained in the prototypes, whereas features that vary depict information that is not contained. a) Features that emerge when pretraining with a power-law prior on the iNat18 8 dataset tend to be associated with high level concepts such specific image classes. b) Features that emerge by pretraining with uniform prior are largely associated with low-level concepts such as texture.
CONCLUSION
In this work, we show that many common self-supervised learning frameworks have prior to capture features that enable uniform clustering of the data, and as such, require class-balanced datasets to learn dlassddiscrmminatiie features. By reformulating self-supervised criteria to prefer power-law distributed features, class-imbalanced datasets.
improve quality of the learned representations real- world

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we provide details on our pretraining and evaluation protocol Ap- pendices D and E. When pretraining using existing methods, we leverage publicly available imple- mentations along with the default hyperparameters; see Appendix D for details. For evaluation, we use the publicly available VISSL codebase (Goyal al., 2021); specific evaluation configurations are provided in Appendix D.2. The training details for the PMSN experiments are provided in Appendix 5. And finally, the proofs for all propositions Section 3 are produced in Appendix C.
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv: /902.09229 2019.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint arXiv:1911.05377 2019.
Mahmoud Assran, Nicolas Ballas, Lluis Castrejon, and Michael Rabbat. Supervision accelerates pre-training contrastive semi-supervised learning of visual representations. arXiv preprint DrXiv:2006.10800 10803, 2020.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-par non-parametrically predicting view assignments with support samples. arXiv preprint arXiv:2104. 13963, 2021.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning, arXiv preprint arXiv:2204.07144 2022.
Pranjal Awasthi, Afonso S Bandeira, Moses Charikar, Ravishankar Krishnaswamyy Soledad Villar, and Rachel Ward. Relax, no need to round: Integrality of clustering formulations. In Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science, pp. 191-200, 2015.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. arXiv preprint arXiv:2205. 1508, 2022.
Hangho Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-iimarrince-covvaaaaa regularzzaionn for self-supervised learning. arXiv preprint arXiv:2105.049900 2021.
Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regularization: Improving deep networks generalization by removing their head. arXiv preprint arXiv:2206. 13378, 2022a.
Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self-supervised representation knows about. Transactions on Machine Learning Research, 2022b URL httptst///peeeeee/////////
net forum? www.//eett//ddeeeeeeeeee borrumidd-rfWbb 7V jV
Paul S Bradley, Kristin P Bennett, and Ayhan Demiriz. Constrained k-means clustering, Microsoft Research, Redmond, 20, 2000.
John Bridle, Anthony Heading, and David MacKay. Unsupervised classifiers, mutual information
and' phantom targets. Advances in neural information processing systems, 4, 1991.
10

---

Published as a conference paper at ICLR 2023
Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature verification using a "siamese" time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7(04))669-668 688, 1993.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint AXXi:22006009882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691- 1703. PMLR, 2020a.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. preprint 2001:2200,.05709, 2020b.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances Neural Information Processing Systems, 34:1 1834-118444 2021.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011. 2..20011.1056 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020c.
Yubei Chen, Adrien Bardes, Zengyi Li, and Yann LeCun. Intra-instance vicreg: Bag of self-supervised image patch embedding. arXiv preprint arXiv:2206.08954 2022.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural
information processing systems, 26, 2013.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2000. /9999 2020.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112.10740, 2021.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning. arXiv preprint arXiv:2206.02574 2022.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. The International Journal of Robotics Research, 32(11):12311137, 2013.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations by predicting bags of visual words. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6928-6953 2020.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. Vissl. https:://// / /ttp:/giitt thub. com/ http://eeboookeeerreeeeeeeeeeeee 2021.
Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Mannat Singh, Ishan Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski. Vision models are more robust and fair when pretrained uncurated images without supervision. arXiv preprint 20Xiv:2202.08830) 2022.
11

---

Published as a conference paper at ICLR 2023
Jean-Bastir Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to sulf-supe learning. arXiv preprint arXiv:2006.07733, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05727 2019.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Mariii-nMr,,i,,nn, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: /800. 066706 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httptt http:tpppppppppoceeneneoo www.eeedddiinnn..nnn...... wwwprrewweeeeeeeeeeeeeeeeeeeeeee
wctppcpppppppcpcccccc///pppp /hash/ html.
Matthew Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1, 2016.
Kurt Hornik, Ingo Feinerer, Martin Kober, and Christian Buchta. Spherical k-means clustering. Journal of statistical software, 50:1-22, 2012.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In International conference on machine learning. pp. 1558-1567. PMLR, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift, 2015. URL wwwps:ttt/s///teeeeeeee www.vvorr//bbba. /ttpr///b///////eee// 102..0.000 031 67.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and
Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Matthäus Kleindessner, Pranjal Awasthi, and Jamie Morgenstern. Fair k-center clustering for data summarization. In ICML, pp. 3448-3457. PMLR, 2019.
Andreas Krause, Pictro Perona, and Ryan Gomes. Discriminative clustering by regularized informa- tion maximization. Advances neural information processing systems, 23, 2010.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06--11 2022.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: /yann.
lecun. com/ exdb i st/.
Jiye Liang. Liang Bai, Chuangyin Dang, and Fuyuan Cao. The k-means-type algorithms versus imbalanced data distributions. IEEE Transactions on Fuzzy Systems, 20(4):728-745, 2012.
Ralph Linsker. Self-organization in perceptual network. Computer, 2153):105-117, 1988.
12

---

Published as a conference paper at ICLR 2023
Yi Ma, Doris Tsao, and Heung- Yeung Shum. On the principles of parsimony and self-consistency for the emergence of intelligence. Frontiers of Information Technology & Electronic Engineering, pp. 1-26, 2022.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the Europeun conference on computer vision (ECCV), pp. 181-196, 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed represen- tations of words and phrases and their compositionality, 2013. URL httphttss///rrii..oooo
abs/131 0.000.4 45
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant represtttooossss.sss In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6707- 6717, 2020.
Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell. Representa- tion learning via invariant causal mechanisms. arXiv preprint arXiv:2010.07922, 2020.
Mark EJ Newman. Power laws, pareto distributions and zipf's law. Contemporary physics, 46(5): 323-351, 2005.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1VXiv:/807.053 2018.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URL pttttps://axxi..ooo 06125.
/2204.
Napat Rujcerapaiboon, Kilian Schindler, Daniel Kuhn, and Wolfram Wiesemann. Size matters: Cardinality-constrin clustering and outlier detection via conic optimization. SIAM J. Optimiza- fion, 29(2):121 1-1299, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-222, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffuusion models with deep language understanding, 2022. URL https: /tttpii://axxi..../ 2205. 11487.
/bs/
Yonglong Tian, Olivier Henaff, and Aäron van den Oord. Divide and contrast: Self-supervised learning from uncurated data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10063-10074, 2021a.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pp. 10268-10278. PMLR, 2021b.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv: 13625, 2019.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8769-8777 2018.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 23(7):1661-1674 201 1.
13

---

Published as a conference paper at ICLR 2023
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pitrre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of machine learning research. 11(12), 2010.
Fei Wang, Ping Li, and Arnd Christian Konig. Learning a bi-stochastic data similarity matrix. In 2010 IEEE International Conference on Data Mining, pp. 551-560. IEEE, 2010.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929-9939. PMLR, 2020.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint arXiv:21 2021.
Junjie Wu, Hui Xiong, and Jian Chen. Adapting the right measures for k-means clustering, In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 877-886, 2009.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1733-37442, 2018.
Qizhe Xie, Zihang Dai, Eduard Hovy. Minh-Tham Luong, and Quoc V Le. Unsupervised data augmentation. arXiv preprint arXiv: 12848, 2019.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 100.0088888888 2017.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03250, 2021.
Hongyuan Zha, Xiaofeng He, Chris Ding, Ming Gu, and Horst D Simon. Spectral relaxation for k-means clustering. In NeurlPS, pp. 1057-1064, 2001.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. Advances in neural information processing systems, 27, 2014.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.078. 2021.
Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference Machine Learning pp. 27367-27317 PMLR, 2022.
14

---

Published as a conference paper at ICLR 2023
A BROADER RELATED WORK
Joint-embedding architectures are an active line of research in self-supervised representation learn- ing (Wu et al., 2018; He al., 2019; Chen & He, 2020; Grill et al., 2020; Chen et al., 2020c; Caron et al., 2021; Bardes et al., 2021; Zhou et al., 2021). These approaches rely on invariance based pretraining where a neural network encoder is trained to output similar embeddings for two or more views of the same image. To avoid pathological solution, joint-embedding approaches use explicit regularization (Chen et al., 2020b; Caron et al., 2021; Bardes al., 2021; Assran et al., 2021) or architectural constraints (Grill et al., 2020; Chen & He, 2020).
Explicit regularization usually maximizes the volume of space occupied by the representations. Regu- larization can be implemented using various strategies such contrasting negative samples (Bromley et al., 1993; He et al., 2019; Chen et al., 2020b), arriance-covarinne regularization (Bardes et al., 2021; Zbontar et al., 2021), or by maximizing the entropy of the representations (Asano et al., 2019; Caron et al., 2020; 2021; Assran et al., 2021; 2022). To boost performance of contrastive methods when pre-training with long-tailed data, Zhou et al. (2022) propose increase the strengths of data-augmentattinss applied to tail samples. Alternative
callapse-prevention approaches based on architectural constraints leverage architectural design avoid collapse such as stopping the gradient flow in one of the Siamese Network branches (Chen et al., 2020b), using momentum encoder to compute the network targets (Grill al., 2020), or using an asymmetric prediction head (Grill et al., 2020; Chen et al., 2020b; Baevski et al., 2022). Recent theoretical work (Tian et al., 2021b) explores why certain joint-embedding methods with architectural constraint avoid representation collapse without explicit use of volume maximization penalty: the implicit collapse prevention mechanisms here are not mutually exclusive. Recent empirical work (Bordes et al., 2022a) studies the invariance properties of these pretrained representations. The work of Mitrovic et al. (2020) has drawn connections between invariance and causality when the data autteeaugmetttionn used during pretraining manipulate specific factors of variation in the data- gettraeenratiig distribution. Other works have studied the computational efficiency of joint-embedding methods, and demonstrated how small amounts of supervision can be used accelerate convergence (Assran et al., 2020).
While joint-embedding architectures are usually leveraged to learn a global image representations, some works explore these use of these architecture for learning local and dense representations (Chen et al., 2022; Gidaris et al., 2020). More recently, LeCun (2022) proposes an architecture based on joint-embedding approaches to learn generic world model, capturing both dense local features as well as global image features.
Orthogonal the contributions of invariance- based pretraining, another line of work attempts to learn representations by artificially masking parts of the input and training a network to reconstruct the hidden content (Vincent al., 2010). Auto-regressive models, and denoising auto-encoders, in particular, predict clean visual inputs from noisy views (Chen et al., 2020a; Vincent et al., 2010; He et al., 2021; Bao et al., 2021; Baevski et al., 2022). A mask-noise is usually used to perturb the images and those approaches predict the masked inputs either at the pixel level (Dosovitskiy et al., 2020; He et al., 2021; Xie ct al., 2019) or at a token-level, using a pixel (often patch-level) tokenizer (Bao et al., 2021; Wei et al., 2021). While these works demonstrate impressive scalability, they usually learn features at a low-level of semantic abstraction compared to joint-embedding approaches (Assran et al., 2022).
More recently, a set of approaches attempt to combine both joint-embedding and reconstruction based approaches Zhou al. (2021); El-Nouby al. (2021), wherein they combine invariance pretraining loss with patch-level reconstruction loss.
B RELATION TO THE INFOMAX PRINCIPLE
A longstanding conviction in unsupervised representation learning is that the resulting representations should be both maximally informative about the inputs, while also satisfying certain simplicity constraints (Linsker, 1988; Goodfellow et al., 2016). The former objective is often referred to as the
principle (InfoMax), while the latter is sometimes referred to as the parsimony principle (Ma et al., 2022), which is crucial to the problem formulation. Indeed, Bridle et al. (1991). i). one of the first works to empirically explore unsupervised representation learning via information- enformation-maximizaae maximization, found that, in the absence of additional constraints, the resulting
15

---

Published as a conference paper at ICLR 2023
InfoMax representations may not be very useful. The more recent analysis of Tschannen al. (2019) also argues that simplicity constraints are essential to the success of modern representation learning methods built on the InfoMax principle. Historically, simplicity constraints were enforced by encouraging the learned representations to be sparse, low-dimensional, or disentangled, i.e., the individual dimensions of the representation vector should be statistically independent.
Modern state-of-the-an approaches for unsupervised representation learning still frequently employ an iooommationmmxiimizaa
formulation (Hjelm et al., 2018; Bachman al., 2019; Krause et al., 2010; Hu et al., 2017; Oord et al., 2018), but with the simplicity constraints manifested in self- supervised loss terms. One example is the widespread riew-invariance penalty (Misra & van der Maaten, 2020), often coupled with with independence (Zbontar et al., 2021; Bardes et al., 2021) or low-dimensionality constraints, e.g., by projecting representations on the unit hypersphere (Chen et al., 2020b; He et al., 2019; Grill et al., 2020).
To better understand these methods, recall that the mutual information, I(.,), between latent vector Z and data X can be written
1(2 = [| H[
where H[Z] is the marginal entropy of Z, and X is the expected entropy of the posterior distribution. In learning representation of the data, Z, that maximizes the mutual information [Z, X). we thereby seek maximize the marginal entropy of Z, i.e., search for uniformly distributed feature embeddings.
However, as we show this work, perhaps the features that we wish our representations to capture are not necessarily those with the highest marginal entropy. It is often the case that the semantic concepts we wish to capture actually follow a long-tailed distribution the wild Such desirable features would be penalized under existing iitormation-maxiniii absence finer grained notions of information, perhaps it necessary to reconsider the longstanding conviction of seeking representations that maximize information content.
frameworks. Thus, in the
C THEORETICAL GUARANTEES
C.I PROOF OF PROPOSITION
We will demonstrate that
K
P Prik 1 P..k
min
min
En Mell2
min PE(O.1) PIk
Σ Rm 11,
Σ
I
2n
Pl....kkk
1N
(10)
where the lent-hand side minimizes over the values of the K centroids and the inner minimization identifies the cluster membership of sample 2n,, To do so, one should first notice that almost surely, fact, for any
for any input Ty, the min.=......tt | v| Hell2 attained for a single centroid.
continuous distribution on the data (and/or centroids) the probability to sample them so that a sample
lies exactly equidistant from two (or more) centroids 0 since this a space of dimension D which has measure O. Hence, we first obtain
N
N
K Ps.
min Mi n=1
min
[n Mel
min Mloeokk
min
| - 14 12
pe
(0.1)
p
1=
and because each sub- buoblroblem is independent, we can pull them out of the sum to obtain
N
K
min
min Dr pel12
min min Pa N0.....NKKPI.
Pa
14 12.-
n=1
n=lk=
now we can switch the minimization order to obtain
K
K Pa. Pr. mill.
min PE(0.1)NXA
min
Σ
P.. k 1
min PE(O.1)NXX,
Σ
Pig Pi
PE 1=1
n= 1=1
16

---

Published as a conference paper at ICLR 2023
with p the mean of the samples within cluster k, i.. (p>1x)s Σ show is that the sum of distances to the cluster means is equivalent to the pairwise distances between all the points within each cluster with an appropriate normalization, which is standard result; see, e.g., Zha et al. (2001).
P.,,A.,. The only thing left to
C.2 PROOF OF PROPOSITION
Notation. Before establishing the theoretical connections between common joint-embedding meth- ods for self-supervised learning and the K-means method, we first define some notation to facilitate the discussion. Given "stand-alone" dataset, X' E R xaиWW Ra''w we construct the pretraining data X E RNX3HW by repeatedly perturbing the elements of the stand-alone dataset. Specifically,
X
- View ( (x')²,... Viewv(X'))
where (-)w; is sample- wise image transformation, e.g., random crop, color jitter, patch masking. We also define the ground-trum similarity matrix, G E {0, 1) Nx3HW, given by
Gi = (2) ~ mz), i ド 0,
i=j
where the operator returns if its two arguments are positively related (i.e., correspond to different views of the same sample). We also define the matrix of embeddings obtained from a model fo()) as
Z -
RRNDD with 2n - fo(x)).
(11)
Now are ready to present our first equivalence relations for the VICReg and SimCLR methods. The VICReg loss (Bardes et al., 2021), which is a function of X and G, can be defined as
K
K
K
N N
L=a max (. (01-//ool (z),
+
Cov(Z)
2 +
(G), ==
-2,1
k.t
N
1m1
but this desired result relies on the simplification employed in equation To prove our statement, we first remind the reader of a common result that we will heavily rely on: the decomposition of the covariance matrix into within- and netween-clustr covariance matrices. Let us assume for simplicity that Z is already centered. We can now decompose the covariance into
1 Cov(Z) (2) - t HZ = N (z"
where two terms are now the between cluster and within cluster covariances, with H the centering matrix defined by I 1 //N. /N. Let's first consider the LHS of the VICReg loss simplify it into
1 ZTZ - N
1 ZT GZ +
z G)Z -
N
N
F
2 +
1
GZ -
2- - G)
N
N
+ 2 Tr
ZT [I TT G) - zz 1)
N
2 - 1 n" 1z GZ -1
1 アタ (1 G)
Tr
+ No
Tr (1-G)ZZTGG
2" 22 GZ
11
+ +Σ>, v
4J (I- -()
| z- - -2]
N
F
2 No
Tr ( (I Zz
17

---

Published as a conference paper at ICLR 2023
where {, returns the 1th eigenvalue of its matrix argument M; the last equality is obtained by noticing that |M I E; A; and that for a symmetric positive semidefinite matrix,
Tr(M) = Ex Aa [M); we thus obtain the following upper and lower bounds
1 1 + +E d (1-
1 Td z - N
Z
GZ
-
N
N
(N Gz ]
z² (I -G)Z -2+ ai (N 2
\
N
1 N
2
1 N
Z GZ -1 -1(² +D M -1|²+//+
T" Z
2
zz (I -G)2
N
x | ( z" -G)Z
-2+ AK+I-i (1- 1 2
11 12 zT -I + +Σ d 1 G)- G)Z - -K,
1 N
N N
from which is becomes clear that to minimize the varianco+covariance terms, one must either maximize the intra-cluster variance, or the inter-cluster variance, both. However, the intra-cluster variance is exactly the invariance term since it can be expressed as
N
Σ j=1
-,,1 (
and thus the only possible solution to minimize the invariance term while minimizing the vari- ance+covariance is to minimize || 2 GZ - I - and thus we recover that VICReg's loss corre- sponds to the K-means loss plus regularizer |1 1 p as in
Σ |
mxl12 + a M Il
ΦEX
with M A [. Mr]. and the number of centroids K is given by min(dim(z)) rank(G + I))
and the centroids are given by 14 = (P (P.A.1 E PAAAE with G = DD,,, and finally X
(an, E X PA, 0}
C.3 PROOF OF PROPOSITION
The proof is relatively straightforward and will follow the same principle the proof showing how GMM recovers K K-means. The only difference is that we will take the zero-noise limit the MSN loss (equation 6) instead of GMM loss (equation 5), and will that recover constrained version of K-means with extra constraints on the cluster distribution. The MSN loss is defined for positive pairs n, a and estimates the corresponding cluster posteriors Pn: P via softmax (M x,,/o) and softmax (M /a+) respectively with M A [uI.... _K] we employed in VICReg, and the two temperature parameters are commonly o > o > O; this asymmetry is know sharpening. Without loss of generality we consider the oSSN loss, i.e., the MSN loss re-scaled by o, this does not alter the training dynamics as the learning rate can be adapted accordingly, but will simplify our derivations below; we thus also replace with A/a. this setting, we see that softmax M / ) = 5(k kk()) where we hereafter denote k(n) arg min, I1/e 2 as the cluster assignment of the positive view of En. With those notations, we can finally derive
(MTx../')
lim a rSNN = lim a+40
H(p Pn)) + ADr Pprior)
m=
11
K
K
NR N
No/N (pprior
lim 0->
8(k- k(Tz)) log (softmax W (TTnn/))k)
A
log
)R
1=
k=
18

---

Published as a conference paper at ICLR 2023
where we could push the limit inside DKL since assumee that cluster is empty and thus the KI. function is continuous as -> 0, and that lim,-+0 p (% ..... N The value of N₂. is the number of samples that are assigned to cluster k. This assumption, which was only used for simplicity, can be removed easily by noticing that lim log(u) O. Now considering the left term of the loss we obtain direct simplification of the log-softmax as follows
N
K
lim 0
ō(k k(n)) log (softmax Settmmxx("TTaa (WT WWww///ll (en/))k)
N
k=1
1
nn (k(x)) 112
min || pa 11
N
arg min [| Hella marg min || -
which is minimized whenever all the samples 2n that have their positive views ox associated to the same centroid belong to the same cluster; thus the overall MSN loss can be written as
Nu log
NN/N (Ppetor)
)
- 1x11
k= xEX
1=1
where Xx {x X arg min
m= | - Mell2 2 k}.
C.4 PROOF OF PROPOSITION 4
The proof for SwAV mainly relies on the same development as done for MSN. That is, we saw that with enough sharpening. the cross-entropy between P and pn. falls back to the K-means like term with an extra margin that is minimized only when the two views are in the same cluster. The difference arises in that SwAV explicitly adds linear constraints on the cluster-membership matrix whereas MSN was employing KL-divergence between the average posterior and the prior. The connection between the linear constraint and the SwAV Sinkhorn- Knopp procedure has been made precisely in Wang al. (2010), where it was shown that the latter solved relaxed optimization problem for which the cluster meunber-membessip is no longer constrained to be 0 or 1.
D PRETRAINING AND EVALUATION DETAILS FOR SECTION
D.1 PRETRAINING PROTOCOL
SimCLR. We the VISSL (Goyal et al., 2021) code base to pretrain a ResNet-50 with Sim- CLR (Chen et al., 2020b), with a batch size of 4096 for 300 epochs. Our pretraining follow the standard hyperparameters defined in Chen al. (2020b). The learning rate follows the default cosine schedule with a 10 epoch warmup. We use temperature of (.. for the contrastive loss and LARS (You et al., 2017) as an optimizer. We modify the sampler to force K different classes inside each mini-batch where K is set to 8 for the class-imbalancee sampling experiments and 960 for the class-balanced sampling experiments.
VICReg. We pretrain ResNet-50 with VICReg (Bardes et al., 2021) using the LARS optimizer with a batch size of 1024 for 300 epochs using the official code base, which is publicly available: https: htt///gtttuu hyperparameters defined in Bardes et al. (2021). The learning rate follows the default cosine schedule with 10 epoch warmup. We the dimensions of the expander MLP to the default 8192 - 8192 8192. Weight decay is set to 10 6, the variance and invariance coefficients are set to 25.0, and the covariance coefficient set to 1.0. We modify the sampler force different classes inside each mini-batch where K is set to 2 for the class-imbalancee sampling experiments and 960 for the class-balanced sampling experiments.
com/ www.bokkreeee wwwearcccviicreee. Our pretraining follow the standard
MSN. We pretrain a ViT-B/16 with MSN (Assran et al., 2022) using the AdamW optimizer with batch size of 1024 for 300 epochs and 1024 prototypes using the official code base, which is publicly available: https: // /i thub. com fonm/faccoookroo resea rch/ /an. Our pretraining follow the
19

---

Published as a conference paper at ICLR 2023
standard hyperparameters defined in Assran et al. (2022). The learning rate follows the default cosine schedule with a 15 epoch warmup. Weight decay is linearly increased from 0.04 to 0.4. Gradient clipping is set to 3.0, the entropy coefficient is set to 1.0, the temperature is set to 0.1, the sharpening exponent is set to 0.25, and the masking ratio is set to 0.5. The number of random-mask views in each iteration is modify the sampler to force K different classes inside each mini-batch where K is set to 2 for the class-imbalanced sampling experiments and 960 for the class-balanced sampling experiments.
1, and the number of focal-mask views in each iteration is set to 10. We
MAE. We pretrain a ViT-L/16 with MAE (He et al., 2021) using the AdamW optimizer with a batch size of 1024 for 800 epochs using the official code base. which is publicly available: https: g github. com/ facebook researe rch maae Our pretraining follow the standard hyperparameters defined in He et al. (2021). The learning rate follows the default cosine schedule with a 40 epoch warmup. Weight decay is set to 0.05, and the masking ratio is set to 0.75. We modify the sampler to force K different classes inside each mini-batch where is set to 2 for the glass-imbalanced sampling experiments and 960 for the class-balanced sampling experiments.
data2vec. We pretrain a ViT-B/16 with data2vee (He et al., 2021) using the AdamW optimizer with batch size of 2048 for 800 epochs using the official code base, which is publicly available: ht tt: / //github. com/ fa cebook researeeeeee aaeee/cc/// datddaaaaeeeeee taavec_ree.c.eee serre_vvssie visionn tree /maii//in///////////m//.a/ / /beet. t.. Our pre- training follow the standard hyperparameters defined in Baevski et al. (2022). Specifically, the learning rate follows the default cosine schedule with a 10 epoch warmup. Path drop is set to 0.05, and the target layers are set to [6. 7, 8, 9, 10, 11). We modify the sampler to force K different classes inside each mini-batch where K is set for the class-imi itus--imallanced sampling experiments and 960 for the Mlas-balancee sampling experiments.
0.25, gradient clipping is set to 3.0, weight decay
D.2 EVALUATION PROTOCOL
For linear evaluation, we use the default linear evaluation configurations of VISSL (Goyal et al., 2021) to evaluate our models on the following datasets: ImageNet (Russakovsky et al., 2015), iNaturalist18 (Van Horn et al., 2018), CIFARI00 (Krizhevsky et al., 2009), Clevr/Count (Johnson al., 2017), Clevr/Dist (Johnson al., 2017), KITTI/Dist (Geiger et al., 2013) and Places205 (Zhou et al., 2014).
For pretrained models based on Vision Transformers (Dosovitskiy et al., 2020), we report the best linear classifier number among the following representations:
the concatenation of the last 4 layers of the class token, (Caron et al., 2021)
the representation of the last layer of the class token.
For pretrained models based on ResNet50 architectures (He al., 2016), we follow the evaluation protocol of SEER (Goyal et al., 2022) and report the best linear classifier number among the following representations:
the final representation layer (of dimension 2048 for a ResNe150),
an adaptive average pooling of the last feature map, concatenated to get 8192 dimensions.
We also follow the default VISSL (Goyal et al., 2021) configuration and attach 2 linear heads per chosen representation, one composed of a single linear layer, and one with an added batch normalization (loffe & Szegedy, 2015) before the linear layer.
E PRETRAINING AND EVALUATION DETAILS FOR SECTION 5
E. PRETRAINING PROTOCOL
E. TOY SETTING
We pretrain a ViT-Tiny/4 with MSN (Assran et al., 2022) using the AdamW optimizer with batch size of 1024 for 300 epochs and 10 prototypes using the official code base, which is publicly
20

---

Published as a conference paper at ICLR 2023
The learning rate follows the default cosine schedule with a 15 epoch warmup. Weight decay is linearly increased from 0.04 to 0.4. Gradient clipping is set to 0.0, the entropy coefficient is set to 100.0, the temperature is set to 0.1, the sharpening exponent is set to 0.25, and the masking ratio is set to 0.05. The number of random-mask views in each iteration is set 1, and the number of focal-mask views in each iteration is set to 0. To accommodate the lower-resolution images, we modify the scale of random-resizee-crrpp data augmentation to (0.5, 1.0) and train without Gaussian-Blur.
available: https: httpps://gtttin ccwm//fccbooreeeeercccccccc
E.1.2 NATURAL CLASS -LASS-IMBALANCED SETTING
We pretrain a ViT-S/16 with MSN (Assran et al., 2022) using the Adam optimizer with a batch size of 4096 for 300 epochs and 8142 prototypes using the official code base, which publicly available: https: / //g/ttttt /github. com/ fa facebookresearch msn. The learning rate follows the default cosine schedule with 15 epoch warmup. Weight decay linearly increased from 0.04 to 0.4. Gradient clipping is set to 3.0, the entropy coefficient is set to 5.0, the temperature is set to 0.1, the sharpening exponent set to 0.25, and the masking ratio set to 0.15. The number of random-mask views in each iteration is set to 1, and the number of focal-mask views in each iteration is set to 10.
E.2 EVALUATION PROTOCOL
We use the exact same evaluation protocol as D.2.
F ALTERNATIVES TO FEATURE PRIORS FOR SECTION 5: DATA SAMPLING
As alternative changing the feature prior, perhaps one devise more intelligence sampling strategies to align the class distribution in the sampled mini-batches with the implicit uniform prior in self-supervised algorithms. Here, we investigate the impact on the mini-batch sampling schemes when pretraining on iNat18. 8. We compare (uniform) random sampling to unthresholded inverse square-root frequency sampling (Mikolov et al., 2013), which is commonly applied in the context of (weakly) supervised learning on internet-scale class-imbalen imbalanced data. Specifically, the prescription of inverse square-root frequency sampling to sample class with probability inversely proportional to the square root of the frequency of the class, and then sample images uniformly within the class. For example, we have D) images in the datasct and Dy images in the 1.th class category, then the probability of sampling class k is equal to vDz/D, opposed to the traditional Di/D probability under uniform sampling. The effect of this sampling strategy is to mitigate class-imbalances by oversampling underrepresented classes in the dataset. While such an effect is desirable, one limitation to this strategy that it induces an implicit reduction in the dataset size; for example, we may see the same images from the tail of the distribution very frequently and, given finite training time, may never see some of the images in the head. Another limitation of this strategy is that requires knowing the class label (or a weak class label) for every image in the training set.
Despite these limitations, we explore inverse frequency sampling in Table and observe that it does improve upon uniform random sampling. We hypothesize the inverse frequency sampling tends to aggressively over-sample classes with very few examples, reducing the effective number of images seen per epoch and thus degrading the quality of the learned representations. This result suggests that changing the feature prior is more viable solution for pretraining with class-imbalanced data.
Table 4: Impact of Sampling Scheme: Changing the mini-ban mini-batch distribution by using an inverse square root sampling strategy on iNat18 does not improve downstream performance of the learned representations.
Sampling iNat18
C0
Place205
Clevr/Count
Clevr/Dist
KITTI
Pretrained on INaturalist18 (VIT-S/16)
Uniform
29.1
59.4
36.9
69.4
56.8
68.2
Inverse Square Root Freq
23.4
58.4
35.1
68.3
59.8
66.6
21

---

Published as a conference paper at ICLR 2023
G MARGINAL SAMPLING PROBABILITIES
The insalaneaee strategy in Section maintains the same marginal probability of sampling individual data points as compared to the sampling a particular data point in each iteration is unchanged. To see this, suppose we have a mini-batch of size B
sampling strategy; i.c., the probability of
960 for some integer O. Under class-balanced sampling, we will thus sample 960 classes in each iteration, and then n images per class. Under class-imbalanced sampling, we will first sample 2 classes in each iteration, and then n 480 images per class so that the overall batch size is B. Now consider the probability of sampling a data point x that comes form a class C in our dataset containing N 2 480 n samples. Under bass-balanced sampling, the probability of sampling the data point z can be factored as
9999
n 960
Ptalanced (I) p(z|C)p(C)=
959
1000
N 1000
960
Under class-imbalancee sampling, the probability of sampling x can be factored as
(480 ,)
(999)
480 n 2
Pimbolanced (2) p<x[C)pp(C)
1000
N 1000*
180
from which it is clear that Pbalanced (I) is equal to Pimbalanced (z)).
H VISUALIZING PROTOTYPES WITH RCDM
We use the RCDM framework (Bordes et al., 2022b) to visualize the representations and prototypes learned with MSN. RCDM trains conditional generative diffusion model, which maps a noise vector to pixel space using a neural network representation as conditioning. Many works (Ramesh et al., 2022; Saharia et al., 2022) have demonstrated the potential of conditional diffusion model for image generation, however they are also very useful, as highlighted by (Bordes et al., 2022b) to get a better understanding of what is learned by neural networks.
During training, RCDM takes as input a noisy image it (corrupted with an e, noise vector such as d = (t) and the representation vector y computed by MSN of the image æ. Then, RCDM is train, with a denoising score matching loss (Vincent, 201 1; Ho et al., 2020), to reconstruct the image that used to compute the representation vector y. More formally, we define RCDM neural network gr (it, y) that learns to predict the noise component e, of ét. ie. by minimizing (,,, y) - ee. As demonstrated by (Bordes et al., 2022b), RCDM extract as many information as possible from the representation vector y in order to reconstruct faithfully the image.
The conditioning vector y is computed from æ using a pretrained and frozen MSN model. MSN generates as output the probability distribution p that an image belongs to given cluster, p softmax softmax(W"'p, (fop,(f(xx)) where W is the matrix concatenating all the prototypes (or cluster centroids), Py the projection head and So the encoder. To visualize the prototype contained in W, we first train a conditional generative diffusion model that take the last linear layer input as conditioning, ie y = P, (=fo())). After training, we replace the image embedding with a learned prototype and generate the corresponding output in pixel space by setting y = W,, where i is randomly selected.
To summarize, we gather for every images in the training set their embedding (with dimension size of 256)* of a trained MSN model. Then, we use these embedding as conditioning for RCDM which train reconstruct the corresponding image associated to given embedding. When training is complete, we replace the projector's ' embedding by the prototype learned with MSN (which also have a dimension of 256). By doing so we can visualize which information is associated to each prototype (or cluster) learned with MSN. For every RCDM training, we used the same defaults settings as the ones on https:/ /ttpttttbb//////////// com/ hatp///bbboooeeerrchh.o 200000 iterations.
We train each network for
'the one that is use perform the clustering with respect the prototypes
22

---

Published as a conference paper at ICLR 2023
3_
2.
3
3
3
3
3
3
3
3
3
3
3
3
3
Figure 5: Each row visualizes the nearest neighbours of the references images (first column), in the embedding space of PMSN model pretrained grayscaled images with MNIST digit in the top left corner. The distribution MNIST digits in the dataset constructed follow long-tailed power-law distribution. Here we consider PMSN model using two projection heads, one with a power-law prior and one with an uniform prior. Nearest neighbours show both consistent digit and object classes. When pretraining PMSN with a mixture of priors, the learned representation is able to capture both power-law features (related to the digits) and uniform features (related to the object classes).
T MIXTURE OF PRIORS
Section 5.1 illustrated the effect feature priors in a toy setting. Specifically, we take grayscaled CIFARI0 10 images and overlay of ten MNIST digits in the top left corner, such that the overall distribution of MNIST digits in the dataset follows power-law distribution with exponent 0.5. When pretraining using MSN with power-law prior, Figure 3b shows that the digit information is encoded by the model, while the CIFARIO 10 class information, which is uniformly distributed, is discarded. By contrast, Figure 3c shows that when pretraining using MSN with uniform prior, the digit information is discarded by the model, whereas the CIFAR 10 class information is relatively well captured. An important question ask is whether it possible to pretrain with mixture of priors, so as to capture multiple sets of features following different distributions in the pretrianing data.
We follow the experimental setup described in Section 5.1, but pretrain using two different heads. The first head enforces a uniform prior, while the second head enforces power-law prior. The total loss is simply the sum of the losses from the two heads. The first column in Figure 5 shows reference images from this toy dataset; images in subsequent columns visualize the corresponding nearest neighbours in the embedding space of the pretrained models. Remarkably, using this simple multi-heat head mixture prior, the model learns capture both digit information (power-law distributed) and the object class information (uniformly distributed). Indeed, the nearest neighbours tend to exhibit both consistent digit and object classes.
23

---

Published as a conference paper at ICLR 2023
J PRIOR MATCHING WITH SWAV
The self-supervised SwAV (Caron et al., 2020) method has an implicit uniform prior through the use of the Sinkhorn- Knop normalization. With minor tweak, we can re-parameterize their default Sinkhorn-Knopp normalization to rebalance the mini-batch prior to a power-law instead of a uniform distribution.
We train SwAV for 100 epochs, once with the default implicit uniform prior, and once with power-law prior with exponent 0.5. We run these two pretraining runs on both the ImageNet dataset, where the class distribution is relatively balanced, and on the iNaturalist datasets, where the class distribution is long-tailed."
Table 6 summarizes these SWAV prior pretraining results on various downstream tasks, and exhibits similar findings as with PMSN: aligning the prior better match the underlying class distribution increases the quality of the learned representations. SwAV pretrained on ImageNet benefits from a uniform prior, while SwAV pretrained on i8at18 benefits from a power-law prior, boosting in- distribution performance by 1.5% as well as boosting performance semantic tasks.
Table 5: Impact of prior SwAV: Changing the prior of SwAV by changing the Sinkhorn Knopp target impacts the representations learned by SwAV. Aligning the prior with the underlying class distribution of the pretraining dataset increases the quality the learned representations.
Prior iNat18
CIFAR100
Place205
Clevr/Count
Clevr/Dist
KITTI
Prefruined INaturalis188 (ResNet50)
uniform
33.8
50.2
41.9
70.7
70.6
70.3
power-law
35.3
51.2
42.1
72.2
68.6
70.5
+1.5
+1.0
+0.2
+1.5
-2.0
+0.2
Pretrained on ImageNet (ResNet50)
uniform power-law
44.4
75.6
54.5
78.4
68.6
72.5
42.6
74.2
54.6
76.6
67.9
72.3
A
-1.8
-1.4
+0.1
-1.8
-0.7
-0.2
K ADDITIONAL RESULTS WITH PMSN
In this section report additional results with PMSN when pretraining larger ViT-B/16 encoder with either pawer-law or a uniform prior on the blassbblaance ImageNet dataset. Results are consistent with Table 3, indicating that the power-law prior result in less semantic representations when pretraining with balanced data.
Table VIT-B/16 with Power-Law prior. Power-law prior can result in less semantic representations when pretraining with class-balanced data.
Prior iNat18
CIFAR100
Place205
Clevr/Count
Clevr/Dist
KITTI
Pretrained on ImageNet (VIT-B/16)
uniform
41.9
81.7
54.0
74.3
57.8
73.3
power-law
22.0
63.8
38.3
71.4
65.8
66.6
A
-19,9
-17.8
-15.7
-2.9
+8.0
-6.7
L FULL SET OF RESULTS FOR SECTION 4
In this section, report the full experimental results for Section 4. Figure 6 shows summary of the main results.
For 88a118 pretraining, we increase the number of prototypes to 8142, corresponding the number of i8 classes.
24

---

Published as a conference paper at ICLR 2023
Table 7: SimCLR: Evaluation of the mini-batch sampling distribution on various downstream task. Sampling imbalanced mini-batches lead to significant drop in image classification tasks.
CIFAR100
CIFAR100 1%
Place205
Clevr/Count
Clevr/Dist
KITTI
960 cls per batch
69.9
31,4
52.1
77.4
65.5
70.5
8 cls per batch
63.4
21.25
47.1
73.8
66.5
71.7
-6.4
-10.2
-5.2
-4.3
+0.9
+1.2
Table 8: MSN: Evaluation of the mini-hatch sampling distribution various downstream task. Sampling imbalanced mini-batches lead to significant drop in image classification tasks.
CIFAR100
CIFAR 100 1%
Place205
Clevr/Count
Clevr/Dist
KITTI
960 cls per batch
84.3
46.2
81.0
56.7
63.7
73.2
2 cls per batch
71.4
26.4
76.4
50.3
65.6
71.5
-12.9
-13.2
-4.6
-6.4
+1.9
-1.6
Table VICReg: Evaluation the mini-batch sampling distribution on various downstream task. Sampling imbalanced mini-batches lead to significant drop in image classification tasks.
CIFARI00
CIFARI00 1%
Place205
Clevr/Count
Clevr/Dist
KITTI
960 els per batch 2 cls per batch
69.7
28.9
51.0
79.8
69.0
73.3
60.8 -12.0
16.9
44.9
76.8
69,84
72.1
A
-8.9
-6.0
-3.0
+0.7
-1.1
Table 10: data2vec: Evaluation of the mini-batch sampling distribution on various downstream task. Sampling imbalanced mini-batches lead to similar performances across tasks.
CIFARI00
CIFAR100 1%
Place205
Clevr/Count
Clevr/Dist
KITTI
960 cls per batch cls per batch
50.3
13.7
37.0
76.8
49.7
65.3
48.6
13.1
37.0
74.7
48.2
65.1
-1.7
-0.5
0
-2.1
-0.2
Table 11: MAE: Evaluation of the mini-hatch sampling distribution various downstream task. Sampling lead to similar performance across tasks.
imbalanced
CIFARI00
CIFAR100 1%
Place205
Clevr/Count
Clevr/Dist
KITTI
960 cls per batch
75.0
28.3
50.4
90.4
72.4
70.0
2 cls per batch
75.4
30.8
50.3
89.6
71.7
70.0
A
+0.4
+2.5
-0.1
-0.8
-0.7
+0.0
Table 12: In-distrihution: Evaluation of the mini-batch sampling distribution on in-distribution ImageNet linear evaluation using 100% of the training set.
SimCLR
MSN
VICReg
data2vec
MAE
class balanced sampling
66.9
77.1
69.1
41.5
65.9
class imbalanced sampling
55.8
59.4
51.4
40,6
65.8
A
-11.1
-17.7
-17.7
-0.8
-0.1
Table 13: In-distribution low-shot: Evaluation of the mini-batch sampling distribution in-distribution ImageNet linear evaluation using only 1% the training set.
MSN
VICReg
data2vec
MAE
class balanced sampling
66.2
48.6
27.4
35.1
class imbalanced sampling
28.0
34.8
-38.2
-30.5
+4.0
-0.3
25

---

Published as a conference paper at ICLR 2023
SimCLR balanced vs imbalanced
MSN balanced vs imbalanced
ViCReg balanced imbalanced
KITTVOit
CLEVR/Dst
CLEVR/SH
CLEVIVDiSt
KITTOiis
CLEVR/Coutt
Places20s
CLEVRitount
Places206
CLEVIUCourt
Paccs205
CIFAR100
CIRAR100
CIFARLCO
INat18
Not18
Natt8
10
1s
accuracy
accuracy
acturacy
Figure 6: Visual representations of the results of Table 1. Methods relying on volume maximization regularizers all exhibit similar performance alteration across diverse transfer tasks.
26


Reviews:
[{"review_id": "X7tIbGd9rR", "reviewer": "Reviewer_1J6n", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper points out a phenomenon that typical self-supervised learning losses implicitly enforces a feature prior that forms clusters of roughly equal size. The authors empirically verifies that this behavior can hurt the performance if the data samples are class-imbalanced. Then the authors proposed a variant that assumes other prior class distribution (i.e. long tail distributions such as power-law), and demonstrated that this could remedy the problem.", "summary_of_the_review": "I like the simplicity and the insight of the paper. The only concern is that only one algorithm and its variant is demonstrated.", "strength_and_weaknesses": "Strength:\n- Class imbalanced data is ubiquitous and deserves attention, especially for self-supervised setting where you don't know the data distribution.\n\nWeakness:\n- The only disappointment I had is that the authors only demonstrated the performance comparison for the MSN algorithm, where less deviation is needed. It would be more comprehensive if the authors can show how long tail distribution prior can be enforced in other algorithms.\n- The experiment is clear and convincing, but it seems only image modality is considered. I'm not sure how much this affects the generalizability of the problem/solution. \n- It would also be a good sanity check to visualize the features learned from class-balanced sampling vs class-imbalanced sampling.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TUU6M3XILRo", "reviewer": "Reviewer_sHjw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper points out that the overlooked uniform class prior would harm the learning of semantic representation for SSL methods, especially those with collapse prevention techniques, on class-imbalanced real-world datasets. The paper formulates and proves the objectives of the recent representative SSL methods to be essentially k-means problems with assumption of uniform prior. The paper shows the negative impact of uniform prior on class-imbalanced dataset through a comparative experiment on sampling strategies and visualizes the learned prototypes. The paper subsequently proposes a remedy by extending the objective of MSN through the recovering of the entropy term back to the relative entropy from an arbitrary prior to the average prediction. The arbitrary prior to the paper’s interest is the power-law distribution which is often close to a typical real-world dataset. Alongside with a toy experiments to showcase the impact of different priors and mini-batch distributions, downstream tasks of classification and object distance detection are compared under different pretraining datasets and priors and shows the performance gain on class-imbalanced dataset with power-law prior. With different impacts of prior mismatch on uniform and non-uniform datasets, the author notes that the class-imbalanced pretraining issue is not completely solved.", "summary_of_the_review": "The paper formulates current SSL as K-means problem and points out the overlooked uniform prior in those methods with volume maximization techniques and demonstrate that it can harm the model pretrained on class-imbalanced datasets, which is usually the case in real world. The proposed method is a straightforward extension of MSN. With power-law prior, it yields improved performance over uniform prior on class-imbalanced dataset. However, experiments in this paper also shows that the cost of the prior mismatch over balanced\ndataset is much more higher than over class-imbalanced dataset, which suggests the uniform prior is a safer prior when there is no access to the label. Furthermore, there is a potential of unfair comparison in the experiment reported Table 3.\n\nOn the whole, my recommendation of the paper is marginal accept", "strength_and_weaknesses": "Strength:\n1. It is probably the first paper in SSL domain to classify and formulate the current typical SSL methods into more or less constrained k-means problems with detailed proofs. which can be referred to as a review from the theoretical perspective.\n2. The paper utilizes smartly designed experiments to demonstrate the negative impact of prior mismatch (in mini-batch) to SSL with volume maximization regularizers.\n3. Performance improvement of pretraining on class-imbalanced dataset is consistent on multiple classification downstream tasks.\n\nWeakness:\n1. Unfair comparison with different models on different pretext dataset. In Table 3. The model pretrained on iNat18 is ViT-S while the one on ImageNet-1K is ViT-B. The capacities of this two model are very different (the latter is 4 times larger than former i.t.o # parameters) compared to the size differences of the two dataset (roughly 2 times). To better demonstrate the gain of power-law prior on the class-imbalanced dateset, the model impact should be controlled.\n2. Limited gain on choosing long-tailed prior over uniform. As shown in Table 2 and Table 3, the performance improvement on classification is minor compared to the loss with power-law prior on balanced dataset. Looks like without the knowledge of class distribution, the uniform prior is a much safer option.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OumtEmOVPcw", "reviewer": "Reviewer_JLgU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper formulates several self supervised learning methods (VICReg, SimCLR, MSN, SwAV) with the volume maximization principle as variants of k-means, analyzes their difference on class-balanced and class-imbalanced data, and indicates that the uniform prior w.r.t. prototype vectors is the curse of performance degeneration of class-imbalanced data. For resolving this problem, this paper develops PMSN, an extension to MSN which replaces the uniform prior with a power-law distribution. Experiments verifies the effectiveness of the proposed method.", "summary_of_the_review": "In general, this paper focuses on an important and interesting problem w.r.t. self supervised learning, and its findings are insightful. Experiments can be further improved to better support the claims and verifies theoretical analysis made in the paper.", "strength_and_weaknesses": "Strength:\n1. The analysis of SSL methods with volume maximization principle on class-imbalanced data is novel and impressive\n2. Experiment analysis of the performance degeneration on class-imbalanced data is thorough and impressive\n3. The findings of the connection between SSL methods and k-means is insightful\n\nWeakness:\n1. The toy experiment can be further analyzed. From Figure 3, the power-law prior encourages the model to retain semantic features related to MNIST but discard semantic features related to CIFAR10 (the nearest neighbors have different CIFAR classes). It seems that the choice of prior decides which types of class conditional semantic features will be retained (uniform or power-law). Is it possible to retain semantic features for both MNIST and CIFAR (e.g., by using a mixture prior?). Hope to see more analysis.\n\n2. The power-law prior is only tested on MSN. As several volume maximization based SSL methods are discussed in the paper, a natural question is that whether introducing the power-law prior will be helpful for these method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6vUWk890P71", "reviewer": "Reviewer_vagN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThe paper reports the effects of a hidden prior (i.e., the uniform feature distribution assumption) used in the existing self-supervised pretraining methods for class-balanced datasets and real world class-imbalanced datasets. Moreover, a power-law distribution assumption to perfer long-tail prior is formulated for self-supervised pre-training and a method called Prior Matching for Siamse Networks (PMSN) is proposed. Extensive experiments are conducted on several benchmark datasets, showing that using the power-law distribution prior yields some improvements on class-imbalanced pretraining dataset (i.e. iNaturalist18) but leads to significant performance degeneration on class-balanced pre-training data (i.e., ImageNet). ", "summary_of_the_review": "The paper is clearly written and the experiments are extensive and convincing. The founding is novel and interesting. However, notice of that when using the long-tail class pre-training dataset and the power-law prior leads to marginal improvements, it potential value is limited. \n\n\n\n=====\nAfter reading the responses and the submission, the reviewer would like to increase the rating due to the contribution on implicit priors in self-supervised learning. ", "strength_and_weaknesses": "The  strengths of the paper: \n\n+ The paper points out an interesting hidden prior used in the existing self-supervised representation learning methods and did extensive empirical evaluation to show the effects on down-stream classification task. \n\n+ MSN is extended Prior Matching for Siamense Networks (PMSN) to incorporate arbitrary prior. \n\n\nThe weaknesses of the paper: \n\n- While using a power-law distribution assumption to account for the long-tail class pre-training dataset leads to some improvements, it is somewhat marginal. However, it leads to significant performance degeneration when the pre-train dataset is class-balanced. In this sense, it potential value in application is limited. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster