Rethinking Positive Sampling for Contrastive Learning with Kernel
Authors: Benoit Dufumier
Carlo Alberto Barbano
Robin Louiset
Edouard Duchesnay
Pietro Gori
Venue: Submitted to ICLR 2023 (2023)
Keywords: contrastive learning
kernel theory
representation learning
deep learning

Abstract:   Data augmentation is a crucial component in unsupervised contrastive learning (CL). It determines how positive samples are defined and, ultimately, the quality of the representation. Even if efforts have been made to find efficient augmentations for ImageNet, CL underperforms compared to supervised methods and it is still an open problem in other applications, such as medical imaging, or in datasets with easy-to-learn but irrelevant imaging features. In this work, we propose a new way to define positive samples using kernel theory along with a novel loss called \textit{decoupled uniformity}. We propose to integrate prior information, learnt from generative models viewed as feature extractor, or given as auxiliary attributes, into contrastive learning, to make it less dependent on data augmentation.  We draw a connection between contrastive learning and the conditional mean embedding theory to derive tight bounds on the downstream classification loss.  In an unsupervised setting, we empirically demonstrate that CL benefits from generative models, such as VAE and GAN, to less rely on data augmentations. We validate our framework on vision and medical datasets including CIFAR10, CIFAR100, STL10, ImageNet100, CheXpert and a brain MRI dataset. In the weakly supervised setting, we demonstrate that our formulation provides state-of-the-art results.

Under review as a conference paper at ICLR 2023
RETHINKING POSITIVE SAMPLING FOR CONTRASTIVE LEARNING WITH KERNEL
Anonymous authors Paper under double-blind review
ABSTRACT
Data augmentation is a crucial component in unsupervised contrastive learning (CL). It determines how positive samples defined and, ultimately, the quality of the representation. Even if efforts have been made to find efficient augmenta- tions for ImageNet, CL underperforms compared to supervised methods and it still an open problem in other applications, such as medical imaging, in datasets with easy-to-learn but irrelevant imaging features. this work, we propose a new way to define positive samples using kernel theory along with novel loss called decoupled uniformity. We propose to integrate prior information, learnt from gen- erative models viewed as feature extractor, given auxiliary attributes, into contrastive learning, to make it less dependent on data augmentation. We draw connection between contrastive learning and the conditional mean embedding theory to derive tight bounds on the downstream classification loss. In an un- supervised setting, models, such as VAE and GAN, to less rely on data augmentations. We validate our framework on vision and medical datasets including CIFARIO, CIFARI00, STL10, ImageNet100, CheXpert and a brain MRI dataset. In the weakly super- vised setting, we demonstrate that our formulation provides state-of-the ofethe-art re- sults.
empirically demonstrate that CL benefits from generative
INTRODUCTION
Incomplete Augmantation Graph
Kernel Graph
Extended Augmentation Graph
Dlss
Dlwox
Figure 1: Illustration of the proposed method. Each point an original image T. Two points are connected if they can be transformed into the same augmented image using distribution of aug- mentations A. Colors represent semantic (unknown) classes and light disks represent the support of augmentations for each sample z, AA(-lz). From incomplete augmentation graph (1) where intra-class samples are not connected (c.g. augmentations are insufficient or not adapted), we recon- nect them using kernel defined prior information (either learnt with generative model, viewed as feature extractor, or given as auxiliary attributes). The extended augmentation graph (3) is the union between the (incomplete) augmentation graph (1) and the kernel graph (2). In (2), the gray disk indicates the set of points z' that are close to the anchor (blue star) in the kernel space.
Contrastive Learning (CL) 4 10) is paradigm designed for representation learning which has been applied to unsupervised (10) 13), weakly supervised(55) 20) and supervised problems 37 It gained popularity during the last years by achieving impressive results in the unsupervised

---

Under review as a conference paper at ICLR 2023
setting standard vision datasets (e.g. ImageNet) where it almost matched the performance of its supervised counterpart (TO 22)).
The objective in CL is to increase the similarity in the representation space between positive sam- ples (semantically close), while decreasing the similarity between negative samples (semantically distinct). Despite its simple formulation, it requires the definition of similarity function (that can be seen as an energy term annd of a rule to decide whether a sample should be considered posi- tive or negative. Similarity functions, such as the Euclidean scalar product (e.g. InfoNCE(44)), take as input the latent representations of an encoder f E F, such as a CNN 1 or a Transformer ( for vision datasets.
In supervised learning (37). positives are simply images belonging to the same class while negatives are images belonging to different classes. In unsupervised learning (IO), since labels are unknown, positives are usually defined as transformed versions (views) of the same original image (a.k.a. the anchor) and negatives are the transformed versions of all other images. As result, the augmenta- tion distribution A used to sample both positives and negatives is crucial (10) and it conditions the quality of the learnt representation. The most-used augmentations for visual representations involve aggressive crop and color distortion. Cropping induces representations with high occlusion invari- ance (46) while color distortion may avoids the encoder f to take a shortcut (IO) while aligning positive sample representations and fall into the simplicity bias (5I).
Nevertheless, learning a representation that mainly relies on augmentations comes at a cost: both crop and color distortion induce strong biases in the final representation (46). Specifically, dom- inant objects inside images can prevent the model from learning features of smaller objects (12)
(which is not apparent in object-centric datasets such as ImageNet) and few, irrelevant and easy-to- learn features, that are shared among views, are sufficient to collapse the representation (12) (a. feature suppression). Finding the right augmentations in other visual domains, such as medical imaging, remains an open challenge (20) since we need to find transformations that preserve seman- tic anatomical structures (e.g. discriminative between pathological and healthy) while removing unwanted noise. If the augmentations are too weak or inadequate to remove irrelevant signal w.r.t. discrimination task, then how can we define positive samples?
In our work, we propose to integrate prior information, learnt from generative models given as auxiliary attributes, into contrastive learning, to make less dependent on data augmentation. Using the theoretical understanding of CL through the augmentation graph, we make the connection with kernel theory and introduce a novel loss with thcoretical guarantees on downstream performance. Prior information is integrated into the proposed contrastive loss using a kernel. In the unsupervised setting, we leverage pre-trained generative models, such as GAN (24 and VAE (38), to learn a prior representation of the data. We provide a solution to the feature suppression issue in CL (12) and also demonstrate SOTA results with weaker augmentations on visual benchmarks. In visual domain where data augmentations are not adapted to the downstream task (e.g. medical imaging), we show that we can improve CL, alleviating the need to find efficient augmentations. In the weakly supervised setting, we use instead auxiliary/prior information, such image attributes (c.g. birds color or size) and we show better performance than previous conditional formulations based on these attributes (55).
In summary, we make the following contributions:
1. We propose a new framework for contrastive learning allowing the integration of prior informa- tion, learnt from generative models or given as auxiliary attributes, into the positive sampling.
2. We derive theoretical bounds on the downstream classification risk that rely on weaker assump- tions for data augmentations than previous works on CL.
3. We empirically show that our framework can benefit from the latest advances of generative
models to learn a better representation while relying on less augmentations.
4. We show that we achieve SOTA results in the unsupervised and weakly supervised setting.
2 RELATED WORKS
In weakly supervised setting, recent studies 20 55) have shown that positive samples can be de- fined conditionally to an auxiliary attribute in order to improve the final representation, in particular for medical imaging (20). From information bottleneck perspective, these approaches essentially
2

---

Under review as a conference paper at ICLR 2023
compress the representation be predictive of the auxiliary attributes. This might harm the perfor- mance of the model when these attributes are too noisy to accurately approximate the true semantic labels for a given downstream task.
In an unsupervised setting, recent approaches (22 63 66
used the encoder f. learnt during optimization, to extend the positive sampling procedure to other views of different instances (i.e. distinct from the anchor) that are close to the anchor in the latent space. In order to avoid rep- resentation collapse, multiple instances of the same sample (2), support set (22), a momentum encoder (43) or another small network (65) can be used to select the positive samples. In clustering approaches (43. (). distinct instances with close semantics are attracted in the latent space using pro- totypes. These prototypes can be estimated through K-means (43) or Sinkhorn- Sinkhorn-Knopp algorithm (8). All these methods rely on the past representation of network to improve the current one. They re- quire strong augmentations and they essentially assume that the closest points in the representation space belong to the same latent class in order to better select the positives. This inductive bias is still poorly understood from theoretical point of view (50) and may depend on the visual domain. For medical imaging, ImageNet self-supervised pre-training was beneficial for all subsequent tasks (2).
43
Our work also relates to generative models for learning representations. VAEs (38) learn the data distribution by mapping each input to a Gaussian distribution that we can easily sample from to reconstruct the original image. GANs (24), instead, sample directly from a Gaussian distribution to generate images that are classified by a discriminator in a min-max game. The discriminator representation can then be used (48) as feature extractor. Other models (ALI (21), BiGAN (18) and BigBiGAN(I9) learn simultancously generator and an encoder that can be used directly for representation learning (63) All these models do not require particular augmentations model the data distribution but they perform generally poorer than recent discriminative approaches (64, [ for representation learning. A first connection between generative models and contrastive learning has emerged very recently (36). In (36), authors study the feasibility of learning effective visual representations using only generated samples, and not real ones, with a contrastive loss. Their empirical analysis is complementary to our work. Here, we leverage the representation capacity of the generative models, rather than their generative power, to learn prior representation of the data.
3 CONSTRASTIVE LEARNING WITH DECOUPLED UNIFORMITY
Problem setup. The general problem in contrastive learning is to learn a data representation using an encoder f E F : X' + Sd-1 that is pre-trained with set of original samples (Z.))e[L..a) X, sampled from the data distribution p( I These samples are transformed to generate positive samples (i.e., semantically similar to X) in X, space of augmented images, using a distribution of augmentations A(. Concretely, for each T; we can sample views of I; using x A(- (e.g.. by applying color jittering, flip or crop with a given probability). For consistency, we assume A(T) = p(z) so that the distributions A(- and p(z) induce marginal distribution p(x) over X. Given an anchor T; all views
A(-(77,) from different samples Zzzt are considered as
negatives. Once pre-trained, the encoder ff is fixed and its representation fX() is evaluated through linear evaluation on a classification task using a labeled dataset D {(T,, y:)} E X x y where
) = []. with K the number of classes.
Linear evaluation. To evaluate the representation of I on a classification task, we train a linear classifier g = (*) (J is fixed) that minimizes the multi-class classification error.
Objective. The popular InfoNCE loss (45) 44), often used in CL., imposes 1) alignment between pos-
itives and 2) uniformity between the views (x iid. A(- of all instances 전 (57) two properties that correlate well with downstream performance. However, by imposing uniformity between all views, we essentially try to both attract (alignment) and repel (uniformity) positive samples and therefore we cannot achieve a perfect alignment and uniformity, as noted in (57). Moreover, InfoNCE has been originally designed for only two views (i.e., one couple of positive) and its extension to multi- ple views is not straightforward (40). Previous works have proposed a solution to either the first (54) or second (60) issue. Here, we propose a modified version of the uniformity loss, presented in (57). that solves both issues since it: i) decouples positives from negatives, similarly to (60) and ii) is
With abuse of notation, we define it as p(7) instead than p& to simplify the presentation, as it is common in the literature
3

---

Under review as a conference paper at ICLR 2023
generalizable to multi-views as in (54). We introduce the Decoupled Uniformity loss as:
Sunsy (f) = log p 1lp
(1)
where jiz = EA(=(a) (x) is called centroid of the views of ã. This loss essentially repels distinct centroids M4 through an average pairwise Gaussian potential. Interestingly, implicitly optimizes alignment between positives through the maximization of | D
so we do not need to explicitly add an alignment term. It can be shown (see Appendix B that minimizing this loss brings
representation space where the sum of similarities between views of the same sample is greater than the sum of similarities between views of different samples.
We will study its main properties hereafter and we will see that, contrary to other contrastive losses, prior information can be added during the estimation step of these centroids using a kernel. First, we define measure of the risk on a downstream task.
Supervised risk. While previous analysis (88) generally used the mean cross-entropy loss (as has closer analytic form with InfoNCE), we use supervised loss closer to decoupled uniformity with the same guarantees as the mean cross-entropy loss (see Appendix C.T). Notably, the geometry of the representation space optimum is the same cross-entropy and SupCon (37) and we can theoretically achieve perfect linear classification.
Definition 3.1. (Downstream supervised loss) For a given downstream task DD X x Y, we define the classification loss as: (J) = log E. g.y'>p(v))(()) ll,, 1l, where My = (x.
This loss depends on centroids rather than f()). Empirically, it has been shown (23 that per- forming feature averaging gives better performance on the downstream task.
3.1 GROMETRICAL. ANALYSIS OF DECOUPLED UNIFORMITY
Definition 3.2. (Finite-samples estimator) For n samples (āx))E(1..)) d.dd p(ā), the (biased) empir-
ical estimator of ( (f) is: (fm = log n(n Σigg Mex, Me, It converges in law to Fumis (f) with rate O (n-1/2). Proof in Appendix E.1
Theorem 1. (Optimality of Decoupled Uniformity) Given n points (5.)1e(1...)) such that S d+ 1, any optimal encoder f minimizing Ainis achieves representation s.t.:
1. (Perfect uniformity) All centroids (Ha,)ie(1... make a regular simplex on the hyper-sphere S 2. (.eeffet alignment) f is perfectly aligned, ie Vz, x' ~ A(-la.), (x) = (x') for all [1..n]. Proof in Appendix E.2
Theorem gives a complete geometrical characterization when the batch size set during training is not too large compared to the representation space dimension d. By removing the coupling between positives and negatives, we see that Decoupled Uniformity can realize both perfect alignment and uniformity, contrary to InfoNCE (57).
Most recent theories about CL 558 28 make the hypothesis that samples from the same seman- tic class have overlapping augmented views to provide guarantees on the downstream task when optimizing InfoNCE (1O) or Spectral Contrastive loss (28). This assumption, known as intra-class connectivity hypothesis, is very strong and only relics on the augmentation distribution A. In partic- ular, augmentations should not be "too weak", so that all intra-class samples are connected among them, and at the same time not "too strong", to prevent connections between inter-class samples and thus preserve the semantic information. Here, prove that we can relax this hypothesis if we can provide a kernel (viewed as a similarity function between original samples F) that is "good enough" relate intra-class samples that generative models (viewed as feature extractor) or auxiliary information can define such kernel. We first recall the definition of the augmentation graph (58). and intra-class connectivity hypothesis before presenting our main theorems. For simplicity, we assume that the set of images X' is finite (similarly to (58 28)). Our bounds and theoretical guarantees will never depend on the cardinality [.)
connected by the augmentations (see Fig. [ In practice, we show
By Jensen's inequality ||,=|| S F FAG FA with equality iff f is constant on supp A(-
4

---

Under review as a conference paper at ICLR 2023
INTRA-CLASS CONNECTIVITY HYPOTHESIS.
Definition 3.3. (Augmentation graph (28 58) Given set of original images X, we define the augmentation graph Ga(V, E) for an augmentation distribution A through 1) a set of vertices V X' and 2) a set of edges such that (a,z))
E E the two original images ž, I' can be transformed into the same augmented image through A, i.e supp A(- n supp A(-1=') & 0.
Previous analysis in CL make the hypothesis that there exists an optimal (accessible) augmentation module A that fulfills:
Assumption 1. (Intra-class connectivity (58) For given downstream classification task D = X' x y/ Vy V, the augmentation subgraph, Gg C G containing images only from class y, connected.
Under this hypothesis, Decoupled Uniformity loss can also tightly bound the downstream supervised risk but for a bigger class of encoders than prior work (not restricted to L-smooth functions (58)).
Definition 3.4. (Weak-aligned encoder) An encoder f E Fis e'-kea (e' 2 0) aligned on A if:
- [(x)]l s'' Va E X. Va. iid. A(-13)
Theorem 2. (Guarantees with AA) Given an optimal augmentation module A", for any e-weak aligned encoder E F we obtain: Limii (f) I (sup(f) S 8Dc + L (f) where D is the maximum diameter of all intra-class graphs Gy (y E D). Proof in Appendix E.5
Contrary to previous work (58), this theorem does not require L-smoothness f E F (strong as- sumption) and provides tighter lower bound. In practice, the diameter D can be controlled by small constant in some cases (e.g., 4 in (58)) but it remains specific to the dataset at hand. Furthermore, observe (see Appendix A.I] that f realizes alignment with small error during optimization of Lumis (f) for augmentations close to the sweet spot A (54) on CIFAR-10 and CIFAR-
In the next section, we study the case when A' is not accessible or very hard to find.
3.2 RECONNECT THE DISCONNECTED: EXTENDING THE AUGMENTATION GRAPH WITH KERNEL
Having access to optimal augmentations is a strong assumption and, for many real-world applica- tions (e.g medical imaging (20). it may not be accessible. If we have only weak augmentations (e.g., supp A(- Ç A (-Iz) for any 3), then some intra-class points might not be connected and we would need to reconnect them to ensure good downstream accuracy (see Theorem in Ap- pendix C.2 Augmentations are intuitive and they have been hand-crafted for decades by using human perception (e.g., rotated chair remains chair and gcay-scale dog is still a dog). However, we may know other prior information about objects that are difficult to transfer through invariance augmentations (e.g., chairs should have 4 legs). This prior information image attributes (e.g., age or sex of a person, color of a bird, etc.) or, in an unsupervised setting, directly learnt through generative model (e.g., GAN or VAE). Now, we ask: how can we integrate this information inside a contrastive framework to reconnect intra-class images that are actually dis- connected in G We rely on conditional mean embedding theory and use kernel defined on the prior pppresentation/iiforaatt
be either given as
This allows us to estimate a better configuration of the centroids the representation space, with respect to the downstream task, and, ultimately, provide theoretical guarantees on the classification risk.
<-KERNEL GRAPH.
Definition 3.5. (RKHS on X)) We define the RKHS (H,z Kx)) on associated with a kernel Kg-
Example. If we work with large natural images, assuming that we know prior 2(x) about our images (e.g., given by a generative model), we can compute K2x using through Kx ((1)) where R a standard kernel (e.g., Gaussian or Cosine).
To link kernel theory with the previous augmentation graph, we need to define kernel graph that connects images with high similarity in the kernel space.
Definition 3.6. (e-Kernel graph) Let 6 > O. We define the e-kernel graph Gi (V, Ex) for the kernel K2 & through 1) a set of vertices V = L and 2) set of edges Exg such that Ex between F, F' E X iff max(K K (5, F), K (x x')) Kx (7. x') S e.
5

---

Under review as a conference paper at ICLR 2023
The condition max((xx(E:))) Kx(',, dx (z,z") = (x2 (x) ) implies that d x²) S 2c where is the kernel distance. For kernels with constant norm (e.g.. the standard Gaussian, Cosine or Laplacian kernel,, it is in fact an equivalence. Intu- itively, it means that connect two original points in the kernel graph if they have small distance in the kernel space. We give now our main assumption to derive a better estimator of the centroid Me in the insufficient augmentation regime.
Assumption 2. (Extended intra-class connectivity) For given task D = X x V the extended (V, U Ex₂) (union between augmentation graph and e-kernel graph)
graph G A U Gig & is class- conneceed for all y E y.
This assumption is notably weaker than Assumption I w.r.t augmentation distribution A. Here, we do not need to find the optimal distribution of augmentations AA", as long as we have a kernel K2 such that disconnected points in the augmentation graph are connected in the e-kernel graph. K is not well adapted to the data-set (i.e it gives very low values for intra-class points), then needs to be large to re-connect these points and, as shown in Appendix AA. the classification error will be high. In practice. this means that we need to tune the Dlyper-paameee of the kernel (e.g., a for RBF kernel) so that all intra-class points are reconnected with small e.
CONDITIONAL MEAN EMBEDDING.
Decoupled Uniformity loss includes no kernel in its raw form. It only depends on centroids As EACeja) (x:) Here, we show that another consistent estimator of these centroids can be defined, using the previous kernel Kxx To show it, we fix an encoder f F and require the following technical assumption in order to apply conditional mean embedding theory (2.39)
Assumption 3. (Expressivity of Kx) The (unique) RKHS (H, defined on X' with kernel K = (f(), f()R fulfills Vig Hy, (())))((Z)) E H2x
Theorem 3. (Centroid estimation) Let (zy, [E).s( tor of the centroid is:
sid A(x, क). Assuming a consistent estima-
Vo & Aoz ()))
K (zj. 3) and K, = [K & (Zi, F, T.)la.e(1..)) It converges to Me with the Eg norm at rate Onn-1/4) for X (n 1/2). Proof in Appendix E.6
where cx(T) -
+ + -)=
This theorem says that we can use representations of images close to an anchor こ, accord- ing to our prior information, to accurately estimate μız. Consequently, if the prior is "good enough" to connect intra-class images disconnected in the augmentation graph (i.e. fulfills Assumption 2). then this estimator allows us to tightly control the classification risk. From this theorem, we naturally derive the empirical Kernel Decoupled Uniformity loss using the previous estimator.
Intuition.
Definition 3.7. (Empirical Kernel Decoupled Uniformity Loss) Let (x.,, F:))e((...)) od A(x, .). Let KKn)s Aæ Στ 0s. (x;;) with at, ((K,, + ( À O(n 1/2) regularization constant and K, = x (Ze, F, s.sep....... We define the empirical kernel decoupled uniformity loss as:
Eumey (J) def log n(n 1)
1
-1x,113)
f===1
Extension to multi-views. If we have V views (a! Dee[1.. for each T. we can easily extend the previous estimator with Ú2z + EV= 14. where p = Ei= [1.55(((()))
The computational cost added is roughly O(n²) (to compute the inverse matrix of size n x n) but it remains negligible compared to the back-propagation time using classical stochastic gradient descent. Importantly, the gradients associated to at, are not computed.
A TIGHT BOUND ON THE CLASSIFICATION LOSS WITH WEAKER ASSUMPTIONS.
We show here that Enmiy()) can tightly bound the supervised classification risk for well-aligned encoders f T F.

---

Under review as a conference paper at ICLR 2023
Theorem 4. We assume
2 and 3 hold for a reproducible kernel Kx and augmentation distribution
itd A. Let (zi, B.).e( ~ AZ,, For any c'-weak aligned encoder f E F:
Ld Cuns (f) - -0 (n (11) S LSu S Lumey (f) + 4D(2'' + B. +0 (n (1/1/4) (4)
where Bn
(Amin((a) (Km) + VTAA)= v/
O(1) for x
O(n-1/2), Kn
(Kx(Bi; Fy)).. D is the maximal diameter of all sub- aub-graphs G & where y V We noted Amin (Kn) > 0 the minimal eigenvalue of K.. Proof in Appendix E.7
Interpretation. Theorem gives tight bounds on the classification loss (f) with weaker as- sumptions than current work (I 588 28). We don't require perfect alignment for f E F or L- smoothness and we don't 't have class collision term (even if the extended augmentation graph may contain edges between inter-class samples), contrarily
(I). Also, the estimation error doesn't depend on the number of views (which is low in practice))-as it was always the case in previous formulations (58 28 but rather the batch size and the eigenvalues of the kernel ma- trix (controlling the variance of the centroid estimator (27 Contrarily to CCLK (55), we don't t condition our representation weak attributes but rather we provide better estimation of the con- ditional mean embedding conditionally to the original image. Eventually, our loss remains in an unconditional contrastive framework driven by the augmentations A and the prior x on input images. Theorem becomes a special case = 0 and A A" (i.e the augmentation graph is class- connected, a stronger assumption than 2). In Appendix A.2 we provide empirical evidence that better kernel quality (measured by k-NN accuracy in kernel graph) improves downstream accuracy, as theoretically expected by the theorem. It also provides a new way to select priori a good kernel.
EXPERIMENTS
Here, we study several problems where Kernel Decoupled Uniformity outperforms current con- trastive SOTA models. In unsupervised learning, we show that we can leverage generative models representation to outperform current self-supervised models when the augmentations are insufficient to remove irrelevant signals from images. In weakly supervised, we demonstrate the superiority of our unconditional formulation when noisy auxiliary attributes are available. Implementation details presented in Appendix D
We systematically use 2 views for faimesss in our main experiments.
Model
0 bits
5 bits
10 bits 20 bits
SimCLR (10)
79.4
68.74
13.67
10.07
BYOL (26)
10.14
19.98
10.33
10,00
3-VAE (8
41.37
43.32
42.94
43.1
3-VAE (8 = 2)
42.28
43.89
43.11
42.19
3-VAE (8 = 4)
42.5
42.5
42.5
39.87
Decoupled Unif (ours)
82.43
53.45
10.08
9.64
KYΛE Decoupled Unif (ours)
82.74
68.75
68.42
68.58 10.17
Table I: Linear evaluation accuracy (%) RandBits-CIFAR1O with ResNet18 for 200 epochs. For
VAE, we use ResNet18 backbone, Once trained, we use its representation to define the kernel KY in kernel decoupled uniformity loss.
Evading feature suppression with VAE. Previous investigations 122 have shown that a few easy- to-learn irrelevant features not removed by augmentations can prevent the model from learning all semantic features inside images. We propose here a first solution to this issue.
RandBits dataset (12). We build RandBits dataset based CIFAR-10. For each image, we add
a random integer sampled in [0, 2* - 1] where k is a controllable number of bits. To make easy to learn, we take its binary representation and repeat it to define k channels that are added to the original RGB channels. Importantly, these channels will not be altered by augmentations, so they will be shared across views. We train ResNet18 this dataset with standard SimCLR augmentations (10) and varying k. For kernel decoupled uniformity, we use a 3-VE representation (ResNet18 backbone, B 1, also trained on RandBits) to define Av I')
K (((()),(7))
7

---

Under review as a conference paper at ICLR 2023
where (()) is the mean Gaussian distribution of F in the VAE latent space and K is standard RBF kernel.
fixed encoder trained with various trastive (SimCLR, Decoupled Uniformity and Kernel Decoupled Uniformity) and non-contrastive (BYOL and 3-VAE) methods. As noted previously (12), /- 3-VAE is the only method insensitive to the number added bits, but its representation quality remains low compared to other discrimina- tive approaches. All contrastive approaches fail for 10 bits. This can be explained by noticing that, as the number of bits k increases, the number of edges between intra-class images in the aug- mentation graph GA decreases. For k bits, on average N/2* images share the same random bits (N = 50000 is the dataset size). So only these images can be connected in GA. For & = 20 bits, < image share the same bits which means that they are almost all disconnected, and it explains why standard contrastive approaches fail. Same trend is observed for non-contrastive approaches (e.g. BYOL) with a degradation in performance even faster than SimCLR. Interestingly, encour- aging disentangled representation by imposing higher 3 >
Table the linear evaluation accuracy computed
in 3-VAE does not help. Only our KY Decoupled Uniformity loss obtains good scores, regardless of the number of bits.
BigBiGAN as prior. We show that very recent advances in gen- erative modeling improve representations of contrastive models in Table 2 with approach. Due to our limited computational re- sources, we study ImageNet100 (54) (100-class subset of ImageNet used in the literature (3)(5) and we leverage BigBiGAN repre- sentation (99 as prior. In particular, we use BigBiGAN pre-trained ImageNet to define kernel KGAN(E, x') K (2()), =(3)) (with an RBF kernel and z(.) BigBiGAN's encoder). We demon- strate SOTA representation with this prior compared to all other trastive and non-contrastive approaches. We use ResNet50 trained for 400 epochs. Please note that in our implementation, use data augmentation for testing (i.e. during linear evaluation).
Model
ImageNet100
SimCLR 10 BYOL
66.52
26
72.26
CMC 53
73.58
DCL. 05
74.6
AlignUnif
IFM [39]
71.88
BigBiGAN
Decoupled Unif
72.24
Keas Decoupled Unif
76.58
Supervised
82.1
Table 2: Linear evaluation ac- curacy (%) on ImageNet100. "Results from papers
do not
CIFAR-10
CIFAR-100 100
STL-10
Model
w/o Color and Crop
w/o Color and Crop
w/o Color and Crop
All
w/o Color
All
w/o Color
All
w/o Color
SimCLR 100
81.75
62.56
34.07
53.02
38.27
15.28
59.01
39.56
BYOL (26
81.97
64.86
45.88
53.65
35.61
22.48
79.61
65.36
11.28
M3Co (4
86.51
67.71
42.12
58.83
36.95
22.11
83.02
64.25
38.38
VAR
41.37
14.34
42.17
42.17
DCGAN* (4
66.71
66.71
66.71
26.17
26.17
26.17
70.06
70.06
70,06
Decoupled Unif
85.82
60.45
39.18
58.89
34.16
14.58
79.89
54.53
36.81
KYAR AR Decoupled Unif
85.84
72.92
50.52
45.59
28.24
79. 10.10
61.39
45.66
KGAN Decoupled Unif
85.85
77.16
69.19
58.42
50.07
35.98
79.97
71.44
68.11
Table 3:
When augmentation overlap hypothesis is not fulfilled, generative models can provide good kernel to connect intra-class points not connected by augmentations. * For VAE and DCGAN, augmentations were used during training. Bold: best result; underlined: second best. All models have been trained for 400 epochs.
Towards weaker augmentations. Color distortion (including color jittering and gray-scale) and crop are the two most important augmentations for SimCLR and other contrastive models to ensure a good representation on ImageNet (10). Whether they are best suited for other datasets (e.g medical imaging (00 multi-objects images (12)) is still open question. Here, ask: can generative models remove the need for such strong augmentations? We use standard benchmarking datasets (CIFAR-10. CIFAR-I00 and STL-10) and we study the case where augmentations are too weak to connect all intra-class points. We compare to the baseline where all augmentations are used.
We use a trained VAE to define KVAE as before and a trained DCGAN (48) KGAN F') def ((x(T), (f²)) where z(-) denotes the discriminator output of the penultimate layer.
In Table 3] we observe that our contrastive framework with DCGAN representation as prior is able to approach the performance of self-snn culf-super models by applying only crop augmentations and flip. Additionally, when removing almost all augmentations (crop and color distortion). we approach the performance of the prior representations of the generative models. This is expected by our theory
8

---

Under review as a conference paper at ICLR 2023
since we have augmentation graph that is almost disjoint for all points and thus only rely on the prior to reconnect them. This experiment shows that our method is less sensitive than all other SOTA selff-uuprriied methods to the choice of the "optimal" augmentations, which could be relevant in applications where they are not known a priori, or hard to find.
Weakly supervised learning ages, In Table [4] we suppose that we have access to image attributes that correlate with the true semantic labels (e.g birds color/size for birds classification). We use three datasets: CUB- 200-2011 (59), ImageNet100 (54) and UTZapp pos (62), following (55). CUB-200-2011 con-
natural im-
Model
CUB
ImageNet100
UT-Zaappss
SimCI R
17.48
65.30
84,08
BYOL
16.82
72.20
85:48
CosKerael CCLK (55)
15.61
74.34
83.23
RRFKernel CCLK [s
30.49
77.24
CosKemal Decoupled Unit
27.77
78.8
85,56
KBFKerne
Unif
32.87
76.34
84.78
tains 11788 images of 200 bird species with 312 binary attributes available (encoding size, color, etc.). UTZappos contains 50025 images of shoes from several brands sub-categorized into 21 groups that tion labels. It comes with seven attributes. Finally, for ImageNet100 we follow (55) and use the pre-trained CLIP (47) model (trained on pairs (text, image)) extract 512-d features considered as prior information. We compare our method with SOTA Siamese models (SimCLR and B BOLL) and with CCLK, a conditional contrastive model that defines positive samples only according the conditioning attributes. The proposed method outperforms all other models on the three datasets.
Table 4: If images attributes are accessible (e.g birds color or size for CUB200), they can be leveraged prior in our framework to improve the representation.
downstream classifica-
Filling the gap for medical imaging. Data augmentations on natural images have been handcrafted over decades. However, we argue they are not adapted in other visual domain such as medical imaging (20). We study 1) bipolar disorder detection (BD), challenging binary classification task, on brain MRI dataset BIOBD (32) and 2) chest radiography interpretation, a 5-class classification task on CheXpert (35). BIOBD contains 356 healthy controls (HC) and 306 patients with BD. We use BHB (20) as large pre-training dataset containing 10k 3D images of healthy subjects. For CheXpert, we use Gloria (34) representation, a multi-modal approach trained with (medical report, image) pairs to extract 2048-d features weak annotations. We show that our approach improve contrastive model in both unsupervised (BD) and weakly supervised (CheXpert) setting for medical imaging.
Model
Atelectusi
Cardismecully
Couselidttion
Picural Effusina
Model
BD vs HC
SinCLR
1i
82.
37.62
90.52
$9.08
$6.83
SinCLR (10
60.466.1.2
BYOL
20
83.04
81.54
90.98
90.18
$5.09
BYOL 26
58.81
McCo CXR
3
75,8
73.7
77.1
56.7
$5.0
MoCo
60.27.1.
GLakiA
3
86. 70
86.99
90.41
90:58
91.82
50.94.nst
CCLK
86.31
83.67
92.45
91.59
91.23
VAE
52.80 es
Katanr Decoupted Unit (ouns)
86.92
80.88
93.03
92.39
91.93
Ky Decoupled Unif (ouns)
62.19) 11
Supersised
81.6
1977
90.5
36.8
39.9
Supervised
17.42
Table 5: AUC scores (%) under linear evaluation for discriminat- ing pathologies on CheXpert. ResNet18 backbone trained for 400 epochs (batch size N = 1024) without labels on official CheXpert training set and results are reported on validation set.
Table 6: BD detection. Linear evaluation AUC scores(%) using 5-fold leave-site-out CV scheme.
5 CONCLUSION
In this work, we have showed that we can integrate prior information into CL to improve the final representation. In particular, we draw connections between kernel theory and CL build our the- oretical framework. We demonstrate tight bounds on downstream classification performance with weaker assumptions than previous works. Empirically, we show that generative models provide a good prior when augmentations are too weak or insufficient to remove easy-to-leam noisy features. We also show applications in medical imaging in both unsupervised and weakly supervised setting where method outperforms all other models. Thanks to theoretical framework, we hope that CL will benefit from the future progress in generative modelling and it will widen its field of application to challenging tasks, such computer aided-diagnosis.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
[1] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A Theoretical Analysis of Contrastive Unsupervised Representation Learning. arXiv:1902.09229 [cs, stat], February 2019. arXiv: 1902.09229.
[2] Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, Vivek Natarajan, and Mo- hammad Norouzi. Big Self-Supervised Models Advance Medical Image Classification. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3458-3468. IEEE, 2021.
[3] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by max- imizing mutual information across views. Advances in neural information processing systems, 32, 2019.
[4] Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 351(6356) 1992.
[5] Sergiy V Borodachov, Douglas P Hardin, and Edward B Saff. Discrete energy on rectifiable sets. Springer, 2019.
[6] Keno Bressem, Lisa C Adams, Christoph Erxieben, Bernd Hamm, Stefan M Niehues, and Janis L Vahldiek. Comparing different deep learning architectures for classification of chest radiographs. Scientific reports, 10(1):1-16, 2020.
[7] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. Advances in neural information processing systems, 6, 1993.
[8] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
(9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[10] Ting Chen, Simon Korbblihh, Mohammad Norouzi. and Geoffrey Hinton. A Simple Frame- work for Contrastive Learning of Visual Representations. International Conference on Ma- chine Learning, pages 1597-1607. PMIR, November 2020. ISSN: 2640-3498.
[11] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big Self-Supervised Models are Strong Semi-Supervised Learners. In arXiv:2006. 10029 [cs, stat], Vancouver, Canada, October 2020. arXiv: 2006.10029.
[12] Ting Chen, Calvin Luo, and Lala Li. Intriguing Properties of Contrastive Losses. arXiv:2011.0288 [cs, stat), June 2021. arXiv: 0011.02803.
[13] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint wwXXi:20003002990 2020.
[14] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[15] Ching- Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. Debiased Contrastive Learning. arXiv:2007.00224 [cs, stat], October 2020. arXiv:
[16] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsu- pervised feature learning. In Proceedings of the fourteenth international conference artifi- cial intelligence and statistics, pages 215-223, JMLR Workshop and Conference Proceedings, 2011.
10

---

Under review as a conference paper at ICLR 2023
[17] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[18] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 1rXiv:1605.09972) 2016.
[19] Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning, Advances in Neural Information Processing Systems, 32, 2019.
[20] Benoit Dufumier, Pietro Gori, Julie Victor, Antoine Grigis, Michel Wessa, Paolo Brambilla, Pauline Favre, Mircea Polosan, Colm McDonald, Camille Marie Piguet, and Edouard Duch- esnay. Contrastive Learning with Continuous Proxy Meta-Data for 3D MRI Classification. arXiv:2106.. 0880000888888/8 [es, stat], June 2021. arXiv: 2106.08808.
[21] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Mar- tin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint 16Xx::6666.00044 2016.
[22] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest- -eiihht contrastive learning of visual representa- tions. arXiv preprint arXiv:2104. 10ti:21044.4444,, 2021.
[23] Adam Foster, Rattana Pukdee, and Tom Rainforth. Improving transformation invariance in contrastive representation learning. arXiv preprint arXiv:2010.09515. 2020.
[24] Ian Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[25] Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised constrastive learning. In International Conference on Machine Learning, pages 3821-3830. PMLR, 2021.
[26] Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et Bootstrap your own latent-a new approach to self-supervised learning. Ad- vances in Neural Information Processing Systems, 33:21271
2020.
[27] Steffen Grünewälder, Guy Lever, Luca Baldassarre, Sam Patterson, Arthur Gretton, and Mas- similano Pontil. Conditional mean embeddings as regressons-supllementayy arXiv preprint arXiv:1205.4656, 2012.
[28] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. arXiv preprint 0rXhv:2106.04156, 2021.
[29] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In CVPR, pages 9729-9738, 2020.
[30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pultern recogni- tion, pages 770-778, 2016.
[31] DP Hibar, Lars Tjelta Westlye, Nhat Trung Doan, Neda Jahanshad, JW Cheung, Christo- pher RK Ching, Amelia Versace, AC Bilderbeck, Anne Uhlmann, Mwangi, et al. Cortical abnormalities in bipolar disorder: an mri analysis of 6503 individuals from the enigma bipolar disorder working group. Molecular psychiatry, 23(4):932-9)) 942, 2018.
[32] Franz Hozer, Samuel Sarrazin, Charles Laidi, Pauline Favre, Melissa Pauling, Dara Cannon, Colm McDonald, Louise Emsell, -ran-François Mangin, Edouard Duchesnay, et al. Lithium prevents grey matter atrophy in patients with bipolar disorder: an international multicenter study. Psychological medicine, 51(7):1201-1211 2021.
11

---

Under review as a conference paper at ICLR 2023
[33] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely nected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattem recognition, pages 4700-4708, 2017.
[34] Shih-Cheng Huang. Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multi- modal global-local representation learning framework for label-efficient medical image recog- nition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942-3951, 2021.
[35] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ileus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597, 2019.
[36] Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. arXiv preprint arXiv:2106.05258, 2021.
[37] Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Samn,, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. Supervised Contrastive Learn- ing. arXiv:2004. / 1362 [cs, stat], December 2020. tex.ids= khosla supervised 2021 arXiv: 2004.11362.
[38] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13X22 14, 2013.
[39] Iaa Klebanov, Ingmar Schuster, and Timothy John Sullivan. A rigorous theory of conditional mean embeddings. SIAM Journal on Mathematics of Data Science, 2(3):583-6008 2020.
[40] Ching-Yun Yun Ko, Jeet Mohapatra, Sijia Liu, Pin-Ye Yu Chen, Luca Daniel, and Lily Weng. Revis- iting contrastive learning through the lens of neighborhood component analysis: an integrated framework. In International Conference on Machine Learning, pages 11387-11412. PMLR, 2022.
[41] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[42] Yann LeCun and Fu Jie Huang. Loss functions for discriminative training of energy-based models. In International Workshop on Artificial Intelligence and Statistics, pages 206-213. PMLR, 2005.
[43] Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint 20Xiv:2005.04966, 2020.
[44] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv: 0877003788 [es, stat), January 2019, arXiv: 1807.03748.
[45] Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A. Alemi, and George Tucker. On Variational Bounds of Mutual Information. arXiv:1905.0699 [cs, stat), May 2019. arXiv: 1905.06922.
[46] Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. Advances in Neural Information Processing Systems, 33:3407-3418, 2020.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pages 8763. PMLR, 2021.
[48] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 1011::5111006433 2015.
12

---

Under review as a conference paper at ICLR 2023
[49] Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in neural information processing systems, 34:4974 2021.
[50] Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporat- ing inductive biases. arXiv preprint arXiv:2202. /4037, 2022.
[51] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
[52] Le Song, Kenji Fukumizu, and Arthur Gretton. Kernel embeddings of conditional distributions: unified kernel framework for nonparametrie inference in graphical models. IEEE Signal Processing Magazine, 30(4):98-111, 2013.
[53] Hari Sowrirajan, Jingbo Yang, Andrew Y Ng, and Pranav Rajpurkar. Moco pretraining im- proves representation and transferability of chest x-ray models. In Medical Imaging with Deep Learning, pages 728-744. PMLR, 2021.
[54] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive Multiview Coding. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020, Lecture Notes in Computer Science, pages 776-794, Cham, 2020. Springer ternational Publishing. tex.ids= "aan_contrastive2000 arXiv: 1906.05849.
[55] Yao-Hung Hubert Tsai, Tianqin Li, Martin Ma, Han Zhao, Kun Zhang, Louis-Philippe Morency, and Ruslan Salakhutdinov. Conditional contrastive learning with kernel. arXiv preprint arXiv:2202.08 05458, 2022.
[56] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 2irds-200-2011 dataset. 2011.
[57] Tongzhou Wang and Phillip Isola. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. UrXiv:2005.10242 [cs, stat], November 2020. arXiv: 2005.10242.
[58] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: A new thcoretical understanding of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203. 2022.
[59] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. 2010.
[60] Chun-Hsiao Yeh, Cheng- Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann Le- Cun. Decoupled contrastive learning. arXiv preprint arXiv:2110.06844 2021.
[61] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprini arXiv:/7085
2017.
[62] Aron Yu and Kristen Grauman. Fine-grained visual comparisons with local learning. In Pro- ceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 192-199, 2014.
[63] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Syl- vain Gelly, and Neil Houlsby. A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark, 2020.
[64] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 191000486777.. 2019.
13

---

Under review as a conference paper at ICLR 2023
[65] Mingkai Zheng, Fei Wang. Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Weakly supervised contrastive learning. In Proceedings of the IEEE/CVF Interna- tional Conference Computer Vision, pages 10042-10051, 2021.
[66] Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Ressl: Relational self-supervised learning with weak augmentation. Advances in Neural Information Processing Systems, 34, 2021.
[67] Zongwei Zhou, Vatsal Sodha, Jiaxuan Pang, Michael Gotway, and Jianming Liang. Models genesis. Medical image analysis, 67:101840, 2021.
14

---

Under review as a conference paper at ICLR 2023
A MORE EMPIRICAL EVIDENCE
In this section, we provide additional empirical evidence to confirm several claims and arguments developed the paper.
A.1 DECOUPLED UNIFORMITY OPTIMIZES ALIGNMENT
CIFARIO
CIFARI00
0189
3m3
512
n=1024
Aignmure
0.65
0.65
0.60
0.80
0.55
0.55
0.50
0.50
50
300
150
200
50
100
150
200
Spacc
Figure 2: Alignment metric Lalign computed the validation set during optimization of Decoupled Uniformity loss with various batch sizes n and a fixed latent space dimension d = 128. We use 100 positive samples per image compute Lulign-
We empirically show here that Decoupled Uniformity optimizes aligmment, even in the regime when the batch size n > d + where the representation space dimension. We use CIFAR- 10 and CIFAR-100 datasets and we optimize Decoupled Uniformity (without kernel) with all SimCLR augmentations with 128 and we vary the batch size n. We report the alignment metric defined fx²
in (57) as falign = E A[
A.2
MEASURING KERNEL QUALITY AND EMPIRICAL VERIFICATION OF OUR THEORY
Accuracy (%)
Accuracy (%)
70
100
60
50
1001
40
10-2
20
10
100
0.20
0.22
0.24
0.26
0.28
for RBF kernel
Kernel Quality (in %)
Figure 3: Empirical verification of our the- ory. The optimal E to add 100 edges be- tween intra-class images in e-Kernel graph is inversely correlated with the downstream ac- curacy, as suggested by Theorem We use k 20 bits and an RBF kernel.
Figure 4: How we can select priori good kernel? Downstream accuracy on RandBits CIFAR-10 is highly correlated with kernel quality measured as fraction of 10 nearest neighbors of the same CIFAR- 10 class (from test set) in the kernel graph.
We provide empirical evidence confirming our theory (Theorem 4 in particular) along with new way to quantify kernel quality with respect to a downstream task for a kernel K. We perform experiments RandBits dataset (based on CIFAR-10) with & 20 random bits (almost all points are disconnected in the augmentation graph) and SimCLR augmentations. For a given kernel K, defined by KT .') RBF, (p(()) -where sa(-) is the mean Gaussian distribution of x in
15

---

Under review as a conference paper at ICLR 2023
VAE latent space trained on RandBits- we train Kernel Decoupled Uniformity with K, on RandBits. In Fig. 3 we vary a and we report downstream accuracy (measured by linear evaluation) along with the optimal c" to add 100 intra-class edges in the e-Kernel graph obtained with K,, The lower e", the better the downstream accuracy, which expected since the upper bound of supervised risk becomes tighter in Theorem 4 It gives first empirical confirmation that tightly bounds the supervised risk on downstream task.
A new way to quantify kernel quality. Based on the concept of kernel graph, we measure the quality of a given kernel K using the nearest-neighhors of each image (a vertex in kernel graph). More precisely, K induces a distance dx (dx(a,,) b) K(a,a) + K(b,b) b) 2K (a, b)) that can be used to define nearest-neighbors in its kernel graph. We compute the fraction of these nearest neighbors that belong the same class. In Fig. [ we plot the downstream accuracy vs kernel quality using 10-nearest neighbors for various kernel K. They are obtained by using latent space of a VAE trained for an increasing number of epochs (2, 50, 100, 150 and 1000) and by setting K (z, RBE, (u(z), ((²))) as before (with 50 fixed). It shows that this new measure of kernel quality highly correlated with final downstream accuracy. Therefore, it can be used as a tool to compare priori (without training) different kernels. One limitation of this metric is that it requires access to labels on the downstream task. Future work would consist in finding unsupervised properties of the kernel graph that correlates well with downstream accuracy (e.g. sparsity, clustering coefficient, etc.).
A.3 MULTI-VIEW CONTRASTIVE LEARNING WITH DECOUPLED UNIFORMITY
When the intra-class connectivity hypothesis is full-filled, we showed that Decoupled Uniformity loss can tightly bound the classification risk for well-aligned encoders (see Theorem 2 Under that hypothesis, we consider the standard empirical estimator of μl R Σ [((x(1)) for V views. Using all SimCLR augmentations, we empirically verify that increasing V allows for: 1) a better estimate of μą which implies faster convergence and 2) better SOTA results both small-scale (CIFARIO. CIFAR 100, STL10) and large-scale (ImageNet100)) vision datasets. We always use batch size 256 for all approaches with ResNet18 8 backbone for CIFAR1O, CIFAR and STL10 and ResNet50 for ImageNet100. We report the results Table
Model
CIFAR-10 10
CIFAR-100
ImageNet100
STL10
200
e 400
200
e 400
e 200
400
200
= 400
1i0 BYOL(66
79.4
81.75
48.89
53.02
65.30
66.52
76.99
79.02
80.14
81.97
51.57
53.65
72.20
72.26
77.62
79.61
Decoupled Unif (2 views)
82.43
85.82
54.01
58.89
71.98
72.24
78.12
79.89
Decoupled Unif (4 views)
84.99
85.34
57.23
59.07
72.08
75.00
78.25
80.47
Decoupled Unif (8 views)
86.50
85.80
59.63
59.74
74.70
75.00
79.82
80.30
Table 7: A better approximation of centroids μı (i.e. increasing number of views) when augmen- tation overlap hypothesis is (nearly) full-filled implies faster convergence. All models are pre- trained with batch size 256. We use ResNet18 backbone for CIFAR10, CIFAR100 STL10 and ResNet50 for ImageNet100. We report linear evaluation accuracy (%) for a given number of epochs e.
A.4 INFLUENCE OF TEMPERATURE AND BATCH SIZE FOR DECOUPLED UNIFORMITY
InfoNCE is known to be sensitive to batch size and temperature to provide SOTA results. In our theoretical framework, we assumed that [xx) Sd-1 but we can easily extend to (x) E Visd-1 where / > 0 is a hyper-parameter. It corresponds to write (f) = (p()))(())) - -1l We show here that Decoupled Uniformity does not require very large batch size (as it is the case for SimCLR) and produce good representations for / [1,
16

---

Under review as a conference paper at ICLR 2023
Datasets 1= 0.1
/
0.5
t=
= 10
CIFARIO
73.91
83.01
84.72
85.82
83.05
74.82
CIFAR100
39.16
51.33
55.91
58.89
56.70
48.29
Table 8: Linear evaluation accuracy (%) after training for 400 epochs with batch size = 256 and varying temperature in Decoupled Uniformity loss with SimCLR augmentations. / gives overall the best results, similarly to the uniformity loss in (7))
Datasets
Loss
72
128
n
512
n
1024
2048
SimCLR
78.89
79.40
80.02
80.06
CIFARIO
Decoupled Unif
82.67
82.12
82.74
82.33
SimCLR
49.53
53.46
54.45
55.32
CIFAR100
Decoupled Unif
54.61
54.12
55.56
55.20
Table Linear evaluation accuracy (%) after training for 200 epochs with a batch size n, ResNet18
backbone and latent dimension = 128. Decoupled Uniformity is less sensitive to batch size than SimCLR thanks to its decoupling between positives and negatives, similarly (60)
A.5 IMPORTANCE OF REGULARIZATION TERM IN CENTROIDS ESTIMATION
Kernel Decoupled Uniformity introduces an additional syper-parameter À for centroid estimation, which should be such that O where is the batch size to full-fill the hypothesis of Theorem We have cross-validated this hyper-parameter À on RandBits CIFAR-10 with k = 10 bits and we show in Table 10 that À= 0.01 yields the best results. We have fixed this value for all our experiments in this study.
Vn
256 A
30
50
0.001
10.25
60.75
0.01
67.21
68.42
0.1
59.09
58.13
1
50.49
60.75
Table 10: Importance of in centroids estimation with Kernel Decoupled Uniformity. We report linear evaluation accuracy after training RandBits-IIFRRO (10 bits) with ResNet18 for 200 epochs using RBFKernll (r) and batch size n 256.
A.6 KERNEL CHOICE ON RANDBITS EXPERIMENT
In our experiments on RandBits, we used RBF Kernel in Decoupled Uniformity but other kernels can be considered. Here, we have compared our approach with cosine kernel Randbits with
= 10 and A = 20 bits. There is no hyper-parameter to tune with cosine. From Table [] we see that cosine gives comparable results for k 10 bits with RBF but bits.
appropriate for 20
Kernel
10 bits
20 bits
RBFKernel(r 1)
66.2540.11 17
9.91
RBFKernel(r 30)
67.21
66.46.0.1
RBFKernel(o = 50)
68.42 0..1
68.58 08.18
CosineKernel
66.564u.45
9.68
Table 11: Linear evaluation after training on RandBins-IIFAR10 with ResNet18 for 200 epochs. RBF and Cosine kernels evaluated.
17

---

Under review as a conference paper at ICLR 2023
A.7 LARGER PRE-TRAME TRAINED GENERATIVE MODEL INDUCES BETTER PRIOR
We argue that using larger datasets (e.g,, ImageNet IK) for pre-training larger generative models will improve the prior smaller-scale datasets and improve even more the final representations with our method. We have tested this hypothesis on CIFAR-10 10 and BigBiGAN as prior, compared
DCGAN and the other approaches without prior.
Model
epochs
CIFAR-10
SimCLR TT)
400
81.75
BYOL (26)
400
81.97
MoCov3 14
400
86.51
Decoupled Unif
200
82.43
Decoupled Unif
400
85.82
KDCGAN Decoupled Unif
400
85.85
K BigB Decoupled Unif
200
84.93 (+2.5)
K BAAN Decoupled Unif
400
86.86 (+1.04)
Table 12:
We evaluate Kernel Decoupled Uniformity with BigBiGAN pre-trained on ImageNet as prior knowledge. We compare this approach with shallow DCGAN pre-trained on CIFAR-10 as prior. We train ResNet18 8 on CIFARIO and we report linear evaluation accuracy. Pre-trained generative models on larger datasets improve the final representation.
B GEOMETRICAL CONSIDERATIONS ABOUT DECOUPLED UNIFORMITY
B.1 ASYMPTOTICAL OPTIMALITY
The assumption n < d + 1 is crucial to have the existence of a regular simplex on the hypersphere sd-1 In practice, this condition is not always full-filled (e.g SimCLR (10) with d 128 and 4096). Characterizing the optimal solution of Lunis for any n > d + is still an open problem 5 but theoretical guarantees can be obtained in the limit case n + cc.
Theorem 5. (Asymptotical Optimality) When the number of samples infinite + co, then for any perfectly aligned encoder f F that minimizes L Lunif, the centroids μt for F p(T) are uniformly distributed on the hypersphere Sd-1, Proof in Appendix E.2
Empirically, we observe that minimizers f of Limiss remain well-aligned when d + on real- world vision datasets (see Appendix [A.I]. Decoupled uniformity thus optimizes two properties that are nicely correlated with downstream classification performance (57)-that is alignment and uniformity between centroids. However, noted in (58) (50,, optimizing these two properties is necessary but not sufficient to guarantee good classification accuracy. In fact, the accuracy can be arbitrarily bad even for perfectly aligned and uniform encoders (50).
B.2 A METRIC LEARNING POINT-OR-VIEW
In this section, we provide geometrical understanding of Decoupled Uniformity loss from metric learning point of view. In particular, we consider the Log-Sum-Exp (LSE) operator often used in CL as an approximation of the maximum.
We consider the finite-samples case with n original samples (T:)sE(1...)) sid p(z) and V views
(x)") (²) DEE[L. A(-12,) for each sample Z, We make an abuse of notations and set bs = I ΣV (() Then we have:
V
Ed = log 1
Σ
exp (-11m4 -1y113)
n(n 1)
itj
(5)
1
=log n(n 1)
Σ exp (-s 5 + 28
18

---

Under review as a conference paper at ICLR 2023
where 5f" (f(), is viewed as similarity measure.
||
Eew
s(a) s(a)")x/))) si En s(x, af)") and s(.-))
From a metric learning point-of-viest looking for an encoder / such that the sum of similarities of all views from the same anchor (s and sz) are higher than the sum similarities between views from different instances (s,y):
shall see that minimizing Eq. is (almost) equivalent to
8 + 5J > 28 + Vi デ
(6)
where a margin that we suppose "very big" (see hereafter). Indeed, this inequality is equivalent to 23, of $j for all i * j. which can be written as:
arg min max(-6. (2s,g s 8 (e(e)))))ss)
This can be transformed into an optimization problem using the LSE (log-sum-exp) approximation of the max operator:
arg min min exp(-e) Σ (f)
exp (-s -3 25
Thus, if we use an infinite margin (lim,-+o)) retrieve exactly our optimization problem with Decoupled Uniformity in (up to an additional constant depending on n).
C ADDITIONAL GENERAL GUARANTEES ON DOWNSTREAM CLASSIFICATION
C.I OPTIMAL CONFIGURATION OF SUPERVISED LOSS
In order to derive guarantees downstream classification task D when optimizing unsuper- vised decoupled uniformity loss, we define supervised loss that measures the risk on downstream supervised task. We prove in the next section that the minimizers of this loss have the same geometry as the ones minimizing cross- coos-eentoppy and SupCon (37): a regular simplex on the hyper-sphere (25). More formally, have:
Lemma Let a downstream task with C classes. We assume that C S + (i.e., a big enough representation space), that all classes are balanced and the realizability of an encoder f* arg min Leup(f) with (fu log the optimal centroids (Hi)yey associated to f make regular simplex on the hypersphere S and they are perfectly linearly separable, i.c min E(R.)))D((Wy m < 0) = Proof in the next section.
- and My (p Then
This property notably implies that we can realize 100% accuracy at optima with linear evaluation (taking the linear classifier g = W f (x) with W = (Pi)EE E RCxd).
C.2 GENERAL GUARANTEES OF DECOUPLED UNIFORMITY
In its most general formulation, we tightly bound the previous supervised loss by Decoupled Uni- formity loss / depending on variance term of the centroids μt, conditionally to the labels: Theorem 7. (Guarantees for given downstream task) For any f E F and augmentation AA we have:
Canis(l) S (f) S Σ Var(usly) Led (f) & 4fF poaaaoee uy Cumer()) (/) (7)
j=1
where Var(w (y) Ep(z)p) (ng component of Mgg EA(M aa) (xx). Proof in the section.
1p(=)))) m )2, y arg max Var(ualv') and pe is the j-th
Intuitively, it means that will achieve good accuracy if all centroids (uz Ja for samples ø X in the same class are not too far. This theorem is very general since we do not require the intra-class connectivity assumption A; any A A can be used.
19

---

Under review as a conference paper at ICLR 2023
D EXPERIMENTAL DETAILS
Code will be released upon acceptance of the manuscript. We provide detailed pseudo-code of our algorithm as well as all experimental details to reproduce the experiments run the manuscript.
D.1 PSEUDO-CODE
Algorithm Pseudo-code of the algorithm
Require: Batch of images (f1,-.., F..) X, augmentation distribution A, temperature i, hyper- parameter À for centroid estimation
K (K(,,, Fz.) F.))),
Compute the kernel matrix b Compute weights for centroid estimation
(Kn n Kn
(1) 2i 3.. Ri
(V) iid A(-(aa)
Sammle V views per image
F t ( Lv f(z;" ))Epp..)))
B. Compute the averaged image representations Centroid estimation
+ af
Ed Lumis + log n[n-1) Eizs exp(-tili: 2/ll3)
Kernel Decoupled Uniformity loss
return Cd unis
D.2 IMPLEMENTATION IN PYTORCH
We provide a PyTorch implementation of previous pseudo-code in Algorithm 2 It is generalizable an arbitrary number of views and kernel.
Algorithm Implementation in PyTorch
loader: generator of images
# batch size
n views: number views
d: latent space dimension
f: encoder (with projection head)
x: Tensor of shape [n, *]
aug: augmentation module gencrating views
kernel defined on image space
lamb: papar-trr to estimate centroids
10
for x in loader:
alphas = (K x) + w...mmxtorch.eeye eye (n) .inverse (x, x)
aug ix, n_views)
shape=
*]
13
z (x) view [n, n views, dI) shape=In, views, di
14
mu alphas detach () 2 z.mean (dim=l) # shape- (n, d] loss L(mu)
15
loss. (ossbbackward
18
def (mu,, t=2): return torch. pdist (mu, p=2) .po (2) .mul (-t) .exp () mean () .1og ()
19
20
D.3 DATASETS
CIFAR (41)
We use the original training/test split with 50000 and 10000 images respectively of
size 32 32.
STL-10 166
In unsupervised pre-training,
all labelled+unlabellde images (105000 images)
for training and the remaining 8000 for test with size 96 x 96. During linear evaluation, we only use the 5000 training labelled images for learning the weights.
20

---

Under review as a conference paper at ICLR 2023
This dataset is composed of 200 fine- fine-grained bird species with 5994 training images and 5794 test images rescaled to 224 x 224.
CUB200-2011 (56)
UTZappos (2) This dataset is composed of images of shoes from zappos.com. In order to be comparable with the literature on weakly supervised learning. we follow (55) and split it into 35017 training images and 15008 test images resized at 32 x 32.
ImageNet100 17 54) It is a subset of ImageNet containing 100 random classes and introduced
in (54). It contains 126689 training images and 5000 testing images rescaled 224 x 224. It notably allows reasonable computational time since we runt all our experiments a single server node
with 4 V100 GPU.
BHB (0) This dataset is composed of 10420 3D brain MRI images of size 121 x 145 x 121 with 1.5mm spatial resolution. Only healthy subjects are included.
BIOBD (32) It is also brain MRI dataset including 662 3D anatomical images and used for downstream classification. Each 3D volume has size 121 x 145 x 121. contains 306 patients with bipolar disorder vs 356 healthy controls and we aim at discriminating patients vs controls. It is particularly suited to investigate biomarkers discovery inside the brain (31
CheXpert (35) This dataset is composed of 224 316 chest radiogaphs of 65240 patients. Each radiograph comes with 14 medical obervations. We use the official training set for our experi- ments, following (34 335 and we test the models on the hold-out official validation split containing radiographs from 200 patients. For linear evaluation on this dataset, we train 5 linear probes to dis- criminate 5 pathologies (as binary classification) using only the radiographs with "certain" labels.
D.4 CONTRASTIVE MODELS
Architecture. For all small-scale vision datasets (CIFAR-10 10 411, CIFAR-100 (41). STL- (6.
CUB200-2011 (56) and UT-Zappos (62)) and CheXpert, we used official ResNet18 (30) backbone where we replaced the first 7 7 convolutional kernel by smaller x 3 kernel and removed the first max-pooling layer for CIFAR-10, CIFAR-100 and UTZappos. For ImageNet100, we used ResNet50 (30 for stronger baselines it is common in the literature. For medical images
brain MRI datasets (BHB (20) and BIOBD 32), we used DenseNet121 (33) as our default backbone encoder, following previous literature on these datasets (20). We use the official
Following (0), we use the representation space after the last average pooling layer with 2048 dimen- sions to perform linear evaluation and use 2-layers MLP projection head with batch normalization between each layer for a final latent space with 128 dimensions.
Kernel choice. In all experiments with Kernel Decoupled Uniformity, we used an RBF kernel and we cross- cross- validated the hyperparameter a within {0.1, 1, 10, 30, 50, 100).
Batch size. We always use a default batch size 256 for experiments on vision datasets and 64 for brain MRI datasets (considering the computational cost with 3D images and since it had little impact on the performance (20).
Optimization. We use SGD optimizer small-scale vision datasets (CIFAR-10. CIFAR- STL-10. CUB200-200 2011, UT-Zappos) with a base learning rate 0.3 x batch size and a cosine scheduler. For ImageNet100, we use LARS (61) optimizer with learning rate 0.02 x /batch size and cosine scheduler. In Kernel Decoupled Uniformity loss, we À Vbatch size
2. For SimCLR, we set the temperature to 0.07 for all datasets following (0)). Unless mentioned otherwise, use views for Decoupled Uniformity (both with and without kernel) and the com- putational cost remains comparable with standard contrastive models.
0.01 and
Training epochs. By default, train the models for 200 epochs, unless mentioned otherwise for all vision data-sets excepted CUB200-2011 and UTZappos where we train them for 1000 epochs, following (55) and ImageNet100 where train them for 400 epochs. For medical brain MRI
21

---

Under review as a conference paper at ICLR 2023
dataset, we perform pre-training for 50 epochs, in (20). As for CheXpert, we train all models for 400 epochs.
Augmentations. We follow (10) to define our full set of data augmentations for vision datasets in- cluding: RandomResizedCrop (uniform scale between 0.08 to 1), RandomHorizomalliin and color distorsion (including color jittering and gray-scale). For medical brain MRI dataset, we use cutout covering 25% of the image in each direction (1 /4 of the entire volume), following (20). For CheX- pert, we follow (2) and we use RandomResizedCrop (uniform scale between 0.08 1), RandomHor- izontalFlip. RandomRotation (up to 45 degrees) however we do not apply color jittering as we work with gray-scale images.
D.4.1 GENERATIVE MODELS AND GLORIA
Architecture. For VAE, we use ResNet18 backbone with a completely symmetric decoder using nearest-neighbor interpolation for up-sampling. For DCGAN, we follow the architecture described in (48). We keep the original dimension for CIFAR-10 10 and CIFAR-100 datasets and we resize the images 64 x 64 for STL-10. For BigBiGAN (9), we use the ResNet50 pre-trained encoder available at https: http/ttffhbbbbb dev features.
htw.:///bbbbioaaaaeeeeeeeett- with BN+CReLU
3 and we optimize VAE for CIFAR-10 and CIFAR-100 for 400 epochs using initial learning rate 10-4 and SGD optimizer with cosine scheduler. For RandBits experiments, the VAE is trained with the same setup as for CIFAR-IO/ on RandBits-CIFARIO For DCGAN, we optimize using Adam optimizer (following 448 and base learning rate 2 x 10-4. Importantly, all generative models are trained without data augmentation, providing fair comparison with other methods.
Training. For VAE, we use PyTorch-lightning pre-trained model for STL-10
GloRIA can encode both image and text through 2 different encoders. is pre- trained on the official training set of CheXpert, in experiments. We only GloRIA image's encoder (a ResNet18 in practice") to obtain weak labels on CheXpert and we leverage this weak labels with Kernel Decoupled Uniformity loss. In practice, use RBF kernel as in our previous experiments.
GloRIA 34
D.4.2 LINEAR EVALUATION
For all experiments, we perform linear evaluation by encoding the original training set (without augmentation) and by training logistic regression on these features. We cross-validate an & penalty term between {0. 1e 2, le - 3, le - 4, 1e - 5} for training this linear probe for 300 epochs with an initial learning rate 0.1 decayed by 0.1 at each plateau.
E PROOFS
E. ESTIMATION ERROR WITH EMPIRICAL DECOUPLED UNIFORMITY
Property Ld (f) fulfills (6d (()ns(() Ld (a)s s O
with a convergence in law. (i
PROOF. For any x E X, since f(x) E Sd-1, then ||u=|| = ||E 2y f(x)] S Ex = 1.
As result, e-llms-Hee! E def -4-, for any F, X' E X. Since log is k-Lipschitz on / then:
Rano ( Ld Lanss (f) S n(n
IlMa, Mayll El
"The official model is available http://///////..i.ooooo//aaaaaan
22

---

Under review as a conference paper at ICLR 2023
For fixed E E R, let (n) 1 +E== e -l and g(Ø) Ep 10 Since (Z)) (e (c-lluz-us g(()) xE(l..m]
iid with bounded support in [-2, 2] and zero
mean then by Berry-Essees theorem we have In. (.) ((ø) - O (+) Similarly, (e().n (X,) 9( (z)) are iid, bounded in [- and with zero mean. So | -i= (T:)
9m (E) I 10 (1)))) by Berry- Esseen theorem. Then we have:
(uuns(() (f) - Limms (f)l & kl
(z,) E)(a)o(T))
(n 1)n i=1
2k -
In (3,) - (pp()) E - [x))
1=1
1
0 1
0 +
0
E.2 OPTIMALITY OF DECOUPLED UNIFORMITY
Theorem 1. (Optimality of Decoupled Uniformity) Given n points (Z)))e(t...)) such that S d+ 1, the optimal decoupled uniformity loss is reached when:
1. (Perfect uniformity) All centroids (14))E(1...)) (Mz,)<E(L..) make regular simplex the hyper-sphere Sd-1
2. (Perfect alignment) f is perfectly aligned, i.e Vx, x' idd A(- = f(x))
PROOF. We will use Jensen's ' inequality and basic algebra show these 2 properties. By triangular inequality, we have || = [|E==A((2)) & [x] = since we assume f(x) E Sd. So all (1s) are bounded by 1.
Let ().)E We have:
r(w)
Σ
||m 11112
Σ 11mal12 + 11111112 2/c Pd
1-11
i,
<
(2 - 2/1s 14))
2n
21 Σ uill? < 2n
with equality if and only if E M4 0nd Vi [1..m]: ||mil|
By strict convexity of - eu
we have:
r(u)
Σ (op(-11u-111111) n(n exe
'nnn (Tn(m- - 1)
2n
2 n(n - exp
with equality if and only if all pairwise distance 1|ا4
Mull equal (equality case in Jensen's
inequality for strict convex function). Σn= 44 = and || = 1. So all centroids must form regular n - 1-simplex inscribed on the hypersphere 1 centered at 0.
Finally, since || = 1 then we have equality in the Jensen's 's inequality || = ||EAc < EA (x))] = 1. Since || || is strictly convex on the hyper-sphere, then ff must be constant on supp A(- [.a)), for all 7; so of must be perfectly aligned.
23

---

Under review as a conference paper at ICLR 2023
Theorem 5. (Asymptotical Optimality) When the number of samples is infinite oo, then for any perfectly aligned encoder f F that minimizes L Lumiy. the centroids μı for x p(z) are uniformly distributed on the hypersphere Sd-1,
PROOF. Let f E F perfectly aligned. Then all centroids Ma: f(z) lie the hypersphere Sd-1 and we are optimizing:
arg min Cunis (f) arg min E
-
2, p, p(x)
So direct application of Proposition 1. in (57) shows that the uniform distribution on S is the unique solution to this problem and that all centroids are uniformly distributed on the hyper-sphere.
E.3 OPTIMALITY OF SUPERVISED LOsS
Lemma 6. Let downstream task D with C classes. We assume that C S d + (i.e. big enough representation space), that all classes are balanced and the realizability of an encoder f arg min (f) with (Sa log Ey. <us'"p(())p(/)) petp-//pep/eelll----/---------- 11 and My En( Then the optimal centroids (MG)vEy associated to f* make regular simplex on the hypersphere Sd-1 and they are perfectly linearly separable, i.e (i)yee)) ((E.))>D( my < 0) = O.
PROOF. This proof is very similar to the one in Theorem We first notice that all "labelled" centroids My = Epta Mh are bounded by (lla)l < p( Ats = 1 by Jensen's inequality applied twice). Then, since all classes balanced, re-write the supervised loss as:
Csup(s) log C
Σ
--llay-- - uy
B-y'=1
We have:
C
Ty(m) = Σ
11 13=
11mll12 + 11 2p Py
9,y==1
V.v
< > (2 24p Hyy)
2C 2 Σ Myll < 202
with equality if and only if Σg My = and Vy E (1..C),lla)) = 1. By strict convexity of + e e we have:
Ty(p)
Σ exp(( - -1 -CC- - ex "
(-c(c-1)
2C
C(C -1 11C-1)expp (
with equality if and only if all pairwise distance ||, My || are equal (equality case in Jensen's inequality for strict convex function), E My 0 and ||m||| || = 1. So all centroids must form regular& C - 1-simplex inscribed the hypersphere gd-1 centered at 0. Furthermore, since ||mil = then we have equality in the Jensen's 's inequality ||w사/| = ((Ep())))((taa)(()))) s ((1 = 1 so f must by perfeetly aligned for all samples belonging to the same class: Vē, ~ p(-l)) f(3)).
24

---

Under review as a conference paper at ICLR 2023
E.4 GENERALIZATION BOUNDS FOR DECOUPLED UNIFORMITY
Theorem 7. (Guarantees for given downstream task) For any / E F and augmentation distribution
A, have:
d
(fn S Lunss (J) = 2
Σ
Var((22l)) + (mme(()) S 4fF pe - Mtx 11-a-1++2 Ld fumis (f) (8)
==1
where Var(u = Ep(2) (ng
p po and ne is the j-th component μt [A(x)E)
PROOF.
Lower bound. To derive the lower bound, we apply Jensen's inequality to convex function u
e:
exp (fn (f) = (p(a))(21)) marll
= ExGewweee -
S Eptu)p(r)) exp (-Ep(F)a)p(e syllez pz-11
Then, by Jensen's inequality applied to 11-12:
@ EpcE | + +E E 2ph My
[x0))))((=)1>)11 - -a
|| #x 1| || 1/ 24y Fy
|| 14y 1|
(1) follows according to the previous lemma. So we can conclude:
exp Cd. ( & Ep(x)r(o') exp(-Ilı, - By = exp Lumes
Upper bound. For this bound, we will use the following equality (by definition of variance):
(| // || + Encmmllu=
Var(uglu) + EpoMlala
So we start by expending:
11ly My _ || M /[ppe/))/>11 2F pt ue
Var(uelv) + Var(uggll)) 2E pooliipeer Wy pa
Eoll -
Σ Var(u
So by applying again Jensen's inequality:
a
)
exp suu = E Ep(uip(s)) exp(-1l1e - 11 < Eppipper exp
(Epprtoeeeeiiillle - Mx 11 + Σ
Var (22a
d
< exp2
Var (ng (mm)
Epciniter) exp (- --Eoctooooce//r>)lle ux
exp
Var Var(csh (mm)
exp Liunss
25

---

Under review as a conference paper at ICLR 2023
We set Vm arg JE(1.dx) Var(ugly) We conclude here by taking the log the previous inequality.
Variance upper bound.
Starting from the definition of conditional variance:
Var (pgg Mm)
E ((=t)))ll || )
= p() (Qllæll p/ Ma + p| wn.. M Marll)) (D)
Sp(tlmm)llee pp Mar (Il + 11 (al
(2)
< 2F M(Blm))llu - Ep(slum))ar!
2F (plel
par
(1) Follows from standard inequality |la bll ||||| ||b||| (from
(2) follows
from boundness of || < and Jensen's inequality. (3) is again Jensen's 's inequality.
E.5 GENERALIZATION BOUND UNDER INTRA-CLAN CONNECTIVITY ASSUMPTION
Theorem 2. Assuming then for any e-weak aligned encoder f F:F
(fnis()) S. Lunes (f) S 8De + Cumis (f) Where D the maximum diameter of all intra-class graphs Gy (y/ E }).
(9)
PROOF. Let E ) and I, p(zly)p(3"()) By Assumption 1 it exists path of length p S D connecting (x, X') Gy- So it exists (d=))e[L.pP1] X and (z: (T:)sel..pee" s.t Vi E [1..p, 24 ~
A(xs(T7s) O A(m 1T+1), I1 = F and Tp+1 = Then:
| u = || Mayll
Mam Malll
|| - Mayll
Σ ||Ma - f(xx) + f(xx)) - -Ma,ll
||Mann - -(x))1++11[(1) +11(xx;) Ms.
F Fp(e(( (1 fx;))1 (pte)))) s(x)]
W
(e + 2ep < 2eD
(1) follows from Jensen's 's inequality and by definition of μg. (2) follows because f is e-weak aligned and x; A(x (ā.) A(z,(2411.
So we have
||4 Mall < 2eD and we can conclude by Theorem 7 (right inequality).
E.6 CONDITIONAL MEAN EMBEDDING ESTIMATION
Let F fixed.
26

---

Under review as a conference paper at ICLR 2023
Theorem 3. (Conditional Mean Embedding estimation) We assume that Vg E Hx. Snan)g(r))
Hx.. Let [(11, 31 ) ... (In, Fn)] iid samples from p(z[T)p([T). Let Φ, = [d( ... 1 [Tn)) and
, (f(3))m (Tx)) )]a An estimator of the conditional mean embedding is:
Va i
Σ (c())
(10)
where (x( = [0=][(""'0 [ E= + + "lu H( converges to μa with the l norm rate O for x 0 (1+)))
PROOF. Let my p( (f(), f()) Hx be the conditional mean embedding operator. Ac- cording Theorem 6 in (52) and the assumption Vig E Hxx (x(=)-1((2) Hxx this estimator can be approximated by:
mi
(F)
with ag defined previously in the theorem. This estimator converges with RKHS norm to my at rate O I + A). So we need to link ma, mg with pay fgg. We have:
Vnd
(max mxx Hx
EmielEy (F))
Σ (E)(()))))))
Hx
n
Σ ax( (e)) f((x))
(Mz, P*)R=
(1) holds by the reproducing property of kernel Kx in Hx. We can similarly obtain:
|| 12x (E ( (()), Hx
(1) (Eppoaaa) F (c))
= [ = |
Again, (1) by reproducing property of Kx. And finally:
|| Paixx
(1))((() Σ a.())((a)))()))
Hx
Σ (f)
i
By pooling these 3 equalities, we have:
|| mel = ||m 12 + ||m 2(m ma)
= =1|pz|1 + || 2(4 Pu)
||ا| 1
We can conclude since ||m mæll < 0() + (n2) 1/2).
27

---

Under review as a conference paper at ICLR 2023
E.7 GENERALIZATION BOUND UNDER EXTENDED INTRA-CLASS CONNECTIVITY HYPOTHESIS
Theorem, Assuming and holds for reproducible kernel Kx and augmentation distribution A. Let F e'aaligned. Let (I....) be samples iid drawn from p(ž). We have:
(fm I unns (f) S Cumis (f) 4D(26' (K x)) + O(n 114)
(11)
where Bn (kx) (Amio(ku) VnA)-1 O(1) for x ( K, ( je
Vn
D is the maximal diameter for all Gy. y y. We noted Amin (.,) is the minimal eigenvalue of Kn.
PROOF Let y E y and T. p(Cly))(F'lv) By Assumption 2 it exists path of length p I connecting ž, I' G. it exists (d.)EE1.p+1) E X and (u;)ier E s. Vi E L, u, ~ max(K((,)) ūz). ((U4+1, (y+1)) (ū uz+1) S e with (T, J) partition of [].p). Furthermore, th = I and up = F. As result, we have:
A(uglūs) n A(u, (1+1)) and Vj
Mg M ||
l - Ma,ll
Mū+t1 Maal
i=1
| Mu.l
| - Mú, |l +
Paaz1 Mayll
iEr
JEJ
Edges in E. As in proof of Theorem 2 we use the e'-alignment of f to derive bound:
Σ
|aa 11 Mnn
|Maa11 [(u)) [us) - m0,,
Matt1 ((w+)l++1ff(u))) paill
)l f(u)ll Eoom)llf f(u)l
RET
+e')= =26==
SET
(1) holds by Jensen's inequality and (2) because f is e' e'aaligned.
Edges in Ex For this bound, we will use Theorem 3 to approximate μs and then derive bound from the property of Gi Let (#*)ke(L.n)
p(xx(==*) n samples iid. By Theorem 3 we know that, for all j E J, Paa converges to Ha, with & norm at rate O(n-1/4) where Nu,, = Ekiz ar2 K Kx x (z) and or21 = [[K, + +nI,,) -k... As a result, for any j E J,
we have:
l|May41 - Mū, ||Mayen Payan + Mayen - Pay + Pag Mayll
(1) S
< ||aay
- M + ||βay4 faa, || + llβa,
Payll
0
+
n1/4
Where (1) holds by Theorem
Then will need the following lemma conclude:
28

---

Under review as a conference paper at ICLR 2023
Lemma. For any a, b, E ((ax((((,,),K K(a,b) [K(a, c) K(b, c)l for any reproducible kernel K.
PROOF. Let a, c E X. We consider the distance d(x, v) = K(x, x)) + K (y, (y.z) y) (x, y) (it a distance since K is a reproducible kernel so it can be expressed as ((,,)) ((()),(())) We will distinguish two cases.
Case 1. We assume K(a,c) 2 K(b,c). c). We have the following triangular inequality:
d(a. b) + d(), 2 d(b,c)
((a,b) K(b,b) - 2K(, -(o K(a,a) - K(a 2 -(b,)) 2 So max(K a), K(b, b)) K(a,b) ((, K(b,))
2 K(bbb) K(c,))- ((0cc-2KK) (k
Case 2. We assume K(b, c) 2 K(a, c). We apply symmetrically the triangular inequality:
d(a. b) + d( c) 2 d(a, c)
((b K(a, b) 2 K(bb c) K(a,e) 20 So max(K a), ((b,)bbb)) 6)) (a. b) 2 K(a, c) (b,, c) concluding the proof. Then, by definition of pay;
|: - Payll = |1
Σ
Σ k==1
an (Z), fa)(l
k===1
= ||AC||
Where A = (Ei=n (x)) E Rdxx (f()) is the i-th component of f()) and C (K(T,, (x+1) - K(Ti, a,)) E Ruxil So, using the property of spectral lg norm we have:
|| - -u, ||AC|| S ||A|22|C||2
Using the previous lemma and because (ūy, 1)+1) E EK. we have: ||C = Σブ= (K 8y-1) K ((;,, ū;)))² $ Σ (max(R) ( (Mj+1, 1, K (ūz, dz))) K(ūj, u)+1))² S ne" To conclude, we will prove that ||Al]2 ||a||2 where (ais E(EL..1. For any E R", we have:
2
(1)
n
(2) = ||au| S |||||||0|
||Au||a 2 =
Σ k-j=1
and "
Σ
k-==1
Where (1) holds with Cauchy- Schwar- inequality and because f(-) s and (2) holds by defini- tion of spectral f2 norm. So we have Yo E Rd, ||ll s ||a|||||u||, showing that ||A]]2 S ||a|22
So conclude that:
Σ || - Ma,ll < Σ JEJ
(Vāl((K, + AnI,)) 112c O(n-1/4)) = |J]II(,, + nAI,) - -1ll-1ll Onn
JEJ
We set (x(Kx) = Vkil(Kn + AmI,))
In order to see that (K,,) (AminiKn) + V
with Amin (Kn) 0 the minimum eigenvalue of Kn, we apply the spectral theorem on the sym- metric definite-positive kernel matrix Kn. Let 0 < A < d ... < An the eigenvalues of K,, According to the spectral theorem, it exists U an unitary matrix such that K2 U with D = diag(A1,..., ).)) So, by definition of spectral norm:
II(Kn + x = Amaz (U(D + nAI,) -U XnL,)-
Amoz (UDUT)
(A1 + nd)
29

---

Under review as a conference paper at ICLR 2023
where D diag( (A1 An (Antnx)z) So conclude that (,,,K)) -12h + -nd)=1=
((1) for = 0 vT
Finally, by pooling inequalities for edges over E and EK, we have:
l|ue Mall - 2c (J) + O(n 1/1) D(2d' + (n(K))c) O(
1/44)
We can conclude by plugging this inequality in Theorem 7
Theorem 4. We assume 2 and 3 hold for reproducible kernel Ke and augmentation distribu- tion A. Let (i)EEL))
Azy, T..) iid samples. Let ßæ, Li=1 j with as,
((Kn AIn)= Knnis and Kn [K ( Fy),, Fy)...e Then the empirical decoupled uniformity
loss Camis f f F:
def log m) Eij=1 exp(- -1Aaa A22 verifies, for any e'-weak aligned encoder
L 0
1 1/4
Lunes (f) S Eaner 4D(2)' +B((K)))) +0
1
(12)
n2/44
PROOF. We just need to prove that, for any f E F, IK - Lamis()) (f)1 5 O0 and we can conclude through the previous theorem. We have:
(f) Ed (f)ms(())
log n(n 1) i,j=1
- Aa,113) - Ep(a)p(2)) llke -lle-ll- e
exp( app(-1laa - Aay log 1 --1 Illsls 1ila,,-a.10 12,11
log
n(n 1) b,==1
n(n - 1)
+ log nnn2
1) 6 11,n; n2zll² ((())(()))) -1ae-
The second term last inequality is bounded by O according to property As for the first term, we use the fact that log k-Lipschitz continuous on [e 1] and exp is Lipschitz continu- ous on [44, 0) so:
log n(n 1)
Σ
- Pz. ps,1 log n(n
1 11,2; M2y
S
[|Ax; Azy -1laa,
n(n 1)
1,j=1
f,1=1
kkb
Σ
||, Pa l1ME, M1,
n(n - 1)
i+=1
Finally, we conclude using the boundness of fag and μi by a constant C:
liu. - "2,1' 122 ||ms; Ms, 1|2
As. - pnnll + llls, - Mæ, (M(l)), puz,) 11 |1us; - 12211)
$ 40(1)2. -2z,) |l l|pz, - M2,
4C11iaz - .ia;; (Mz, - -M2,)ll
40 4C(1)zz. -Mzlll + |1 112
0
114
30


Reviews:
[{"review_id": "gufxBC1wpW", "reviewer": "Reviewer_khWQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a contrastive learning method that generalizes the previous CL loss in a novel kernel-based prototypical loss uniform loss form, and integrates kernel methods for positive distribution modeling with generative models to show the effectiveness in the case where crop and coloring augmentations are not used. The authors provide theoretical analysis to show the proposed loss is able to bound the downstream classification loss. The authors conduct the empirical studies on both natural images like CIFAR10/100, STL, ImageNet100 and medical images. ", "summary_of_the_review": "The problem pointed out in this paper is important in the study of contrastive learning. The proposed method is also simple and shows improvements in the case when crop and coloring are not applied. However, some key perspectives regarding the proposed method are not comprehensively studied and thus remain unclear to the readers, which weakens the contribution of this paper, and I prefer to vote this paper below the acceptance threshold. ", "strength_and_weaknesses": "**Strength**\n\nThe paper is well-written and organized, making it easy to follow. The proposed loss is simple and is well presented in a mathematical way. Following the theoretical results, some interpretations are provided to help the readers to understand. The authors leverage the kernel graph theory to generalize the positive sample distribution and is empirically justified to be good when crop and coloring augmentation is not applied. From the empirical perspective, the authors have shown their main experiments reasonably, and the details included should be sufficient for reproducibility. \n\n**Weakness**\n\nThe paper uses several paragraphs to discuss the weak and strong augmentation, but the definition of weak and strong is heuristic, and it seems the crop and coloring augmentations are empirically regarded as strong augmentations regardless of the dataset and the data distribution. \n\nAlthough formulated as a general method, the proposed method is still justified on image datasets. For some other data domains like graph, the effectiveness of this method is unknown. \n\nAs the method involves a kernel estimation in calculating the centroid, it introduces a new hyperparameter $\\lambda$, but I did not find the study regarding the effect of this hyper-parameter in the experiment (please correct me if I missed out this part). Moreover, the choice of the kernel is also a part that needs further tuning in practical usage.\n\nThe downstream task is only assumed to be classification, but there could exist various other options. When using generative models, it requires additional training cost if there are no applicable pre-trained models.\n\nFrom the perspective of empirical studies, the improvements only appear in the case when crop and coloring are not applied; while combining these two augmentations, the proposed method underperforms previous methods. \n\nThe baselines of multi-view experiments are relatively weak. If compared with stronger methods that use multiple positives (SWAV, Dino, CACR, F-CL), there is a significant performance gap. \n\n(SWAV: https://arxiv.org/abs/2006.09882\nDino: https://arxiv.org/abs/2104.14294\nCACR: https://arxiv.org/abs/2105.03746\nF-CL: https://arxiv.org/abs/2011.11765\n)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Z-sufNnCiU7", "reviewer": "Reviewer_rmip", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Based on the perspective of Decoupled Uniformity, the authors propose a new loss and new perspective in contrastive learning. Introducing centroid based loss opens up a couple of interesting results like  1) the loss leads to uniformity and alignment (the idea of uniformity and alignment is from earlier paper), 2) the loss could be used to bound the classification loss (related to downstream task) and 3) exploiting the connection to kernel methods and mean embedding, prior information can be incorporated to design better contrastive learning. Using the pretrained VAE and GANs authors show how their contrastive learning is more robust and accurate by exploiting the kernel formulation", "summary_of_the_review": "The paper is well presented and contains enough high quality results. So, I recommend accept. There are some minor concerns. I hope authors can clarify those.", "strength_and_weaknesses": "Overall, the paper is quite interesting and is presented in a complete and rigorous manner. The paper develops everything is a very principled way, which is one good feature of this paper. Theorems are insightful, supports intuition and lead to an empirically powerful algorithm. Overall, very nice paper. \n\nThe only weakness, perhaps, from my perspective is the following. Contrastive learning is supposed to be a method of representation learning. If we need yet another representation learning method like VAE or GAN, then it looks quite a lot of work just to do classification. I understand that the authors tested their method on downstream task of classification because that is what is shown in the literature. However, it would improve the paper a lot if the authors could show the importance of features learnt from this kind of contrastive learning are far superior to both the VAE  and regular contrastive learning. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IfCc6QL3Ae", "reviewer": "Reviewer_YGqm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the setting of contrastive learning the authors proposed to define positive samples using the kernel theory, and  introduced a new loss function called the decoupled uniformity loss. In addition, the authors suggested to exploit the prior information learned from generative models or given as auxiliary attributes to make it less dependent on data augmentation. Theoretical analysis regarding the downstream classification loss was provided. Experiments were implemented in both unsupervised and weak-supervised settings.  ", "summary_of_the_review": "The authors introduced a new loss function, re-defined the positive samples via the kernel, and suggested to boost the performance with prior knowledge. However, as mentioned in the Weaknesses part, \n\n. the introduced assumptions need to be further validated; and\n\n. the proposed method shows no advantage compared to the SOTA under the same condition (ie, without the prior information). When using the prior information, the proposed method essentially needs to learn two representation models based on each dataset. So, the comparison is a bit unfair. ", "strength_and_weaknesses": "Strength:\n\n A new loss function, the decoupled uniformity loss, is provided and further validated theoretically. Prior information is exploited to boost the CL performance.\nBoth the theoretical and empirical results have been provided regarding the downstream performance. \n\n#################\n\nWeaknesses:\n\nLoss function: The proposed Decoupled Uniformity loss extends the uniformity loss in [54] and does not explicitly include the alignment loss in [54], which is elegant. The authors claimed that the proposed loss function can implicitly encourage alignment (also showed theoretically in Theorem 1 with insufficient training samples). The reviewer wonder what if you add the alignment loss similar to [54]. Would that increase or degrade the performance in practice, ie., when the number of the training samples is neither too low nor infinity? As shown in the experiment, without the prior information, the proposed method cannot beat the SOTA. \n\n\nAssumptions: The authors introduced the Weak-aligned encoder, which is claimed to be weaker than previous assumptions such as L-smoothness. In the implementation, how to ensure this assumption is satisfied?\n\nAssumptions: How to ensure Assumption 2 in implementation?\n\nDefinition 3.7: How to find the value of \\lambda?\n\n\nExperiments: \n\n. From the experimental results, without the prior information (the same as the benchmarks) the proposed method has no advantage compared to the SOTA. The advantage only shows when using the prior knowledge. Such comparison is a bit unfair, because in this case the proposed method essentially requires two representation models learned based on each dataset, ie., VAE/GAN + CL. Such extra complexity and cost need to be considered. \n\n. Kernel quality is crucial to the downstream performance as shown experimentally in Appendix Fig. 4. In the implementation, how to choose a proper kernel with optimal parameters? Are these parameters jointly learned or picked via the validation sets?\n\n. In Table 2, the performance of the proposed method is shown with 4 views. Do the benchmarks also use 4 views? If not, please provide results with 2 views for the fair comparison.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FZmPMD3aYoH", "reviewer": "Reviewer_6Wk9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The success of contrastive learning has been heavily relying on the quality of data augmentations.\nThis paper tries to reduce this reliance by augmenting the augmentation graph with some other \"kernel graph\", where the kernel is defined with prior knowledge, such as features of generative models or attributes of data points. The paper also proposes a \"decoupled uniformity loss\" to allow incorporating such kernel.\n\nThe paper theoretically justify that the inclusion of such kernels can relax the assumption on data augmentation, and empirically show that adding such a kernel helps to improve contrastive methods' performance especially on poor augmentation. The quality of the kernel (e.g. features from $\\beta$-VAE vs from DCGAN) affects the accuracy as expected.", "summary_of_the_review": "This paper proposes to use prior information to make contrastive learning more robust to the choice of data augmentation.\nI find the overall message interesting, but believe that the paper could be strengthened with more carefully conducted experiments and better organization.\n", "strength_and_weaknesses": "Strengths:\n- I find the idea of incorporating prior knowledge in form of kernels to be reasonable (\"unsurprising\"), yet still interesting.\n- The empirical results adds one more piece of evidence that leveraging generative models in contrastive representation learning may be promising.\n\nConcerns and questions\n- The empirical comparison could be done more thoroughly, and the design choices could be better justified.\n  - Table 2: what accuracy does SimCLR achieve?\n  - What's the reason for using BigBiGAN for Table 2, but DCGAN for Table 3?\n- The paper should cite and compare with \"Can contrastive learning avoid shortcut solutions?\" (Robinson et al. 2021)\n  -  Also in Sec 3, the comment on InfoNCE that the \"extension to multiple views is not straightforward\": the paper \"Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Framework\" seems related.\n- Table 3: the comparison between 1) the gap between \"w/o Color\" and \"All\" and 2) the gap between (\"w/o Color and Crop\" and \"w/o Color\") seems to vary across setups. Did you find any consistent trend, e.g. a particular type of generative model is more robust to the absence of certain type of augmentation?\n- Improvement over generative features may be marginal: the proposed framework can be viewed equivalently as using generative features to improve contrastive learning, or using contrastive learning to improve feature quality of generative modeling. The latter view may be more appropriate sometimes, since the generative features may already achieve competitive accuracy.\n    - Table 2: the BigBiGAN features themselves already achieve good accuracy, so the accuracy gain from training on decoupled uniformity loss seems marginal.\n    - Table 3, STL-10: when there is no augmentation, DCGAN is better than Decoupled loss+DCGAN (do you know why?).\n- How to choose the kernel in practice? Related to the previous point, it seems that if the kernel comes from a generative model, then the quality of the generative model's feature is strongly correlated with the quality of the kernel; can we use contrastive learning and generative model together, either training jointly or iteratively boosting each other?\n\nClarifications/comments:\n- \"...we see that Decoupled Uniformity can realize both perfect alignment and uniformity, contrary to InfoNCE\": why can't InfoNCE realize both?\n- The comments under Thm 3 says it \"provides tighter lower bound\"; how is Thm 3 a lower bound?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject