Contrastive Learning with Consistent Representations
Authors: Zihu Wang
Yu Wang
Zhuotong Chen
Hanbin Hu
Peng Li
Venue: Accepted by TMLR ()

Abstract: Contrastive learning demonstrates great promise for representation learning. Data augmentations play a critical role in contrastive learning by providing informative views of the data without necessitating explicit labels. Nonetheless, the efficacy of current methodologies heavily hinges on the quality of employed data augmentation (DA) functions, often chosen manually from a limited set of options. While exploiting diverse data augmentations is appealing, the complexities inherent in both DAs and representation learning can lead to performance deterioration. Addressing this challenge and facilitating the systematic incorporation of diverse data augmentations, this paper proposes Contrastive Learning with Consistent Representations (CoCor). At the heart of CoCor is a novel consistency metric termed DA consistency. This metric governs the mapping of augmented input data to the representation space. Moreover, we propose to learn the optimal mapping locations as a function of DA. Experimental results demonstrate that CoCor notably enhances the generalizability and transferability of learned representations in comparison to baseline methods. The implementation of CoCor can be found at https://github.com/zihuwang97/CoCor.

Published in Transactions on Machine Learning Research (09/2024)
Contrastive Learning with Consistent Representations
Zihu Wang
zibu wong@ucsb.ect edu
Department of Electrical and Computer Engineering University of California, Santa Barbora
Yu Wang
yu950ucsb. edu
Deportment Electrical and Computer Engineering University of California, Santa Barbara
Zhuotong Chen
stchen@ucsb.. edu
Department of Electrical and Computer Engineering University of California, Santa Barbara
Hanbin Hu
hambinhu@uucsh edu
Depariment of Electrical and Computer Engineering University of California, Santa Barbara
Peng Li
lpprucsb. edu
Department of Electricol and Computer Engineering University of California, Santa Barbara
Reviewed on OpenReview: https: // openrevee.... net/ formf? id- gKeSI8w632
Abstract
Contrastive learning demonstrates great promise for representation learning, Data augmen- tations play critical role in contrastive lcarning by providing informative views of the data without necessitating explicit labels. Nonetheless, the elficacy of current. methodolo- gies heavily hinges on the quality of employed data augmentation (DA) functions, often chosen manually from limited set options. While exploiting diverse data augmentations appealing, the complexities inherent in both DAs and representation learning can lead to performance deterioration. Addressing this challenge and facilitating the systematic in- corporation of diverse data augmentations, this paper proposes Contrastive Learning with Consistent Representations (CoCor). At the heart of CoCor is novel consistency metric termed DA consistency. This metric governs the mapping of augmented input. data to the representation space. Morcover, we propose to learn the optimal mapping locations as a function DA. Experimental results demonstrate that CoCor notably enhances the gener- alizability and transferability learned representations in comparison baseline methods. The implementation of CoCor can be found at https://gitie commziluwang3//oooor
1
Introduction
Data atggeentation (DA) is widely used in supervised learning in computer vision Ho et al.
2019 Lim al.
(0019:
Cubuk et al. 22019 2020
Li & Li (2023).
achieving excellent results popular datasets Ciregan
ct al.
2012];
Sato ct al. (2015); Wan et al. (2013): Krizhevsky et al.] (017)). DA is also key component in
recent contrastive learning techniques Chen et. al. 2020a);; Tian et. al. (2020b); He et al. (2020); Chen & He
(202 Xiao et al. 2020 : Lee & Shin (002)). An encoder that learns good visual representations of the input
data is trained with a contrastive loss. The contrastive loss is characterized by the following principle: in the
feature space, two views of given data example, transformed by distinct DA functions, exhibit correlation
(similarity), whereas transformed views of different input examples manifest dissimilarity. The effectiveness

---

Published in Transactions on Machine Learning Research (09/2024)
inconsistent
consistent
Neighhory
MoCo
Query Image
MoC
CoCor
SimSiam
Query Image
SimSiam
CoCor
stronger augmentatiin
stronger augmentatiin
(
(b)
Figure (a) Left: An encoder trained with the standard contrastive loss can exhibit inconsistency, as different views of an instance are encouraged to be represented similarly in the feature space, irrespective of the actual difference between them. Right: A consistent encoder positions the vector of more strongly augmented data further away from that of the raw data. Here the rings represent points with varying similarities to the central representation vector. (b) Nearest-neighbor retrieval in the feature space on CUB- 200 Wah et al. (2011) and Flowers100 Nilsback & Zisserman (2008) using pre-trained encoders. Existing contrastive methods He et al./ (2020); Chen & He (2021), which enforce invariance to all data augmentations, may inadvertently cluster dissimilar data closely in the feature space. However, by applying consistency, CoCor ensures that only data sharing similar latent semantics are distributed closely in the latent space.
of the encoder, trained on unlabeled data, is pivotal to the overall performance contrastive learning and is contingent upon the choice of employed DAs.
To learn effective and transferable representations, numerous studies have focused enhancing contrastive learning by selecting suitable DAs. Chen et al. (2200a); Tian et al. (2000b He et al. 2020 Li et al. (2023); Wang et al. (2023); Van der Sluijs et al. (2024) have shown that combining different augmentations with appropriate intensities can improve performance. However, these works confine augmentations to a random composition of specific types within a limited intensity range. Contrastingly, research has indicated that employing diverse DAs effectively enhances the model's ability to capture the invariance of the training data, thereby boosting the model's performance Cubuk et al. (2020), transferability Lee al. (2021). and ro- bustness Lopes et al. (2019). Consequently, the exploration of diverse DAs has garnered increased attention recently. PIRL Misra & Maaten (2020) adopts additional augmentations, SwAV Caron et al. (2020) intro-
duces multiple random resized-crops to provide the encoder with broader range of data reviews. Moreover:
some recent works adopt combinations of stronger DAs in contrastive learning Tian et al. 2020b
Wang &
Qi 2021
Lee & Shin 2023).
While the idea of leveraging diverse data augmentations appealing, to date, there lacks a systematic approach for integrating a substantial number of augmentations into contrastive learning. It is crucial to note that the intricacies of data augmentation and representation learning may lead to performance degradation if augmentation functions not judiciously chosen Tian et al. (2020b); Chen et al. 2020a
Wang & Qi 2021
To lackle the aforementionedd challenges, this paper introduces Contrastive Learning with Consistent Repre- sentations (CoCor). First, we define set of composite augmentations, denoted as Ωc formed by combining multiple basic augmentations. Each composite augmentation represented using a composition vector that encodes the types and frequencies of the basic augmentations employed. This composite set Ωe facilitates the incorporation of diverse DAs, encompassing wide range of transformation types and overall intensity.
2

---

Published in Transactions on Machine Learning Research (09/2024)
Crucially, we introduce the concept of consistency for data augmentations, termed DA consistency, which imposes a desired property on the representations of views generated by various composite data augmenta- tions. Specifically, for a given input example, stronger data augmentation should move its feature space view farther from the representation of the original example. In other words, the similarity in feature space between the translormed view and the original input decreases with the strength of the data augmentation, as depicted in Figure Ta). An encoder failing to satisfy this property is considered inconsistent from a data augmentation perspective.
We propose the integration of a novel consistency loss to enforce DA consistency within the representation space of the encoder. Unlike contrastive loss, which does not account for the type or strength of the data augmentations used Chen et al. (2020a); Tian et al. (020bb;; [e et al. (2020), our consistency loss strategically guides encoder training to ensure alignment of the resultant feature space with the strength of the applied
data augmentations. As demonstrated in nearest-neighhor retrieval tasks shown in Figure [b), existing contrastive learning approaches like He et al. (2020); Chen et al. (2020b); Chen & He (2021) may cluster dissimilar views due to their agnostic to sugmentatioonindduee data variance. However, the consistency enforced by CoCor ensures proximity in the feature space only for data with similar latent features, while distinctly scparating variants of the data. Furthermore, our experimental studics affirm that maintaining DA consistency is crucial for mitigating discrepancies among diverse data augmentations, thereby preventing potential performance degradation or even dimensional collapse in the feature space Jing et. al. 20221) et al. 2022a)).
The proposed consistency loss hinges on determining the optimal similarity between the representations of input example and its transformed view, value contingent on the strength of the composite data aug- mentation. However, this optimal similarity is not known priori. To address this challenge, we adopt a data-driven approach, training a neural network to map from the composition vector of each data augmenta- tion to the desired similarity. Recognizing the monotonic relationship between similarity and augmentation strength, we introduce and enforce a monotonicity constraint on the neural network. This constraint cnsures that stronger composite data augmentations correspond to a strictly smaller valued similarity.
The proposed Contrastive Learning with Consistent Representations (CoCor) has the following contributions:
Systematically explore large set of diverse composite data augmentations to improve the encoder's performance on various downstream tasks.
Propose novel concept, DA consistency, which quantifies the monotonic dependency of latent-space similarity between input example and its transformed view on the strength of the augmentation.
Introduce a new consistency loss designed to train the encoder to satisfy DA consistency and to utilize diverse augmentations without suffering inconsistency-induee performance loss.
Train a monotonically constrained neural network for learning the optimal latent-space similarities.
With consistent representations, CoCor achieves state-of-the-art results for various downstream tasks. More- over, it can be readily integrated into existing contrastive learning frameworks, effectively imposing DA consistency the encoder.
2 Related Work
Contrastive Learning demonstrates significant potential in computer vision tasks and other domains in recent years Chen et al. (2020a): He et al. (2020); Wang et al. (2023): ; Caron et al. (2020); Wang et al. (2024). It is a common practice to utilize data augmentations in forming both positive and negative pairs of data for defining the contrastive loss Chen et al. (0200)); Oord el. al. (2018). Some mod highlight the significance of negative pairs in learning distinguishable features from the data Robinson et al. (2020): Awasthi al.] (2022). MoCo introduces a moving-averaged encoder paired with a large negative memory queue He et al. (2020): Misra & Maaten 2202)). In contrast, Chen He (2021): Grill et al. (2020) exclusively learn features from positive pairs, achieving state-of-the-art performance without incorporating negative
3

---

Published in Transactions on Machine Learning Research (09/2024)
pairs. Furthermore, supervised contrastive methods introduce annotation-related information to learn better representations ions Khosla et al. (2020]; Li et al. (202bb). Recent research endeavors aim to comprehend the effectiveness and limitations of contrastive learning via the lens of representation distribution Wang & Isola
(2200) and delve into issucs such as dimensional collapse Li et al. (2022a); Jing et al. 20211)
Data Augmentation Data augmentations applied to natural images have demonstrated efficacy in en- hancing the genera lizibiiiy and robustness of models trained in supervised learning Cubuk et al. (0019 p020 Lopes et al. He et al. 2020), careful selection of data augmentations from a limited set becomes crucial for ensuring optimal performance. Instead of treating both views in positive pairs equally, recent proposals by Zhang &
2019 Krizhevsky et al. (0117) However, in self-supervised learning Chen et al. 2020a
Ma 2022 Lee et al. [2021); Zhang et al. (002)); Devillers & Lefort 2023) suggest capturing the variance
between two views caused by the application of different random augmentations, termed augmentation-awaat
information. AugSelf et al. (2021) achieves this by training an auxiliary network to predict the difference
between augmentations applied to generate positive pairs. LoGo Zhang et al. (0022) proposes learning the differences between variously sized crops of cach image. Zhang & Ma (2022) encodes an augmented image along with the DA parameters using two networks, and combines the two embeddings to form a feature vector for contrastive loss. EquiMod Devillers & Lefort (2023) trains a network predict the representation of an augmented view from the original data and the applied DA. While methods like InfoMin [ian et al.) (2020b) and JOAO You et al. (2021) propose scarching for optimal data augmentation policies for forming pairs in contrastive loss, their empirical studies reveal that stronger and more diverse augmentations can, in fact, degrade encoder performance. Conversely, some recent works aim to explore the benefits of strong augmentations Wang & Qi (2021); Caron et al. (2020); Lee & Shin (2028). However, as of now, there is a lack of a systematic approach to leveraging diverse data augmentations while considering the impact of augmentation strength.
3 Preliminaries
The goal of contrastive representation learning is to learn encoder parameterized by θe that maps input data a e Ra to an &2 normalized feature vector of dimension m in the feature space E, i.e., (-) R² + Sm-1 The encoder is trained by minimizing contrastive loss. Specifically, it defines two different augmented versions of the same input example as a positive pair, which are expected to have similar representations in the feature space. Meanwhile, the encoder shall be trained to discriminate any two instances augmented from different input examples, i.c., a negative pair, the feature space. Minimizing the contrastive loss pulls together positive pairs and pushes apart negative pairs Oord al. (2018); Chen
et al. (020a)) Li et al. 2020
4
Method
4.1
Diverse Composite Augmentations
The proposed Contrastive Learning with Consistent Representations (CoCor) improves contrastive learning performance by exploring diverse data augmentations composed from a set of basic augmentations.
Definition 1 (Composite Data Augmentations). A composite augmentation with length of l, namely, Arl>, is defined as the composition of 1 randomly sampled augmentations from a given set of Na basic augmentation functions Ω, = i(k) E N.). = 1,2.....
,aN,(.); A = d:(1) ai2) ... a:()) where
We denote the set of all composite augmentations with a length of / by N and denote the set of all composite augmentations with different. lengths by Ω. = 2 U 2-2>... .... Our studies show that the ordering of the basic augmentations performance. To simplify the discussion, we assume that composite augmentations are order invariant, e.g., a o aj Oj a. We represent a composite augmentation with an unique composition vector as shown
a composite augmentation does not have a significant impact on
Figure 2 b).
4

---

Published in Transactions on Machine Learning Research (09/2024)
A weaker
A stronger
A2 stronger
A weaker
Basic Duta Augmentations
Sazuration
Bei tnggttnes
composite
Eaturstion
Batursticn
Saturation
Baturation
Saturation Saturaticn
Soturation
Bosuratton
Augmentation (DAA
Data
Rotato
Solazioe Rosato
Bolazris
Hotata
vectors
(2,
Baturation
DA composition Vections
Vcs 0,l
Brightnees
Sdarre
c2
6,
solartes
Besate
Encoder f()
B
Ropresentatiin co Space
Monotonic Layer
Monotonic Mapping Neural Network (MMNN) g
Langer similarity
Monotonic Layer
sloug
Smaller latent similarity
Imposes desired
Predicted
consistency g(va) 9 (Vos)
Similarity g
(a)
(b)
Figure Overview of the Proposed Method. (a) The proposed property, DA consistency, ensures that data augmented with stronger augmentation is positioned farther from the original data compared a weakly augmented view in the representation space. (b) The Monotonic Mapping Neural Network (MMNN) predicts the optimal latent similarity between an augmented view and the original data, using augmentation composition vectors as input. Stronger augmentation results in a smaller predicted latent similarity by the MMNN.
Definition 2 (Composition Vector). The composition vector of a length-l composite augmentation A,1> is defined di-iimensional vector v(A E Nº" (No is the set of all natural numbers), where the jeh entry 0, the number of times that the basic augmentation aj is applied A and Σn= vi(i] = l.
4.2 Data Augmentation Consistency
Data augmentation plays a crucial role in contrastive learning, requiring careful design to expose the latent structure of the original data. However, our key observation is that the standard contrastive learning frame- work Chen al. (2020a): He et al. (2020): Chen & He (2021) does not inherently imply data augmentation (DA) consislency, fundamental property proposed by this work. In the absence of this consistency, the encoder might simply bring views in positive pairs together in the feature space Z C Sm-1, irrespective of the actual difference between them. However, a weakly augmented view and a strongly augmented view of an input example may not necessarily share the same latent structure, and could be advantageous to encode them into different regions the feature space, as illustrated in Figure 1 (a). In such cases, an encoder attempting to pull positively paired views together would learn an incorrect representation distribution.
To this end, introduce the concept of DA consistency. With the DA consistency imposed, the encoder is trained not only to cluster positive pairs together but also to map them to their respective optimal locations in the latent feature space Z based on the strength and types of applied augmentations. Consequently, the encoder preserves critical information introduced by data augmentations, such as variances related to brightness, sharpness, and rotation, as shown in Figure This characteristic of the pre-trained encoder enhances its performance on various downstream tasks, where such information is crucial for recognition.
We quantify mapped feature space location using latent similarity with respect to the raw input data.
5

---

Published in Transactions on Machine Learning Research (09/2024)
Definition 3 (Latent Similarity). The latent similarity La(æ;; A) of an imput æ, given encoder f and composite augmentation A, is defined as the cosine similarity wetween the normalized representations of a and A(x) in the feature space Z C smm-1 La(2; A) A)= f(x)" f(A())).
To be DA consistent, latent-space similarities of different augmented views of the same data shall be mono- tonically decreasing with the augmentation strength. We learn mapping, i.e., parameterized neural network (+) that takes composition vector v(A) of composite augmentation A input and map it to the optimal latent similarity ()()) as detailed in Section 4.3 An optimal encoder is considered to be fully DA consistent, if its latent similarity for any given composite augmentation is considered optimal. For encoders that are not fully DA consistent, we further define DA consistency level, as the degroe at which the learned representations are consistent in terms of data augmentation.
Definition 4 (Consistency Level). Given an encoder f, set of composite augmentations Ne and raw input z, the DA consistency level (DACL. is define as the encoder's deviation from optimal latent similarity:
(. = (l J, A) (
(1)
DACL measures the level of consistency of given encoder f by comparing the latent similarity of the augmented view of a with the corresponding optimal latent similarity (²()( over all possible composite augmentations.
With DACL, an encoder fe((.) parameterized by O. that is not fully DA consistent can be trained to more DA consistent by minimizing the consislency loss defined below:
Ccousistent (Oe))
Ex (B fo.: ste]l]
[DACL(x;
E A- (1r Sa A) (()(l)))]]
(2)
Ez (1la(;;; Sac, (o())
Our empirical study shows that alternative consistency loss provides better performance, details of this alternative is included in Appendices.
4.3
Learning Optimal Latent Similarities
4.3.1 Learnable Model for Optimal Latent Similarities
To train the encoder on the proposed consistency loss equation g we need to provide the optimal latent similarity ((A)) for a given composite augmentation A, which contingent on the strength of A. Nev- ertheless, defining augmentation strength poses challenge, two composite augmentations may comprise different basic angmentation types, making it unclear how to define and compare their strength.
To tackle this problem, we learn mapping, i.e., parameterized neural network gex(:) that takes a compo- sition vector as input and map to the optimal latent similarity (²))) This approach relaxes the need for directly modeling the strength of different augmentations.
4.3.2 Imposing Monotonicity to the Learnable Model
Taking step further, not only approximate the optimal latent similarity L with neural network but also enforce important monotonic property to better learn the desired optimal latent similarities.
As the overall strength of composite augmentation increases, e.g,, by incorporating additional basic augmen- tations, the similarity between the augmented data and the raw data is expected to decrease. For instance, although directly comparing the effects two different augmentations CaussianBlur and Sharpness (which blurs the image by a Gaussian kernel and adjusts edge contrast, respectively.) may be challenging, applying Sharpness twice composite DA would certainly distort the raw input more than applying it once.

---

Published in Transactions on Machine Learning Research (09/2024)
[DA Strength Comparison Operator]Thus, illustrated in Figure 3 formally, we define op- erator for comparing the strength of compo- sition data augmentations in set Se: A, iff v (())) where 3 operates element wise on the composition vectors: v(A.))<(AA)) implics that (A.)[(]] [k),)kk vk [1, and 3K E Na. such that [k) [k)
Saturstion
Saturation
Saturation
Solarize
Solarize
Stronger
Weaker
Sharpness
Gsussisa Blur
Solarize
Solarize
Not directly comparable
In other words, composite augmentation A, is con- sidered to be stronger than Aj (i.e., Aç)A,) if and only if the number of times any basic augmentation is applied in A, is at least that of the same basic augmentation is applied in Aj, and there at least one basic augmentation used in A; more times than
Figure A composite augmentation considered stronger than another if and only it includes all ponents of the latter, along with additional basic aug- mentations.
in Az.
Our learnable neural network model g0, takes the composition vector of an augmentation A; as the input, and map it to the optimal latent, similarity of (((() in the feature space: ()()( = go, (vA.)). We enforce monotonicity w.r.t. input v(A.) on ge, go, (o(A)) > (v)()))))) ((A))Su(A)) AA,, VA, E Qc In other words, stronger data augmentations result in a smaller latent similarity. The monotonicity is enforced by incorporating monotonic linear embedding layers You et al. (2017) in g0, (.), shown in Figure bb. We refer to go,(.) monotonic mapping neural network (MMNN).
4.4 Encoder Training with Consistency
The proposed CoCor approach incorporates the consistency loss equation F form the overall loss function Cy for optimizing the encoder parameter 0e given the MMNN's parameter 0a on unlabeled data:
(0(0)(0a) = Lcontrast (0e) ) + Loousistent (0e 0a)
(3)
In equation 3 the optimization of the encoder parameters, Oe, is contingent upon the MMNN parameters, Od- Consoquently, we denote the dependency of 6e on Oa as (fa) and solve bi-level optimization problem optimize Og:
mino. CE((r((2)) (0a)) = arg ming, (0.(00)
(4)
s.t.
(5)
The top-level problem in equation optimizes the MMNN and linear classifier (not explicitly shown equation 4 based on a cross entropy loss CE on small amount of labeled data. During each iteration, the bottom-level problem in equation 5 solved to update the encoder parameter 0, the unlabeled data. We defer the derivation of 0.'s dependency on 02 and update rule for 0g to Appendices.
We conclude the algorithm flow of the proposed approach in Algorithm [
The design of CoCor makes compatible with various state-of-the-a self-supervised learning frameworks. CoCor can be integrated into an existing method by replacing Coontrast equation 3 with the mothod's specific loss function. The added consistency loss imposes DA consistency and improves the generalizability and transferability of the pre-trained encoder within these frameworks.

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm 1: Algorithm flow of CoCor
initial encoder, MMNN, and classifier parameters ( ((0) and o number of training
Input:
epochs N, unlabeled dataloader Du, labeled dataloader Dj.
for i=1 to N do
1.Sample unlabeled Eu and labeled data (æ), yt) from Du and Dj, respectively.
2. C.Cl equation 5 update the encoder's parameter o on Uu with MMNN parameters 0 fixed.
3.Call equation 4 to update MMNN o and classifier (()) on (a), with the encoder parameters
_ fixed.
Output: Trained encoder with parameters ON
5
Experimental Studies
We demonstrate the performance and generality of CoCor by integrating it into several state-of-the-art contrastive learning methods: MoCo Chen et al. (2020b); He al. (2020), a dual-encoder approach with a large negalive memory queue, SimSiam Chen & He (2021), which exclusively leverages positive pairs, and SupCon Khosla et al. (2020), supervised contrastive learning method that uses labeled data to enhance reprosentation learning. We refer to cach of the three methods as a bascline. We compare the performance of encoders pre-trained using these baselines against those pre-trained using a combination of CoCor with the baselines across a variety of datasets for linear evaluation and object detection. CoCor is also compared information into consideration Zhang & Ma (2022) Lee (2023) and works which use stronger data augmentations than conventional contrastive learning Lee & Shin (2023): ; Wang & Qi (202T). To shed more light on the working mechanism of
with recent works which take augmentation-awaa
et al.] (2021); Devillers & Lefort
CoCor, we perform post-hoc analysis on latent similarity and potential dimensional collapse, and number of ablation studies.
Unless otherwise noted, all experimental results presented in this paper are reproduced by us.
5.1 Experimental settings for encoder pre-training
The encoders of the baseline methods pre-trained the large ImageNet-11K Russakovsky et al. (2015) dataset and its subset ImageNet-100 Tian et al. (2020a), under two different backbone encoder archilectures ResNet-50 and ResNet-34 He et al. (2016). The encoders of MoCo and SimSiam based methods are pre-
trained for 200 epochs, while SupCon's encoder undergoes 100 epochs of pre-training. Batch size of all
pro-training experiments is set to 256. We follow Khosla et al. (2020): : Chen et al. (2020b); Chen Hc
2222 for other settings of these baseline methods.
For the sake of fair comparison, pre-training with CoCor incorporated follows the same experiment settings their corresponding baselines. The MMNN in CoCor is 3-layer MLP with monotonic lincar embedding layers You et al. (017)) and ReLUU units. While most of the results are demonstrated when using only 1% of the pre-training dataset labels for tuning the MMNN, we also demonstrate the good performance of CoCor in an ablation study where the labeled data usage is significantly further reduced in Table 5.4 CoCor utilizes a basic data augmentation set Pa, consisting of 14 commonly used augmentation functions as detailed in Appendices. We impose the DA consistency constraint to several composite DA sets Ω of specific lengths. For most pre-training experiments, we take / 1, 2, i.e., for each example, we sample three composite DAs of l 2, and 3, and apply them to get three views of data to calculate the consistency loss. More details of data augmentations and pre-training setup are provided in Appendices.
5.2 Main Results
Linear evaluation Each pro-trained encoder is parameter-frozen and paired with a linear classifier, which is fine-tuned, following the linear evaluation protocol of Krizhevsky al. (2017), as detailed in Appendices. Linear evaluation is conducted the following datasets: Cifar-10/100 Krizhevsky al. (2009), CUB-
8

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Top-1 accuracies (%) of linear evaluation. All ResNet-50 backbone encoders are pre-trained on
ImageNet-100.
Method
Epochs
IN-100
Cifar10
Cifar100
CUR200
Caltech101 SUN397
Food101
Flowers102
Pets
MoCo Chen et al.
2020b
200
67.04
82.15
59.17
21.52
79.23
39.43
52.97
71.77
56.94
MoCo + CoCor urs
200
71.66
83.87
59.64
22.57
81.51
41.95
54.81
75.65
60.23
SimSiam Chen & He
2021
200
73.92
84.66
61.82
26.13
85.11
45.96
58.73
82.40
65.60
SimSiam + CoCor urs
200
83.70
86.89
66.36
34.50
87.85
49.92
62.48
87.95
76.04
SupCon Khosla et 2020
80.16
84.30
61.33
26.46
85.04
41.95
52.45
76.99
73.75
SupCon + CoCor (Ours)
100
82.14
85.49
62.18
27.40
87.12
42.87
53.93
79.04
75.95
Table 2: Top-1 accuracies (%) of linear evaluation. All ResNet-50 backbone encoders are pre-trained on ImageNet-1K.
Method
Epochs
IN-1K
Cifar10
Cifar100 CUB200
Caltech101 SUN397
Food101
Flowers102
Pets
MoCo Chen et al.
2020b
200
67.56
92.24
74.33
41.54
92.00
58.28
69.84
88,86
81.74
MoCo CoCor Ours
72.83
93.28
76.19
44.11
93.10
60.15
71.10
90.39
83.08
SimSiam Chen & He (2021
200
70.93
93.08
76.21
51.05
93.13
60.81
71.18
92.11
85.01
SimSiam + CoCor urs
200
73.25
94.04
78.12
54.50
94.51
63.09
73.31
92.84
87.24
SupCon Khosla ct 2020
74.18
93.17
76.55
52.30
93.28
61.03
71.54
91.34
85.35
SupCon + CoCor (Ours)
100
76.24
94.61
77.91
55.73
94.66
63.80
74.09
92.17
87.21
200 Wah al. (2011), Caltech-101 Fei-Fei et al. (2004), SUN397
Xiao et al. (0010). Food101
Bossard et al.
2201 4.. Flowers 102 Nilsback & Zisserman (2008), Oxford-IIIT Pet (Pets) Parkhi et al. 2012 Aircraft Maji
et al. 2013), and StanfordCars Krause et al. (2013).
The classification accuracies for 100 and ImageNet-IK pre-trained encoders are presented in Ta- ble and Table Notably, CoCor demonstrates a substantial enhancement in the performance of the pre-trained encoder across diverse classification lasks. This improvement suggests that CoCor contributes to an enhanced generalizability of the encoder, achieved by incorporating a wider range of augmentations and by complementing the semantics within the feature space.
Object detection We fine-tune the pre-trained encoders on VOC2007+2012 Everingham et al. (2009) and COCO2017 Lin et al. (2014) datasets for object dotection downstream tasks. The pro-trained encoders are converted to generalized R-CNN Girshick et al. (2014) detectors with a ResNet50-C4 backbone by using Detectron2 Ww al.] (2019). The models trained on VOC2007 and COCO2007 are subsequently evaluated on the correspouding dataset's test set.
In comparison with the baselines, Table shows that CoCor substantially improves accuracy under standard metrics such as AP, APso, and APrs, suggesting that CoCor is effective in achieving the pre-trained encoders' transferability. Notably, CoCor shows substantial improvements in detecting challenging small and medium object detection, as evidenced by the increased AP, and AP,, scores.
Post-hoc study We visualize the effect of the proposed consistency loss of equation 2 during pre- trainini by comparing the latent similarity of encoders pre-trained with and without the consistency loss in Table (a), showing the averaged latent similarity of each specific length of composite augmentations on the ImageNet- 100 dataset. Clearly, the use of CoCor results in larger latent similarity values across a wide range of DA lengths, demonstrating a lighter clustering of image views with the same identify. This aligns with the analysis presented in [Wang & Isola (2020), which, even without taking into account the dependency on augmentation strength, suggests that stronger clustering of views generated by weaker augmentations is correlated with improved encoder performance. Additionally, the image retrieval task results Figure [b) suggest that. CoCor learns more semantically consistent feature space compared to the baseline methods.

---

Published in Transactions on Machine Learning Research (09/2024)
Table 3: Transfer learning results on VOC and COCO object detection tasks.
Pre-train
VOC07+12
COCO
Method
Epochs
AP
APto
APrs
AP
APs0
APrs
AP.
APm
APt
MoCo He et al.
2020
200
50.23
76.52
54.42
34.23
55.08
36.60
16.67
$7.27
50.97
MoCo
CoCor
(Ours)
200
51.11
77.41
54.88
39.13
58.30
42.19
23.19
43.45
52.13
SimSiam Chen & He 20021
200
52.04
77.86
57.11
34.15
55.21
36.50
15.30
37.37
50.49
SimSiam + CoCor O Ours
200
54.31
80.60
59.58
34.80
56.29
37.05
17.53
39.88
51.74
SupCon Khosla ot al. 2020
100
47.41
75.83
53.17
34.26
55.11
36.35
14.34
37.40
50.56
SupCon + CoCor (Orr))
100
49.03
77.22
54.19
35.67
56.10
37.98
16.63
38.71
51.29
Normallznn wwwddallddddd Singular Value
Moto MoCo CoCor
Sendiam
Sinsiem CoCor
similarity
similarity
Singuler 1
Cocor
Louar
Composite augmentation length
Composite sugmentation length
Sorted singular value index
Sorted singular value index
(b)
Figure 4: (a): Latent similarities under different composite augmentation lengths of encoders pre-trained with and without consistency loss, and (b) singular values of the learned latent presentations, both evaluated on ImageNet-100.
This introduced consistency ensures that data sharing similar latent semantics is positioned close, while dissimilar instances are kept distant in the feature space.
Dimensional collapse evaluation We illustrate the singular values of the latent representations of pre- trained encoders on ImageNet-100 by conducting principal component analysis (PCA) on the feature vectors. The occurrence of dimensional collapse typically associated with observed small singular values. However, as depicted in Figure 4 (b), this less likely be the case for the encoders pre-trained by CoCor compared with the two baselines. While stronger data augmentations Jing et al. (2021) and larger datasets Li et al. (2022a) have been suggested to contribute to dimensional collapse, this issue is not observed in the CoCor encoders. In conventional contrastive learning methods, the encoder is forced to be invariant to fcature variance introduced by augmentations, potentially leading to constant values in some feature space dimen- sions. The dimensional collapse problem is mitigated in CoCor, possibly due to the imposed consistency that captures this variance.
Table 4: Comparison of top-1 accuracies (%) in linear evaluation of SimSiam, existing augmentation-awwrr methods, and CoCor. Encoders (ResNet-34) are trained on ImageNet-100 for 200 epochs.
Method
ImageNet100
CUB200
Flowers102
StanfordCars
SimSiam Chen & He 2021
63.90
29.10
76.10
19.60
Augsolf Lee al. 2021
75.96
29.96
82.96
29.41
Hierarchical Zhang
2022
67.10
33.90
83.60
20.70
EquiMod Devillers & Letort
22033
65.45
30.09
77.91
19.06
SimSiam CoCor (Ours)
77.76
34.07
83.88
29.93
10

---

Published in Transactions on Machine Learning Research (09/2024)
Table 5: Comparison of top-1 accuracies (%) linear evaluation of methods that leverage stronger data augmentations. Encoders (ResNet-50) are trained on ImageNet-IKe for 200 epochs.
Method
ImageNet
CUH2000
Flowers102
Aircraft
CLSA Wang Qi 2021
69.4
40.20
89.97
61.24
RényiCI. Lee & Shin 2023
72.6
45.05
90.17
61.80
MoCo+CCoCon (Ours)
72.8
44.11
90.39
63.18
5.3 Comparison with the state-of-the-art
Comparison with augmentaaiion-aware methods Recently, several works Devillers & Lefort (0223
Lee el al. (2021): Zhang & Ma 2022) have taken inlo account augmentalionn-wwar information cont cont iae learning. AugSelf Lee et al. (2021) achieves this by training an auxiliary network predict the difference between augmentations applied to generate positive pairs. Zhang & Ma (2022) encodes an augmented image along with the DA parameters using two networks, and combines the two embeddings to form a feature vector for contrastive loss. EquiMod Devillers & Lefort (2023) trains a network to predict the representation of an augmented view from the original data and the applied DA. A performance comparison between these methods and CoCor on lincar evaluation is presented in Table 4 All the encoders have a ResNet-34 architecture and are pre-trained on ImageNett-100 for 200 epochs. Compared to Lee et al. (2021) : Zhang & Ma (2022); Devillers & Lefort (2023), CoCor introduces much stronger DA and more diverse DA types. CoCor consistently outperforms them, which may be attributed to its effective utilization of significantly broader set of augmentations while maintaining the well-specified monotonic property in augmentation strength.
Comparison with methods using stronger data augmentations Several recent contrastive learning approaches incorporate the use of stronger data augmentations. CLSA Wang & Qi (2021) employs the distribution divergence between weakly angmented views as supervision for the divergence between a weakly and a strongly augmented view. RényiCL & Shin 2022)) proposes new contrastive objective using Rényi divergence to address strongly augmented data. Table shows the comparison between these methods and ours. Here all encoders are ResNet-50 trained for 200 epochs on ImageNct- Unlike RényiCL and CLSA, which do not explicitly distinguish between different types of data augmentations (DAs), CoCor addresses this by using the MMNN to differentiate various DA types. Furthermore, CoCor introduces DAs that not only stronger but also more diverse, implementing consistency across different lengths of DAs, in contrast to RényiCL and CLSA, which apply a single length of DA per pre-training process.
5.4 Ablation Studies
Basic Data Augmentations define the augmentation--ware information to be learned by the encoder. To better understand the role of these data augmentations during pre-training, we separate them into two groups: Ocolor and Pattiee Pcolor includes augmentations that change the color of each pixel while maintaining its position, such as Brightness and Contrast. Pattine comprises affine transformations, such as Rotate and Shear. Composition of Pcotor and Qattine is provided in the Appendices. Using CoCor, we pre- trained three encoders for 500 epochs on ImageNet-100 with Reolor, Qattine; and Pestor Uaftine, respectively. Table presents the linear evaluation results on ImageNet-100 and two fine-grained datasets, CUB-200 and Flowers102. The results indicate that CoCor achieves better overall performance than LooC Xiao et al. (2020) and AugSelf Lee et al. (2021), which have improved transferability to various downstream recognition tasks. Notably, while Dcalor Ramiine delivers the best overall results across the tested datasots, refining the basic data augmentations can enhance CoCor's performance on specific downstream tasks. For example, color- aware information is crucial for discriminating flower images in the Flower102 dataset Nilsback & Zisserman (2008), whereas position-related information introduced by affine transformations is less relevant. Thus, applying only color-related augmentations while maintaining invariance to affine transformations improves performance 100
11

---

Published in Transactions on Machine Learning Research (09/2024)
CoCor is pre-trained using basic data augmentation pools, which include color-related augmentations, affine transformations, and their combination. Top-1 Linear evaluation accuracy is reported. Encoders (ResNet-50) are trained on ImageNet-100 for 500 epochs.
Table 6: Comparison between LooC, AugSelf, and CoCor.
Method
ImageNet100
CUB200
Flowers(5-shot)
Flowers(10-sho))
MoCoft Chen et al.
2020b
81.0
36.7
67.9
77.3
(ooo(c Xiao et al.
2020
81.1
40.1
68.2
77.6
Looo(rotation] Xiao
2020
80.2
38.8
70.1
79.3
Looor, rotation] Xiao et al.
2020
79.2
39.6
70.9
80.8
MoCo + AugSelf* ft Lee al. 2021
82.4
37.0
81.7
84.5
SimSiam + AugSell Lee ol. a..
12021
82.6
45.3
86.4
88.3
SimSiam CoCor (color)
83.5
46.5
89.1
90.2
SimSiam + CoCor (affine)
81.8
83.5
86.1
SimSiam + CoCor (color+affine)
83.7
47.9
88.9
89.6
*Sinee the official implementation LooC has not been released, we adopt the results from Lee et al. (2021) ; Xiao
et 2020
ensure a fair comparison, we maintain the same experimental settings for all methods this table.
Table 7: Linear evaluation of CoCor models trained with and without MMNN. Encoders (ResNet-50) are trained on ImageNet-110 for 200 epochs.
baseline
w/o MMNN
w/ MMNN
MoCo CoCor
67.04 73.92
60.51 82.18
71.66
SimSiam + CoCor
83.70
Effectiveness of the MMNN CoCor utilizes an MMNN model to learn the optimal latent similarities la of varions composite augmentations. To see the benefits of this learnable MMNN model, we replace it by a manually optimized model: we manually search for the best L value for each composite augmentation length, determined by extensive trial-and-error, to achieve optimal linear evaluation performance. More detailed experimental settings and results on la selection are included in Appendices.
We pre-train ResNet-50 encoders built upon SimSiam and MoCo with the MMNN and the manual model based on composite augmentation lengths 1 = 1, on ImageNet-100 for 200 epochs. The resulting perfor- mances are reported in Table 7 which also includes the performances of the MoCo and SimSiam baselines. There is the clear advantages of the encoders trained with the MMNN over the ones without, i.c. with the manual model.
Effect of the amount of labeled data in MMNN training We further study the impact of the amount of supervision used to train the MMNN on the encoder performance. We pre-train ResNet-50 CoCor encoders built upon SimSiam on ImageNet-100 for 100 epochs while training the MMNN using 100%, 10%, 1%, and 0.1% of the labeled data in the dataset. The pre-trained backbones are evaluated by linear evaluation on TmageNet-100. Table g shows that drastically reducing the amount of labeled MMNN training data from 100% to 0.1% only results in 0.94% performance drop, and all four CoCor models outperform the bascline SimSiam by more than 15%.
Composite augmentation length in pre-training The composite augmentations introduced in our work produce diverse informative views. We test the effect of the strength of composite augmentations by running experiments where only one length of composite augmentation is adopted per pre-training experi- ment. We run CoCor on SimSiam and MoCo with single augmentation length 1, 2, 3, 4, and a combination of lengths 1, 2, and 3. All encoders are ResNet-50 pre-trained on ImageNet-100 for 50 epochs, and are then evaluated on ImageNet-100.
As seen in Table 5, CoCor is able to leverage augmentations of varying lengths, and combining DAs at all these lengths further improves performance. Different. from the results presented in Chen et. al. 2020a
12

---

Published in Transactions on Machine Learning Research (09/2024)
4.40
MrCaH with singed
0.44
Seröamn LiKorwith (-112-5
StrSien CoCor win sogle!
(a)
(b)
Figure 5: Linear evaluation of encoders trained by CoCor on (a) MoCo and (b) SimSiam with single length
of composite augmentations and combination of length 1, 2, and
Performance of CoCor models trained with different amounts of labeled data. Encoders (ResNet-50) Table 8:
are trained on ImageNet-100 for 50 epochs.
Percentage labels
100%
10%
1%
0.1%
SimSiam
Top-1 Acc
75.97
75.82
75.79
75.46
59.20
Tian et al. (020b)), which show that strong augmentations degrade performance, we have conducted additional experiments with much stronger composite augmentations to demonstrate CoCor's ability in lever- aging such augmentations. These results, presented in the Appendices, demonstrate CoCor's capability to benefit from even stronger augmentations compared to recent works that introduce strong augmentations in their methodsLce & Shin (0033: Wang & Qi 2021
6 Discussion
In this paper, introduce CoCor, a systematic approach exploring diverse data augmentations in con- trastive learning. Our contribution includes the introduction of DA consistency to quantify the dependency of the desired latent similarity on the applied data augmentation. We propose a consistency loss to guide the encoder training towards improved DA consistency. To enforce the DA consistency, we employ a data-driven method to learn the optimal dependency and apply it to the encoder training. The effectiveness of CoCor is substantiated through extensive experimental results on various tasks and datasets. Further details on the MMNN update rule, additional experimental settings, and extended results are provided in the Appendices.
As potential future direction, we suggest exploring the learning of optimal latent similarity with respoct to data examples. Specifically, extending the MMNN to take data instances input could allow for considering the variance of latent similarity caused by data variance. This approach has the potential to learn a more accurate mapping from data augmentation to latent similarity.
7 Acknowledgement
This material is based upon work supported by the National Science Foundation under Grant No. 1956313.
References
Pranjal Awasthi, Nishanth Dikkala, and Pritish Kamath. Do more negative samples necessarily hurt in contrastive learning?, 2022.
Lukas Bossard, Matthicu Guillaumin, and Lue Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-1 6-12, 2014, Proceedings, Part VT 13, pp. 446-461. Springer, 2014.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Aduances in neural information processing systems, 33:9912 9924, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprinl urXiv:2002.057 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Xinlci Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved basclines with momentum contrastive learning. Xix preprint arXiv:2003. vrXiu:203304297, 2020b.
Dan Ciregan, Ueli Meier, and Jürgen Schmidhuber. Multi-column deep neural networks for image clas- sification. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3642-3649. IEEE, 2012.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 113 123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. Tn Proceedings of the IERE/CVF conference on computer vision and pattern recognition workshops, pp. 702 703, 2020.
Alexandre Devillers and Mathicu Lefort. Equimod: An equivariance module to improve visual instance discrimination. In International Conference on Learning Represenlations, 2023.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-308 --000-08,, 2009.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 580-587, 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing syslems, 33:21271 21284, 2020.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual Icarning for image recognition. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 770 778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learning, pp. 2731- 2741. PMLR, 2019.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arxin:2 10. 09348, 2021.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661 13:11661-11663, 2020.
Simon Kornblith, Jonathon Shlens, and Quoc Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2661-2671, 2019.
Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine-grained cars. 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, lyy Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Hankook Lee, Kibok Lec, Kimin Lec, Honglak Lee, and Jinwoo Shin. Improving transfcrability of represen- (alions via augmenttation-wwar self-supervision. Advances Newral Information Processing Systems, 34: 11710-17721 2021.
Kyungmin Lee and Jinwoo Shin. Renyicl: Contrastive representation learning with skew renyi divergence, 2023.
Alexander C Li, Alexei A Efros, and Deepak Pathak. Understanding collapse in non-contrastive siamese representation learning. In European Conference on Computer Vision, pp. 490-505. Springer, 2022a.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations ions. arXiv preprint arXim:2005.4496: 049666 2020.
Lujun Li and Anggeng Li. A2-aug: Adaptive automated data augmentation. In Proceedings of the IEEE/CVE /CFF Conference on Computer Vision and Pattern Recognition, pp. 2266 2266-2273, 2023.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVE /CVF Conference on Computer Vision and Pattern Recognition, pp. 8918-6928 2022b.
Wei Li, Jiahao Xie, and Chen Change Loy. Correlational image modeling for self-supervised visual pre- training. In Proceedings of the IEFE/CVI /CVF conference on computer vision and pottern recognition, pp. 15105 15115, 2023.
Sungbin Lim, Ildoo Kim, Tacsup Kim, Chihcon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Newral Information Processing Systems, 32, 2019.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C LLwwrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint arXiv:1906. 02611, 2019.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. P'ine-graine visual classification of aircraft. arXiv preprint arXiv:1306.5111 15011,116, 2013.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the SEEE/CVE conference on computer vision and pattern recognition, pp. 6707-6717, 2020.
Maria- Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. Tn 2008 Sirth Indian conference computer vision, graphics & image processing, pp. TEFF, 2008
15

---

Published in Transactions on Machine Learning Research (09/2024)
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXi 11i::807.000 807.03748, 2018.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE IEEE, 2012.
conference on computer vision and pattern recognition, pp.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 11537-11568, 2021.
Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint ar orXtt:20100045000 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2535 252, 2015.
Tkuro Sato, Hiroki Nishimura, and Kensuke Yokoi. Apac: Augmented pattern classification with neural networks. ar preprint arxi::150, 03229, 2015.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020b.
Rogier Van der Sluijs, Nandita Bhaskhar, Daniel Rubin, Curtis Langlotz, and Akshay S Chaudhari. Exploring image angmentations for siamese representation learning with chest x-rays. In Medical Imaging with Deep Learning, pp. 444 467. PMLR, 2024.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-uesd birds- 200-2011 dataset. 2011.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pp. 1058 1066. PMLR, 2013.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929 9939. PMLR, 2020.
Xiao Wang and Guo-Jun Qi. Contrastive learning with stronger augmentations. arXio preprint arXiv:2104.07713, 2021.
Zihu Wang, Hanbin Hu, Chen He, and Pong Li. Recognizing wafer map pattorns using semi-supervised contrastive learning with optimized latent representation learning and data augmentation. In 2023 IEEE International Test Conference (ITC), pp. 141-150 150. IEEE, 2023.
Zihu Wang, Lingqiao Liu, Scott Ricardo Figueroa Weston, Samuel Tian, and Peng Li. On learning discrim- inative features from synthesized data for self-supervised fine-grained visual recognition. arXie preprint arXiv:2407. 14676, 2024.
Xi Weng, Yunhao Ni, Tengwei Song, Jie Luo, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan, and Lei Huang. Modulate your spectrum in self-supervised learning. arXiv preprint orXiu:2305.16788 2023.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan Yen Lo, and Ross Girshick. Detectron2. https: //ithub. /rt.taa...//aabboooeeerrrreeaoor 2019.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to 200. In 2010 IEEE computer socicly conference on computer vision and pattern recognition, 3485 3492. IEEE, 2010.
Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. arXiv preprint arXiv:2008.0500 05665,, 2020.
Yuxuan Yin, Yu Wang, and Peng Li. High-dimensional bayesian optimization via semi-supervised learning with optimized unlabcled data sampling. arXiv preprint 0rXiv:2305.0261 2023.
Seungil You, David Ding, Kevin Canini, Jan Pfeifer, and Maya Gupta. Deep lattice networks and partial monotonic functions. Adoances in newral information processing syslems, 30, 2017.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning, pp. 22121-12133 PMLR, 2021.
Junbo Zhang and Kaisheng Ma. Rethinking the augmentation module in contrastive learning: Learning hierarchical augmentation invariance with expanded views. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 16650-16659, 2022.
Tong Zhang, Congpei Qiu, Wei Ke, Sabine Süsstrunk, and Mathieu Salzmann. Leverage your local and global representations: A new self-supervised learning strategy. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16580-16666 11880-1689,, 2022.
17

---

Published in Transactions on Machine Learning Research (09/2024)
A Improving CoCor with an Alternative Consistency Loss
The original consistency loss is defined as:
Consisteett (0e Ee-x [Na(<;; (0, g0s (vc((a))
(6)
However, in our preliminary experiments, we find the following consistency loss leads to better downstream tasks performance than equation 6
Lconsistent (0.(0s) En ~e
ssoftpuu Na >n [la(:; S, <!>) 90. (velae <t> )II
(7)
Figure 6: Images from same category of a dataset. Target items may be of different sizes and colors, and may be located at different places in images. Thus same augmentation can have different effects on different images. ImageNet is used for illustration.
The motivation is to reduce the high variance among images in visual datasets. The exactly same composite augmentation hence can lead to different latent similarity in the feature space. For instance, as illustrated in Figure 6 target items of images may be of different size and shape, and may be located at different places. Therefore, enforcing all augmented views to share the same optimal similarity may not lead the best performance. To this end, equation calculates the averaged latent similarity in minibatch of each length of augmentation composition. Therefore, equation imposes DA consistency constraint the averaged latent similarity in batch instead of on each image, which help better handle the variance.
Additionally, data augmentation that can greatly distort an images may not be able to largely distort the identity of another one. However, the absolute value loss pushes apart those loss distorted views from the representations the original data. Thus, poftp used to replace the absolute value calculation because softplus produce small gradient for negative numbers.
Derivation of the Update Rule of the MMNN
In our implementation, the parameter of the encoder (0e) and the parameter of MMNN (0a) is updated in an alternate manner, as equation
0' = 0g Co..
(8)
0 Oa nl Vo, CE(e
With the overall loss function optimize the encoder, the update of 0g with learning rate ne can be written
0' = 0c - n) (Vo. Lcontrast + Vo. Cconsistent)
(9)
18

---

Published in Transactions on Machine Learning Research (09/2024)
By applying chain rule, the update of da with learning rate na can be rewritten as:
0' 0s ---- "Ia" ee CF((
à CE((a)
00
(10)
where: Ve, (o))
a0
d0
In equation 10 a CE(O2) can be computed via back-propagation. Thus, we focus on the computation of 80,
0e
80a
First, we substitute 6" in equation 10 with the one-step updated 0 in equation 9 following the approach
used in Pham et. al. (2021] Vin al. (2023) We also neglect the higher order dependency of Og 0a
00
adopted Pham et 2021
can thus be expanded as below:
30
00
0
10.
Te Ccontrass + Vo Cconsistent)
d0
a
0
0
fconslatent
(11)
00
00.
0
0
Booppsootp (a) (ao)) (ve ))]
Te a 00.
where u! <>>>>> (x:) denotes the augmented data, l denotes the set of lengths of composite augmentations that are used, and B denotes the set of data indices in minibatch.
de.t
For simplification in later section, we define:
=Eies[(Jo, (x:e) yod (va( sim' (eo So (2:1))
(12)
We then combine equation 11 with equation
12
ae
i ed'
Te Biedl
sim
a0,
a0
cke
00.
(13)
a sim
T/e Bied
Eieslyo (volaet ((2)))
(1 + ektyz
00_
d0
By incorporating equation 13 with equation 10
we have:
endl
0
a
Vo, CO, = CE(A")) (~Te Fied
sim'
E (go, (va(a <t> III)
(14)
00
(1 + ekizz
d0_
d0a
Additionally, in equation
14
CE(O') can be approximated with first order Taylor expansion: ae
CE CE(O.) ~ (0's - -0e) 00
CE((
(15)
Thus, we have:
CE(O') R CE((') CE((2)
CE((')) CE((e)) - (Ve, Loontrast + Ve, Cooneistent)
(16)
00
0' 0e
19

---

Published in Transactions on Machine Learning Research (09/2024)
We can now rewrite equation 14
CE - CB((2)
ek
0
Vo, Vo,C (R() ~ (Vo. Coontrast + Vo, ccousistent)
Beed
sim
E [1o, (ve(a <1> ))]]
(1 +le ehiyz
00.
d
(17)
Further more, equation 17 can be further simplified. Note that Va.Ccontras Vo Connsistent can be approx- imated by:
Lu((') Lu((e))
0, coontrast + Vo, cooniistnt Ve. Culee
(18)
O' te
Similarly, the approximation of a sim' can be written as:
00
a sim Rd
sim' (0%) sim'
00.
0' Oe
(19)
where:
sim (0')) = Eiess(So (2!)² (oc (x:)))
Finally, by combining equation follows:
18 and equation
19 with equation
Vo, CE((((O)))) can be rewritten as
eli
sim (0') sim (0c)
Ve. (e)) = (CE(e')) CE((2)) Exeel
Eie (V(A))]I (20)
(1 222
(O)) Cu
00
Note that in equation 20 all the derivative terms can be calculated directly via back propagation, resulting in a scalable algorithm.
C Candidate Augmentations in CoCor
The of basic augmentations Q2 used to form composite augmentation consists of different types data augmentations, i.e, Q2 (AutoContrast, Brightness, Color, Contrast, Rotate, Equalize, Identity, Posterize, Sharpness, ShearX, ShearY, Solarize, TranslateX, TranslateY).
In Section 5.4 we conduct ablation studies on the composition of basic data augmentations, as illustrated in Table 6 In these experiments, Ωt is divided into two sets: Rcotor and Qatinee The compositions of the two sets Caadooo=AAAoCooiaaaaaaaa (AutoContrast, Brightness, Color, Contrast, Equalize, Identity, Posterize, Sharpness, Solarize,} and Qatfine = (Rotate, Identity, ShearX, ShearY, TranslateX, TranslateY).
are as follows: Qecooor =
Experimental Setup for Pre-training
All pre-training experiment are conducted using stochastic gradient descent (SGD) as the optimizer. A cosine decay scheduler is adopted for scheduling the learning rate. All pre-training adopt training data batch size of 256. Detailed setups used for each method are follows.
MoCo He et al. (020);; Chen et al. (2220b:: We use starting lcarning rate of 3 10-2, a weight decay of x 10 and a momentum of 0.9 for the optimizer. For the dual encoders of MoCo, feature space dimension is 128, memory queue size 65536, and the momentum of updating the key encoder 0.999. For the consistency loss, the query encoder takes both the original data and its augmented view as input. However, the query encoder is stop-gradient when encoding the original input. The encoder receives gradient while encoding the augmented views.
20

---

Published in Transactions on Machine Learning Research (09/2024)
SimSiam Chen & He [0221 A starting learning rate of 5 10-2, 3, weight decay of 1 x 10- and momentum of 0.9 are used for the SGD optimizer. Predictor and projector and output dimension are both 2048. For the consistency loss, representations of the original data are obtained by letting the data go through the stop-gradient backbone and the projector. The augmented views go through backbone, projector, and predictor for the represenlations of them. All the three models receive gradient while calculating the representations of augmented views.
SupCon Khosla et (0220; We use a starting learning rate of 10-2,, a weight decay of x 10-4, and momentum of 0.9 for the optimizer. Feature space dimension of the projection head output is 128. In the consistency loss, the encoder takes both the original data and its augmented view input. The encoder is stop-gradient when encoding the original input. The encoder receives gradient while calculating representations of the augmented views.
E Experimental Setup for Linear evaluation
We follow the linear evaluation protocol of Chen et al. (2020a); Lee et al. (2021); Kornblith et al. (2019). A onc-layer lincar classifier is trained upon the frozen pro-trained backbone. Resize, CentorCrop, and Normalization are used as the training transformations. An L-BFGS optimizer is adopted for minimizing the éz-regularized cross-entropy loss. The optimal regularization parameter is selected on the validation set of each dataset. Finally, models are trained with the optimal regularization parameter, and the obtained test accuracies are reported.
MoCo composite aug. length I=1
MoCo composite length I=2
MoCo composite aug. length /=3
Linear evaluation atc
Linear evaluation acc
Linear evaluation
e41
ain selo mill edlo adts alte alts ana Constant value
Constant value
Constant l value
SimSiam composite aug. length [-1
SimSiam composite aug. length I=2
SimSiam composite aug. length [=3
Linear evaluation ace
Linear evaluation acc
Linear evaluation
320
140 +No <ita
660
Constant value
Constant value
Constant
value
Figure 7: Lincar performance accuracies of CoCor models trained with only one length of composite aug- mentation. CoCor with each length of composite augmentation is run with multiple different constant L values.
F Ablation Study on CoCor without the MMNN
To evaluate the effect of MMNN, we run CoCor where 1 is replaced by fixed constants. In practice, constant is used for each length of composite augmentation in calculating the consistency loss. We run multiple experiments to select the constant for each length of composite augmentation. All models are ResNet-50 pre-trained on ImageNet-100 for 50 epochs. Figure the lincar evaluation results of CoCor running with one length of composite DA and with different constant 15 For each length of composite augmentation, there exists optimal constant which leads to the best result. in linear evaluation. These
21

---

Published in Transactions on Machine Learning Research (09/2024)
optimal constants are then used for the comparison with the MMNN. The best constants of MoCo and SimSiam are listed in Table g
l
1
MoCo
0.80
0.75
0.65
SimSiam
0.75
0.70
0.60
Table 9: Constant la used for the comparison with the MMNN.
G CoCor with much stronger data augmentations
CoCor is a systematic approach of leveraging the information provided by diverse augmentations. In order further test the effect of composite augmentations of different strength, we run CoCor on MoCo and SimSiam with single augmentation length of 2, 3, 5, and 10. We train ResNet-50 ImageNet-100 for 50 epochs. Pre-trained backbones are evaluated on ImageNet-100 for linear classification. Table 10 shows the
accuracies of evaluation.
1
2
3
5
10
baseline
MoCo
47.90
47.70
48.08
45.16
43.07
42.66
SimSiam
54.01
54.20
55.16
53.50
52.70
44.60
Table 10: Linear evaluation of encoders trained by applying consistency loss on only one length I of composite
augmentations.
As is shown in Table 10 introducing relatively weaker (shorter) data augmentation ion in pre-training leads to better performance on downstream tasks. When evaluating the pre-trained models on various downstream tasks, the test data for testing model performance are natural and realistic images. Thus, providing the encoder with less distorted views in pre-training can help the encoder better recognize natural images during inference.
It is also noteworthy that CoCor can leverage very strong augmentations of a length of 10 to improve the encoder's performance over baseline, which is even stronger than the data augmentations used in recent works that include strong augmentations such as CLSA Wang & Qi (2021 and RényiCT. Lee & Shin 2023 Although it has been shown that very strong augmentations which can distort the identity of data too much may bring too much noise the training process. The noise may result. in performance degradation Chen al. (2000a); Tian al. (2020b), partial dimensional collapse, or even complete dimensional collapse )022a Jing ct al. 20211. However, CoCor able to capture essential information from the highly distorted views and enrich the semanticc of the feature space using these strong augmentations.
et al.
H CoCor's compatibility with more baseline contrastive learning methods.
To demonstrate CoCor's compatibility with additional baseline methods, e.g. BYOL, Grill et al. 2020 and INTL Weng et al. 2023 we implement CoCor on these additional baseline methods. Lincar evaluation results of these encoders across various datasets are presented in Table 11 These results indicate that CoCor not only enhances the generalizability of learned representations from contrastive methods like MoCo, SimSiam, and SupCon but also extends these benefits to other existing contrastive learning approaches.
Training Time Comparison
CoCor applics the proposed consistency loss to views producod by various data augmentations. To address the concern of extra training time consumed by CoCor over its corresponding bascline methods, we compare the pre-training time and model performance in Table 12 Although CoCor requires more time per epoch,
22

---

Published in Transactions on Machine Learning Research (09/2024)
Table 11: Top-1 accuracies (%) of linear evaluation. All ResNet-50 backbone encoders are pre-trained on ImageNet-100.
Method
IN-100
Cifar
CUR200
Caltech101
SUN397
Food101
BYOL Grill al. 2020
68.05
60.13
20.27
78.90
39.03
53.01
BYOL + CoCor (Ours)
72.30
62.42
22.75
80.51
42.64
55.32
INTL Weng et al. 2023
74.58
62.73
29.05
87.45
47.06
60.38
INTL
CoCor (Ours)
76.98
63.06
31.40
88.81
48.27
64.27
CoCor achieves better performance than its corresponding baselines with shorter training time. Additionally, CoCor takes similar amount of time to train as CLSA Wang & Qi (2021), which uses strong multi-view augmentations. However, CoCor achieves significantly better performance than CLSA with similar training time.
Table 12: Training Time Comparison machine with 4 A100 GPUs. Training time (in hhurr//Top- linear evaluation results (in %) on ImageNet-1K is provided for comparison.
Epochs
50
100
200
Methods
MoCo He al. 2020
13. 13.4b/64.78
26.8h/666.1
53.6h/67.0
SimSSaa Chen He 2021
13. 13.2h/66.86
26.4h/68.12
12.8h/70.93
CLSA Wang & Qi 2021
18. 15..1h67.995
36.2h/09.15
72.4h
MoCo + CoCor
18.5h/69.22
37.0h/70.76
74.0h/72
SimSiam CoCor
18.1h/70.90
36.2h/72.03
72.4h/T3.25
J Impact of the Data Augmentation Ordering in CoCor.
To evaluate the impact of the ordering of DAs in CoCor, we conducted experiments with various fixed DA orderings. We trained encoders using MoCo CoCor with specific DA orderings. Results shown in Table 13 For instance, in configurations CoCor-fixed 1/2, the order of DAs is predetermined. In a fixed order scenario, such as in CoCor-fixe 1, rotation is consistently applied before brightness whenever both augmentations are selected for an image. Results in Table 13 indicate that in CoCor, the ordering of DAs does not significantly affect the encodor's performance in downstream tasks. Thus, in final experiments, we use random DA ordering to explore wider range of DA variation.
Table 13: Linear evaluation results on ImageNeet-100 of CoCor models trained with different fixed DA orderings. Encoders are trained on ImageNet-100 for 200 epochs.
MoCo
CoCor fixed
CoCor fixed 2
Top-1 Ace
67.04
71.32
71.47
23


Reviews:
[{"review_id": "Gfw1e0E3kC", "reviewer": "Reviewer_KPsZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The idea of incorporating data augmentations into the loss function and making the encoder aware of the augmentations is commendable. Unlike AugSelf, this paper delves deeply into designing a metric to rank the augmentations, which is a novel and valuable contribution.\n\n\nCons:\n1. Writing Clarity: The writing of this paper is somewhat confusing. For example, it begins by introducing augmentation in a supervised setting, but this is not connected to the rest of the paper. The sentences in the introduction are somewhat disjointed and the authors tend to use very long and complex sentences. Missing commas exacerbate this issue, making the text difficult to understand. For instance, in Definition 3, the lack of a comma after \"A\" made it challenging to understand what \"A\" represents.\n\n2. Formulation. The formulation presented in the paper seems a bit confusing, and the star representing the optimal should not be on the $l_d$, but on the parameter $\\theta_d$.\n\n3. Literature Review: The paper misses a very relevant piece of literature. The formulation (learning another neural network as a new metric) and the idea of the proposed method are quite similar to [1], which focuses on crop size as a data augmentation method. This paper, however, extends the focus to general data augmentations. Including and discussing this related work would provide a more comprehensive context for the contributions of the current paper.\n\n4. The justification of stronger augmentations. Simply taking more operations as stronger augmentation is a bit unreasonable, and at least from my humble view, the intensity should be a key value. For example, conducting twice saturations with smaller parameters should be less than one saturation operation with a larger value. \n\n5. The experiments. I found the results in all the tables are much lower than the MoCov2 and SimSiam are much lower than the original paper and results reported by [1] and Augself. \n\n\n[1] T. Zhang C. Qiu, W. Ke, S. Süsstrunk, and M. Salzmann, \"Leverage Your Local and Global Representations: A New Self-Supervised Learning Strategy\" CVPR 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an approach to addressing a key challenge in contrastive learning: the dependency on the quality of manually chosen data augmentation functions. Recognizing the critical role of data augmentations in providing informative views without explicit labels, the authors introduce Contrastive Learning with Consistent Representations (CoCor). Central to CoCor is the Data Augmentation consistency, which ensures optimal mapping of augmented input data in the representation space, aligned with the intensity of the applied augmentation. The proposed method systematically incorporates diverse data augmentations and learns optimal mapping locations while preserving a monotonic relationship with DA intensity. The experimental results are noteworthy, demonstrating that CoCor significantly enhances the generalizability and transferability of learned representations compared to baseline methods. This approach not only addresses the complexities inherent in data augmentations and representation learning but also offers a systematic solution that could be highly beneficial for future research in the field.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Discussion on the Difference with Reference[1].\n\n2. The authors should address the reasons for any degraded results observed. It would be beneficial to explain why the proposed method might underperform in certain scenarios and attempt to replicate the results under the same settings and baseline conditions. This would provide a clearer understanding of the method's strengths and weaknesses and offer insights into potential areas for improvement.\n\n3. The authors should strive to improve the clarity and readability of the paper. Simplifying complex sentences and ensuring proper punctuation would significantly enhance comprehension.", "broader_impact_concerns": "No need to discuss broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "abFW62yIOV", "reviewer": "Reviewer_q2iP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe motivation of the proposed method is clear. The proposed method intuitively can improve the performance of SSL methods, if it is well designed to exploit the magnitude (of consistency) of data augmentation. The introduce of data augmentation consistency is new to me. \nThis paper is overall well organized, the description is also clear. \nThe experimental results of “Composite augmentation length in pre-training” is interesting.\n\n\n**Weaknesses:**\n\n1.\tThis paper should provide the computation cost of the CoCor. E.g., The overall time cost to train baselines+CoCor, compared to the baselines. Based on the description of CoCor, I feel the proposed method of this paper will introduce significant additional computation cost, due to the bilevel-optimization. I doubt the practicability of the proposed method.   \n2.\tThis paper claims that “ CoCor achieves state-of-the-art results for various downstream tasks.”. I donot think the current experimental results can support this claim. The experimental results currently are weak:\n (1) In Section 5.2 ‘Main results”, Why not show the results of linear evaluation on ImageNet-1K itself?, like the experimental setup in Simsiam paper (Chen&He 2021). It is important to show this result; \n(2) In Table 3, the Simsiam method is significantly lower the Simsiam paper reported, e..g, Simsiam obtains 57.0 AP on VOC07+12  and 39.2 AP on COCO detection reported in Simsiam paper (Chen&He 2021) while 50.23 AP and 34.23 AP only in this paper. I think this paper should clarify it. \n(3) I donot think MoCO and SimSiam is currently the state-of-the-art SSL baselines, there are many better methods, e..g, Barlow Twins[1], INTL[2] (INTL [2] can obtains 75.2 accuracy of linear evaluation on ImageNet using 200epoch training with strong data argumentation) and 40.7 AP on COCO). This paper should conduct experiments to compare these methods.   \n3.\tThis paper claims that “Moreover, it can be readily integrated into existing contrastive learning frameworks, effectively imposing DA consistency on the encoder”. I think this paper should conduct more experiments to support this claim, since the experiments are only based on Moco and SimSiam. This paper should conduct more experiments on SSL method using regularization, e.g., Barlow Twins[1], VICReg[3], and INTL [2]\n\n\n4.  This paper claims “Our studies show that the ordering of the basic augmentations in a composite augmentation does not have a significant impact on performance.”. I think this paper should provide some evidence (in Appendix) to support this claim. E.g., how this paper gets the observations., \n\n5.   Some notation is not well explained (If I am correct), e.g., (1) What is $N_a$ in the last row of page 4? (2) what is $N_0$ in the Definition 2?\n\n**Ref:**  \n\n[1] Barlow Twins: Self-Supervised Learning via Redundancy Reduction, ICML 2021  \n[2]Vicreg: Variance-invariance-covariance regularization for self-supervised learning, ICLR 2022   \n[3 ] MODULATE YOUR SPECTRUM IN SELF-SUPERVISED LEARNING, ICLR, 2024", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on investigating the data augmentation (DA) of contrastive learning (typically the self-supervised learning for vision data). The motivation is that the intricacies of data augmentation and representation learning may lead to a performance degradation if augmentation functions are not judiciously chosen. This paper proposes Contrastive Learning with Consistent Representations (CoCor), including introducing the set of composite augmentations, defining the DA consistency. It further proposes to learn the optimal mapping locations as a function of DA, all while preserving a desired monotonic property relative to DA intensity. Experimental results demonstrate that effectiveness of CoCor in enhancing the generalizability and transferability of learned representations comparing to baseline methods", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My main concern is the practicality of the proposed method, it seems the proposed method is complicated in computation, and this paper does not provide strong results in experiments (see weaknesses) . It is better to add more experimental results to support the claims or reword the claims.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xyk4iP7o55", "reviewer": "Reviewer_e3Gj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths：\n1. The motivation is clear that the view of stronger augmentation has a smaller similarity with the original sample, which is thoroughly introduced.\n2. The experiments show the effectiveness of the proposed DA loss and the monotonic mapping neural network (MMNN).\n3. The visualization in Fig. 4(a) verified the latent similarity is larger than the baseline when stronger DA is adopted.\n\nWeaknesses：\n1. The generalization of the proposed DA consistent is limited as only basic DA could be used, for example, color-related augmentations and affine transformations.\n2. The teaser of Fig. 1(a) is overclaimed that the proposed approach cannot force the view of different DA to a regular structure, i.e., the circular ring. It’s better to explain the circular ring which indeed is the contour line of similarity.\n3. Why is labeled data used for the monotonic mapping neural network? In my opinion, the consistency loss can be computed directly by ld(x; f, A1) - ld(x; f, A2), where A1 is stronger than A2 from the perspective of Fig. 3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Data augmentation (DA) is important in contrastive learning for visual representation. Various kinds of data augmentation have been elaborately studied. However, how to efficiently compose different kinds of DA is rarely considered. In this paper, a measurement of DA consistency is proposed to make the view of stronger augmentation get smaller similarity with the original sample in the feature space, and the strength of the DA is estimated by an auxiliary network named monotonic mapping neural network (MMNN). Experimental results show that the proposed approach can be used as a plug-in module with different contrastive learning frameworks and achieves performance gain compared with them.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The MoCo, SimSiam, and SupCon are used in Table 1 to show the proposed DA consistent can be used together with different contrastive learning frameworks. All three frameworks are better compared in Table 2 to Table 7. In Table 2, as the SupCon is not compared, I would doubt that the DA consistently didn’t work.  \n2. It’s hard to understand the illustration of dimensional collapse evaluation in Fig. 4(b).\n3. The strength of DA is measured with the length of DA in Section G in the appendix. However, it is measured by the monotonic mapping neural network. It’s not consistent.\n4. Some numbers in the experiment section are non-corresponding, like 83.8 in Table 6 and 83.70 in Table 7.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Despite expressing some concerns in their final evaluation, including w.r.t. the generality/uniformity of the approach across different augmentations, some reported baseline numbers, and the computational cost of the method, the reviewers all acknowledge that the paper introduces some interesting ideas.
Decision: Accept (poster)