LeOCLR: Leveraging Original Images for Contrastive Learning of Visual Representations
Authors: Mohammad Alkhalefi
Georgios Leontidis
Mingjun Zhong
Venue: Accepted by TMLR ()

Abstract: Contrastive instance discrimination methods outperform supervised learning in downstream tasks such as image classification and object detection. However, these methods rely heavily on data augmentation during representation learning, which can lead to suboptimal results if not implemented carefully. A common augmentation technique in contrastive learning is random cropping followed by resizing. This can degrade the quality of representation learning when the two random crops contain distinct semantic content. To tackle this issue, we introduce LeOCLR (Leveraging Original Images for Contrastive Learning of Visual Representations), a framework that employs a novel instance discrimination approach and an adapted loss function. This method prevents the loss of important semantic features caused by mapping different object parts during representation learning. Our experiments demonstrate that LeOCLR consistently improves representation learning across various datasets, outperforming baseline models. For instance, LeOCLR surpasses MoCo-v2 by 5.1% on ImageNet-1K in linear evaluation and outperforms several other methods on transfer learning and object detection tasks

Published in Transactions on Machine Learning Research (10/2024)
LeOCLR: Leveraging Original Images for Contrastive Learn- ing of Visual Representations
Mohammad Alkhalefi
m. m.olkhaleft1.1 @aaiiii.....@@dd.....ooo. @abdnn.ac.
Deportment of Computing Science University of Aberdeen
Georgios Lcontidis
Ctongios.lemmidds@aaaddd.ccc
Deportment Compuling Science & Interdisciplinary Institute University of Aberdeen
Mingjun Zhong
wiingioo..cenng@@@abdd..ccc
Department of Computing Science University of Aberdeen
Reviewed on OpenReview: htips: // openreview. net/ forum? id- y8gGOuUn1r
Abstract
Contrastive instance discrimination methods outperform supervised learning in downstream tasks such as image classification and object detection. However, these methods rely heavily on data augmentation during representation learning, which can lead to suboptimal results if not implemented carefully. A common augmentation technique in contrastive learning is random cropping followed by resizing. This can degrade the quality of representation learning when the two random crops contain distinct semantic content. To tackle this issue, we introduce LeOCLR (Leveraging Original Images for Contrastive Learning of Visual Rep- resentations). a framework that employs a novel instance discrimination approach and an adapted loss function. This method prevents the loss of important semantic features caused by mapping different object parts during representation learning. Our experiments demon- strate that LeOCLR consistently improves representation learning across various datasets, outperforming baseline models. For instance, LeOCLR surpasses MoCo-v2 by 5.1% on ImageNet- 1K in linear evaluation and outperforms several other methods on transfer learn- ing and object detection tasks.
1 Introduction
Self-supervised learning (SSL) approaches based on instance discrimination (Chen et al. 2020b Chen & He
2021 Chen et al.) 2020a Misra Maaten 2020 Grill et al. 2020 heavily rely on data augmentations, such as random cropping, rotation, and colour Jitter, to build invariant representation for all the instances in the dataset. To do so, the two augmented views (positive pairs) for the same instance are attracted in the latent space while avoiding collapse to the trivial solution (representation collapse). These approaches have proven efficient in learning useful representations by using different downstream tasks, e.g., image classification and object detection, proxy evaluations for representation learning. However, these strategies ignore the important fact that the augmented views may have differont semantic content due to random cropping, which can lead to a degradation in visual representation learning (Song el. al. 2023 Zhang el al. 2022 Liu et al. 2020 Mishra et al. 2021). On the one hand, creating positive pairs through random cropping and encouraging the model make them similar based on the shared information the two views makes the
SSL task harder, ultimately improving representation quality (Mishra al. 2021 Chen et al. 2020a). addition, random cropping followed by resizing leads model representation to capture information for the

---

Published in Transactions on Machine Learning Research (10/2024)
object from varying aspect ratios and induce occlusion invariance (Purushwalkam & Gupta 2020 On the other hand, minimizing the feature distance in the latent space (i.e., maximizing similarity) between views
containing distinct semantic concepts tends to result in the loss of valuable image information (P Purushwalkam
& Gupta 2020
Zhang et al. 2022: Song et al. 2023
(a)
(b)
(c)
(d)
Figure 1: Examples of positive pairs that might be created by random cropping and resizing.
Figure T (a and b) show examples of incorrect semantic positive pairs (i.e., positive pairs that contain mismatched semantic information for the same object) that may result from random cropping. In example (a), when the model forced to align the representations of a dog's head and leg closer the latent space, it may discard important semantic features. This happens because the model bases its representations on the shared region between the two views. If this shared region does not contain semantically consistent information, the reprosentations bocome trivial. For random cropping to be effective and achieve occlusion invariance, the shared region must convey the same semantic information in both views. In contrast, Figure (c and d) shows positive pairs where the shared region contains similar semantic content. For example, in (c), both views contain the dog's head, which helps the model capture fcatures of the dog's head across different scales and angles. It worth noting, though, that there is value in contrasting pairs that might include different semantic information about the same object (e-g., dog our case), as can aid learning global features.
As the examples show, creating random crops for one-centric object does not guarantee obtaining correct semantic pairs. This consideration is important for improving representation learning. Instance discrimina- tion SSL approaches, such as MoCo-v2 (Chen et al. 2020b) and SimCLR (Chen et al. 2020a), encourage the model to bring positive pairs, i.c., two views for the same instance, closer in the latent space regardless of their semantic content (Xiao al. 2021 Zhang al. 2022 This could limit the model's ability to learn reprosentations of different object parts and may impair capability learn representations of semantic features (Song al. 2023: Zhang al. 2022 (see Figure (left)).
It has been shown that undesirable views containing different semantic content may be unavoidable when employing random cropping (Song et al. 2023 Therefore, we need method to train the model on different object parts to develop robust represenlations against natural transformations. such as scale and occlusion, rather than simply pulling the augmented views together indiscriminately (Mishra et al.) 2021). This issue should be addressod, as the performance of downstrcam tasks depends on high-quality visual representations learned through superviid learning (Alkhalefi et al.] 2024 Donahue al. 2014 Manová al. 2023
Girshick et al. 2014: Zeiler & Fergus 2014 Kim & Walter 2017 Zhang & Ma
2022
Xiao et. al. 2020
2

---

Published in Transactions on Machine Learning Research (10/2024)
Our work introduces new instance discrimination SSL approach to avoid forcing the model to make similar representations ions for the two positive views regardless of their semantic content. As shown in Figure 2 (right); we include the original image X in the training process, as it encompasses all the semantic features of the views X and X². our approach, the positive pairs (i.e., X and X²) are pulled to the original image X in the latent space, unlike contrastive SOTA approaches like SimCLR (Chen el. al. 2020a) and MoCo-v2 (Chen et al. 2020b), which attract the two views each other. This training method ensures that the information in the shared region between the attracted views (X,X²) and (X,X²) semantically correct. As result, the model learns better semantic features by aligning with the correct semantic content, rather than matching random views that may contain different semantic information. In other words, the model learns the representations of diverse parts of the object because the shared region includes correct semantic parts of the object. This is contrary to other approaches, which may discard important semantic features by incorrectly mapping object parts in positive pairs. Our contributions are as follows:
We introduce a new contrastive instance discrimination SSL method, LeOCLR, designed to reduce the loss of semantic features caused by mapping two random views that are semantically inconsistent.
We show that approach improves visual representation learning in contrastive instance discrim- ination SSL, outperforming state-of-the-art (SOTA) methods across various downstream tasks.
We demonstrate that our approach consistently enhances visual representation learning for con- trastive instance discrimination across different datasets and contrastive mechanisms.
x2
x1
Existing approach
Our approach
Figure 2: The figure on the left shows the embedding space of established approaches (Chen al. 2020a bj where the two views attracted to each other regardless of their content. In contrast, the figure the right illustrates our approach, which clusters the two random views together with the original image in the embedding space.
2
Related Work
SSL approaches are divided into two broad categorics: contrastive and non-contrastive learning. While all these methods aim to bring positive pairs closer in latent. space, each uses a different strategy to avoid representation collapse. This section provides brief overview of some of these approaches, and encourage readers to refer to the respective papers for more details.
Contrastive Learning: Instance discrimination methods, such as SimCLR, MoCo, and PIRL (Chen et al. 2020a; He et al. 2020 Chen et al. 2020b Misra & Maaten 2020) employ a similar idea. They attract the positive pairs together and push the negative pairs apart in the embodding space, albeit through different mechanism. SimCLR (Chen al. (020a) uses an end-to-end approach where large batch size is used for the negative examples and both encoders' parameters in the Siamese network are updated together.
3

---

Published in Transactions on Machine Learning Research (10/2024)
PIRL (Misra & Maaten 2020) uses memory bank for negative examples and both encoders' parameters are updated together. MoCo (Chen et al.) 2020b; He et al. 2020) takes a momentum contrastive approach where the query encoder is updated during backpropagation, and, in turn, updates the key encoder. Negative examples are stored in a separate dictionary, allowing for the use of large batch sizes.
Con-Contrastive Learning: Non-contrastive approaches use only positive pairs to learn visual represen- tations employing different methods to avoid representation collapse. The first category is clustering based methods, where samples with similar features are assigned to the same cluster. DeepCluster (Caron et al, 2018) uses pseudo-lbbels from the previous iteration, which makes it computationally expensive and hard to scale. SWAV (Caron et al., 2020) addresses this issuc by using online clustering, though it requires de- termining the correct number of prototypes. The second category involves knowledge distillation. Methods like BYOL (Grill et al. 2020) and SimSiam (Chen & He 2021 draw on knowledge distillation techniques, where a Siamese network consists of an online encoder and a target encoder. The target network parameters are not updated during backpropagation. Instead, only the online network paramcters are updated while being encouraged to predict the representation of the target network. Despite their promising results, how these methods avoid collapse is not yet fully understood. Inspired by BYOL, Self-distillation with no labels (DINO) CCaron et al. 2021 uses centering and sharpening, along with different backbonn (ViT), which allows it to outperform other self-supervised methods while being more computationally efficient. Another approach, Bag of visual words (Gidaris et al., 2020 2021), uses a teacher-student framework inspired by natural language processing (NLP) to avoid representation collapse. The student network predicts a his- togram of the features for augmented images, similar to the teacher network's histogram. The last category is information maximization. Methods like Barlow twins (Zbontar al. 2021 and VICReg (Bardes et al., 2021 do not require negative examples, stop gradient clustering. Instead, they use regularization to avoid representation collapse. The objective function of these methods tries to eliminate the redundant informa- tion in the embeddings by making the correlation of the embedding vectors closer to the identity matrix. While these methods show promising results, they have limitations, such as the representation learning being sensitive to regularization and decreased effectiveness if certain statistical properties are missing in the data.
Instance Discrimination With Multi-Crops: Different SSL approaches introduce multi-crop methods to enable models to learn visual reprosentations of the object from various perspectives. However, when generating multiple cropped views from the same object instance, these views may contain distinct semantic information. To address this issue, LoGo (Zhang et al. 2022) creates two random global crops and N local views. They assume that global and local views of an object share similar semantic content, increasing similarity between these views. At the same time, they argue that different local views have distinct seman- tie content, thus reducing similarity among them. SCFS (Song et al., 2023) proposes a different solution for handling unmatched semantic views, scarching for semantic-consistent features between the contrasted views. CLSA (Wang& & Qi 2022) generates multi-crops and applies both strong and weak augmentations to them, using distance divergence loss to enhance instance discrimination in representation learning. Previous approaches assume that global views contain similar semantic content and treat them indiscriminately as positive pairs. However, our approach suggests that global views may contain incorrect semantic pairs due to random cropping, as shown in Figure Therefore, we aim to attract the two global views to the original (intact and uncropped) image, which fully encapsulates the semantic features of the crops.
3 Methodology
Mapping incorrect semantic positive pairs (i.e., positive pairs containing different semantic views) leads to the loss of semantic features, which degrades model representation learning (Mishra et al., 2021 Purushwalkam & Guptal 2020 Song et al. 2023). To address this, we introduce new contrastive instance discrimination SSL strategy called LeOCLR. Our approach aims to capture meaningful features from two random positive pairs, even if they contain different semantic content, to enhance represenlation learning. To achieve this, it is crucial to ensure that the information in the shared region between the attracted views is semantically correct. This is because the choice of views controls the information captured by the representations lcarned in contrastive learning (Tian et al.) 2020). Since we cannot guarantee that the shared region between the two views includes correct semantic parts of the object, we propose to involve the original image in the training
4

---

Published in Transactions on Machine Learning Research (10/2024)
process. The original image X, which remains uncropped (i.e., random cropping), encompasses all the semantic features of the two cropped views, X and X².
Encoder
X
t-T
fq
NG
X
Momentum Update
L
RC1
t
T-T
RC2
X2
fx
Momentum Encoder
Embedding Space
Figure 3: LeOCLR: Overview of the proposed approach. The left part illustrates that the original image X is not cropped (NC), but is resized 224 224, before applying transformations. The other views (X' and X2) are randomly cropped (RC1 and RC2) and resized 224 224, followed by the application of transformations. The embedding space of our approach is depicted the right side of the Figure.
As shown in Figure 3 (left), our methodology generates three views (X, X and X²). The original image (X) is resized without cropping, while the other views (X1 and X²) are randomly cropped and resized. All views are then randomly augmented to provent the model from lcarning trivial features. We use data augmentations similar to those employed MoCo-v2 (Chen et al. 2020b). The original image (X) is encoded by the encoder fq, while the two views (X), xx² are encoded by a momentum encoder fss whose parameters are updated using the following formula:
0g + m8 (1 m)0g
(1)
where m is a coefficient set to 0.999, 0g are encoder parameters of updated through backpropagation, and 0g are momentum encoder parameters of fe updated by 0g. Finally, the objective function forces the model to pull both views all other instances, as illustrated in Figure 3 (right).
closer to the original image (X) in the embedding space, while pushing apart
3.1 Loss function
Firstly, briefly describe the loss function of MoCo-v2 (Chen et al. 2020b), since we momentum contrastive learning in approach. We will then explain modification to the loss function.
(u, log exp(u o /T)
2
Σ exp(u - vn/T)
where similarity is measured by the dot product. The objective function increases the similarity between the positive pairs (u u+) by bringing them closer in the embedding space, while pushing apart all the negative samples (v,) in the dictionary to avoid representation collapse. is the temperature hyperparameter of softmax. In our approach, we increase the similarity between the original image (i.c., query's feature representation) u = (x) with the positive pair (i.e., key's feature representation) at = fa(x*) (i = 1,2) and push apart all the negative examples (w,). Therefore the total loss for the mini-batch is:
= Σ lu, sg( + E(u, sg(1)))
(3)
5

---

Published in Transactions on Machine Learning Research (10/2024)
where sg()) denotes the stop-gradient trick, which is crucial for avoiding representation collapse. As shown in Equation 3 the total loss 4, attracts the two views (ut and of) to their original instance u4, This allows the model to capture semantic features from the two random views, even if they contain distinct semantic information. Our approach captures better semantic features than previous contrastive approaches (Chen He et al.) 2020), as we ensure that the shared region between the attracted views contains correct semantic information. Since the original image contains all parts of the object, any part contained the random crop is also present in the original image. Therefore, when we pull the original image and the two random views closer in the embedding space, the model learns representations of the different parts, creating ooclusion-invariant representation of the object across varying scales and angles. This is in contrast to previous approaches, which pull the two views together in the embedding space regardless of their semantic content, leading to the loss of semantic features (Liu et. al. 2020 Purushwalkam & Gupla 2020 Song et al.
et al. 2020a b
2023 (see Algorithm for the implementation of our approach).
A Algorithm 1 Proposed Approach
1: for X in dataloader do
2:
X2, x²= rc(X)
b random crop first and second views apply random augmentation for all the views D encode the original image encode the first view by momentum encoder encode the second view by momentum encoder >> computed shown in eq.1 b computed as shown in eq.1  computed the total loss as shown in eq.2
3:
X,X", ,x2 axgment(XX,X,_aaa
4:
X =X)(X)
X fx(X))
X (x(22)
lossl )
8: loss2 = 2
9: 4 loss1 + loss2
10: end for
11:
12: def rc(x):
13: x= -.T.RandommessieeCCC0(2212,,..1
T transformation from torchvision module
14: return
Equation and Algorithm 1 illustrate the key differences between our approach and previous multi-crop
approaches, such as CLSA (Wang & Qi 2022). SCFC (Song al. 2023 and DINO (Caron al. 2021 The key differences are follows:
Previous approaches assume that two global views contain the same semantic information, encourag- ing the model to focus on similarities and create similar representations for both views. In contrast, our approach uses the original images instead global views, as we argue that global views may con- tain incorrect semantic information for the same object. While they may help capture some global features, this could limit the model's ability to learn more universally useful semantic features, ultimately affecting performance.
Previous approaches use several local random crops, which might be time- and memory-intensive, while our approach uses only two random crops (Caron et al. 2020 Wang & Qi 2022
Our objective function employs different methods to enhance the model's visual representation learn- ing. We encourage the model to make the two random crops similar to the original image, which contains the semantic information for all random crops while avoiding forcing the two crops to have similar representations if they do not share similar semantic information. This approach differs from previous methods, which encourage all crops (global and local) to have similar representations re- gardless of their semantic information. As result, although useful for learning some global features, those methods may discard relevant semantic information, potentially hindering the transferability of the resulting representations to downstream tasks.

---

Published in Transactions on Machine Learning Research (10/2024)
4 Experiments and Results
Datasets: We conducted multiple experiments on three datasets: STL-10 'unlabeled" with 100K training images (Coates & Ng 2011), CIFAR-10 with 50K training images Krizhevsky 2009 and ImageNet-IK with 1.28M training images [Russakovsky et al., 2015).
Training Sctup: We used ResNet50 as the backbone, and the model was trained with the SGD optimizer, with a weight decay 0.0001, momentum of 0.9, and an initial learning rate of 0.03. The mini-batch size was 256, and the model
trained for up to 800 epochs ImageNet-1K.
Evaluation: We used different downstream tasks to evaluate the LeOCLR's representation learning against leading SOTA approaches on ImagcNet-1K:: linear evaluation, scmi-supervised learning, transfer learning, and object detection. For linear evaluation, we followed the standard evaluation prolocol (Chen et al. 2020a He et al. 2020 Huynh et al., 2022 Dwibedi et al. 2021), where linear classifier was trained for 100 epochs on top frozen backbone pre-trained with LcOCLR. The ImageNet-1K training set was used to train the linear classifier from scratch, with random cropping and left-to-right flipping angmentations. Results are reported the ImageNet-1K validation set using center crop (224 224). Tn the semi-supervised setting, fine-tuned the network for 60 epochs using 1% labeled data and 30 epochs using 10% labeled data, Also, we evaluated the learned features on smaller datasets, such as CIFAR (Krizhevsky 2009), and fine-grained datasets (Krause et al. 2013; Parkhi et al.; 2012 Berg et al. 2014), using transfer learning. Finally, we use the PASCAL VOC (Excringham et al. 2010) dataset for object detection.
Comparing with SOTA Approaches: We used vanilla MoCo-v2 (Chen et al. 2020b) as a baseline to compare with our approach across different benchmark datasets, given our use of momentum contrastive learning framework. Additionally, compared our approach with other SOTA methods the ImageNet-1K dataset.
Table 1: Comparisons between approach LeOCLR and SOTA approaches on ImageNet-1KK IK..
Approach
Epochs
Batch
Accuracy
MoCo-v2
Chen et al.
2020b
800
256
71.1%
BYOL
Grill et al.
2020
1000
4096
74.4%
SWAV
(Caron
2020
800
4096
75.3%
SimCLR
CChen al.
2020a
1000
4096
69.3%
HEXA
Li et al.
2020)
800
256
71.7%
SimSiam Chen & He
2021
800
512
71.3%
VICReg Bardes et al.
2021
1000
2048
73.2%
MixSiam Guo et. al.
2021
800
128
72.3%
OBoW Gidaris al.
202
200
256
73.8%
DINO Caron et al. 2021
800
1024
75.3%
Barlow Twins (Zbontar al.
2021
1000
2048
73.2%
CLSA (Wang & Qi
2022
800
256
76.2%
RegionCL-M [Xu a.l
2022
800
256
73.9%
UnMix Shen et al. 2022
800
256
71.8%
HCSC Guo et a..
2022
200
256
73.3%
UniVIP Li et al. 2022
300
4096
74.2%
HATEV Zhang
Maj
2022
200
256
70.1%
SCFS
Song et al.
2023
800
1024
75.7%
LeOCLR (ours)
800
256
76.2%
Table presents the linear evaluation of our approach compared other SOTA methods. As shown, approach outperforms all others, surpassing the bascline (i.c., vanilla MoCo-v2) by 5.1%. This supports our hypothesis that while two global views can capture some global features, they may also contain different semantic information for the same object (e.g., a dog's head versus its leg), which should considered to improve representation learning. The observed performance gap (i.c., the difference between vanilla MoCo- and LeOCLR) demonstrates that mapping pairs with divergent semantic content hinders representation learning and affects the model's performance in downstream tasks.

---

Published in Transactions on Machine Learning Research (10/2024)
Semi-Supervised Learning ImageNet-1K: In this part, evaluate the performance of LeOCLR un- der a semi-supervisel setting. Specifically, we use 1% and 10% of the labeled training data from ImageNet-1K for fine-tuning, following the semi-supervised protocol introduced in SimCLR (Chen et al. 2020a). The top-1 accuracy, reported Table 2 after fino-tuning with 1% and 10% of the training data, demonstrates LeO- CLR's s superiority over all compared methods. This can be attributed to LeOCLR's improved representation learning capabilities, especially compared other SOTA methods.
Table 2: Semi-supervised training results on ImageNet- IK: Top-1 performances are reported for fine-tuning pro-trained ResNet-50 with the ImageNet-11g 1% and 10% datascts. denotes the results are reproduced in this study.
Approach Fraction
ImageNet-IK 1%
ImageNet-IIe 1K 10%
MoCo-v2 Chen et al.
2020b
47.6%
64.8%
SimCLR Chen et al.
2020a
48.3%
65.6%
BYOL Grill et al.
2020
53.2%
68.8%
SWAV Caron et a..
2020
53.9%
70.2%
DINO
Caron et a.. 2021
50.2%
69.3%
Region CL
C
M [u al.
2022
46.1%
60.4%
SCFS (Sng et al.)
2023
54.3%
70.5%
LeOCLR (ours)
62.8%
71.5%
Transfer Learning on Downstream Tasks: We evaluate our self-supervised pretrained model using transfer learning by fine-tuning on small datasets such as CIFAR (Krizhevsky 2009 Stanford Cars (Krause
et al. 2013), Oxford-IIIT Pets (Parkhi et al., 2012), and Birdsnap (Berg
2014]. We follow the transfer learning procedures outlined (Chen et al. 2020a Grill et al. 2020 find optimal hyperparameters for each downstream task. As shown in Table various downstream tasks. This demonstrates that model learns useful semantic features, enabling it to generalize better to unseen data in different downstream tasks compared to other approaches. Our method preserves the semantic features of the given objects, thereby improving the model's representation learning capabilitics. As result, it more effective at extracting important features and predicting correct classes on transferred lasks.
approach, LeOCLR, outperforms all compared approaches
Table 3: Transfer learning results from ImageNet- IK with the standard ResNet-50 architecture. denotes the results are reproduced in this study.
Approach
CIFAR-10
CIFAR-100
Car
Birdsnap
Pets
MoCo-v2
Chen el
2020b
97.2%
85.6%
91.2%
75.6%
90.3%
SimCLR
Chen et al.
2020a.
97.7%
85.9%
91.3%
75.9%
89.2%
BYOL
Grill et al.]
2020)
97.8%
86.1%
91.6%
76.3%
91.7%
DINO
Caron el al.
2021
97.7%
86.6%
91.1%
91.5%
SCFS
Song et a..
2023
97.8%
86.7%
91.6%
91.9%
LeOCLR (ours)
98.1%
86.9%
91.6%
76.8%
92.1%
Object Detection Task: To further evaluate the transferability of the learned representation, compare approach with other SOTA approaches using object detection the PASCAL VOC. We follow the same settings as MoCo-v2 (Chen et al. 2020b), finc-tuning the VOCO7+12 trainval dataset using Faster R-CNN with an R50-C4 backbone, and evaluating on VOCO7 test datasel. The model is fine-tuned for 24k iterations (+= 23 epochs). As shown in Table ! our method outperforms all compared approaches. This superior performance can be attributed to our model's ability to capture richer semantic features comparod the bascline (MoCo-v2) and other approaches, leading to better results in object detection and related tasks.

---

Published in Transactions on Machine Learning Research (10/2024)
Table 4: Results (Average Precision) for PASCAL VOC object detection using Faster R-CNN with ResNet- 50-C4.
Approach
50
AP
AP7s
MoCo-v2 (Chen et al.
2020b
82.5%
57.4%
64%
CLSA (Wang & Qi
2022
83.2%
SCFS (Song et al
2023
83%
57.4%
63.6%
LeOCLR (ours)
83.2%
57.5%
64.2%
5 Ablation Studies
In the following subscctions, we further analyze our approach using another contrastive instance discrimi- nation approach (i.e., end-to-end mechanism) to explore how our method performs within this framework. Additionally, we perform studies on the benchmark datasets STL-10 and CIFAR-10 10 using a different backbone (ResNet-18) to assess the consistency of our approach across various datasets and backbones. Furthermore, employ random crop test to simulate natural transformations, such as variations in scale occlusion of objects in the image, to analyze the robustness of the features learned by our approach, LeOCI.. We also compare our approach with vanilla MoCo-v2 by manipulating their data augmentation techniques to determinc which model's performance is more significantly affected by the removal of certain augmentations. In addition, we experiment with different fine-tuning settings to evaluate which model learns better and faster. Furthermore, we adapt the attraction strategy and cropping method of the original image, as well as compute the rumning time of our approach. Finally, we examine our approach on non-centrie object dataset where the probability of mapping two views containing distinct information is higher.
5.1 Different Contrastive Instance Discrimination Framework
We utilize an end-to-end framework in which the two encoders fy and fs updated via backpropagation to train model with our approach for 200 epochs and 256 batch size. Following this, we conduct a linear evaluation of our model against SimCLR, which also employs end-to-end mechanism. As shown in Table our approach outperforms vanilla SimCLR by a significant margin of 3.5%, demonstrating its suitability for integration with varions contrastive learning frameworks.
Table Comparing vanilla SimCLR with LeOCLR after training our approach 200 epochs on ImageNet- Approach
ImageNet-1K
SimCLR (Chen et al. 2020a LeOCLR (ours)
62%
65.5%
5.2 Scalability
In Table we evaluate our approach on different datasets (STL-10 and CIFAR-10) using ResNet-18 backbone ensure its consistency across various backbones and datasets (i.e., scalability). We pre-trained all the approaches for 800 epochs with batch size 256 both datasets and then conducted linear evaluation. Our approach demonstrates superior performance on both datasets compared to all approaches. For instance,
Table 6: SOTA approaches versus LeOCLR on CIFAR-10 and STL-10 with ResNet-18. Approach
STT-10
CIFAR-10
MoCo-v2 (Chen al.
2020b)
80.08%
73.88%
DINO
Caron et al.
2021
84.30%
78.50%
CLSA Wang & Qi
2022
82.62%
77.20%
BYOL
Grill et al.
2020
79.90%
73.00%
LeOCLR (ours)
85.20%
79.59%

---

Published in Transactions on Machine Learning Research (10/2024)
our approach outperforms vanilla MoCo-v2, achieving accuracies of 5.12% and 5.71% on STL-10 and CIFAR- 10, respectively.
5.3 Center and Random Crop Test
In Table we reported the top-1 accuracy for vanilla MoCo-v2 and our approach after 200 epochs on ImageNet-1K, focusing two tasks: a) center crop test, similar to (Chen et al., 2020a b) where images are resized lo 256 pixels along the shorter side using bicubic resampling, followed by a 224 x 224 center crop; and b) random crop, where images are resized to 256 x 256 and then randomly cropped and resized to 224 x 224. We obtained the MoCo-v2 center crop result directly from (Chen et al. 2020bb, but the random crop result was not reported. To ensure a fair comparison when reporting the random crop results, we replicated MoCo-v2 using the same hyperparameters from the original paper that were used to report the center crop. According the results, the performance of MoCo-v2 dropped by 4.3% with random cropping, whereas approach experienced smaller drop of 2.8%. This suggests that approach learns better semantic features, demonstrating greater invariance to natural transformations such as occlusion and variations in object scales, Additionally, we compare the performance of CLSA (Wang & Qi 2022) with our approach, given that both perform similarly after 800 epochs (see Table 1 Note that the CLSA approach uses multi-crop (i.e., five strong and two weak augmentations), while our approach employs only two random crops and the original image. As shown Table 7 LeOCLR outperforms the CLSA approach by 2.3% after 200 epochs on ImageNet-1K. To address concerns about the increased computational cost associated with training LeOCLR compared to MoCo V2, we include the training time for both approaches in Table We trained both models three A100 GPUs with 80GB for 200 epochs. Our approach took additional 13 hours to train over the same number of epochs, but it delivers significantly better performance than the baseline.
Table 7: Comparing LeOCLR with vanilla M2 and CLSA after training 200 epochs ImageNet- Center Crop
Approach
Random Crop
Time
MoCo-v2 (Chen et al. 2020b)
67.5%
63.2%
68h
CLSA (Wang & Qi
2022
69.4%
LeOCLR (ours)
71.7%
68.9%
81h
5.4 Augmentation and Fine-tuning
Contrastive instance discrimination approaches are sensitive to the choice of image augmentations Grill ct al. 2020). This sensitivity necessitates further analysis comparing our approach to Moco-v2 (Chen ct al. 2020b). These experiments aim to explore which model learns better semantic features and produces more robust representations under different data augmentations. As shown in Figure ] both models are affected by the removal of certain data augmentations. However, our approach shows more invariant representation and exhibits less performance degradation due to transformation manipulation compared to vanilla MoCo- v2. For instance, when we apply only random cropping augmentation, the performance of vanilla MoCo-v2 drops by 28 percentage points (from baseline of 67.5% to 39.5% with only random cropping). In contrast, our approach experiences a decrease of only 25 percentage points (from a baseline of 71.7% to 46.6% with only random cropping). This indicates that our approach learns better semantic features and produces more effective representations for the given objects than vanilla MoCo-v2.
In Table 2 presented in Section 4, we fine-tune the representations over the 1% and 10% ImageNet- 1K splits from (Chen et al. 2020a) using the ResNet-50 architecture. In the ablation study, we compare the tunet reprosentations of our approach with the reproduced vanilla MoCo-v2(Chen et al. 2020b) across 1 1%, 2%. 5%, 10%, 20%, 50,, and 100% of the ImageNet- dataset, following the methodology in (Henall 2020: Grill et al. 2020). In this setting, we observe that tuning a LeOCLR representation consistently outperforms vanilla MoCo-v2. For instance, Figure (a) demonstrates that LcOCLR finc-tuned with 10% of ImageNet- 1K labeled data outperforms vanilla Moco-v2 (Chen et al. 2020b) finc-tuned with 20% of labeled data. This indicates that our approach is ad advantageous when the labeled data for downstream tasks is limited.
10

---

Published in Transactions on Machine Learning Research (10/2024)
Decrease accuracy from baseline
LeOCLR
-5
-15
-20
-25
Baseline
grayscale
color Augmentation
Crop
Crop.only
Figure 4: Decrease in top-1 accuracy (in % points) of LeOCLR and our reproduction of vanilla MoCo-v2 after 200 epochs, under linear evaluation on ImageNet-IK, Grayscale refers results without grayscale augmentations, while R color refers to results without color jitter but with grayscale augmentations.
LeOCLR
LeOCLR
ImageNet Top-1 Accuracy (%)
McCov2
ImageNet Top-5 Accuracy (%)
92.5
MoCov2
75
90.0
70
87.5
65
77.5
75.0
10
100
Percentage of training data
Percentage of training data
(a) Top-1 accuracy
(b) Top-5 accuracy
Figure Semi-supervised training with fraction of ImageNet-1K labels on ResNet-50.
5.5 Attraction Strategy
In this subsection, we apply random crop to the original image (x) and attract the two views (x+ (x²) toward it evaluate its impact on our approach's performance. We also conducted an experiment where all views were attracted to each other. However, in our method, avoid attracting the two views to each other, enforcing the model to draw the two views toward the original image only (ie., the uncropped image containing semantic foatures for all crops). For these experiments, we pre-trained the model on ImageNet-1K for 200 epochs using the same hyperparameters employed in the main experiment.
The experiments in Table g underscore the significance the information shared between the two views. They also highlight the importance of leveraging the original image and avoiding the attraction of views containing varied semantic information to preserve the semantic features of the objects. When we create random crop for the original image (x) and enforce the model to make the two views similar to the
11

---

Published in Transactions on Machine Learning Research (10/2024)
Table Comparisons of augmentation strategies using our proposed approach after 200 epochs. Approach
Accuracy
LeOCLR (Random original image) LeOCLR (attract all crops)
69.3%
67.7%
LeOCLR (ours)
71.7%
original image (i.e., SEOCCLR(Raddoo original image)), the model performance decreases by 2.4%. This drop occurs because cropping the original image and enforcing the model to attract the two views toward it increases the likelihood of having two views with varied semantic information, leading to a loss of semantic features of the objects. The situation worsens when we attract all views (x,x" (x,) to each other in LeOCLR (attract all crops), causing performance to drop closer to that of vanilla MoCo-v2 (67.5%). This is due to the high probability attracting two views containing distinct semantic information.
5.6 Non-Object-Centric Tasks
Non-object-centric datasets, such COCO (Lin et al. 2014), portray real scenes where the objects of interest are not centered or prominently situated, as opposed to object-centric datasets like ImageNet- this case, the probability of creating two views containing distinct semantic information for the object is higher, exacerbating the problem of losing semantic features. Therefore, we train both our approach and the MoCo-v2 bascline from scratch on the COCO dataset to evaluate how our method handles the discarding of semantic features in such datasets. We used the same hyperparameters for ImageNet-1K, training the models with a batch size of 256 over 500 epochs. Subsequently, fine-tuned these pre-trained models the COCO dataset for object detection.
Table 9: Results for pre-training followed by fine-tuning on COCO for object detection using Faster R-CNN with ResNet-50-C4.
Approach
APso
AP
AP15
MoCo-v2 (Chen et al. 2020b
57.2%
37.6%
41.5%
LeOCLR (ours)
59.3%
39.1%
43.0%
Table g shows that our approach captured better semantic features for the given object than the bascline. This highlights that our method avoiding the attraction of two distinct views is more effective at preserving semantic features, even a non-object-centric dataset.
6 Conclusion
This paper introduces novel contrastive instance discrimination approach for SSL to enhance repre- sentation learning. Our method mitigates the loss of semantic features by incorporating the original image during training, even when the two views contain distinct semantic content. We demonstrate that our approach consistently improves the representation learning of contrastive instance discrimination in various benchmark datasets, backbones, and mechanisms, such momentum contrast and end-to-cnd methods. In linear evaluation, we achieved an accuracy of 76.2% on ImageNet-1K after 800 epochs, outperforming several SOTA instance discrimination SSL approaches. Additionally, we demonstrated the invariance and robustness our approach across different downstream tasks, including transfer lcarning and semi-supervised fine-tuning.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Acknowledgments
We would like to thank the University of Aberdeen's High Performance Computing facility for enabling this work and the anonymous reviewers for their constructive leedback.
References
Mohammad Alkhalefi, Georgios Leontidis, and Mingjun Zhong. Semantic positive pairs for enhancing visual representation learning of instance discrimination methods. Transactions on Machine Learning Research, 2024.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variace-mmarinne--vvvaaac regularization for self-supervised learning. arXiv preprint arXiv:2105.0400 04906, 2021.
Thomas Berg, Jiongxin Liu, Soung Woo Lee, Michelle I. Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2011-2018, 2014.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132 149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:0912-9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervieed vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Xinlci Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. arXio preprind arXiv:2003.04290 2020b.
Adam Coates and Andrew Y Ng. Analysis of large-scale visual recognition. In Advances in neural information processing systems, pp. 284 292, 2011.
Jeff Donahuc, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pp. 647-655. PMLR, 2014.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the ICEE/CVE International Conference on Compuler Vision, pp. 9588 9588-9597, 2021.
Mark Everingham, Lue Van Gool, Christopher KT Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303 -3300,33,, 2010.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthicu Cord. Learning represen- tations by predicting bags of visual words. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6928 6938, 2020.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthicu Cord, and Patrick Perez. Obow: Online bag-of-visuullwords generation for self-supervised learning. In Proceedings of the IEEE/CVP Con- ference on Computer Vision and Pattern Recognition, pp. 6830-68388800 6840, 2021.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE conference on compuler vision and pottern recognition, pp. 580-587, 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 1001222--12844 21224, 2020.
Xiaoyang Guo, Tianhao Zhao, Yutian Lin, and Bo Du. Mixsiam: a mixture-based approach self-supervised representation learning. arXiv preprint 21v: 111.016999 2021.
Yuanfan Guo, Minghao Xu, Jiawen Li, Bingbing Ni, Xuanyu Zhu, Zhenbang Sun, and Yi Xu. Hesc: Hierar- chical contrastive selective coding, In Proceedings of the IEEE/CVI Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 9706-99 9715, June 2022.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International confer- ence machine learning, pp. 4182 4192. PMLR, 2020.
Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. Boosting contrastive self-supervised learning with false negative cancellation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2785 2795, 2022.
Dong-Ki Kim and Matthew Walter. Satellite image- based localization via learned embeddings. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp.
TEEE, 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fino-grained cate- gorization. In 2013 IEEE International Conference on Computer Vision Workshops, pp. 554-561, 2013. doi: 10.1109/ICCVW 2013.77.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Cileseer, 2009.
Chunyuan Li, Xiujun Li, Lei Zhang, Baolin Peng, Mingyuan Zhou, and Jianfeng Gao. Self-supervised pro-training with hard examples improves visual representations. arXiv preprint arXi::2012.13405 2020.
Zhaowen Li, Yousong Zhu, Fan Yang, Wei Li, Chaoyang Zhao, Yingving Chen, Zhiyang Chen, Jiahao Xie, Liwei Wu, Rui Zhao, et al. Univip: A unified framework for self-supervised visual pre-training. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14627-14636, 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 19th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Songtao Liu, Zeming Li, and Jian Sun. Self-emd: Self-supervised object detection without imagenet. arXiv preprint ar 1rXi:22011.13677 133677, 2020.
Alžbëta Manová, Aiden Durrant, and Georgios Leontidis. S-jea: Stacked joint embedding architectures for self-supervised visual representation learning. arXiv preprint arXio:2305.1 1701, 2023.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Shlok Mishra, Anshul Shah, Ankan Bansal, Abhyuday Jagamnatha, Abhishek Sharma, David Jacobs, and Dilip Krishnan. Object-aware cropping for self-supervised learning. ar preprint. orXiu:2 20t::211200033 12.00319 2021.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarien representations. In Proceedings of the SEEE/CVE conference on computer vision and pottern recognition, pp. 6707-6717, 2020. 2020.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pp. 3498 3505. LEEE, 2012.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biascs. Advances in Newral Information Processing Systems, 33:3407-3418, 2020.
Olga Russakovsky, Jia Dong, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211 (222222111222, 2015. doi: 10.1007 /11007/1110 /t////1122333 111263-015-0816..
Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing. Un-mix: Rethinking image mixtures for unsupervised visual representation learning. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 36, pp. 2216-2224, 2224, 2022.
Kaiyou Song, Shan Zhang, Zimeng Luo, Tong Wang, and Jin Xie. Semantics-consisteni feature search for self-supervised visual representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16099-16108, October 2023.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Newral Information Processing Systems, 33:6827 23:6827-6839,
2020.
Xiao Wang and Guo-Jun Qi. Contrastive learning with stronger augmentations. IEEE transactions pattern analysis and machine intelligence, (5(5):5549-5566) 2022.
Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. arxiv preprint arXiw:2008. N00:2008800659, 2020.
Tete Xiao, Colorado J Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity repre- sentation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10539-10548, 2021.
Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Regionel: exploring contrastive region pairs for self- supervised representation learning, In European Conference on Computer Vision, pp. 477 494. Springer, 2022.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy roduction. In International Conference on Machine Learning, pp. 12310-12320. PMLR, 2021.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCo ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 818 833. Springer, 2014.
Junbo Zhang and Kaisheng Ma. Rethinking the augmentation module in contrastive learning: Learning hierarchical augmentation invariance with expanded views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16650 16659, 2022.
Tong Zhang, Congpei Qiu, Wei Ke, Sabine Süsstrunk, and Mathieu Salzmann. Leverage your local and global representations: A new self-supervised learning strategy. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognilion, pp. 18580-16556 2022.
15


Reviews:
[{"review_id": "PLaWE4INMm", "reviewer": "Reviewer_rwjE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The writing of the paper is clear, with the intuition behind the design of the method well explained and making sense.\n2. The research problem is important, as unmeaningful positive pairs can impact the robustness and reliability of the trained model.\n\nWeaknesses:\n\n1. One of the fundamental assumptions of strong augmentations is that the semantic information is not completely distorted. For example, in the illustration, although the two crops capture different parts of the dog, from a human perspective, it is still easy to recognize that the cropped components belong to a dog (i.e., the semantic information is not entirely distorted). The authors should demonstrate why less related positive pairs are not beneficial to contrastive training, as many studies have shown that this is actually advantageous.\n2. The proposed method essentially creates a weak augmentation (the original image) and contrasts it with its strong augmentations twice. I fail to see the benefit of doing this.\n3. There are prior works that have investigated the problem of semantic distortions during strong augmentations for contrastive learning [1,2]. It is recommended to discuss the relationship and superiority of the proposed method to prior approaches, and to include comparisons in the experiments.\n\n[1] Xiao, T., Wang, X., Efros, A.A. and Darrell, T., What Should Not Be Contrastive in Contrastive Learning. In International Conference on Learning Representations.\n\n[2] Zhang, J. and Ma, K., 2022. Rethinking the augmentation module in contrastive learning: Learning hierarchical augmentation invariance with expanded views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16650-16659).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\n\nunsupervised contrastive learning usually relies heavily on data augmentations. In this paper, the authors suggest that one common augmentation strategy—random cropping—might negatively impact contrastive learning because the two augmented views after cropping might contain different semantic information. As a result, encouraging them to become closer can lead the model to learn incorrect semantic representations. To address this issue, the authors propose a new training/augmentation paradigm, which utilizes the original image as the anchor in contrastive learning. Instead of pulling the two cropped views towards each other, they are pulled towards the anchor image in the latent space. The experimental results show that the proposed training paradigm outperforms several classic contrastive learning frameworks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0DfjzYGbOD", "reviewer": "Reviewer_NHig", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nInnovative Approach: The introduction of leveraging original images in the training process to preserve semantic features is a significant improvement over traditional methods.\nConsistent Performance: The proposed method shows consistent improvement across different datasets and tasks, including linear evaluation, semi-supervised learning, and transfer learning.\nRobust Evaluation: The paper provides a comprehensive evaluation of the method, comparing it with various state-of-the-art approaches and demonstrating its effectiveness in different settings.\nClear Methodology: The methodology and the rationale behind the approach are well-explained, making it easier for readers to understand the improvements made.\nWeakness:\nLimited Ablation Studies: While the paper includes some ablation studies, more extensive ablations could strengthen the claims about the individual contributions of each component of the proposed method.\nScalability Concerns: The paper does not discuss the computational complexity or scalability of the proposed approach in detail, which could be a concern for large-scale applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel self-supervised learning approach that addresses the limitations of current contrastive instance discrimination methods. Traditional methods like MoCo-v2 and SimCLR rely heavily on data augmentation, particularly random cropping, which can result in incorrect semantic pairs and degraded representation learning. LeOCLR mitigates this by including the original image in the training process, ensuring that the shared semantic features between augmented views are preserved. This method consistently outperforms existing state-of-the-art approaches across various datasets and downstream tasks, demonstrating superior representation learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Extended Ablation Studies: Include more detailed ablation studies to isolate and evaluate the impact of each component of the proposed method.\n\nComputational Complexity Analysis: Provide a discussion on the computational complexity and scalability of LeOCLR, particularly in comparison to other state-of-the-art methods.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1WRmfGc63S", "reviewer": "Reviewer_mf3A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n\nThe proposed method is simple and straightforward to implement.\n\nThe review of SSL methods is detailed and well-written.\n\n**Weakness:**\n\nSome explanations for the motivation of the method are unclear (see requested changes).\n\nThere is insufficient evidence demonstrating the effectiveness of the proposed method.\n\nThe paper appears to lack thorough proofreading, as spelling and grammar mistakes, misuse of capitalization, and incorrect usage of \\citep/\\citet, and parentheses are frequently observed. The paper could benefit from improved writing and presentation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Conventional contrastive learning methods use two randomly cropped and resized images as a positive pair during training. This work argues that these images may contain different semantic information, leading to sub-optimal representation learning. To address this issue, the paper proposes using one original, uncropped image along with a randomly cropped version. Experiments with linear probing and in semi-supervised settings demonstrate that the proposed method outperforms existing approaches on ImageNet and other downstream tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There is a ratio parameter when using RandomResizedCrop, which sets the 'lower and upper bounds for the random aspect ratio of the crop.' Can adjusting this parameter address the issue discussed in this paper?\n\nConventional contrastive methods generate one positive pair per training sample, whereas LeOCLR uses two positive pairs per training sample. The experiments and ablation studies do not rule out the possibility that the improvement from LeOCLR is simply due to using an additional positive pair. Would it be a fairer comparison if other methods also used two positive pairs per training sample? I hope the authors can either justify not performing such a comparison or include additional results on modified baseline methods.\n\nThe paper repeatedly uses the terms 'correct' and 'incorrect' semantic positive pairs. There is nothing inherently incorrect about one image focusing on the leg of the dog and another on the head, as both are representations of the dog. The paper mentions occlusion invariance. My understanding is that different views of the dog are precisely how occlusion invariance is achieved. In Figure 1, conventional methods treat all four as positive pairs, and my understanding is that the real issue is that images with different semantic information are treated indiscriminately (also mentioned in Song et al.). I hope the authors can clarify this.\n\nFirst sentence of Section 3 requires either a reference or should be verified in the paper.\n\nPlease carefully review all references, as some of them are either missing information or straight up incorrect: Leverage Your Local and Global Representations: A New Self-Supervised Learning Strategy should be Zhang et al, instead of Qiu et al.\n\nIs $L_t$ in Figure 3 the TotalLoss in (3)? Please ensure that the notation for such crucial definitions is clear throughout the paper.\n\nPlease fix the hyperlink at the end of page 5 and at the begining of page 6.\n\nIn the second bullet point before Section 4, it is mentioned that there might be time and memory issues with multiple crops. Can the authors verify this claim?", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel approach to self-supervised learning called LeOCLR that addresses limitations in current contrastive instance discrimination methods. The quality of the work is demonstrated through:

- Experiments across various datasets and tasks
- Performance improvements over state-of-the-art methods
- Some theoretical justification for the proposed approach

*Clarity:*
While the overall methodology is well-explained, there are some areas that could benefit from improved clarity:

- Some explanations of the motivation could be clearer
- The paper would benefit from more thorough proofreading to address spelling, grammar, and formatting issues

*Originality:*
The work shows originality in its approach to preserving semantic features in contrastive learning:

- Novel use of the original, uncropped image in the training process
- New attraction strategy and loss function to address semantic feature loss

*Significance:*
- Consistent improvement over existing methods across multiple tasks and datasets
- Addressing a fundamental issue in contrastive learning approaches
- Potential for broader impact in self-supervised learning applications

*Pros:*

- Novel approach to preserving semantic features in contrastive learning
- Performance improvements across various tasks and datasets
- Some theoretical foundation

Cons:

- Some clarity issues in explanations and writing
- Limited ablation studies
- Lack of detailed computational complexity analysis

*Reasoning behind decision:*

The paper presents a contribution to the field of self-supervised learning. The proposed LeOCLR method addresses a limitation in current contrastive learning approaches and demonstrates consistent performance improvements. The theoretical justification is sound, and the experimental results are convincing. While there are some minor issues with clarity and presentation, these do not significantly detract from the overall quality and importance of the work.

The reviewers' discussions highlight the strengths of the paper in terms of its innovative approach and consistent performance improvements. While there were some concerns raised about clarity and the need for more ablation studies, the authors have addressed many of these issues in their responses.

*Revision*
However, we request that you address the following minor revisions before final publication:

- Improve the clarity of some explanations, particularly regarding the motivation of the method. E.g., a pending request is:

"The authors have made some incorrect claims regarding the distinction between global and local features. While I agree that conducting contrastive learning on a dog's body and its face may not necessarily help the model learn the local features of each specific part, it does indeed aid in learning global features. This is because there are various relationships between a dog's body and its face, such as spatial and topological connections, which contribute to understanding the overall context.

I appreciate the attempt at introducing this novel method; however, based on the authors' explanations, I feel that certain claims in the paper are not sufficiently validated."

- Conduct proofreading to address spelling, grammar, and formatting issues.
- Include a brief discussion on the computational complexity and scalability of LeOCLR compared to other state-of-the-art methods.
- Consider adding ablation studies to isolate and evaluate the impact of each component of the proposed method.
Decision: Accept (poster)