Trip-ROMA: Self-Supervised Learning with Triplets and Random Mappings
Authors: Wenbin Li
Xuesong Yang
Meihao Kong
Lei Wang
Jing Huo
Yang Gao
Jiebo Luo
Venue: Accepted by TMLR ()

Abstract: Contrastive self-supervised learning (SSL) methods, such as MoCo and SimCLR, have achieved great success in unsupervised visual representation learning. They rely on a large number of negative pairs and thus require either large memory banks or large batches. Some recent non-contrastive SSL methods, such as BYOL and SimSiam, attempt to discard negative pairs and have also shown remarkable performance. To avoid collapsed solutions caused by not using negative pairs, these methods require non-trivial asymmetry designs. However, in small data regimes, we can not obtain a sufficient number of negative pairs or effectively avoid the over-fitting problem when negatives are not used at all. To address this situation, we argue that negative pairs are still important but one is generally sufficient for each positive pair. We show that a simple Triplet-based loss (Trip) can achieve surprisingly good performance without requiring large batches or asymmetry designs. Moreover, to alleviate the over-fitting problem in small data regimes and further enhance the effect of Trip, we propose a simple plug-and-play RandOm MApping (ROMA) strategy by randomly mapping samples into other spaces and requiring these randomly projected samples to satisfy the same relationship indicated by the triplets. Integrating the triplet-based loss with random mapping, we obtain the proposed method Trip-ROMA. Extensive experiments, including unsupervised representation learning and unsupervised few-shot learning, have been conducted on ImageNet-1K and seven small datasets. They successfully demonstrate the effectiveness of Trip-ROMA and consistently show that ROMA can further effectively boost other SSL methods. Code is available at https://github.com/WenbinLee/Trip-ROMA. 

Published in Transactions on Machine Learning Research (05/2023)
Trip-ROMA: Self-Supervised Learning with Triplets and Random Mappings
Wenbin Li', Xuesong Yang', Meihao Kong', Lei Wang', Jing Huo',Yang Gao'; Jiebo Luo State Key Laboratory for Novel Software Technology, Nanjing University, China Uniiveriity of Wollongong, Australia, *University of Rochester, USA
Reviewed on OpenRevicw:
Abstract
Contrastive self-supervised learning (SSL) methods, such as MoCo and SimCLR, have achieved great success in unsupervised visual representation learning. They rely on large number of negative pairs and thus require either large memory banks or large batches. Some recent non-contrastive SSL methods, such as BYOL and SimSiam, attempt to discard negative pairs and have also shown remarkable performance. To avoid collapsed solutions caused by not using negative pairs, these methods require non-trivial asymmetry designs. However, in small data regimes, effectively avoid the over-fitting problem when negatives are not used at all. To address this situation, we argue that negative pairs are still important but one is generally sufficient for each positive pair. We show that a simple Triplet-based loss (Trip) can achieve surpris- ingly good performance without requiring large batches or asymmetry designs. Morcover, to alleviate the over-fitting problem in small data regimes and further enhance the effect of Trip, we propose simple aun-andd-lay RondOm MApping (ROMA) strategy by randomly mapping samples into other spaces and requiring these randomly projected samples to sat- isfy the same relationship indicated by the triplets. Integrating the triplet-based loss with random mapping, we obtain the propased method Trip-ROMA. Extensive experiments, in- cluding unsupervised representation learning and unsupervised few-shot learning, have been conducted on ImageNet-1IK and seven small datasets. They successfully demonstrate the effectiveness of Trip-ROMA and consistently show that ROMA can further effectively boost other SSL methods. Code is available at httpp:://iithuu.cmmm http:///gttttt.ccoeeveaaaeeeettp
not obtain a sufficient number of negative pairs or
1 Introduction
Unsupervised visual representation learning aims learn good image representations without human su- pervision. To achieve this, self-supervised learning (SSL.) has received considerable attention and shown promise in recent years [Wu et al.
2018 He et al. 2020 Chen et al. 2020a Chen & Hej 2021 Grill et al, 2021 The recent advances. can be classified into two categories (Tian et al. 2021]: contrastive SSL and non con-contrastive SSL. Contrastive SSL, such as Sim-
2020 Tian et al. 2020 Caron et al. 2020 Zbontar el. al.
CLR (Chen et al. 2020a and MoCo (He et al. 20200 Chen et al. 2020b). learns representations by closing the latent representations of two views of the same image together (positive pairs), while pushing the latent representations of different images farther away (negative pairs or inter-image). These methods normally rely on large number of negative pairs by using large batches or memory banks. Recently, non-contrastive SSL, such as BYOL Grrill et al. 2020) and SimSiam (Chen & He 2021). has attempted to learn represen- tations without using negalive pairs. To avoid collapsed representation ions caused by removing negative pairs, these methods usually have to employ some non-trivial asymmetry designs, such as asymmetric predictor
* *Corresponding author.

---

Published in Transactions on Machine Learning Research (05/2023)
network (Grill et al.
2020; Chen & He 2021 stop-gradients (Grill et al. 2020 Chen & He 2021 and
2020 However, in the small data regimes, e.g., unsupervised sho- learning, it is not able to use large number of negative pairs, and it is also difficult to avoid the over-fitting problem in the case of small data setting, especially when the negative pairs are discarded at all. Therefore, this raises an interesting question " "How to effeclively ulilize negalives in self-superossed learning, especially on small '
momentum encoder /rill et al.
In addition, to learn view-invariant representations, both contrastive and non-contrastive SSL normally use data augmentation to obtain pseudo labels, considering that data augmentation can well maintains the semantics of the original examples. Unfortunately, such an advantage could make models be prone to the over-fitting problem during training. The similar point has also been mentioned in SimCLR (Chen et al.) 2020a) and InfoMin (Tian et al. 2020), which show that stronger data augmentation or reducing the mutual information between augmented views of the same image can bring more benefits. In fact, the asymmetric designs in non-constrastive SSL can also be regarded as a way to alleviate the over-fitting problem. Unfortunately, this over-fitling problem will be even worse on small data where the number of training samples is scarce. This naturally raises another question "Do we have more efficient ways to alleviate the over-fitting problem in data augmentation based SSL methods?"
In this paper, we propose a new method, Trip-ROMA, as responses to the above two questions. For the first question, we find that even when the number of negative pairs is reduced to one, i.e., just one negative, it will be generally sufficient for each positive poir (i.e., triplet). In the literature, some methods (e.g., SimCLR (Chen et al., 2020aa) have ever tried to use the margin triplet loss (Schrooff et al. 2015) based on triplets, but did not achieve satisfied results. In this work, we revisit the effect of triplets, but differently we employ a triplet + binary cross-entropy loss built on triplets constructed within each minibatch for training, Compared with other methods, such a simple triplet-based loss (Trip) requires neither large batches (Chen et al. 2020a) nor memory banks (He et al. 2020), which is therefore more suitable for small data. Also, it docs not require any asymmetric design like asymmetric prodictor network (Grill et al. 2020 Chen & He 2021) or stop-gradients (Grill et al.] 2020: Chen & He 2021]. In contrast, it enjoys advant advantages of both existing contrastive and non-contrastive SSL methods: (1) working well with small batches, (2) learning from both intra- and inter-image information, and (3) naturally avoiding collapsed reprosentations.
As for the second question, we propose "random mapping preemplive measure" into SSL to help unsuper- vised deep models efficiently deal with over-fitting In other words, we can learn more robust representations by forcing models maintain the required sample relationship even under random mappings. This is because sometimes the relationship of samples during the training process could be satisfied by chance or over- wver-fitting Taking RandOm MApping (ROMA) as a preemptive measure can alleviate such a problem to some extent, and thus bring more robust way to evaluate the relationship between samples. In addition, random map- pings could enhance the diversity of samples' features, which can also alleviate the over-fitting caused by the scarce of samples in the small data regimes. In this work, we provide interpretation to the effect of the proposed random mapping strategy from the perspective of perturbation of the coordinate system. Because the same relationship of samples in triplets is required to be satisfied for all random mappings, we name our method ROMA, taking the meaning from "all roads lead to a trip to Rome".
Typically, the existing SSL methods are mainly designed on largo-scale data, e.g., ImageNot-1K (Deng et al. 2009), while paying less altention to small data. Therefore, in this work, we mainly focus on small data and investigate the transferability of SSL. models on small data. Through extensive experiments, we show that our Trip-ROMA can achieve a new state of the art on multiple benchmarks. Specifically, under the linear evaluation protocol, ROMp achieves 83.05% top-1 accuracy on ImageNet-100 which is a 3.07% absolute improvement over SimCLR (Chen et. al.] 2020a). Using the k-NN evaluation protocol, Trip-ROMA achieves 58. 58.02% top-1 accuracy on mindmmageNeet under the 5-way 1-shot few-shot setting, which is a 5.26% absolute improvement over SimSiam |Chen & He| 2021). Moreover, when extended to the large-scale dataset, Trip-ROMA can also obtain a superior top-1 accuracy of 71.1% on ImageNet- 1K under the linear evaluation protocol compared to the state-of-the-art methods. In addition, as plug-and-play strategy, the ROMA strategy can also consistently improve the performance of existing SSL methods.
Our main contributions of this work are summarized as follows:
2

---

Published in Transactions on Machine Learning Research (05/2023)
We demonstrate that negative pairs are still important but one negative pair is generally sufficient for each positive pair in contrastive SSL, especially on small data. A simple and effective triplet-based loss (Trip) is designed accordingly to validate this approach.
We propose a novel plug-and-play random mapping strategy, ROMA, to enforce SSL models to learn under random mappings and demonstrate that unsupervised representation learning can effectively benefit from incorporating randommess.
We experimentally demonstrate that the proposed Prip-ROMa achieves new state of the art on both unsupervised representation learning and few-shot learning.
2
Related Work
Contrastive learning. Recently, contrastive learning based SSL. (Bachman et al. 2019 Ye al. 2019
Hjelm et al. 2019 Misra& & Maaten 2020 Oord et al.
2018 Wu et al.
2018
He et al. |020 C Chen et al. 2020a
Wei et al. 2021 Xie et al. 2021 has drawn increasing attention owing to its simple design and excellent performance. The core idea is to discriminate
Hénaff et al. 2020 Chen et al., 2020b Tian et al.
2019
2020
among different images, by maximizing the similarity between two augmented views of the same image and repulsing other different images, i.e., contrastive learning (Hadsell et al. 2006). Along this way, Dosovitskiy et al. (2014) and Wu et al. (2018) propose to take each patch/imas immge as an individual class via instance-level discrimination. MoCo (He et al. 2020) and MoCo v2 (Chen et al., 2020b) match an encoded query to a dictionary of encoded keys with a slow-moving average network using an InfoNCE loss (Hénaff et al., 2020). ReSSL. (Zheng et al., 2 proposes relational SST. framework to model the relationship between different images, which is built on MoCo v2. Differently, SimCLR (Chen et al.] (020a) directly constructs negative samples within a much larger minibatch without requiring additional memory banks. Instead of using an instance-level contrastive lask, Pixpro (Xie et al. 2021) proposes a pixel-to-propaga ion consistency pretext task from the perspective of pixel level. Although this type of methods performs excellently, they usnally requires a large number of negative examples to work well, by using either memory banks or large batches.
Non-contrastive learning. Different from the above contrastive learning based SSL, recent literature has attempted to only use positive pairs and completely discard the negative examples, named non-contrastive learning based SSL (Caron et al., 2020 Grill et al. 2020 Chen & He| 2021 Zbontar et al. 2021). BYOL (Grill et al. 2020) directly predicts the representation of one augmented view from another view of the same image with a Siamese network (Bromley et al. 1993). where one branch of the network is a slow-moving average (momentum encoder) of another branch. After that, based on BYOL, RSA (Bai et al. 2022) proposes a new multi-stage augmentation pipeline to reduce the severe semantic shift. of using aggressive augmentations.. SimSiam (Chen & He 2021)) discards the momentum encoder, directly maximizes the similarity between two augmented views of one image, and demonstrates that the stop-gradient operation is critical for preventing collapsing. W-MSE (Ermolov et al. 2021 scatters all the sample representations into spherical distribution using whitening transform and collapse problem. In addition, SwAV (Caron et al. 2020) performs online clustering to learn prototypes and indirectly compares two augmented views of the same image by using their cluster assignments built on the learned prototypes. Recently, Barlow Twins (Zbontar et al., 2027 maximizes the similarity between two augmented views of one image while reducing redundancy between their components, by relying on very high-dimensional representations. Although these methods successfully discard the negative examples and thus somewhat alleviate the computation, they introduce new collapse problem and require further effort to address this new problem.
penalizes the positive pairs that are far from each other to avoid the
As mentioned above, the existing rastivee SSL is confined to requiring massive negative examples, while the non-contrastive SSL easily suffers from the collapse problem. As a compromise, Trip-ROMA employs one negative example to naturally avoid the collapse problem. Importantly, although only one negative example is used, Trip-ROMA can still achieve remarkable performance. In addition, a new plug-and-play random mapping strategy is proposed, which can further boost the unsupervised representation learning.
3

---

Published in Transactions on Machine Learning Research (05/2023)
Triplet Loss CE Loss
NT-Xont Loss
Cosine Similarity
gad
grdd
grdd
gnd
god
seg-gad
random M
random M
rundom
random
random
random
predictor
encoder f
encoder f
encoder f
encoder f
encoder f
encoder f
X;
X₂
image1 & image2
imagel X, imuge2 X", ....
image X
(a) Trip-ROMA
(b) SImCLR+ROMA
(c) SimSiam-ROMA
Figure (a) Trip-ROMA architecture with a tropeero-crosseettrrr (CE) loss. (b) SimCLR architecture introducing the random mapping strategy, named SimCLR+ROMA. (c) SimSiam architecture using the random mapping strategy, named MunSiam-RRMAA All three models employ the same encoder f (a back- bone plus a projection ML.P) in both branches except SimSiam uses an additional predictor h in one branch. For each input positive image, SimSiam needs no negative examples, SimCLR requires a large number of negative examples, while Trip-ROMA only needs one negative example. A random PSD matrix M > 0 is used to calculate the final loss, which will be discarded during test.
3
The Proposed Method
3.1 Trip: Triplet-based Loss Function
Following the existing contrastive and non- con-cantra SSL methods He et al. 2020 Chen et al. 2020a
Chen & He 2021: Grill et al. 2020 Tian et al.] 2020; Caron et al.] 2020; Zbontar et al. 2021), we use data augmentation to obtain the image-level pseudo labels, t.e., each image is regarded as one individual class. As shown in Fig. [(a) given any two different images (X, within a minibatch, we can construct triplet (X,, X1, X3) via data augmentation, where both X; and X; (a positive pair) are augmented from X, and Xj is augmented from X' (a negative example). After that, this triplet will be processed by an encoder network f that consists of backbone and projection multilayer perceptron (MLP), obtaining the corresponding feature vectors (zi, 2i, zz), can be formulated as
Rd. A simple triplet + binary cross-entropy loss for triplet
exp(2, 2:/T) exp(2 2./T) +exp(z,] 2j/7)*
+1 f, 22 -
A. log
(1)
where zi, 2, and 2j have been lg no-orimll [=]+ ma, X is a hyper-parameter trading off the importance of the two loss terms, and T is a temperature parameter.
As seen, the first term of Eq. 1 is a triplet loss (Weinberger Saul, 2009,, and the second term a cross- entropy loss for binary classification. In general, the triplet loss will suffer from a limitation of being sensitive to the training triples because of using fixed margin (Manmatha et al., 2017). Therefore, the cross-entropy loss can be seen as soft triplet loss but with an adaptive margin, compensating for the limitation of the triplet loss with a fixed margin. Note that each part alone of Eq. [ is not new and has been used in previous work (Parkhi et al. 2015 Koch et al., 2015 Wu et al.] 2018 He et al. 2020 Chen el al. 2020a Tian et al. p020). Nevertheless, our key contribution here to demonstrate that such a combination of both losses can achieve surprisingly good performance in SSL on small data. This has not been elcarly shown the literature and it reveals that negative pairs are still important but one may be sufficient.
3.2 ROMA: Similarity Measure with Randomness
The existing contrastive and non-contrastive SSL methods normally use cosine similarity Sim(u, u l/|||||||| to caleulate the similarity between points Rd and E Rd. During the optimization
4

---

Published in Transactions on Machine Learning Research (05/2023)
process of training, the relationship of points through such a measure could be satisfied by chance due to overfitting. An interesting question is whether we can slightly modify this measure to deal with this situation so as to obtain a more robust evaluation on the relationship of points. Our answer is yes. To be specific, we assume both u and v have been lz-normalizod for simplicity. Then Sim(u, v) reduccs to u v and can be expressed as Sim(u, v) = u Iu, where is the identity matrix. To obtain a more robust evaluation on the relationship between u and v, we replace I with a random positive semi-definite (PSD) matrix M E Raxd, M ž 0 as follows,
Sim(u, v) = =u Mu.
(2)
In this way, we can conveniently introduce randomesss into this similarity measure and then further into the objective function in Eq. ( by using series of random matrices {M&)lk- M, E gadd (n is the number
of random matrices). More specifically, when using the proposed similarity measure in Eq. (2), the loss in
Eq. can be rewritten as
M2./T exp(z, M + exp(z, M,z//T)
Crandom = [2] M2, 2, MEM - A- A-log
(3)
As will be demonstrated in the experimental study, the adoption of the random matrix M consistently lcads to better feature representation after the learning process. To gain better understanding on the use of M, we interpret its effect from the following perspective of perturbation.
Porturbation of the coordinates. We explain the effect of using the random matrix M from perspoc- live of linear projection, i.e., perturbing the coordinate system Rd with random rotation and scaling. In particular, because M is positive semi-definite (PSD) in Eq. (2, M can be mathematically decomposed
L L, where L E Reed where e is the rank of M (c S d). After that, Eq. (2) can be converted as
Sim(u, u, v) u Mu n T T. T (T..)T(T.))
(4)
where L is random linear transformation matrix, randomly mapping the data points into new (perturbed) coordinate system. In the new system, cosine similarity is then evaluated. By employing multiple different transformation matrices in Eq. ( we hope to obtain a more reliable evaluation on the relationship of the two points. In this sense, we essentially advocate a "random mapping preemplive measure" for SSL. That is, to avoid the relationship among points from being satisfied by chance or due to overfitting, we shall take a more aggressive approach to evaluate their relationship, i.e., a stable similarity or dissimilarity relationship shall more likely survive random perturbation.
Considering that directly generating a random PSD matrix M is difficult due to the requirement of the property of positive semi-definiteness, we instead randomly generate the linear projection matrix HR making cach L LA naturally positive semi-definite. The illustration of random mappings for a triplet is shown in Fig. 2 Note that for each triplet in a minibatch, we only use single random linear projection matrix Lg. Also, the frequency of using the random mapping will be discussed in Section 5.3 of experiments.
3.3 Generality of ROMA
We highlight that RandOm MApping (ROMA) is a general strategy. In other words, as plug-and-pla strategy, ROMA can be applied to other state-of-the-art SSL methods, no matter they are contrastive based non-contrastive based. Taking SimCLR (Chen et al. 2020a) as example, when the random mappings are incorporated, named SSCCRR+ROMA its handom-based NT-Xent loss can be easily formulated as
exp(x M2./T)
C SimCLRRROMA log Σ3 Li exp(2 Mzz/T)²
where Les is an indicator function, N is the mumber of batch size, and M is a random PSD matrix. Its architecture can be seen in Fig. 1(b)
Similarly, introducing the random mappings into SimSiam (Chnn & He 2021), we can obtain Sim- Siam+ROMA and its symmetrized cosine similarity loss can be easily converted as
SimSlam ROMA = 2
1
Mx - p2 Mæ
(6)
5

---

Published in Transactions on Machine Learning Research (05/2023)
original space
Iu
L2
Ln-i"
C+ Pull
Dust
null L
pull
pull
random subspace
I
random subspace 2
random subspace n-l
random subspace
Figure Illustration of random mappings for a triplet. For each triplct, the original feature space can be projected into another feature space with random matrix L4- In each random subspace, the relevant structure of the triplet, i.e., the similarity between the positive pair should be larger than the negative pair, will be enforced by pulling and pushing operations. Rectangles and triangles denote positive and negative samples, respectively. Note that for one certain minibatch, only one single random matrix Lg is used.
where (21, z2) a positive pair, outputs of the encoder f, and (P1, P2) is the subsequent representations of (21, #2) by further processing via an additional predictor h. See Fig, [[c] for more intuitive understanding. In addition, both SimCLR+ROME and SimSiam ROMA will be further investigated and discussed in the experimental part. Please see Section 5.1 for more details.
4 Implementation Details
Datasets. The main experiments are conducted on seven small benchmark datasets, i.e., CIFAR- 10 Krizhevsky et al. 2009), CIFAR-100 (Krizhevsky et al.] 2009). STT-10 (Coates et al. (001]) ImageNet-
100 (Tian et al. 2019), minilmageNet (Vinyals et al. 2016). CIFAR-10OFS (Bharti et al. 2020) and
FC100 (Oreshkin al.
2018 whose details can be found in the appendix.
Image augmentation. For fair comparison, use the same data augmentation operations Sim-
CLR (Chen et al.y 2020a), consisting of random cropping and resizing, horizontal flipping, color distortion, Gaussian blurring, and an optional grayscale conversion. Please refer to SimCLR (Chen el. al. 2020a for
more details.
Architecture. We utilize ResNet-50 (He et al. 2016 ResNet-18 and ResNet-12 (Lee et al. 2019) fol- lowed by ML.--baed projector network as our encoder f, respectively, where ResNel-12 used for the unsupervised few-shot learning (FSL) task. For the generic unsupervised representation learning task, the MLP-based projector network consists of three fully-connected (FC) layers, cach with 2048 output dimen- sions. Specifically, the first two FC layers are followed by batch normalization (BN) layer (Ioffe & Szegedy 2015) and a leaky rectified linear unit (LeakyRel.U) layer (Maas et al. 2013), while only the BN layer is used in the last FC layer. As for the unsupervised FSL task, there are only two FC layers in the MLP-based projector. Furthermore, different from the mainstream methods (Chen et al. 2020a Grill et al. 2020 Chen & He 2021b, we add random projection matrix with size of 2048 x 1024, i.e., 20-1X1002 afterr the
encoder module learn more robust represcntations.
Optimization. We use SGD optimizer with momentum of 0.9 and cosine decay learning rate schedule to optimize ROMA. On 1000,,, we train Trip-ROMA for 200 epochs with the base learning rate of 0.2, batch size of 128 and weight decay of 1e 4. For CIFAR-10, STL-10, CIFAR-100 and mintImageNett, the models are trained for 1000 epochs with the base learning rate of 0.03 and weight decay of 5e - 4. The batch size is set to 64 for CIFAR-10 and CIFAR-100, but 128 for STL-10, ménilmageNet,, CIFAR-100ES and FC100. Additionally, each learning rate lincarly scaled with the batch size. The hyper-paramcters X and 7 in Eq. (3) are set to and 0.5, respectively. For the random mapping setting, linear transformation matrix L. with size of 2048 x 1024 is randomly sampled from standard normal distribution.

---

Published in Transactions on Machine Learning Research (05/2023)
Linear evaluation. Following the SSL literature (Zhang al. 2016 Oord et al. 2018 Chen al. 2020a
Grill et al. 2020 Chen & He 2021). we employ a linear evaluation protocol to evaluate the features learned by our model for the unsupervised representation learning task. To be specific, given the pre-trained ResNet backbone after unsupervised learning, a linear classifier is further trained on top the frozen backbone and global average pooling layer. In our experiments, the linear classífier trained for 100 epochs using a SGD optimizer in cosine docay schedule, where the base learning rate is 30, weight decay 0, momentum is 0.9 and batch size is 128. After training, we perform the evaluation on the center cropped images in the evaluation test set. In addition, avoid the randomness of single and make the comparison more fair, we run three times for each method (i.e., unsupervised pre-training for three times) and report the top-1 mean accuracy as well as 95% confidence interval (i.e., linear evaluation) for all comparison methods.
Table 1: Linear evaluation (top-1 mean accuracy) on four small benchmark datasets, all at 95% confidence intervals. Evaluation is on single center crop. All unsupervised methods are trained 1000 epochs CIFAR- 10/100 and STL-10, and 200 epochs on ImageNet-100 all of which are trained for three times. In addition, SimCLR, SimSiam and Trip-ROMA are further trained 1000 epochs on ImageNet-100 for once. Neg. pair indicates whether negative pairs are used. Stop. and Pred. denotes whether an additional predictor network and stop-gradient are used.
Image
Batch Size
Neg. Stop. Pair Pred.
DataSet
Method
Epochs
Backbone
Acc. (%)
Size
SimCL.R Chen et al. SimCLR ROMA
2020a.
1000
512
ResNet-18
91.45
1000
512
ResNet-18
91.93 (t 0.48)
MoCo
Chen et
2020b
1000
512
ResNet-18
89.46-2032
CIFAR-10
x2
SimSiam Chen He
2021
1000
512
ResNet-18
91.122a.12
CinSiamFFOMM
1000
512
ResNet-18
91.61 #0.09 (f 0.49)
Trip
1000
64
ResNet-18
92.06.0.221
Trip-ROMA
1000
64
ResNet-18
92.29
SimCLR Chen al.
2020a
1000
512
ResNet-18
62.66
SimCLR
ROMA
1000
512
ResNet-18
63.05 (f 0.39)
MoCo v2
Chen et al.
2020b
1000
512
ResNet-18
59.7720.60
CIFAR-100
32x
SimSiam Chen He
2021
1000
512
ResNet-18
64.04=0.3z
SimSiam
1000
512
ResNet-18
+..35 (1 2.31)
Trip
1000
64
ResNet-18
66.06
Trip-ROMA
1000
64
ResNet-18
66.32
SimCLR Chen et al.
2020a.
1000
512
ResNet-50
87.14-0.19
SimCLR ROMA
1000
512
ResNet-50
87.28 (1 0.14)
MoCo Chen et
2020b
1000
512
ResNet-50
86.6920.50
STL-10
64
SimSiam Chen & He
2021
1000
512
ResNet-50
85.91
MinSiamFRONe
1000
512
ResNet-50
86.18-0044 (1 0.21)
Trip
1000
128
ResNet-50
37.37:0.42
Trip-ROMA
1000
128
ResNet-50
87.80=0.27
SimCLR Chen al.
2020a
200
256
ResNet-50
76.15
SimCLR ROMA
200
256
ResNet-50
76.53 (1 0.38)
MoCo v2
Chen et al.
2020b
200
256
ResNet-50
68.59-0.51
SimSiam
Chen & He
2021
200
256
ResNet-50
75.15
SimSiam
200
256
ResNet-50
76.08 (f 0.93)
Trip
200
128
ResNet-50
78.62
ImageNet-100 224x224 Trip-ROMA
200
128
ResNet-50
80.21 10.27
SimCLR Chen el al.
2020a.
1000
256
ResNet-50
79.98
SimCLR ROOMA
1000
256
ResNel-50
80.98 (1 1.00)
SimSiam Chen & He
2021
1000
256
ResNet-50
81.72
FiOSia
1000
256
ResNet-50
82.10 (1 0.38)
Trip
1000
128
ResNet-50
81.97
Trip-ROMA
1000
128
ResNet-50
83.05

---

Published in Transactions on Machine Learning Research (05/2023)
5 Experiments
5.1 Comparison with the State of the Art
We thoroughly compare the proposed methods with the closely related state-of-the-art methods, i.e., Sim- CLR (Chen et al. 2020a), , MoCo v2 (Chen et al. 2020b) and SimSiam (Chen & He 2021), on four benchmark datasets, including CIFAR-10, CIPAR-100, STL-10 and ImageNet-100. Specifically, two variants of Trip- ROMA are constructed by introducing the random mapping strategy into SimCLR and SimSiam, named Sim CLM+ A and SimSiam-+RMMi triplet-based loss in Eq. (L), another variant of Trip-ROMA is also constructed by removing the random map- pings, i.e., Trip. For fairness, we reimplement all the comparison methods into the same framework being
respectively. In addition, to verify the effect of the proposed
faithful to the original paper. Following the literature (Tian et al.) 2019 Chen et al. 2020a Chen & He, 2021), we adopt a ResNet-50 backbone on STT-10 and ImageNet- and adopt a ResNet-18 18 backbone on CIFAR-10 and CIFAR-100 for all comparison methods. The linear evaluation results are reported in Table [
Effectiveness of the triplet-based loss. One of our concerns is how to effectively utilize negatives in SSL. From Table [ we see that Trip (which only uses the proposed triplet-basod loss) can achieve remarkable performance on all datasets. For example, on CIFAR-10, Trip can obtain 0.61%, 2.60% and 0.94% improvements over SimCLR, MoCo v2 and SimSiam, respectively. More importantly, Trip adopts much smaller batch size (64), compared to other competitors (512). Similarly, on ImageNet-100, Trip can even gain 2.47%, 10.03% and 3.47% improvements over SimCLR, MoCo v2 and SimSiam, respectively. It verifies that negatives are still important and one is sufficient, i.e., triplet is sufficient. Also, the above concern, t.e., the first question in the introduction, has been answered.
Effectiveness of random mappings. As seen in Table 1 SimCLR+ROMA consistently performs better than SimCLR, and obtains 0.48% 0.39%, 0.14% and 0.38% improvements over SimCLR on four datasets, respectively. Similarly, SimSiam+ROMA can also consistently performs better than SimSiam, and achieves 0.49%, 2.31%, 0.24% and 0.93% improvements, respectively. This successfully demonstrates the effectiveness and generality of the proposed random mapping strategy. Mcanwhile, the second question in the introduction can also be answered.
Effectiveness of Trip-ROMA. As expected, Trip-ROMA, a combination of the triplet-based loss and random mapping strategy, achieves remarkable performance on all datasets, and performs much better than other competitors. For example, Trip-ROMA achieves new state-of-the-art. result of 92.29% on CIFAR-10 with much smaller batch size of 64, which is 0.84%, 2.83% and 1.17% higher than SimCLR, MoCo v2 and SimSiam with batch size of 512, respectively. Similarly, on CIFAR-100, Trip-ROMA still performs better than other competitors, gaining 3. 3.66%, 6.55% and 2. 2.28% improvements over SimCLR, MoCo v2 and SimSiam, respectively. Specifically, on the more challenging ImageNet-100, Trip-ROMA still gain significantly improvements over SimCLR, MoCo v2 and SimSiam by 4.06%, 11.62% and 5.06%, respectively, even with a smaller batch size of 128.
Longer training process. We also compare TOiA with SimCLR and SimSiam in longer training process (1000 epochs) on ImageNet-100.. As shown in Fig. 3(a) in the first 400 epochs, Trip-ROMA obtains similar accuracy as SimCLR. However, after 400 epochs, Trip-ROMA improves significantly and achieves 2.05%, 3.34%, 3.07% improvements over SimCLR at the 600th, 800th, 1000th epochs, respectively. Although SimSiam can converge faster than Trip-ROMA in the initial stage, its performance will becomes saturation or even slightly degraded after 400 epochs. On the contrary, the performance of Trip-ROMA can steadily improve with longer training. Their results trained for 1000 epochs are also reported in Table D As scen, Trip-ROMA obtains a new stale-of-the-art result of 83.05% on ImageNet-100,, which is 3.07% and 1.33% higher than SimCLR and SimSiam, respectively. This verifies that Trip-ROMA has a strong ability to continuously learn powerful representations from randomness with longer training.
Performance on large-scale data. Although our focus in this work is on small data, we also perform Trip and Trip-ROMA. on a large-scale dataset, i.e., ImageNet-IK (Deng et al., 2009) to fully investigate the performance of Trip and TOiA- Different from the small datascts, as a largo-scale datasct, ImageNet- IK needs more computing resources to put effort into efficiently tuming the esper-parameters and training tricks. Unfortunately, we are not able to have access to sufficient computing resources. Therefore, we simply
8

---

Published in Transactions on Machine Learning Research (05/2023)
Table 2: Linear evaluation (top-1 accuracy) on ImageNet-1K, All the methods are trained with a ResNet-50. means that the multi-crop strategy is used.
Method
Epochs
Batch Size
Acc. (%)
NPID (Wu ot al.
2018
200
256
54.0
LA
(Zhuang et a..
20199)
200
128
60.2
MoCo
He al.
2020
200
256
60.6
SeLa
Asano al.)
2019
200
256
61.5
PIRL
Misra & Maaten
2020
200
1024
63.6
CPC v2
Henaff et al.
2020
200
512
63.8
SimCLR
Chen et al.
2020a.
200
256
64.3
SimCLR
Chen et
2020a
200
1096
66.8
MoCo v2
CCen et al.
20205
200
67.5
SwAV
Caron et at.
2020
200
1096
69.1
SwAV
Caron
al. 2020
200
1096
72.7
SimSiam
Chen &c He
2021
200
256
70.0
BYOL (rril
a..
2020
200
4096
70.6
Trip Trip-ROMA
200
256
69.9
200
256
70.2
Trip-ROMA*
200
256
71.1
Table 3: K--N evaluation (top-1 mean accuracy) on minilmageNeet, CIFAR- and FC100 under the unsupervised few-shot setting. denotes that the results quoted from the original papers.
mineImageNet
CIFAR-10OF'S
FC100
Type
Method
Backbone
5wls
5w5s
5wls
5w5s
5wls
5w5s
Supervised FSL
ProtoNet
(nell
2017
ResNet-12
57.10
74.20
57.64
81.24
35.16
49.34
UBC-FSL,
Chen et
2021
ResNet-12
47.8
68.5
Unsupervised FST.
No-labels
Bharti et al
2020
ResNet-50
50.1
60.1
53.0
62.5
37.1
43.4
UBC-FSL
Chen et
2021
ResNet-50
56.2
75.4
SimSiam
Chen & He
2021
ResNet-12
52.76
73.30
50.78
69.54
35. 11
46.93
Self-supervied Learning
SimCLR
Chen et
2020a
ResNet-12
57.13
74.54
55.27
73.56
35.01
47.26
Trip-ROMA ROMA
ResNet-12
58.02
76.41
55.87
74.00
36.30
49.06
follow the latest works (Chen al. 2020a Grill et al. 2020 Caron al. 2021
Bardes al.) 2022). adopt
ResNed-50 backbone on ImageNet- 1K and introduce some training tricks, e.g., momentum encoder (Grill) et al. 2020 and multi-crop (Caron et al., 2020 into the training process. In addition, also perform variant of Trip-ROMA without using the multi-crop strategy. As seen in Table 2 Trip-ROMA* obtains a top-1 accuracy of 71.1% with small batch size of 256 under a 200-epoch training setting, which is better than MoCo (e et al.) 2020), SimCLR (Chen et al.) 2020a), SimSiam (Chen & He 2021) and BYOL (Grill et al. 2020), and is very close to SwAV* of using a much larger batch size of 4096, This successfully demonstrates the offectiveness and applicability of the proposod methods on the largo-scale dataset.
5.2 Performance on Unsupervised FSL Tasks
We further apply Trip to unsupervised few-shot learning tasks to investigate its effectiveness on the sccnario of few-shot Icarning (FSL). Following the FSL. literature (Lee al., 2019) Chen et al.] 2019). we adopt ResNet-12 the backbone and set the input image size 84 84. At the training stage, we perform unsupervised pre-training each data (i..,, miniImageNet, CIFAR-10OFS and FC100) for 1000 epochs by using Trip-ROMA, SimCLR and SimSiam, respectively. At the test stage, we use a k-NN evaluation, i.e., cosine similarity based ProtoNet (Snell et al. 2017) over 3000 5-way 1-shot / 5-shot test tasks and report the mean top-1 accuracy.
As shown in Table 3, compared with the closcly related SSL methods (i.e., SimCLR and SimSiam), Trip- ROMA achieves the best results on all three datasets with k-NN evaluation. Specifically, on min'mageeett, Trip-ROMA gains 1.87% and 3.11% improvements over SimCLR and SimSiam under the 5-shot setting,
9

---

Published in Transactions on Machine Learning Research (05/2023)
CIFAR-10
ImageNet-100 100
CIFAR-100
-D- SarCCLL
01:32
01:41
80
Accuracy
75.04
68
75-
866
70
65
100
200
400
600
800
1000
64
512
1024
255
512
1024
2048
4096
8192
epochs
Batch Size
Dimension
(a) Longer training process.
(b) Effect of batch sizes.
(c) Random output dimensions.
Figure 3: (a) Performance on ImageNet-1000 for longer training, L.c., 1000 epochs. Here each point is intermediate linear evaluation (top-1 accuracy) result during the 1000-epoch training. (b) Effect of different batch sizes for Trip-ROMA, (c) Effect of random output dimensions for Trip-ROMA.
Table 4: Effects of different loss functions.
Table 5: Effects of different random frequencies.
Triplet
CE
Trip
NoRandom IBatch
1Epoch 10Epoch
CIFAR-10
91.67
91.71
92.29
CIFAR-10 ImageNet-100
91.85
92.19
92.26
92.29
Ima
78.12
77.28
80.21
77.14
79.11
80.21
79.07
Table Effects of using different random strategies.
Table
Effects of using the same batch size for all methods on ImageNet-1000
Bernoulli Uniform
Normal
Method
Epochs
BatchSize
Backbone Acc.(%)
CIFAR-10
91.28
91.57
92.29
SimCLR
200
128
ResNet-50
76.85
MoCo v2
200
128
ResNet-50
69.32
ImageNet-100
77.66
78.67
80.21
SimSiam
200
128
ResNet-50
74.12
Trip
200
128
ResNet-50
78.62
Trip-ROMA
200
128
ResNet-50
80.21
respectively. Also, compared with the existing unsupervised FSL methods, i.e., UBC-FSL (Chen et al, 2021) and No-labels (Bharti et. al. 2020), Trip-ROMA is still able to obtain significant improvements no matter under the 1-shot nor 5-shot settings on both minilmageNNet and CIFAR- 100FS. No-labels performs better than Trip-ROMA only on FC100 under the 1-shot sctting, but note that No-labels uses a much deeper backbone than TOMA, In addition, it interesting see that Trip-ROMA, an SSL method, has already been able to achieve much better results over ProtoNet (Snell et al., 2007)). a supervised FST, method, on minélmageNet under the k-NN evaluation criterion. This reveals that the SSL methods alone are greatly promising in the field of FSL.
5.3 Ablation Studies
Batch size. For contrastive SST, methods, e.g., SimCLR (Chen al. 2020a): the excellent performance is largely dependent on a large batch size, while the performance will suffer drops when batch size is reduced. In contrast, our Trip-ROMA is more robust to small batch sizes, like some non-contrastive SSL methods, e.g., SimSiam (Chen & He 2021). To empirically verify this point, train Trip-ROMA on four datasets using different batch sizes from 64 to 1024. As seen in Fig. 3(b), Trip-ROMA works very well with small batch size on all datasets, especially on CIFAR-10 and CIF.R-100. When using a batch size of 128 (or 256), Trip-ROMA performs the best on STL-10 (or TmageNet-100).
In addition, further verify the superiority on small batch size of the proposed methods, we conduct an experiment on ImageNet-100 by using the same small batch size of 128. From Table 7, we can see that SimCLR, MoCo v2 maintain the results, but SimSiam will lose 1.03% performance compared to their
10

---

Published in Transactions on Machine Learning Research (05/2023)
results in Table 1 In this sense, performance improvements of the proposed Trip and Trip-ROMA over these existing methods are robust and significant.
Random dimension. The output dimension of the random projection matrix, iee, the dimension of the random subspace, is also an important factor in Trip-ROMA Note that the feature dimension is usually set as 2048 the literature. Clearly, we can not only randomly project samples into a lower-dimensional subspace (e.g., 256), but also project them into a higher-dimensional subspace (e.g., 8192). To systematically study the impacts of different random dimensions, we conduct experiments on four benchmark datasets by varying the random output dimensions. As seen in Fig. 3(c) on different datasets, the impacts of output dimensions are relatively different. For example, on S0,- a higher-dimensional projection performs better, but the difference is somewhat limited. In contrast, on CIFAR-100 and ImageNet-1000, a 1024-dimensional projection performs the best. Intuitively, such an appropriate dimensionality reduction, i.e., 2048 1024, could reduce some redundant features, benefitting the final performance. In general, a random projection matrix with the size of 2048x1022 can obtain stable good results for TriA-
Loss function. Because there are two parts in our objective loss function in Eq. (3), t.e., triplet loss and a binary cross-entropy (CE) loss, it will be interesting to analyze the influence of each individual part. Without loss of generality, we conduct experiments on both CIFAR-10 and ImageNet-100 100 with the same settings in Section 5.T. In particular, three variants of Trip-ROMA are constructed, i.e. Triplet, CE and Trip, where Triplet means only the triplet loss is used, CE indicates only binary cross-entropy loss is used, and Trip means both parts are used. Note that all the three variants still employ the random mapping strategy. From Table 1 we can see that (1) each individual loss can achieve competitive results, compared to the state-of-the-art methods (see Table 1 (2) Triplet+CCE performs much better than each individual loss alone (i.e., Triplet or CE). As have been explained in Section 3.1 the triplet-based cross-entropy loss can be regarded as a supplementary to the standard triplet loss, both of which benefit the final performance of TOMA-
Random frequency. The frequency of random mapping determines how much knowledge can our method acquire from the latent random subspaces. An inappropriate random frequency may cause our method to fail to acquire enough knowledge or waste too much time in a random subspace. Therefore, we compare the performance of using different frequencies, i.e., using no random (NoRandom), ) using random every mini batch (1Balch), using random every epoch Eppoc) and using random every 10 epochs (10Epoch), on both CIFAR- 10 and ImageNet- 100. As seen in [ compared to NoRandom, all three kinds of random frequencies can boost the performance, which further verifies the effectiveness of the random mapping strategy. In addition, we can see that Trip-ROMA is not significantly sensitive to the random frequency. For example, on CIFAR- 10, using random every 10 epochs ( (10Epoch), i.e., a lower frequency, performs the best, while using random every epoch (1 (iopp))) higher frequency, achieves the highest accuracy on ImageNet-100. Therefore, in our other experiments, we 1Eooc frequency for ImageNet-100 100 and mintImageNet, and 10Epoch frequency for other five datasets which are trained for 1000 epochs.
Random strategy. We further explore the impacts of different random strategics. Specifically, three distributions are employed, including Bernoulli distribution (Bernoulli), uniform distribution of 1) (Uniform), and standard normal distribution (Normal), where the first is a discrete distribution, while the later two are continuous distributions. For example, if we employ the standard normal distribution to generate a random projection matrix L, it means that all entries of L are identically and independently sampled from standard normal distribution. Table G presents the results on CIFAR-10 and ImageNet- From the results, we can observe that the normal distribution strategy outperforms Bernoulli and Uniform strategies on both of datasets. In addition, we note that the continuous random strategy (i.e., the Uniform and Normal distributions) performs better than the discrete random strategy (i.e., the Bernoulli distribution) to some extent, which implies that the random continuous subspace is more conductive to learn more robust representations.
6 Conclusion
In this paper, we present Trip-ROMA, a simple and effective SSL method for unsupervised representation learning and unsupervised few-shot learning. By using a simple triplet-based loss and random mapping
11

---

Published in Transactions on Machine Learning Research (05/2023)
strategy, Trip- ROip can achieve new state-of-the-art results seven small datasets and large-scale dataset. From the results, we have the following findings: (1) Contrary to the recent trend of completely discarding the negative pairs in SSL, negative examples are still important but one is sufficient with a properly designed loss function; (2) Unsupervised representation learning can benefit significantly from randomness, such as using random mappings. We hope our work will draw the community to rethinking the impact of negative examples and exploiting randomness for benefits. In the future, we will explore how effectively accelerate the convergence of Trip-ROMA.
Acknowledgements. This work is supported in part by the Science and Technology Innovation 2030 New Generation Artificial Intelligence Major Project (20212D0113300 the National Natural Science Foun- dation of China (62106100, 62192783, 62276128), Jiangsu Natural Science Foundation (BK20221441), the Collaborative Innovation Center of Novel Software Technology and Industrialization, and Jiangsu Provincial Double-Innovation Doctor Program (JSSCBS20210021).
References
YM Asano, C Rupprecht, and A Vedaldi. Self-labelling via simultaneous clustering and representation learning. In Proceedings of the international conference on machine learning (ICLR), 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Proceedings of the conference on neural information processing systems (NeurIPS), 2019.
Yingbin Bai, Erkun Yang, Zhaoqing Wang, Yuxuan Du, Bo Han, Cheng Deng, Dadong Wang, and Tongliang Liu. Rsa: Reducing semantic shift from aggressive augmentations for supe-suue learning. In Proceed- ings of the Conference on Neural Information Processing Systems (NeurIPS), 2022.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: supervised learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
regularization for self-
Aditya Bharti, Vineeth N. Balasubramanian, and C. V. Jawahar. Few shot learning with no labels. In arxiv preprinl Xiv:220012, 2. 13751, 2020.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a' siamese' time delaybromley199366. 10sinlyyy11siggttuueeeeerrr network. In Proceedings of the conference on neural information processing systems (NeurIPS), volume 6, pp. 737-744, 1993.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In Proceedings of the conference on neural information processing systems (NewrIPS), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE international conference on computer vision (ICCV), pp. 1630-9640, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Tn Proceedings of the international conference on machine learning (ICML), pp. 1597 1607, 2020a.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few- shot classification. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE conference compuler vision and pallern recognilion (CVPR), 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. ar 00::2003304209 2020b.
12

---

Published in Transactions on Machine Learning Research (05/2023)
Zitian Chen, Subhransu Maji, and Erik G. Learned-Miller. Shot in the dark: Few-shot learning with no base-class labels. In Proceedings of the IEFE Conference on Compuler Vision and Pattern Recognation (CVPR) Workshop, pp. 2668 2677, 2021.
Adam Coates, Andrew Ng, and Honglak Lee, An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the international conference on artificial intelligence and statisties (AISTATS), pp. 215-223, 2011.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchi- cal image database. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 248-255, 2009.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsu- pervised feature learning with convolutional neural networks. In Proceedings of the conference on newral information processing systems (NeurIPS), 2014.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In Proceedings of the international conference on machine learning (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 5015-3024 2021.
Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In Proceedings of the conference neural information processing systems (NewrIPS), 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In Proceedings of the TEFE conference on computer vision and pattern recognition (CVPR), pp. 1735 1742, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pallern vecognition (CVPR), pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual reprosentation learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 9729 9738, 2020.
Olivier J. Hénalf, Aravind Srinivas, Jelfrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. Dfta-efficient image recognition with contrastive predictive coding. In Proceedings of the international conference on machine learning (ICML), pp. 4182-4192, 2020.
R Devon Hjelm, Alex Fodorov, Samuel Lavoic-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In Proceedings of the international conference machine learning (ICLR), 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the international conference on machine learning (ICLR), pp. 448-456, 2015.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In Proceedings of the international conference on machine learning (TCML. deep learning workshop, volume 2. Lille, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Mcta-learning with differ- entiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10657-10665, 2019.
13

---

Published in Transactions on Machine Learning Research (05/2023)
Andrew L Maas, Awni Y Hanmun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the international conference machine learning (ICML), pp- 3, 2013.
R. Manmatha, Chao- Yuan Wu, Alexander J. Smola, and Philipp Krähenbühh. Sampling matters in deep embedding learning. In Proceedings of the TEEE international conference on computer vision (ICCV) pp. 2859 2867, 2017.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE conference on computer vision and pallern recognition (CVPR), pp. 2020.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv:1807.0371 03748, 2018.
Boris N. Oreshkin, Pan Rodríguez López, and Alexandre Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), pp. 719-729, 2018.
Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. 2015.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the TEFE Conference on Computer Vision and Pattern Recognation (CVPR), pp. 815 823, 2015.
Jake Snell, Kevin Swersky, Richard Zemel, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), pp. 4077-4087, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, arXiv:1906.052: 05849, 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. In Proceedings of the conference on neurul information processing systems (NeurIPS), 2020.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In Proceedings of the international conference on machine learning (ICML), volume 139, pp. 10268-10278, 2021.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), pp. 3630 3638, 2016.
Chen Wei, Huiyu Wang, Wei Shen, and Alan L. Yuille. CO2: consistent contrast for unsupervised visual representation ion learning. In Proceedings of the international conference on machine learning (ICLR), 2021.
Kilian Q Weinberger and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. Journal of machine learning rescarch, 10(2), 2009.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 3733 3742, 2018
Zhenda. Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Explor- ing pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEFE conference computer vision and pattern recognition (CVPR), pp. 16684-16693, 2021.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervisod embodding learning via invariant and spreading instance feature. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 6210 6219, 2019.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Proceedings of the international conference machine learning (TCML), volume 139, pp. 12310 12320, 2021.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Proceedings of the European conference on computer vision (ECCV), pp. 649 666. Springer, 2016.
Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Ressl: Relational self-supervised learning with weak augmentation. In Proceedings of the Conference on Neural Information Processing Systems (NewrIPS), pp. 2543 2555, 2021.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE international conference on computer vision (ICCV), pp. 6002 6012, 2019.
A Details of Datasets
As mentioned in the main paper, the major experiments of this paper are conducted on seven small benchmark datasets, i.e., CIFAR-10 (Krizhevsky et al. 2009), CIFAR-100 Krizhevsky et al. 2009). STL- (Coates et al.] 2011p, ImageNet-Iie (Tian et al. 2019) minilmageNett (Vinyals et al. 2016), CIFAR- 100FS (Bharti et al. (0200), and FC100 (Oreshkin et al. 2018). A largo-scale benchmark datasct, i.e.,
ImageNet-1K (Deng al. 2009), is also used. The details of these datasets are follows:
CIFAR-10 consists of 60000 32 32 colour images in 10 classes, in which there 6000 images in each class. Following the original splits, we take 50000 and 10000 images for training (without labels) and test, respectively.
CIFAR-100 has total number of 100 classes, containing 500 training images and 100 test images per class. We take 50000 images ignoring the labels for training, and the remaining images for test.
STL-10 contains 10 classes, where there are 100, 000 96 96 unlabeled training images, and 500 labeled training images and 800 test images in each class.
ImageNet-100 is subset of ImageNet-1K with 100 well-balanced classes and total number of 0.13 million images, which is first introduced in TTian et al. 2019).
méndImageNee is popular benchmark dataset in the filed of few-shot learning (FSL), which contains 100 classes selected from ImageNet-1K,, each class has 600 images with image size of 84 x 84.
CIFAR-100FS is version for few-shot learning conducted based the CIFAR-100 dataset (Krizhevsky et al. 2009). It also contains 100 classes like CIFAR-100, where 64, 16 and 20 classes are used for training, validation and test, respectively. In addition, because the average inter-class similarity of this dataset is generally high and the image resolution of 32 x 32 is low, CIFAR-10OFS generally a challenging data in FSL.
FC100 is another version built on CIFAR-100 for FST. with a new splits that is different from COFAR-10OFS. Specifically, 20 high-level super classes are divided into 12, 4 and 4 classes, which are corresponding to 60, 20 and 20 low-level specific classcs, for training, validation and test, respectively.
ImageNet-1K a widely-used large-scale dataset in the field of image classification. It contains 1000 classes, with 1.2 million images for training and 50000 images for validation. All images are resized 224 224 for unsupervised training without labels.
B Pseudo-code of Trip-ROMA
The pseudo-code of the proposed TriA is shown in Algorithm
15

---

Published in Transactions on Machine Learning Research (05/2023)
Algorithm Trip-ROMA Pseudocode, PyTorch-liie
backbone projection mlp
x, x_ in loader:
load samples
x1, x2,
aut(X), (X)(X), aug (x_)
random augmentation, obtaining triplet projection, N-by-D
z1, z2, z3 f(x1), f(x2), fx33)
rendom_matrix randn (D, D)
# random matrix: D-by-D
loss L(11, 22, 23, random_matrix) loss. backardc u1)
# loss
# backward
# SCD update
10 def (z21 22, 23, random_ (addon_matrix):
11
21 normalize (21 8 random_ random_matrix matrix, dim-1)
* randon projection and normalize
z2 normalize (z2 0 candom_matrix: dim=1)
23 = normalize (23 randon_matrix dim=1)
pos_sim (z1 22)..un sun (dim=1)
positive pair similarity negative pair similarity
sim (z1 z3) sun (dim=1)
Triplet_ loss clamp (natgin neg_sim poa_sim, min=0.) Triplet loss
18
20
legits ([pos_sin, neg_sim]) temperature
by temperature
21
labels [1, 0]
22
CE_loss crose_entroph (logite, labels)
# Cross entropy loss
23
24
return Triplet_ weight co_l
C Explanation for ROMA from Perturbation of the points
In addition to the explanation from the perturbation of the coordinates in Section 3.2 in the main paper, we can also explain the random projection, i.e., ROMA, from another perspective, i.c., perturbation of the points. Specifically, given a pair of points u and v, we are able to add a small perturbation Au, Av E IRd to u and v, respectively. Conceptually, we can always generate them by Au = Mju and Av = M2v, where Mj and M2 are two random matricos with sufficiently small Frobenius norm. This because the norm of Miu can be proved to be upped bounded by the Frobenius norm of M1 as follows: [| u M, Miu s Amar [M [ < trace(M) M || 1/ This proof uses the property of Rayleigh quotient generate
and the definition of Frobenius norm. Therefore, by using M1 with a sufficiently small norm, sufficiently small perturbation by Au = Myu. This proof applies to the case of Av = M2v. In this case, their cosine similarity can be expressed as
Sim(u + Au, Av) (u Au) T (v + Av)
=u + u Av + Au + Au Av
=uT + u M2 u Mi + u JTM, M2U
(7)
-u M2 Mi Mi M2)0
(Let M I Mi Mr Mr M2)
-oMMo.
As seen, when interpret random matrix M (T M1 M2 + M, M2), its effect be thought of adding random perturbation to the points before evaluating their similarity usual. This provides explanation on why our insertion of random matrix M could lead to better results.
D Eigenvalue Spectrum Analysis
To gain more insight on the effect of the proposed random projection, i.e., ROMA, we examine the eigenvalue spectrum of the covariance matrix computed with the feature representations before and after a random projection is applied. To be specific, train Trip-ROMA Ima for 200 epochs with RestNet-550
16

---

Published in Transactions on Machine Learning Research (05/2023)
-20
log of eigenvalue
-40
500
1000
1500
2000
epoch 150
-20
-40
500
1000
1500
2000
200
sorted elgenvalue index
Figure 4: Evolution of eigenvalues of the covariance matrix computed with the feature representations learned by Trip-ROMA on ImageNet-100. The solid and dotted lines indicate the eigenvalues obtained before and after using random projection, respectively.
and select the effectively trained models at the 150th and 200th epochs, respectively. With each model, we extract 2048-dimensional feature vector for each training image, and then use all feature vectors calculate the covariance matrix before and after random projection is applied to them, respectively. For each of the two covariance matrices, its eigenvalues are sorted and linearly scaled to make the largest eigenvalue to be one for the ease of comparison. The resulted eigenvalue spectrums from the two covariance matrices are compared in Figure
It observed that after a random projection applied, the eigenvalue spectrum becomes more balanced, that is, the rapid attenuation at the late stage of the spectrum has been well deferred. This change indicates that the variance of the feature representations along the directions of the corrosponding eigenvectors is increased. This increased variance could help to reduce the odds that the relationship (i.e. similar not similar) of samples is satisfied by chance when it is measured in this new projected space.
E Implementation of Trip-ROMA on ImageNet-1K
Different: from the small datasets, as largo-scale dataset, ImageNet-1K needs more computing resources put effort into efficiently tuning the lyper-parameters and training tricks. Unfortunately, we are not able to have access to sufficient computing resources. Therefore, we follow the latest works (Chen et al. 2020a
Grill et al.] 2020 Caron et al. 2021 Bardes et al. 2022). adopt ResNet-50 backbone ImageNet- 1K and introduce some training tricks, e.g., momentum encoder (Grill et al. 2020) and multi-crop (Caron et al. 2020) into the training process. Specifically, there are two encoders used lo extract features, one is directly updated by the gradients, and the other momentummuupdteed by the former via the exponential moving average operation. As seen Table our Trip-ROMA achieves a top-1 accuracy 71.1% with 200 training epochs, which much better than the current state-of the-art SSL methods, such SimCLR (Chen et al. 2020a), SwAV (Caron et al. 2020 SimSiam Chen & He. 2021 and BYOT (Grill et al. 2020
17

---

Published in Transactions on Machine Learning Research (05/2023)
(a) SimCLR pretrained
(b) SimSiam pretrained
(c) ROMA pretrained
Figure 5: Feature visualization CIFAR-10 with pretrained ResNet-18 by SimCLR, SimSiam and Trip- ROMA, respectively, Each color indicates one class.
(a) SimCLR pretrained
(b) SimSlam pretrained
(c) ROMA pretrained
Figure Feature visualization on STL-10 with pretrained ResNet-50 by SimCLR, SimSiam and Trip- ROMA, respectively. Each color indicates one class.
F Feature Visualization
To further demonstrate the effectiveness of the proposed Trip-ROMA in a more intuitive way and show that the learned features are conducive to the classification, we visualize the feature spaces learnt by different methods on two datasets, f.e., CIFAR-10 and STL-10, respectively.
As soen Figure 5 First, three models are trained on CIFAR-10 with ResNet-18 by using SimCLR, SimSiam and Trip-ROMA, respectively. After that, all test samples CIFAR-10 are represented accordingly and then are reduced to two-dimensional space by t-SNE. As seen, the samples in the feature space learned by Trip- ROMA are more soparable than both SimCLR and SimSiam, showing that Trip-ROMA can learn better feature representation.
Similarly, as seen in Figure 0 Trip-ROMA can also make the learnt features of different categories more sep- arable than SimCLR and SimSiam, showing the effectiveness of Trip-ROMA unsupervisod representation learning. In addition, features learnt by SimSiam are more chaos than the features learnt by SimCLR and Trip-ROMA, showing that negative pairs beneficial to learn more discriminative feature representations.
18

---

calculate the gradients of Lig with respect to the Published in Transactions on Machine Learning Research (05/2023)
G Gradients Analysis
To further explain the effectiveness of the proposed loss function in Eq. (3) espocially the complementary property of the two loss terms, we present the gradients analysis of this loss function. Recall that the formula of the proposed simple triplet + binary cross-entropy loss for a triplet
exp(2, 2./T) exp(z, +ep(2) 25/T)
Cg 2 2, z,
+
1]+ A log
(8)
To explain the effectiveness of each part in Eq. positive sample zi as below:
8
[zj - Ei) +
(AFE 2j -
(
x+x)2;) if if 2i < 2 2j +
0z,
A
(
B )
otherwise,
1
where == exp(=, zj/T) and B = exp(z) 2 T./T). From Eq. (9), we have the following observations: (1) the gradient of the cross-entropy loss with respect to 2; has the same formula as the triplet loss, which means that these two loss items are complementary; (2) when the gradient of the triplet loss is zero, we still have the gradient of the binary cross-entropy loss to optimize the model.
19


Reviews:
[{"review_id": "Gwe71RF9SD", "reviewer": "Reviewer_zTHE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* The methods are clearly exposed.\n* The experiments are extensive. I appreciate in particular the 3-fold repetition and the addition of standard errors in Table 1.\n\n# Weaknesses\n* The paper feels like a combination of two unrelated methods - the use of a batch-based triplet loss combined with cross-entropy on one side, and the use of random projections of the other. Combining these two unrelated methods make the assessment of the significance of these two contributions more difficult in experiments where the ablation of using only one of the two is not done (e.g.: \"Longer training process).\n* The intuition to ROMA is somewhat unclear (see Requested changes 1. below)\n* Some additional experiments would strenghten the experimental section significantly by reducing the unstudied hyperparameters (e.g. influence of $\\lambda$, and dimension of projected subspaces in ROMA; see points in \"Requested changes\").", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose two main extensions to existing contrastive self-supervised learning methods. First, they experiment with the use of a triplet loss (Trip.) with a single negative, by opposition with methods using a large number of negative examples or methods using only positive examples. Second, they propose the introduction of random mappings (ROMA) added on top of the computation of the embeddings in SSL methods.\nThe authors study the impact of their method in linear evaluation and few-shot-learning settings on different ResNet architectures and various datasets. Through ablation studies, the authors claim that both Trip and ROMA bring significant improvements in SSL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Open questions\nIn the spirit of TMLR, I believe the provided methods and results can already interest some readers and the claims experiments are technically correct. However the work would be greatly strenghtened by addressing the following open questions.\n\n1. Intuition for ROMA: The justification for ROMA is unclear to me and I would like to have more certainty that the observed effects are consistent with the explanation. First, it is unclear what the authors call \"overfitting\" or \"by chance\" in the triplet relation, and this argument could be made more expicit. Second, it would be good to know why the random projections approach is actually beneficial, as my intuition would be that adding triplet relationships on the higher-dimensional space should already imply some triplet relationships on the projected images. Clarifying this intuition would be important before publication. Additionally, the authors could complement their experiments with varying the dimension of the projected subspaces, or by using axis-aligned subspaces (i.e. subsampling the embedding coordinates).\n\n2. Impact of $\\lambda$ (eq. 1): $\\lambda$ seems arbitrarily set to 8 in the experiments. The work would be strenghtened by an ablation on the impact of $\\lambda$ including the edge cases where we have only the triplet loss ($\\lambda=0$) or only the cross-entropy loss (would require the addition of e.g. a ($1-\\lambda$ factor for the first term and set $\\lambda=1$) would be valuable in assessing if both loss terms contribute.\n\n3. Influence of batch size: Trip. is compared with other methods with larger batch size. It would be beneficial to compare when using the same batch size and see if the gains of Trip are more important. Alternatively, it would be interesting to know if the performance of competing methods decreasses when using a smaller batch size.\n\n# Misc.\n4. Introduction: I do not agree with the author's claim that the asymetric momentum encoder or stop-gradient approaches of BYOL or SimSiam correspond to a \"sophisticated\" design. Indeed, these approaches are conceptually simple and easy to implement. I would however agree that requiring less batch size and could be a positive for using only a batch-based triplet loss.\n\n5. Please distinguish between `\\citep` to add parentheses to the citations, or `\\citet` when the citations can be integrated in the text. At the moment the uniform use of `\\citet` hinders the reading.\n\n6. Sec. 5.2: please remind the reader of what \"FSL\" means (few-shot learning).\n", "broader_impact_concerns": "The paper raises no appearent ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8VBLhw8sHt", "reviewer": "Reviewer_6v58", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The exploration of the triplet loss is important for reducing the sizes of batch-size and memory banks in SSL.\n2. The overall writing is good and can be easily followed.\n3. The article gives a lot of experimental results, especially ablation experiments.\n\n\nWeaknesses:\n1. Why is the combination of two triplet losses better than any one of them? I think this part should give more explanation.\n2. Critical experiments are missing. For example, because the result in ImageNet-1K uses multi-crop (which is lower than SwAV in the original paper), the effectiveness of the proposed method on large datasets cannot be verified.\nFor more details, please check \"Requested Changes.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a method that employs a Triplet-based loss and a random mapping strategy (ROMA). For the Triplet-based loss. There are two parts: a triplet loss and a cross-entropy loss for binary classification. ROMA employs a random matrix, which can be seen as many linear projections for vectors. For each projection, ROMA brings two positive vectors closer and pushes away the negative vector. The experimental results show the proposed method can largely improve the performance of small datasets and FSL tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In page 1, \"the existing SSL methods are mainly ... while paying less attention to small data.\" There are some existing SSL methods that work well on small datasets.\n\n[1] ReSSL: Relational Self-Supervised Learning with Weak Augmentation NerualIPS 2020\n[2] RSA: Reducing Semantic Shift from Aggressive Augmentations for Self-supervised Learning. NerualIPS 2021\n[3] Whitening for Self-Supervised Representation Learning. ICML 20212.\n\n2. On page 4, \"τ is a temperature parameter that is set as 0.5 in our work.\" Hyper-parameters should be given in the experimental part.\n\n3. In the ImageNet-1K experiments, the authors adopt a multi-crop strategy, which can significantly improve the performance and increase the training cost. I think the authors should mention it in the paper, not supplimentary. For a fair comparsion and showing improvements, I suggest the results on ImageNet-1K without multi-crop should be given. BTW, SwAV with multi-crop is 72.7 in the original paper.\n\n4. I found the matrix in ROAM to be relatively large, and it is used in each pair calculation. Thus, the training time comparison is important for ROAM. In addition, the most improvements of ROAM are less than 0.5 in table 1, and some important experiments are missing, e.g., ImageNet-100 for 1000 epochs, and ImageNet-1K w/wo ROAM.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GRLllCQudl", "reviewer": "Reviewer_HcGJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n* Their observation is very interesting and practical. Although many existing SOTA methods are training SSL models with a large computational cost, the observation implies that we can dramatically reduce the cost by using their approaches. \n* Their proposed random matrix projection, ROMA, is simple, yet useful for various SSL methods. This is also a great strength of this submission. \n* The evaluation is extensive. The experiments cover almost everything I wanted to see. \n* The submission is well-organized and easy to follow. \n\n ### Weaknesses\n* The reason why simple triplet + cross-entropy loss is so effective on this task is not very well analyzed. I think it is a very interesting observation. But, if there is any analysis to understand the reason, the paper will be more solid. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces observations related to contrastive self-supervised learning and a new approach that can reduce the training cost of the contrastive SSL, yet improve the quality of learned representations. Specifically, they show two observations: (i) the simple triplet loss plus cross-entropy loss, which requires only one negative sample per one positive sample, can achieve comparable or better performance than the existing contrastive SSL approaches requiring many negative samples, (ii) in addition to (i), projecting samples by a random matrix can also boost the performance. Their proposed approach combining (i) and (ii) can outperform many existing SSL methods. They conduct evaluations on diverse datasets and provide wide analysis including ablation studies. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I do not see any major concerns with this submission. The submission is already very well-written and has content that can be very impressive to the audience. My only concern is about the justification for why the proposed method is working so well. The existing papers' assumption that we require many negative samples to boost performance on contrastive SSL may not be true from this submission's result. But, there is not much empirical analysis on this point. ", "broader_impact_concerns": "No. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The results in this paper may encourage others to investigate approaches that require few negative examples, making for scalable learning procedures. Two of the reviewers had no hesitations in recommending acceptance after discussion with the authors. The third reviewer pointed out that in a particular setting, without multi-cropping, performance was below SOTA, and that the method was somewhat complex. Overall i dont think this is of sufficient concern to overrule the other two assessments. Even if the model is a little complex, it still may have useful insights that later papers can use in more elegant ways.
Decision: Accept (poster)