SemPPL: Predicting Pseudo-Labels for Better Contrastive Representations
Authors: Matko Bošnjak
Pierre Harvey Richemond
Nenad Tomasev
Florian Strub
Jacob C Walker
Felix Hill
Lars Holger Buesing
Razvan Pascanu
Charles Blundell
Jovana Mitrovic
Venue: ICLR 2023 poster (2023)
Keywords: contrastive learning
representation learning
semi-supervised learning

Abstract: Learning from large amounts of unsupervised data and a small amount of supervision is an important open problem in computer vision. We propose a new semi-supervised learning method, Semantic Positives via Pseudo-Labels (SEMPPL), that combines labelled and unlabelled data to learn informative representations. Our method extends self-supervised contrastive learning—where representations are shaped by distinguishing whether two samples represent the same underlying datum (positives) or not (negatives)—with a novel approach to selecting  positives. To enrich the set of positives, we leverage the few existing ground-truth labels to predict the missing  ones through a k-nearest neighbors classifier by using the learned embeddings of the labelled data. We thus extend the set of positives with datapoints having the same pseudo-label and call these semantic positives. We jointly learn the representation and predict bootstrapped pseudo-labels. This creates a reinforcing cycle. Strong initial representations enable better pseudo-label predictions which then improve the selection of semantic positives and lead to even better representations. SEMPPL outperforms competing semi-supervised methods setting new state-of-the-art performance of 76% and 68.5% top-1accuracy when using a ResNet-50 and training on 10% and 1% of labels on ImageNet, respectively. Furthermore, when using selective kernels, SEMPPL significantly outperforms previous state-of-the-art achieving 72.3% and 78.3% top-1accuracy on ImageNet with 1% and 10% labels, respectively, which improves absolute +7.8% and +6.2% over previous work. SEMPPL also exhibits state-of-the-art performance over larger ResNet models as well as strong robustness, out-of-distribution and transfer performance. We release the checkpoints and the evaluation code at https://github.com/deepmind/semppl.

Published as a conference paper at ICLR 2023
SEMPPL: PREDICTING PSEUDO-LABELS FOR BETTER CONTRASTIVE REPRESENTATIONS
Matko Bošnjak, Pierre H. Richemond, Nenad Tomasev, Florian Strub, Jacob C. Walker Felix Hill, Lars Holger Buesing, Razvan Pascanu, Charles Blundell, Jovana Mitrovic DeepMind {matko, richemond,
ABSTRACT
Learning from large amounts of unsupervised data and a small amount of super- vision is an important open problem in computer vision. We propose a new semi- supervised learning method, Semantic Positives via Pseudo-Labels (SEMPPL), that combines labelled and unlabelled data to learn informative representations. Our method extends self-supervised contrastive learning-whan where representations are shaped by distinguishing whether two samples represent the same underlying datum (positives) or not (negatives) tives. To enrich the set of positives, leverage the few existing ground-truth labels to predict the missing ones through a k-nearest neighbours classifier by using the learned embeddings of the labelled data. We thus extend the set of positives with datapoints having the same pseudo-labe, and call these semantic positives. We jointly learn the representation and predict bootstrapped pseudo- labels. This creates a reinforcing cycle. Strong initial representations enable better pseudo-label predictions which then improve the selection of semantic positives and lead to even better representations. SEMPPL outperforms competing semi- supervised methods setting new state-of-the-art performance 68.5% and 76% top-1 accuracy when using a ResNet-50 and training on 1% and 10% of labels on ImageNet, respectively. Furthermore, when using selective kernels, SEMPPL sig- nificantly outperforms previous state-of- otate-of-the-arr achieving 72.3% and 78.3% top-1 accuracy on ImageNet with 1% and 10% labels, respectively, which improves absolute +7.8% and +6.2% of-the-art performance over larger ResNet models as well as strong robustness, out-of-distribution and transfer performance. We release the checkpoints and the evaluation code at https: /ttt:s::/githeee com/ deepmind/ semppl.
novel approach to selecting posi-
previous work. SEMPPL also exhibits state-
INTRODUCTION
In recent years, self-supervised learning has made significant strides in learning useful visual fea- tures from large unlabelled datasets [Oord al., 2018; Chen et al., 2020a; Mitrovic et al., 2021; Grill et al., 2020; Caron et al., 2021]. Moreover, self-supervised representations have matched the performance of historical supervised baselines on the ImageNet-Ik 1k benchmark [Russakovsky al., 2015] in like-for-like comparisons as well as outperformed supervised learning in many transfer settings [TTomasev al., 2022]. While such results show exciting progress in the field, in many real-wold applications often there exists small amount of ground-trude labelled datapoints making the problem of representation learning aami-supevvised.
In this work we propose a novel approach to semi sumi-supervieed learning called Semantic Positives Pseudo-Labels (SEMPPL) which incorporates supervised information during the representation learning stage within self-supervised loss. Unlike previous work which uses the available super- vision as targets within a cross-entropy objective, we propose to use the supervised information to help inform which points should have similar representations. We propose learn representations using a contrastive approach, i.e. we learn the representation of a datapoint (anchor) by maximizing the similarity of the embedding of that datapoint with a set of similar points (positives). while simul- tancously minimizing the similarity of that embedding with a set of dissimilar points (negatives). As such, the appropriate construction of these sets of positives and negatives is erucial the success of

---

Published as a conference paper at ICLR 2023
contrastive learning methods. While strategies for sampling negatives have been extensively studied in the literature [Schroff et al., 2015; Harwood et al., 2017: Ge et al., 2018; Wang et al., 2019a; 1a; He et al., 2020; Chen et al., 2020c], the sampling of positives has received far less attention.
We propose a novel approach to selecting positives which leverages supervised information. Specif- ically, we propose using the small amount of available ground truth labels in order non- parametrically predict the missing labels (pseudo-labels) for the unlabelled data. Note that many previous semi-supervised approaches use pseudo-labels targets within a cross-entropy- jective [Van Engelen & Hoos, 2020; Yang et al., 2021]. In SEMPPL we use pseudo-labels in a very different way, i.e. we use them select positives based on whether two datapoints (we call these semantic positives) share the same (heuddo)labell By maximizing the similarity of datapoint with its semantic positives we expect learn representations that are more semantically aligned and as a consequence encode more abstract, higher-level features which should generalise better. To predict informative pseudo-labels, we compare the representations of the unlabelled data with those of the labelled subset and use k-nearest neighbours (k-NN) classifier to impute the missing labels.
ob-
We simultaneously learn the representation, predict pseudo-labels and select semantic positives. This creates virluous cycle: better representations enable better pseudo-label prediction which in turn enables better selection of semantic positives and thus helps us learn better representations. Im- portantly, as the prediction of eseudo-labela and selection of semantic positives does not depend on the exact form of the contrastive objective employed, SEMPPL is compatible with and complements all contrastive losses, e.g. [Chen al., 2020a;b; Caron al., 2020; He et al., 2020; Mitrovic al., 2021] and may even be extended to non-contrastive losses [Grill et al., 2020; Chen & He, 2021].
We evaluate the representations learned with SEMPPL across a varied set of tasks and datasets. In particular, SEMPPL sets new mate-of-the-ar in semi-supervised learning on ImageNet with 1% and 10% of labels on the standard ResNet-50 (1x) architecture with respectively 68.5% and 76.0% top- 1 performance and across larger architectures. When combined with Selective Kernels [Li et al., 2019b], we achieve 72.3% and 78.3% top-1 performance with 1% and 10% labels, respectively, significantly outperforming previous state-of-the-are by absolute +7.8% and +6.2% in top-1 perfor- mance. We also outperform previous state-of-the on robustness and distrf-distribution (OOD) generalisation benchmarks while retaining competitive performance in transfer learning.
Our main contributions are:
We extend contrastive learning to the semi-supervised setting by introducing the idea of es- timating pseudo-labels for selecting semantic positives as key component especially in the low-label regime,
We propose a novel semi-supervised method SEMPPL that jointly estimates pseudo-labele selects semantic positives and Iearns representations which creates virtuous cycle and enables us to learn more informative representations,
We extensively evaluate SEMPPL and achieve state-of-ther thef-tte-rtt in semi-supervised learn- ing, robustness and out-of-distributioo generalisation, and competitive performance in transfer.
2 SEMANTIC POSITIVES VIA PSEUDO-LABELS
The selection of appropriate positive and negative examples are the cornerstone of contrastive learn- ing. Though the research community has mainly focused on the selection of negatives, positives are equally important as they play a vital role in learning semantic similarity. We thus leverage labelled information as it encodes semantic information to improve the selection of informative pos- itives. Specifically, we expand self- self-supervised model to use this labelled data to non- non-parametrically predict paeudo-labels for the remaining unlabelled data. Using both ground-truth labels and the pre- dicted pscudo-labels, we expand the set positives with semantic positives.
Notations Let D DrUD, be dataset consisting of labelled training data Dj - (Zi, v4) Jo and unlabelled training data D,, = (z with M > N. Let B be a batch of data of size B with
B = ((x, 1n) })) U { - where (II, E D; and x D.,. where the indices i, j and to denote labelled, unlabelled, and all datapoints, respectively. Following established supe-supe learning practices [Chen et al., 2020a;b; Caron et al., 2020; Mitrovic et al., 2021; Dwibedi et al., 2021; Tomasev al., 2022], we create different views of the data by applying pairs of randomly
2

---

Published as a conference paper at ICLR 2023
Loss
Loss
hogativas
Predicton
h
Projection
Projection
Projection
Passido
Projection
Projection
g
St
Embeddling
Emboddling
Projection
Emoedding
Projection
Query
Sketch of SEMPPL. (Left) Standard contrastive pipelines. (Middle) Unlabelled data are tagged with pscudo-labels by using k-NN over projected labelled data. (Right) Semantic positives are queried from the queue and processed to compute an additional contrastive loss.
Figure 1:
sampled augmentations 01,43 42 A from the augmentation distribution A proposed in Chen et al. [2020a]. For every datapoint vm we denote the corresponding augmentations as 2 4m
Augmentation positives We embed one data view x via online encoder network f and embed the other data view 2a with a targef encoder network ft, i.e. we get latent representations am = (x ) and 2,,,- ft(2%)) Note that the weights of f. weights of f. Next, we pass these latent representations through projection and prediction multi- layer perceptrons. Specifically, we use online projector g and target projector ft, well an online predictor h, to further transform 241 and 2 again, the weights of 9t are an exponential moving average of the weights of g. We then get 2 = h(g(= and = ge(2m, and l2- normalise these; we use 2 =m, onward as the normalised latent embeddings.
exponential moving average of the
In order learn the representation of 2 contrast it against the augmentation-baset positive 202 as well as against negatives. For this, we use the contrastive loss: "m,t
(22m 202
LACOM Σ log
60 2mm -
+. (N()m) 50( 201 2n,t
m=1
where N(xx) is the set of negatives, randomly uniformly sampled from the current batch, E ge (ff(())) the target network projection of the negative sample; s0(1, = exp((x1 (01, x2) the scoring function,
is a scalar temperature, and (: denotes the Euclidean dot product. Since the representations we contrast are Iy-normalised, the dot product effectively turns into cosine similarity.
Pseudo-labe prediction and semantic positives Since we have access to a small labelled dataset, we can use the label information select more informative positives beyond just augmentations of the original image. Specifically, we can associate images with the same label as positives and we call these semantic positives. We want to select semantic positives for all the data, not just the labelled subset. For this purpose, we propose compute pseudo-labels for the unlabelled data and use this to select semantic positives. To compute pseudo-labels we compare the current latent embeddings a mest--n-first-on
of the unlabelled data to those of the labelled data. Specifically, queue Q with capacity C' for storing labelled embeddings which we use for computing the pseudo- labels. At the start of training, simply initialise the queue with random vectors, and use the queue
propose
3

---

Published as a conference paper at ICLR 2023
from the first step. For each batch B, add the target projection of only the labelled data to the queue, i.c. Q + (2 Vr). To predict a pseudo-label for an unlabelled datapoint zy, we first compute the online predictor output 2yt before retrieving its k-nearest neighbours {( Ms) in cosine similarity from the queue Q. Finally, we compute the pseudo-label By of xy as:
Mj mode (Z2, Ne)
(2)
where mode is the mode of the set, tasked with obtaining the most frequent class in the k-nearest neighbours. We use the ground-truth labels (for the labelled data) or the computed pseudo-labels (for the unlabelled data) select semantic positives for every datapoint in B. For each vm E B, we uniformly sample over all the embeddings in Q that share the same (pseudo-) label as Im to get a semantic positive " ((( 31) E Q yi = pl(zm))), where pl( ) = ymm if Im is labelled and (zm)) Fm if mm is unlabelled. Next, we include these semantic positives within our representation learning process through the contrastive objective
B Σ m=1
(0( (mm 202.
SEMPOS =
log
(3)
202.1) + Enn, (N(mm EaneN(am (p(mm
2
( 2 pn.t)
Taking these two losses (1) and (3) together, we propose to learn representations in our method SemPPL. by minimising the following total loss
CSuMPPL = LAVGM + aC
(4)
where a controls the ratio between these sub-losses.
2.1 IMPLEMENTATION DETAILS
Architecture We use Residual Networks [He et al., 2016] (vl; pre-activation as customary in the literature) for f and ft and use either 50 or 200 layers deep networks and with width multiplier ranging from x to 4x. As in [Grill et al., 2020: Tomasev et al., 2022], we use multi-layer per- ceptrons with layers of size 4096 and 256, with batch normalisation [loffe & Szegedy, 2015] and rectified linear activation.
Self-supervised learning method We use RELICV2 [Tommaev et al., 2022] as our default self- supervised training objective due to its competitive performance. Therefore, we add an invariance penalty on top of Equation to further enforce the similarity constraints and regularize the learning process detailed in Appendix B. We also explore other supervidd learning objectives in Section 4.
Algorithm parameters We queue of capacity C 20B, with batch size B 4096, and temperature = 0.2 while randomly sampling negatives from the current batch; we take [W(x) = 10 negatives in total. For augmentations, 2020a] and the RELICV2 multi-crop and saliency-based masking [TTomasev et al., 2022]; we use large views and small views for augmentation positives and 3 semantic positives. The semantic positives are computed with a k-NN with A: (see the analysis section in Appendix D): we build single K-NN instance per augmentation a queried with all the augmentations where |a| = 4. This produces |a|2 = 16 k-NN induced pseudo-labels in total for each unlabelled image among which we then perform majority voting to compute the final pseudo-label.
the standard SIMCLR augmentations [Chen al.,
Optimisation Our networks are optimized with LARS [You et al., 2017]. Our base learning rate 0.3 and we train our models for 300 epochs with a learning rate warm-up period of 10 epochs and cosine decay schedule thereafter. We use a weight decay of 10 and batch size B 4096. We exclude the biases and batch normalisation parameters both from LARS adaptation and weight decay. The exponential moving average parameter for target networks is 0.996. Our pseudo-code described in the appendix along with precise architectural and implementation details. Pretrained model checkpoints and code will be made available on GitHub.
We use the cosine similarity as the embeddings are normalised.
4

---

Published as a conference paper at ICLR 2023
3 EXPERIMENTAL RESULTS
To evaluate SEMPPL, we pre-train representations using 1% and 10% labelled data from the Ima- geNet dataset [Russakovsky et al., 2015] based on the splits from Chen et al. [2020a]. We then test SEMPPL in semi-supervised classification, robustness and out-of-distribution generalisation tasks. Lastly, we probe the transfer capabilities of the representations to other image classification datasets. For complete of results and experimental details, please see the Appendix A.
3.1 SEMI-SUPERVISEN LEARNING
In Table report top-1 aceuracy the ImageNet test when either 1% 10% of the data labelled for the ResNet-50 architecture as well as deeper and wider ResNets. SEMPPL achieves top-1 accuracy of 68.5% with 1 % of labels, significantly outperforming the previous state-of-the-nn SimMatch [Zheng et al., 2022] by an absolute + +1.3% in ImageNet test accuracy. With 10% of label data, our top-1 accuracy
ResNet-50 reaches 76.0%. outperforming the previous state-of-the-An PAWS [Assran et al., 2021]in in semi-supervised learning. SEMPPL outperforms competing represen- tation learning methods across the board, achieving state-of-the-an performance on all ResNet-50 2x, ResNet-50 x and, in both the 1% and 10% labelled settings. SEMPPL does not use, and there- fore excludes from comparison, distillation from larger networks as [Chen et al., 2020b; Pham et al., 2021].
Similar to [Chen et al., 2020b], we also tested SEMPPL on ResNets with Selective Lernels (SK) [Li et al., 2019b]. This increases the encoder parameter count to 27.5M. We thus achieve new absolute state-of-the-art of 72.3% and 78.3% top-1 accuracies, respectively, when using 1% and 10% of labelled data. Finally, SEMPPL reaches new ofate-of-the-ar using 76.0 and 80.5 on 1% and 10% of labels without self-distillation with ResNet-200 2x + SK architecture.
For implementation details of the semi-supervised results and additional results, see the Ap- pendix A.I.
Table 1: Top-1 accuracy (in %) for ResNet encoders with different depth and width.
ResNet-50 1x
ResNet-50 2x
ResNet-50 4x
ResNet-200 2x
Method
Top-1
Top-1
Top-1
Top-1
10%
1%
10%
1%
10%
10%
SimCLR [Chen al., 2020a]
48.3
65.6
58.5
71.7
63.0
74.4
BYOL [Grill et al., 2020]
53.2
68.8
62.2
73.5
69.1
75.7
71.2
77.7
RELICv2 [Tomasev al., 2022]
58.1
72.4
64.7
73.7
69.5
74.6
72.1
76.4
SimCLRv2 [Chen al., 2020b]
57.9
68.4
66.3
73.9
CoMatch [Li et al., 2021a]
66.0
73.7
PAWS [Assran et al., 2021]
75.5
69.6
77.8
69.9
79.0
SimMatch [Zheng et al., 2022]
74.4
SemPPL (ours)
76.0
71.9
78.6
72.5
79.3
74.8
80.4
SimCLRv2 + SK [Chen et al., 2020b]
64.5
72.1
70.6
77.0
SemPPL + SK (ours)
72.3
78.3
74.5
79.8
76.0
80.5
3.2 ROBUSTNESS AND OOD GENERALISATION
We evaluate the robustness and generalisation abilities of SEMPPL on ImageNetV2 [Recht al., 2019], ImageNet-C [Hendrycks & Dietterich, 2019], ImageNet-R [Hendrycks et al., 2021] and Ob- jectNet [Barbu et al., 2019] which have all been purposefully constructed to test different robustness and generalisation aspects. We evaluate all three variants on ImageNetV2: matched frequency (MF), Threshold 0.7 (T-0.7) and Top Images (TI). When evaluating PAWS, we used the publicly available checkpoints. Table 2 shows good robustness and generalisation ability of the representations learned with SEMPPL. SEMPPL sets the new state-of-the-ar performance (outperforming even the super- vised baseline) 4 out of 5 datasets, while outperforming PAWS across all datasets. SEMPPL also outperforms SimMatch on 4 out of 5 datasets. For more details on the evaluation protocols and results for ImageNet-C see the Appendix A.2.
5

---

Published as a conference paper at ICLR 2023
Table Top-1 accuracy (in %) for ImageNetV2, ImageNet-R and ObjectNet.
Robustness
OOD generalization
Method
MF
T-0.7
Ti
ImageNet-R
ObjectNet
Supervised (100% labels) [Lim et 20191
65.1
73.9
78.4
24.0
26.6
Semi-supervised (10% labels)
PAWS [Assran al., 2021]
64.5
73.7
23.5
23.8
SimMatch [Zheng al., 2022]
63.8
73.2
78.3
25.0
24.5
SemPPL. (ours)
65.4
74.1
79.6
24.4
25.3
Table 3: Top-1 accuracy (in %) the full suite of transfer tasks.
Method
Food101
CIFARIO
CIFAR100
Birdsnop
SUN397
Cars
Aireraft
DTD
Pets
Caltech Flowers
Supervised-IN IN [Chen ol., 2020a]
72.3
93.6
78.3
53.7
61.9
66.7
61.0
74.9
91.5
94,5
94.7
Semi-superiieed (10% labely)
PAWS [Assmmn al,, 202.1]
79.1
92.3
76.3
62.0
66.1
75.7
61.4
77.0
92.2
91.9
96.5
SimMatch [Zhong 2022]
93.6
78.4
93.2
SEMPPL (ours)
80.2
92.5
77.6
64.2
66.3
75.5
63.9
77.8
92.5
93.0
96.3
3.3 TRANSFER LEARNING
We evaluate the generality SEMPPL representations by testing whether the features learned on ImageNet are useful across different datasets and tasks. Specifically, we evaluate the transfer per- formance of SEMPPL on set of image classification datasets commonly used in the contrastive literature under the linear protocol [Grill et al., 2020; Chen et al., 2020a; Dwibedi et al., 2021; Mitro- al., 2021; Tomasev et al., 2022]. For the linear protocol, the pretrained encoder is frozen and randomly initialized linear classifier is trained on top using the training data from the target dataset. We report standard metrics for each dataset as well as performance on held-out test set. For more details on the evaluation protocol see the Appendix A.3. Table 3 compares the transfer performance of representations pretrained using the supervised baseline [Chen et al., 2020a], PAWS [Assran et al., 2021]. SimMatch [Zheng al., 2022] and method SEMPPL. SEMPPL outperforms the super- vised baseline on 8 out of 11 datasets, PAWS on 9 out of 11 datasets, while showing competitive performance SimMatch, outperforming it out of datasets.
3.4 FULL LABELLED DATASET
We also assess how SEMPPL. be- haves a fully supervised setting- For this purpose, we select semantic positives based on the ground-truth labels and fine-tune the learned rep- resentations with the full ImageNet dataset. We compare against strong supervised baselines on ResNets as well as against recent performant net- work architectures that are extensions of the ResNet, e.g. [Liu et al., 2021b; 2022]. Our method reaches 79.7% top-1 accuracy on a ResNet 50 out- performing number of strong super- vised baselines. When we add se- lective kernels to ResNet 50, we achieve 82% top-1 accuracy outper- forming recent transformers architec-
Figure Top-1 accuracy for ResNet50 with 100% of the labels across augmentations, initializations and networks.
Method
Params
Top-1
Supervised (ResNei-50)
Automnt [Cubuk et al., 2019]
27M
77.6
Max [Gong et al., 20211
27M
78.9
Representation Learning (ResNet-50)
SEMPPL (SimCLR basc)
27M
76.0
SEMPPL (BYOL base)
27M
77.7
SEMPPL (ReLICv2 base; ours)
27M
79.7
Other Architectures
Swin-T [Liu al., 2021b]
29M
81.3
ConvNeXt [Liu et al., 2022]
29M
82,1
SEMPPI. + SK (ours)
29M
82.0
ture [Liu al., 2021b], and matching highly tuned ConvNext [Liu et al., 2022]. Therefore, SEMPPL may also be considered promising pretraining method in the supervised learning setting.

---

Published as a conference paper at ICLR 2023
4 ANALYSIS
We analyse the impact of different design choices in SEMPPL on downstream performance. In this section, we focus the behaviour and impact of pseudo-labels and semantic positives on learning representations. For further analyses and experimental details, please see Appendix D.
Semantic positives across self-supervised learning objectives With SEMPPL we extend the set of positives include semantic positives based on predicted pseudo-labels; we can combine these ideas with other self- supervise methods. In Table 4a, we additionally evaluate SEMPPL on the non- contrastive self supervised method BYOL [Grill et al., 2020]. BYOL replaces the contrastive loss in Equation 4 with an l2 loss. Importantly we follow the training pipeline (e.g. augmentation, hyper- parameters etc.) from [Grill et al., 2020] to fairly highlight the impact of SEMPPL. We observe a drastic improvement when adding semantic positives. With 1% labels ImageNet BYOL improves by absolute +3.9% and by absolute +3.6% when using 10% labels. For completeness we have also highlighted the contribution of SEMPPL when using RELICV2 as the base self-supervised objec- tive which is our default implementation. For 1% labeled data, we see an absolute improvement of a gain of absolute +3.6% in top-1 accuracy.
+ +10.4% in top-1 accuracy, while for 10% labels In summary, we see that SEMPPL can be easily combined with other self-supervised objectives to yield significant improvements and can be used a plug-and-play module semi-supervised learning.
The contribution of pseudo-labels and semantic positives We examine the impact of omitting pseudo-label prediction and semantic positives from learning representations. Specifically, we ab- late the use of pseudo-labels when selecting semantic positives for unlabelled datapoints, i.e. we only use labelled images when retrieving semantic positives. In Table 4b (middle row), removing pseudo-label prediction significantly decreases performance both in the 1% and 10% label settings. In addition, the low-label regime (1% labels) suffers a stronger performance decrease -6.6% than the 10% labels regime, -4.9%. This underscores the importance of pseudo-lahel estimation and subsequent selection of semantic positives for unlabelled data especially in the low-data regime. Going a step further, we remove semantic positives even for the labelled data, falling back to vanilla RELICV2. In Table 4b (bottom row), we again significant drop in performance for both the 1% and 10% label settings with a sharper drop for the low-label regime. Together these highlights the importance of both including semantic positives for labelled data as well as using pseudo-labe prediction for selecting semantic positives for unlabelled data in order to learn informative represen- tations in semi-supervised setting in label-efficient way.
Precision and Recall of pseudo-labels. In Figure 3, we analyse the behaviour of pseudo-labels by looking at the precision and recall as training progresses. We train a ResNet-50 for 100 epochs using 10% labels with SEMPPL on ImageNet. As we have 4 large views there will be in total 16 votes cast and then the pseudo-label will be estimated using majority voting. We want to measure how often these 16 votes agree disagree; least l votes have been cast for one class. We see that as training progresses the precision across all thresholds increases as expected. This means that the pseudo-label prediction is bootstrapping itself to become more accurate, which enables us to select better semantic positives and thus learn more informative representations as training progresses, i.e. we have virtuous cycle of representation leamning and pseudo-labe prediction. Furthermore, precision is an increasing function of the voting
denote as voting threshold the number l where
Table 4: Top-1 test accuracy (in %) with ResNet50 pretrained on ImageNet with 1 1% and 10% labels.
Top-1
1%
10%
1 1% labels
10% labels
BYOL [Grill et al., 2020]
53.2
68.8
SEMPPL
68.5
76.0
SEMPPI. with BYOL
57.1
72.4
Pseudo-labels
61.9
71.1
Semantic Positives
58.1
72.4
ReLICv2 [Tomasev et al., 2022] SEMPPL with RELICv2 (ours)
58.1
72.4
68.5
76.0
(b) Removing pseudo-labelling and semantic posi-
tives in SEMPPI.
(a) Trained on different self- self-supervised objective.

---

Published as a conference paper at ICLR 2023
Table 5: Top-1 accuracy (in %) with ResNet50 pretrained on ImageNet 10% labels for 100 epoches when using ground truth labels instead of pseudo-label, while retrieving semantic positives (oracle); PL. accuracy holds for pseudo-label accuracy.
10% labels
Top-1
PL accuracy
SEMPPL SEMPPL (+oracle)
69.9
69.7
71.6
76.9
threshold throughout training and is highest for the biggest voting threshold. This indicates how confident we can be in the accuracy pseudo-label prediction, and thus how confident we can be that an appropriate semantic positive has been selected. Yet, see that the recall for individual thresholds is also increasing as training progresses but that the recall decreases as we increase the voting threshold. This is expected as there always a trade-off between precision and recall.
Vating threchold
0.4
0.4
5000
10000
15000
20000
25000
30200
5000
10000
15050
20000
25000
30000
Stops
Stops
Precision and recall for pseudo-labels computed based k-nearest neighbours when trained on ImageNet with 10% labels over 100 epoches.
Figure 3:
Noise in pseudo-label prediction In Figure 3, we observe that the proportion of correctly pre- dicted pscudo-labels the end of training reasonably high (60% accuracy of voting threshold 0). Yet. it also means that 40% of pseudo labels are still incorrectly predicted. As the incorrect prediction results suboptimal semantic positives selection, i.e., SEMPPL does not select semantic positives from the same class the datapoint, this behavior may ultimately worsen the quality of ex- tracted features. To quantify this phenomenon, we train the representation with SEMPPL where an oracle replaces the pseudo-labe prediction with ground ground-truth labels and those are used for selecting semantic positives. Table 5, we train the representations for 100 epochs on ImageNet with 10% labels. There, the oracle increases the top-1 performance of 1.7% in the test set with 10%. Besides, the pseudo-label accuracy also gets 6.2% higher. thus confirms that incorrect pseudo-label predic- tions, and incorrect semantic positives retrieval, hurts learning informative representations and the downstream performance. Yet, the oracle performance remains close to the actual performance of SEMPPL, illustrating the method's robustness.
Further ablations on other design choices, such as the number of semantic positives, the use of view voting, the choice in k-NN, queue length and training duration can be found Appendix D.2.
RELATED WORK
Semi-supervised learning In the semi-supervised regime [Cheplygina al., 2019; Van Engelen & Hoos, 2020; Yang et al., 2021; Alizadehsani et al., 2021], one can either pre-train model on unlabelled data and subsequently fine-tune it on labelled data, or train both jointly. Joint training on labelled and unlabelled data often involves combining the two losses [Grandvalet & Bengio, 2004; Miyato et al., 2018; Zhai et al., 2019; Verma et al., 2019; Berman et al., 2019; Xie et al., 2020a]. Pseudo-label self-training approaches [Zoph et al., 2020] present important alternative, first inferring approximate pseudo-labels for the unlabelled examples, and then incorporating them in supervised losses. Pseudo-labels either be generated prior subsequent supervised learning
8

---

Published as a conference paper at ICLR 2023
phase [Yarowsky, 1995; Riloff, 1996; Lee al., 2013] or jointly in an online fashion [Berthelot et al., 2019; 2020; Sohn et al., 2020]. These methods may benefit from pseudo-label confidence measures [Sohn et al., 2020; Rizve al., 2021; Zhang et al., 2021] as well as thresholding [Xu et al., 2021], temporal ensembling [Laine& & Aila, 2017], or stronger regularization to mitigate bias in early model training [Sajjadi al., 2016; Arazo al., 2020]. The use of pseudo-labels with rebalancing has shown improvements, both in class-imbalance problems [Wei et al., 2021] and in a general context [Wang et al., 2022]. Teacher-student network configurations for generating and utilising pseudo-labels have also shown promise (Tarvainen & Valpola, 2017; Luo et al., 2018; Ke et al., 2019; Xie et al., 2020b; Cai et al., 2021; Pham et al., 2021]. Co-training uses different feature extractors for different data views and alternates between pseudo-labelling and training phases [Blum & Mitchell, 1998; Qiao et al., 2018]. Good performance has been reached by using consistency losses between pseudo-labels of different inputs [Verma et al., 2019; Hu et al., 2021].
Predicting view assignments with support samples [Assran et al., 2021] (PAWS) has resulted in substantial performance improvements, with the idea that the assigned pseudo-labels ought to be similar across multiple views of the same image. Recent work has shown that incorporating label information in positive selection in contrastive methods is highly promising, compared to the cross- entropy loss in the fully supervised case [Khosla et al., 2020]. Our method demonstrates a similar utility of pseudo-labels for semi-supervised problems, and differs from competing ones in the fol- lowing ways. Unlike DebiasPL. [Wang et al., 2022] that uses an adaptive margin loss, SemPPL. does not seek to directly address or re-shape the distribution of pseudo-labels. Unlike SimCLRv2 [Chen et al., 2020b]. we do not rely on self-distillation procedures, In contrast with PAWS [Assran et al., 2021], we fully leverage the contrastive approach for semi-supervised learning: not using positives only for training means SEMPPL does not require specific care like pseudo-labels sharpening to stabilize learning and avoid representational collapse. SEMPPL is more closely related to CoMatch [Li et al., 2021a] that also uses bootstrapping to improve pseudo-labels representational quality, but is conceptually much simpler, avoiding phases of distributional alignment and of performing graph- based contrastive learning. In similar vein, SimMatch [Zheng al., 2022] also uses a memory buffer to propagate pseudo-labels, but has a more complex objective than SEMPPL and equally requires additional phases of pseudo-labels unfolding and aggregation to function.
Self-supervised learning Major advances in learning useful representations from unlabelled data [Liu et al., 2021a; Goyal et al., 2021] can be seen as a paradigm shift, since these methods have recently been competitive with supervised training baselines [Tommaev et al., 2022]. A number of self-supervised learning methods involve contrasting multiple views of the data [Oord et al., 201 8; Bachman et al., 2019; Chen et al., 2020a; He et al., 2020; Grill et al., 2020; Dwibedi et al., 2021]. Similar performance were also achieved by bootstrapping based multi-view learning [Grill et al., 2020; Richemond et al., 2020; Chen & He, 2021; Zbontar et al., 2021; Wang et al., 2021], or involy- ing explicit clustering steps [Caron et al., 2020; Asano et al., 2020; Li et al., 202 An explicit causally- mottivaed invariance loss, when used in conjunction with the contrastive objective, has been shown to lead to more compact representations, and desirable generalisation properties [Mitro- vic et al., 2021; Tomasev et al., 2022]. Contrastive approaches are not always used in self-supervised methods [He et al., 2021; Ermolov et al., 2021; Chen et al., 2022]. Transformer-specffic methods have been devised [Caron al., 2021; Chen et al., 2021; Zhai et al., 2022].
CONCLUSION
In this work, we propose SEMPPL, novel semi-supervised learning method to incorporate semantic positives in self-supervised objectives by taking advantage of pseudo-labels. Through extensive empirical evaluation, we demonstrated that our approach achieves state-of-the-art semi-supervised performance on ImageNet across several ResNet architectures as well as on the robustness, out-of- distribution generalization and transfer tasks. We also show that SEMPPL can be easily combined with other existing self-super methods and is promising direction to pre-train networks also in a fully supervised learning regime. Our analyses suggest that the role of pseudo-labels in selecting positives for semi-supervised contrastive methods might be underappreciated. Despite widespread use in cupi-supereieed applications, pseudo-labels are less understood and have been explored far less in the context of self-supervised methods. We hope this study, which shows empirically that prior work has under-utilizee pseudo-labele may help bridge that gap.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We documented the model and experimental evaluation in the main body of the paper and added further details in the appendix. Coneretely, we explain implementation details and sweep parame- ters in Appendix A and Appendix D.I, the invariance loss in Appendix B and give details data augmentations in Appendix E. The model pseudo-code is Appendix C. The datasets used in the experiments are freely available from their respective sources. We also open source SemPPL pre- trained checkpoints from our experiments, namely ResNet-50 1x, 2x and 4x as well as ResNet-200 2x together with the evaluation code at https: /ggthub.. com/ deepmind/ semppl.
REFERENCES
Roohallah Alizadehsani, Danial Sharifrazi, Navid Hoscini Izadi, Javad Hassannataj Joloudari, Afshin Shoeibi, Juan M Gorriz, Sadiq Hussain, Juan E Arco, Zahra Alizadeh Sani, Fahime Khozeimeh, et al. Uncertainty- aware seme-super method using large unlabeled and limited labeled covid-19 19 data, ACM Transactions on Multimedia Computing. Communications, and Applications (TOMM), 17(ss)1-24, 2021.
Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudo-labeling and con- firmation bias in deep semi-supervised learning. In Proc. of the International Joint Conference on Neural Networks (ICNN), 2020.
Yuki M. Asano, C. Rupprecht, and Andrea Vedaldi. Self-lahelling via simultancous clustering and representa- tion learning. In Proc. of International Conference on Learning Representations (ICLR), 2020.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning visual features by non-parametrically predicting view assign- ments with support samples. In Pruc. of Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Proc. of Advances in Neural Information Processing Systems (NeurPPS), 2019.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenhaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Proc. of Advances in Neural Information Processing Systems (NeurlPS). 2019.
Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, A. Srinivas, Tsung-Yr Lin, Jonathon Shlens, and Barret Zoph. Revisiting resmets: Improved training and scaling strategies. In Proc. of Advances Neural Information Processing Systems (NeurIPS), 2021.
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle I, Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
Maxim Berman, Hervé Jégou, Andrea Vedaldi, lasonas Kokkinos, and Matthijs Douze. Multigrain: a unified image embedding for classes and instances. arxiv preprint (00Xi::902..550000 2019.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mix- match: A holistic approach to semi-supervised learning. In Proc, of Advances in Neural Information Pro- cessing Systems (NewrIPS), 2019.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In Proc. of International Conference on Learning Representations (ICI.R), 2020.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proc. of the Anmual Conference on Computational Learning Theory, 1998.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 1oa-minmminiig discriminative components with random forests. In Proc. of the European conference on computer vision (ECCV), 2014.
Zhaowei Cai, Avinash Ravichandran, Subhransu Maji. Charless Fowlkes, Zhuowen Tu, and Stefano Soatto. Exponential moving average normalization for self-supervised and semi-supervised learning. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
10

---

Published as a conference paper at ICLR 2023
Mathilde Caron, Ishan Misra, Julien Mairal. Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégon, Julien Mairal, Piotr Bojanowski, and Armand Joulin, Emerging properties in self-supervised vision transformers. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proc. of International Conference on Machine Learning (ICML), 2020a.
Ting Chen, Simon Komblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-superviser learners. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang. Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning, arxiv preprint arXiv:2202.03000 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR). 2021.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arxiv preprint arXiv:2003.042205 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proc. of the International Conference Computer Vision (ICCV), 2021.
Veronika Cheplygina, Marleen de Bruijne, and Josien PW Pluim. Not-so-superrised: survey of semi- supervised, multi-instance, and transfer learning in medical image analysis. Medical image analysis, 54: 280-296, 2019.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing tex- tures in the wild. In Proc. Conference Computer Vision and Pattern Recognition (CVPR), 2014.
Ekin D Cubuk, Barret Zoph, Dandelion Mane. Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation policies from data. In Proc. of Conference Computer Vision and Puttern Recognition (CVPR), 2019.
Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data augmentation with reduced search space. In Proc. of Conference on Computer Vision and Pattern Recog- nition Workshop (CVPRW), 2020.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With lit- tle help from my friends: Nearest-neighbor contrastive learning of visual representations. Proc. of the International Conference on Computer Vision (ICCV), 2021.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In Proc. of International Conference on Machine Learning (ICML), 2021.
Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categorics. In Proc. of Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2004.
Weifeng Ge, Weilin Huang. Dengke Dong, and Matthew R. Scott. Deep metric learning with hicrarchical triplet loss. In Proc. of Eumpean Conference on Computer Vision (ECCV), 2018.
Chengyue Gong, Tongzheng Ren, Mao Ye, and Qiang Liu. Maxup: Lightweight adversarial training with data augmentation improves neural network training. In Proc. of Conference on Computer Vision and Patter Recognition (CVPR), 2021.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch. Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arxiv preprint arXiv: 0ee.l77666.221 02677, 2017.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vi- taliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski. Self-supervised pretraining of visual features in the wild. arxiv preprint arXiv:2103.01998 2021.

---

Published as a conference paper at ICLR 2023
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Proc. of Advances Neural Information Processing Systems (NeurIPS), 2004.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self- self-supervised learning. Proc. of Advances in Neural Information Processing Systems (Neur[PS), 2020.
Bo Han. Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Wai-Ilung Tsang, and Masashi Sugiyama. Co-teaching: Robust training deep neural networks with extremely noisy labels. In Proc. of Advances in Neural Information Processing Systems (NeurlPS), 2018.
Ben Harwood, B. V. Kumar, G. Cameiro, Ian D. Reid, and Tom Drummond. Smart mining for deep metric learning. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Pruc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast unsupervised visual representation learning. Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Kaiming Hc, Xinlci Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, arxiv preprint 1..6377 2021.
Tong He. Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proc. of Conference on Computer Vision and Pottern Recognition (CVPR). 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In Proc. of International Conference on Learning Representations (ICLR), 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of- distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8840 8349, 2021.
Zijian Hu, Zhengyu Yang. Xuefeng Hu, and Ram Nevatia. Simple: Similar pseudo label exploitation for semi- supervised classification. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
S. Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. of International Conference on Machine Learning (ICML), 2015.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 733):335-547, 2019.
Zhanghan Ke, Daoye Wang, Qiong Yan, Jimmy Ren, and Rynson W.H. Lau. Dual student: Breaking the limits of the teacher in semi-supervised learning. In Proc. of the International Conference Computer Vision (ICCV), 2019.
Mahmoud Khairy. Tpu vs gpu vs cerebras vs graphcore: fair comparison between ml hard- ware. https://kharry201 .mddium.com/ppu socduum..o//ppuu-eggnwwwwwweeeee 2020. Accessed: 2022-11-16.
sesp:p/ww.crrrrrsse-rrrpeeorreee hatrrmame.........oommmmaeeeeeee betwenn--
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive leamiig. Proc. of Advances in Neural Information Processing Systems (NeurIPS), 33:18661- -1666, 2020.
Youngdong Kim, Junho Yim, Juseung Yun. and Junmo Kim. Nlnl: Negative learning for noisy labels. In Proc. of the International Conference on Computer Vision (ICCV),201 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained catego- rization. In Proc of the International Conference Computer Vision Workshops, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Toronto, 2009.
12

---

Published as a conference paper at ICLR 2023
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Proc. of International Conference on Learning Representations (ICLR), 2017.
Dong-Hyun Lee et al. Pseudo-lahel: The simple and efficient semi-supervised learning method for deep neural networks. In Proc. of the ICMI. Workshop on Challenges in Representation Learning, 2013.
Kuang-Huei Lee, Anurag Arnab, Sengio Guadarrama, John Canny, and Ian Fischer. Compressive visual repre- sentations. Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2021.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Learning to learn from noisy labeled data. In Proc. of Conference on Computer Vision and Pottern Recognition (CVPR), 2019a.
Junnan Li, Caiming Xiong, and Steven C. H. Hoi. Comatch: Semi-supervised learning with contrastive graph regularization. In Proc. of the International Conference on Computer Vision (ICCV), 202
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven C. H. Hoi. Prototypical contrastive learning of unsupervised representations. In Proc. of International Conference on Learning Representations (ICLR), 2021b.
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proc. of Conference on Computer Vision and Puttern Recognition (CVPR), 2019b.
Sungbin Lim, Iddoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Proc. of Advances in Neural Information Processing Systems (NeurlPS), 32, 2019.
Sheng Liu, Jonathan Weed, Narges Razavian, and Carlos Femandez- Early-learning regulariza- tion prevents memorization of noisy labels. Proc. of Advances in Neural Information Processing Systems (NeurlPS), 2020.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- self-supervised learning: Generative or contrastive. JEEE Transactions on Knowledge and Data Engineering, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proc. of the International Conference on Computer Vision (ICCV), 202
Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhoffer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi- supervised learning. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual clas- sification of aircraft. arxiv preprint 106.51130..515,, 2013.
Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell. Representation learn- ing via invariant causal mechanisms. In Proc. of International Conference on Learning Representations (ICLR), 2021.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regular- ization method for supervised and superviie learning. Proc. of Transactions on Pattern Analysis and Machine Intelligence, 2018.
Maria- Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes.
In Proc. of Indian Conference on Computer Vision, Graphics & Image Processing, 2008
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arxiv preprint arXiv:1807.03777 03744,, 2018
Omkar Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. Proc, of Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proc. Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan Loddon Yuille. Deep co-training for semi- supervised image recognition. In Proc. of European Conference on Computer Vision (ECCV), 2018.
Benjamin Recht, Rehecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Proc. of International Conference on Machine Learning (ICML., 2019.
13

---

Published as a conference paper at ICLR 2023
Pierre 11 Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, Byol works even without batch statisties. In Proc. of Advances in Neural Information Processing Systems Workshops, 2020.
Ellen Riloff. Automatically generating extraction patterns from untagged text. In Proc. of the National Con-
ference on Artificial Intelligence, 1996.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh Rawat, and Mubarak Shah. In defense of pscudo- labeling: An uncertainty-ave aware pseudo-label selection framework for semi superviue learning, arxiv preprint 1xXi::2001006229 2021.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, An- drej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 115(3)211-252. 2015.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Proc. of Advances in Neural Information Processing Sys- tems (NeurIPS), 2016.
Florian Schroff,, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consis- tency and confidence. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Proc. of Advances Neural Information Processing Systems (NeurIPS), 2017.
Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell. Jovana Mitrovic. Pushing the limits of self--upervised resnets: Can we outperform supervised learning without labels on imagenet? arxiv preprint wrXiv2201.051//3 2022.
Jesper E Van Engelen and Holger H Hoos. A survey on semi sune-supe learning. Machine Learning, 109(2): 373-440, 2020.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for seni-supervised learning. In Proc. the International Joint Conference on Artificial Intelligence (ICCAI), 2019.
Xiang Wang, Xinlei Chen, Simon Shaolei Du, and Yuandong Tian. Towards demystifying representation learning with non-contrastive self-supervision. arxiv preprint arXiv:2 10.04947, 2021.
10.097,7
Xudong Wang. Zhirong Wu, Long Lian, and Stella X Yu. Debiased learning from naturally imbalanced pseudo- labels. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R. Scott. Multi-similarity loss with general pair weighting for deep metric learning. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR). 2019a.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 322-332--3333 330, 2019b.
Chen Wei, Kihyuk Sohn, Clayton Mellina. Alan Yuille, and Fan Yang. Crest: A class-iebalancing self-training framework for imbalanced supervise learning. Proc. of Conference on Computer Vision and Pattern Recognition (CVPR). 2021.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data augmentation for consistency training. In Proc. of Advances in Neural Information Processing Systems (NeurlPS), 2020a.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proc. of Conference Computer Vision and Pattern Recognition (CriittinnCCCR (CVPR), 2020b.
14

---

Published as a conference paper at ICLR 2023
Yi Xu, Lei Shang. Jinxing Ye. Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In Proc. of International Conference on Machine Learning (ICML)g (ICML), 2021.
Xiangli Yang. Zixing Song, Irwin King, and Zenglin survey on deep semi-supervised learning. arxiv preprint arXiv: 103:::21133055550 2021.
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. Proc. of Annual Meeting of the Association for Computational Linguistics (ACL), 1995.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training convolutional networks. arxiv preprint arXiv: 1e1./7000000000 03888, 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Sulf-supervised learning via redundancy reduction. In Proc. of International Conference on Machine Learning (ICML), 2021.
Shuangfei Zhai, Navdeep Jailly, Jason Ramapuram,
Busbridge, Tatiana Likhomanenko, Joseph Yitan Cheng, Walter Talbott, Chen Huang, Hanlin Goh, and Joshua Susskind. Position prediction as an effec- tive pretraining strategy. In Proc. of International Conference on Machine Learning (ICML), 2022.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi-supervised learning. In Proc. of the International Conference Computer Vision (ICCV), 2019.
Bowen Zhang, Yidong Wang. Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shi- nozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Proc. of Ad- vances in Neural Information Processing Systems (NeurIPS). 2021.
Zhilu Zhang and Mert Rory Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2018.
Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu. Simmatch: Semi-supervised learning with similarity matching. In Proc. of Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self training. In Proc, of Advances Neural Information Processing Systems (NeurIPS). 2020.
15

---

Published as a conference paper at ICLR 2023
A ADDITIONAL RESULTS AND IMPLEMENTATION DETAILS
A.1 DEME-SUPERVISED DETAILS AND RESULTS
Implementation details. In this work, we follow the protocol of Chen et al. [2020b]; Assran et al. [2021] for fine-tuning from the first layer of the projector and initialize both the encoder and the first layer of the projector with the parameters of the pretrained model. We add the randomly initialized classifier on top of the first layer of the projector (after the non-linearity). We train all the weights (pretrained and classifier weights) using either 1% or 10% of the ImageNet- 1 training data, and use the splits introduced in Chen et al. [2020a] and used in all the methods to compare to Grill et al. [2020]; Caron al. [2020]; Dwibedi et al. [2021]: Lee et al. [2021]; Mitrovic et al. [2021]; Tomasev et al. [2022]; Assran et al. [2021].
At training time we randomly crop the image, resize it 224 x 224, and then randomly apply a horizontal flip. At test time we resize images to 256 pixels along the shorter side with bicubic resampling and apply a 224 224 center crop to Both training and testing times we subtract from the color channels the average channel value and divide it by the standard deviation of the channel value (as computed on ImageNet-1k).
We use cross entropy loss and stochastic gradient descent with Nesterov momentum of 0.9 to fine-tune the model. For both 1% and 10% settings, we train for epochs and decay the initial learning rate by factor 0.2 at 18 and 24 epochs. Following the approach of Caron al. [2020], we pick different learning rates for the encoder (and the first projector layer) and for the classifier weights. We do not niques. We sweep over batch sizes values in (512, 1024, 2048]. encoder base learning rate val- ues in {0.005, 0.0035, 0.003, 0.0025, 0.002, 0.001}, and linear layer base learning rate values in (0.5, 0.3, 0.2, 0.1, 0.05, 0.025).
any weight decay other regularization tech-
Table 6: Top-1 and Top-5 accuracies (in %), after semi-supervised fine-tuning with a fraction of ImageNet labels, for ResNet-50 encoder across a number of representation learning methods.
Method
Top-1
Top-5
1%
10%
1%
10%
Supervised (Zhai et al., 2019]
25.4
56.4
48.4
80.4
Pseudo labels in classification: MPL [Pham et al., 2021]
73.9
Representation learning methods:
SimCLRv2 [Chen et al., 2020b]
57.9
68.4
SimCLRv2 + self distillation [Chen et al., 2020b]
60.0
70.5
CoMatch [Li et al., 2021a]
66.0
73.7
86.4
91.6
PAWS [Assran et al., 2021]
66.5
75.5
DebiasPL. [Wang et al., 2022]
67.1
85.8
SimMatch [Zheng et al., 2022]
67.2
74.4
87.1
91.6
SEMPPL (ours)
68.5
76.0
88.2
92.7
SimCI.Rv2 Selective Kernels [Li al., 2019b]
64.5
72.1
86.7
91.4
SEMPPI. (ours) + Selective Kernels
72.3
78.2
90.6
93.9
Additional results and larger networks. When the architecture of the ResNet-50 modified to include selective kernels [Li et al., 2019b], we see significant gains in performance at the expense selective kernels standard and follows rigorously Li et al. [2019b] for a total of 27.5 million weights instead of of 25.5 million for regular ResNet-50. Specifically, we use 2 channels, two convolution kernels of (3,3) and (5,5) with the latter imple- mented as a (3, 3) dilated convolution with rate 2, and 32 grouped convolutions. Unlike SimCLRv2 [Chen et al., 2020b]. we implement our group convolutions explicitly, and do not use the additional ResNet-D architectural modification from He al. [2019]. When using selective kernels our per- formance after finetuning with 1% of labels is the same as that of Sim2LRv2 after finetuning with 10% of labels.
of additional weights. Our implementation
16

---

Published as a conference paper at ICLR 2023
Additionally, in order to investigate the robustness and scalability of these results, further test the generality of SEMPPL by learning representations on larger (both deeper and wider) ResNet en- coders. Table testifies to SEMPPL outperforming the competing representation learning methods across all the architectures, both in the 1% and the 10% labelled settings. Also, as our flagship result reach 80.4% top-1 accuracy ResNet-50 1 x case this figure is comparable with the fully supervised accuracy attained by historical methods. 80.1% top-1 is defined in Grill et al. [2020] with standard RandAugment [Cubuk al., 2020] data augmentation. However it's certainly few percentage accuracy points away from results obtained with optimal current training protocols [Bello al., 2021]. We also note that SEMPPL is pre-trained for 300 epochs in all cases. This, rather than the 1000 epochs used as standard by most other representation learning methods, again compares with a typical figure of 200 epochs used in supervised learning. Overall this hints at SEMPPI. having achieved close an order of magnitude gain in label efficiency (compared to supervised learning) at a similar epochs budget.
ResNet-200 2x with 10% of ImageNet-1k labels. Just as in the
Our final networks were optimized using tranches of between 128 (for ResNet-50) and 512 (for the largest ResNets) Cloud TPUv3s all during 300 epochs each irrespective of size. This required around a day of computation time per run and tranche for ResNet-50 128 devices, time which scaled approximately linearly with the number of parameters on larger networks, depending on the actual network.
A.2 ROBUSTNESS AND OOD GENERALIZATION
We test the robustness and out-of-distribution (OOD) generalization abilities of representations learned via SEMPPL on several detasets. We use ImageNetV2 [Recht al., 2019] and ImageNet- C [Hendrycks & Dietterich, 2019] datasets to evaluate robustness and the datasets ObjectNet [Barbu al., 2019] and ImageNet-R [Hendrycks et al., 2021] to evaluate the OOD generalization.
The ImageNetV2 dataset [Recht et al., 2019] has three sets of 10000 images (matched frequency (MF), Threshold 0.7 (T-0.7) and Top Images (TI) that were collected to have similar distribution to the ImageNet test set. The ImageNet-C dataset [Hendrycks & Dietterich, 2019] consists of 15 synthetically generated corruptions of different severities (e.g. blur, noise) that are applied to the ImageNet validation set. The ImageNet-R dataset [Hendrycks et al., 2021] consists of 30000 different renditions (e-g- paintings, cartoons) of 200 ImageNet classes; the aim of this dataset is to test the generalization ability to different textures and other naturally occurring style changes that are outtof-distributtin to the ImageNet training data. The ObjectNet dataset [Barbu et al., 2019] has 18574 images from differing viewpoints and backgrounds compared the ImageNet training set.
On all datasets we evaluate the representations learned on a standard ResNet50 encoder under a linear evaluation protocol. We freeze the pretrained representations (no gradient updates) and train a linear classifier on top of the output of the ResNet-50 encoder using the full labelled ImageNet training set. We perform the test evaluation zero-shot, i.e the above datasets are not seen during the training of the representation or classifier.
We provide detailed breakdown across the different ImageNet-C corruptions in Table 7. Our proposed approach SEMPPL outperforms both the supervised baseline, on 12 out of 15 corruptions, as well as the competing semi-supervised representation learning model PAWS, on 12 out of 15 corruptions (notably, over all Blur, Weather and Digital corruptions).
Table 7: Top-1 accuracies (in %) for OOD generalisation on Gauss, Shot, Impulse, Blur, Weather, and Digital corruption types of ImageNet-C.
Blur
Weather
Digital Blastic
Method
Ganes
Shx
Impuise 20.5
Delocas
Glass
Motion
Zoom
Snow:
Frost
Fog
Bright 68.1
Contrasi
Pisel
JPBG 35.11
Supersisal 2019]
56.9
32.6
mperviodd aytreseniatvee::
PAWS
33.5
40.3
64.0
20.5
38.5
$5.1
SpMPPI
34.0
64.4
71.9
45.1
A.3 TRANSFER
To further evaluate the usefulness of the learned representations, we evaluate how well they transfer across datasets. For this, follow the standard evaluation protocol outlined in Grill et al. [2020];
17

---

Published as a conference paper at ICLR 2023
Chen al. [2020a]. We evaluate SEMPPL across the linear evaluation protocol which consists of freezing the encoder and only training a randomly initialized linear classifier on top of the encoder. In line with prior work [Chen et al., 2020a; Grill et al., 2020; Dwibedi et al., 2021], test SEMPPL representations on the following datasets: Food101 [Bossard et al., 2014], CIFAR 10 [Krizhevsky al., 2009], CIFART 100 [Krizhevsky al., 2009]. Birdsnap [Berg et al., 2014), SUN397 (split 1) [Xiao et al., 2010]. DTD (split 1) [Cimpoi et al., 2014], Cars [Krause ct al., 2013] Aireraft [Maji al., 2013], Pets [Parkhi al., 2012], Caltech101 [Fei-Fei et al., 2004], and Flowers [Nilsback & Zisserman, 2008], where we compare the downstream performance of SEMPPL to that of other reported semi-supervised methods on 10% of labels. Across these datasets there are differences in terms of metries used for selecting the best hyper-parameters as well the reporting of the final results. In line with prior work [Chen et al., 2020a; Grill et al., 2020; Dwibedi et al., 2021J, for Food101 [Bossard et al., 2014], CIFARIO [Krizhevsky et al., 2009], CIFAR 100 [Krizhevsky et al., 2009], Birdsnap [Berg et al., 2014]. SUN397 (split 1) [Xiao et al., 2010], DTD (split 1) [Cimpoi et al., 2014], and Cars [Krause al., 2013] we report the Top-1 accuracy Aireraft [Maji et al., 2013], Pets [Parkhi et al., 2012], Caltech101 [Fei-Fei et al., 2004], and Flow- [Nilsback & Zisserman, 2008] report the mean per-class aceuracy. For DTD and SUN397 only use the first split of the 10 provided splits in the dataset as per Chen et al. [2020a]; Grill et al. [2020]; Dwibedi et al. [2021].
the test set, and for
In these experiments, models are initially trained on the training sets of the individual datasets, and the validation sets are used select the best hyperparameters from the executed hyperparameter sweeps. Once the best hyperparameters have been selected, the final models are trained on merged dataset containing both the training and the validation split and evaluated on the held-out test split. The final results of the transfer experiments are reported in Table 3. The performed hyperparameter sweeps involved sweeping over the learning rates {.001, .01, 0.1, 0.2, 0.25, 0.3, 0.35, 0.4, 1., 2.). batch sizes {128, 256, 512, 1024, 2048), weight decay {1e-6. 1e-5, 1e-4, 1e-3, 0.01, 0.1). warmup epochs (0, 10), momentum (0.9, 0.99). Nesterov {True, False]. and the number of training epochs. For the linear transfer protocol considered setting epochs among {20, 30, 60, 80, 100). Models were trained by stochastic gradient descent with momentum.
18

---

Published as a conference paper at ICLR 2023
B INVARIANCE REGULARIZATION
We define the short-hands
=02,+)
p( (m: sm.t))
(5)
(0(2m 22m
mm, EEn (0 2m 20,
and
(0 _mitt)
201
p ; 2m
(6)
(
2 + Ee, (a 60( 2m 2
where (axx) is the set of negatives, randomly uniformly sampled from the current batch, kn
ge (x.()) is the target network projection of the negative sample; ( x2) = T cxp( (21 x2)/r)
the scoring function, T is scalar temperature, and (+, denotes the standard Euclidean dot
product.
We rewrite the components of the overall loss
LSEMPPL. LACOM + acCsmmPOs
as
B log 301 Fmmt
SCMPOS
Σ
+)
(8)
and
B Σ m=1
AUGM
log p( (2 km.))
(9)
As discussed in the main text we add the invariance penalty introduced in Mitrovic al. [2021] to further increase the similarity between the anchor and positives and regularize the learning process. We add this invariance penalty both augmentation positives and semantic positives. In particular, compute
Laugm = -D (p( (m),) || p(=02 sm.))
BBgBBBp pup :,,, log p( (2 s0t, - pptt i5 log p(3:; mm..)
and
Isempos =Dx 281, 202.t) || p( (202 =m.)))
=sg
E
log pl : 20a )]
p p go2d) log p( (0 2
p(= :
where sg denotes the stop- gro-igradient operation. Taking all this together, this gives the final form of the loss
CSEMPPL. C(Caua + aLsempos) X (Iaugm + Isempos
(10)
with À the invariance scale and is the contrastive scale. We use À = and e 0.3 in all our experiments irrespective encoder size or training time as our method robust to the choice of these hyperparameters.
19

---

Published as a conference paper at ICLR 2023
C PSEUDO-CODE OF SEMPPL
Listing provides PyTorch-like pseudo-code for SEMPPL, detaalin how we compute pseudo-labels and use them select the additional semantic positives, which are then used in the contrastive loss, along the augmentation positives.
k: The number of neighbors in K-NN when computing pscudolabels.
T.o: online network: Encoder comparison.nel.
target network: Encoder comparisen.net.
gamma: Target EMA coefficient.. Number of negativos.
m: Mask apply probability
for in range(num.aarge.vien)
10
queue.i queue init (queue.size, FIFO)
12
for x, in batch: # Load batch of B samples each with data and (maybe) label y. background for i in range num large_ views):
14
15
Stochastically apply background Bernoulli (p-m)
16
17
à Create an augmented large view.
augment(crop ((amen((rrop.larg((k)) (x)) ol_i f-o(xl.i)
19
11_i g-I(xl.i)
21
# Enqueue the labeled images in the batch. y not queue.i.enqueue ((Il.i. y-i))
22
24
for i mage((numssmal num.small_viewe
26
xs_1 = augment(crop (ro)-smallxx))
Small views only go through online os.i [_o(xs_i)
28
29
30
Pscudo-label computation for unlabolled examples. if is None: # Missing label. [kan(k, queue.i. ol.j) for i, all. pairs ) y mode(votes)
11
33
35
lass
36
Compute the loss between the pairs large views.
37
for i in range (num large. views):
38
for in range(num) (ange(mnllarge.eeeess += contrastive_lossclLL. tL. n.e) Augmentation positives. for in range wum..eemnticcpoottiie
40
Sample semantic positives
queue. add loss. 2, a.o)
42
sample (queue.j. j. (yl)
43
loss + contrastive.loss
44
45
Compute the loss between the small and large views.
for sango(num.mmlllvvieee...
47
for in range(nunn (nm...aaa...re.ieveee..
tl.j n_e) Augmentation positives. for in rango wtum.seaattic.psiieee
+s contrastive_losss
49
50
Sample semantic positives from the queue. add to the loss. (umple(queue... filler
52
loss += contrastive.loos
2, n.c)
54
lass J= ((num,larg
num swall-vieww.......
num large views (1 num semantic positives))
56
57
Compute the gradients. and update the online and target network. lass backward update(f.o)
59
8-1
gammaa g-1
gammu)
Listing Pseudo-code for SEMPPL.
20

---

Published as a conference paper at ICLR 2023
D ANALYSIS
D.1 IMPLEMENTATION DETAILS
We perform all the ablation experiments using 10% of labelled data and train a standard ResNet- 50 encoder with SEMPPL. for 100 epochs (except in the training duration ablation). We report the top-1 accuracies on the ImageNet
after fine-tuning from the first layer of the projector. As in Grill et al. [2020] and for the main results in this paper, we multi-layer perceptrons for the projector and predictor with linear layers-the first one followed by batch normalization [loffe & Szegedy, 2015] and rectified linear activation with output sizes 4096 and 256 for the two layers respectively. We use the same augmentations as for the experiments in the main paper-tthe the stan- dard SIMCLR augmentations [Chen et al., 2020a] and the
and saliency- based masking (TTomasev et al., 2022]. Following the hyperparameter settings of the main results, we use
batch size: B 4096
queue capacity C 20B (unless specifically ablated)
number of nearest neighbours k 1 (unless specifically ablated)
view voting is used (unless specifically ablated)
weight decay: le 6 (exclude biases and batch normalization parameters)
optimizer: LARS (exclude biases and batch normalization parameters from adaptation)
base learning rate: 0.3 (scaled linearly with batch size [Goyal et al., 2017])
warm-up: 10 epochs
cosine decay schedule for learning rate
exponential moving average parameter: 0.996
views: 4 large views size 224 x 224 and 2 small views of size 96 x 96
temperature: T = 0.2
number of semantic positives: (unless specifically ablated)
10 randomly subsampled negatives per anchor
1/5 (unless specifically ablated), A 5 and c 0.3.
21

---

Published as a conference paper at ICLR 2023
D.2 ADDITIONAL ANALYSES
Number of semantic positives We study the effect of varying the number of semantic positives in SEMPPL. Table 8a shows that increasing this number from to only has an effect on the amount of correctly predicted pseudo-labels, but no effect downstream performance. On the other hand, using or 10 semantic positives significantly improves performance and also yields much more accurate pseudo-labels prediction.
Training duration Next, we vary the length of training representations and examine downstream performance. can be seen from Table 8b, both the proportion of correctly predicted pseudo- labels and downstream performance improve with longer training up to 300 epochs but decrease if we continue training up to 500 epochs. This indicates with training longer than 300 epochs SEMPPL. is starting to overfit, an observation consistent with phenomena reported elsewhere in the literature involving noisy labels [Li et al., 2019;; Kim et al., 2019; Liu et al., 2020; Han et al., 2018; Zhang & Sabuncu, 2018; Wang et al., 2019b].
Table 8: Top-1 accuracy (in %) on the ImageNet- Ik test set, and accuracy (in %) of correctly pre- dicted pscudo-labels at the end of training for semantic positives and training length experiments.
Num. positives
Top-1
Pseudo-label
69.9
68.6
Training time (epochs)
Top-1
Pscudo-labes acc.
2
69.9
70.9
100
69.9
69.2
3
69.9
69
200
72.4
76.8
71.0
72.8
300
72.7
77.9
10
70.7
72.9
500
72.3
75.6
(a) Varying the number of semantic positives.
(b) Varying the length of training.
View voting SEMPPL generates multiple views from single instance image in order to learn rep- resentations. Those different views can be leveraged towards better pseudo-labels prediction. Rather than only picking one randomly selected data view compute single pseudo-label, we perform majority voling over (noisy) pseudo-labels computed from all available image views. Specifically, we compare the online predictor embedding of one view with the queue of the target projector em- beddings of the same data view from previous batches in the first setting; in the second setting we compare the online predictor embedding of each view with the queue of the target projector embed- dings of each other data view from previous batches.
Since SEMPPL relies on large views, this yields up to 16 different pairs of views to compare and compute pseudo-labels from, i.e. we get 16 pseudo-label predictions; this setting we call view voting. Table shows that using all available views to compute pseudo-labels significantly increases pseudo-labels accuracy which in turn significantly improves downstream performance.
Table 9: Top-1 accuracy (in %) on the ImageNet-Il Ik test set, and accuracy (in %) of correctly pre- dicted pseudo-labels at the end of training for view voting experiments (using all views to compute pseudo-labels using just single view).
View voting
Top-1
Pseudo-label acc.
On
69.9
68.6
Off
69.0
62.5
Number of nearest neighbours In order to compute pseudo-labels we use k-nearest neighbour lookup on the queue. While in the main results Section 3 we consistently assume = here we ablate the effect of varying & on downstream performance. As can be seen Table 10a, increasing actually leads to a small decrease in performance. This is attributable to the decrease in the proportion of correctly predicted pseudo-labels as k increases.
22

---

Published as a conference paper at ICLR 2023
Queue length How long should the queue of target projector embeddings for computing pseudo- labels be? As the queue grows in size, contains increasingly stale embeddings and threatens to hamper the accuracy of predicted pscudo-labels. On the other hand, increasing queue size increases the amount and diversity of labelled embeddings available which we would expect to be beneficial. the accuracy with which
Those opposing sioces-stalenesss and increasing coverage-gooee can correctly predict pseudo-labels in turn directly affecting the selection of semantic positives and their quality. We resolve this ambiguity empirically. As seen in Table 10b, increasing the coverage and diversity of labelled embeddings has a strong positive effect on representation learning and downstream performance. Staleness of embeddings is far less of a problem at practical (i.e., not very large) queue sizes, showing diversity and coverage to be the dominant factor.
Table 10: Top-1 accuracy (in %) on the ImageNet-1k test set, and accuracy (in %) of conrectly predicted pseudo-labels at the end training for k--N and queue length experiments.
Queue size (C)
Top-1
Pscudo-label
4000
67.9
53.1
k-nn
Top-1
Pseudo-label acc.
8000
68.6
60.4
70.0
70.5
12000
68.8
62,0
2
69.9
69.5
20000
69.2
64.3
3
69.8
70.0
10000
69.6
66.9
69.8
70.0
80000
69.9
68.8
68.0
200000
69.8
69.5
(a) Varying the number of nearest neighbours.
(b) Varying queue size.
The effect of the loss weight We in Equation 4 weight the contribution of the loss coming from semantic positives against the loss coming from augmentation positives. In SEMPPL multiple semantic positives for learning and thus need to adjust appropriately weigh the contribution of the individual semantic positives. In our main experiments, we use 3 semantic positives and for this reason is not equal to
In Table 11 we vary from 0.0 to 1.0, with = 0.0 effectively recovering RELICv2. The results indicate that the SEMPPL loss notably improves over RELICv2, but that the exact choice of should be treated as hyperparameter. Note that the value 0.2 is very close to the 3 which is exactly 1/number of semantic positives. Thus, the optimal choice of is very close that fraction.
Table 11: Top-1 accuracy (in %) on the ImageNet-Ie test set for varied values the loss weight a
Loss weight (a)
Top-1
0.0 (REL.CCV)
72.4
76.0 0.2
1.0
74.6
23

---

Published as a conference paper at ICLR 2023
E AUGMENTATIONS
In this work, we follow the established data augmentations protocols and pipelines of Chen et al. [2020a]; Grill al. [2020]; Caron al. [2020]; Mitrovic al. [2021]; Chen al. [2020b]. Specifi- cally, SEMPPL uses a set of augmentations to generate different views of the original image which has three channels, red r, green and blue with r, g, E [0, 1].
The augmentations we use are generated by applying the following sequence of operations in the following order
1. Crop the image: Randomly select patch of the image, between minimum and maximum crop area of the image, with aspect ratio sampled log-uniformly in [3/4, 4/3). Upscale the patch, via bicubic interpolation, to square image of size 8.
2. Flip the image horizontally.
3. Colour jitter: randomly adjust brightness, contrast, saturation and hue of the image, in random order, uniformly by value in [-a, a) where is the maximum adjustment (speci- fied below).
4. Grayscale the image, such that the channels are combined into one channel with value 0.2989r 0.5870g + 0.1140b.
5. Randomly blur. Apply a 23 x 23 Gaussian kernel with standard deviation sampled uni- formly in [0.1,
6. Randomly solarize: threshold each channel value such that all values less than 0.5 are replaced by 0 and all values above equal to 0.5 replaced with 1.
Apart from the initial step of image cropping, each subsequent step is applied with a certain proba- bility generate the augmented view of the original image. These probabilities and other augmen- tation parameters are given in Table 12. SEMPPL uses 4 large views of size 224 x 224 pixels and 2 small views of 96 96 pixels; to get the first and third large views and the first small view we use the parameters listed below for odd views, while for the second and fourth large view and the second small view we use the parameters for even views. Note that these are the same augmentations used also in Chen et al. [2020a]; Grill et al. [2020]; Caron al. [2020]: Mitrovic et al. [2021]; Chen et al. [2020b].
In addition to these augmentations, also randomly apply the saliency masking augmentation proposed in Tomasev et al. [2022] which enables us to remove large part of the background. We follow the protocol described in Tomasev et al. [2022] for computing the saliency masking for an image and we apply this augmentation with probability 0.1 to the large views. In keeping with Tomasev et al. [2022], we fill out the removed background of the image with homogeneous grayscale noise with the grayscale level randomly sampled for each view. We only apply the saliency masking when the remaining foreground covers at least 20% of the total image.
24

---

Published as a conference paper at ICLR 2023
Parameter
Even views
Odd views
Probability of randomly cropping
50%
50%
Probability of horizontal flip
50%
50%
Probability of colour jittering
80%
80%
Probability of grayscaling Probability of blurring
20%
20%
100%
10%
Probability of solarization
0%
20%
Maximum adjustment a of brightness
0.4
0.4
Maximum adjustment of contrast
0).4
0.4
Maximum adjustment a of saturation
0.2
0.2
Maximum adjustment a of hue
0.1
0.1
Crop size
224
96 (small), 224 (large)
Crop minimum area
8%
5% (small), 14% (large)
Crop maximum area
100%
14% (small), 100% (large)
Table 12: Parameters of data augmentation scheme. Small/large indicates small or large crop.
F COMPUTATIONAL COST OF SEMPPL
Added cost of SEMPPL As SEMPPL is based on RELICv2, the computational overhead of SEMPPL over RELICv2 comes from three factors: i) the queue maintenance, ii) the k-NN exe- cution, and iii) the computation of the added loss term. We carefully analyzed this overhead, and concluded the following:
The k-NN and the queue maintenance take 5.6% of the step time, with the k-NN itself taking 4.5%.
The total loss (both forward and backward passes of loss terms coming from both aug- mentation and semantic positives) takes 3.9% of the step time, with 2.9% belonging to the additional loss coming from semantic positives.
In total, SEMPPL takes 8.5% of the total step time.
Note that in our work we use naive implementation of the k-NN. If needed, the k-NN can be further sped up, at fraction of the accuracy, with a fast approximate k-NN model such as FAISS [Johnson al., 2019]. This would further reduce the computation cost of using the k-NN to predict pseudo- labels and select semantic positives. Additionally, since our code is not heavily optimised, there is room for additionally lowering these numbers with targeted optimisations.
Comparing SEMPPI. to related models The direct comparison of training times of SEMPPI. to related models can be deceiving, since there are three different sources of difference that influ- ence the comparison. These summarise to differences in: i) accelerator architectures used, ii) deep learning frameworks used, and iii) concrete code, which can be assumed, is not optimised for any of the models, given their research nature. Being aware of these sources of difference, contrast the running times pre- pre-training for SEMPPL and related models:
PAWS [Assran et al., 2021] report 8.5 hours per 100 epochs on 64 V100s. Assuming continuity in speed, this is equivalent to 25.5 hours for 300 epochs.
SimMatch [Zheng et al., 2022] report 2.34 hours per epoch on 8 V100s. If we (generously) assume perfect scaling (which is difficult in reality) to 64 V100s, this is equivalent to 87.75 hrs for 300 epochs.
SEMPPL trains in 13 hours on 64 TPUv3 cores.
Given empirical evidence that TPUv3 is roughly 23% faster than V100 on ResNet-50 train- ing [Khairy, 2020], one could infer that the implementation of our model is significantly more computationally efficient. However, given that we cannot control all the aforementioned differ- ences mentioned above, more careful analysis is needed to establish the right cost for each of these methods.
25


Reviews:
[{"review_id": "97fuj9bjCQ", "reviewer": "Reviewer_U91Z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an elegant way of combining pseudo labels (semantic similarity) and instance similarity. This method achieves state of the art performance on 1% and 10% ImageNet settings using ResNet-50 architecture. Transfer learning and OOD generalization experiments show the usefulness of this approach.  ", "summary_of_the_review": "The paper provides an elegant solution to the idea introduced by SimMatch (Zheng et. al. 2022). It is well written and there is extensive evaluation done w.r.t. semi-supervised learning approaches in the literature, along with transfer learning and OOD generalization benchmarks. The novelty of the idea is not large. But it is an interesting contribution to the literature.", "strength_and_weaknesses": "**Strengths**  \n_S1_. The paper provides an elegant solution to combine pseudo labels (semantic similarity) and instance similarity while learning visual representations. This approach is more elegant than SimMatch (Zheng et al, 2022), beating their performance with lesser number of training epochs.   \n_S2_. It achieves state of the art on 1% and 10% ImageNet settings using ResNet-50 architecture.  \n_S3_. The paper beats the transfer learning performance w.r.t. SimMatch (Zheng et al, 2022).\n\n\n**Weaknesses**  \nAs such the paper is well written with a good set of ablation studies. But certain things aren't clear.  \n_W1_. The paper says that is uses the Relic objective (Mitrovich et. al., 2021). It is not clear how important this objective would be in terms of the performance gain. This objective could be used for the competing approaches as well.  \n_W2_. It would be good to have the precision and recall plots of the pseudo label quality (Fig. 2) for the competing approaches. That would give good insight into whether there is a correlation with this elegant way of optimizing to the quality of pseudo labels generated. It would also justify the point the authors are marking about the virtuous cycle of learning better representations while improving quality of pseudo labels. Currently, that point has not been justified empirically. We know the approach helps learning better representations. \n_W3_. It would be good to also show how this approach does on limited annotation settings introduced in PAWS (Assran et. al., 2021) and Masked Siamese Networks (Assran et. al., ECCV 2022) with 1-10 annotations per class. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7TAww9da_52", "reviewer": "Reviewer_iXGB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends contrastive learning to semi-supervised settings. To do so, it estimates pseudo-labels for the unlabeled data during the training process of contrastive learning and adds a supervised contrastive loss according to the labels of the labeled data as well as the pseudo-labels of the unlabelled data to the original contrastive loss. Experiments demonstrate that it achieves a new state-of-the-art performance in semi-supervised benchmarks and has better robustness and generalizability. ", "summary_of_the_review": "This paper addresses an interesting topic, i.e., how to utilize the supervised information in the semi-supervised setting to help contrastive learning to learn a better embedding space. However, as discussed above, the proposed method lacks novelty and some experiments are missing. Therefore, I feel this paper is below the acceptance threshold.", "strength_and_weaknesses": "Pros:\n\n- Interesting topic. It is interesting and important to utilize the supervised information in semi-supervised setting rather than a simple cross-entropy objective. This paper proposes to use that information to help decide the similarity relationships in the embedding space for contrastive learning.\n- Good experiments. The proposed method outperforms state-of-the-art semi-supervised learning methods on ImageNet with 1% and 10% labels. The analytical and supplemental experiments are also comprehensive.\n\nCons:\n\n- The proposed method is naive. It just uses the nearest labeled datapoint as the pseudo-label of each unlabeled datapoint and then adds a supervised contrastive loss using the labels and the pseudo-labels directly. Moreover, the description is confusing. For example, what’s the purpose of first retrieving the k-nearest neighbors and then finding the nearest one from the k datapoints (Equation 2)? In 2.1 Algorithm parameters, what does it mean by |a| = 4 (what is a?) and why are there 16 pseudo-labels for each unlabeled datapoint?\n\n- The diversity of the datasets used is too low. It compared with previous methods for semi-supervised learning only on ImageNet, which makes the empirical evaluation less convincing. Most of the baselines included in the tables of this paper (e.g., DebiasPL [Wang et al., 2022], SimMatch [Zheng et al., 2022], etc) present the results on other datasets (e.g., CIFAR) in addition to ImageNet. This is no reason to exclude those results and compare the results on only one dataset.\n\n- This is no computation cost analysis in the paper. Since the proposed method has much more steps than the traditional contrastive learning framework, it would be helpful to evaluate how much computation overhead it brings. Comparison to other semi-supervised methods (e.g., as SimMatch [Zheng et al., 2022] does) is also necessary.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5b7UICEDG6", "reviewer": "Reviewer_JoKU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to integrate the idea of pseudo-labeling to the contrastive self-supervised learning paradigm. To be specific, the method uses a subset of labeled data to pseudo-label the unlabeled data point through $k$-NN. The pseudo-labeled data can therefore be used as a semantically positive sample and joint optimized using the contrastive learning objective. The paradigm achieves outperforms previous methods by a noticeable margin on ImageNet with 1% or 10% data. It also shows strong performance on robustness, out-of-distribution and transfer learning settings.", "summary_of_the_review": "Excellent performance and well-written technical contribution.", "strength_and_weaknesses": "### Pros: \n\n+ The construction of semantic positives introduces additional class-invariant objective in addition to augmentation-invariant objective as in the instance-level contrastive training.\n\n+ New state-of-the-art performance on semi-supervised learning with 1% and 10% ImageNet dataset.\n\n### Cons:\n\n- In SemPPL, the pseudo-labeling stage relies on a $k$-NN to retrieve semantically close labeled samples. What is the speed of $k$-NN compared with the rest of the training?\n\n- What's the effect of loss weight$\\alpha$? I am particularly interested in the case of $\\alpha=0$, meaning that you drop the augmentation-invariant self-supervised objective and $\\alpha=1$, meaning that you are reproducing a baseline from pure self-supervised representation (since I notice the augmentations used in SemPPL are stronger than the original SimCLR so the baseline might be higher than the reported number as well). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4bvlESBo280", "reviewer": "Reviewer_vzyU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "There are two successful directions in DL nowadays to build models based on both labeled and large amount of unlabeled data: pseudo-labeling (semi-supervised) and SSL (self-supervised learning) with further fine-tuning. There are few papers which try to properly combine both SSL and supervised loss right from the beginning of training showing that maybe we should use labeled data earlier to ground ourselves or guide the training. Current paper is extending SSL with positive samples mining: we use supervised data to generate pseudo-labels based on KNN in representational space to find closest labeled data and use their label as pseudo-label. The pair is positive if their labels/pseudo-labels are the same. In this way SSL is grounded by some extra knowledge from labeled data. Also pseudo-labeling is used in non-standard way as we don't compute cross entropy on pseudo-labels but we use them to decide what is positive sample. Authors probe this idea with resnet model and 1/10% of labeled data from ImageNet. They achieve SOTA results with for resnet architecture. In extra ablations authors showed that the method works for large resnet-based models, for out-of-distribution generalization and transfer learning.", "summary_of_the_review": "I think the main strong contribution of the paper is demonstration that self-supervised learning (SSL) (via successful and popular contrastive learning) can be significantly improved if supervised data are used not only at finetuning phase but also at pre-training phase, which is more aligned in the way how babies are learning with additional (weak) supervision and active learning. I hope this paper could push research community in the direction of SSL and semi-supervised learning synergy but not as two-phase training but more natural way, e.g. how this paper proposes.\n\nOverall the idea is presented in a clear way and with extensive ablations. There are several concerns on the clear presentation and discussion of the queue component in the method as well as some extra ablations to strengthen the point \"we should use labeled data as earlier as possible and integrate/ground SSL with them\".\n\n\nUpdate: Based on the revision and additional experiments and analysis I am changing my score from 6 to 8.", "strength_and_weaknesses": "**Strength**\n- I like a new idea proposed in the paper in terms of how to perform positive mining and use some pseudo-labeling for this.\n- Clear presentation and extensive ablations of the parameters like queue size, KNN size, OOD and domain transfer, large architecture, etc.\n- SOTA results for 1/10% of supervision\n- Analysis of pseudo-labels quality and how it changes over the training\n\n\n**Weaknesses**\nPart of weaknesses is just clarification into the main text which should be done to resolve some ambiguities. Others are some extra experiments to better understand the contribution on top of simCLR and strongly demonstrate that adding labeled data from the beginning helps SSL loss.\n- there are no ablations on how penalty influences training and what is its contribution in vanilla simCLR v2\n- Any thoughts / experiments if method is applicable to ViT? And how it is comparable with recent work of pseudo-labeling https://openreview.net/forum?id=7a2IgJ7V4W&referrer=%5BReviewer%20Console%5D(%2Fgroup%3Fid%3DNeurIPS.cc%2F2022%2FConference%2FReviewers%23assigned-papers)?\n- It is not clear from the text if we do fine-tuning on labeled data of the full network or just linear probing. Thus not clear if the baselines are in the same setting as proposed method. Also not clear if simCLR v2 benchmark is done also with EMA for target branch of the SSL as it serves as the main purely comparable baseline having the same structure of the SSL except additional positive mining.\n- I think it is worth to mention some limitations of the proposed method as overall discussion about pseudo-labeling could be misleading. Pseudo-labeling is applicable for **any** data type, including sequential like speech recognition and machine translation. Proposed method is not directly applicable to speech and text domain as there we operate with sequence level labels without segmentation. \n- why did authors decide to use EMA? SimCLR v2 worked without it too. I wonder what is the impact of EMA overall on the method. It should work without it as authors maintain the queue of labeled data representations and there could be the same sample but with representations obtained from different model snapshots - kind of ensembling which performs same as EMA.\n- there could be different ways (4 cases) to compute embedding for labeled / unlabeled data and then perform KNN. Why do authors select the way described in formula 2? any ablations or at least motivation (better to add it into the text)?\n- Table 3, 8: I like results, this is great! But what are the results for simCLR here? Could it be that the method performs better because it is using SSL loss and not PAWS / other semi-sup. loss?\n- Table 5: Why do we not get results of simCLR v2 from Table 1 for the last row in Table? What is the difference then as we removed both pseudo-labels and semantic positives? Also not clear if simCLR v2 baseline is also done with multi-crop strategy \n- There is no any discussion on the proportion of labeled / unlabeled data in the main text and in the experiments? How do we balance data?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2tC0p6piL-P", "reviewer": "Reviewer_15Rd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper adapts the contrastive learning from self-supervised learning to semi-supervised learning with two core designs: 1) performing pseudo-labeling based on the similarities to the labeled data in the encoded feature space; 2) select positives for a datapoint based on the pseudo-labels.\n\nPros: The idea is simple and extensive experiments demonstrate its effectiveness.\nCons: The novelty of the proposed method is relatively limited. \n", "summary_of_the_review": "While the idea is simple yet effective and the experiments are extensive, the novelty is limited, not adequate for ICLR.", "strength_and_weaknesses": "1) The idea is straightforward and sound.\n2) The experiments are sufficient to demonstrate its effectiveness. \n3) paper is written well and easy to follow.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster