Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training
Authors: Utku Ozbulak
Hyun Jung Lee
Beril Boga
Esla Timothy Anzaku
Ho-min Park
Arnout Van Messem
Wesley De Neve
Joris Vankerschaver
Venue: Accepted by TMLR ()

Abstract: Although supervised learning has been highly successful in improving the state-of-the-art in the domain of image-based computer vision in the past, the margin of improvement has diminished significantly in recent years, indicating that a plateau is in sight. Meanwhile, the use of self-supervised learning (SSL) for the purpose of natural language processing (NLP) has seen tremendous successes during the past couple of years, with this new learning paradigm yielding powerful language models. Inspired by the excellent results obtained in the field of NLP, self-supervised methods that rely on clustering, contrastive learning, distillation, and information-maximization, which all fall under the banner of discriminative SSL, have experienced a swift uptake in the area of computer vision. Shortly afterwards, generative SSL frameworks that are mostly based on masked image modeling, complemented and surpassed the results obtained with discriminative SSL. Consequently, within a span of three years, over $100$ unique general-purpose frameworks for generative and discriminative SSL, with a focus on imaging, were proposed. In this survey, we review a plethora of research efforts conducted on image-oriented SSL, providing a historic view and paying attention to best practices as well as useful software packages. While doing so, we discuss pretext tasks for image-based SSL, as well as techniques that are commonly used in image-based SSL. Lastly, to aid researchers who aim at contributing to image-focused SSL, we outline a number of promising research directions.

Published in Transactions on Machine Learning Research (05/2023)
Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training
Utku Ozbulak1,2
utha.. ozbulak @ggentt wew.eettt.......... oc.kr hyunjung. lee @ghent. ac.kr beril. bogo @bshg com eslatimothy, anzaku Gigeett ac.kr homin. park k oighnnt kr arnorl. varmessem @uliege. be wesley, deneve Gghent, oc. kr jorís. www..ooonnkechhaaeeeoeee Gghent. ac..
Hyun Jung Lee
Beril Boga"
Esla Timothy Anzaku
Homin Park
Arnout Van Messem
Wesley De Neve
Joris V
Chent Universily, Belgium
Chent Universily Global Campus, South Korea
BSH Hausgeräte GmbH, Germany
University of Liège, Belgium
Reviewed on OpenReview: https: opennevwe,,,reeeeuu net/ formm id Mo2554lud) luddd
Abstract
Although supervised learning has been highly successful in improving the state-of-the-art the domain of image-based computer vision in the past, the margin of improvement has diminished significantly in recent years, indicating that plateau in sight. Meanwhile, the use of self-supervised learning (SSL) for the purpose of natural language processing (NLP) has seen tremendous successes during the past couple of years, with this new learning paradigm yielding powerful language models. Inspired by the excellent results obtained in the field of NLP, self-supervised methods that rely on clustering, contrastive learning, distillation, and information maaatiionmmiiiizaaiiin which all fall under the banner of discriminative SST., have experienced a swift. uptake the area of computer vision. Shortly afterwards, generative SSL frameworks that are mostly basod on masked image modeling, complemented and surpassed the results obtained with discriminative SSL. Consequently, within span of three years, over 100 unique general-purpee SSL, with focus on imaging, were proposed. In this survey, we review plcthora of rescarch efforts conducted on image-oriented SSL, providing historic view and paying attention to best practices as well as useful soft packages. While doing so, we discuss pretext tasks for image- based SSL, as well as techniques that are commonly used in image- based SSL. Lastly, to aid researchers who aim at contributing to image-focused SSL, we outline number of promising research directions.
frameworks for generative and discriminative
1 Introduction
The remarkable feature extraction capabilities of deep neural networks (DNNs) have enabled their effective utilization in numerous visual tasks. Although the core building blocks that are in common today were already proposed two docades ago (LeCun ct al. 1998), DNNs only became the go-to models after the introduction of AlexNet. (Krizhevsky el al. 2012), a DNN architecture that was able to obtain exceptional results for the ImageNet Large Scale Visual Recognition Challenge (Russakovsky et al. 2015)) that took place in 2012, by leveraging vast amounts of computational resources (at that time) and large amounts of laboled data. Since then, the availability of standardized datasets in the image domain such MNIST (Leunn
et al. 1998), CIFAR (Krizhevsky & Hinton 2009) SVHN (Netzer et al. 2011), COCO
(Lin et al. 2014)

---

Published in Transactions on Machine Learning Research (05/2023)
and ImageNet enabled standardized experimentation, with these datasets acting catalysts for major advancements in the area of supervised learning. Starting with Alex the classification accuracy of DNNs on ImageNet improved year after year thanks to better and novel architectural designs (e.g., VGG (Simonyan
& Zissorman 2015 ResNet (He ct al.) 2016), InceptionNet (Szegody et al.) 2015: 2016), ViT (Dosovitskiy al. 20200), augmentation Lechniques, optimizers, and activation functions, as well as methods for smoother
jet
training (Loshchilov & Hutter 2017: Yun et al. 2019 Toffe & Szegody 2015; Kingma & Ba 2014 Clevert
et al. 2015).
Unfortunately, not all datasets come with an abundance of labeled training data. In order to overcome this hurdle and to facilitate the application of DNNs to smaller datasets, transfer lcarning was introduced and soon became the dominant method to transfer knowledge across image datasets (Tnn et al. 2015). Although transfer learning enables the usage of DNNs for smaller datasets thanks to features extracted from larger datascts, models trained in this way are known to be brittle and sensitive to small changes in the data (Jain) ct al. 2022) due to the use of supervised pre-training. Furthermore, shortcomings of supervised learning also became apparent when improvements obtained with these methods came to halt in recent years (see Figure I for top-1 accuracy on ImageNet), thus calling for research efforts that go beyond the use of supervised learning Zisserman 2018). In order to overcome the limitations of supervised lcarning, countless studies investigated the line of unsupervised learning, which aims at enabling robust feature extraction through the training of models without label information (Celebi & Aydin 2016). U Unfortunately, results obtained by these methods on image datasets fell short until recently (Noroozi & Favaro 2016 Pathak et al. 20166. while
the use of celf-supervised methods in the field of natural language processing (NLP) achieved the--ar
results, compared to supervised learning techniques (Devlin et al. 2018 Radford et al. 2019).
As mentioned above, the field of NLP enjoyed the success of self-supervised models over supervised ones ear- lier than the field of computer vision, with models such as BERT, GPT, and their variants achieving state-of-the art results (Devlin et al.
2018 Radford et al. 2019 Brown et al. 2020). One reason which explains the suc- cess of SSL in NLP is the abundance of unlabeled text data, such as books, online websites, and blogs (Chen) et al. 2017 Hamilton et al. 2017), which prompted researchers to investigate SSL over supervised training. Another reason that explains their success, as discussed by [e et al.] (2020), is the fundamental difference between the signal space of NLP and the signal space of computer vision, given that language data are discrete and structured (i.e., words), whercas image data are high dimensional, continuous, and unstruc- tured. Nevertheless, we can state that the success of SST in the field of NT.P prompted the computer vision community to put more investigative efforts into this learning paradigm.
In order to alleviate issues regarding label requirements, as well as to enable robust feature extraction, self- supervised learning in computer vision emerged as a method for extracting robust features from unlabeled data using the properties of images themselves (He al. 2020 Chen et al., 2020b). The idea behind SSL is straightforward: devise an experimental setting in which the task that provides the supervisory signal can be solved without human annotation and then train DNNs to solve it.
Note that the description provided above for SSL also covers number of additional approaches including autoencoders |Gogna & Majundar] 2016), generative models, and clustering- based methods that leverage self-labeling (Caron et al., (2018), and that these approaches also fall into the category of unsupervised learning (since human annotation is not necessary). Furthermore, most of the training routines described in this manuscript also use the term "self=supervised learning" interchangeably with "representation learning" when supervision is provided by the data, while representation learning is described by [engii et al. (2013) as "learning representations of the data that make it easier to extract useful information when building classifiers or other predictors", irrespective of the supervisory nature of the learning methodology. So, how did "self-supervision" become such a popular term in recent years?
Resurgence of the term "self-supervised learning" in computer vision Beyond a number of niche use cases such as image colorization (Larsson et al., 2017), , image inpainting (Yang et al. 2017), and puzzle- solvers (Trnn et al., 2019) that explicitly use self-supervision, the term "self-supervised learning" was previ- ously not employed to describe many techniques. Furthermore, compared to other learning paradigms, the use of SSL was not popular until recently (see Figure] D. In fact, research elforts that are now considered to be pioneers in SSL and that are used for SSL benchmarking, such as Deep Cluster (Caron et al. 2018).
2

---

Published in Transactions on Machine Learning Research (05/2023)
Accuracy
Search interest
parstrad learing
Saper Spervied awriing
pesaperbii loaring
Top-1
53
2012
2013
2014
2015
2016
2217
2010
2010
2020
2021
2022
2092
2012
2014
2016
2018
2022
2002
Year
Year
(a) ImageNett top-1 accuracy
(b) Interest over time for different learning paradigms
Figure 1: (a) ImageNet top-1 accuracy for DNNs proposed between 2012 2022 and (b) interest over time for three popular learning paradigms between 2004 2022, as measured with Google Trends.
InstDist (Wu et al. 2018b), CPC (Oord et al. 2018), and Local Aggregation (Zhuang et al. 2019 were published as unsupervised training methods, distancing themselves from SSL.
The resurgence of interest in self-supervision and the re-branding of corresponding methodologies can be attributed to the popularization of the term by both authoritative researchers and tech giants in the field
between 2018 and 2020 (Zisserman] 2018: Efros 2019: Bachman 2019 LeCun Misra, 2020 Chen 2020; Howard 2020). The reason for this ro- rranding straight most of the tasks discussed above that fell under the banner of unsupervised learning were deemed misleading, since the training was not completely unsupervised. Instead, the supervision
provided by the data itself, without explicit human labeling (Zisserman 2018 LeCun 2019). As result of this re-branding, while most papers published before 2020 use unsupervised learning to describe their work, those that are published after 2020 use the description self-supervised learning, hence the conflict between the use of the two terms.
An interesting moment in this timeline, and the one that furthered the popularity of the term SSL, is the revision by Yann LeCun of his now- fow-famous cake analogy from NeurIPS-16, during a talk he gave at ISSCC-19 and later at AAAT-20 (LeCun] 2020 : "If intelligence is cake, the bulk of the cake is unsupervised self- supervised learning, the icing on the cake is supervised learning, and the cherry the cake is reinforcement learning" [LeCun 2016).
In summary, we can say that self-supervised learning refers to a recently popularized learning paradigm, encompassing predictive tasks where the supervisory signal is provided by the data, without relying on the explicit use of human labels.
Generative and discriminative SS general, self-supervision approaches can be grouped into two categories: generative and discriminative (Docrsch et al. 2015). In generative self-supervision, the task is to build appropriate distributions over collection of data while operating in the pixel space. A common criticism of generative self-supervision is that it is computationally expensive, does not work well with high- resolution images, and that it may be superfluous for representation learning (Chen et al.) 2020b Grill et al. 2020 Typical models relying on this kind of self-supervision are auloencoders (AEs) and generative adversarial networks (GANs) (Kingma & Welling 2013 Vincent et al., 2008 Goodfellow et al.] 2020). Tt should be noted that although both AEs and GANs are categorized as "generative" models, they achieve self-supervision in different and distinct ways.
In contrast to generative SST, in discriminative self-supervision, the task is to learn good representations of the data in order to perform specified pretext task (which we will explain shortly) that does not require a human annotation effort (Doersch et al.) 2015). Discriminative self-supervision is similar to supervised learning the sense that the objective function is often a scoring function that evaluates the discriminative power of learned represcntations. Most of the SSL frameworks we will cover in this manuscript refer to the works of [ecker & Hinton| (1992) and BBrolee discriminative self-supervision in the form it is used nowadays, with the above research efforts investigating representation alignments across different inputs.
al. (1993) as the earliest research efforts that use
Purpose of this survey Thanks to the excellent results obtained by SSL in computer vision, numerous SSL frameworks were proposed within the span of a couple of years. Although most of these frameworks are often specialized in nature, addressing a select number of tasks (such as depth estimation, face recognition, remote sensing, and pose estimation), we could trace their origin to roughly 100 general-purpose SSL frameworks that are applicable to images. Even though several in-depth surveys are available the topic of image-
3

---

Published in Transactions on Machine Learning Research (05/2023)
DAN
(a) Colorization
(b) Inpainting
(c) Geometric transformations
Full together
DNW
DNN
Contrnst
(d) Puzzle solvers
) Instance discrimination
(f) Context prediction
LANS
(g) Masked image modeling
(h) Corrupted image modeling (i) Masked feature prediction Figure 2: Illustrations of various image-bascr pretext tasks for solf-supervised learning.
based contrastive SSL (Albelwi 2022 Khan et al. 2022 due to the fast-paces nature of research in SSL, they do not cover recent non-contrastive SSL mothods that transformed the field. As such, major goal of this survey is to cover the aforementioned image-oriented frameworks for generative and discriminative SSL which benefited from tremendous research and development efforts in recent years, hereby presenting concise and aggregate work to readers who take an interest in this field.
In Section 2 we describe popular pretext lasks for self-supervssion, subsequently detailing a number of relevant technical concepts that are commonly used in Section 3 Diving deeper into SSL as is used nowadays for image-related tasks, Section | we cover recently proposed SSL frameworks for image- training in a chronological order and discuss methods of evaluation in Section In Section G we cover relevant. libraries, repositories, and publicly available implementations that aim at assisting researchers. Finally, in Section 7 we review number of shortcomings of SSL, identify open problems, and conclude our survey.
2 Pretext tasks for self-supervised learning
The image domain allows mumber of unique pretext tasks that enable self-supervision. Below describe the most popular ones and illustrate them in Figure
Image colorization Automated colorization of grayscale images is line of research that was investigated even before the widespread usage of DNNs (Luan et al. 2007 Charpiat et al. 2008). However, the availability of large-scale colored datasets such as ImageNet, combined with the versatility of DNNs, further strengthened the interest high-quality image colorization, especially for the purpose of coloring historical pictures. In parallel to research efforts that aimed at. increasing the quality of colorization, such (Cheng et. al. 2015 lizuka et al., 2016), the idea of using image colorization
pretext task for representation learning also investigated [Larsson et al. 2017 2016 Zhang et al. 2016). Although this task alone was revealed to be too simple to force DNNs to learn complex representations (Caron et al. 2020), colorization is still used in tandem with other tasks to boost the effectiveness of SSL models.
Inpainting-775 task of prodicting a missing part of an image is referred as image inpainting [Bertalmio
al.
2000). With the widespread usage DNNs, inpainting problems also found numerous solutions (Yang
et al.
2017 Yu et al. 2019 One such solution, and the one that allows for the of SSL, is proposed
4

---

Published in Transactions on Machine Learning Research (05/2023)
by Pathak et al. (2016). leveraging context encoders that aim at inpainting large parts of images that are missing, forcing models to learn the image context.
Geometric transformations Inspired by research efforts that bring together geometric transformations and neural networks (Kanazawa et al. 2016 Rocco et al.) 2017). and taking advantage of image- based datasets that almost. always contain upright images, Gidaris et al. (2018) proposed the idea of predicting image rotations as method of self-supervision. Following the success of this method, other types of geometric
transformations were proposed by Novotny et al. (2018); Zhang et al. (201)); Chen et al. (2019)
Puzzle solvers-A A unique image-based task that can be formulated in a SST setting is solving a Jigsaw puzzle (Noroozi & Favaro, 2016), where the goal is to correctly predict the relative location of nine puzzle pieces. This unusual pretext i (ask, as well as a number of derivations, is employed in support of a variely of tasks, including domain generalization (Carlucci et al., 2019), generation of image embeddings (Trinh et al. 2019), image retricval (Pang et al. 2020), and auxiliary learning (Li et al. 2021b).
Instance discrimination Given differently augmented views (i.e., instances) originating from one image, instance discrimination refers to the idea of recognizing these views originating from the same image, while discriminating any other image with a different origin (Wu et al. 2018b). Different from the previously described pretext tasks which achieve representation learning as by-product of the optimization objective, instance discrimination optimizes for representation learning by directly matching the representations of similar images while contrasting the representations of dissimilar ones. In this context, images that are contrasted to similar ones are called negative samples (c.g., the gecko image in Figure 2e The main idea behind representation matching between similar images and contrasting different images is to help DNNs learn reprosentations that are invariant to commonly used image transformations, since most of these transformations do not alter the visual semantics MMisra & Maaten 2020). The origins of this approach can be traced back to the research efforts presented in Hadsell et al. (2006), Sohn & Lee (2012), and Hui 2013
Masked image modeling- The adaptation of masked language modeling in NLP to computer vision as a new pretext lask for self-supervised training was a groundbreaking discovery in generative SSL (Chen el al. 2020a; Bao et al.) 2021). This technique is referred to as masked image modeling (MIM) (Bao et al.) 2021] The idea behind MIM is simple: divide an image into collection of equal-sized patches, mask some of the patches, and task the model with generating their corresponding pattern. As we will discuss in later parts of this paper, while the usage of MIM has popularized generative SSL,, this pretext task can be thought of as variant of image inpainting. The primary difference between MIM and the inpainting method proposed in the work of Pathak et al. (2016) is that MIM uses non-overlapping patches of equal size. After the rise in popularity of MIM (which is often used in conjunction with vision transformers (ViT) (Dosovitskiy el al. 2020)), a number of its variants emerged, with corrupted image modeling (Fang et al. 2023) (sce Figure 2h and masked feature prediction (Wei al. 2022) (see Figure 2i the two most prominent ones.
Others - AAart from the mainstream pretext tasks described above, there are a number of unique tasks that do not fit into one of the above categorics such as: the split-brain approach which trics to predict a subset of image channels from other channels (Zhang et al. 2017), feature consistency method involving synthetic images (Ren & Lee 2018), context prediction (Doersch et al.) 2015), adversarial feature learn-
ing (Donahue et al.] 2016 Donahue & Simonyan 2019). exemplar networks (Dosovitskiy et al. 2014), and
object counting (Noroozi et al.) 2017].
Effectiveness of pretext tasks Given the abundance of pretext tasks for self-supervision, which of these tasks enable DNNs to learn the most useful representations? Although there is no clear answer to this question, ever since the works of Dosovitskiy et al. (2014). Wu et al. (2018b), and|Oord et al. (2018). instance
discrimination was established as the dominant pretext task for image- based discriminative SSL thanks to
the superb results achieved using this type of superviuprvi (He et al. 2020 Grill et al. 2020 Chen et al.
2020b). On the other hand, MIM has been recognized as a tremendously powerful pretext task
that enables
generative SSL to reach and even surpass the results obtained with instance discrimination.
Dosovitskiy
et al. 2020 Bao et al.) 2021 He et al. 2022
5

---

Published in Transactions on Machine Learning Research (05/2023)
Loss
Loss
SSL frame woneworkk
Linear training
Linear layer
ri
SS1 training
Gradient
Gradlent
Gradient
fo(-)
(-(-)
fo(-)
fo(.)
Backbone
Trained bocebboue
ajj
22
Bj
2;
T(-)
T(-)
T(-)
T(-)
a
(a) SST. training and linear evaluation
(b)
Siamese network
(c) Stop-grad
qi
Loss
pred(.)
i
Sadign labols
Zi
Loss
Loss
Gradient
proj (-)
projz()
Tt
r
pred(.)
Gradient
fo
(-)
Gradient
fo
fo)(.)
Weight update
Ci
Ij
22
so(.)
T(-)
T(-)
T(-)
T(-)
x
(d) Delayed weight update
(f) Pseudo-labeling
(e) Projector and predictor
Figure 3: Illustrations of some of the important concepts related to SSL described in Section 3
3 Important concepts in self-supervise learning
In this section, we briefly describe number of commonly used concepts that are relevant the forthcoming SSL frameworks. Although these concepts were key elements of carly individual SSL frameworks, newer frameworks make use of mixture of them.
Notation For clarity, we briefly detail the notation used to describe several core SSL concepts. Given an image x E RP and its categorical association y E RM sampled from a dataset (æ, y) ~ D, with ye = 1 and Vm = 0, vm E (O...., MJMc), let fo(.) be an encoder (i.e., a feature extractor) that maps an image augmented with a stochastic augmentation function T(.) with parameters These features can then be mapped onto set of projections and predictions q using the proj(-) and pred(-) functions, respectively. In this context, projectors and predictors simply multi-layer perceptrons (MLP).
of features r E R using a neural network
Backbone network In the context of SSL, the term "backbone" refers to the feature extractor(s) (i.e., fe(-)) that are trained with SSL frameworks. Typically, a backbone network is a task-agnostic DNN (e.g., a ResNet-50 without the final fully connected layer). The majority of the frameworks we will cover use either variant ResNet (e.g., vanilla ResNet-50, ResNext, or Wide ResNet) or, very recently, vision transformers the backbone.

---

Published in Transactions on Machine Learning Research (05/2023)
SSL training and evaluation In traditional supervised learning, the feature extractor (e.g., convolutional layers) and the predictor (e.g,, linear layers that map features to classes) are trained at. the same time. However, SSL is only concerned with the training of the feature extractor. After the SSL training is complete, the linear layer that maps the features to classes is trained separately.
In Figure 3a we provide a simplified illustration of (left) SSL training and (right) linear evaluation. SSL frameworks are placed on top of backbone networks and are trained in conjunction with the backbone. After the SSL training is complete, the framework is discarded and only the trained backbone is used. Note that this backbone is merely feature extractor. Then, depending on the problem at hand, a new layer that maps features to classes is initialized and trained. It is crucial to keep in mind that the SSL training is only concerned with the quality of features obtained from the feature extractór. As such, the majority (if not the entirety) of the forthcoming concepts as well as frameworks tackle feature extractor training. Nevertheless, for the sake of completeness, Section 5 we will also describe evaluation methods.
Vision transformers Vision transformers represent novel deep learning paradigm that leverages the transformer architecture developed initially for NLP and applies it to image classification tasks.
VIT adopts a preprocessing stop that involves partitioning the input image into non-overlapping patches, which are linearly embedded to create se- quence of tokens. The transformer encoder is then applied to these tokens, with the self-attention mechanism allowing the model to selectively focus on different patches and learn intricate correlation structures among them.
VIL
An essential element of the ViT architecture the [CLS] token, which is prepended to the input and subsequently leveraged for downstream classifi- cation tasks. However, in addition to the [CLS] token, ViT's also generate patch representation tokens that encapsulate information about the corre- sponding patch and its relationship with other patches, based on the atten- tion mechanism (see Figure ! These representation tokens can be utilized for various MIM-based self-supervised lasks, which are relevant for generative SSL frameworks.
:
CLS
CLS
Figure 4: An illustration of ViT input-output rela- tions.
Siamese networks- A form of dual-backbone networks called Siamese networks (Bromley et al. 1993 consisting of two identical neural networks that share the same set of weights are popular architectures for SSL (see Figure 3b). Although this type of networks was useful in solving variety of problems (Chopra
et al. 2005 Bertinetto et al. 2016 Chicco 2021). in the context of SSL, they are mostly employed to achieve consistency between representation wwtennaaiiona when, for example, two instances of the same image are provided.
Apart from Siamese networks, a majority of SSL frameworks use dual backbones that may not share weights due to recently discovered beneficial properties. Tn such cases, the weights of one model are updated via backpropagation, while the weights of the other model can be updated using variety of techniques which we discuss next.
Stop-grad- - Siamese networks generally propagate errors from both branches after the loss calculation. As illustrated in Figure Sel the term "stop-grad" refers to stopping the gradient flow from one branch of a dual-bbckboone network, while allowing this gradient flow to alter the weights of the other branch (Chen & He 2021).
-Siamese
Delayed weight Podates-Assunne a Siamese-like dual-backbone network where one branch is called the teacher and the other one the student. However, different than the Siamese architecture, weights of these models are not shared. In this scenario, delayed-weight updates refer to the idea of propagating the error through only one branch via backpropagation and updating the trainable parameters of the other branch via a predetermined rule (see Figure 3d). Popular implementations of this operation are Mean Teacher (Tarvainen & Valpola. 2017). momentum encoding HHe et al. 2020), and exponential moving average (Grill et al, 2020).
Projection and prediction MLPs The usage of multi-layer perceptrons in the form of projection and prediction heads following feature extractor (e.g., a dual backbone) is acknowledged as powerful technique that greatly improves the effectiveness of SSL methods (Chen et al. 2020d). We visualize this technique in
7

---

Published in Transactions on Machine Learning Research (05/2023)
implemented in BYOL framework (Grill et al 2020 Note that this visualization illustrates asymmetric architecture but the asymmetry is not a necessity for projection/prediie ppeedicteiioonnn ML.ss.
Figure 3e|
The InfoNCE loss (discussed in depth in Section 3.1 aims at maximizing the similarity between representations of two augmentations the same image, while minimizing the same metric across different images. In such cases, the "different" images are referred to as negative samples (Chen et al. 2020b). This concept, which has been the focus of many research efforts (which we will discuss later on), will be particularly relevant for contrastive SSL (He et al. 2020
Negative samples
Memory bank Given a set of images, storing the corresponding image representations, as computed with fo(x) [ri...., Tul, and to subsequently using this memory bank for various lasks (for example, to use the obtained image representations as negalive samples in InfoNCE) (Wu et al. 2018a He et al. 2020
[.1..... Fnl, a memory bank refers to the simple idea of
Pseudo-labeling-AA number of SSL methods discussed below employ pscudo-labeling strategics enable self-supevvision (Caron et. al.) 2018 Asano et. al.; 2019). Such approaches can be visualized shown in Figure 3f where label is assigned to an image based on its feature representation (through the use of, for example, K-means clustering) and where that label is then used to calculate a loss.
3.1 Loss functions to train SSL frameworks
The forthcoming SST. frameworks utilize a wide range of loss functions to enable self-supervised training. Although the usage of these loss functions is often specific to certain frameworks, in this section, we will cover the most prominent losses that see common use across different frameworks.
Cross-entropy loss Cross-entropy loss (CE) is a commonly used loss function in classification tasks which measures the difference between the predicted probabilities and the true probabilitios of categorical variable. Given a prediction ÿ for C-class classification problem, CE for the class is calculated as follows:
exp( Ut
LCr(D,t) -log
Ee exp(re)
In clustering-based SSL, CE and its variants are mainly used with the target label / bbeinng assigned via
self-labeling mechanism such as k-means clustering (Caron et al.) 2018 Asano et al.] 2019 Qian et al. 2022). More recently, distillation-based SSL frameworks also make use of CE where the output of the student
network is matched to that of the teacher (Caron et al. 2021 Gidaris et al. 2021 Li et al. 2021a).
Cosine similarity- Cosine similarity measures the similarity be- tween two non-zero vectors in a high-dimensional space, formalized as a dot product between & normalized vectors U1 and V2 as follows:
sim v2) = V1
a |/2
In the context of SSL, cosine similarity is often employed in combination with noise-contrastive estimation (NCE) for contrastive-learning based based discriminative SSL frameworks. It is also employed by a mumber of prominent distillation networks to quantify representation similarity (Grill et al.) 2020 Chen & He 2021 Given an image æ and two views 2(1,2) ~ T(x) obtained with an augmentation T, let 2(1,2) and q(122) be the outputs of the projection and prediction layers, respectively, obtained by using a Siamese-like backbone similar to the one depicted in Figure SimSiam, for example, then employs negative symmetric cosine simi- larity between projections and predictions defined as - sim stop (22)) 5 sim q2. stop- -oe-ggrrdd (21))
with stop-grad(() referring to the stop-grad operation described above (Chen & He 2021
coiss-contrastive estimation contrastive loss is loss that has a low value when the two input images are similar and a large value when they are dissimilar (Chopra et al. 2005 Hadsell et al.
2006 A funda-
mental loss that enables contrastive training for image-based SSL is InfoNCE [Sohn] 2016
Oord et al. 2018).
which is modification of NCE (Gutmann & Hyvärinen] 2010)- Following Chen ct al.
(2220b). InfoNCE
can be defined using 2n instances of n images in a single batch: [T(x)), T(x),. ((n,),((n))) with
T(-) a stochastic image augmentation function. In this scenario, the InfoNCE loss for a single positive pair is defined as follows:
exp(sim(r;, rT))
(1))))))(((()))) = log Exe k= (R exp(sim(r:. rx))
(1)
8

---

Published in Transactions on Machine Learning Research (05/2023)
X
MLP Head
Backbone
Proji
+
Poodi
4:
VIT
Loss
Lass
Bockbooe
Proja
Preda
q:
Figure 5: Illustrations of (left)) dual--ackbone discriminative and (right) MTM-based generative frameworks.
where f(x;) = r, denotes the feature representation of the ith data point. InfoNCE is the most employed
loss function for SSL frameworks that use contrastive learning (Chen al.) 2020b He al., 2020).
Mean squared error- Defined MSE(v, 0) Ei= 6,)2 the mean squared error (MSE) is
ployed in number of prominent distillation-based SSL frameworks to measure feature alignment Grill
et al. 2020 Tian et al. 2021b Caron et. al.] (021). More recently, MSE has also been adopted to measure
the correctness of reconstruction targets for MIM-based generative frameworks (He et al. 2022 Hou et al.
2022
Tian et al. 2023
Σ 6/1, the mean absolute error (MAF) was rarely used error measurement metric until the resurgence of MIM-based generative SSL, in which is employed to measure the correctness of reconstruction targets (Xie et al. 2022b Tian et al. 2023).
Mean absolute error Defined as MAF(U, 6)
- Proposed by Ermolov et al. (202 Zbontar et al. (2021) Bardes et al. 2202], unique method of self-supervision is to maximize the information content of the embeddings (i.e., projections/ pprriiionsspdiciioosss.... Compared to the previously discussed losses, losses that maximize information content of embeddings are not only unique but also much more complicated.
intormation-mxxmiinni
For example, the loss of VicReg (Bardes et al. 2021 popular information infornatior-maxxinizzionn framework can be defined using two batches of n image embeddings coming from two branches of Siamese-like network, q (qi..... 9n] and q' [g'....,,gT. ..... Then, the VicReg loss is defined as follows:
Cvic(q, 9') = s() tu [v(q) + +o4')) +u (c(q) c(q')]
(2)
Invariance Variance
Covariance
where A, nt, and v are hyperparameters, and the three constituent expressions in this complex loss function play the following role: (1) The invariance term s(q. = Σ l1q 2/? aims to learn invariance to data transformations by making and q' similar. (2) The variance term v(q) aims to prevent norm collapse by giving the components of q and q' a standard deviation equal to y (a fixed hyperparameter). It is defined as a hinge loss v(q) = max(0, (0, 7 S(q, <)), with 5(q, = / Var(qq) + the regularized standard deviation. (3) The covariance term c(q) strives remove correlations between the different components of q, and is given by the sum Eitg(C(a)], over the off- diagonal elements of the d-dimensional covariance matrix C(q).
4 self-supervised learning frameworks
Although most recently proposed frameworks make use of a variety of techniques from both generative and discriminative SSL, the frameworks that we will discuss shortly can be typically calegorized as either generative or discriminative. In the case when a framework leverages techniques that belong to multiple categorics and may thus fall into more than one category, we adopt the designation used by its crcators. Since most of the frameworks are known by their acronyms, we use their abbreviated names the main text and provide their full names in Section of of the the appendix.
9

---

Published in Transactions on Machine Learning Research (05/2023)
SSL framework
Proposed by
Unique property
Deep Cluster
Caron ct al.
2018
Avoids trivial solutions for clustering based SSL
Local Aggregation
Zhnang et a
2079
aggregation metric for cluster assignments
Deeper Cluster
Caron
Al
2019
Integrates rotation-based SSL into clustering
SeLa
Asano el 82
2019
Iuproves Deep Cluster with the Sinkhorn- Sinkhoru-Kuoop algorithu
SCAN
Gansbe
2020
Decouples feature learning and clustering using two-step approach
Deap Cluster-v2
Caron 01 m1 2020
Incorporates various SSL, improvements into Deap Cluster
Sela-v2
M1. 2020
Incorporates various SSL improvements Online clustering with consistency across assignments Converts Deep Cluater into an online method
Sela
Swav
Caron ct ml. 2720
DDC
Zhan 6l. 2020
Coke
Qiau el
2022
Improves the clustering phase with online coustrained k-means method Siugle-stage end-to -o-eet- chustering combined with contrastive learning
Gelf-Claseifior
Amram & Broustein 22021)
Table 1: SSL frameworks that rely on clustering- based self-supervision and their unique properties.
4.1 Discriminative SSL
In terms of discriminative SSL, frameworks can roughly be grouped by their reliance on the following tech- niques: clustering, contrastive learning, distillation and infornation-maxiiizatio. In what fol- lows, we detail discriminative SSL frameworks that fall under the aforementioned categories.
4.1.1 Clustering
Self-labeling via clustcring is one of the most straightforward ways to achieve self-supervision, with clustering being one of the most popular methods for unsupervised learning (Bishop 2006). For neural networks, the usage of clustering based methods for training can be traced back the seminal works of Coates et al.] (2011), ; Coates & Ng (2012), and Yang al. (2016), which paved the way for the use of such methods for SSL. Unfortunately, clustering-bassee methods have to solve number of well-documented issues such as: (1) offline training that prevents their usage for large-scale data, (2) large clusters dominating the majority of the labels or small clusters leading to extremely granular labels, (3) empty clusters, (4) requiring knowledge about the number of clusters beforehand, and (5) trivial solutions where all data are gathered in a single cluster which causes the network to collapse (Xu et al. 2004 Joulin et al.) 2016). Since these issues are fundamental problems of clustcring, all of the clustering- based SSL methods have to tackle these problems in their own unique way when trying perform self-supervision.
The pioneering work of Caron et al. 2018 put forward Deep Cluster, one of the first clustering based SSL methods that achieves results comparable to supervised models. This method solves the issues listed above with an offline training approach and by forcing a uniform distribution across clusters, both of which limit the usage of Deep Cluster. Following that, getting rid of the tricks applied in Deep Cluster became the primary focus of a number of subsequent studies, leading to improved clustering based SSL methods such as SeLA (Asano et al., 2019), Online Deep Cluster (Zhan al. 2020), and Self-Classifier (Amraní & Bronstein 2021). h. SeLa tackles the issue of model collapse by incorporating more principled loss using the Sinkhorn- Knopp algorithm (Cuturi 2013). Online Deep Clustering on the other hand addresses the aforementioned offline training issue to enable online training for large datascts.
Conversely, Van Gansbeke et al. 2020 argue that an end-to-end approach with online training may lead to various problems and propose an approach called SCAN that replaces the use of K-Means for the purpose of clustering with the use of au advanced neighbor search. When it comes to the state-of-the-art, the clustering-base based method proposed in Caron et al.) (2020). known as Swav, which also leverages a number of contrastive elements, is currently considered to be the most stable and accurate approach. Table provides sunmarizing overview of several clustering- based SSL methods, detailing their unique traits.
4.1.2 Contrastive learning
Contrastive learning with the InfoNCE loss (or an exlension of it) is the most popular approach for self- supervision and also the one that received the most research contributions in the past years. Contrastive methods can be traced back to the works of BBrollyy al. (1993) and Chopra et al. (2005), but in terms of modern usage of SSL, Wu et al.) (2018b) and Oord al. (2018) popularized this line of research by proposing InstDist and CPC, respectively. Hjelm et al. (0188) and Bachman et al. (2019) investigated different ways to
10

---

Published in Transactions on Machine Learning Research (05/2023)
SSL, framework
Proposed by
Unique property
InstDist (MPID)
Wa et al
201 8b
Non par-peammttiic softmax calculation
CPC
Gord es
2018
Usage InfoNCE lnss arross multiple tasks
DIM
Hielm of 41
2018
Messures represeutation quality with two novel losses (MINE and NDM)
CPC-v2
Hensil 2020
Iaproves CPC architecture and training
AMDIM
Bachtnan
2019
Extends DIM for mixture- based representations
CMC
Tian Ft R
2020mg
Information-maxiiiaatin
across different: sensory views
MoCo
SSL with momentuum encoder and memory bank Contrastive learning with jigsaw puzsiles
PIRL
Kissa Mansen
2020
SinCLa
Chen el. 6l 20/06
Usage projection heads and new angmentations Improves NoCo with the design of SinCLR Iaproves SimCLR with memory bank and deeper projector MLPs Formulates contrastive learning with clustering using EM One-branch parametric instance classification
NoCo-v2
Chen et al
2020
SinCLR-v2
Chen et ml 20200
PCL & FCL-v2
121
PIC
Cao el di.
20220
DCL
Chiang al: 120
Negative sample section with debiased contrastive objective Learns transformation dependent and invariant represntations SImCLR with negative sample selection using paeudo-labels Imposes iuvariance constraints during SSL training Mixes self eeff-trained uegative adversaries into SSL Dense contrastive loss for SSL.
Lood
Xian ot al.
2020
G-SInCLR
Chakraborty et
2020
ReLIC
Kitrovic sL
2020
AdCo
Hu et al. 2221
DensaCt.
Wang [202
PixPro
Aie et a1:
PixContrast and PixPro losses for contrastive SSL Improves NoCo-v2 with symmetrized loss and without memory bank Usage of stronger augmentations for contrastive learning Attempts solve under- and over-clustering in contrastive learning gearret-neighbors positive samples in contrastive loss
MoCo-v3
Chen
CLSA
Wang Q
2012
Truncated Triplet
Wang et al.
20210
MHCLR
Dwibedt
2021
NoBY
Xie ct al.
2021 b
Combines design principles of MoCo and BYOL for transformers
DNC
Tian
(222T
Alternation of contrastive learning and clustering- hased hard negative mining Maintains the relational cousistency between different instances of images Unifies contrastive learning, distillation, and ietorrnttiio-maxiiinttttt Improves ReLIC with inductive blases to learn more informative representations Simpliftes MoCo with momentum removal Simpliiies MaCo with dictionary removal
ReSSL
Zheng et a 2021
UniGrad
at. 20z24
ReLIC-V2
Tomasev et Rl 2022
SimCo
Zhiang et s..
2072a
SinMooo
Zimang
20/22
VII
La cf al.
2022
Scene-based SSL based on similarity, correlation, and discrimination Explicitly learns multi-gramular visual features
Muga
Zhou
2022
CaCo
Wang et al
20222b
Learns both positive and negative samples end-to-end with an encoder Replaces instance contrastive learning with group contrastive learning Instance discrimination using UniGrad and masked images
Pong al.
2022
SiameseIM
T30 et
20226
Table 2: SSL frameworks that rely on contrastive learning based self-supervision and their unique prop-
erties.
measure representation quality for contrastive learning and proposed DIM and AMDIN respectively, while Tian et al. (2220a) extended contrastive learning for multiple sensory inputs with CMC. After the aforementioned works contrastive SSL attracted significant rescarch interest but it the groundbrcaking results obtained with MoCo which used memory banks with delayed weight updates that pul contrastive SSL really into the spotlight (He et al. 2020). Shortly after, Chen et al. (2020b) proposed SimCLR and with it, further improved the statc-of-the-art with the help of projection heads and strong augmentations and cemented the importance of contrastive self-supervision as a learning paradigm. Incorporating the enhancements of SimCLR into MoCo, Chen et al.
(2020d) proposed MoCo-v2 and showed that there still exists a large margin (021)) later introduced a third version of MoCo, exploring the usage of vision transformers as backbones. The reliable design of MoCo and its improved versions were the foundation of
for improvement. Chen et al.
many subsequent contrastive SSL frameworks, such AdCo (Hu et al. 2021
MocHi (Kalantidis et al. 2020
and DenseCL (Wang et al. 2021c/.
While the above architectures mostly use dual backbones, Cao et al. 2020 proposed PIC and demonstrated
the viability a single-branch backbone architecture for contrastive learning. Kalantidis et al. (020)) experi- mented with hard nogative samples for improving the effectiveness of contrastive learning and Wang & Qi (2022) demonstrated the usefulness of stronger augmentations. After the success of Moco-v2 and Moco-v3, and with the increased availability of unique SSL methods, frameworks like G-SimCLR (Chakraborty et al., 2020), MoBY (Xie et al.] 2021b), SimCo, and SimMoCo (Zhang et al. 2022a), which combine multiple SSL meth- ods into a single one, gained traction. More recently, SSL frameworks such as UniGrad (Tao et al. 2022a) claim to combine four self-supervision methodologies (clustering, contrastive, distillation, and information- maximization) into a single framework and to unify discriminative SSL training.
Although contrastive methods garnered more attention than clustering- based methods, they are also sub- ject. to similar problem that, needs to be mitigated: network collapse (Jing et al. 2021). Contrastive
11

---

Published in Transactions on Machine Learning Research (05/2023)
SST, framework
Proposed by
Unique property
BYOL
Grill et al
2020
Avoids trivial solutions through network asymuectr
SimS1an
Chen
with simple Siamese networks without negative samples
0BoW
Gidaris ot
202
Ouline wog-o for SSL
DirectPred
Jan el SL.
20216
Adjusts linear predictor with gradieut-free approach
SEED
2021
Knowledge distillation from large to small models
DisCo
Lao PT 13.
2021
Combines contrastive and distillation learning for lightweight models
DIND
aron
Knowledge distillation with vision transformers
EaViT
La et ml.
212 m
Multi-stage architectures with sparse self-attentions and region matching for efficient SSL Distillation- SSL for small-scale models
BINGD
Xu el. Bil
2021
TinyMIN
Ren et al.
(2023
Distillation to transfer knowledge from large MIM-based models to amall models
Table 3: SST frameworks that rely on distillation-base self-supervision and their unique properties.
SSL framework
Proposed by
Unique property
KNSE
Ermolov al.
2021
Whitening Mean Squared Error loss for information- mowiiiiiaaiiiiiiiiinniii SSL with redundancy reduction
Barlow Tvins
Zbontar et a.
20ZT
VicReg
Bardes
2021
Variance- inw.riance--ooaaiaaaan
regularization for avoiding collapse
TWIST
Wang ot al.
m
Thcoretically explainable TWIST loss that avoids collapse Improves Barles Twins with TLDR encoder
TLDR
Kalantidis
[222
ARB
Zhisng et RL
20/2
Aligns feature representations with nearest orthonormal basls
VicRagl.
Hardos
22022
Improves VicRag with lncation- and feature- matching
Table 4: SSL frameworks that rely on informmaiion-maxiiizaaiiiooooo
sell-supervision and their unique
properties.
methods prevent complete collapse of a network through the use of negative samples. However, Hua et al. (2021) surprisingly demonstrated that contrastive SSL frameworks can suller from another type of collapse, namely dimensional collapse, wherein representations collapse into low-dimensional manifold. Given the importance of negative samples in preventing collapse in contrastive SSL, understanding the effects of nega- tive samples and finding better sampling techniques became active research topic shortly after (Chuang et al. 2020 Robinson et. al.] 2020 Zhang et al. 2022a). A summarixing overview of several contrastive SSL
frameworks can be found in Table 2
4.1.3 Distillation
Can the collapse of networks be prevented without the use of self-labeling contrastive loss that relies negative samples? Through an asymmetric framework called BYOL, Grill et al. (2020) demonstrated that neither of those techniques are necessary to achieve self-supervision when the proposed method relics on distillation (Hinton et al., 2015). The general idea behind distillation is to train a network (student) to predict representations
another one (teacher) (Tarvainen & Valpola 2017). Shortly after the proposal of BYOL, Chen & He (2021) proposed SimSiam am, a symmetric (Siamese) framework that uses neither negative projection/predietion
samples nor clustering. but leverages instead stop-grad and projection/ prredictio MLPs. This was followed by OBOW (Gidaris et al. 2021), in which the task is to reconstruct bag-of-visuallwords representation.
Similar to the trends witnessed for clustering and contrastive-learinng distillation-based SSL frameworks were experimentally combined with other frameworks in an attempt to obtain boosts overall effectiveness. Frameworks such as DisCo (Gao et. al., 202T) and MoBY (Xie et al. 2021b merged multiple frameworks together, while others tried improve the effectiveness of established methods, such as MSF (Koohpaycgani et al. 2021) and ORL (Xie al.] (221aa), improving upon BYOL.
How do distillation methods avoid network collapse? Tian et al. 2020c) and Fetterman & Albrecht (2020 argued that methods that incorporate batch statistics into training (c.g., batch normalization) aid BYOL (and potentially other distillation--assed methods) in preventing collapse, but this hypothesis was prompily refuted by Richemond et al. (2020). Recently, Li et al. (202aa) scrutinized SimSiam and found it to be highly sensitive model size. Nevertheless, a definite answer the way distillation-base SSL methods avoid collapse is not yet found. Table provides a summarizing overview of several SSL frameworks that rely on distillation.
12

---

Published in Transactions on Machine Learning Research (05/2023)
Module
Proposed by
Unique property
Infolin
Tion et al
2020b
InfoMin principle and evaluation of augmentations
InterCLR
Xie
Inter-image invariance contrastive learning
HEXA
et al
2720A
Proposes new data augmentation methods that are harder to predict
MocHl
Kalanidis el
2020
Hard negative image mixing approach
ReSin
Enhances SSL representations with region similarities
MSF
Koon payyora
111
al.
2021
Enhances BYDI. by shifting the embeddings to be close to the mean of its instances
ORL
X1e al.
Utilizes BYOL object-level training
CEB
Lid RL. 2021
Mrasures the amount of compression in the learned representations
SEX
Lavole el all 2122
Employs simplicial emboddings to map unnormalized representations onto simplices Investigates optimal ensemble models for discriminstive SSL frameworks
EXS
Ruan el 2029
MRCI
Liu et al.
3022b
Uses MIM as a method to avoid the discriminative information overfitting
Kukleya
2022
Assists contrastive methods learn group-wise features and instance-specifie details Enhances contrastive learning with domain- invariant features representations Data angmentatiin strategy that enriches backgrounds of crops
ARCI
ARRO ob Al.
20233
MosRop
Wang et
Table 5: Enhancements for existing discriminative SSL frameworks and their unique properties.
4.1.4
Information maximiizaonn
The fourth and final discriminative self-supervision category we cover is information-maximiaatte maximization, having as primary idea the maximization of the information conveyed by decorrelated embeddings. Such approaches come with a number of advantages, in particular, they neither require negative samples nor require an asymmetric architecture to avoid collapse. Instead, they completely rely innovative loss functions to avoid collapse. As a result, most of the frameworks that fall under this category can be characterized by the novel loss function that is used.
Information-maximizatoo as a method for self-supervision was put forward by Ermolov et al. (2021) and Zbontar et al. (2021), , where the former proposed W-MSE loss, which constrains the batch samples to dissipate a spherical distribution, and where the latter (Barlow Twins) aims at making the normalizod cross-correlation matrix of the embedding vectors be close to the identity matrix. Bardes et al. (2021) further improved the loss of Barlow Twins with the VicReg framework, proposing loss based on variance, invariance, and covariance (described above in equation 2p. Successor frameworks such as TWIST (Wang et al.) 2021a), TLDR (Kalantidis et al. 2021), and ARB (Zhang et al.) (022b) followed the path paved by the previous frameworks and aim at improving the losses in different ways. Due to the complex nature of the losses used in
as a mothod for self-supervision, we refer the interested reader to the respective research papers underlying those frameworks. Table 4 provides a sunmarizing overview of several SSL frameworks that rely on information maximization.
infomatioo-maxiiizzaiin.
4.2 Enhancements for existing frameworks
So far, we have covered a large number of discriminative SSL frameworks, all of which have consistently improved state-of-the-arl results across various computer vision tasks. However, we have observed trend that emerged towards the end of 2020: framework-agnosticc enhancements. These modules are small tweaks to existing frameworks that can improve their performance in various ways, such as utilizing harder im- ages/ agges/augmont /angmmntatiinns (Kalantidis et al. 2020: Li et al. 2020a), improving object-level representations (Xiao let al. 2021 Xie et al., 2021a), or enabling optimal ensemble models (Ruan et al. 2022). For completeness,
we have listed these modules separately in Table 5
4.3 Generative SSL
From 2018 onward, generative SSL was largely dismissed in favor of discriminative training methods with contrastive learning holding the prime spot for research (He et al. 2020 Chen et al. 2020b). Popular discriminative frameworks such as MoCo, SimCLR, and BYOL were employed for a variety of unique tasks and were further improved with enhancements taken from each other (Chen el al. 2020d[c). In an unexpected turn of events, towards the end of 2021, generative SSL came to dominate image-based self-supervised lcarning and became the primary rescarch focus, dethroning contrastive lcarning as well as discriminative SSL (Bao et al.] 2021 Zhou et al.] (2021). The advances in the field from the last quarter of 2021 the first quarter of 2023 were so rapid that state-of-the-art results in generative SSL were improved
13

---

Published in Transactions on Machine Learning Research (05/2023)
SSL framework
Proposed by
Unique property
B1GAN
Donahuc et al.
2016
Bidirectional CAX with additional encoder modela
Dumonha A
20T6
framework
an additional inference model
BigBIGAN
Donahue Simonyan
2019
BIGAN with the generator of BigGAK GAX with auxiliary rotation loss SS-GAX with label augmentation VQGAN with label quantivation and VIT backbone
S3-GAM
Caen al.
SS-CAN-LA
Hon 202T)
V1t-VQGAN
TD ot al.
Table 6: SSL frameworks that rely on GAN- generative self-supervision and their unique properties.
monthly basis. This rapid expansion also came with large category of unique approaches which resulted in frameworks of generative SSL becoming much less standardized opposed to frameworks in discriminative SSL where the latter mostly contains straightforward Siamese-like stal-bbccbooes as shown in Figure 5 In order to improve readability, we will group generative SST frameworks into two categories: the ones that use generative adversarial networks (GANs) and others that use a form of masked image modeling.
4.3.1 GAN-based generative SSL
While the usage of generative neural networks can be traced back to the work of Hinton et al. (2006), it the seminal work of Goodfellow et al. (2020) that popularized generative models with the newly proposed
GAN framework. Since the work of Goodfellow et al. 20200), numerous GAN variants were proposed with
some of them recently taking advantage of advances in SSL, such as incorporating rotation prediction (Chen
ctt al. 2019). jigsaw puzzles (Baykal & Unal 2020 Baykal et al. 2022). and self-labeling (Lučić et al.) 2019)
However, most of the research the GAN space has primarily locused on enhancing the fidelity of images generated by the generator network, which typically evaluated using metrics such as the Fréchet Inception Distance (Heusel ct al. 2017). As result, these studics largely ignore the discriminative network and lack comparative evaluations on downstream tasks, leaving them out of the scope of SSL. In what follows, we focus on those research efforts that evaluate the discriminative power of GANs on downstream tasks.
With a unique twist to GANs, Donahue et al. (0066) proposed BIGAN, a GAN framework that contains an additional encoder network trained in conjunction with the generator and discriminator networks with the objective of inverting the generator. After the training is completed, this encoder can be used as a feature extractor for downstream tasks. Independently, Dumoulin et al. (2016) proposed generative framework called ALI that is almost identical to BiGAN. Leveraging the improved generator of BigGAN (Brock et. al. 2018 in BiGAN, Donahue & Simonyan (2019) proposed BigBiGAN which comes with better downstream transfcrability results. Taking inspiration from the developments in the area of discriminative SSL, Chen
ltt al. 2019)) proposed SS-GAN which exploits rolation as an auxiliary task to achieve self-supervision with
GANs. This framework was further improved with the addition of label augmentation by Hou et al. (0221
One of the most recent approaches within GAN-based SSL is ViT-VQGAN (Yu et al. 2021 which improves VQGAN Esscr et al. 2021 using ViT backbones.
Overall, the usage of GANs in SST has not become a mainstream method due to a number of GAN-related limitations, ranging from mode collapse to limitations related to scalability, as well as lack of flexibility in backbone networks.
4.3.2 Generative SSL with masked image modeling
Although only couple of years have passed since the discovery of ViT's (Dosovitskiy al. 2020), these archi- tectures have been shown to achieve state-of-the-art results in variety of vision tasks. In their work, Doso- vitskiy et al. 2220)] demonstrated the feasibility of SSL using MIM as pretext task (although it was called masked patch prediction by the authors), where this pretext task is scamlessly supported by the pasch-based image intake of ViTs. Developing this technique further, BEiT was one of the first frameworks that success- fully employed MIM with vector quantized images and ViTs (Bao et al., 2021). It is important to note that BE1T does not directly predict the pixel values of the image but learns to predict discrete visual tokens which are created from image patches (Wu al. 2020). BEiT uses the tokenizer a discrete variational autoen- coder (dVAE) - of DALL-E (Ramesh et al.
2021 which requires offline training before training the final
14

---

Published in Transactions on Machine Learning Research (05/2023)
SST, framework
Proposed by
Unique property
1GPT
Chen et al
2020a
MIM with 9-bit pixel clastering per patch
BEST
Bao et. m
2021
Patch-based MIM with ViTs using offline DALL-E tokenizer
MAE
He A
2022
MIM with sutoencoders using lightweight VIT encoders and pixel-based reconstruction BEIT with an online tokenizer trained using the DINO objective
180T
Zhou et a
2021
SinMIN
20
1226
BEIT with pixel reconstruction target without tokenizer Proposes a new rodehook to replace the DALI-E tokenizer
PaCD
17mng ot 20221
YaskPeat
a1:
MIM training
HOG as reconstruction target
data2vec
Baevvki ct
2222
BE1T with a teacher-student model and stronger augmentations
CAE
Chen al: 2022a
MAE with DALL-E token target reconstruction
CIM
Fang el sl 2023
Corrupted image modeling for generative SSL with additional discriminative objective Multi-scale hybrid
MCMAE
Gao et 10 2022
for improved MIM performance
ConMIN
Y1
2022)
Contrastive learning MIM patches
CNAE
Hilaa e ni 2022
MTM + contrastive learning with shifted image views Self-distillation with high-level feature reconstruction
SdAE
NILAX
Hoi ct ml
2022
MTM with semantic-aware mask sampling and CLIP-assisted feature reconstruction BEIT with CLIP tokenizer as a teacher and patch aggregation strategy
BEIT-V2
Peng el W 2172
BELT-v3
Wang et
21226
BEiT-v2 with text fusion
CAB-v2
Zhang ct at.
20222
CAE with CLIP tokenizer
CAN
Mishhn
Combines contrastive learning, MIM, and image denoising with symmetric backhones Progressively drops reconstruction tokens in MAE for better speed/ performanee trade-off networks Uses pixels, DIND features, and CLIP features for reconstruction
PCAE
Li et al. 2028
SparX
&L 2023
MIM for convolutional
NRMAE
GBo et
2025
Table 7: SSL frameworks that rely on MIM-based generative self-supervision and their unique properties.
model. BEiT gave rise to BEiT-v2 (Peng et al. 2022 and BEiT-v3 (Wang et al. 2022a which obtain better results using the CLIP tokenizer (Radford et al. 2021) with a patch aggregation strategy. Mcanwhile, the requirement of an external lokenizer for BEiT was alleviated by iBOT (Zhou el al. 2021 which introduced an online tokenizer trained with the distillation routine of BYOL, thus leveraging the advances made on the side of discriminative SSL. Xie et al. (2022b) got rid of the tokenizer of BEiT and proposed SimMIM, which directly operates over pixel values and predicts them.
Narrowly predating BE1T, Chen al. (2020a) proposed iGPT by leveraging GPT-2 (Radford al. 2019) and adapted it to vision, which represents images with tokenized patches using 9-bit color palette by clustering
RGB pixels, and then training this model with the MLM objective of BERT (Devlin et al. 2018). The primary
difference between the training objective of iGPT (i.e., BERT-style MLM) and MIM of BEiT is that the latter directly uses image patchos as an input, therefore not losing any pixel-level information.
With a unique lake on MIM, He el al. (2022) proposed MAE, an asymmetric autoencoder framework that directly learns to reconstruct image patches. What unique to MAE is that its encoder (ViT) only processes unmasked patches (c.g., 25% of all patches) without any tokenizer, making it much faster than the frameworks we have discussed thus far. He el al. (2022) also evaluated the effect iveness of different reconstruction targets and found no statistically significant difference between reconstructing DALL-E tokens and pixels, suggesting simple pixel reconstruction to be a viable reconstruction target. Building upon MAE, Chen et al. (002aa) proposed CAE which comes with a latent contextual regressor and uses the DALL-E tokenizer, which was replaced in the next iteration of this framework (CAE-v2) (Zhang et al. 2022d) by a CLIP tokenizer.
At this point we believe it is important to reiterate that MIM was explored using different reconstruction targets such as (1) dVVEE-baeed patch tokens (Bao et al. 2021 (2) clustering- based patch tokens (Chen et al. 2020a), and (3) pixel values (Xie et al. 2022b). Expanding this corpus, Wei et al. (2022) proposed MaskFeat in which the task is to predict Histograms of Oriented Gradients (HOGs) a hand-crafted feature descriptor (see Figure 2ib and argued that a broad spectrum of image features can be used as largets in MIM. Following their work, SdAE (Chen et al. 2022b) and MILAN (Hou et al. 2022 demonstrated the fcasibility of reconstructing high-level features. For an overview of reconstruction targets for Masob generative SSL frameworks, see Table
BEiT-v3 is framework that fuses vision and text but we include for completeness.
15

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 6: ImageNet top-1 accuracy with linear probing on frozen representations for discrimina- tive SSL frameworks plotted against the mum- ber of parameters in the trained backbone. The con- necting lines indicate different backbone networks trained with the same framework. In both figures, nodes with cireles indicate CNN-based architectures, whereas triangles indicate transformer- based archi- lectures. Note that a few frameworks with overlap- ping results are omitted from the figures, and that axis values scaled independently to improve vi- sual clarily.
81
Maps "ESVIT
ImageNet top-1 accuracy (%6
SLMCLR-V2
PeLIC-v2
78
Svaly
paln-v3
Stores e
SINGLR
75
75.1.M: UKCERECS
Vicflep
1h 1prottperroolee Conilaiss
71
6tLa-v2
gansion
KPC
<PC-v2
Noto
61
50
Resttlic
20
100
200
500
1000
Number of parameters (Millions)
Experimenting with the input side Fang al. (2023) proposed CIM where an auxiliary generator (in their use-case, BEiT) corrupts the imput image and the proposed framework aims to (1) discriminate patches (classification for each patch) and (2) generate the original image.
Table 8: Reconstruction targets for generative SSL frameworks that use MIM.
The current trend for generative SSL to combine both generative and dis- criminative losses together improve the quality of representations. In par- ticular, contrastive learning has become a popular task to combine with MIM with frameworks such as CAN (Mishra et al. 2023) CMAE (Huang et al. 2022 and ConMIM oYi et al., 2022) leveraging advances made on the side of con- trastive SSL.
Framework
Reconstruction
1GPT
9-bit pixela
BELT
DALL-E tokens
Pixels
180T
Distilled tokens
SimMIN
Pixels
PeCD
PeCD tokens
MaskFeat
HOG
data2vec
DALL-E tokens
All of the generative frameworks we have discussed thus far use some form of vision transformer backbone as opposed to the majorily of the discrim- inative frameworks, which make use of ResNets. In an attempt to leverage masked convolutions, Gao et al. (2022) proposed MCMAE, a framework that employs the hybrid convolution-transtoon
CAE
DALL-E tokens
DALL-E tokens
NCMAE
Pixels
ConMIN
Patch features
CNAE
Pixels
S4AE
High-level features
MAE which is able to learn dis- criminative representations. Very recently, Tian et al. (2023) showed that classical (ResNets) and modern (ConvNext (Liu et al. 2022a) CNNs can be trained with MIM and achieve state-of-the-are results that can rival those of ViIs. The results obtained by Tian et al. (2023) indicate that ViTs, which have been considered a prerequisite for MIM, are not irreplaccable and that CNNs can still compete with ViTs in generative SSL.
MILAX
High-level features
BE1T-v2
CLIP tokens
BEiT-v3
tokens
CAE-v2
CLIP tokens
CAN
Pixels
PCAE
Pixels
SparK
Pixels
MRMAE
CLIP tokens
5 Evaluating SSL models
As briefly noted in Section 3 the SSL frameworks covered thus far are concerned with the training of feature extractors that can extract robust and useful features from images. Regardless, those feature extractors must be evaluated for a fair comparison of performance, which is the focus of this section.
In the SST literature, three types of evaluations are commonly used: (i) fine-tuning the entire model, (ii) linear ovaluation, also known as lincar probing or lincar protocol, and (iii) K-ncarest neighbors (KNN) evaluation using extracted features. A further distinction can be made based on the dataset the trained model is evaluated on: either (a) on the same dataset, typically ImageNet, that was used for the self-supervised training or (b) on different datasets to test downstream transferability.
KNN evaluation After the SSL training is complete, features of the training images are generated from the backbone and matched to their corresponding labels, thus creating a feature bank. Next, predictions are made for the test images based on the KNN labels of this feature bank. For KNN-based evaluation, following Wu et al. (2018)), 200 is often used. Although this form of evaluation was popular carly on, the majority of recently proposed frameworks evaluate their models using linear evaluation by fine-tuning.
16

---

Published in Transactions on Machine Learning Research (05/2023)
ImageNet top-1 accuracy (%
ImageNet top-1 accuracy (96
80
pEIT-2
BETT-2
OAN
2017
85
69
#askiestt
101
CorPPNN
Spark
Slatt
MackTest
CAI-V2
SiaMD 2park
100
500
6000
Number of parameters (Millions)
Number of parameters (Millions)
(a) Linear probing
(b) Fine-tuning
Figure The ImageNet Top-1 accuracy of backbones that are trained with MIM-based generative SSL frameworks measured using (a) linear probing and (b) fine-tuning. The connecting lines indicate different backbone networks trained with the same framework. In both figures, nodes with circles indicate CNN-based archilectures, whereas triangles indicate transformer-based architectures.
Linear evaluation - For this type of evaluation, all trainable parameters (e.g,, weights) in the model are frozen and new linear layer, which maps features to prodictions, introduced to the trained model. Then, only this linear layer is trained on the training to achieve an optimal performance.
Fine-tuning For this type of evaluation, once again, linear layer is introduced to the SSL-trained model which maps features to predictions. Then, in supervised fashion, the entire model is (re)trained the training dataset, after which an evaluation is performed on the test/validation set.
Benchmarks- In order to provide an aggregate view of the field, we provide the benchmarking results below.
For the majority of the discriminative SSL frameworks covered in this survey, we provide compar- ison of model size to linear probing accuracy on ImageNet in Figure 6
For MTM-based generative SSL frameworks, provide comparison of model size to linear probing and fine- fine-tuingg accuracy ImageNet in Figure 7
From Table 21 to Table
we provide the datasets used the respective papers SSL. frameworks.
From Table 24
Table
30
we provide benchmarks on ImageNet-IK,
In Table 31
provide benchmarks on COCO.
Evaluation preference for discriminative and generative frameworks A noteworthy observation in the evaluation self-supervised learning frameworks is that while most discriminative SSL frameworks tend favor lincar evaluation, the majority of generative SSL frameworks tend prefer fine-tuning. This is primarily due to the poor results obtained with linear probing with generative frameworks that only use MTM the pretext task (see MAE, BEiT, SimMIM, iGPT in Figure 7). When discriminative elements such contrastive learning, distillation, the CLIP tokenizer which is trained contrastively, used, linear probing accuracy shows dramatic increase (see MILAN, iBOT, CAE-v2, and BEiT-v2 in Figure
6 Availability and comparability of SSL frameworks
Most of the frameworks covered in Soction perform experiments on ImageNet, COCO (Lin et al., 2014). and Pascal VOC (Everingham al. 2007), thus enabling straightforward benchmarking and comparability. Moreover, many SST. frameworks come with implementations and trained models that are publicly available,
17

---

Published in Transactions on Machine Learning Research (05/2023)
contributing to speeding up research SSL, For example, the availability and the straightforward adopt- ability of the MoCo framework enabled a number of follow-up studies that used the code of MoCo (Kalantidis
et al.
2020:
Hu et al. 2021 Wang et al., 2021c). For the SSL frameworks covered in this survey, in Table 17
Table
18
and Table 19 WC provide details on the availability of official implementation as well as trained
models.
Apart from the availability of official implementations, the availability of third-party repositories also accel- erated the adoption of SSL, enabling unified experimentation. Alas, not all third-party repositories are up to date, and some of them have already been abandoned. In Table 20 we provide number of useful SSL repositorios that have been updated within the third quarter of 2022.
7 Conclusions, current trends, and directions for future research
In this survey, we reviewed general-purpose frameworks that use images for SSL training, with the goal of bringing interested researchers up to speed with the field of SSL. In what follows, we highlight a number of directions for future research that are ripe for contribution.
Theoretical understanding of the requirements of SSL As detailed in Section 4 the successful implementation of discriminative supe-sipe learning frameworks requires several prerequisites. To this end, several studies have investigated the efficacy of these requirements, covering topics such as the necessity of nogative samples [Kalantidis et al. 2020 Xic ct al. 2022a), the importance of image augmentations (Zhang
ett al. 2022c Wang el. al.] 2022d), and architectural tricks (Chen & He 2021 Furthermore, a number of
research efforts have attempted to explain the underlying mechanisms behind collapse avoidance (Garrido
ct al. 2022b Chen ct al. 2023b). Because they were the carliest self-supervision methods, clustering (Assran
et
2023) and contrastive learning (Hu et al.) 2023 Johnson et al.) 2023 Tian 2023) have received
significant. attention in terms of theoretical contributions. However, other self-supervision paradigms are
areas where theoretical explanations are still lacking and are open for further research efforts.
Domain- and task-specific SSL-The majority of the frameworks covered in this survey are task-agnostic and evaluate their performance on the ImageNet dataset and a number of various downstream tasks focusing on natural images. However, the effectiveness of these models on natural image datasets may not necessarily generalize to other datasets that contain different image modalities or to other tasks. Therefore, investigating the effectiveness of SSL frameworks that leverage the unique characteristics of data in other domains such
medical imaging (Ramesh et al. 2022 Chen et al. 2023a) other tasks such classification in the
wild (Goyal et al. 2021a Tian et al. 2021a), object detection MMishra et al. 2021 Li et al. 2022b) pose
estimation CChen et al. 2023c), action detection as well as human-object interaction (Wei et al. 2022 Shah
et al.
2023 represents a relevant area of research.
Calibration, interpretability, and adversarial robustness Initial findings suggest that models trained using SSL exhibit distinct properties for robustness and interpretability when compared to models trained via supervised learning (Hendrycks et al.) 2019 Zhong et al.) (022).. However, many of the beneficial and detrimental effects of self-supervised training on downstream tasks remain unclear.
Efficient SSL- The training of SSL models demands a substantial amount of computational resources in comparison to supervised learning. For instance, as reported by Chen et al. (2021), the training of MoCo-v3 with a vision transformer backbone requires approximately 625 TPU days. Consequently, SSL has significantly increased the computational demands of DNN training. This observation explains why a vast majority of the contributions to the frameworks discussod in this survey have at least one author with an industry affiliation (see Table 17 to Table [9]. Moreover, the majority of these contributions (> 80%) come from industry labs such as Facebook AI Research, Microsoft Research, DeepMind, Google Rescarch, SenseTime, ByteDance, and Huawei. To mitigate the high costs of training, rescarchers have started exploring techniques for efficient training and evaluation (Li et al. 2021a Garrido et al. 2022a He et al. 2022 Despite the progress made, there is still a considerable amount of work to be done in this field.
KNN-based evaluation, linear probing, or fine-tuning?- - As we mentioned in Scction most gener- ative SSL frameworks prefer to use fine-tuning as the method of evaluation while discriminative frameworks prefer to use linear probing. In favor of those two, KNN-based evaluation has been mostly abandoned.
18

---

Published in Transactions on Machine Learning Research (05/2023)
(2022) argue that there is no correlation between the aceuracy of linear probing and fine-tuning or downstream transferability. [e et al. (2022) further argues that linear probing misses the opportunity to utilize strong but non-linear features, and this sentiment is repeated by number of follow-up research efforts (Yi et al. 2022 Chen et al.) 2022a). The most recent research effort on this topic is the work of (Park et al., 2023 where the authors argue that models trained with MIM exhibit a bias towards texture whereas contrastive learning leads to bias towards shape, suggesting that this may be the explanation for the difference in linear probing accuracy. Nevertheless, thorough investigations on SSL model evaluations and convincing explanations for their differences are largely missing.
Chen & He (2021) and He et al.
On the usage of tokenizers in SSL-Frow Table 8 it can be scen that a number of MiM-basod generative SSL frameworks make use of a previously trained tokenizer in order to enhance learned representations. In particular, (Peng et al. 2022: Gao et al., 2023 Zhang et al.) 2022d) demonstrate that the usage of CLIP tokenizers (either as-is or as a teacher for distillation) boosts the performance of frameworks considerably. However, using a tokenizer that is trained on a large corpus of images to demonstrate statc-of-the-al results against other frameworks that do not. have access to this level of supervision has recently attracted critieism from the community (ICLR) 2023). We believe that the investigation of the efficacy and necessity of tokenizers in generative SSL is an area that is largely unexplored.
Moving forward: SSL seem to favor generative approaches over discriminative ones, it is important to note that generative SSL has not only benefited from the discovery of vision transformers, but also from advancements made in the area of discriminative SSL. The most recent comparative studies suggest that the answer to the discriminative versus generative question is not that straightforward and that both approaches have their merits and limitations (Park et al. 2023 As mentioned, some of the newly proposed generative frameworks also leverage discriminative features such as contrastive learning (Yi et al. 2022 Zhang et al., 2022d) or distillation (Zhou et al.) 2021 Peng et al., (022). Therefore, we speculate that this trend will continue and that the newly proposed frameworks in the upcoming years will leverage improvements from both sides order to further improve the slate-of-the-art.
generative or discriminative SSL? Although recent. results obtained with generative
Cadence of research in SSL and the extent of this survey Before we conclude our survey, we would like to bricfly discuss the cadence of rescarch in SSL and the breadth of topics covered in this survey.
It is widely recognized that the field of machine learning has experienced an unprecedented growth in research and development over the past decade, particularly following the groundbreaking results achieved with AlexNet (Krizhevsky et al. 2012). During this period, significant advances have been made not only in the architectural design of deep neural nerworks, but also in optimizers, training routines, normalization techniques, data augmentation, and various other arcas. While these improvements have steadily advancod the state-of-the-art in supervised learning, self-supervised learning has also benefited from the majority of these advancements from the beginning. Thanks to the integration of these advancements into SSL from its early stages, as well as the availability of computational resources, the state-of-the-art in SSL has improved rapidly since 2018. The pace of rescarch has been so fast that some frameworks have been improved upon even before their predecessors got published (e.g., CAE (Chen el al. 2022a) and CAE-v2 (Zhang et al. 2022d) BEiT-v2 (Peng et al. 2022) and BEiT-v3 (Wang et al. 2022a))
Given the aforementioned observation, to cover the most up-to-date rescarch efforts, we have decided to include papers that have not yet been published in conference proceedings or journals. Consequently, the majority of the papers cited in this survey, from 2021 onward, are preprints. At the time of submitting this survey, the latest conference surveyed for relevant papers was the 11th International Conference on Learning Representations (ICLR), held in May 2023, from which a portion of the papers cited in this survey have
been rejected, withdrawn, or published (Wang et al. 2023 Fang et al. 2023 Park et al. 2023 Chen et al. 2022a
Peng et al.] 2022 Chen et al.] 2023b Gao et al. 2023
Mishra et al. 2023 Li et al. 2023 Kukleva
et al. 2023 Zhao et al. 2023 Chen et al. 2023c Park et al. 2023).
Funding Information
This work was supported by grant from the Special Research Fund (BOF) of Ghent University (BOF/STA/202100))
19

---

Published in Transactions on Machine Learning Research (05/2023)
Appendix for:
Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training
The content of appendix detailed below.
A list of abbreviations is provided in Section
Metadata for the frameworks such the primary affiliation of authors, publication date, and source code as well availability of trained models provided for:
Discriminative SSL frameworks in Table 17
Enhancements to existing SSL frameworks in Table 18
Generative SSL frameworks in Table 19
Repositories that are useful for vision-based SSL are listed in Table 20
Datascts used for the evaluation in the respective papers of the frameworks are provided for:
Discriminative SSL frameworks in Table 21
Enhancements discriminative SSL frameworks in Table 22
Generative SSL frameworks in Table 23
Benchmarks ImageNet-1K provided for:
Clustering-based SSL frameworks in Table 24
Contrastive--eemming--aas SST. frameworks Table 25
Distillation-based SSL frameworks in Table 26
Infornattior-mmxiiniitttobbbeedd SSL frameworks Table 27
Enhancements to discriminative SSL frameworks in Table 28
GaN-based SST, frameworks in Table 20
MIM-based SSL frameworks in Table 30
Benchmarks on COCO for all frameworks are provided in Table
20

---

Published in Transactions on Machine Learning Research (05/2023)
A List of abbreviations
Clustering frameworks
Enhancement modules
Self labeling
InfoMin
Mutual information principle
SCAM
Semantic cluatering by adopting pearest neighbors Swapping assiguments between multiple views of the same image
InterCIR
Inter-image contrastive learning
HEXA
Hard examples
MocBi
Mixing of contrastive hard negatives
ODC
Online deep clustering
Resim
Region similarity representation learning
CoKe
Coustrained K-means
MSF
Mean shift for self-supervised learning
DRL
Object-level representation learning
CE3
Conditional entropy bottleneck
Contrastive frameworks
SEN
Simplicial embeddinge
ENS
Ensemble self-supervised learning
InstDist
Instance discrimination
MRCL
Masked reconstruction contrastive learning
NPID
Nou-parames con-parametric instance discrimination
Temperature schedules
Contrastive predictive coding
ARCL
Augmentatioon-oous contrastive learning
DIM
Deep InfoMax
MoaRep
Mossic representation learning framework
Augmented multi-scale
CNC
Contrastive multi-view coding
GAN-based frameworks
MoCo
Momentum contrast
PIRL
Pretext-invariat representation learning
SinCLa
A simple framework for contrastive learning Prototypical contrastive learning
Ridiroctional
ALI
Adversarially learned inference
PIG
Parametric instance classification Dehiased contrastive learning
BigBiGAN
BIGAN with BigGAN generator
SS-GAN
sule-sue GAN
LooC
Lesve-one-out contrastive learning
S8-GAM-LA
SS-CAN with label angmentation
D-SimCLR
Self-supervised contrastive learning with
VQGAN
Vector-quentized GAN
guided projection
Vit-VQGAN
VIT ViT- based VQGAN
ReLIC
Representation learning via
invariant
mechanisms
MIM-based frameworks
Addo
Adversarial contrast
DenaeCL,
Dense contrastive learning
PixPro
Pixel-level consistency propagation
1GPT
Image GPT
CLSA
Contrastive learning with
Bidirectional encoder representation from
stronger augmentations
image transformers
INCLR
Nearest-neighbor contrastive learning
MAE
Masked autoencoders
representations
180T
Image BENT pre-training with online tokenizer
NoBY
MoCo + BYOL
SimMIX
simple framework for MIM
DMC
Divide contrast
PaCD
Perceptual codcbook
ReSSL
Relational self-supervised learning
MaskFeat
Masked feature prediction
UniGrad
unified gradient framework
Context sutoencoder
SinCo
Simplifier MoCa without
CIN
Corrupted image modeling
BinMooCo
Simplified MoCo
MCNAE
Masked convolution meets MAE
unified framework selE-supervised
CoaMIX
Denoising contrast masked image modeling
visual pre-training,
CKAE
Contrastive MAE
Muge
Kulti-gramular self-eupervised learning
Self-distillated masked autoeneoder
CaCo
cooperative-advenaae contrastive learning
MILAK
Masked image pretraining on language
SNoG
Synchromous momentum grouping
representation
SianeseIM
Slamese image modelling
CAN
Contrastive learning, masked autoencuders, and noise prediction
PCAE
Progressively compressed autoencoder
Distillation frameworks
SparK
Sparse masked modeling
MRNAE
Mimic before reconstruct
BYOL
Build your-own latent
SinSian
Simple Siamese representation learning networlss
Others
0BoW
Online bag-of-visual-word.
DirectPred
Direct linear predictor
SEED
Self-supervised distillation visual representation
BERT
Bidirectional encoder representations from transformers Generative pro-trained transformer
DisCo
Distilled contrastive learning
GPT
DIHO
Self-distillation with no labels
EsVit
Efficient self-supervised vision transformer
BINGO
Bag of instances aggregation
TinyMIN
Tiny
Information- inoommttion-mxximiatiiin frameworks
WHSE
Whitening squared
VicReg
Variance-invariant coveooo
regularization
Twin class distribution estimation
TLDR
Twin learning for dimensionality reduction
Align representations
base
VicRegl.
VicReg with local visual features
21

---

Published in Transactions on Machine Learning Research (05/2023)
B Metadata for frameworks
SSL,
Primary
Publication date
Expariments on on ImogeNot 1K
Downetream caperimenta
Official implemonnaatin
Trained
Trumcwork
atfliation
woddle
Blaster
Pacsbook Al Research
Mar 2019
Yea Yes
Yea
Available
Avatlable
Looal Aggregatião
Siinaood University
Apr 2010
Yes
Dosper Clussse
Facebook Al Research
Ang 2019
Yes
Yoe
Avatlaae Avarlable
Available
Sela
University of Oxford
Nov 2019
Yoa
Yea
Avathabi
SCAN
KU Leuven
Jul 2020
Yoa
No
Deap Chasterm22
Facebook Al Research
Jun 2020
You
Yes
deca-12
Pazebook Research
Jun
Sonr
Pansbook Research
2022
Yes
AVETAFI
Sime
Treettine (MP
Ch Chinese Usiv. of Hong Kong
May 2018
Yes
Ync
Available
Avallable
CPC
Desphind
Jul 2018
Yea
No
Ner AVATIE
Not avamia
DJM
Mierosoft Research
Aug 2018
No
No
Available
Noi available
070-12
DeepMied
May 2010
Yes
Yea
Nut mes bloble
INDT
Microsoft Research
Jun 2010
Yes
No
Availablle
Avallable
AVatlaPie
Fanebaok Research
Google
Passbaok
BLaiUR-12
Google Researela
Saleeforce Research
2020
PIC
Microuoft Research
Jun 2620
Yea
Yea
Avarlable
Avatiaiie
DCL
MIT
Jul 2020
Yea
LouG
UC Berlcley
Aus 2020
Yo
Yes
G-StecL.
Walmart Labe
Sep 2020
No
No.
Avaclable
Avatlable
SeLIC
Desphind
Occ 2020
Yea
Yoa
Peking University
Nov 2020
Ye5
Yes
Availabile
Avaliable
Densect.
The University Adelaide
Nov 2020
Yos
Microroft Reearrch
Nov 5030
Yoe
KoComeS
Fasebook Resourch
Apr
Avadlable
CL.b
Purduc University
Apz
Triplet
Univensity
Google Remaarch
Microsoft Research
nic
DeepMind
May 9021
Yee
Swddn
Senze
Jul 2021
No
Available
Avsllable
Senze Time
Dec 2021
Yea
No
hol30-/2
DeepMind
Jan 2022
Yos
Yea
Not mvazzinle
Slade
KAIST
Mar 2022
No
Available
Noi available
Stetoco
KAIST
Mar 2022
No
No
Avarlabde
Not asssllible
Un171P
University of Chinase AoS
Mar 2022
Yoa
Yea
Nar assallale
Tugs
Sea Al Lab
Mar 2022
Yoa
Yea
Available
Amsilable
Caco
Purdun University
2022
2NcB
2022
Shoughai Artificial Intelligence Lab.
Not available
available
2001
Deephind
Jan 2620
Yos
Yes
Avadlable
Avatlable
SiwBlax
Facebook Al Research
Aug 2020
Yes
Yoe
Avatien
Valeo.al
Dec 2020
Yos
Yoe
Microsoft Ressarch
Jan 2021
Yes
Yoe
NWt AosiTii
DaseciPzed
Facebook Al Rescurch
Feb 2021
You
Yea
Availablee
Noi avsilabd bde
Dus00
Tenccat
Apz 2021
Yos
Yea
Amstinbl
2000
Pacebook Research
Apr
Microsoft Research
B1102
Huawei
AN PAA
TinyMIN
Microeoft
WIBE Tartow Tutne
University of Trenio
Jul 2020
Yo
No
Availall bole
Nui Losa Didde
Pacabook Al Ressarch
Mar 2021
Yes
YAd
Vicilig
Pacebook Al Research
May 2021
Yos
Yoa
TN1IT
Tainghus Univeresty
Oct 2021
Yos
Yoa
TLOK
(Navor Labs RU
Oet 2021
Yes
ABB
Shanghai Jiso Tong Universlty
Now
No
Vichogl
Fasebook Research
Oct
Yeu
Available
Available
Table 17: Publication information as well as implementation details for discriminative SSL frameworks covered in this survey,
22

---

Published in Transactions on Machine Learning Research (05/2023)
SSL
Primary affiliation
Publication date
Experiments on ImageNet
Downstream experiments
Official
Trained
framework
implementation
models
InfoNin
MFT
May 2020
Yes
Yes
Available
Available
InterCLR
Nanyang Technological Univ.
Aug 2020
Yes
Yes
Not available
Not. available
hEXa
Microsoft Research
Dec 2020
Yes
Yes
Not. available
Not: available
Nocifi
Naver Labs EU
Oct
Yes
Yes
Not available
Available
ReSin
UC Berkeley
Mar 2021
Yes
Yes
Available
Available
MBF
University Maryland
May 2021
Available
Available
ORL
Nanyang Technological Univ.
Jun 2021
Yes
Yes
Available
Available
CEB
Google Research MILA
Sep 2021
Yes
Yes
Available
Available
SBM
Apr 2022
Yrs
Yes
Not. available
EX3
Google Research
Nov 2022
Yes
Yes
Not. avanlable
Not: available
University of Chinese AnS
Nov
Yes
Not: available
Not available
T8
Max Planek Institute
Mar 2023
No
No
Available
Available
ARCL
Shanghai Jiso Tong Univ.
Mar 2023
Not available
Not availbble
MosRep
University of Sydney
Mar 2023
Yes
Yes
Available
Available
Table 18: Publication information well as implementation details for enhancements proposed existing SSL frameworks covered in this survey.
SSL framework
Primary
Publication date
Experiments on ImageNet IK
Downstreum experiments
Official
Trained
affiliation
implementation
models
BiGAN
University of California
May 2016
Yes Yes
Yes
Aveilable
Available ble
B1gB1GAN
DeepMind
Yes
Ala
Avaala
ALI
MILA
Jun 2016
No
Yes
Ala
Available
93-GAN
University California
Nov 2018
Yes
Yes
Available
Not availab
S8-GAN-LA
University of Chinese AuS
Oct 2021
No
Yes
Available
Availuble
Vit-VQCAM
Gongle Research
Oct 2021
Yes
Yes
Not available
Not availanle
1GPT
OpenAl
Jul 2020
Yes
Yes
Available
Available
BEIT
Microsoft Research
Jun 2021
Yea
Yes
Availa ble
Availa
MAE
Facebook Al Research
Nov 2021
Yes
Yes
Avaiiu ble
Avaria ble
1B0T
ByteDance
Nov 2021
Yes
Yes
Avaslable
Available
SinMIN
Microsoft Research
Nov 2021
Yes
Yes
Available
Available
PeC0
Microsoft Research
Nov 2021
Yes
Yes
Not available
Not available
MaskFeat
Facebook Research
Yes
Yes
Available
Available
data2vac
Facebook Al Research
Feb 2022
Yes
No
Avaallable
Avaalable
CAE
Peking University
Feb 2022
Yes
Yes
Not available
Not available
CIN
Microsoft Research
Feb 2022
Yes
Yes
Nut available
Nut available
MOMAR
SenseTime
May 2022
Yes
Yes
Available
Available
ConMIN
Tencent
May 2022
Yes
Yes
Availabbe
CHAE
ByteDance
Jul 2022
Yes
Yes
DE available
oo available
SdAE
Huawei
Jul
Yes
Yes
Available
Available
MILAN
Alibaha
Aug 2022
Yes
Yes
Avia bic
Av bic
BBiT-r2
Microsoft Research
Aug 2022
Yes
Yes
Avaala ble
Avaalable
BEiT-73
Microsoft Research
Aug 2022
Yos
Yes
Avania
Avaiia
CAE-v2
Baidu
Nov 2022
Yes
Yes
Not available
Not available
CAN
Google Research
Jan 2023
Yes
Yes
Available
Available
PCAE
Huawei
Jan 2023
Yes
Yes
Avaiable
Avarabbee
Spark
ByneDance
2023
Yes
Yes
MRMAR
Shanghai AT Laboratory
Mar 2023
Yos
Yes
Avaniaa
Table 19: Publication information well implementation details for generative SSL frameworks covered
in this survey.
Repository name
Maintainer
Purpose
Awesome SST
Independent.
comprehensive reading list for SSI.
Independent da Costa et al
2022
SSL frameworks, benchmarking, and model zoo SSL frameworke, benchmarking, and model 2oo SSL frameworks, benchmarking, and model zoo SSL frameworks benchmarking
VISSL
Facebook Gooya
MINSS SSup
OpenMMLa5 ab ( Contributors
2021
Lightiv
Lightly.ai Susmet
EV
Alibaba
Contributors
SSL frameworks and benchmarking
Unittd SSL Benchmark
Microsoff (Wang ef al.
20226
SSL frameworks and benchmarking
Table 20: Github repositorics related to SSL, their maintainer, and purposc.
23

---

Published in Transactions on Machine Learning Research (05/2023)
C
Framework dataset usage
8SL,
Used
framework
Deep Cleater
ImageNat-ikk Pascal VOC. Places, YFCC100M
C, D,
Losal Apgeegation
ImageNet-22. Pascul VOC
benger niter
ImageNat-1k: Parcal VOC, Placee
Gata ImageNat-1k; CIPAR-100, CIPAΠ-10, Pascal VOC, 8VHN ImageNat-1k, CIPAR-100, CIPAR-10, STL-10
Deep
InnggNett1kk
ImagoNet-1k
Places
ImaggeNet1kk Pascaal
Placea
D C.D.S
InoogoNet-1k COCO, VOC
Sale- Clanetfier
ImageNat-ik, COCO, Pascal VDC
C,
DootDiot (MP10)
Ne.geNet-1kk Placco ImageNat-ik
Thay ImageNatt CIFAR-100, CIFAR-10, STL-10, CelebA
ImageNet-1k,
ImageNet-1k, C1PAR-100, CIRAB-10,
ST1-10
ImageNat-1k, STI.10
COCO, Pascal VOC
ImogoNet-3k. Pascal VOC,
iNat
ImageNat-ik, CIPAR-100, CIFAR-10, Pascal VOC, Foad, Birdsnap SUN39T, Cuza, Aireratt, DTD, Pets, Caltech-101, Flower
SeacLa
C,
moggeNat-1k. Pascal VOC
C.
maggnaat-1k, C1PAR-10
pc. e Potur
ImageNat-ik. Pascal VOC. Places
C.
P3C
ImageNat-ik. Pascal VOC. Cityscaper, iNati8
C. D,
ImageNet-100. CIPAR-10, STL-10 30L
GrBtacLa
Laoc
ImageNet aubsct, CIFAR-10
ImageNat- INas-1K, CUBB-200 Flowere-102
00000
IntgeNet-lk. LetggeNet-R. ImageNNt.C
AdD
Ien COCO, VOC, Pascal
Puscal
Placos
69
ImageNet-3k, CIPAR-100 CIPAR-10, Oxford Flowers-102 Oxford-IlI--oo
ImageNet-1k; C0CO, Pascal VDC
Trancated
ImageNat-1k, C0CO, Pascal VDC, SYSU-30k
0,
Ill CIFAR-100, 100, CIFAR-10, Puscul VOC, Food, Birdsnup SUN197, Cazs. Aitron, DTD. Pots, Callech-101, Flower
C,
Nour
ImageNat-ik, CO0O, ADE20K ImageNat-ikk CIPAR-100, CIFAR-10, COCO, Paseal VOC, Places, Food Birdanap, SUN397, Cars, Airerut, DTD. Pets, Caltech-101, Flower, NYU x2 IaageNat-1k, R000 CIRAIL-10, ST1-10
C, D,
30C
C, D.
Untorad
ImageNat-1k
10
InngooettSleidh.
ImngeNet-CC, IntgeNet-R InngeNet-1k. V2
Cityscapea
C, D.
ImageNet-100, C1PAR-100,
STL-10,
ImageNet-lk. COCO
ImageNat-1k, 6O00
Cato
ImageNat-3k, CIPAR-1100 CIFAn-11, Pascal VOC, Road SUN39T. Cars, Asreraft, DTD. Peta, Caltech-101, Flower
C.
gl00
InageNeet11k, COCO, Pascal VOC, Cityscapes
D.
ImageNet-1k, COCO, ADE20X
C, D.
Riedenap,
ImageNet-1k,
C, D.D,
ImageNat-1k, Pascal Places
Bbvi
ImogoNet-1k, CIPAR-10. STL-10
mum
ImageNat-ik, CIPAR-10, STL-10. COCO, Pascal VOC
C, D.
Di.00
ImageNat-Ik. CIPAR-100, CIFAR-10, COCO, Paresi VOC
C, D.
ImageNat-ik, CIFAR-180, CIFAR-10, Puscal VOC, iNai18, iNat19 Plawers, Cars, (Net, Google Landmarke v2, DAVI8 2017 videos
DINO
0, D.
Eatst
ImagsNet-Ik. CIFAR--10, CIFAN--0, COCO, Pazes) VOC, STL-10, MINST, Food SUN397, Cars, Aiteou,, DTD, Pets, 1011 Flower, PER2013, GTSRB, HotefulMemea, PatchConndbon, UCF101
D,
REDO0
ImogeNet, CIPAR-100, CIPAR-10, CO0O ImageNet-1k, ADB20K
C, D.
TIM
0,
ImageNat-1k, ImageNets 100, Thny ImageNet, CIFAR-100, CIFAR-10, STL-10 ImageNat-1k, 0000, Pawci--VDC Places, INac18
Tartou
D, 8 C, D.
Viclog
MaageNat-1k.CCOO Pascal VOC, Plase, iNatls
SUN197, Cars, Airezaft, DTD, Pets, Caltech-101, Flower
lt-l C1PAR-100, CIFAR-10, COCO, Pascal VOC, Food
0, D,
TID
ImageNat-ik
ImageNet-ikk ImmgeNett-10 100, CIFAR-100, CIPAR-10
ImageNet-13 1k, Paecal VOG, Cityarages
Table 21: Employed datasets for experiments used in the research articles of the discriminative SSL
frameworks. The third column summarizes the evaluated tasks in the respective papers of frameworks: [C)sssiifcaatio, (D)etection/locaal localiiiiioo,iiiiiiioooooo and (S)egmentation.
24

---

Published in Transactions on Machine Learning Research (05/2023)
SST.
Used
framework
datasets
InfoMin
ImageNet- 1k, COCO, Pascal VOC; Colorful Moving-MNIST
C, D. S
InterCLa
ImageNet-1k, Pascal VOC, Places
НEXA
ImageNet-1k, CIFAR-100. CIFAR-10, Pascal VOC
C,
Nocili
ImageNet-1k, ImageNet-100, COCO, Pascal VOC
C, D, $
ReSin
ImageNet-1k, ImageNet-100. COCO, Pascal VOC
ImageNet-11k, CIFAR-100, CIFAR-10 10, Pascal VOC, Food, SUN397, Cars
MSF
Aircraft, DTTD, Pots, Caltoch-101, Flower
C,
DRL
ImageNet-1k, COCO, Pascal VOC, Places, INat ImageNet-1k, ImageNet-A, ImageNet-CC ImageNet-R ImageNet-v2, ImageNet-Vir, YouTube-BB, ObjectNet ImageNot-1k, ImageNet-A, ImageNet-CC, ImageNet-Ri
C, D
CBB
SEN
ImageNet-v2, CIFAR-100, CIFAR-10, Food, Sun397, DTD, Flower ImageNect-1k, CIFAR-100, CIPAR-10, 10, Food, SUN397, Cars
ENS
DTD; Pets, Caltech-101, Flower
C
MRCL
ImageNet-1k, COCO, ADE20k ImageNet-100, CIFAR-100, CIFAR-10
C, D
T3
ImageNet-1k, CIFAR-100) CIFAR-10, Food, SUN397
Cars, Aircraft, DTD, Pots, Caltech-101, Flower
ImageNet-1k, ImageNet-100. CIFAR-100, CIFAR-10, COCO Fond, Cars, DTD, Pets, Caltech-101, Flower
NosRap
C, D,
Table 22: Employed datasets for experiments used in the research articles of the enhancements to discrim- inative SSL frameworks. The third column summarizes the evaluated tasks in the respective papers of frameworks: (C)lassification, (D)etection/ loocllizaiiaaiiinnn,. and (S)egmentation.
SST.
Tasks
framework
datasets
BigBiGAN
ImageNet-1k
C
B1GAN
ImageNet-1k, Pascal VOC, MNIST
C, D,
Tiny ImageNet, CIFAR-10,
CelebA
88-GAN
ImageNet-Ik, 1k, CIFAR- 10, CelebA-HQ, LSUN-Bedrrom
88-GAN-LA
Tlay- ImageNet, CIFAR-10, STL-10, CelebA
Vit-VOGAN
ImageNet-1k, CelebA-HQ, FFHQ
1GPT
ImageNet-1k, C1FAR-100, CIFAR-10, STL-10
BE1T
ImageNet-1k, ADE20K
C, s
MΛE
ImageNet-1k: COCO, Places, INat17, INat18, INsc19
10:00 C, D.
1B0T
ImageNet-Ik. CIFAR-100. CIFAR-I0, COCO, ADE20k, iNat18, iNat19, Flower, Cars
G. D.
C. D.
SimMIN
ImageNet-1ke 1k, COCO, ADE20K, INat18
PeC0
ImageNet- COCO, ADE20k
C. D.
YaskFeat
ImageNet-Ik, Kinetica-100, Kinetics-600, Kineticc-700
ImageNet-Ik
CAE
ImageNet-1k, COCO, ADE20K
C. D,
ImageNet-11 COCO,
C.
NCMAE
ImageNet-Ik, COCO, ADE20K
C. D, &
ConMIN
ImageNet-1k, COCO, ADE20K
C. D. s
CMAE
ImageNet-Ik, COCO, ADE20K
C. &
SdAE
ImageNet-lk, COCO, ADE20K
C. D.
ImageNet-lk, COCO, ADE20K
C.
BEiT-v2
ImageNet-1k, ADE20K
C. S
BE1T-V3
ImageNet-Ik, COCO, ImageNet-1k, COCO, ADE20K
C, D.
MRCL
C. D,
CAE-v2
ImageNet- COCO, ADE20K
C, D.
ImageNet-1k. ImageNet-v2. ImagoNet-Real, ImageNettAddveearilll Netaeeeet-enndliion
CIFÁR-100, Birds, Cars, DTD, Pets, UC-Merced, Col-Hist, Caltech, ObjectNet
ImageNet-1k, COCO
D,
Spark
ImageNet- 1k, COCO
C. D.
MRMAE
ImageNet- COC0
C,
Table 23: Employed datasets for experiments used in the research articles of the generative SSL frameworks. The third column summarizes the evaluated tasks in the respective papers of frameworks: (C)lassificaaton, (e)etection/localiiaee and (S)egmentation.
25

---

Published in Transactions on Machine Learning Research (05/2023)
D ImageNet benchmarks
SSL
Backbone network
SSL
Fine-tuning accuracy
Linear probing accuracy
Additional
framework
epochs
notes
Deep Cluster
AlexNet
Used conv4 output
LA
ResNet-50
200
60.2
Deeper Cluater
VGG-16
100
48.4
SaLA
ResNet-50
90
61.5
SCAN
ResNet-50
90
39.9
Unsupervised evaluation
Deep Cluster-v2
ResNet-50
400
71.3
2x160 1x96 crops
Deep Cluster-v2
ResNet-50
800
71.9
SeLA-v2
ResNet-50
400
71.8
2x160 1x96 crops
Swav
RosNet-50
800
77.8
75.3
ResNet-50
57.6
CoKe
ResNet-50
800
76.4
views
Self-C.
ResNet-50
800
74.1
Table 24: ImageNet-Ik 1k linear probing and fine-tuning benchmarks for clustering-based SSL frameworks.
26

---

Published in Transactions on Machine Learning Research (05/2023)
Backbone network
SSL epochs
Fine-tuning accuracy
Linear probing accuracy
Additional
framework
notes
InstDist (NPID)
ResNet-50
200
54.0
Using conv5
ReeNet-v2
101
130
CPCV2
ResNet-50
200
61.8
AMDIM
ResNet-50
150
68.1
Large AMDIN model
CNC
ResNet-50
200
66.2
Randengmnnt
MoCo
ResNet-50
200
77.3
60.6
HoCo
ResNet50 (x2)
200
65.4
MoCo
ResNet50 (x4)
200
68.6
ResNet-50
SiaCLR
ResNet-50
100
63.6
S1xCLR
ResNet-50 (x2)
100
74.2
SimCLR
ResNet-50 (x4)
100
76.5
McCo-v2
ResNet-50
800
75.5
71.1
SimCLR-v2
ResNet-50
400
76.3
71.7
SinCLR-V2
ResNet-50 (x2)
400
79.1
75.6
SinCLR-V2
ResNet-101
100
78.2
73.6
SimC1R-v2
ResNet-101 (x2)
400
80.7
77.0
PCL
ReaNet-50
200
61.5
POL-v2
ResNet-50
200
67.6
ResNet-50
ReLIC
ResNet-50
800
74.8
AdCo
ReaNet-50
800
75.7
Multi-crop
AdCo
HesNet-50
PixPro
ResNet-50
100
66.3
with SinCLa
McCo-v3
ResNet-50
800
73.8
McCo-v3
ViT-B
83.2
McCo-v3
VIT-I.
300
84.1
77.6
McCo-v3
VIT-H
300
78.1
C1.SA
ResNet-50
200
73.3
Multi-crop
CLSA
ResNet-50
800
76.2
Multi-crop
Triplet
ResNet-50
XNCLA
ResNet-50
1000
75.6
Multi-crop
MoBY
DefT-S
300
72.8
MoBY
Swin-TT
DMC
ResNet-50
3000
78.2
75.8
Re88t
ReaNet-50
200
74.7
crops
UniGrad
ResNet-50
800
75.5
with CutMix + multi-crop
ReLIC-v2
ResNet-50
1000
77.1
UniVIP
ResNet-50
300
71.2
Muga
ViT-S
3200
82.6
78.9
Huga
VIT-B
1600
84.3
80.6
84.3
Muga
VIT-L
1000
82.1
CaCo
ResNet-50
200
75.3
SMoG
ResNet-50
400
78.3
76.4
Multi-crop
SiameseII
VIT-B
1600
84.1
78.0
Table 25: ImageNet-Ik 1k linear probing and fine-tuning benchmarks for contrastive-learning.. based SSL franeworks.
27

---

Published in Transactions on Machine Learning Research (05/2023)
SSL
Backbone network
SSL
Fine-tuning accuaracy
Linear probing accuracy
Additional
framework
epochs
notes
BYOL
ResNet-50
BYOL
ResNet-50 (x2)
1000
77.4
BYOL
ResNet-50 (x4)
1000
BYOL
ResNet-200 (x2)
1000
79.6
SimSiam
ResNet-50
DBeW
ResNet-50
DirectPred
ResNet-50
300
72.4
SEED
ResNet-34
800
58.5
Distilled from ResNet-50
DisCo
ResNet-34
800
62.5
Distilled from ResNet-50
DING
ResNet-50
300
DINO
VIT-S
300
81.5
77.0
VTT-B
EsVIT
Swin-T
300
78.
EsViT
Swin-S
300
79.5
EaVET
Swin-B
300
80.4
BINGO
ResNet-34
200
66.1
Distilled from ResNet-50
TinyMIM TinyMIM
VIT-S
300
83.0
Distilled from Vit-B
VTT-B
300
85.0
Distilled from Vit-L
Table 26: ImageNet-1k linear probing and fine-tuning benchmarks for distillation-base SSL frameworks.
Backbone network
Fine-tuning accurary
Linear probing accuracy
Additional notes
framework
epochs
WABE WABE
ResNet-50
100
69.4
4, corresponding 6 positive pairs
ResNet-55
400
72.5
4, corresponding to 6 positive pairs
Barlow Toins
ResNet-50
1000
73.2
VicReg
ResNet-50
1000
73.2
ResNet-50
800
75.5
Multi-crop
TWIST
VIT-B/16
300
82.8
78.4
TLDR
VIT-S/16
100
74.8
ARB
ResNot-50
100
68.2
VicRogl VicRegl
ConvNext-S
150
75.9
ConvNext-B
150
77.1
Table 27: ImageNet-1k linear probing and fine-tuning benchmarks for information-maatiiizatoo.....
SSL frameworks.
28

---

Published in Transactions on Machine Learning Research (05/2023)
SSL
Backbone network
Fine-tuning accuracy
Linear probing accuracy
Additional
framework
opochs
notes
Infolin
ResNet-50
800
73.0
InterCIR
ResNet-50 NPID-v2 HesNet-50 + BYOL
InterCLR
1000
74.5
ВEXА
ResNet-50 MoCo-v2
800
75.7
71.7
НEXA
ResNet-50 + DeepCluster.v2
800
78.3
75.5
8-crops
MocHii
ResNet-50 + MoCo-v2
1000
70.6
NSF
ResNet-50 BYOL-asym
200
72.4
Weak /strong variation
KSF
ResNet--50 BYOL-asym
200
66.3
Weak/ weak variation
ORL
ResNet-50 + BYOL
800
60.7
Pre-train on COCO+
CEB
ResNet-50 SimCLR
1000
71.6
CEB
ResNet-50 + BYOL
1000
75.6
CEB
ResNet-50 (2x) SimCLR
1000
75.0
CEB
ResNet-50 (2x) + BYOL
1000
78.8
SEM
ResNet-50 + BYOL
200
74.1
ENS
VTT-B/16 + DINO
400
79.1
ENS
VIT-B/16 MSN
400
78.9
VIT-B/8 DINO
300
$1.0
ENS
VIT-B/8 + MSN
300
80.8
MRCL
VIT-B SimCLR VIT-B + BarTwins
600
80.0
NRCL
600
80.4
ARCL
ReaNet-50 + MoCo
900
70.9
3 views
MosRep
ReaNet-50 MoCo-v2
200
72.3
NoeRep
ResNet-50 + BYOL
200
76.2
Table 28: ImageNet-Il Ik linear probing and fine-tuning benchmarks for enhancements to discriminative SSL frameworks.
SSL framework
Backbone network
SSL epochs
Fine-tuning accuracy
Linear probing accuracy
Additional
notes
BiGAN
100
56.2
Bi BigBiGAM
ResNet-50
800
76.3
55.4
B1 B1AN
ResNet-50 (x4)
800
76.6
60.8
ResNet
Vit-VOGAN
VIT-B VTT-L
100
65.1
Vit-VQGAN
100
73.2
Table 29: ImageNet-Il 1k linear probing and fine-tuning benchmarks for GAN- SSL frameworks.
29

---

Published in Transactions on Machine Learning Research (05/2023)
SSL
Backbone
SSL epochs
Fine tuning accuracy
Linear probing accuracy
Additional notes
framework
network
1GPT
GPT-I
100
72.6
66.2
1GPT
GPT-XL
68.7
GPT-XI.
100
72.0
Concatenation five layers
VIT-B
BEIT
VIT-I
300
85.2
73.5
BEIT
VIT-H
300
85.1
MAR
VIT-B
1600
83.6
68.0
VIT-L
1600
85.9
76.8
MAE
VIT-H
1600
87.8
76.6
1BUT
VIT-S
3200
82.3
77.9
180T
VIT-B
1600
84.0
79.5
VIT-L
1000
84.8
81.0
VIT-B
56.7
PeCD
VIT-B
800
84.5
PeCD
VIT-L
800
86.5
PeCD
VIT-H
800
88.3
MaskPeat
VIT-L
1600
84.0
67.7
date2vec
VIT-B
800
84.2
data2vec
VIT-L
1600
86.6
CAE
VIT-S
300
82.0
51.8
VIT-B
70.4
CAE
VIT-L
1600
86.3
78.
CIN
VIT-S
300
81.6
CIN
VIT-B
300
83.3
ResNet-50
epochs
ИСНАВ
ConVIT-B
1600
85.0
70.9
CouMIN
VIT-S
800
83.9
384 x 384 images
VII-B
384 384 images
CocMIN
VIT-L
1600
86.5
384 384 imagess
CMAE
CONVIT-B
1600
85.3
73.9
VIT-B
300
84.1
64.9
VIT-B
79.9
MILAN
VIT-L
400
87.8
84.3
BE1T-v2
VIT-B
300
85.0
80.1
BELT-v2
VIT-L
1600
87.3
BE1T-v3
BB
N/A
89.6
Uses IN-21k
CAB-v2
VIT-S
300
83.1
77.5
CAB-v2
VIT-B
300
85.3
80.6
VIT-L
81.7
CAN
VIT-B
1600
6
74.8
CAN
VIT-L
80U
84.7
76.2
VIT-S
300
81.9
PCAE
VIT-B
300
83.6
VTT-B
Spark
ConvX-S
1600
84.1
Spark
ConvX-B
1600
84.8
54.7
ConViT-B
85.8
Table 30: ImageNet-1k 1k linear probing and fine-tuning benchmarks for MIM-based generative SSL frame- works. "SSL epochs" denotes the number of epochs for the SSL training.
30

---

Published in Transactions on Machine Learning Research (05/2023)
E
COCO benchmarks
SSL
Backbone
AT performance
Apm
Additional
framework
network
performancr
notes
Sway
ResNet-50
41.6
37.8
ResNet-50
37.2
Self-Clesaifier
ResNet-50
41.5
36.1
InstDiat
ResNet-50
37.4
34.1
MoCo
ResNet-50
40.9
35.5
ResNet-50
34.0
SInCLR
ResNet-50
39.6
34.6
MoCo-V2
ResNet-50
39.8
DenseCL
ResNet-50
39.6
35.7
PixPro
ResNet-50
41.4
40.5
CLSA
ResNet-50
42.3
24.4
NoCo-v3
VIT-B
45.5
40.5
Truncated Triplet
ResNet-50
NoBY
Swin-T
48.1
41.7
UniVIP
ResNet-50
42.2
38.2
Muge
ViT-S
49.8
43.0
SNoG
ResNet-50
40.1
36.9
SianeseIM
VIT-B
02.1
46.2
SimSian
ResNet-50
SEED
ResNet-18
35.3
31.1
Distilled from ResNet-50
Dia
ResNet-34
40.0
34.9
Distilled from ResNet-50
DIND
VIT-B
46.8
41.5
EsVit
Swin-S
46.2
41.6
BINGO
ResNet-18
Barlow Twins
ResNet-50
40.0
36.7
VicReg
ResNet-50
39.4
36.4
InfeMin
ResNet-50
40.4
38.8
Noclii
ResNet-50 + MoCo-v2
39.4
31.5
ReSin
ResNet-50 MoCo
41.9
37.9
ORL
ResNet-50 + BYOL
40.6
36.7
Pre-trains COCO+
XBCL
VIT-B BarlowTTwins
53.3
46.6
MRCL.
ViT-B SimCLR
53.7
46.9
MoaRep
ResNet-50 + BYOL
41.1
37.2
MosRep
ResNet-50 + Moco-v2
36.6
data2vec
VIT-B
37.0
MAE
VIT-B
48.4
12.6
180T
VIT-S/16 16
49.4
42.6
SinMIM
VIT-B/16
43.6
PeCO
VIT-B
43.9
39.8
VIT-B
44.0
MOMAE
ConViT-B
53.2
47.1
ComMIM
VIT-B/16
43.6
CNAE
VIT-B
52.9
47.0
SdAE
VIT-B
48.9
43.0
MILAN
VIT-B
45.5
MILAN
VIT-L
55.9
48.2
BEiT-v3
extra Ohject336
MRCL
SimCLR
53.7
46.9
CAE-V2
VIT-B
52.0
11.9
PCAE
VIT-B
48.8
43.1
Spark
ConvX-B
51.2
45.1
MRMAE
ConViT-B
Table 31: Downstream transferability results on COCO dataset.
31

---

Published in Transactions on Machine Learning Research (05/2023)
References
Salch Albelwi. Survey self-supervised learning: auxiliary pretext tasks and contrastive learning methods in imaging. Entropy, 24(4):551, 2022.
Elad Amrani and Alex Bronstein. Self-supervised classification network. arxiv preprint arXiv:2103.10994 2021.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint orXiv:1111.05371, 2019.
Mido Assran, Randall Balestriero, Quentin Duval, Florian Bordes, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, and Nicolas Ballas. The hidden uniform cluster prior in self-supervised learning. In The Fleventh International Conference on Learning Regresenlations, 2023. URL httpps:/ het/foormm2dt=0OA3PMMMGG
Philip Bachman. Going meta: learning algorithms and the sclf-supervised machine. In Microsoft AT Podcast, 2019. URL https://www. youtube. wwt. hoom/watch??err cctm:/watchhv==3333333......
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning represenlations by maximizing mutual information across views. Advances in Neural Information Processing Systems, 32, 2019.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pp. 1298-1312. PMLR, 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: Bert pre-training of image transformers. arXiv preprint arXiv:2106. 08254,, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variane-iinarrancc-ccraaaaa regularization for self-supervised learning. arXiv preprint arXiv:2105. 04906, 2021.
Adrion Bardes, Joan Ponce, and Yann LeCun. VICRegL: Sclf-supervised lcarning of local visual features.
arXiv preprint xrX::2220000157/ 2022.
Gulcin Baykal and Gozde Unal. Deshulllega: A self-supervised gan to improve structure learning. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 708-712. IEFE, 2020.
Gulcin Baykal, Furkan Ozcelik, and Gozde Unal. Exploring deshufflegans in self-supervised generative adversarial networks. Pattern Recognition, 122:108244, 2022.
Suzanna Becker and Geoffrey Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nalure, 355(6356):161-16;) 1992.
Yoshua. Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798 1828, 2013.
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Pro- ceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 417-424, 2000.
Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea. Vedaldi, and Philip HS Torr. Fully-convolttional siamese networks for object tracking. In European Conference on Computer Vision pp. 850-865. Springer, 2016.
Christopher M. Bishop. Pattern Recogration and Machine Learning. Springer, 2006.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. Xio preprint arx::8809. 11096, 2018.
32

---

Published in Transactions on Machine Learning Research (05/2023)
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a "siamese" time delay neural network. Advances in Newral Information Processing Systems, 6, 1993.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Newral Information Processing Systems, 33:1877 1851877-190, 2020.
Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu. Parametric instance classification for unsupervised visual feature learning. Advances in Neural Information Processing Systems, 33:15614-15624, 2020.
Fabio M Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2229 2238, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132 149, 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. In Proceedings of the IEEE/CVF Internalional Conference on Compuler Vision, pp. 2959-2968, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9999 -924,99244 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CV International Conference on Computer Vision, pp. 9650-9660, 2021.
M Emre Celebi and Kemal Aydin. Unsupervised learning algorithms. Springer, 2016.
Souradip Chakraborty, Aritra Roy Gosthipaty, and Sayak Paul. G-SimCLR: Self-supervised contrastive learning with guided projection via pseudo labelling. In 2020 International Conference on Data Mining Workshops (ICDMW), pp. 912-916 IEEE, 2020.
Guillaume Charpiat, Matthias Hofmann, and Bernhard Schölkopf. Automatic image colorization via multi- modal predictions. In European Conference on Computer Vision, pp. 126 139. Springer, 2008.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. Xix preprint ar arXin:1 704.00051, 2017.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and 1lya Sutskever. Gen- crative pretraining from pixels. In International conference on machine learning, pp. 1691-1703. PMLR, 2020a.
Qingzhong Chen, Shilun Cai, Crystal Cai, Zefang Yu, Dahong Qian, and Suncheng Xiang. Colo- scrl: Self-supervised contrastive representation learning for colonoscopic vidco retricval. arXio preprint arXiv:2303.156% 2023a.
Ting Chen. Advancing self-supe and semi-superriised learning with SimCLR. In Google Al Blog, 2020.
URL httppp//a....oooooooocc/c//////a
and-semi. html
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised GANs via auxiliary rotation loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12154-12163, 2019.
33

---

Published in Transactions on Machine Learning Research (05/2023)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self- supervised models are strong semi-supervised learners. Advances in Neural Information Processing Sys- tems, 33:22243- -22255, 2020c.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervieed representation learning. arXiv preprint arXie:2202. 03026, 2022a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 18750-157550 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arxiv preprint arXiv:2003.4430 2020d.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9640 9649, 2021.
Yabo Chen, Yuchen Liu, Dongsheng Jiang, Xiaopeng Zhang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. In Computer Vision ECCV 2022: 17th European Conference, Tel Avio, Israel, October 23-27, 2022, Proceedings, Part XXX, pp. 108-124. Springer, 2022b.
Yubei Chen, Adrien Bardes, Zengyi Li, and Yann LeCun. Intra-instance VICRog: Bag of self-supervised image patch embedding explains the performance. 2023b. URL http://ppperreveeeeteeeeeeeeeeee
J923GzIz8Sh
Ziwei Chen, Qiang Li, Xiaofeng Wang, and Wankou Yang. LiftedCL: Lifting contrastive learning for human- centric perception. In The Eleventh International Conference on Learning Representations, 2023c.
Zezhou Cheng, Qingxiong Yang, and Bin Sheng, Deep colorization. In Proceedings of the IEEE international conference on computer vision, pp. 415-423, 2015.
Davide Chicco. Siamese neural networks: An overview. Artificial Newral Networks, pp. 73 94, 2021.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with applica- tion to face verification. In 2005 IEFE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR (05,, volume 1, pD. 539 546. IEEE, 2005.
Ching- Yao Chuang, Joshua Robinson, Yen-Chen Lin, Autonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. Advances in Neural Information Processing Systems, 33:8765 8775, 2020.
Djork-Arué Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast And accurate deep network learning by exponential lincar units (Elus). arXiv preprint arXiv:1511.07289, 2015.
Adam Coates and Andrew Y Ng. Learning feature representations with k-means. In Neural Networks: Tricks of the Trade, pp. 561 580. Springer, 2012.
Adam Coates, Andrew Ng, and Honglak Lec. An analysis of lingle-layer networks in unsupervised feature learning. In Proceedings of the fourleenth international conference on artificial inlelligence and slatislics, pp. JMLR Workshop and Conference Proceedings, 2011.
EasyCV Contributors. EasyCV.
2022.
MMSelfSup Contributors. MMSelfSup: OpenMMLab self-supervised learning toolbox and benchmark. http://hhh...ooo/ooooommmma/aaaa
2021.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems, 26, 2013.
34

---

Published in Transactions on Machine Learning Research (05/2023)
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of sell-supervised methods for visual representation learning. Journal of Machine Learning Research, 23 (56):1 6, 2022. URL http://jmlr http:///jil..og//pppppaeaa 2ttpa//rro////pppe///////1111110 1155.html
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0448 2018.
Carl Docrsch, Abhinav Gupta, and Alexci A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on compuler vision, pp. 1422-142200 1430, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advonces in neural information processing systems, 32, 2019.
Jeff Donahue, Philipp Krähenbühh, and Trevor Darrell. Adversarial feature learning. ar preprint arXiv:160500978, 2016.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pro-training of vision transformers. orXiv preprint wrXiv:2111.12710 2021.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsu- pervised feature learning with convolutional neural networks. Adoances in Neural Information Processing Systems, 27, 2014.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXio:2010. 1 11929, 2020.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv:1606. FTf:::606000000,, 2016.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual reprosentations. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9588 9597, 2021.
Alyosha Efros. The gelato bet. 2019. URL https://peoppee.eess berkeley wwt.rieellyy.e//eefoo////////l.e bet html
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised
representation learning. In International Conference on Machine Learning, pp. 3015-3024. PMLR, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883 2021.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pasce network.org/ www.orkkorg/challlnggge. Soieeee//////aae //tp////C////////oaaaaaaa
2007.
Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self- supervised visual pre-training. In The Eleventh International Conference on Leurning Representations, 2023. URL /tttpsss//penrevieeeeeeeeeeeee
aett/ForumrKKo--OOVVVseeeee
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervisel distillation for visual representation. arXie preprint Miii::2011.0731., 2021.
Abe Fotterman and Josh Albrecht. Understanding sclf-supervised and contrastive learning with "bootstrap your own latent" (BYOL). In Generally Intelligent AI Blog, 2020. /ttp////gg////orrraaaiaaaeegggg az/blog/2020-000 08- 2118-24-unddreeee http://www..aaaaaaaaaaaaaaaaoana
35

---

Published in Transactions on Machine Learning Research (05/2023)
Peng Gao, Teli Ma, Hongsheng Li, Jifeng Dai, and Yu Qiao. ConvMAE: Masked convolution meets masked autoencoders. arViv preprinl or Xiv:2200 Xix:2205.03892, 2022.
Peng Gao, Renrui Zhang, Hongyang Li, Hongsheng Li, and Yu Qiao. Mimic before reconstruct: En-
hance masked autoencoders with feature mimicking, 2023. URL http:///ppeneeieee.eeeffoooooooe
UoBJm4V21md
Yuting Gao, Jia-Xin Zhuang, Ke Li, Hao Cheng, Xiaowei Guo, Feiyue Huang, Rongrong Ji, and Xing Sun. Disco: Remedy self-supervised learning lightweight models with distilled contrastive learning. anXiv preprint arXiv:2104.00 09124, 2021.
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. arXiv preprint wrXiv:2210.02856 2022a.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning. arXiv preprint or arXiv:2006.0 02577, 2022b.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rolations. arxiv preprint wrXiv:1803.07725 2018.
Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Online bag-of-visuallwordd generation for unsupervised representation learning. orXiv preprint or arxi::2012. 221122215552 11552, 2021.
Amupriya Gogna and Angshul Majumdar. Semi supervised autocncoder. In International Conference Newral Information Processing, pp. 82 89. Springer, 2016.
Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11); 139 144, 2020.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengehao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint ar arXi::200 00.10103.9989 01988, 2021a.
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Rois, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. VISSL. https:
//ithub. comm//aaccboccrreerrc///sss
2021b.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent new approach to self-supervised learning. Advances in Newral Information Processing Systems, 33:21271-2128. 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnor- malized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Procoodings, 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR 06). volume 2, pp. 1735-1742. IEEE, 2006.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems, 30, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognation, 2016.
36

---

Published in Transactions on Machine Learning Research (05/2023)
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IERE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9729 9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 16000-16009, 2022.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Confer- ence on Machine Learning, pp. 4182 4192. PMLR, 2020.
Dan Hendrycks, Mantas Mazcika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainly. Advonces in Newral Information Processing Systems, 32, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in newral infor- mation processing systems, 30, 2017.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge in neural network. arXiv preprinl arXiv:1503.0955 2(7), 2015.
Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Newral computation, 18(7):1527 1554, 2006.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildonn Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 180: 06670, 2018
Liang Hou, Huawei Shen, Qi Cao, and Xueqi Cheng. Self-supervised GANs with label augmentation. Ad- vances in Neural Information Processing Systems, 34:13019 13031, 2021.
Zejiang Hou, Fei Sun, Yen-Kuang Chen, Yuan Xie, and Yun-Yuan Kung. Milan: Masked image pretraining on language assisted representation. arXiv preprint arXiv:2208..66 2022.
Jeremy Howard. Self-supervised learning and computer vision. In fast.ai Blog, 2020. URL https://www fast. 2ttt.aa///20//1////ees5 /ts///00000/1///eeeessuurrrra///
Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition, pp. 1074-1083, 2021.
Tianyang Hu, Zhili LIU, Fengwei Zhou, Wenjia Wang, and Weiran Huang. Your contrastive learning is secretly doing stochastic neighbor embedding. In The Eleventh International Conference on Learning Representations, 2023. URL httpps://peerrevien
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On leature decorrelation in self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9598-999960 9608, 2021.
Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming- Ming Cheng, Dongmei Fu, Xiaohui Shen, and Ji- ashi Feng. Contrastive masked autoencoders are stronger vision learners. arXiv preprinl or orXim:2207.135.. 135332. 2022.
Ka Yu Hui. Direct modeling of complex invariances for visual object features. In International Conference on Machine Learning, pp. 352 360. PMLR, 2013.
ICLR. ICLR 2023 Reviews for: BETT v2: Masked Image Modeling with Vector- Vector-Quante Quantized Visual Tokenizers. 2023. Accessed: 2023-04-04.
37

---

Published in Transactions on Machine Learning Research (05/2023)
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Transactions on Graphics (TbG), 35(4):1-11 11, 2016.
Sergey Toffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. CoRR, abs/ 1bs/1502.03167, 2015.
Saachi Jain, Hadi Salman, Alaa Khaddaj, Eric Wong, Sung Min Park, and Aleksander Madry. A data-based perspective on transfer learning. arXiv preprint arXiv:2207.0739, 2022.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprinl arxiv:21 10. 09348, 2021.
Daniel D. Johnson, Ayoub El Hanchi, and Chris J. Maddison. Contrastive learning can find an optimal basis for approximately view-invariant functions. In The Fleventh International Conference on Learning Representations, 2023. URL https::/
nwt htt///ooumnnddddode iMu
Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In European Conference on Computer Vision, pp. 67-84. Springer, 2016.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Newral Information Processing Systems, 33:21798 21809, 2020.
Yannis Kalantidis, Carlos Lassance, Jon Almazan, and Diane Larlus. TLDR: Twin learning for dimensionality reduction. arXiv preprint ar arXim:2100.08 09455, 2021.
Angjoo Kanazawa, David W Jacobs, and Manmohan Chandraker. Warpnet: Weakly supervised matching for single-view reconstruction. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3253-3261, 2016.
Adnan Khan, Sarah AlBarri, and Muhammad Arslan Manzoor. Contrastive self-supervised learning: a survey different architectures. In 2022 2nd International Conference on Artificial Intelligence (ICAI), pp. 1-6. TEFE, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/ abs//a//11/0 1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Mean shift for self-supervised lcarning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10326- 10335, 2021.
Alex Krizhevsky and GeofTrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012.
Anna Kukleva, Moritz Böhle, Bernt Schiele, Hilde Kuehne, and Christian Rupprecht. Temperature schedules for self-supervised contrastive mothods on long-tail data. arXiv preprint arxi::2009. 19664, 2023.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic col- orization. In European Conference Computer Vision, pp. 577 593. Springer, 2016.
Gustav Larsson, Michael Maire, and Gregory Shakhmarovich. Colorization as a proxy task for visual under- standing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 6874-6883, 2017.
38

---

Published in Transactions on Machine Learning Research (05/2023)
Samuel Lavoie, Christos Tsirigotis, Max Schwarzer, Ankit Vani, Michael Noukhovitch, Kenji Kawaguchi, and Aaron Courville. Simplicial embeddings in sell-supervised learning and downstream classification. ar preprint arXiv:2204.00616 2022.
Yann LeCun. Predictive learning. In NIPS 2016, 2016.
URL http:///ww..yotbibbeccc//aaaaaa
0unt2Y4qx0okt=111505
Yann LeCun. I now call it "sel--sueerviee learning". 2019. URL http:p//ww..facbbboo.cco/mmmmmm
posta/101553340422111
Yann LeCun. Self-supervised learning. In Proceedings of the Fhurth-Fourth AAAT Conference on Artificial In-
telligence, AAAL, Invited Tolk, 2020. URL https:://rrve..goorr 100g
w17Z207700zBB//veen
Yann LeCun and Ishan Misra. Self-supervised
learning: telligence. In Facebook Al Blog, 2020. URL lttps://ai.facebbo.. combblog...oooo// blog/ selffsuppervisedleerre Learniinn-dernninn---e the-thh--a-- dark miakk--ttee--eee of of-intelligenne/
The dark matter of in-
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 1998.
Kuang-Huei Lee, Amurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. Advances in Neural Information Processing Systems, 34:19538- 2021.
Alexander C Li, Alexei A Efros, and Deepak Pathak. Understanding collapse in non-contrastive siamese representation learning. In European Conference Computer Vision, pp. 490-505. Springer, 2022a.
Chunyuan Li, Xiujun Li, Lei Zhang, Baolin Peng, Mingyuan Zhou, and Jianfeng Gao. Sclf-supervised pre-training with hard examples improves visual representations. arXiv preprint arxiv:2012.133 13493, 2020a.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. ar preprint arXiw:2166 0670978. 2021a.
Jin Li, Yaoming Wang, Xiaopeng Zhang, Yabo Chen, Dongsheng Jiang, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Qi Tian. Progressively compressed auto-encoder for self-supervised representation learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevier:
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervisod representations arXiv preprinl rxXin:205504466 2020b.
Ru Li, Shuaicheng Liu, Guangfu Wang, Guanghui Liu, and Bing Zeng. JigsawGAN: Auxiliary learning for solving jigsaw puzzles with generative adversarial networks. IEEE Transactions on Image Processing, 31: 513 524, 2021b.
Zhaowen Li, Yousong Zhn, Fan Yang, Wei Li, Chaoyang Zhao, Yingying Chen, Zhiyang Chen, Jiahao Xie, Liwei Wu, Rui Zhao, et al. Univip: unified framework for self-supervised visual pre-training. Tn Proceedings of the IEEE/CK Conference on Computer Vision and Pattern Recognition, pp. 14627 14636, 2022b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft Coco: Common Objects In Context. In Proceedings of the IEEE European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976 17776-198666 11986, 2022a.
39

---

Published in Transactions on Machine Learning Research (05/2023)
Ziwen Liu, Bonan Li, Congying Han, Tiande Guo, and Xuecheng Nic. Masked reconstruction contrastive learning with information bottleneck principle. arViv preprint AXN:22211.00015 2022b.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. are preprint arXiu:1711.05101, 2017.
Qing Luan, Fang Wen, Daniel Cohen-Or, Lin Liang, Ying-Qing Xu, and Heung-Yeni Shum. Natural image colorization. In Proceedings of the 18th Eurographics conference on Rendering Techniques, pp. 309-320, 2007.
Mario Lučié, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High- fidelity image generation with fewer labels. In International conference on machine learning, pp. 4183 4192. PMLR, 2019.
Shlok Mishra, Anshul Shah, Ankan Bansal, Abhyuday Jagannatha, Abhishek Sharma, David Jacobs, and Dilip Krishnan. Object-aware cropping for self-supervised learning. arXiv preprint orXiv:2112.00319, 2021.
Shlok Kumar Mishra, Joshua David Robinson, Huiwen Chang, David Jacobs, Weicheng Kuo, Aaron Sarna, Aaron Maschinot, and Dilip Krishnan. CAN: A simple, efficient and scalable contrastive masked autoen- coder framework for learning visual representations, 2023. URL http://oppenrviee.eeteeoeoooee
BBV_09H
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. Tn Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6717, 2020.
Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell. Representation learning via invariant causal mechanisms. arXiv preprinl or 2rXi:22010.07922, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69 84. Springer, 2016.
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. In Proceedings of the IEEE international conference on computer vision, pp. 1898-5906 2017.
David Novolny, Samuel Albanie, Diane Larlus, and Andrea Vedaldi. Self-supervised learning of geometri- cally stable features through probabilistic introspection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3637 3645, 2018
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXi::1807.0317 2018.
Bo Pang, Yifan Zhang, Yaoyi Li, Jia Cai, and Cewu Lu. Unsupervised visual representation learning by synchronous momentum grouping. In Computer Vision-ECCV 2022: 17th European Conference, Tel Avie, Israel, October 23-27, 2022, Proceedings, Part XXX, pp. 265 282. Springer, 2022.
Kaiyue Pang, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song. Solving mixed-modal jigsaw puzzle for fine-grained sketch-based image retrieval. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 10347-10355, 2020.
Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. What do self-supervised vision transformers learn? In The Eleventh International Conference on Learning Representations, 2023.
Decpak Pathak, Philipp Krahenbuhl, Jeff Donahuc, Trevor Darrell, and Alexci Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 2536-2544, 2544, 2016.
40

---

Published in Transactions on Machine Learning Research (05/2023)
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BETT v2: Masked image modeling with vector-quantized visual tokenizers. arx preprint or Nin:2208.06366 2022.
Qi Qian, Yuanhong Xu, Juhua Hu, Hao Li, and Rong Jin. Unsupervised visual representation learning by online constrained k-means. In Proceedings of the IREE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16640 16649, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Nya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramosh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748- 8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821 8831. PMLR, 2021.
Sanat Ramesh, Vinkle Srivastav, Deepak Alapatt, Tong Yu, Aditya Murali, Luca Sestini, Chinedu Innocent Nwoye, Idris Hamoud, Antoine Fleurentin, Georgios Exarchakis, et al. Dissecting self-supervised learning methods for surgical computer vision. arXiv preprint or xrXi:22077.04449 2022.
Sucheng Ren, Fangyun Wei, Zheng Zhang, and Han Hu. TinyMIM: An empirical study of distilling MIM pre-trained models. arXiv preprint axXtv:2301.01996, 2023.
Zhongzheng Ren and Yong Jae Tee. Cross-domain self-supervised multi-task feature learning using synthetic imagery. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 762 771, 2018.
Pierre H Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, el al. BYOL works even without batch statistics. arXiv preprint Xrc::22000. 102/1, 2020.
Joshua Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04590 04592, 2020.
Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. Convolutional neural network architecture for geometric matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6148-6157, 2017.
Yangjun Ruan, Saurabh Singh, Warren Morningstar, Alexander A Alemi, Sergey loffe, Ian Fischer, and Joshna V Dillon. Weighted ensemble self-supervised learning. arXiv preprint ar Xrx:::2211.0 2EY::2211.0998,, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-2555 2015.
Anshul Shah, Aniket Roy, Ketul Shah, Shlok Kumar Mishra, David Jacobs, Anoop Cherian, and Rama Chellappa. HaLP: Hallucinating latent positives for skeleton-based self-supervisod learning of actions. arXiv preprint Xiv: 000387,0077 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in Neural Information Processing Systems, 29, 2016.
Kihyuk Sohn and Honglak Lee. Learning invariant representations with local transformations. arXiv preprint 12X66:1206.61 6418, 2012.
41

---

Published in Transactions on Machine Learning Research (05/2023)
Igor Susmelj, Matthias Heller, Philipp Wirth, Jeremy Prescott, and Malte Ebner et al. Lightly. GitHub. Note: http///llab..oom//ggheee oi/lightly, 2020.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE/CVI Conference on Computer Vision and Pattern Recognition, pp. 9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, 2016.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International Conference on Artificial Neural Networks, pp. 270 279. Springer, 2018.
Chenxin Tao, Honghui Wang, Xizhou Zhu, Jiahua Dong, Shiji Song, Gao Huang, and Jifeng Dai. Exploring the equivalence of siamese self-supervised learning via a unified gradient framework. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 14431-14440, 2022a.
Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning. arXiv preprint OrXii:2206.01200 2022b.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in Neurol Information Processing Systems, 30, 2017.
Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, and Zehuan Yuan. Designing bert for convolu- tional networks: Sparse and hierarchical masked modeling. arxiv preprint arXiv:2301.cm 035580, 2023.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, In Europeon Conference on Computer Vision, pp. 776-794. Springer, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-6839, 2020b.
Yonglong Tian, Olivier J Henaff, and Aäron van den Oord. Divide and contrast: Self-supervised learning from uncurated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 10063 10074, 2021a.
Yuandong Tian. Understanding the role of nonlincarity in training dynamics of contrastive learning. In The Eleventh International Conference on Learning Represenlations, 2023. URL httpa:///penneeriee.nnss net/ forum?id=s13orTEEN X.
Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. orXiv preprint arXiv:2010.000 00578,, 2020c.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervieed learning dynamics without contrastive pairs. In International Conference Machine Learning, pp. 10268 10278. PMLR, 2021b.
Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Bucsing, Razvan Pascanu, Charles Blundell and Jovana Mitrovic. Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet? arXiv preprint 2ix::220.00111111 2022.
Trieu H Trinh, Minh-Thang Luong, and Quoc V Selfie: Self-supervised pretraining for image embedding. arXiv preprint ar 1exxi:9900 02040,2 2019.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Procsmans, and Luc Van Gool. Scan: Learning classify images without labels. In European Conference on Computer Vision, pp. 268 285. Springer, 2020.
42

---

Published in Transactions on Machine Learning Research (05/2023)
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoin Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference Machine Learning, pp. 1096 1103, 2008.
Feng Wang, Tao Kong, Rufeng Zhang, Huaping Liu, and Hang Li. Self-supervised learning by estimating twin class distributions. arXiv preprint arXiv:211000744 2021a.
Guangrun Wang, Keze Wang, Guangeong Wang, Philip HS Torr, and Liang Lin. Solving inefficiency of self-supervised representation learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9505-9515, 2021b.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit. Som, al. Image foreign language: BFiT pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022a.
Xiao Wang and Guo-Jun Qi. Contrastive learning with stronger augmentations. Pullern Analysis and Machine Intelligence, 2022.
IEEE Transactions
Xiao Wang, Yuhang Huang, Dan Zeng, and Guo-Jun Qi. Caco: Both positive and negative samples are di- rectly learnable via cooperative-adversariaa contrastive learning. arXiv preprint arXiv:2203 14370, 2022b.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self- supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 3024 3033, 2021c.
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Wei Ye, Marios Savvides, Bhik- sha Raj, Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, and Yue Zhang, USB: A unified semi-supervised learning benchmark for classification. In Thirty-sirth Conference on Neural Informa- tion Processing Systems Dalasets and Benchmarks Track, 2022c. doi: 11.8550/axxi..220.000r https://arxin org/ Beigg/bb//2000 2208 07204
URL
Zhaoqing Wang, Qiang Li, Guoxin Zhang, Pengfei Wan, Wen Zheng, Nannan Wang, Mingming Gong, and Tongliang Liu. Exploring set similarity for dense self-supervised representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16590-16599, 2022d.
Zhaoqing Wang, Ziyu Chen, Yaqian Li, Yandong Guo, Jun Yu, Mingming Gong, and Tongliang Liu. Mosaic representation learning for sulf-supervisod visual pro-training. In The Eleventh International Conference on Learning Representations, 2023.
Chen Wei, Haoqi Fan, Saining Xie, Chao Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVT Conference Computer Vision and Pattern Recognition, pp. 14668 14678, 2022.
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision. arXiv preprint arXie:2006. N00::2006.03677, 2020.
Zhirong Wu, Alexei A Efros, and Stella X Yu. Improving generalization via scalable neighborhood component analysis. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 685-701, 2018a.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the TEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018b.
Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive contrastive learning. arXio preprint 2r00::008.. 2005:2000.05650, 2020.
43

---

Published in Transactions on Machine Learning Research (05/2023)
Tete Xiao, Colorado J Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity repre- sentation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10539 10548, 2021.
Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Unsupervised object-level representation learning from scene images. Advances in Neural Information Processing Systems, 34:28864- 28876, 2021a.
Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Delving into inter-image invariance for unsupervised visual represenlations. International Journal of Compuler Vision, pp. 1 20, 2022a.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self-supervised learning with swin transformers. arxiv preprint ar arXiv:210500558, 2021b.
Zhenda Xie, Yulong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16684-16693, 2021c.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognalion, pp. 9053995539690 2022b.
Haohang Xu, Jiemin Fang, Xiaopeng Zhang, Lingxi Xie, Xinggang Wang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Bag of instances aggregation boosts self-supervised distillation. In International Conference Learning Representations, 2021.
Linli Xu, James Neufeld, Bryce Larson, and Dale Schuurmans. Maximum margin clustering. Advances in Newral Information Processing Systems, 17, 2004.
Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. High-resolution image inpainting using multi-scale neural patch synthosis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6721-6729, 2017.
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE/CVT Conference on Compuler Vision and Pallern Recognition, pp. 5147-5156; 2016.
Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. arXio preprint or arXin:2205.006 09616, 2022.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 4471-4480, 2019.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantizod image modeling with improved VQGAN. arXiv preprint ar Xrvi::2110 10.00000000 10.04627, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023 6032, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310 12320. PMLR, 2021.
Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 6688 6697, 2020.
44

---

Published in Transactions on Machine Learning Research (05/2023)
Chaoning Zhang, Kang Zhang, Trung X Pham, Axi Niu, Zhinan Qiao, Chang D Yoo, and In So Kweon Dual temperature helps contrastive learning without many negative samples: Towards understanding and simplifying MoCo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14441-14450, 2022a.
Liheng Zhang, Guo-Jun Qi, Liqiang Wang, and Jiebo Luo. AET vs. AED: Unsupervised representation learning by aulo-encoding translormations rather than data. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 2547-22552 2555, 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Conference Computer Vision, pp. 649-666. Springer, 2016.
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prodiction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognilion, pp. 1058 1067, 2017.
Shaofeng Zhang, Lyn Qiu, Feng Zhu, Junchi Yan, Hengrui Zhang, Rui Zhao, Hongyang Li, and Xiaokang Yang. Align representations with base: new approach to self-supervised learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognation, pp. 16600-16609, 2022b.
Tong Zhang, Congpei Qiu, Wei Ke, Sabine Süsstrunk, and Mathieu Salzmann. Leverage your local and global representations: A new self-supervised learning strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16580-16589, 2022c.
Xinyu Zhang, Jiahui Chen, Junkun Yuan, Qiang Chen, Jian Wang, Xiaodi Wang, Shumin Han, Xiaokang Chen, Jimin Pi, Kun Yao, et al. CAE v2: Context autoencoder with CLIP target. arXiv preprint arXiv:2211.0979 1.7999 2022d.
Xuyang Zhao, Tianqi Du, Yisen Wang, Jun Yao, and Weiran Huang. ArCL: Enhancing contrastive learning with sugmentattion-robustt representations. arXiv preprint ar Xr:::22303.000 01092,, 2023.
Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. ReSSL: Relational supersupe learning with weak augmentation. Advances Neural Information Processing Systems, 34:2543 2555, 2021.
Yuanyi Zhong, Haoran Tang, Junkun Chen, Jian Peng, and Yu-Xiong Wang. Is self-supervised contrastive learning more robust. than supervised learning? In First Workshop on Pre-Iraining: Perspecläves, Pilfalls, and Paths Forward ICML 2022, 2022.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBOT: Image BERT pro-training with online tokenizer. arXiv preprint arXi::2111.07888 2021.
Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: A multi- granular self-supervised learning framework. arXiv preprint arXiv:2203.14415, 2022.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6002 6012, 2019.
Andrew Zisserman. Self-supervised learning. 2018. URL 07/21sserman-sell supervised.
45


Reviews:
[{"review_id": "FD8s3YkE78", "reviewer": "Reviewer_xZKh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. This survey is well organized. Each section is easy to follow.\n2. This submission gives a good overview of the recent discriminative self-supervised learning on images.\n\nWeakness\n1. One large weakness is that they focus on discriminative self-supervised learning and do not pay much attention to recent methods, such as Masked Autoencoder (MAE). Such approaches are shown to perform well on this task, they need more discussion to clarify the difference from other approaches. Considering that survey in this field already exists, the contribution from this paper can be marginal.\n2. Overall, this survey is a bit superficial in that they provide only the list of methods and do not show any comparison in the performance. It is hard to know what approaches are promising for this task. Also, each approach can show strengths and weaknesses depending on the training dataset or downstream task, yet, this paper does not discuss much of it. Although their focus may be to give an overview of this field, I am unsure whether such a survey can be above the bar of acceptance.\n3. They lack a discussion on what kind of dataset is used for training and evaluation and the impact of such factors. SSL methods should rely a lot on the dataset they are trained on. ImageNet1K is probably standard in this task, yet increasing the dataset size should further improve the performance, which they do not discuss much.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper surveys image-based discriminative unsupervised learning. This survey aims to cover image-oriented frameworks for discriminative SSL, some of which may not be covered by the previous survey due to the fast progress in this field. They briefly explain the SSL pretext task, a list of clustering approaches, contrastive learning approaches, distillation, and information maximization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Including MAE papers for more discussion is important. Also, see weaknesses for other changes. ", "broader_impact_concerns": "No concern. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "k0bNmPxd5y", "reviewer": "Reviewer_UR1h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) They have a comprehensive survey of image-based instance discrimination methods capturing both classic and state-of-the-art techniques. \n\nWeakness:\n1) The main area of research which is missing is Masked-Autoencoders which has come out recently and has shown great effectiveness.\n2) The second area of research is pre-training on multi-object images COCO[1] , OpenImages[2]. Citing all the relevant works in these areas will be important. \n3) Methods that have dealt with scaling SSL tasks should also be discussed in the survey.\n4) Masked image modelling and it’s related papers should be discussed as well. \n\nReferences: \n[1] Dense-CL CVPR 2021 https://arxiv.org/abs/2011.09157 .\n[2] Object-Aware Cropping for Self-Supervised Learning TMLR 2022 https://openreview.net/pdf?id=WXgJN7A69g .\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary: This paper discusses how supervised learning has improved computer vision for images, but the rate of improvement has slowed down. In contrast, self-supervised learning (SSL) has seen great success in natural language processing (NLP) in recent years. This has led to the adoption of SSL methods, such as clustering, contrastive learning, distillation, and information-maximization, in the area of computer vision. The paper reviews research on image-based SSL, covering best practices, software packages, pretext tasks, and techniques commonly used in discriminative SSL. The paper also outlines relevant research directions for those interested in contributing to image-focused SSL.\n\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Adding papers in MAE,  MIM and pre-training on multi-object datasets will be most critical to the survey paper.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "635jsb0ZL6", "reviewer": "Reviewer_M91J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n– The paper is very well-written and easy to follow\n\n– The categorization presented is intuitive and fairly comprehensive\n\n– Altogether, the paper is a good one-stop resource for those seeking a summary of recent progress in discriminative SSL on images \n\nWeaknesses \n\n– The paper conceptually explains the different SSL frameworks well but does not provide sufficient formalism, apart from describing the InfoNCE loss, despite introducing considerable notation. I think adding equations describing the generalized loss objective of each framework would help ground many of the concepts that are described much better, even if they may not apply to every method listed within that framework.\n\n– The paper focuses provides a good conceptual comparison of recent SSL approaches but does not provide an adequate head-to-head comparison of their performance, beyond including a table of results on ImageNet. I think additionally compiling results on (atleast a subset of) transfer performance of different approaches, performance v/s model capacity, training time, compute requirements etc. in the appendix would strengthen the paper. Additionally, it would be good to include atleast a brief description of the main takeaways in the main paper, ideally so a reader might be able to judge which method might be suitable for their application of interest.\n\n– Adding \"pseudocode\" for the different SSL frameworks would be a very nice addition.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a survey of research in image-based discriminative self-supervised learning (SSL). It first provides some historical context and technical background, and then summarizes prior work falling under one of the following SSL learning frameworks: clustering, distillation, contrastive, and information-maximization. It also includes additional resources including links to code implementations and ImageNet benchmark numbers (in appendix).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses above. I would consider addressing the first two weaknesses as crucial to my recommending acceptance, while addressing the third would strengthen the work but is not crucial.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RTiz4TzlVq", "reviewer": "Reviewer_euRx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n## Strengths\n- The important concepts related to SSL are nicely illustrated in Figure 3, with description in Section 3.\n- It is good to see the one sentence summary of each of the 50+ SSL techniques. \n\n## Weaknesses\n- The topic of discriminative SSL is a bit outdated, due to the fast research pace nowadays. We may use Table 8 as an evidence (the authors summarize the publication information as well as implementation details for the SSL frameworks covered in this survey), the number of papers in 2022 is significantly smaller than 2020 and 2021. In Figure 1, though SSL is shown an increasingly popular topic for the all areas according to Google Trends does not necessarily mean a similar trend for SSL in computer vision, not to mention the discriminative model family in SSL in computer vision. For the information of authors, the community seems more interested in generative SSL paradigm like MAE & BEIT, and vision-language pre-training for general and transferable visual representation learning.\n\n- There are not much new insights and discoveries in this survey, including the future directions section. Though I appreciate the hard work to compile the list and provide the summary, I am afraid that the survey would encourage much more interests in discriminative SSL. The current form of this survey can fit the format of a GitHub readme to maintain the update-to-date papers on this topic. \n\n- Some related papers on discriminative SSL are missing. One simple way to add more related papers is to check out the leaderboard \"Self-Supervised Image Classification on ImageNet\" on the paper-with-code website: https://paperswithcode.com/sota/self-supervised-image-classification-on\nAs a survey paper, please at lease add all the methods from this leaderboard, which is crowdsourced by the most related community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey paper reviews a number of general-purpose frameworks for image-based self-supervised learning (SSL) that fall into the category of discriminative training. This includes clustering, contrastive learning, distillation, and information-maximization. 50+ SSL techniques are covered.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "My major concern is how to attract more attention for this survey (in another word, the topic of discriminative SSL), based on the up-to-date literature and the community interests. Some suggestions: \n\n- Try to make more concrete connections with the current community interests, such as (1) generative methods for representation learning & content creation, (2) the possible integration with vision-language pre-training, which is arguably the most empirically effective in visual representation learning for recognition problems.\n\n- Think about why discriminative SSL is not so popular. One reason could be the evaluation metrics. Sticking with the so-called standard metric on \"linear probing on ImageNet-1K\" does not really present the full power of discriminative SSL methods. Other tasks in computer vision that are aligned with the pre-training objectives could be better downstream evaluation scenarios.", "broader_impact_concerns": "Probably add broad discussions with other related topics, and their shared and unique impact concerns, such as compute cost.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The survey does an excellent job of covering a large amount of literature, without being overly long or verbose. It introduces the core concepts, and provides a concise overview of this broad topic. The paper is well-organized, and uses tables effectively to provide a useful reference/overview highlighting the "unique properties" of the different papers.

There are important topics that are not covered by the survey, such as the role of different pre-training datasets, and transfer of learned features to diverse downstream benchmarks. However, in order to not make the survey too long, I think the current scope is appropriate.

Overall, I recommend acceptance, congratulations to the authors!
Decision: Accept (poster)