Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning
Authors: Florian Bordes
Randall Balestriero
Quentin Garrido
Adrien Bardes
Pascal Vincent
Venue: Accepted by TMLR ()

Abstract: One unexpected technique that emerged in recent years consists in training a Deep Network (DN) with a Self-Supervised Learning (SSL) method, and using this network on downstream tasks but with its last few layers entirely removed. This usually skimmed-over trick of throwing away the entire projector is actually critical for SSL methods to display competitive performances. For example, on ImageNet classification, more than 30 points of percentage can be gained that way. This is a little vexing, as one would hope that the network layer at which invariance is explicitly enforced by the SSL criterion during training (the last layer) should be the one to use for best generalization performance downstream. But it seems not to be, and this study sheds some light on why.
This trick, which we name Guillotine Regularization (GR), is in fact a generically applicable method that has been used to improve generalization performance in transfer learning scenarios. In this work, we identify the underlying reasons behind its success and challenge the preconceived idea that we should through away the entire projector in SSL. In fact, the optimal layer to use might change significantly depending on the training setup, the data or the downstream task. Lastly, we give some insights on how to reduce the need for a projector in SSL by aligning the pretext SSL task and the downstream task.

Published in Transactions on Machine Learning Research (05/2023)
Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning
Florian Bordes
Soovian.bondes@@moontteee
Meta AI Reseorch
Mila, Universilé Montréol
Randall Balcstriero
Meta AI Research
Quentin Garrido
Meta AI Research
Université Gustave Biffel,
CNRS, LIGM
Adrien Bardes
Meta AI Research
Inria
Pascal Vincent
Meta Research
Mila, Université de Montréal
CIPAR
Reviewed on OpenReview: https: // openreview. net/ forum? id= 2gX
Abstract
One unexpected technique that emerged in recent years consists in training a Deep Network (DN) with Self-Supervieed Learning (SSL. method, and using this network on downstream tasks but with its last few projector layers entirely removed. This trick of throwing away the projector is actually critical for SSL methods to display competitive performances on ImageNet for which more than 30 percentage points can be gained that way. This is a little vexing, as one would hope that the network layer at which invariance is explicitly enforced by the SSL criterion during training (the last projector layer) should be the one to use for best generalization performance downstream. But it seems not to be, and this study sheds some light on why. This trick, which we name Guillotine Regularization (GR), is fact a generically applicable method that has been used improve generalization performance in transfer learning scenarios. In this work, identify the underlying reasons behind its success and show that the optimal layer to might change significantly depending the training setup, the data or the downstream task. Lastly, we give some insights on how to reduce the need for projector in SSL by aligning the pretext SSL task and the downstream task.
1 Introduction
Many recent. self-supervised learning (SSL) methods consist in learning invariances to specific chosen relations
between samples implemented through data-augun iitaa-aagmenttaiions while using a regularization strategy to avoid
collapse of the representations (Chen al. 2020a Grill et al. 2020 Lee et al. 2021 Caron al. 2020:
Zbontar al. 2021 Bardes al. 2022
Tomasev al.) 2022
Caron al. 2021 Chen et al. 2021
al.
2022 Zhou et al. 2022a Incidentally SSL learning frameworks also heavily rely on simple trick

---

Published in Transactions on Machine Learning Research (05/2023)
Projector head trick used in SSL
Training on pretext task
Downstream task
ImageNot Top-1 Accuracy
70
Layors !....
Trunk
Representation
Layers E-1
Head
Layer I.... f
Trunk
Repressent rntpresentttoon
new shallow hear
Loss
S807
mistioo
H
50
SSL: Barlow Twins
SSL: VICReg
H,
H
SSL: SimCLR
FROZEN
30
40
SSL. Byol
Supervised
Head
Trunk
Head
Head
Hoad
Layer
(b)
Figure 1: a) An illustration of projector head trick used in SSL. During training, small neural network named the Head (also coined projector in the SSL literature (Chen et al. 2020a)) is added top of another deep network referced as the Trunk. This Head can be viewed as a buffer between the training loss and the Trunk that can absorb any bias related to a ill optimisation. When using such network on downstream tasks, we throw away the Head. b) We measure with linear probes the accuracy at different layers on a Resnet50 (as Trunk) (see Figure 10 for vision transformers) on which we added a small 3-layer MLP (as Head) for various supervised and sclf-supervised methods. For each method, we show the mean and standard deviation across 3 runs (The std between different runs is low). With traditional supervised learning, there is significant drop in performances when using the trunk layer instead of the last projector layer. However, when looking at self-supervised methods, the gap in performances between the linear probe trained at the trunk and projector can be as high as 30%.
to improve downstream task performances: removing the last few layers of the trained deep network depicted in Figure la From a practical viewpoint, this technique emerged naturally (Chen et al. 2020a) through the
search of ever increasing SSL performances. In fact, on ImageNet (Deng et al. 2009 such tochnique can improve classification performances by around 30 points of percentage (Figure lbb.
Although it improves performances in practice, not using the layer which the SSL training was applied is unfortunate. It means throwing away the representation that was explicitly trained to be invariant to the chosen set of data augmentations, thus breaking the implied promise of using a more structured, controlled, invariant represontation. By picking instcad a representation that was produced an arbitrary number of layers above, SSL practitioners end up relying on a representation that likely contains much more information about the input (Bordes et al., 2021 than should be necessary to robustly solve downstream tasks.
Although the use of this tochnique emerged independently in SSL, using intermediate layers of a neural network-instr of the deepest layer where the initial training criterion was applied- has long been known to be useful in transfer learning scenarios YYosinski et al. 2014). Features in upstream layers often appear more general and transferable to various downstream tasks than the ones at the deepest layers which are too specialized towards the initial training objective. This strongly suggests a related explanation for its success SSL: does removing the last layers of a trained SST, model improve performances because of misalignment between the SSL training task (source domain) and downstream task (target domain)
In this paper, we examine that question thoroughly. We first place the SSL trick of removing the projector post-training under the umbrella of generically applicable method that call Guillotine Regularization. We argue that it is important to distinguish the action of removing layers during evaluation from architecturo modifications because the optimal layer to use for a given downstream task is not always the backbone and could be intermediate projector's 's layers. Then, we explore how changes in the training optimization, training data and downstream task impact the optimal layer in both supervised and self-supervised sctting. Lastly, we demonstrate that increasing the *alignment* * between the pretext and downstream task in SSL decreases the need to use projector in SSL.
2

---

Published in Transactions on Machine Learning Research (05/2023)
To summarize, this paper's main contributions are the following:
Since the optimal layer to use in Self-Superiise-Learniie might not always be the backbonn, we suggest coining the action of removing layer as general method called Guillotine Regularization to distinguish it from the architectural modification which is the addition of a projector.
To show through experiments that the optimal layer to cut heavily depend on the training optimiza- tion, training data and downstream task for both supervised and self-supervised models. We hope that this result will encourage the research community to run more systematic evaluations through different layers.
The need to use Guillotine Regularization in SSL depends heavily on how the positives views are defined. When these views are aligned with the downstream-task, the optimal layer to use become closer to the last layer.
2
Related work
Self-supervised learning Many recent works on self-supervised learning CChen et al. 2020a Grill et al.
2020 Lee et al. 2021: Caron et al., 2020; Zbbontrr et al. 2021 Bardes et, al. 2022] Tomasev et al., 2022
Caron et al. 2021 Chen et al., 2021 Li ct al. 2022 Zhou et al. 2022a b) rely on the addition of few non linear layers (MLP) termed projection head on top of a well established neural network termed backbone during training. This addition is done regardless of the neural network used as backbone, it could be a ResNet50 (He et al., (2016) or Vision Transformer (Dosovitskiy et al. 2021]. After training, the projector is usually threw away to evaluate the model using the backbone representation. Even if Chen et al.] (020bb) demonstrated that the optimal layer to use might not always be the backbone when using few labelled data, most recent works introducing new SSL methods have continued to use only the backbone for ovaluation. Some works also tried to understand why a projection head is needed for self-supervised learning. Appalaraju et al. (2020) argue that the nonlinear projection head acts as filter that can separate the information used for the downstream task from the information useful for the contrastive loss. In order to support this claim, they used deep image prior {Ulyanov et al.) 2018) to perform features inversion to visualize the features at the backbone level and also at the projector level. They observe that features the backbone level seem more suitable visually for downstreem classification task than the ones at the projector level. Another related work (Bordes et al.] 2021) similarly tries to map back the representations to the input space, this time by using a conditional diffusion generative model. The authors present visual evidence confirming that much of the information about a given input is lost at the projector level while most of is still present at the backbone level. Another line of work tries to train self-supervised models without the use of a projector. Jing al. (2022) shows that by removing the projector and cutting the representation vector in two parts, such that SSL criteria is applied on the first part of the vector while no criterion is applied on the second part, improves considerably the performances compared to applying the SSL criteria direetly on the entire representation vector. This however works mostly thanks to the residual connection of the resnet. In ront with these approaches, our work focus on identifying which components of iraaltional SSL training pipelines can explain why the performances when using the final layers of the network are so much worse than the ones at the backbone level. This identification will be key for designing future SSL setups in which the generalisation performance doesn't drop drastically when using the embedding that the SSL criterion actually learns.
Transfer learning The idea of using the intermediate layers of a neural network is very well known in the transfer learning community. Work like Deep Adaptation Network (Long et al., 2015) freeze the first layers of a neural network, fine-tune the last layers while adding a head which is specific for cach target domain. The justification behind this strategy is that deep networks learn general features (Caruana] 1994| Bengio 2012 Bengio et al.) 2011), especially the ones at the first layers, that may be reused across different domain (Yosinski et al.] (014)). Oquab et al.] (2014) demonstrate that when limited amount of training data are available for the target tasks, using the frozen features extracted from the intermediate layers of a deep network trained on classification can help solve object and action classification tasks other datasets.
3

---

Published in Transactions on Machine Learning Research (05/2023)
Another line of work training with random noisy labels also studied how the use of intermediate layers improves significantly downstream performances (Maennel et al. 2020) Baldock et al. (2021) introduced a measure of example difficulty that leverages the number of intermediate layers that are aligned towards a given prediction. In this paper, we show that SSL trained models fall under the realm of transfer learning, in consequence we can expect that all the observations made in the transfer learning literalure about the use of intermediate layers are also valid for SSL. When viewing the projector SSL trick and cutting layer for transfer as general machine learning trick to improve generalization, it's not surprising anymore that work
as Wann et al.] [ Sanyyıdaı et al. (0223 have been able to show that adding a projector can also be highly beneficial for supervised training.
Out of distribution (OOD) generalization Kirichenko et al. (002)) demonstrates that retraining only the last layer with a specific reweighting helps to "forget' the spurious correlations that were learned during the training. Such work emphasizes that most of the spurious correlation due to the training objective is contained in the last layers of the network. Thus, retraining them is essential to remove such spurious correlation and generalize better on downstream tasks. Similarly Rosenfeld al. (2022) show that retraining only the last layers is most of the time as good retraining the entire network over a subset of downstream tasks. Lastly, Evei et al. (2022) demonstrates the usefulness of using intermediate layers for OOD. Our si slud also confirms that Guillotine Regularization show important properties with respect to OOD generalization.
3 Guillotine Regularization: A regularization scheme to improve generalization of deep networks
In this section, we provide a definition for Guillotine Regularization. Then, through experiments, we show that the optimal layer to use changes significantly depending on different factors. Finally, we show that the performances at a given layer are not always correlated with the performances one can have at another layer.
3.1 (Re)Introducing Guillotine Regularization From First Principles
We distinguish between a source training fask with its associated training set, and a target dounstream task with its associated datase [ It is the performance on the downstream task that is ultimately of interest. In the simplest of cases both tasks could be the same, with their datasets sampled i.dd. from the same distribution. But more generally they may differ, as in SSL or transfer learning scenarios. In SSL we typically have an unsupervised training task, that uses a training set with no labels, while the downstream task can be a supervised classification task. Also note that while the bulk of training the model's parameters happens with the training task, transferring to a different downstream task will require some additional, typically lighter, training, at least of a final layer specific for that task. In our study we will focus on the use of a representation computed by the network trained on the training lask and then frozen, which gets fed to a simple linear layer that will be tuned for the downstream task. This 'linear evaluation' procedure is typical in SSL and aims to evaluate the quality /usefulness of an unsupervised-trained representation. Our focus is to ensure good generalization to the downstream task. Note that training and downstream tasks may be misaligned in several different ways.
Informally, Guillotine Regularization consists in the following: for the downstream task, rather than using the last layer (layer L) representation from the network trained on the training task, instead use the representation from a few layers above (layer t, with < L). We thus remove a small multilayer 'head' (layers + to L) of the initially trained network, hence the name of the technique. We call the remaining part (layers 1 to t) the P
Formally, we consider deep network that takes an input X and computes a sequence of intermediate through layer functions ((),... f(l) such that He = (H,, starting from
representations
Termiinoooyy pretext-training / downstream comes from SSL while source / target used in transfer learning Phead / trunk are also known as projection head / backbone in the SST literature
4

---

Published in Transactions on Machine Learning Research (05/2023)
SimC1.
Supervised
0.22
Floor
6.10
Spat
rotadion
0.05
0.00
Wierinss
0.08
0.66
0.04
0.0
0.02
Cb ect rotation
Heael
heard
cowe
alook
Mork
Layere
Layers
Figure 2: Training a linear regression to predict latent variables from pooled intermediate representations of network trained with a self-supervised objective (using SimCLR) or a supervised objective (trained to predict 3D rotations of an object). The data used consists of renderings of 3d objects from 3D Warehouse (Trimble Inch where control the floor, lighting and object pose with latent variables, see samples the right. The dimension of the intermediate representations increases throughout the layers and is kept constant in the head, if there is one. In the supervised setling, when looking at the Validation Mean Squared Error for object rotations prediction, the lowest error is obtained with the linear probe at the last layer of the neural networks. In contrast, the lowest error for other attributes like the Spot prediction are obtained with the linear probes localized 3,4 or 5 layers before the output of the networks. In the self-supervised setting, we also see that the predictor is responsible for a lot of the invariance augmentation, and that the information is most easily retrievable before These results highlight the need to use Guillotine Regularization i.e removing the last layers of the neural network to generalize better other tasks.
Ho = X. The entire computation from input X to last layer representation Hy is thus a composition of layer functions
Hy = fo, (f(L) o... f(+-1) p(t) o....off f()))
head f/ f1+
trunk fiit
The parameters 0 and & of trunk f and head fo are then trained on the entire training set of examples Xsource of the training task (optionally with associated targets Ysource that we may have in transfer scenarios, but will typically be absent in SSL), to minimize the training task objective L source,
0, 3 arg min source (ft+ (T+1::1ff
11d sooure)), Ysorrce)
0,0
Then the multilayer head f f++1 is discarded, we add to the trunk a (usually shallow) new head sw and we train its parameters using the training set of examples for the downstream task (Xtarget, ylarget), to minimize the downstream task objective I target.
t = arg min larget (swl
el:t fl:t target
(1
llrggt))
representation Htange
PPrcciscly, 'layer function" (() correspond to standard neural network layer (fully-connected, convolutional) with residual shorteut connections between them, entire blocks (as in densenet, transformers) which may have internal shortcut connections, but none between them.
5

---

Published in Transactions on Machine Learning Research (05/2023)
3.2 An empirical analysis of situations in which cutting layers is beneficial
There are several situations that can create a misalignment between training and downstream task. Here name of few:
Misalignment between the training (source) and downstream (target) task while using the same input data distribution. The potential effectiveness of GR for transfer is not surprising since this tochnique has been used for years in the transfer learning research literature (Yosinski et al. 2014) to improve generalization across different tasks. As a simple illustration, we present Figure 2 which show how much performances on a given task can vary depending on which layer has been chosen as features extractor. In this figure, we used an artificially created object dataset in which of variations. The dataset consists of renderings of 3D models from 3D warehouse (Trimble Inc]. Each scene is built from 3D object, a floor and a spot placed on top of the object to add lighting. This allows us to control every factor of variation and produce complex transformations in the scene. We vary the rotation of the object defined as a quaternion, the hue of the floor, and the spot hue as well as it position on a sphere using spherical coordinates. We provide more details on the dataset and rendering samples the appendix. We observe Figure 2 that when training a supervised model on the object rotation prediction task and evaluating the linear probe on the same lask across different layers, the best results are obtained on the last layer. However, when using the same frozen neural network to predict other attributes like the Spot 6, the best performances are obtained few layers before the last one. Similarly, when training with a self-supervised objective (SimCLR), we can see that the different factors of variation are most easily retrievable before the projector. This means that representations before the projector will be more versatile as they will contain information that was removed by the pretraining task. For example our downstream task is to predict the rotation, the representation at block4 will be optimal while if the downstream task is to predict the spot hue, the representation at the block 3 will be optimal. Such results highlight the need to Guillotine Regularization when there is shift in the prediction task. Moreover, the optimality of layer depends on the downstream task.
able to play with different factors
Misalignment due to badly optimized network It can be expected that the op- timal layer to use to train a downstream task readout function might be differ- ent depending on how much the pretrained network To test this hypothesis, we train headed supervised Resnet50 ImageNet with two different types of optimization. The first one using only AdamW with small learning rate of 1e-4 with- out any additional regularization. The second one using SGD and the recommended
is overfitting on the pretext task.
80
70
parameeressssss for a supervised train- ing (with cycling learning rate, weight docay and momentum). In Figure [3] we observe that the Adamn trained network that is overfitting on the classification task has readout function per- formances that are very close across different layers. However, when looking the well-regularized model with SGD which does not overfit, on the lask, the readout performances across layers vary significantly. In second experiment, we study more in-depth the effect of overfitting by training the Resnet50 over only a random subset of 250 classes. Then we use the remain- ing 750 classes as an OOD validation set that is split. randomly in other subset of 250 classes. In Figure 3b we clearly see that the training readout is overfitting the training set while the readout performances across layers are similar on the corre- sponding in distribution validation set (which is similar to the previous experiment over the full ImageNet). Then, we train linear probes over the OOD splits and observe that the performances are radically different from the in distribution validation set. In fact, in this instance the best layer to use for every of these split is the backbone layer whercas the best layer to use for the in-distribution split is the projector layer. This result highlights that the optimal layer to discard can vary depending on the optimization techniques and downstream data distribution, even when the same training objective is used.
55
Val
330
Berath
Clzug
Trunk
Head
Head
Head
Layer
Linear probe accuracy on several downstream tasks. The opti- mal layer to cut is not the same for different downstream tasks.
Figure 4: SimCLR:

---

Published in Transactions on Machine Learning Research (05/2023)
Traizing splitt
95
Validation split.
ImsgeNet Top- Accuracy
ImogeNet Top-1 Accuracc
90
Top-1 Val Accuracy
80
85
Validation splits with classes
60
Spli. 1
Splic Split 4
Split 5
Split 2
Split 6
75
80
40
75
Imagelet
Imagelee 1%
70
Eurosat
CITARIO
Wall regularzaed
Training
Sudly regulazized
Validation
Places200
Trunk
Head
3
Head
Head
Head
Trank
Head
Head
Layer
Layer
(a) Training setups
(b) Random split
(c) Downstream tasks
Figure 3: Supervised: The optimal layer to cut might change depending of the training optimization, the data and the downstream task. The best accuracy for each curve is show experiments, we trained a headed supervised Resnet50 over ImageNet (with 3 layer MLP as projection head). For a) and c) we trained this network over the full training set whereas for b) we use random subset of 250 classes. Then, we froze the model parameters and trained linear probes over representation at different layers. a) We trained two models with different optimization pipeline: the first one in blue was trained with SGD using cycling learning rate, along with momentum and weight decay. The second one in gray was trained with AdamW without additional regularization. This model is overfitting the training set, which leads to similar validation performances across the backbone and projector. In contrast, the first one generalize much better but the performances across layers change significantly. b) Validation accuracy given linear probes each layers. The validation split in gray corresponds to the same subset of classes that was used for training w Split 1-6 corresponds to different OOD random split. In this instance, we that the optimal layer use is the first layer of the projector. e) Validation performances different downstream tasks. We have used the well regularized model from a) and evaluate it across different downstream tasks. For some datasets, the optimal layer to use is the last one, while for some other the optimal layer is the second layer of the projector.
big square. For each
different random subset of 250 ImageNeet's classes for
Misalignment between the training and downstream tasks while using different data distribu- tions. When using pretrained model to predict classes, there is bias in the data distribution well as in the fine-tuning objective (with respect to the training settings). We did first experiment in Figure 3e in which we train a supervised Resnet50 over ImageNet. Then, we freeze the weights of the model and train a linear probe over ImageNet (Deng et al. 2009), CIFAR10 (Krizhevsky) 2009), Place205 (Zhou et al. 2014). CLEVR (Johnson et al., 2017) and Eurosat (Helber et al.) 2019) at different layers. We obscrve that the readout performances on ImageNNtt are the best at the last layer but for datasets like CLEVR or Place205 the best performances are obtained the second projector layer. In Figure 4 we performed the same ex- periment but this time using SimCLR. In this instance, the best performances for ImageNet are obtained the backbone whereas the best performances for Eurosat, CLEVR and Imagenet 1% are obtained at the first projector layer. This result. challenges the common practice of discarding the entire projector in SSL since the layers to cut depend on the downstream task.
Misalignment between the training input data distribution and testing input data distribution while using the same training and downstream tasks. Another type of bias can arise when using
wrongful data distribution after training of the model. This scenario is often referred to Out Of Distribution (OOD)
since the distribution of the data used by the model becomes different from the one seen during training. We took the super- vised model trained on ImageNet along with the linear probe trained at different layers and evaluate the performances of these readouts on ImageNet-C (Hendrycks Dictterich 2019) which is modified version of the validation set of ImageNet
Head 3 Head 2 Head 1 Trunk
59.0
58.8
58.0
63.3
Table 1: ImageNet-C mCE (unormalized) across layers.
which different data transformations were applied. Our experiment in Table 1 demonstrates that the
7

---

Published in Transactions on Machine Learning Research (05/2023)
ImageNet Acouracy
ImageNes Top- Accuracy
60
ImageNa Tap-l Acsurway
65
50
60
33
50
Backbone Projector
40
Backbone Projector
55
30
Image
85
40
50
20
80
30
Projector
35
Projector
L.0
0.25 0.5 0.75
2.5
7.5
0.4
0.6
0.7 0.8
1.0 15 1.7
SimCLR: Temperature
Byol: Momentam Teacher
Barlow Twims Adamm Learning Rate 11e-30
Ryol LARS Learning Iate
(a) SimCLR
(b) Byol
(c) Barlow Twins
(d) Byol
Figure The performances at the projector level aren't always correlated with the perfor-
mances at the backbone level. We train SimCLR, Barlow Twins and Byol with different hyper-paramoters and evaluate with linear prob, the performances the backbone but also the projector level on Im- ageNet classification task. For each model, observe that the accuracy given by the linear probe at the backbone level isn't always correlated with the performance at the projector level.
performances are better after cutting layers from the head of the network which highlight that it might be a good practice to probe intermediate representations when evaluating OOD tasks.
3.3 The readout performances at the projector and backbone level are not always correlated
In Figure we study the effect of Guillotine Regularization with respect an hyper-paramcter grid scarch for various SSL methods (SimCLR, Barlow Twins and Byol). When looking the performances on ImageNet using a linear probe at the backbone level, one observe almost stable classification task performance for different pypeepparameters such as SimCLR temperature, Barlow Twins and Byol learning rate while the corresponding performances at the projector level change significantly. This highlights that the performances at the projector level are not always correlated with the performances at the backbone level. In consequence, knowing the performances of a linear probe at the projector level cannot give in advance insights about the performances at the backbone level.
4 Reducing the Need for Projector in Gelf-Supervised Learning by increasing the alignment with the downstream task
Self-Supervised Learning is often considered a distinct learning paradigm in between supervised and unsu- pervised learning. reality, the distinction not as sharp, and much of SST can be understood as solving pretext-tasks akin to supervised task Wu et al.) 2018 Khosla et al., (220)). merely with pseudo-labels obtained in another way than by human annotation. In this section, show that different data selection process in SSL influences the alignment between the downstream and pretext task, which heavily impact the need of using a projector head in SSL.
To confirm the hypotheses that SSL methods need to use a projector because of a misalignement between the pretext and downstreem task, we have to verify that reducing this misalignement, results reducing the performance gap between the Trunk and Head representations. Idcally, we would like get close to the supervised scenario in Figure | for which the optimal readout function is obtained at the last layer. To do so, devise two experimental setups in which replace the traditional data augmentation pipeline used in SSL, which consists of using handcrafted augmentation on each image to create a set of pairwise positive samples.
In the first setup, while using the exact same SSL criterion (SimCLR), we use as positive examples pairs images that belong the same class, and as negative examples images that don't belong to the same class. Note that the SSL training criteria will push towards a collapse in the representation space of all the images belonging to the same class, while pushing further apart the different
8

---

Published in Transactions on Machine Learning Research (05/2023)
Backbone Acc. Projector Acc.
H.T.
20
10
-10
-20
Normal Aug.
-30
KNN Data Aug.
Class Data Aug.
IN1K
101
INIK-1P
CIFAR10
CIFAR100
CLEVR-C
CLEVR-D
Eurosat
Places205
Inat18
Datasets
Figure 6: Difference accuracy with linear probing between the projector and backbone representation with different aligmments with respect to the classification downstream lask. In this experiment we used SimCLR and change how the positive pair are defined to better aligned with a classification downstream task. In blue, our baseline, we trained SimCLR with the traditional SSL data augmentations which defines the positive view as two augmentations of a same image. In orange, we use the embedding of a pretrained model to define the positive pair as two nearest. neighbors under pretrained model (while using the same data augmentation the baseline). In green, supervised class label selection to define the positive examples. In this scenario, SimCLR should learn to produces similar embedding to all images belonging we evaluate them with a linear probe across a wide range of downstream tasks at the backbone and projector level and show the difference in accuracy between both. When the difference is positive, the accuracy at the backbone level is higher than the one at the projector level, highlighting the benefits of Guillotine Regularization. In contrast when the difference is negative, the accuracy at the projector level is higher than the one at the backbone level. In this instance, Guillotine Regularization is not needed. When positives pairs are defined as belonging to given class, there is no misalignment with the imagenet classification downstream task. Thus on ImageNet- IK, Ima (10% of the training set to train the linear probe) and ImageNetlk-IP (1% of the training set to train the linear probe), we observe that the performances at the projector level are much higher than the ones at the backbone level. Interestingly, the nearest neighbors heuristic reduces considerably the impact of Guillotine Regularization across several downstream tasks.
given class. All three models are trained ImageNet (INIK), then
class clusters. By doing the training SSL objective becomes perfectly aligned with the downstream classification task, despite using SSL training criteria instead of a traditional cross entropy loss.
In the second setup, we use as positive pairs the closest neighbors found by pretrained SSL model trained with the traditional SSL handcrafted data augmentation pipeline. The reasoning is that if instead of considering each image of the dataset as its own specific class, we use clusters of many images to define the positive pairs, we might be able to close the gap with respect to a supervised baseline without the need of labels.
In Figure 6 we show the differences in accuracy between the backbone and the projector with respect to these two new data augmentation scenarios. The baseline, using the traditional SimCLR positive pairs based on data augmentations is in blue, the nearest neighbors sctup in orange and the class based sctup in green. We observe for SimCLR that using the nearest neighbors based heuristic is helping in reducing the gap between the pretext and downstream task while having a purely supervised heuristic to define the positive pair is removing the need to perform Guillotine Regularization across several downstream tasks. Hence confirming the hypothesis that the effectiveness of a projector depends of the alignment between the pretext and ream task in self-supervised learning.
9

---

Published in Transactions on Machine Learning Research (05/2023)
Head 3
Head
Head
Trunk
(a) SimCLR trained with SSL augmentations.
(b) SimCLR trained with class labels.
conditional generative model to visualize what information is decodable at different layers. The leftmost column images (before the red line) is the conditioning image that was used to compute the representation that is fed to RCDM. The subsequent columns generated by the model using this representation. The first row correspond to the last projector layer, the second row to the second projector layer, the third one to the first projector layer and the last row to the backbone layer. As show in Figure O changing the alignment with the pretext task change significantly the information encoded by the neural network. When using SSL augmentations at the projector level, the information about the dog's breed seem to have been lost whereas when looking at a network trained with supervised augmentations, the information is preserved throughout each layers.
Figure 7: In this ligure, we used RCDM [Brddes et al. 2022
samples
4.1 Visualizing the information across layers for different alignments
In this section, we use RCDM (Bordes al. 2022), a conditional generative model to visualize what infor- mation is retain not in the representation. We train RCDM on ImageNet with blurred faces( et al.), using the representation given by SimCLR model trained on handcrafted SSL views and another which trained on class based views. In Figure 7 we show that when looking at different decoding corresponding to different layers in the network, the information encoded vary a lot depending on the layer to use. When going deeper, RCDM is not able to reconstruct as much as information about the images than when using the backbone representation (which contain much more low level features). When looking at the generated samples that were conditioned on the representation of the model trained with supervised views, we observe that the breed of the dog stay the same across layers. However when using traditional data augmentations, the information about the specific golden retriever breed is lost in the last projector layers. This is correlated with the fact that this model get. lower classification performances when using the projector.
4.2 Experimental details
We use Pytorch (Paszke al.) 2019 and FFCV-SSL [Bordes et al. 2023
Leclere al. 2022 as data loader.
All the experiments backbone. For cach model, we use batch of size 2048 and AdamW (Loshchilov & Hutter 2019 optimizer with an adaptive learning rale schedule. We run the training for 100 epochs. For each model, we add as head a small MLP 3 layers of size 2048 (same dimension as the backbone) with ReLU (Glorot et al., 2011h as activation and batch normalization [Toffe & Szegedy 2015). When training different. SSL mothods, we always used the same set of data augmentations (with cropping, color-jitter, random grayscale, gaussian blur and solarization).
performed with Resnet50 (He al. 2016) (except if mentioned otherwise)
10

---

Published in Transactions on Machine Learning Research (05/2023)
5 Conclusion
Through empirical evaluations, we demonstrated that the optimal layer to use for downstream evaluation vary depending on several factors: optimization, data and downstream task. These results highlight the need for SSL. practitioners to run syslematic evaluations al several layers instead of using always the backbone as reference. We also demonstrated that the use of a projector in SSL depends on the alignment between the downstream and pretext task. Despite, its usefulness, having to rely to increase performances reveals an important shortcoming of current self-supervised learning methods: the inability to design experimental setups and training criteria that learn structured and truly invariant representations with respect to an appropriate set of factors of variation. As future work, in order to escape from Guillotine Regularization, we should focus on finding new training criteria and data augmentations that will be more aligned with the downstream tasks of interest.
trick like Guillotine Regularization
References
Srikar Appalaraju, Yi Zhu, Yusheng Xie, and István Fehérvári. Towards good practices in self-supervised representation learning. In NeurIPS Self-Supervision Workshop, 2020.
Robert J. N. Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. In Neural Information Processing Systems, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varraane-invvriinne-ovvaar regularization for self-supervised learning. In TCLR, 2022.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Isabelle Guyon, Gideon Dror, Vincent Lemaire, Graham Taylor, and Daniel Silver (eds.), Proceedings of ICML Workshop on Unsupervised and Transfer Learning, volume 27 of Proceedings of Machine Learning Research, pp. mlr. presa/v27/
17 36, Bellevue, Washington, USA, 02 Jul 2012. PMLR. URL htttps://proceeeee bengio12a. html
Yoshua Bengio, Fródéric Bastien, Arnaud Bergeron, Nicolas Boulangor-Lewanddwwer Thomas Breuel, Yous- souf Chherawala, Moustapha Cisse, Myriam Côté, Dumitru Erhan, Jeremy Eustache, Xavier Glorot, Xavier Muller, Sylvain Pannetier Lebeuf, Razvan Pascanu, Salah Rifai, François Savard, and Guillaume Sicard. Deep Iearners benefit more from out-or-distribution examples. In Gooffrey Gordon, David Dunson, and Miroslav Dudik (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 164-172, Fort Landerdale, FL, USA, 11 13 Apr 2011. PMLR. URL htttps://proceedingg mlr pttpss//1///eegioool.o... html
Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self- supervised representation knows about. CoRR, abs/21 abs/2211..99144 2021. URL htttps://arxit wwwpzxivvorg//ba..
09164
Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self- supervised reprosentation knows about. Transactions on Machine Learning Research, 2022. URL https:
Florian Bordes, Randall Balestriero, and Pascal Vincent. Towards democratizing joint-embedding self- supervised learning, 2023. URL
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In NewrIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herve Jegou, and Julien Mairal Piotr Bojanowski Armand Joulin. Emerging propertics in self-supervised vision transformers. In ICCV, 2021.
Rich Caruana. Learning many related tasks at the same time with backpropagation. G. Tesauro, D. Touretzky, and T. Leen (ods.). Advances in Neural Information Processing Sys- tems, volume MIT Press, 1994. URL http:://ppoeenngg..errrrscccccpp 0f520beeeaaa233aa22222221111---2
In
11

---

Published in Transactions on Machine Learning Research (05/2023)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for con- trastive learning of visual representations. In ICML, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. In NewrIPS, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentun contrastive learning. Xix preprint ar arXim:2003.042: 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. ICCV, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2Toe: Utilizing intermediate representations for better transfer learning. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 6009 6033. PMLR, 17-23 Jul 2022. URL https: bttb:::/pproceedingg mlr,
html
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 315- 323, Fort Lauderdale, FL. USA, 11-13 Apr 2011. PMLR. URL https: https://pproeeedings mlr httpa/nnw........s/miseee press/v15/ g glorot1la.htm
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self- supervised learning. In NeurIPS, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Patrick Helber, Benjamin Bischke, Andrcas Dengel, and Damian Borth. Eurosat: A novel datasct and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topies in Applied Farth Observations and Remote Sensing, 12(7):2217-222 2226, 2019. doi: N0.1009/ITAARSS2209222000
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Sergey lofle and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448 456, Lille, France, 07-09 Jul 2015. PMLR. URL
mlr. pptess//37/offell...om html
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
12

---

Published in Transactions on Machine Learning Research (05/2023)
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Had- sell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18661-18673. Curran Associates, Inc., 2020. URL httpps:/prroeeedinss hewpocaadiingg..uuipp.........p htt1://546667r7022221aaaaaaaaas2
neurips httph//gsw.oiirr.cr//eeeeeeeeeee pappr//22/0//
pdf
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson Last layer re-training is sullicient for robustness to spurious correlations, 2022. URL https://arxiv...
02937
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32 33, 2009. URL https:
Guillaume Leclerc, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry, ffev.
2022. commit XXXXXXX
Kuang-Huce Lee, Amurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. In NewrIPS, 2021.
Chunyuan Li, Jianwei Yang, Pongchuan Zhang, Mci Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient sele-supervised vision transformers for representation learning. In ICLR, 2022.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML'15, pp. 97-105. JMI.R.org, 2015.
Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Tn 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net 2019.
URL
Hartmut Maennel, Ibrahim Alabdulmohsin, Ilya Tolstikhin, Robert J. N. Baldock, Olivier Bousquet, Sylvain Gelly, and Daniel Keysers. What do neural networks learn when trained with random labels? In Pro- ceedings of the 34th International Conference on Newral Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations ions using convolutional neural networks. In Proceedings of the IEEE conference on compuler vision and pattern recognition, pp. 1717-1724, 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, bigh-performanc deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alcé-Buu,, F. Fox, and R. Garnett (eds.), Admances in Neural Informa- tion Processing Systems 32, pp. 8024-8033 Curran Associates, Inc., 2019. URL httt::/:p::pprr/// http://paapss...u..r/.ee neurips. cc/
learning- libraryibbrrr... -pdf
Elan Rosenfeld, Pradeop Ravikumar, and Andrej Ristoski. Domain-adjusted regrossion or: Erm may already learn features sufficient for diitt-idistiiitto generalization, 2022. URL httpss//arxxv.rrg wrw.:/s///////.ee.ecoo... abs/ ab2g/22.. 2202.
06856
Mert Bülent Sanyyldız, Yannis Kalantidis, Karteek Alahari, and Diane Larlus. No reason for no supervision: Improved generalization in supervised modols. In The Eleventh International Conference on Learning Representations, 2023. URL hattps://ppenrevien wttpo/vwwwwwwneeffneeeeee httt/forumndd-3Y5unn..
Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet? orXiv preprint ar xxxx:22001.05111 2022.
Trimble Inc. 3d warehouse. httpp://Sddvrreoous..kkeee stt.:/uusse..eeeeecu......ooooo
Accessed: 2022-03-07.
13

---

Published in Transactions on Machine Learning Research (05/2023)
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In CVPR, 2018.
Yizhou Wang, Shixiang Tang, Feng Zhu, Lei Bai, Rui Zhao, Donglian Qi, and Wanli Ouyang, Revisiting the transferability of supervised pretraining: an MLP perspective. In IEEE/CVF Conference on Computer Vision and Puttern Recognation, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 pp. 9173 9183. IEEE, 2022. doi: 10.1109 (C0000//CVVP2228820222222 00897.. URL https://dol.orr//0.. wtt..////00..110///P222222222220 2022. 00897
Zhirong Wu, Yuanjun Xiong, X Yu Stella, and Dahua Lin. Unsupervised feature learning via non con-paramctric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pollern Recog- nition, 2018.
Kaiyu Yang, Jacqueline Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky. A study of face obfuscation in imagenet. In International Conference on Machine Learning (ICML).
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.). Advances
in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL https:// pdf
proceedings weapiiiinnnneuuuis...e. www.neus.....pprre//eee https///....ae..///.////////aaaa
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning
via redundancy reduction. arXiv preprint 2rzi::203.03323. 2021.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Newral Information Processing Systems, vol- wme 27. Curran Associates, Inc., 2014. URL https://proceeddngg newriiiis.nn.................. wwweurpps.ccppppe///////. pc/ppp///////////iilel/ (tss4a002311651222511866622
pdf
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong, ibot: Image bert pre-training with online tokenizer. In ICLR, 2022a.
Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: A multi- granular self-supervised learning framework. 2022b.
A Datasets
In this work, we use ImageNet (Deng al. 2009) (Term of license on htlps://www.inest ret.org/downloar
for our experiments. We also used synthetic 3D dataset that will be described in the next subsoction.
A.1 3D models dataset
We will now discuss the dataset used for figure 2 As previously mentioned, this dataset consists of 3D models from 3D Warehouse (Trimble Incp, freely available under General Model License, and rendered with Blender's Python API. We alter the scene by uniformly varying the latent variables described in table 2
The varicty in the scenes that can be generated is illustrated in figure g We can see that cach latent variables can significantly impact the scene, giving significant variety in the rendered images.
B Reproducibility
Our work does not introduce a novel algorithm nor a significant modification over already existing algo- rithm. Thus, to reproduce our results, one can simply use the public github repository of the following models: SimCLR, Barlow Twins, VicReg the PyTorch Imagenet example (for supervised learning) with the following twist: adding linear probe at each layer of the projector (and backbone) when evaluating the
14

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 8: Rendered views of skateboard generated by randomly sampling latent variables. The influence of each parameter is easily visible, which is expected to make their prediction easier.
Table 2: Latent variables used to generate views of 3D objects. All variables are sampled from a uniform distribution.
Latent variable
Min. value
Max. value
Object yaw
-7/2
π 2
Object pitch
-T/2
π 2
Object roll
-T/2
π /
Floor hue
0
Spot 0
0
π //4
Spot &
0
2m
Spot hue
0
model. However, since many of these models can have different syper--arammters, or data-augimentations especially for the SSL models, we recommend to use a single code base with a given optimizor, a given set of data augmentations so that comparisons between models are fair and focus on the effect Guillotine Regularization. In this paper, except if mentioned otherwise, mensions 2048 each (which match the number of dimensions at the trunk of Resnet50) along with batch normalizaton and ReLU activations.
Head, MLP with 3 layers of di-
C Additional experimental results
In this section, we present additional experimental results. The first one in Figure 9 an extended version of Figure with additional results the training set.. Figure T0 is similar setup to the in Figure g
15

---

Published in Transactions on Machine Learning Research (05/2023)
ImageNet Top-T Accuracy
ImageNet Top-1 Accuracy
70
60
SSL: Barlow Twins
50
SSL: Barlow Twins
SSL: VICReg
SSL: VICReg
SSL: SimCLR
SSL: SimCLR
40
SSL: Byol
SSL:
Supervised
Supervised
Trunk
Head 1
Head
Head
Trunk
Head 1
Head 2
Head
Layer
Layer
(a) Accuracy on the training set
(b) Accuracy on the validation set
Figure 9: We mcasure with lincar probes the accuracy different layers on a resnet50 (as Trunk) on which added small layers MLP (as Head) for various supervised and self-supervised methods on the training and validation set. For each method, show the mean and standard deviation across 3 runs (The std between different runs is low). When looking at self-supervised methods, the gap in performances between the linear probe trained at different levels can be as high as 30 points of percentage.
where we compared the performances at different layers for SSL methods and a supervised one except that we use a VIT-B instead of Resnet50. We observe an important gap on the classification performances reached with linear probe on different layers with the VIT-B when using SSL methods.
In Figure 11 show how the performances different layers change during training by using online linear probing. At the beginning of the training the gap of performances between layers is low, however it increases significantly after 10 epochs.
In Figure 12 we show the accuracy computed with linear probes trained using projector and backbone representations. This figure similar to Figure exceppt that we present the absolute accuracy value instead of the difference in accuracy with respect the backbone.
Limitations
In this work we focused mostly on analyzing the use of Guillotine Regularization the context of Self- Supervised Learning. However, this kind of regularization might be useful for a variety of other types of training methods which don't investigate in this paper. We also mostly focus on generalization for classification tasks, but other tasks could also been worth exploring.
16

---

Published in Transactions on Machine Learning Research (05/2023)
70
ImageNet Top-1 Acouracy
ImageNet Top-1 Accuracy
70
60
55
60
55
SSL: Barlow Twins
SSL: Barlow Twins
SSL: VICReg
SSL: VICReg
SSL: SimCLR
45
SSL: SimCLR
Supervised
Supervised
45
Trunk
Head 1
Head 2
Head
Trunk
Head 1
Head 2
Head
Layer
Layer
(a) Accuracy on the training set
(b) Accuracy the validation set
Figure 10: Same experiment in Figure g but this time, measure with linear probes the accuracy different layers on a VTT-B (as Trunk) which we added a small 3 layers MLP (as Head) for various supervised and self-supervised methods. Since the outputs of the VIT-B has a lower number of dimensions than a Resnet, we added at the trunk of the VIT-B a linear layer with ReLU activation to project into a 2048 dimensional vector. In the supervised learning setting, the best performances are obtained when using the last layers the model. But, when looking sclf-supervised methods, the gap in performances between the linear probe trained at different levels can be high as 20 points of percentage. Interesting, seems for the VIT-B that got the best performances at Head 1 for SimCLR whereas for the ResNet, the best performances were obtained at the Trunk. It is likely that for different architectures, the optimal number of layers on which to apply Guillotine Regularization will vary.
ImageNet Top-1 Val. Accuracy
Val. Accuracy
60
60
50
40
30
Trunk
Trunk
ImageNet
Head
20
Head 1
Head 2
Head 2
10
10
Head 3
Head 3
20
40
60
80
100
20
40
60
80
100
Epochs
Epochs
(a) Accuracy on the training set
(b) Accuracy on the validation set
Figure 11: a) Accuracy of Barlow Twins through epochs computed with online linear probing at different layers. At the beginning of the training the gap in performances between the probes small however after epochs, the gap becomes larger and larger both on the training and validation set.
17

---

Published in Transactions on Machine Learning Research (05/2023)
Normal Data Aug.
KNN Data Aug.
70
Class Data Aug.
Backbone Accuracy
60
50
40
30
20
10
IN1K
INIK-10P
INIK-1P
CIFARIO
CIFAR100
CLEVR-C
CLEVR-D
Eurosat
Places205
Inat18
Datasets
70
Normal Aug.
KNN Data Aug.
Class Data Aug.
Projector Accuracy
50
40
30
20
10
IN1K
INIK-10P
INIK-1P
CIFAR10
CIFAR100
CLEVR-C
CLEVR-D
Eurosat
Places205
Inat18
Datasets
Figure 12: Backbone and projector accuracy with lincar probing with different alignment with respect the classification downstream task. In this experiment we used SimCLR and we change how the positive pair are defined to better aligned with a classification downstream task. In blue, our bascline, trained SimCLR with the traditional SSL data augmentations which defines the positíve view as two augmentations of same image. In orange, use the embedding of pretrained model to define the positive pair as two nearest neighbor under this pretrained model (while using the same data augmentation as the baseline). In green, we use supervised class label selection to define the positive example. In this scenario, SimCLR should learn to produces similar embedding to all images belonging to given class. All three models are trained on ImageNet. (IN1K), then we evaluate them with a linear probe across wide range of downstream tasks the projector and backbone level. When positives pairs are defined belonging to a given class, there is no misalignment with the imagenet classification downstream task. Thus ImageNet- IK, ImageNetlk-1ln 10P (10% of the training set to train the linear probe) and ImageNet (1% of the training set to train the linear probe), we observe that the performances at the projector level are much higher than the ones using the traditional SSL augmentations. Interestingly, the nearest neighbors heuristic reduces considerably the impact Guillotine Regularization across several downstream tasks.
18

---

Published in Transactions on Machine Learning Research (05/2023)
ImageNNet Top-1 Accaracy
ImageNet Top-1 Accuracy
ImageNet Top-1 Accuracy
ImageNei Top-1 Accuracy
60
60
60
50
Backbone
50
Backbone Projector
Projector
30
40
40
Backbone
Backbone
20
Projector
Projector
30
10
30
10
15
0.2
0.4
0.6
0.00
0.02
0.04
0.6
0.7
0.8
0.9
1.0
VICReg: Cov coeff
SimCLR:
Temperature
Barlow Twins: Lambd
Byol: Momentum Teacher
(a) VICReg
(b) SimCLR
(c) Barlow Twins
(d) Byol
Figure 5.1: Loss pepeerprrrmtter...
Insgudet Top- Accurvcy
ImagieNet Top-1 Accurvcyy
Imngee Toge-1 Arsurruyy
65
66
54
64
60
53
58
ImagoNet
56
50
54
Projector
Hacidhone
Projector
Haciboon
45
Backbore Projector
Backboce Projector
0.4
0.6
0.7
0.8
1.0
1.5
1.7
0.4
0.6
0.7
0.8
1.0
1.5
1.7
0.25 0.5 0.75
2.5
7.5
0.4 0.6
0.7
0.8
1.0
1.5 1.7
VICReg Learning Rate
SimCLR: Leaming Bate
Adamn Learing Rata (1m33)
Ryol: Learinng Rata
(e) VICReg
(f) SimCLR
(g) Barlow Twins
(h) Byol
Figure 5.2: Learning rate.
65
Backbone Projector
67.5
ImareNett Top Accurscy
ImareNet Arcuray
60
60
65.0
30
50
Inwuattd Toygl
66.00
45
57.5
40
Imaqe/Ntt
53.0
35
52.5
Rackbome
Backbone
Backbone
Projector
Projector
Projector
30
50.0
120
32
64
128
16
32
64
128
16
32
64
128
VICReg: Batch Size
SinCLR
Batch Size
Barlow Twins: Batch Size
Dyol: Datch Size
ii)
VICReg
(i) SimCLR
(k) Barlow Twins
(1) Byol
Figure 3.3: Batch size.
Figure 13: How different hyper-perameters impact the gap in performances between the backbone and projector representation. We train SimCLR, VicReg, Barlow Twins and Byol with diffcrent hyper-parametorr and evaluate with a linear prob, the performances the backbone but also at the projector level on ImageNet classification task. For each model, we observe that the accuracy given by the linear probe at the backbone level is fairly stable across the grid search of hyper-parameters while the linear probe at the projector level can reach very low accuracy. This highly that the probes performances at different layers might not be al always correlated with each others.
19


Reviews:
[{"review_id": "bib3lE8V1d", "reviewer": "Reviewer_hcdo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The question of why the projection head improves linear evaluation performance in the context of self-supervised learning is an interesting one and is not satisfactorily addressed in existing literature.\n- The submission contains several interesting experiments looking at properties of different layers of networks trained with a projection head.\n- The gains in OOD accuracy by using the backbone vs. projection head output in Figure 3c are particularly interesting to me, although as noted below, I am not sure I understand the details of the experiment.\n\nWeaknesses:\n\n- The intro lists as one of the paper's contributions \"To formalize and motivate theoretically the common trick of using a projector in Self-Supervised Learning\" but I do not believe it makes a meaningful contribution here. The \"formalization\" in Section 3.1 is just notation and the Information Theoretic Motivation on p. 5 is conceptually similar to the justification for the projection head provided in Section 4.2 of Chen et al. (2020).\n- Since Chen et al. (2020)'s justification for the projection head is informal, the Information Theoretical Motivation could still be a contribution of interest to TMLR's audience if it were correct. However, the argument that neural networks lose information as information propagates from one layer to the next is not convincing, as previously noted by Goldfeld et al. (2019). For the case of continuous inputs, the mutual information between the input and the output of a deterministic neural network is usually infinite. To demonstrate this in a ReLU net, it suffices to show that there is at least one region of the input over which the entire network is linear. For the case of discrete inputs, in order to be information-preserving, the network need only be injective, not bijective. For networks with strictly monotonic activations, the network is injective except at a measure zero set of weights. In principle it is possible for ReLU networks to lose information, but that would require that there are two inputs that produce the same output.\n- The paper seems to provide two contradictory descriptions of what is shown in Figure 3c. At the bottom of p. 6, the paper says that the linear probes are trained on the standard ImageNet classification task and evaluated on ImageNet-C. However, the caption of Figure 3 says the probes are trained on ImageNet-C.\n- I wonder whether how robustness measured on ImageNet-C at the backbone of a network trained with a projection head compares with robustness of a standard ResNet-50 with no projection head and would like to compare the numbers with other work. For that purpose, it would be useful to report mCE on ImageNet-C as suggested by the ImageNet-C dataset creators.\n- The relationship between contrastive learning and classification in Section 4.1 is, I think, well-known, e.g. Wu et al. (2018) motivate their work in this way. The statement that “the SimCLR criteria is equivalent to a negative crossentropy [sic] loss over the output of a classifier which predict the index of a given image with respect to a random transformation” does not seem quite correct because this classifier would conventionally have different weights for each example rather than using another example’s representation as the weights. I also do not understand the meaning of the brackets in Eq. 2.\n- There are a couple of papers that explore the use of a projection head in supervised learning that are not cited but probably should be. Off the top of my head, I know of Wang et al. (2022) and Sarıyıldız et al. (2023).\n- “Guillotine Regularization” is a fun name, but at this point a new name for the use of a projection head is unlikely to get widely adopted, and I am not entirely convinced that it is a form of regularization.\n\nReferences:\n\nChen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020, November). A simple framework for contrastive learning of visual representations. In International conference on machine learning (pp. 1597-1607). PMLR.\n\nGoldfeld, Z., Berg, E. V. D., Greenewald, K., Melnyk, I., Nguyen, N., Kingsbury, B., & Polyanskiy, Y. (2019). Estimating Information Flow in Deep Neural Networks. In International Conference on Machine Learning.\n\nSarıyıldız, M. B., Kalantidis, Y., Alahari, K., & Larlus, D. No Reason for No Supervision: Improved Generalization in Supervised Models. In International Conference on Learning Representations.\n\nWang, Y., Tang, S., Zhu, F., Bai, L., Zhao, R., Qi, D., & Ouyang, W. (2022). Revisiting the transferability of supervised pretraining: an mlp perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9183-9193).\n\nWu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018). Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3733-3742).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main contributions in this paper are a series of empirical studies of the properties of the projection head of self-supervised learning methods. Specifically:\n- When varying several parameters when synthesizing 3D renderings of objects and then trying to recover the value of these parameters using a linear probe, the optimal layer varies.\n- When training linear probes on classification datasets on different head layers of a supervised network trained with a projection head, the backbone is usually better than later layers.\n- When training linear probes on different head layers on ImageNet and then evaluating on OOD datasets, the backbone is again better than the later projection head layers.\n- When training networks with contrastive loss using different sets of positives, the backbone is much better than the projector when the positives are augmented views of the same image, but its advantage is not as clear when positives are nearest neighbors in the representation space of a pretrained SSL model, and when the positives are other members of the same class the projector is often better than the backbone.\n- ImageNet linear eval accuracy measured from the backbone is more robust to choice of hyperparameters (especially temperature) than linear eval accuracy measured at the projection head.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Unless there is a way to fix the theoretical claims, they should be stated only as intuition or removed, and they should not be claimed as a contribution.\n- The description of the experiments in Figure 3c should be made consistent. Ideally the paper would also report mCE for each layer in addition to performance on the individual corruptions.\n- If the authors want to keep Section 4.1, it should cite previous work that has made similar observations, but in my opinion it does not contribute much to the paper and could be removed.\n- The papers above should be cited.\n- I think it's fine to use \"Guillotine Regularization\" as a catchy title but I'd suggest minimizing the use of the term elsewhere. It's unnecessary and potentially confusing to rename the use of a projection head given that it is already a common technique. The abstract should also explicitly refer to an \"MLP projection head\" or \"MLP projector\" since that is what people commonly call this trick in previous literature.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FzI3GPGTd6", "reviewer": "Reviewer_JDve", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. **Clean writing**: This paper is very well-written and fun to read. \n2. **Correctly executed experiments**: The experiments are well-designed and executed. They mostly support the discussed findings.\n3. **A few novel results**: As far as I know, this particular set of experiments has not been conducted before, and the discussed findings regarding the specialization ability of the last layers of SSL pretrained networks are probably new.\n\n## Weaknesses\n\n1. **Little added value over prior work**: In my opinion, the main weakness of this work is the fact that the discussed findings are a direct translation of those already known for supervised learning to SSL. In this regard, the proposed view of SSL as a specific subinstance of supervised learning is not new, and therefore, one could have probably expected the presented results even without having performed these experiments. In fact, some of the presented experiments (e.g., training and evaluation on different subsets of ImageNet) are copied almost verbatim from prior work (Yosinski et al., 2014), which further adds to the feeling that the contributions of this work are very small. The evaluation of these experiments on SSL are still technically new, however, so this might be enough to meet the bar for acceptance to TMLR.\n2. **Insights on effect of GR in hyperparameter variability are a bit subjective**: Personally, looking at the plots in Fig. 5.3., I fail to agree with the authors that GR ensures \"a fairly stable accuracy across hyperparameters\". In my opinion, there is indeed a smaller difference in performance, but there are still big fluctuations with respect to different hyperparameter values. The way the authors phrase this section it sounds as though GR allows for no hyperparameter tuning, and in light of these results this is clearly not true.\n3. **Shallow findings**: Considering that most findings could have been predicted in hindsight, and that most experiments are not very novel, I would have appreciated if the paper had studied this problem in more depth, and provided further intuitions. This could be done by visualizing feature maps/embeddings, studying GR in light of the results in Evci et al. (2022), or describing the effectiveness of GR at different pretraining stages, for example.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper delves into the circumstances under which the removal of the final layers of a pre-trained network, named Guillotine Regularization (GR), can improve performance. Through experimentation on various datasets, the authors argue that GR yields the best results when the target and source tasks are significantly different. GR also seems to reduce fluctuations in performance resulting from untuned hyperparameters.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would appreciate if the authors could try to alleviate the weaknesses I listed before by trying to delve deeper in their main insights with different techniques. \n\nI also feel that toning down the message regarding importance of hyperparameter-tuning with GR is important to improve the factuality of this work. \n\nFinally, I believe this paper could also mention/discuss prior work on layer specialization from the noisy label literature, and the example difficulty literature, e.g., (Maennel et al. 2020) and (Baldock et al. 2021).\n\n- Maennel et al. What Do Neural Networks Learn When Trained With Random Labels? NeurIPS 2020\n- Baldock et al. Deep Learning Through the Lens of Example Difficulty. NeurIPS 2021", "broader_impact_concerns": "I see no clear broader impact concerns stemming from this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3mKhBTfzR4", "reviewer": "Reviewer_mA1b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The experiments were interesting – especially the empirical benefits of GR across various datasets and head layers removed – and would be a good contribution. I liked the experiments shown in Figure 3, as well as the experiments in Section 4 that are trying to assess the impact of source/target task mismatch with the particular choice of the pretext task (positive pairs are from the same class) (Khosla et al. 2021).\n\nWeaknesses:\n- The paper claims to provide a theoretical motivation (Sections 3.1 - 3.2) as to why GR works so well in practice – this part of the paper is very limited. In particular, the analysis is a handwave-y explanation of information bottleneck (and the data processing inequality). However, it wasn’t clear to me how the information bottleneck formulation relates to the relationship between the source task and the target task. This part seems critical. Is GR saying that “processing the data less” (removing more layers of the head) leads to less compression of X via H_L, and that this is beneficial because there is more information about Y_target potentially stored in H_L rather than H_{l+1}? But if this were true, wouldn’t a somewhat silly argument here be that GR should work better as we remove more layers in the head? I think this is what the authors may have been trying to get at with the 3D experiments (with the observation that removing more layers is beneficial for improving performance as the gap between the source and target tasks increase), but it’s very hard to reason about Y_source and Y_target in this way. \n\n- There is also an observation of how certain SSL objectives such as SimCLR reduce to the traditional cross entropy loss in the full batch setting (Section 4.1). This is also not new, and the observation that SSL can also become an “image index classification task” is already known (Wu et al. 2018). There are also several existing works looking at pretext tasks that look at class membership when considering data augmentations, such as (Khosla et al. 2021).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper formalizes Guillotine Regularization (GR), a regularization technique which involves removing the last head of a pre-trained neural network to improve generalization in downstream tasks for self-supervised learning (SSL). They demonstrate that GR greatly improves performance in situations where the (pre-) training procedure is misaligned with the downstream task on Imagenet, a variety of transfer learning tasks (e.g. ImageNet -> Place205), and a synthetic 3D dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I don’t think the paper necessarily requires a theoretical explanation for why GR works well in practice. So I believe it’s fair for the submission to rephrase claims of providing theoretical justifications of the method to something more along the lines of providing conjectures. However, if the authors do actually want to corroborate their conjectures, it would be interesting to perform experiments similar to (Tian et al. 2020) where they compute neural estimates of mutual information (MI) between H_L, H_{l+1}, y_source, y_target, etc. to assess whether removing more layers in the head (e.g. representation H_L) indeed has higher MI with y_target than H_{l+1}, etc.\n\nI also think that the next version of this paper should include missing citations, such as (Wu et al. 2018), (Tian et al. 2020), and (Khosla et al. 2021). There are also a variety of mutual information-inspired SSL methods such as (Tschannen et al. 2019), (Wu et al. 2020), and (Taghanaki et al. 2021).\n", "broader_impact_concerns": "There are no immediate ethical concerns or broader impact considerations that are evident in this submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors provide a series of empirical studies of the properties of the projection head of self-supervised learning (SSL) methods, which AE think is useful for the TMLR SSL community. There are several concerns of reviewers initially (including over-claims, limited theoretical parts, lack of clarification/motivation), but the authors succeed in revising the draft for resolving them. Hence, all reviewers suggested acceptance finally. AE thinks that the results in this paper are not super novel, but some of them can be useful for SSL practitioners. Nevertheless, for preparing the camera-ready draft, AE strongly recommends the authors going through the draft again from the beginning to the end, polishing it further, e.g., fixing typos and toning-down some strong sentences for avoiding some potential confusion.
Decision: Accept (poster)