No Reason for No Supervision: Improved Generalization in Supervised Models
Authors: Mert Bülent Sarıyıldız
Yannis Kalantidis
Karteek Alahari
Diane Larlus
Venue: ICLR 2023 notable top 25% (2023)
Keywords: supervised learning
transfer learning
representation learning

Abstract: We consider the problem of training a deep neural network on a given classification task, e.g., ImageNet-1K (IN1K), so that it excels at both the training task as well as at other (future) transfer tasks. These two seemingly contradictory properties impose a trade-off between improving the model’s generalization and maintaining its performance on the original task. Models trained with self-supervised learning tend to generalize better than their supervised counterparts for transfer learning; yet, they still lag behind supervised models on IN1K. In this paper, we propose a supervised learning setup that leverages the best of both worlds. We extensively analyze supervised training using multi-scale crops for data augmentation and an expendable projector head, and reveal that the design of the projector allows us to control the trade-off between performance on the training task and transferability. We further replace the last layer of class weights with class prototypes computed on the fly using a memory bank and derive two models: t-ReX that achieves a new state of the art for transfer learning and outperforms top methods such as DINO and PAWS on IN1K, and t-ReX* that matches the highly optimized RSB-A1 model on IN1K while performing better on transfer tasks.
Code and pretrained models: https://europe.naverlabs.com/t-rex

Published as a conference paper at ICLR 2023
No REASON FOR No SUPERVISION: IMPROVED GENERALIZATION IN SUPERVISED MODELS
Mert Bulent Sariyildiz 1.2
Yannis Kalantidis'
Karteek Alahari
Diane Larlus'
NAVER LABS Europe
Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK
ABSTRACT
We consider the problem of training deep neural network on given classification task, e.g., ImageNet- IK (), so that it excels at both the training task as well as at other (future) transfer tasks. These two seemingly contradictory properties impose trade-off between improving the model's is generalization and maintaining its performance on the original task. Models trained with self-supervised learning tend to generalize better than their supervised counterparts for transfer learning; yet, they still lag behind supervised models on INIK. In this paper, we propose supervised learning setup that leverages the best of both worlds. We extensively analyze supervised training using multi-scale crops for data augmentation and an expendable projector head, and reveal that the design of the projector allows us control the trade-off between performance on the training task and transferability. We further replace the last layer of class weights with class prototypes computed on the fly using a memory bank and derive two models: t-ReX that achieves a new state of the art for transfer learning and outperforms top methods such as DINO and PAWS on INIK, and t-ReX* that matches the highly optimized RSB-AI model on INIK while performing better on transfer tasks.
Code and pretrained models: https: /eurrper.eee//ee naver www.aaa.......oommmmmmmmm..eee.e rex
INTRODUCTION
t-keX *
Mean Transfer Acc. (Log odds)
1.3
Deep convolutional neural networks trained on large annotated image sets like ImageNet- 1K (INIK) (Rus- sakovsky et al., 2015) have shown strong generaliza- tion properties. This motivated their application to broad range of transfer tasks including the recog- nition of concepts that are not encountered during training (Donahue et al., 2014; Razavian al., 2014).
DINO PAWS
New SotA (this paper)
1.2
SL-MLP
(+multt-crop)
LOOK (+mulri-crop)
1.1
Xe
Previous SotA
SupCon
Recently, models trained in a sule-super learning (SSL) framework have become popular due their ability to learn without manual annotations, as well as their capacity surpass supervised models in the con- text of transferable visual representations. SSL mod- els like MoCo (He et al., 2020), SwAV (Caron al., 2020), BYOL (Grill et al., 2020) or DINO (Caron et al., 2021) exhibit stronger transfer learning performance than models (Wightman et al., 2021) trained on the same data with annotations (Sariyildiz et al., 2021).
RSB-AI
A
75
76
78
79
80
ImageNet-IK Accuracy (%) Figure 1: We present t-ReX and t-ReX* two ResNet50 models trained with an im- proved supervised learning setup on Ima- geNet (INIK), with strong performance on both transfer learning (y-axis, averaged over 13 tasks) and INIK (x-axis).
This achievement is on the one hand exciting, as SSL. approaches do not require an expensive and error- prone annotation process, but also seemingly counter-intuitive (Wang et al., 2022b) as suggests that access to additional information, i.e., image labels, actually hinders the generalization properties of model. Models learned via SSL are however not able to match their supervised counterparts on INIK classificatio,, i.e., the concepts seen during training. Top-performing SSL and semi-supervised methods like DINO (Caron et al., 2021) or PAWS (Assran et al., 2021) still result 3-5% lower top-1 accuracy compared optimized supervised models such RSB-A1 (Wightman al., 2021).

---

Published as a conference paper at ICLR 2023
In this paper, we argue that access to more information (in the form of manual annotations) should not hurt generalization, and we seek to improve the tramsferability of encoders learned in supervised manner, while retaining their state-of-ther task. The mismatch observed between IN1K and transfer performance suggests that this goal is not trivial. It has been shown, for example, that popular regularization techniques such Label Smoothing (Szegedy et al., 2016), Dropout (Srivastava et al., 2014) or CutMix (Yun et al., 2019), which improve INIK performance, actually lead to less transferable representations (Kornblith et al., 2021; Sariyildiz et al., 2021), and that representations learned on top of models underfitting their original task transfer better (Zhang et al., 2022).
performance on the supervised training
We identify two key training components from the most successful SSL approaches that may lead to more transferable representations: multi-crop data augmentation (Caron et al., 2020) and the use of an expendable projector head, i.e., an auxiliary module added after the encoder during training and discarded at test time (Chen et al., 2020a). We study the impact of these two components on the transfer performance together with the performance on the training task, and present novel insights on the role the projector design in this context. Furthermore, inspired by recent work on supervised learning (Feng et al., 2022; Khosla et al., 2020), we introduce Online Class Means, memory memory-eficiiet variant of the Nearest Class Means classifier (Mensink et al., 2012) that computes class prototypes in "online" manner with the help of a memory queue. This further increases performance. We perform an extensive analysis on how each component affects the learned representations, and look at feature sparsity and redundancy as well as intra-class distance. We also study the training dynamics and show that class prototypes and classifier weights change in different ways across iterations.
efficient
We single out the two ResNet50 instantiations that perform best at one of the two dimensions (transfer learning and K), denoted as t-ReX and t-ReX* t-ReX exceeds the state-of-the-ar transfer learning performance of DINO (Caron et al., 2021) or PAWS (Assran et al., 2021) and still performs much better than these two on INIK classification. t-ReX* outperforms the state-of-ther of-the-art results of RSB-A1 (Wightman et al., 2021) on INIK while generalizing better to transfer tasks. We visualize the performance of these two selected models, together with those of other top-performing configurations from our setup in Fig. 1, and compare it to state-of-the-ar art supervised, semi- semi-supervised and self- supervised learning methods, across two dimensions: INIK accuracy and mean transfer accuracy across 13 transfer tasks. This intuitively conveys how the proposed training setup pushes the envelope teiniing-veruu--e versus- transsfr performance trade-off (from the "Previous SotA" region, to the "New SotA" one in Fig. 1) and offers strong pretrained visual encoders that future approaches could build on.
of the training-
Contributions. We propose supervised training setup that incorporates multi-crop data augmentation and an expendable projector and can produce models with favorable performance both on the training task of INIK and diverse transfer tasks. We thoroughly ablate this setup and reveal that the design of the projector allows to control the performance trade-off between these two dimensions, while number of analyses of the features and class weights give insights on how each component of our setup affects the training and learned representations. We also introduce Online Class Means, a prototype- based training objective that increases performance even further and gives of models for transfer learning (t-ReX) and INIK (t-ReX*).
2 RELATED WORK
Visual representations learned by deep networks for INIK classification can transfer other tasks and datasets (Donahue et al., 2014; Razavian et al., 2014). This generalization capability of networks has motivated researchers to propose practical approaches for measuring transfer learning (Goyal et al., 2019; Pándy et al., 2022; Zhai et al., 2019) or contribute to a formal understanding of generalization properties (Kornblith et al., 2019; Tripuraneni al., 2020; Yosinski et al., 2014). Recent work in this context (Kornblith et al., 2021; Sariyildiz et al., 2021) shows that the best representations for IN1K are not necessarily the ones transferring best. For instance, some regularization techniques or loss func- tions improving INIK classification lead to underwhelming transfer results. A parallel line of work based on self-supervised learning (Caron et al., 2020; Chen et al., 2020a; Grill et al., 2020) focuses on training models without manual labels, and demonstrates their strong generalization capabilities to many transfer datasets, clearly surpassing their supervised counterparts (Sariyildiz et al., 2021). 1). Yet, as expected, SSL models are no match to the supervised models the IN1K classification task itself.
2

---

Published as a conference paper at ICLR 2023
global crop
local crops
50 fe
Projector 9
Class weights
ResNer50
Lincar clasaifier
Log Reg
protonypes
C
fo (frozen)
toucan
knitted
(a) Supervised learning using multi crop and a projector.
(b) Transfer learning with a frozen model.
Figure 2: Our proposed supervised learning setup borrows multi-crop (Caron et al., 2020) and projectors (Chen al., 2020a) from SSL to train on INIK (left). The projector g is discarded after training, and the ResNet backbone f is used as feature extractor in combination with a linear classifier trained for each task, e.g., for texture classification on DTD (Cimpoi et al., 2014) (right).
few approaches tackle the task of training supervised models that also transfer well and share motivation with work. SupCon (Khosla et al., 2020) extends SimCLR (Chen et al., 2020a) using image labels to build positive pairs. As such, its formulation is close to neighborhood component analysis (NCA) (Goldberger et al., 2004). circumvents the need for large batches by adding momentum and a memory similar to MoCo (He et al., 2020). Supervised-MoCo (Zhao et al., 2021) filters out false negatives in the memory bank of MoCo using image labels, while LOOK (Feng et al., 2022) modifies the NCA objective to only consider the closest neighbors of each query image. We experimentally observe that our model design leads to better transfer than these works.
In this work, we propose an effective training setup, which leverages multi-crop augmentation (Caron al., 2020) and an expendable projector head (Chen et al., 2020a), two key components in many successful SSL approaches. Creating multiple augmented versions (a.k.a. crops) of images in a batch was first proposed by Hoffer al. (2020). Caron et al. (2020) further consider crops with different scales and resolutions in self- supervuue learning setting, creating challenging views of an image for which the model is encouraged to learn consistent representations (Assran et al., 2021; Caron et al., 2021). Recent work argues that multi-crop increases representation variance, is useful for online self-distillation (Wang et al., 2022a), and improves vision and language pretraining (Ko & Gu, 2022). We show that multi-crop over different resolutions works out-of-the-box also for supervised training on INIK.
Using features from intermediate layers of networks has been considered before, e-g., for training object detectors (Lin et al., 2017) and image classification models (Lee et al., 2015), or evaluating the transferability individual layers (Zhang et al., 2016; Gidaris et al., 2018) or groups of layers (Evci et 2022). However, selecting optimal layers for each problem is infeasible due to the computa- tional nature of this selection. SimCLR (Chen et al., 2020a) proposed instead to rely on an expendable projector, a design that is now common practice in SSL (Zhou et al., 2022; Zbontar et al., 2021), and starting to be adopted by supervised approaches like SupCon (Khosla et al., 2020) and LOOK (Feng et al., 2022). The impact of these projectors the representation quality has only seldomly been studied. Wang et al. (2022b) have looked at the impact of projectors, but only for transfer and in isolation. Our work goes one step further and studies how projectors affect performance both the training task and for transfer. We ablate many projector designs and study them jointly with multi-crop. Through our study, we uncover how useful projectors are navigating the trade-off between training and transfer performance, leading to state-of-the-an results on both dimensions.
3
AN IMPROVED TRAINING SETUP FOR SUPERVISED LEARNING
We now present an improved training setup for learning supervised models that achieve high perfor- mance on both INIK classification and diverse set of transfer tasks.
Our setup trains a model (or encoder) fe, parameterized by 0. This model encodes an image into a transferable representation x E d.d. We follow the common protocol (Feng et al., 2022; Kornblith et al., 2021) and train all variants of our model on INIK using a ResNet50 (He et al., 2016) encoder. This choice of encoder is influenced by recent observations (Wightman et al., 2021) that carefully optimized ResNet50 models perform on par with the best Vision Transformers (ViTs, Beyer et al. (2022)) of comparable size on INIK. After training our models, perform transfer learning. We
3

---

Published as a conference paper at ICLR 2023
freeze the model's parameters they are only used to produce transferable representations (x), to be appended with a linear classifier for each transfer task (e.g., INIK or any other dataset, see Fig. 2b).
Our improved training setup enriches the standard supervised learning paradigm with nulti-crop augmentation and an expendable projector head (see Fig. 2a). We train our models with one of the two following training objectives: the standard softmax cross entropy loss that learns class weights, or an online variant of nearest class means that is based on class prototypes computed on-the-fly from a memory bank. We detail all the proposed improvements below.
Multi-crop data augmentation. Caron et al. (2020) leveraged many image crops of multiple scales and different resolutions when learning invariance data augmentation in the context of SSL. Their data augmentation pipeline, termed multi-croop, is defined over two sets of global and local crops that respectively retain larger and smaller portions of an image. These crops are processed at different resolutions. We adapt this component to our supervised setup. Given an input image I, we define two scale parameters, for global and local crops, which determine the size ratio between random crops and the image I. We follow Caron et al. (2021) and resize global and local crops to 224 x 224 and 96 x 96, respectively. We extract multiple global and local crops, respectively Mg and M. Fig. 2a illustrates one global Mg = 1 and four local Mj- = 4 crops. In Sec. 4, we explore the use of multi-crop for supervised learning, and study the effect of different
under that setting.
Expendable projector head. To countervail the lack of annotations, SSL approaches tackle proxy tasks, such as learning augmentation invariance. In order to prevent the encoder from learning representations that overfit to potentially unimportant pretext task, SSL architecttss often introduce an expendable projector between the encoder and the loss function. On the contrary, for supervised learning, performance on the training task is major goal in its own right. Here, we aim to learn supervised models that perform well on the training and on transfer tasks. These two requirements are not aligned and it necessary to find a trade-off (Kornblith al., 2021).
We argue that one can control this trade-off using an additional projector in the context of supervised learning. Similar to SSL methods (Chen et al., 2020a;b; Chen & He, 2021) and to the recent SL-MLP (Wang et al., 2022b) we introduce Multi Layer Perceptron (MLP) projector as part of our supervised training pipeline. Let go IR + Rd denote this projector, parameterized by Φ) Ie is composed of an MLP with L hidden layers of d, dimensions followed by a linear projection to a bottleneck of do dimensions. Each hidden layer is composed of a sequence of a linear fully-comnected layer, a GeLUU (Hendrycks & Gimpel, 2016) non-linearity. We further apply fz-normalization to the output of 9a and optionally also the input. We illustrate this architecture in Fig. 3. Note that SL-MLP (Wang et al., 2022b) uses similar head but with only one hidden layer and no input or output f--normalzzation, so SL-MLP can be seen as a special case of our projector architecture. We compare to their design in Sec. 4 and investigate how the number and dimension of hidden layers among other design choices affect the transfer performance of the learned models, verifying and extending the findings of Wang et al. (2022b). On top of this, we study transfer performance in juxtaposition to performance on the training task, and derive the novel insight that projector design allows to control the trade-off between performance on the training task and transferability.
96 (x)
l nz-nommalization
Fully-connected (dg)
Lx
BatchNorm GELU
(loffe & Szegedy, 2015) and
Fully-connected (dp)
Eg-normalization
X
Figure 3: Architecture of the projector ga.
Cosine softmax cross-entropy loss. Incorporating both the components described above in a standard supervised learning paradigm, we can train with the standard softmax cross entropy loss using class labels. The training pipeline is illustrated in Fig. 2a. It uses multi-crop data augmentation on each input image I to produce M = M, + Mt crops I,. j 1,..., M. Each crop is individually input to the network composed of the encoder followed by the projector, and produces an embedding Zj (I,)) To predict class labels, we multiply embeddings with trainable class weights W (w. E R Te-1 where C' is the number of classes. We train the whole pipeline using the cosine softmax loss as it was shown to improve INIK performance (Kornblith et al., 2021):
LCB
1
Σ
M C
Σ
Vic
log exp(z) Twe/T) Ik=1 exp(z)" (wa/)
(I)
M
c=
4

---

Published as a conference paper at ICLR 2023
where y E 1) is the C'-dim one-hot label vector corresponding to image I, T is temperature hyper-parameter and w. = ww/ll Note that projector outputs z are already fg-normalized.
Online Class Means. Motivated by the recent success of momentum encoders online memory banks for large-scale training (He ct al., 2020), we revisit the prototype-basee Nearest Class Means (NCM) approach of Mensink et al. (2012) and introduce a memory -eficccent variant that computes class prototypes in an "online" manner with the help of a memory queue.
way of maintaining
Concretely, following Mensink al. (2012), define u, to be the class prototype class mean for class i.e., the mean of all embeddings from that class, and define U = {u,)a= Given that jointly learn class means and the embeddings, computing the exact mean at each iteration is computationally prohibitive. Instead, we formulate an online version of NCM that uses a memory bank Q which stores fz-normalized embeddings z output by the projector, similar the memory bank from MoCo (He et al., 2020). Given the memory Q, we do not learn class weights, but instead compute prototype for each class, on-the-fly, as the average of the embeddings in the memory which belong to that class. Formally, if Q. denotes samples in memory that belong to class and No = 12cl. the loss function becomes: exp(z) uu/T)
M C
LOOM = M
1
Via log
with u. = Uc u
and U. = Σz. (2)
Ek= exp(z) T ux/T))
No
==
xE
We refer to the above training objective Online Class Means or OCM. To make sure the embeddings stored in the memory remain relevant as the encoder is updated during training, we follow MoCo (He et al., 2020) and store in memory embeddings from an exponential moving average (EMA) model trailing fo and Jø- As we show in our analysis in Sec. 4.2, estimating class prototypes using only the relatively small subset of samples in the memory bank leads to class prototypes that drift more across iterations compared to SGD-optimized class weights that converge faster. Further details as well as other variants are discussed in Appendix A.
EXPERIMENTS
In Sec. 4.1, we exhaustively study the design of the main components of our setup, i.e., multi-crop augmentation, projectors, and OCM. This leads to a summary of our main findings. We then analyze the learned representations, class weights, and prototypes in Sec. 4.2. There, we explore how each component affects several facets like feature sparsity and redundancy, average coding length, as well as intra-class distance. We also study training dynamics like gradient similarity for multi-crop or how prototypes and classifier weights change across iterations for OCM. Finally, in Sec. 4.3 we plot the performance of multiple variants of the proposed training setup on the training-vewee versus- versus-transfer transfer performance plane, empirically verifying its superiority over the previous state of the art.
Protocol. All our models are trained on the training set of ImageNet- IK (INIK) (Russakovsky et al., 2015). Due to the computational cost of training models on IN1K, each configuration is trained only once. Given an IN model, we discard all the training-speciing modules (c.g., the projector Jo. the class weights W), and use the encoder fe as feature extractor, similar to Kornblith et al. (2019); Sariyildiz et al. (2021). For each dataset we evaluate on, we learn a linear logistic regression classifier with the pre-extracted features and independently optimize each classifier's hyper-paramette for every model and every evaluation dataset using Scikit-learn (Pedregosa et al., 201 and Optuna (Akiba et al., 2019) (see details in Appendix B). We repeat this process times with different random seeds and report the average accuracy (variance is negligible). Note that the feature extractor is never fine-tuned, and, because we start from pre-extracted features, no additional data augmentation is used when learning the linear classifiers. This protocol is illustrated in Fig. 2b.
Evaluation datasets and measures. We measure performance on the training task by evaluating classification accuracy on the INIK validation set. To evaluate transfer learning, we measure classification performance on 13 datasets: the 5 ImageNet- CoG datasets (Sariyildiz et al., 2021)
Although in their evaluation Caron et al. (2021): Zhai et al. (2019) train linear classifiers with data aug- mentation or fine-tune the encoder while training classifiers, we found that such protocols make a proper hyper-parameter validation computationally prohibitive. We instead follow the linear evaluation protocol from Korbllith al. (2019) and Sariyildiz al. (2021).
5

---

Published as a conference paper at ICLR 2023
Figure 4: Impact of the number of local crops (M) on the performance on INIK (left) and transfer datasets (right) when varying the number of hidden layers L in the projector and M,=1.
Table I: Impact of the projector size on per- formance, via the number of hidden layers L (left) and hidden units d,, (right). The default configuration: L=1, d,,2008,, d,=2566 and with fg-normalization of the input (highlighted rows). We use Ma=1 and Mi=8 ("Base+Mc").
Image-Net 1K- Ace. (96)
80
Mean Transfer Acc.
1.3
78
(Log odds
1.2
INIK
Transfer
dp
INIK
Transfer
L=1
1.1
L=1
Buse
76.6
0.10
512
80.0
0.82
76-
1=2
1=2
Me
79.7
0.25
1024
80.0
1.06
1.0
1=3
L
79.8 +
1.15
2048
79.8
1.15
74-
0.9
1
2
78.6
1.31
4096
79.8
1.20
0
4
8
2
6
L
3
77.5
1.33
8192
79.4
1.22
Local Crops (M₃)
Local Crops (M))
that measure concept generalization, and 8 commonly used smaller-scale datasets: Aircraft (Maji et al., 2013), Cars196 (Krause et al., 2013), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), Flowers (Nilsback & Zisserman, 2008), Pets (Parkhi et al., 2012), Food101 (Bossard et al., 2014) and SUN397 (Xiao et al., 2010). We report two metrics: Top-1 accuracy on INIK and transfer accuracy via log-odds (Kornblith et al., 2019) averaged over the 13 transfer datasets. Note that we provide details on the datasets, the exact log-odds formulation, and per dataset results in the Appendix, respectively in Tab. 4, Appendix B.3, and Tab. 7. Appendices C.7 and C.8 present additional evaluations KNIK-Sketch (Wang et al., 2019), INIK-v2 (Recht et al., 2019) and two long-tail datasets: i-Naturalist 2018 and 2019 (Van Horn et al., 2018).
Implementation details. fo is a ResNet50 (He et al., 2016) encoder, trained for 100 epochs with mixed precision in PyTorch (Paszke et al., 2019) using 4 GPUs where batch norm layers are synchronized. We use SGD optimizer with 0.9 momentum, batch size of 256, le-4 weight decay and a learning rate of 0.1 x batch /256, which is linearly increased during the first 10 epochs and then decayed with a cosine schedule. We set 7 = 0.1 and, unless otherwise stated, we use the data augmentation pipeline from DINO (Caron et al., 2021) with 1 global and 8 local crops (My I and Mj == 8). A detailed list of the training hyper-pane srpee-parameters given in the Appendix (Tab. 3). Training one of our models takes up to 3 days with 4 V100 GPUs depending on its projector configuration.
4.1 ANALYSIS OF COMPONENT DESIGN AND LYPER-PARAMETERR
Multi-crop data augmentation. We first study the effect of the number of local crops on INIK and transfer performance. We train supervised models using Eq. (1) with I global and 2, 4, 6 8 local crops, and projectors composed of 1, 2 or 3 hidden layers, and report results in Fig. 4. Our main observations are: a) training with local crops improves the performance on both IN1K and transfer tasks, and b) although increasing the number of local crops generally helps, performance saturates with 8 local crops. We set M, = 1 and M, = 8 for all subsequent evaluations. Further ablations using different scale and resolution parameters are presented in Appendix C.3.
Note that using local crops increases the effective batch size, which, in turn, increases training time. We therefore conduct two experiments to see if a longer training or a larger batch size would lead to similar gains. We train two models using a single crop sampled from a wide scale range (i.e., able to focus on both large and small image regions), one with 9x larger batch size, the other for 800 epochs. Unlike multi-crop, these models bring no significant gain (see Tab. 8 in the Appendix for details).
Expendable projector head. We study the impact of different architectural choices and hyper- parameters for the projector. We vary the number of hidden layers (L), the dimension of the hidden (d,) and bottleneck (db) layers, and whether or not to Ez-normalize the projector input (f2). We start from a default configuration: T== 1, dn = 2048, d, = 256 and with eg-normalized inputs. We ablate each parameter separately by training models optimizing Eq. (I). We use multi-crop in all cases.
The most interesting results from this analysis are presented in Tab. 1. We see that the number of hidden layers (L) is an important hyper-parameter that controls the trade-off between INIK and transfer performance. Adding a projector head with a single hidden layer not only improves the already strong INIK performance of multi-crop (Base+Mc in Tab. 1), but also significantly boosts its average transfer performance. More hidden layers seem to increase transfer performance, at the cost of decrease in IN accuracy. The same can be said about the dimension of the hidden

---

Published as a conference paper at ICLR 2023
Base
DV
Intra- mutra-elass Dist.
+Mc
Spursity (%)
80
0.9
60
0.8
+Mc+Pr
0.7
0.0
ImageNet-18 IK Acc. (%6 Mean Transfer Acc. (LO)
ImageNet-IK Acc. (%) Mean Transfer Ace. (LO)
Figure 5: Average intra-class éz-distance between samples from the same class (leff) and sparsity as the percentage of feature dimensions close to zero (right), on INIK and averaged over transfer datasets.
Gray and Orange arrows denote changes due to adding multi-crop and projectors, respectively.
Coding Lengti
2.5k
2k
Gradient Sinn.
0.8
Base (BS=2K)
Frohanius Norm
10a
Baso+Mc
1.5k
0.6
1k
0.4
10
Prototypes AU
0.5k
0.2
Weights
Basc
+Pr
+Mc
+Mc+Pr
10"
100
50
100
Epoch
Epoch
Figure 6: (leff) Average coding length per sample (Yu et al., 2020) over all transfer datasets. (middle) Average similarity between class weight gradients Vw. LCe during training. (right) Change in class weights W and prototypes U at every iteration across all classes (see text for details) for models trained using Eq. (I) and Eq. (2), respectively.
layer, yet we further see that a larger d₁ significantly increases transfer performance, and moderately decreases INIK accuracy. On the contrary, we observe that the bottleneck dimension de and input ez-normalization only have small influence on INIK and transfer performance (see Appendix C.4). Overall, our observations verify and significantly extend the ones recently presented by Wang et al. (2022b). We not only study the design of projectors jointly with multi-crop, but also analyse transfer performance jointly with performance on INIK, revealing a trade-off between the two, that fully controlled through the design of the project head.
Online class means. There are two main hyper-parametert in OCM: the size of the memory bank and the momentum of the EMA models that populate the memory bank and provide the embeddings for class prototypes. We explored momentum values 0 and 0.999 (in the former, we directly use fo and 96 to compute prototypes) and see that trailing EMA is essential for maintaining high performance, ie., momentum = 0 performs poorly, aligning with the observations for MoCo (He et al., 2020). However, unlike MoCo or other recent methods such LOOK (Feng et al., 2022), OCM does not require a large memory bank to achieve the highest performance. We experimented with memory sizes between 2048 and 65546, and found that 8192 works best (see Fig. 12b in Appendix).
4.2 ANALYSIS OF LEARNED FEATURES, CLASS WEIGHTS AND PROTOTYPES
We now investigate how different components of our setup affect training or the learned representa- tions. We analyse the features produced, class weights and prototypes from the following models: a) Base: a model trained using cosine softmax loss without multi-crop and projector, b) Base (BS=2K): Base but with 9xlarger batch size, c) Base+Mc: Base with multi-crop. d) Base+Pr: Base with projector, e) Base+ Pr, and f) OCM: a model trained using Eq. (2); details in Appendix C.6.
Intra-class distance. We start by analysing the fz-normalized features for the four models, Base, Base+Me, Base+Pr and Mase+Mc+Pr, by computing the average fz-distance between samples from the same class (i.e., intra-class distance). We see in Fig. 5 (left) that multi-crop reduces intra-class distance on INIK, while projectors increase Not surprisingly, this correlates with training task performance, i.e., lower intra-class distance translates to better performance INIK. On the transfer datasets, however, we found no strong correlation between the two, i.e., transfer performance does not necessarily depend on intra-class distance.
Sparsity. In Fig. (right) we report feature sparsity ratio, i.c., the percentage of feature dimensions close to zero for Ez-normalized features from the four models. We see that: a) the average sparsity
7

---

Published as a conference paper at ICLR 2023
I-ReX
New SotA
Mean Transfer Acc. (Log odds)
1.3
(this paper)
DINO +
PAWS
1.2
SL-MLP
(+multi-cro)
LOOK
(+multticropp)
Previous
SotA
1.1
SupCon
t-ReX*
t-ReXi
t-ReX2
t-ReX3
O-MM
L-ReX,-OCM
t-ReXg-OCM
RSB-AI x
75
76
77
78
79
80
ImageNet- IK Accuracy (%)
Figure 7: Comparison on the training task vs transfer task performance for ResNet50. We report INIK (Top-1 accuracy) and transfer performance (log odds) averaged over 13 datasets (5 ImageNet-CoG levels, Aircraft, Cars196, DTD, EuroSAT, Flowers, Pets, Food101 and SUN397)
for a large number of our models trained with the supervised training setup presented Sec. 3. Models the convex hull are denoted by stars. We compare the following state-of-the-art (SotA) models: Supervised: RSB-A1, SupCon, SL-MLP and LOOK with multi-crop: self-supervised: DINO; semi-supervised: PAWS.
ratio on the transfer datasets is inversely correlated with performance, i.e., linear classifiers trained on less sparse features achieve better transfer performunce, and b) projectors drumatically reduce sparsity. We find this last observation intuitive: features from the layer right before the cross-entropy loss encouraged to be close one-hot vector possible and therefore sparse. Introducing projectors in between allows the encoder to output less sparse features, which improves transfer.
Coding length. To further investigate our observations sparsity, follow Yu et al. (2020) and compute the average coding length per sample on the transfer datasets (see Fig. 6 (left)). We see that projectors largely increase the "information content" of representations. This was also verified by analysing singular values per dimension for models with and without projectors. We observed that the feature variance more uniformly distributed over dimensions when projector used (see Ap- pendix C.6). These observations might explain why projectors reduce overfitting to INIK concepts.
Gradient similarity. To understand why using multi-crop increases performance for the same batch size, we examine the gradients of class weights Vw for two models that have the same effective batch size, with and without multi-crop. At each training iteration, we compute the average cosine similarity between individual gradients of every pair of class weights V Vw., LCE and Vw., LeR for any e; * Cy. As we from Fig. 6 (middle), cosine similarity increases substantially with multi--ropp. In other words, on average, classifier gradients (and therefore the class weights themselves) are more entangled. We attribute this to the fact that some of the local crops (e.g.. the ones that mostly cover background and hence are not really discriminative for the class at hand) are harder to classify. This leads to a harder task and gradients of higher variance (shown also in Fig. 16 in the Appendix).
Change in class weights and prototypes. To understand the differences between the training objectives in Eq. (1) and Eq. (2), we measure how much class weights W and prototypes U change during the training phase. In Fig. 6 (right), we plot the average change over all classes by computing the Frobenius norm between before and after each iteration, i.e., A AW | W [|2 and AU [ I1-. where t is the training iteration, and W and U are the class weight w. and prototype u, lg eo-nrmmalieed per class and concatenated, respectively. Interestingly, we observe that prototypes U change orders of magnitude more than class weights W throughoul training. We believe this is because we compute class prototypes using only the small subset of images from our memory bank. The average number of samples per class INIK is 1281. whereas, average have only 8 per class in the memory bank. We argue that this prevents OCM from overfitting, leading to higher INIK performance, as show next.
8

---

Published as a conference paper at ICLR 2023
4.3 PUSHING THE ENVELOPE OF PRAINING-VERSUS-TTE
PERFORMANCE
In this section we report and analyse results from more than 100 different models trained on INIK, all different instantiations of the proposed training setup. We varied hyper-parameters such as the number of hidden layers in the expandable projector head or the training objective. The most important results depicted in Fig. 7, while an extended version of this figure is presented in Fig. 10 in the Appendix.
Previous state of the art RSB-A1 (Wightman et al., 2021) is a highly optimized supervised ResNet50 model with top performance INIK. The self-supervised DINO model (Caron et al., 2021) has shown top transfer learning performance, while also performing well on INIK. The semi-supervised PAWS (Assran al., 2021) model matches DINO in transfer performance, with improved INIK accuracy. To our knowledge, RSB-AI and DINO/PAWS are the current state-of-the-art ResNet50 models for INIK classification and transfer learning respectively. We also compare to three recent supervised models: SupCon (Khosla et al., 2020), LOOK (Feng et al., 2022) and SL-MLP (Wang et al., 2022a). For all models except LOOK and SL-MLP, we evaluate the models provided by the authors. Due to the absence of official code we reproduced LOOK and SL-MMLP ourselves, enhancing them with multi-crop. Our reproductions achieve higher performance than the one reported in the original papers. In both cases, we use projector with hidden layer.
Notations. Models trained with the basic version of the proposed training setup, i.e., using multi-crop, projector with L hidden layers and a cosine softmax cross-entropy loss are reported as t-ReX,, For models using the OCM training objective we append -OCM. Models on the "envelope" (i.e., the con- vex hull) of Fig. 7 are highlighted with star (exact configurations are in the Appendix: Tabs. 3 and 7).
Main results. Our main observations from results presented in Fig. 7 can be summarized as follows.
Pushing the envelope. Many variants from our supervised training setup "push" the envelope beyond the previous state of the art, across both axes. Several of these models improve over the state of the art on one or the other axis, but no single model outperforms all the others on both dimensions. As the number of hidden layers of the projector increases, models gradually move from the lower right to the upper left corner of the plane. This shows again that increasing the projector complexity improves transfer performance at the cost of INIK (training task) performance.
No reason for supervision. A large number of supervised variants outperform the DINO method with respect to transfer learning, while also being significantly better on INIK. We therefore show that training with label supervision does not necessarily require to sacrifice transfer learning performance and one should use label information if available.
State-of-the-art INIK performance with three simple modifications.. A number of t-ReX, models outperform the highly optimized RSB-A1 on INIK, essentially by using only three components over the "vanilla" supervised learning process that is considered standard practice: a) cosine softmax with temperature, b) multi-crop data augmentation, and c) an expendable projector. Training with class prototypes brings further gain. Given the same projector configuration, training models with the OCM objective (Eq. (2)) has small advantage over training with cosine softmax (Eq. (I)). We see that of the 6 points on the convex hull in Fig. 7 are t-ReX-OCM models.
This suggests that using class prototypes is viable alternative to learning class weights end-to-end. Introducing R-ReX and t- ReX* We single out the two instantiations that respectively excel on the transfer learning and INIK axes, i.c., t-ReX,-OCh and t-ReX,-OCM We rename them t-ReX and t-ReX*, respectively. We envision these two transferable ResNet50 models and their corresponding training setups serve as strong supervised baselines for future research transfer learning and INI,. All the
for these two models are in Tab. 3 in the Appendix.
5 CONCLUSION
We present a supervised training setup that leverages components from self-supervised learning. and improves generalization without conceding on the performance of the original task, i.e., INIK classification. We also show that substituting class weights with prototypes used an online class mean classifier over a small memory bank boosts performance even further. We extensively analyze the design choices and parameters of those models, and show that many variants push the envelope on the WNIK-transfer performance plane. This validates our intuition that image-level supervision, if available, can be beneficial to both IN1K classification and transfer tasks.

---

Published as a conference paper at ICLR 2023
This work was supported in part by MIAI@Grenoble Alpes (ANR-19-)) 19-PA- 0003), and the ANR grant AVENUE ANN-18-CC23-000
Acknowledgements.
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proc. ICKDDM, 2019.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabhat, Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In Proc. ICCV, 2021.
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain VIT baselines for ImageNet-Ik arXiv:2205.01580 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 Mining discriminative components with random forests. In Proc. ECCV, 2014.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proc. NeurIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proc. ICCV, 2021.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proc. ICML, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proc. CVPR, 2021.
Xinlci Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. (tt::2003.0429)) 2020b.
Mircea Cimpoi. Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proc. CVPR, 2014.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proc. ICML, 2014.
Alexey Dasovitskiy, Lueas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In Proc. ICLR, 2021.
Utku Evei, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2toe: Utilizing intermediate representations for better transfer learning. In Proc. ICML, 2022.
Yutong Feng, Jianwen Jiang, Minggian Tang, Rong Jin, and Yue Gao. Rethinking supervised pre-training for better downstream transferring. In Proc. ICIR, 2022.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In Proc. ICLR, 2018.
Jacob Goldherger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov. Neighbourhood components analysis. In Proc. NeurIPS, 2004.
Priya Goyal, Dhruv Mahajan. Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self-supervised visual representation learning. In Proc. ICCV, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukeuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In Proc. NewrlPS, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. CVPR, 2020.
10

---

Published as a conference paper at ICLR 2023
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. JSTAEORS, 2019.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). /60.v 2016.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: the marginal value of training the last weight layer. In Proc. ICLR, 2018.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Proc. CVPR, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. ICML, 2015.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Proc. NeurIPS, 2020.
Byungsoo Ko and Geonmo Gu. Large-scale bilingual language-image contrastive learning. arXiv:2203, 14463, 2022.
Simon Kornblith, Jonathon Shlens, and Quoe Le. Do better imagenet models transfer better? Proc. CVPR. 2019.
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why better loss functions lead less transferable features? In Proc. NeurIPS, 2021.
Jonathan Krause, Jia Deng, Michael Stark, and Fei-Fei Li. Collecting large-scale dataset of fine-grained cars. In Proc. ICCV-W, 2013.
Chen-Yo Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In Proc. AISTATS, 2015.
Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proc. CVPR, 2017.
Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. MP, 45(1), 1989.
Yi Ma, Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. PAMI, 29(9), 2007.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. 106.5:1306.5151, 2013.
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Proc. ACM-ICM. 2010.
Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Metric learning for large scale image classification: Generalizing to new classes near-zero cost. In Proc. ECCV, 2012.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Proc. ICVGIP, 2008
Michal Pándy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink. Transferability estimation using Bhattacharyya class separability. In Proc. CVPR, 2022.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. Jawahar. Cats and dogs. In Proc. CVPR, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Py Porchh: An imperative style, high-performance deep learning library. In Proc. NeurIPS, 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-lcam: Machine learning in Python. JMLR, 12, 2011.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. CNN features off-the-shelf: An astounding baseline for recognition. In Proc. CVPR-W, W, 2014.
11

---

Published as a conference paper at ICLR 2023
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Proc. ICML, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJV,, 115(3), 2015.
Mert Bulent Sariyildiz, Ramazan Gokberk Cinbis, and Erman Ayday. Key protected classification for collabora- tive learning. PR, 2020.
Mert Bulent Sariyildiz, Yannis Kalantidis, Diane Larlus, and Karteek Alahari. Concept generalization visual representation learning. In Proc. ICCV, 2021.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. JMLR, 2014,,
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proc. CVPR, 2016.
Nilesh Tripuraneni, Michael Jordan, and Chi Jin. the theory of transfer learning: The importance of task diversity. In Proc. NeurlPS, 2020.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proc. CVPR, 2018.
Haohan Wang, songwei Ge. Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Proc. NeurlPS, 2019.
Xiao Wang. Hanqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning. In Proc. CVPR, 2022a.
Yizhou Wang. Shixiang Tang, Feng Zhu, Lei Bai, Rui Zhao, Donglian Qi, and Wanli Ouyang. Revisiting transferability of supervised pretraining: an MLP perspective. In Proc. CVPR, 2022b.
Ross Wightman, Hugo Touvron, and Hervé Jégon. Resnet strikes back: An improved training procedure timm. In Proc. NewrIPS-W. W, 2021.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proc. CVPR, 2010.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features deep neural networks? In Proc. NeurlPS, 2014.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. In Proc. NeurIPS, 2020.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. In Proc. ICCV, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny, Barlow twins: Self-supervised learning via redundancy reduction. In Proc. JCML, 2021.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djo- longa, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark.
2019.
Jianyu Zhang, David Lopez-Paz Paz, and Léon Bottou. Rich feature construction for the optimization-genernee generalliznnnininnnnnn dilemma. In Proc. ICML, 2022.
Richard Zhang, Phillip Isola, and Alexci Efros. Colorful image colorization. In Proc. ECCV, 2016.
Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? In Proc. ICLR, 2021.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image BERT pre-training with online tokenizer. In Proc. ICLR, 2022.
12

---

Published as a conference paper at ICLR 2023
APPENDIX
CONTENTS
A Further details on the improved supervised training setup
13
A.I Data augmentation
13
A.2 Online Class Means (OCM)
14
A.3 Variant: Random orthogonal classifiers (I-ReX,-orth))
15
A.4 Variant: Online Component Analysis (I-ReX-OCA)
16
Further details on the evaluation process
17
B.1
Hyper-parameters for INIK training
17
B.2 Evaluation datasets
17
B.3 Evaluation metrics: the average log odds transferability score
17
B.4 Evaluation protocols
B.5 Hyper-parameters for t-ReX and t-ReX* on transfer datasets
19
B.6 List of publicly available pretrained models used for comparisons
19
Extended results and evaluations
20
C.I Results per dataset
20
C.2 Extended version of Fig. 7
21
C,3 Extended ablations on multi-crop.
................................
C.4 Extended ablations the projector design
22
C.5 Extended ablations for Online Class Means (OCM)
22
C.6 Extended analyses of learned features, class weights and prototypes
23
C.7 Results on INIK-v2 and
24
C.8 Results on clas--imbllanced transfer datasets
24
D Limitations
26
A FURTHER DETAILS ON THE IMPROVED SUPERVISED TRAINING SETUP
A.1 DATA AUGMENTATION
Our training setup for improving the generalization performance supervised models (described in Sec. 3 of the main paper) includes the multi-crop augmentation initially proposed in SwAV (Caron al., 2020). our experiments, we use the multi-crop implementation from DINO (Caron et al., 2021), which consists of three augmentation branches (two for global crops and one for local crops). The pipeline this multi-crop augmentation is illustrated Fig. Sa. We also illustrate the pipelines of the "vanilla" Torch and SimSiam augmentations that are used in some of our ablation studies in Fig. 8b and Fig. Sc respectively. For our experiments with two global crops, i.e., when Mg = in Fig. 11,
all three branches. In all the other experiments, which have only global crop M, = 1, we use two of the branches: the second global crop and the one for local crops. Tab. summarizes the parameters of the augmentation operations used in experiments.
13

---

Published as a conference paper at ICLR 2023
(DING)
(PyTocs))
(SionSian)
Tat Eloobal
2nd Gilibal
Locals
Normalise
Nocmaliae with mein and snd
RundemSolarizziion
Noratalize with diean and
Normalize with menn and sid
GuusinmBlur
RandomGinnsize
RandernCumssinllee
RendomGGunsianllon
RundomGragseale
RandomGrsyscale
RandesCinysaal
RandomGrsyscale
RundonCollelitter
RundomCooerlitter
RamkmnColoditier
Normaliz
Colorlittor
RandommHoniontttFFiiy
RandkefiierionllFiig
RandomHoorzeetiiiiiii
Randem
RandomfforiionalFlig
RanbomieeizzeCC top sizzem22:1s
RangomBeesizeo Trup (222222
RinkmBeezeee
Ranskmieeizedd rop 1.00)
RandomikeszzedCCe 224x224)
sizm(16x56)
(a) DINO augmentations with multi-crop
(b) PyTorch
(c) SimSiam
Figure 8: nata-augmentation pipelines considered in our work. We the multi-crop augmentation implemented in DINO (Caron et al., 2021) (a) as part of our improved training setup that presented Sec. of the main paper. We also compare the PyTorch (Marcel & Rodriguez, 2010; Paszke et al., 2019) (b) and SimSiam (Chen & He, 2021) (c) augmentations in our ablations, see Appendix C.3. The parameters of the operations in (a) are given in Tab. We use the default values for the operations in (b) and (c). These pipelines are implemented using torchvision (Marcel & Rodriguez, 2010) and Python Imaging Library. Accordingly, the operation and parameter names follow the conventions from these open-source libraries.
Table 2: Default parameters of the DINO augmentations used in our experiments. We imple- ment the RandomGaussianBluu and RandomSolarization using Python Imaging Library, and use the torchvision (Marcel & Rodriguez, 2010) implementations for the remaining ones. For RandomGaus- sianBlur, "radius" denotes range from which uniformly sample radius values. Note that some of these operations involve other parameters, and these cases we use their default values. The scale parameter of RandomResizedCnnn is different for t-ReX, see Tab. 3 for details.
Augmentation Operation
Parameters
RandomResizedCropp for global crops
sizc=(224 x 224), scalc=(0.4, 1.0)
RanzomReeizzeCCropp for local crops
sizc=(96 x 96), scalc=(0.05,, 0.4)
RandomLlorizontaln lorizontalFlip
probabilit=00.5
RandomColozlitter
probability=0.8. brightness=0.4, comtrast-.,, saturation=0.2, hue=0.1
RandomGrayScake
probability=0.2
probability=0.2, radius=(0.1,
RandomSolarization Normaiization
probability=0.2, threshold=128
mcan=(0.485,, 0.456, 0.406), std=(0.229, 0.224, 0.225)
A.2 ONLINE CLASS MEANS (OCM)
In Sec. of the main paper, we introduce the "online" version of the Nearest Class Mean classi- fier (Mensink et al., 2012), referred to as Online Class Mean (OCM). In this section, we give schematic illustration of this model and describe its implementation details.
Illustrations of the different loss functions. In Fig. 9 we visualize the supervised models. We train using the two loss functions defined Sec. of the main paper. Fig. 9a and Fig. 9b correspond to the models for Eq. (I) and Eq. (2) of the main paper, respectively. As seen from Fig. 9b, and as explained in Sec. 3 of the main paper, OCM follows SupCon (Khosla et al., 2020) and LOOK (Feng al., 2022) and uses the momentum network and memory queue proposed in He et al. (2020). We explain them in detail below.
Momentum encoder fe and projector 9ç. To keep memory bank of slowly evolving features, we keep an exponential moving average (EMA) the encoder fo and projector Is parameters. Concretely, momentum encoder fs and momentum projector 9c, whose parameters respectively
14

---

Published as a conference paper at ICLR 2023
C
C
softmax
softmax
W
memory bank
9
80
0
ema
(a) For Eq. (1)
(b) For Eq. (2), OCM
Figure 9: The supervised models we train using our proposed setup. Ig and Lg,1 represent only global crops or both global and local crops. Our f-ReX and t-ReX* variants have the form shown Fig. 9b, with projector configurations
(for t-ReX*) and = (for t-ReX), d, = 2048, d. 256, input Ez-normalization enabled, memory bank size Q = 8192 and with no predictor, i.e., hy is an identity mapping.
EMA of and 6, are defined as: Ç f m & m = 0.999 is the momentum parameter. As shown in Fig. 9b, we only feed global crops Ig through these momentum networks f& and dC during training. Both global and local crops g.1 are passed through the encoder fo and projector J6.
m) x and Ç x m) Φ, where
Expendable predictor hy after projector Ig. In several SSL methods with dual-network architec- tures, e.g., BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021), breaking the architectural symmetry, by adding a multi-layer perceptron one of the branches, was shown improve the generalization of representations. Following this practice, we also experimented with training OCM models optionally using an expendable predictor head he R Rd with parameters vi, added after the projector head 9 (as shown in Fig. 9b). These predictor heads contain fully-connected, normalizaaiinn (loffe Szegedy, 2015) and GeLU (Hendrycks Gimpel, 2016) layers, fol- lowed by another fully-con cullycconnected layer and &g fg-normalization. The first (resp. second) fully-connected layer maps from Rdd to Rdz (resp. Rd, to Rd.). our experiments d, is generally 2048 dimensions. See Appendix C.5 for a discussion on the impact of predictors on performance; Note that neither t-ReX nor t- ReR is trained with an expendable predictor head hg.
Memory bank. The original NCM (Mensink et al., 2012) formulation require access to the entire dataset at each SGD training iteration, which is not possible in our case. To circumvent this, we use a memory queue Q which stores & normalied momentum projector outputs Q {gc (fe (Ig) })} for global crops. In OCM, we compute a "prototype" for each class c, as the mean over all memory points from that class u. 1/x. Exeo. z, where Qe denotes samples in the queue that belong to class and N. Qel- Then, for given training crop I (can be either global or local crop), compute its predictor outputs hg (J (I))) and obtain class prediction scores by taking the inner product between this predictor output and the set of all class prototypes = {uz} 1000 as defined in Eq. (2) of the main paper. In Appendix C.5 study the impact that the size of the memory bank has on IN1K and transfer performance.
Class prototype neighbors. In Tab. 12 we provide visualization of the top-5 nearest prototypes for a random set of classes for t
A.3 VARIANT: RANDOM ORTHIOGONAL CLASSIFIERS CEEEEEEERS-RREEEEEEEEEEE (T-REX,,orrt)
In Sec. 3 of the main paper, propose the OCM variant where class weights W are replaced by class means = [u,]a which are obtained an online manner using a memory bank (see also Appendix A.2 for details). Inspired by Hoffer et al. (2018); Sariyildiz et al. (2020),
15

---

Published as a conference paper at ICLR 2023
I-ReX
New SotA (this paper)
Mcan Transfer Acc. (Log odds)
DINO
PAWS
1.2
SL-MLP (+multi-crop)
LOOK (+multi-crop)
Previous
SotA
1.1
SupCon
t-ReX;
I-ReX,
I-ReX, AReXa-OCM t-ReXa-orth
t-RM
t-ReXa-OCM t-ReXa-orhh
t-ReX
t-Reh--orth
I-ReX;-OCA
RSB-AI* x
75
76
77
78
79
80
ImageNet- IK Accuracy (%)
Figure 10: Extended version of Fig. 7 of the main paper, featuring the t-ReX-OCA and t-ReX variants. We additionally plot six -- variants with and without projector input & and L [1, 3), and four I-ReX-OCA variants with different temperature parameter 7 values and L = 1. We report INIK (top-1 accuracy) and transfer performance (log odds) averaged over 13 datasets (see Tab. 4 for the list) for a large number of our models trained with the supervised training setup presented in Sec. of the main paper. Models on the convex hull are denoted by stars. We compare to the following state-of-the SupCon (Khosla et al., 2020), and SL-MLP (Wang et al., 2022a) and LOOK (Feng et al., 2022) with multi-crop; Self-supervised DINO (Caron et al., 2021); Semi-supervised PAWS (Assran et al., 2021).
(SotA) models: Supervised RSB-A1 (Wightman et al., 2021),
explore another variant of our setup where class weights W [ 1000 are initialized with random orthogonal vectors and kept frozen while the encoder fo and projector Jo networks are trained. Our motivation is that such vectors may lead to higher class separation obtaining higher accuracy on the training task, as also noted by Korbblith et al. (2021).
To this end, without using momentum networks, predictor head or memory bank, simply optimize Eq. (1) in the main paper with "fixed" class weights, and denote these models with the --ReX,,-orte notation. Due
limited computational budget, we train 6 t-ReX, -oorth variants with and without projector input -o-om, L = [1, 2, 3]. d, 2048 and d, = 256, and plot their INIK and average transfer accuracies along with the others from the main paper Fig. 10. We observe that none of those six models were able to expand the envelope of
vernssrrr performance.
A.4 VARIANT: ONLINE COMPONENT ANALYSIS (T-REX-OCA)
Recent supervised learning methods like SupCon (Khosla et al., 2020) or LOOK (Feng et al., 2022) are variants of the soft k-NN loss introduced in Neighborhood Component Analysis (NCA) (Goldberger et al.. 2004). We therefore explore a variant of our training setup where the loss directly minimizes the log NCA probabilities. Specifically, as the OCM variant, we use memory bank Q which stores fz-normalized embeddings output by the projector instead of using the full dataset to compute the soft k-NN, as the latter would be intractable. This variant then optimizes the following loss:
Vlel log Σz.co. exp(z) exp(z]ze/T) ze/T) Ex,c9 exp(z) zn/T)²
LOCA
where M is the number of crops, C' is the number of classes, y E 10 C is the C-dim one-hot label vector, 7 is temperature for NCA probabilities and Q. denotes samples in the memory bank that refer to this training
belong class c. Since this is an "online" variant of the NCA objective, objective as Online Component Analysis or OCA. Similar to OCM, we use momentum networks in OCA, i.e. momentum encoder fs and projector gç. For more details, see Appendix A.2.
16

---

Published as a conference paper at ICLR 2023
We experimentally show in Fig. 10 that our t-ReX-OCA models achieve a balance between INIK and transfer performances, ie., they are towards the middle of the envelope in Fig. 10. Moreover, in evaluations we that this variant is better overall than LOOK difference between I. -multi-t- and the t-ReX-OCA variants is that the former restricts the soft k-NN loss (Goldberger et al., 2004) the close neighhorhood of each training sample in the memory bank. Our observation suggests that relying on more points in the memory bank is beneficial for learning better representations. We further observed that for this variant the size of the memory bank [ does not severely affect the performance of learned representations, but the smoothness of NCA probabilities computed in Eq. (3) does. Therefore, is important to tune the temperature parameter r for a given memory bank size to carefully control this smoothness.
Note that the main
B FURTHER DETAILS ON THE EVALUATION PROCESS
B. HYPEE-PARAMETERR FOR INIK TRAINING
As discuss in the previous section, train supervised models on INIK with different objectives and architecture configurations. We observe that the hyper-panee prper-parameters for IN1K training that we share in Tab. work well for the most effective models we studied.
B.2 EVALUATION DATASETS
Once we train our models on INIK, we evaluate their encoder representations fI) by training linear logistic regression (Log.Reg) classifiers on 13 transfer datasets which include 5 ImageNet-CoG levels (Sariyildiz et al., 2021) and 8 commonly used small-scale datasets: Aircraft (Maji et al., 2013), 1ars196 (Krause et al., 2013). DTD (Cimpoi al., 2014). EuroSAT (Helber et al., 2019). Flowers (Nilsback & Zisserman, 2008), Pets (Parkhi et al., 2012), Food101 (Bossard et al., 2014) and SUN397 (Xiao et al., 2010). To test the generalization of models to INIK concepts, we use the three test sets of INIK-v (Recht et al., 2019) and out of domain images of IN1K-Stttch (Wang et al., 2019). Finally, to test performance on long-tail classification tasks, we use iNaturalist 2018 and 2019 (Van Horn et al., 2018). Statistics of all these datasets are provided in Tab. 4.
When a val split is not provided for a dataset, we randomly split its train set into two, following the size of tr in and val splits from either Feng et al. (2022) or Grill et al. (2020). We also created different t train/val splits when tuning syper-parameter with different seeds, thus further increasing the robustness of our scores. Other notes the datasets
follows:
(i) For DTD (Cimpoi et al., 2014) (resp. EuroSAT (Helber et al., 2019)), there are 10 official arrannval/test (resp. train/test) splits. Following Feng et al. (2022); Grill et al. (2020), we use the first split.
(ii) For EuroSAT (Helber al., 2019), we are aware of either an official dataset split or the exact splits used in prior work, e.g., by Feng et al. (2022). So, we create random t ra n/va e/test splits following the number of samples in each split from Feng et al. (2022), ensuring that the val and test splits are balanced to contain the same number of samples for each class.
(iii) For iNaturalist 2018 and 2019 (Van Horn al., 2018), we use the official val split as the test split and create a random val 1 split each time with different seed.
(iv) We use the Reg. classifier trained on INIK for predicting image labels in the three test sets of K-v2 or the test set of K-Seetch This is because INIK-v2 and K-KeeSt are only composed of test sets and no training data is provided for them.
B.3 EVALUATION METRICS: THE AVERAGE LOG ODDS TRANSFER WINNSERRBILITY SCORE
Following Kornblith et al. (2019), compute log odds over all transfer datasets and use their average as a transferability score. This is the main metric we report in the different plots of the main paper. Denoting Recoreet and Nincorrect as the number of correct and incorrect predictions for a dataset, compute the accuracy p and log odds score as follows:
neoorett
log odds log p
P
Mecrreet + Dincorrest
p
Then we report log odds averaged over all transfer datasets. See Appendix C.2 for per-dataset top-1 accuracies and average log odd scores for the models compare in the main paper.
17

---

Published as a conference paper at ICLR 2023
Table 3: Hyper-parameters for training our models on INIK. Hyper-parameters shared by all models are given on the top part while the ones specific to t-ReX and t-ReX* are shown on the bottom part. Note that neither t-ReX nor t-ReX* trained with an expendable predictor head hg.
Configuration
Value for all models
Optimizer
SGD
Base learning rate
0.1
Learning rate rule
0.1 x batch size/250
Learning rate warmup
Linear, 10 epochs
Learning rate decay rule
Cosine schedule
Weight decay
0.0001
Momentum
0.9
Number of GPUs
4
Batch size per GPU
64
Batch size total
256
Epochs
100
Synchronized batch norms
V
Mixed precision
V
in Eqs. (I) and (2)
0.1
Augmentation pipeline from
DINO
Number of Global crops (Mg)
Number of Local crops (M))
8
Global crop resolution
224
Global crop scale range
(0.4, 1)
Local crop resolution
96
Local crop scale range
(0.05, 0.4)
Value for t-ReX
Value for t-ReX*
Projector input mo-m
V
V
Projector
3
Projector dn
2048
2048
Projector dp
256
256
Global crop scale range
(0.25, 1)
(0.4, 1)
Local crop scale range
(0.05, 0.25)
(0.05, 0.4)
Memory bank size [Q]
8192
8192
Loss function used for training
LOCM
LOCM
B.4 EVALUATION PROTOCOLS
We perform image classification on each evaluation dataset with logistic regression (Log.Reg.) classifiers following of the protocols proposed by Sariyildiz et al. (2021) (for the CoG levels) or by Kornblith et al. (2019) (for the 8 small-scale datasets). In all cases, we first extract and store (single) feature vector for each image and then learn the Log.Reg. classifiers on top of those features. Our classifiers are therefore trained without data augmentation, and this is why we report lower performance for the RSB model than the one presented by Wightman al. (2021). We extract image representations from the encoders fe by resizing an image with bicubic interpolation such that its shortest side is 224 pixels and then taking central crop of size 224 224 pixels. The protocols for training Log. Reg. classifiers on the 5 CoG levels and on the other 8 datasets are different and detailed below.
Reg.Reg. on the ImageNet- CoG levels. We apply & -onommalization to the pre-extracted features using the aublicly-available source code of Sariyildiz al. (2021), and then train Log. classifiers
18

---

Published as a conference paper at ICLR 2023
Table 4: Datasets used for training (INIK) and evaluating (the others) the quality of visual representa- tions. We report top-1 accuracy for each dataset. Further implementation details on the utilization of the datasets are in Appendix A.1. CCAS-4.0 denotes the Creative Commons Srrrbution-ShareAlie 4.0 international license.
# Train
Val
Test samples
Val provided provided
Test
Dataset
Classes
License
samples
samples
For training models
INIK
1000
1281167
50000
Research-only
For evaluating models INIK concepts
INI
1000
x 10000
Research-only
INIK-Sketch
1000
50889
Research-only
For evaluating models on transfer tasks
CoG Lı
1000
895359
223445
50000
Research-only
CoG L2
1000
892974
222814
50000
Research-only
CoG Lıg
1000
876495
218708
50000
Research-only
CoG T.4
1000
886013
221115
50000
Research-only
CoG Tap
1000
873630
218024
50000
Research-only
Aircraft
100
3334
3333
3333
Research-only
Cars196
196
5700
2444
8041
Research-only
DTD
47
1880
1880
1880
Unclear
EuroSAT
10
13500
5400
8100
Research-only
Flowers
102
1020
1020
6149
Unclear
Pets
37
2570
1110
3669
CCAS-4.0
Food101
101
68175
7575
25250
Unclear
SUN397
397
15880
3970
19850
Research-only
For evaluating models on long-tail classification tasks
iNaturalist 2018
8142
437513
24426
Research-only
iNaturalist 2019
1010
265213
3030
Research-only
using SGD with momentum = 0.9 and batch size 1024 for 100 epochs. To treat each model as fairly as possible, we set the learning rate and weight decay hyper-parameters using t t i n/val splits (val splits are randomly sampled using 20% of the original train in splits). We use Optuna (Akiba et al., 2019) and sample 30 different pairs. We train the final classifiers with these hyper-parameters on the union of the train in and val splits, and report top-1 accuracy the test splits. We repeat this process 5 times with different seeds and report averaged results.
Reg.Reg. on the smaller-scale transfer datasets. Following Kornblith et al. (2019), we train Leg.Reg. classifiers with pre-extracted features using L-BFGS (Liu & Nocedal, 1989). To this end, we use the implementation in Scikit-learn (Pedregosa et al., 2011). We set the inverse regularization coefficient ("C") each dataset using Optuna and their tra train/val splits over 25 trials. If a dataset does not have fixed validation set, then we repeat seeds and report the average result.
selection times with different
B.5 HYPER-PARAMETERS FOR T-REX AND T- T ON TRANSFER DATASETS
In Tab. 5 we present the hyper-parameters found by Optuna (Akiba al., 2019) for the Log.Reg. classifiers trained on the transfer datasets for t-ReX and t-
B.6 LIST OF PUBLICLY AVAILABLE PRETRAINED MODELS USED FOR COMPARISONS
In Sec. 4.3 of the main paper, we compare our models to several prior works which are -fate-of-the-art either for IN1K classification or for transfer learning. These include self-supervised DINO (Caron et al., 2021), semi-supervised PAWS (Assran et al., 2021) and supervised SupCon (Khosla et al., 2020), RSB-AI (Wightman et al., 2021), LOOK (Feng et al., 2022) and SL-MLP (Wang et al., 2022b). Additionally, we evaluated another self-supervised model Barlow Twins (Zbontar et al., 2021) but found it to be inferior to DINO on both INIK and transfer datasets. For all the models except LOOK and SL-MLP, authors. Since there was neither publicly-availabe model nor source code for these two models, reproduced the methods ourselves and found that they can perform significantly better when
evaluate the best ResNet50 encoders trained INIK by their respective
19

---

Published as a conference paper at ICLR 2023
Table 5: on transfer datasets found by Optuna (Akiba et al., 2019). "Learning rate" and "Weight decay" are the parameters the SGD optimizer used for training Log.Reg. classifiers as in the ImageNet-CoG protocol Sariyildiz et al. (2021). C is the inverse regularization coefficient used when training Log.Reg. classifiers with L-BFGS implemented in Scikit-learn (Pedregosa et al., 2011).
for evaluating our t-ReX and I-ReX* models using Log.Reg. classifiers
INIK
CoG
CoG Lg
CoG Lg
LoG
CoG
Aireraft
Carv196
EuroS
Flowers
Food101
SUN397
DTD
Pets
Configuration
t-ReX t-Re
Learning rate
12.8
11.0
10.8
11.0
11.0
11.0
Weight decay
2.4e-10
1.1e-8
7.3e-10
1.1e-8
1.1e-8
1.1e-8
42169
5109
1.1
9.1
14678
1778
0.4
1.1
Learning Weight decay
16.9
23.6
28.6
21.4
75.3
75.3
1.2c-8
6.3e-10
4.3e-9
1.8e-9
4e-8
4e-8
42169
14667
1.1
75
14678
42169
3.2
3.2
Table Compared models with public ResNet50 encoder weights trained on INIK by the authors.
Additional
Model
Fppcch
Repository URI. and License
Notes
DINO
http::/github.com.///. hrt.h/hhh..cm/sccccoossseeecciin Apache-2.0
800
Supe-sup
(Caron et al., 2021)
PAWS
Semi-superrised
300
(Assran al., 2021)
(With 10% of annotations)
MIT
Supervised
SupCon
(Khosla et al., 2020)
800
(with momentum encoder and memory bank)
BSD-2-Clause
RSB-AI
http://fthi...ccomm/ee//a//a/aar Apache-2.0
Supervised
(Wightman et al., 2021)
combined with multi-crop. We compare to these enhanced version which we call LOOK+multi-crpp and SL-MLP+multi-crapy In Tab. 6, give list of the compared models with public encoder weights.
C EXTENDED RESULTS AND EVALUATIONS
C.I RESULTS PER DATASET
In Tab. 7, report top-1 accuracy each transfer dataset and on INIK. Results are obtained by Log.Reg classifiers, for all the models listed in Tab. and for the ones that belong to the "convex hull" or envelope, denoted by stars in Fig. 7 of the main paper.
ImageNer-115 Acc. (96)
80
Mean Tran Acc.
1.3
Meam Than fer Acc.
0.2
78
(Log oddss
ImageNet.1K Aoc.
Log odds)
0.1
1.1
0.0
76
M2=2
M,=1
1.0
M
M,=1
M,=1
-0.1
M,=1
M
M,=2
0.2
2
2
10
10
Local Crops (M))
Local Crops (M))
Local Craps (M)
Local Cross (M))
(a) Using vanilla softmax, i.e. encoder features and class weights [we]aur and 7 = 1.0 in Eq. (1) of the main paper.
(b) Using cosine softmax. following Eq. (1) of the main paper, i.e., encoder features x and class 1000 weights [2e) are fz-normalized and 0.1.
1000 are not fz-normalized
Figure II: Ablating the number of global and local crops for different softmax losses without using a projector head, i.e., Is is identity mapping in Eq. (1) of the main paper.
20

---

Published as a conference paper at ICLR 2023
Table Top-1 Reg. accuracy per dataset. Mean LO is average log odds computed over all transfer datasets (i.e.. all datasets except INIK). In the main paper, we only plot INIK and Mean LO scores for each model. For the datasets without a fixed validation set (see Appendix B.2) we repeat each evaluation 5 times with different seeds; variance is generally negligible.
CoG L
CoG 1/
CnG La
Airematt
Cars 196
EuroSAT
Flowers
Food10
SUN397
Mean LO
IVIK
CoG
C60
DTD
Pets
Model
Previous SolA
DINO (Caron et al. 2021)
74.8
71.1
67.2
632
62.6
$7.6
62.5
67.4
77.7
97.7
95.6
88.9
78.7
66.0
1.256
PAWS (Assrun ul., 2021)
76.4
71.2
67.3
63.1
62.1
56.6
63.2
71.6
76.2
96.9
95.8
91.2
77.5
65.4
1.256
SL-MLP (Wang et al., 2022a)
75.1
70.1
66.1
61.6
60.4
54.5
63.1
70.9
75.0
96.7
94.8
91.6
74.9
63.7
1.189
Look--pd--cropp (Feng et al., 2022)
78.0
70.2
65.9
61.7
60.4
54.7
62.4
71,
73.5
96.3
94.9
93.3
75.1
64.1
1.195
SupCon (Khosla al., 2020)
78.8
69.9
64.7
60.6
59.1
53.1
57.3
60.9
74.6
95.7
91.6
92.8
62.8
1.053
RSB-AI (Wightman ol., 2021)
79.8
69.9
65.0
60.9
59.3
52.8
47.1
54.0
73.9
95.7
88.7
93.1
71.2
63.3
0.978
Our models on the convex kull in
Fig.
the
moain paper
I-ReX
78.0
72.0
68.3
63.9
63.4
57.2
67.3
74,2
77.7
97.5
96.2
92.6
66.7
1.357
t-ReX-OCM (Z=2, =K=
78.8
72.3
68.2
63.7
63.0
56.8
64.7
70.8
75.8
97.3
95.3
93.2
79.1
66.9
1.305
t-ee RReX-OCM (/=1, Ng. 13]=131K)
79.6
71.7
67.3
62.8
61.6
55.3
61.9
68.8
75.2
96.7
94.0
93.6
76.6
66.
1.224
t-ReX, (Es, [=l. 409 dG=256)
79.8
71.7
67.1
63.0
61.8
54.8
61.1
66.7
74.4
96.8
93.2
76.7
66.2
1.201
t-Rex [=1, 2048, d,=256)
80.0
66.4
62.3
60.6
53.9
67.5
75.2
96.4
91.6
75.4
65.4
1.150
LReX*
80.2
70.7
66.0
61.5
59.8
53.4
55.5
64.7
73.2
96.2
90.1
93.0
73.2
64.8
1.078
Table Varying scale and resolution of global and local crops as well as the batch size. We train models using different minimum and maximum scales for global and local crops, and batch size, Mi is the number of local crops. We use global crop. i.c., Mg = 1, for each experiment. PyTorch from Paszke et al. (2019), Chen & He (202 1) and
Simsiam and DINO augmentation pipelines Caron et al. (2021), respectively. Note that for the experiments presented in this table we train models using vanilla softmax, Appendix C.3 for discussion.
Augmentation
Global
Local Scale
Local
Batch Size
INIK
Mcan Transfer
M
Epoch
Pipeline
Scale
Resolution
Top-1
Log odds
1
PyTorch
(0.08, 1.00)
100
256
76.0
1.07
2
SimSiam
(0.20, 1.00)
100
256
76.0
1.06
DINO
(0.05, 1.00)
100
256
76.4
1.08
4
DINO
(0.05, 1.00)
100
2304
76.5
1.07
5
DINO
(0.05, 1.00)
800
256
76.5
1.13
6
DINO
(0.15, 1.00)
(0.05, 0.15)
96 x 96
8
100
256
78.4
1.19
7
DINO
(0.25, 1.00)
(0.05, 0.25)
96 x 96
8
100
256
78.6
1.21
8
DINO
(0.40, 1.00)
(0.05, 0.40)
96 x 96
8
100
256
78.9
1.23
9
DINO
(0.40, 1.00)
(0.05, 0,40)
224 x 224
2
100
256
77.5
1.08
10
DINO
(0.40, 1.00)
(0.05, 0.40)
192 x 192
2
100
256
77.5
1.10
11
DINO
(0.40. 1.00)
(0.05,
160 160
2
100
256
77.8
1.13
12
DINO
(0.40, 1.00)
(0.05,
x 128
78.1
1.17
13
DINO
(0.40, 1.00)
(0.05, 0.40)
96 x 96
2
100
256
78.3
1.19
14
DINO
(0.40, 1.00)
(0.05, 0.40)
64 x 64
2
100
256
77.8
1.17
15
DINO
(0.40, 1.00)
(0.05, 0.40)
32 x 32
2
100
256
74.8
1.02
C.2
EXTENDED VERSION OF FIG.
In Fig. 10 present extended version of Fig. in the main paper that further includes results for the variants presented in Appendices A.3 and A.4.
C.3 EXTENDED ABLATIONS ON MULTI-CROP
In See. 4.1 of the main paper, we ablate the hyper-parametee This is because cosine softmax loss, when used without projector head, suffers from overfitting to INIK, and yields much worse transfer performance, as we show Fig. 11. Note that this phenomenon has also been observed by Kornblith et al. (2021) and explained by the fact that cosine softmax increases class separability of seen concepts in the feature space which reduces transferability. Therefore, we believe that using cosine softmax loss when training "multi-crop-only" models is clearly sub-optimal to set multi-crop hyper- parameters. However, we note that, as shown in Sec. 4 of the main paper, this overfitting of cosine softmax is alleviated by projector heads. In the remaining of
multi-crop using a projector head.
21

---

Published as a conference paper at ICLR 2023
Table 9: Extended results for the effeet of the projector's design. ration, i.e., L = 1, d, 2048, de 256 and with Ez normalization of the input (corresponding to highlighted rows in the tables), and ablate L, dy and &z- n--omaalization separately, training models with Eq. (I). These two tables complement Tab. I of the main paper.
We start from a default configu-
(a) Bottleneck dimension de
(b) Hidden layer L and input &orm
dg
INIK
Transfer
I. f
INIK Transfer
128
79.8
1.14
79.4
1.17
1 V
79.8
1.15
256
79.8
1.15
2
78.6
1.33
512
79.9
1.16
2 V
78.6
1.31
1024
79.6
1.15
3
77.9
1.35
2048
80.0
1.15
3 V
77.5
1.33
this section, we train "multi-crop-only" models using vanilla softmax, i.e., encoder features x and 12200 are not nn-normalieed and T = 1.0 in Eq. (1) of the main paper.
class weights (wo)en1
In the main paper, we use (0.05, 0.4) and (0.4, 1.0) as the scale range for local and global crops, i.e., we sample scale values from these intervals. Tab. 8 provides an ablation on the maximum and minimum scales for local and global crops, respectively (see rows 6-8 in Tab. 8). We evaluate 3 different values (0.15, 0.25 and 0.4) for the maximum scale of local crops, which is also set as the minimum scale for global crops. Although the results are comparable, we see that 0.40 produces slightly better results.
We further verify if the improvements from local crops are due to the fact that a) models are trained using more variants for each image, b) the effective batch size is increased during training (as use one batch of images for each crop). To test a), we train a model for 800 epochs using single crops with scale range (0.05, 1.0), and observe that (see row 5 in Tab. 8) it performs comparably training models without multi-crop, i.c., it barely improves over training models with Py TyTrrhh and SimSiam augmentations (rows and 2) and performs similar when DINO augmentations are used without multi-crop (row 3). To test b), we train a model for 100 epochs using single crops with scale range (0.05, 1.0) but increase batch size by 9x (see row 4). Yet, we again observe no improvements over training models without local crops.
Finally, we ablate the resolution of local crops (rows 9-15 in Tab. 8). Using two local crops Mi = 2 and single global crop M, = O we try several values as for the resolution of local crops [224 x 224, 192 192, 160 x 160, 128 128, 96 96, 64 x 64,32 x 32) and find that 96 x 96 works best on both INIK and transfer datasets.
C.4 EXTENDED ABLATIONS ON THE PROJECTOR DESIGN
We present additional ablations on the projector design, complementing our discussion in Sec. 4.1 of the main paper. We analyse the effect of the bottleneck layer dimension (d&), and whether or not to & Ez-normalize the projector input (f2) to the performance on IN1K and transfer tasks. As see in Tab. 9, they have only a small influence on the performance.
C.5 EXTENDED ABLATIONS FOR ONLINE CLASS MEANS (OCM)
Varying the projector and predictor head architecture. In Tab. 1 of the main paper, we study the impact of the projector head parameters for the models we train with Eq. (1). Here, we replicate this study for the OCM variant. Specifically, we ablate the L and da parameters for the OCM model defined by Eq. (2). We show the results in Fig. 12a, and see that our observation still holds, i.e., in- creasing the complexity of projector improves transfer performance at the cost of INIK performance. We also tested the impact of predictors in OCM models, and observed similar behavior: Small predictors improve the transferability of encoder representations by sacrificing INIK performance; allowing these variants to move along the envelope. Consequently, training OCM models with no predictor, i.e., when h is an identity mapping in Fig. 9b, improves INIK performance. Indeed, our best model on INIK, i.e., t*ReX*, is a 1-ReX;-OCh, variant with no predictor head.
Varying the memory bank size. Fig. 12b shows how the size of the memory bank impacts perfor- mance for the OCM variant, where we see that even a moderately-sizee memory bank of [ = 8192, i.e., containing only 8 points per class on average is sufficient to obtain high performance on both axes.
22

---

Published as a conference paper at ICLR 2023
Mean Transfer Acc. (Log odds)
1.35
Mean Transfer Acc. (Log odds)
1.35
1004
2048
L=1, d,=2048
L=1, 40,6
400%
1.25
L=2, 20.4
1.32
=812
L=2, 4096
Q = 16388
L=3, 2048
=65536
1.15
T==3, d, 4096
1.29
78
79
77
78
ImageNet-Ile IK Accuracy (%)
ImageNet- IK Accuracy (%)
(a) Projector parameters L and di.
(b) Memory size ((2)
Figure 12: OCM ablations for (a) the projector head parameters L and d3 when Q = -8192 and (b)
the size memory bank |2] when L and d₃ = 2048.
C.6 EXTENDED ANALYSES OF LEARNED FEATURES, CLASS WEIGHTS AND PROTOTYPES
In Figs. 5 and 6 of the main paper, we compare six models: a) Base: model trained using cosine softmax loss without multi-crop and projector, b) Base (BS=2K): Base but with 9 larger batch size, c) Mase+Mc: Base plus multi-crop. d) Base+Pr: Base plus projector, e) Base+Mc+Prr and OCM: model trained using Eq. (2), from multiple aspects to get more insights on how our proposed training setup improves performance. We extend these analyses below.
When comparing models, we use either features extracted using these models on each dataset, their class weights, or their prototypes. For the analyses with features, we use only the samples in the test splits of INIK and CoG levels, but all the samples in the others (as these datasets small-scale).
Pairwise &2 distnnee We fo-normalize features and compute average pairwise di-distance either between samples from the same class (intra-class) or between all samples in a dataset (all-sample). In Fig. (left) of the main paper, we show intra-class distance for the four models (a, c, and e) and for completeness, in Fig. 13 (left), we provide all-sample lg -ddisttacsss for them.
Sparsity. We compute sparsity as the percentage of feature dimensions close to zero. To compute this metric, we apply threshold to individual dimensions of fz-normalized features. In Fig. 5 (right) of the main paper, we show the results obtained with c = 10 but we found that the conclusions obtained with range of values logarithmically sampled between 10 and 10 were consistent.
Coding length. We compute average coding length (Ma et al., 2007; Yu et al., 2020) of samples in a dataset as R(X,<) <) 1/2 log det Ia + d/Ne>X X). where Ia is d-by-d identity matrix, e² is the precision parameter set to 0.5 and X E R Nxd is a feature matrix containing N samples each encoded into a d-dimensional representation (2048 in our case). In Fig. (left) of the main paper, we show average coding length for the four models (a. c, d and e) obtained on transfer datasets (i.e., coding length values are further averaged over transfer datasets). In Fig. 14 we show coding length for the same four models plus DINO Caron et al. (2021) obtained on each dataset, separately. Moreover, we compute singular values per dimension for models with and without projectors (Base+Pr and Base+Mc). For each model, we compute singular values on each transfer dataset which are normalized by their sum so that they sum and average them over transfer datasets. As can be seen in Fig. 13 (right), feature variance is more uniformly distributed over dimensions when a projector is used.
We then sort these normalized singular values by decreasing order,
Gradient similarity and variance. To get better understanding on why multi-crop improves performance significantly for the same effective batch size, we compare the models Base+Mc and Base (BS=2K) by analyzing gradients of their class weights V LeE each SGD update. We compute absolute cosine similarity between "class gradients" for a pair of classes e; and ej as |sim(V Vo,, LCE, V Vw., fCE)], where sim(:,) denotes cosine similarity. Then average them over all class pairs (C,, cj) ~ P in a dataset, which is reported in Fig. 6 (middle) of the main paper. Additionally. in Fig. 16 we show Frobenius norm and standard deviation computed over V w LeB. We observe that norm and variance of gradients increase with multi-crop, which we believe is due to increasing variance in embeddings as noted by Wang al. (2022a).
23

---

Published as a conference paper at ICLR 2023
All-sample Distance
Base Me +Pr
Singular Value
10"4
1.2
Mcc+Pr
10-
1.1
1.1
10
74
78
50
0.5
500
1,000
ImageNet-1K Acc. (%)
Transfer Acc. (L.))
Feature Dim. Index
Figure 13: (left) All-sample le distance vs. performance IN1K and transfer datasets. These two sub-plots extend the intra-class distance plots shown in Fig. (left) of the main paper. Gray and Orange arrows denote changes due adding multi-crop and projectors, respectively. (right) Singular values across dimensions, averaged over the transfer datasets. We show the first 1000 dimensions (of 2048) for clarity.
Base
Basc+Pr
Basc+Mc
Basc+Mc+Pr
DINO
IN1K
L
L
L
2000
2000
2000
2000
2000
Ls
Aircraft
Cars196
EuroSAT
2000
2000
2000
1000
Flowers
Pets
Food101
SUN397
2000
2000
2000
Figure 14: Average coding length (Yu et al., 2020) obtained on each dataset separately. See Ap- pendix C.6 for a description of models. In Fig. (middle) of the main paper, we show average coding length over the transfer datasets (excluding INIK) for all the models except DINO.
Feature redundancy. In addition the metries described above, following Wang et al. (2022b), compute "redundancy" features X g (defined as in the "coding length" paragraph above) R = 1/8 E, E, Ip(X., X,,),, where, p(X.,;, X.,) is the Pearson correlation between pair of feature dimensions and j. In Fig. 15 we show redundancy score for the four models (a, d and e) plus DINO Caron et al. (2021) obtained on each dataset, separately.
C.7 RESULTS ON INIK-v2 AND INIK-SKETCH
We compare t-ReX and t-ReX* to the previous state of the art on INIK-v2 and INIK-Sketch. As before, for each model, classifier trained on INIK and apply it directly to the test images of INIK-v2 and INJK-Sketch. Note that there 3 test sets for IN 1-12 and we evaluate over all of them. Tab. 10 presents our results. Looking at the mean top-1 accuracy over the three test sets of INIK-v2, we observe that t-ReX* also matches the performance of RSB-AI on INIK-v2 outperforming all others, showing strong domain generalization capabilities. On the other hand, SupCon performs the best on INIK-Sketch, where t-ReX* is the second best. We think that the contrastive loss used SupCon might have improved its out-of-distribution robustness for the training concepts.
the trained encoder
feature extractor, and
the linear
C.8 RESULTS ON LLASS-IMBALANCED TRANSFER DATASETS
We evaluate the long-tail transfer classification performance of DINO, PAWS, RSB-AL, t-ReX and t-ReX* on two imbalanced datasets, iNaturalist 2018 and iNaturalist 2019 (Van Horn et al., 2018). For these evaluations, follow the Log. protocol from the MmageNet-CoCi benchmark
24

---

Published as a conference paper at ICLR 2023
Base A Base+Pr
Base+Mc
Base+Mc+Pr
DINO
IN1K
1.2
1.3
44
0.05
0.05
0.00
0.00
0.00
0.00
0.00
Ls
Aircraft
Cars 196
DTD
EuroSAT
0.1
0.1
0.05
0.1
0.05
0.0
0.00
Flowers
Pets
Food101
SUN397
0.1
0.05
0.0
0.00
0.00
0.00
Figure 15: Feature redundancy scores (Wang et al., 2022b) obtained on each dataset separately. See Appendix C.6 for description of models.
Base (BS=2304)
Base+Mc
Cosine Similarity
Std
Norm
0.0004
0.75
0.50
0.4
0.0002
0.25
0.2
0
100
100
100
Epoch
Epoch
Epoch
Figure 16: (left) Average cosine similarity between gradients of class weights Vw, LCE (also shown in Fig. 6 (middle) of the main paper). (middle and right) Standard deviation and Frobenius norm, respectively, computed gradients Vw LeF. See Appendix C.6 for description of models.
Table 10: Results on IN1K concepts. For each model, we report results on the original IN1K "Val" set (the x-axis of Fig. 7), as well as on the test of K-KK-sketch (Wang et al., 2019) and K-1 (Recht et al., 2019), using in all cases the encoder, and the linear classifier trained on the original INIK training set.
INIK
INIK
INIK-v2
Model
Val
Sketch
Matched-frequency
Threshold-0.7
Top-images
Mean
DINO
74.8
19.8
61.9
71.2
76.7
69.9
PAWS
76.4
24.2
63.6
73,0
78.3
71.6
LOOK multi-crop
78.0
27.8
65.8
75.3
80.7
73.9
SupCon
78.8
30.9
66.8
75.5
80.5
74.3
RSB-AI
79.8
27.9
68.
76.6
81.6
75.4
t-ReX
78.0
26.8
65.6
74.9
80.2
73.6
t-ReX
80.2
29.1
69.0
77.5
82.0
76.2
25

---

Published as a conference paper at ICLR 2023
Table 11: Transfer results on long-tail classification. For each model, train linear classifiers on the iNaturalist 2018 and iNaturalist 2019 datasets ( (Van Horn et al., 2018) with class-imbalanced data.
Model
iNaturalist 2018 iNaturalist 2019
DINO
41.9
51,4
PAWS
40.8
49.8
RSB-AI
34.9
43.2
t-ReX
45.8
54.2
t-ReX*
36.0
44.2
(see Appendix B.4). Results are reported in Tab. 11. We see that our t-ReX and t-ReX* models still outperform RSB-Al and DINO respectively, despite challenging long-tail class distribution.
D LIMITATIONS
Requirement for annotations. Firstly, our training setup tailored for supervised learning, and therefore, its performance on both training and transfer tasks depends on the availability of high- quality and diverse annotations for the training images. Image-level annotation, when involving a large number of potentially fine-grained classes is expensive and error- error-prone prone process. In this work, we show that given a large-scale curated and annotated dataset, more precisely given INIK, which is composed of 1.28M images annotated for 1000 different concepts, it is possible to learn more generic representations than self- or semi-supervised models. When only a handful concepts annotated, when annotations contain much noise, these conclusions might not be accurate anymore, and both the pretraining classification task and the transfer tasks results might be degraded. In this case, self- or vemi-supervisse approaches might become more relevant. Yet, those scenarios are out the scope of our study.
Specitic encoder architecture. Secondly, we develop our training setup based on a single encoder architecture, ResNet50, and do not test it on other architectures. This was motivated by the fact that ResNet50 still very commonly used architecture. Also, we note that our training setup components, multi-crop, a data- augmentaionnnnno operator, and the expendable projector, added after the encoder, are architecture-agnotiic so those contributions can be seamlessly applied to any other archi- tecture of choice. Therefore, it is reasonable to expect that our setup would consistently improve other architecture familics, such as Vision Transformers ( (ViTs) Dosovitskiy et al. (2021). In fact, both com- ponents were previously successfully used with ViTs for self self-supervieed learning Caron et al. (2021). We leave studying the applicability of our training setup to other encoder architectures as future work.
26

---

Published as a conference paper at ICLR 2023
Query concept
5 nearest concepts (4 samples for each), according cosine similarity
Goldfish
Coral reef
Tench
Roch beauty
Axolotl
Snowplow
Fire salamander
Common newl
Spotted salamander
Eft
Tailed frog
Tree frog
Peacock
Cock
Gallinule
Vulture
Hen
Bustard
Flamingo
Crane
Spoonbill
Pelican
American egret
Black swan
Papillon
Japanese spaniel
Chihuabua
Pomeranian
Blenheim spaniel
Toy terrier
Whippet
Italian greyhound
Gazelle hound
Great Dane
Ibizan hound
Russian wolfhound
Jaguar
Leopard
Cheetah
Snow lcopard
Tiger
Cougar
Table 12: Visualization of the nearest neighbors of random class prototypes for t-ReX*, For each
row, the first column is the query concept and the next 5 columns the closest other concepts according to cosine distance of their prototypes. All concepts are from NIK and we show their images in the validation set.
27


Reviews:
[{"review_id": "y75KQCpCKCw", "reviewer": "Reviewer_p7Yg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about the performance of models trained on ImageNet-1K. The authors take the premise that performance on IN-1K and on other transfer tasks are desirable, and they examine how to achieve these two objectives simultaneously. They have traditionally been achieve distinctly by supervised and self-supervised approaches, respectively. The main concrete contribution is a suite of enhancements to a supervised model.", "summary_of_the_review": "Not sure about the scientific value of this paper. But the contributions should nevertheless be of interest to some of the audience, and I can see no obvious flaws. Therefore I recommend acceptance!", "strength_and_weaknesses": "### Strengths (S) and weaknesses (W)\n\n- S1: Interesting setting: focus on the *trade-off* between performance on training task / transfer tasks.\n- S2: The empirical evaluation is extensive and do adequately support the claims.\n- S3: The proposed enhancements to a supervised model lead to two SOTA models on each of the two settings, with competitive performance on the other respectively.\n\n- (W1) This is mostly an empirical/engineering paper. A lot of different pieces are put together and evaluated empirically. There is surely value in these results and SOTA models, but I am not sure what findings are applicable/relevant beyond IN-1k. Can the authors comment on this?\n- (W2) What does \"cosine\" mean in \"cosine softmax\"? The only difference with a standard softmax seems to be a summation over crops, and an additional vector of learned class-wise weights, that follow an L2 normalization of the outputs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "wtyz2CJkgB", "reviewer": "Reviewer_63Ba", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Motivated both by the fact that Self-Supervised Learning (SSL) models have better transferability than supervised models and by the expectation that additional information on labels should not impair generalization performance, this paper attempts to achieve both good supervised classification accuracy and good transferability to other tasks. Regarding the trade-off between good performance on the training task and transferability, the paper carried out extensive experiments and claims that multi-crop data augmentation is one of the key ingredients for transferability. It also finds that the trade-off can be controlled by the design of the projector head. Deeper head leads to better generalization and shallower one leads to better IN1K performance. The paper also incorporates a class prototype to improve performance and achieves SOTA regarding training-versus-transfer performance. ", "summary_of_the_review": "Overall, the paper was an enjoyable read. Although only empirical findings are presented, the experiments are careful and reliable. The findings on transfer learning will be valuable to a wide range of audience.", "strength_and_weaknesses": "The strengths of this paper are (1) finding that generalization can be obtained from multi-crops data augmentation (DA) that SSL is based on, (2) finding that deeper head leads to better generalization and shallower head leads to better IN1K performance, (3) presenting SOTA models (t-ReX and t-ReX*) for the training task vs transfer task performance, and (4) an interesting comparison between the use of class weights and the use of prototypes for training, showing that prototypes with a memory bank performs slightly better.\n\nThe first and second findings are supported by several careful analyses as well as direct experiments. They show that the intra-class L2 distance between samples increases with the projector and decreases with the multicrop DA, and that the fraction of feature dimensions close to zero (which they call sparsity) is larger without the projector. These results indicate that deeper projectors improve generalization performance because the representation is more distributed and less sparse. Those analyses were interesting to me. The SOTA models will be released, and I think the models as well as the details of the learning presented in this paper will be valuable to the community.\n\nThe weaknesses of this paper are that the methodology is one that the community is already familiar with and the findings are still based on experiments, so the findings have not been shown to be theoretically correct. We can make hypotheses, but we do not know if the findings are correct for other large data sets.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_1A1N8qAyNW", "reviewer": "Reviewer_yEpr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors revisit the paradigm of supervised learning with recent successful breakthroughs in self-supervised learning, including multi-crop data augmentation [Caron et al., 2020], expendable projector head [Chen et al., 2020a], and a variant of nearest class means classifier [Mensink et al., 2012]. They found that those components, while working well with self-supervised settings, also work well with conventional supervised settings. Combined with those 3 components, the authors show improvement over other supervised, self-supervised, and semi-supervised approaches for ResNet-50.", "summary_of_the_review": "Please provide a short summary justifying your recommendation of the paper.\nThe proposed modifications are simple and easy to understand. 3 components used in this work are well known from self-supervised learning and other fields. I do not see major novelty in the current form of the manuscript and encourage the authors to expand more on the class-imbalanced setting to see if the proposed changes can benefit for that scenario. \n", "strength_and_weaknesses": "Strength:\n- The proposed modifications are straightforward and easy to follow. They are easily reproducible as most of the components have been released by previous works.\n- Various figures (2 & 3) are helpful in comprehending the proposed changes.\n- I appreciate the ablation study of those 3 components with detailed performance analysis.\n\nWeaknesses\n- As discussed in related work by the authors themselves, all 3 components have been proven useful. The multi-crop data augmentation and expendable projector head are now widely used in various self-supervised learning approaches. Directly applying and combining 3 well known components in my opinion does not necessarily guarantee meeting the novelty bar at ICLR.\n- For example, since AlexNet, VGG, and ResNet, various different image cropping methods have been proposed. With the recent semi-supervised learning advances, we now know that more aggressive image cropping can generally make the training task harder, making the network learn harder and generalize better. The authors found that multi-crop data augmentation from [Caron et al 2020] also works well on ResNet under supervised learning setting should not be a huge surprise.\n- Also, expendable projector head essentially forces the downstream tasks to use feature representations from the earlier layer (not the last layer) has also been largely explored such as [A1, A2]. One can essentially use any combination of any previous layers for the downstream task training. \n- The major challenge in transferring features learned from supervised learning is that it generally does not perform well on class-imbalanced downstream tasks. This is due to during supervised learning, the features are learned to tailor the class distribution from the pre-training set. It would be great that the authors can shed more light on this for any future improvement of the manuscript.\n\n[A1] Lin et al. Feature Pyramid Networks for Object Detection. CVPR’17\n[A2] Lee et al. Deeply-Supervised Nets. AISTATS’15\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZBL060eAc3", "reviewer": "Reviewer_o6eH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a supervised learning framework, named T-Rex, which is a combination of several training strategies, including 1) multi-scale cropping for data augmentation 2) a better designed projection head 3) an auxiliary loss function based on protypical contrastive learning. With the combination of these training strategies, the proposed method achieved improved performance on transfer learning and supervised learning benchmark on image classification tasks. The paper further analyse each component of the proposed training strategy and its corresponding hyper-parameters, shines some insights of the effectiveness of the overall improvement.", "summary_of_the_review": "The paper is improved compared to the last version. The paper is clear in terms of explaining and providing detailed experiments on each training trick. The technical contribution is limited.", "strength_and_weaknesses": "For the full transparency, I reviewed the prior version of this work. The paper was rejected was mainly because of i) lack of insights of the motivation of each design and ii) lack the detailed analysis of each method. I am glad the current version has mostly resolved these issues and expanded the experiments significantly with more analysis and visualisations.\n\nStrength.\n\n++ A comprehensive analysis of training tricks for supervised image classification, which may benefits the community.\n\n++ A detailed hyper-parameter ablative study on each training strategy to understand its contribution to the overall improvement.\n\nLimitation.\n\n-- All proposed training strategy is not new, and therefore the technical contribution is rather limited.\n\n--The comparison for DINO and PAWS might be a bit unfair, since they are unsupervised learning methods?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%