Efficient Training of Visual Transformers with Small Datasets
Authors: Yahui Liu
Enver Sangineto
Wei Bi
Nicu Sebe
Bruno Lepri
Marco De Nadai
Venue: NeurIPS 2021 Poster (2021)
Keywords: Vision Transformers
Transformers
Computer vision
Self-supervision

Abstract: Visual Transformers (VTs) are emerging as an architectural paradigm alternative to Convolutional networks (CNNs). Differently from CNNs, VTs can capture global relations between image elements and they potentially have a larger representation capacity. However, the lack of the typical convolutional inductive bias makes these models more data hungry than common CNNs. In fact, some local properties of the visual domain which are embedded in the CNN architectural design, in VTs should be learned from samples. In this paper, we empirically analyse different VTs, comparing their robustness in a small training set regime, and we show that, despite having a comparable accuracy when trained on ImageNet, their performance on smaller datasets can be largely different. Moreover, we propose an auxiliary self-supervised task which can extract additional information from images with only a negligible computational overhead. This task encourages the VTs to learn  spatial relations within an image and makes the VT training much more robust when training data is scarce. Our task is used jointly with the standard (supervised) training and it does not depend on specific architectural choices, thus it can be easily plugged in the existing VTs. Using an extensive evaluation with different VTs and datasets, we show that our method can improve (sometimes dramatically) the final accuracy of the VTs. Our code is available at: https://github.com/yhlleo/VTs-Drloc.

Efficient Training of Visual Transformers with Small Datasets
Yahui Liu University of Trento Fondazione Bruno Kessler yahui diuuunitt.ii
Enver Sangineto University of Trento
Wei Bi Tencent AI Lab
enver wsaggineto@uiitn.it
victoriiabireeneentcoo com
Nicu Sebe University of Trento niculae. www.laeesseeeunit...
Bruno Lepri
Marco De Nadai
Fondazione Bruno Kessler depri@fbk..
Fondazione Bruno Kessler
work@marcodena. it
Abstract
Visual Transformers (VTs) are emerging as an architectural paradigm alternative to Convolutional networks (CNNs). Differently from CNNs, VTs can capture global relations between image elements and they potentially have larger representation capacity. However, the lack of the typical convolutional inductive bias makes these models more data hungry than common CNNs. In fact, some local properties of the visual domain which are embedded be learned from samples. In this paper, empirically analyse different VTs, comparing their robustness in small training set regime, and we show that, despite having comparable accuracy when trained on ImageNet, their performance on smaller datasets can be largely different. Moreover, we propose an auxiliary self- supervised task which can extract additional information from images with only negligible computational overhead. This task encourages the VTs to learn spatial relations within image and makes the VT training much more robust when training data is scarce. Our task is used jointly with the standard (supervised) training and it does not depend on specific architectural choices, thus it be easily plugged in the existing VTs. Using an extensive evaluation with different VTs and datasets, show that method improve (sometimes dramatically) the final accuracy of the VTs. Our code is available at: https: //gi thub. com/ yhlleo/VTss Drloc.
CNN architectural design, in VTs should
Introduction
Visual Transformers (VTs) are progressively emerging architectures in computer vision as an alter- native to standard Convolutional Neural Networks (CNNs), and they have already been applied to many tasks, such as image classification [17, 53, 61, 35, 58, 60, 33, 59], object detection [4, 66, 14], segmentation [50], tracking [36], image generation [30, 28] and 3D data processing [65], to mention few. These architectures are inspired by the well known Transformer [55], which is the de facto standard in Natural Language Processing (NLP) [15, 45], and one of their appealing properties is possibility to develop a unified information pwr.oosiinooccnsnnnngo paradigm for both visual and textual domains. A pioneering work in this direction is VIT [17], in which image is split using grid of non-overlapping patches, and each patch is linearly projected the input embedding space, so obtaining "token" After that, all the tokens are processed by series of multi-head attention and foed-forward layers, similarly to how (word) tokens are processed in NLP Transformers.
Work done intern at the Tencent AI Lab.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

A clear advantage of VTs is the possibility for the network to use the attention layers to model global relations between tokens, and this is the main difference with respect to CNNs, where the receptive field of the convolutional kernels locally limits the type of relations which can be learned. However, this increased representation capacity comes at a price, which the lack of the typical CNN inductive biases, based on exploiting the locality, the translation invariance and the hierarchical structure of visual information [35, 58, 60]. As a result, VTs need a lot of data for training. usually more than what is necessary to standard CNNs [17]. For instance, ViT is trained with JFT-300M [17]. a (proprictary) huge dataset of 303 million (weakly) labeled high-
images, and performs worse than ResNets [24] with similar capacity when trained on ImageNet- 1K (~ 1.3 million samples [48]). This is likely due to the fact that ViT needs to learn some local proprieties of the visual data using more samples than CNN, while the latter embeds these properties in its architectural design [47].
To alleviate this problem, a second generation of VTs has very recently been independently proposed by different groups [61, 35, 58, 60, 59, 33, 28]. A common idea behind these works is to mix convolutional layers with attention layers, in such a way providing local inductive bias to the VT. These hybrid architectures enjoy the advantages of both paradigms: attention layers model long-range dependencies, while convolutional operations can emphasize the local properties of the image content. The empirical results shown in most of these works demonstrate that these recond-generation VTs can be trained on ImageNet outperforming similar-size ResNets on this dataset [61, 35, 58, 60, 59, 33]. However, it is still not clear what is the behaviour of these networks when trained medium-smal datasets. In fact, from an application point of view, most of the computer vision tasks cannot rely on (supervised) datasets whose size is comparable with (or larger than) ImageNet.
In this paper, we compare to each other different -econd-generation VTs by either training them from scratch or fine-tuning them on medium- small datasets, and we empirically show that, despite their ImageNet results are basically on par with each other, their classification accuracy with smaller datasets largely varies. We also compare VTs with same capacity ResNets, and we show that, in most cases, VTs can match the ResNet accuracy when trained with small datasets. Moreover, we propose to use an auxiliary self-supervised pretext task and corresponding loss function to regularize training in a small training set or few epochs regime. Specifically, the proposed task is based on (unsupervised) learning the spatial relations between the output token embeddings. Given an image, we densely sample random pairs from the final embedding grid, and, for each pair, we ask the network to guess the corresponding geometric distance. To solve this task, the network needs to encode both local and contextual information in each embedding. In fact, without local information, embeddings representing different input image patches cannot be distinguished the one from the others, while, without contextual information (aggregated using the attention layers), the task may be ambiguous.
Our task is inspired by ELECTRA [12], in which the (NLP) pretext task is densely defined for each output embedding (Section 2). Clark et al. [12] show that their task is more sample-efficiee than commonly used NLP pretext tasks. and this gain is particularly strong with caall-capacity models or relatively smaller training sets. Similarly, we exploit the fact that an image is represented by a VT using multiple token embeddings, and we use their relative distances to define a localization task over a subset of all the possible embedding pairs. This way, for a single image forward pass, we can compare many embedding pairs with each other, and average our localization loss over all of them. Thus, our task is drastically different from those multi-crop strategies proposed, for instance, in SWAV [6], which need to independently forward each input patch through the network. Moreover, differently from "ordering" based tasks [42], we can define pairwise distances on a large grid without modeling all the possible permutations (more details in Section 2).
Since our auxiliary task is self-supervised, our dense relative localization loss (Carloe) does not require additional annotation, and we use it jointly with the standard (supervised) cross-entropy as regularization of the VT training. Lartoe is very casy-to-be-reproducee can largely boost the accuracy of the VTs, especially when the VT is either trained from scratch on a small dataset, or fine-tuned on a dataset with a large domain-shift with respect to the pretraining ImageNet dataset. In our empirical analysis, based on different training scenarios, variable amount of training data and different VT architectures, Lartoe has always improved the results of the tested baselines, sometimes boosting the final accuracy of tens of points (and up to 45 points).
and, despite this simplicity, it
In summary, our main contributions are:
1. We empirically compare to each other different VTs, showing that their behaviour largely differs when trained with small datasets few training epochs.
2

---

2. We propose a relative localization auxiliary task for VT training regularization.
3. Using extensive empirical analysis, we show that this task is beneficial to speed-up training and improve the generalization ability of different VTs, independently of their specific architectural design or application task.
2
Related work
In this section, we briefly review previous work related to both VTs and self-supervised learning.
Visual Transformers. Despite some previous work in which attention is used inside the convolutional layers of CNN 26], the first fully- transforree architectures for vision are iGPT [8] and ViT [17]. The former is trained using "masked-pixxl" self-supervised approach, similar in spirit to the common masked-word task used, for instance, in BERT [15] and in GPT [45] (see below). On the other hand, ViT is trained in a supervised way, using a special "class token" and classification head attached to the final embedding of this token. Both methods are computationally expensive and, despite their very good results when trained huge datasets, they underperform ResNet architectures when trained from scratch using only ImageNet-IK [17, 8J. VideoBERT [51] is conceptually similar to iGPT, but, rather than using pixels as tokens, each frame of video is holistically represented by feature vector, which is quantized using an off-the-shelf pretrained video classification model. DeiTT [53] trains ViT using distillation information provided by a pretrained CNN.
The success of VIT has attracted a lot of interest in the computer vision community, and different variants of this architecture have been recently used in many tasks [53, 50, 30, 11]. However, as mentioned in Section 1, the lack of the typical CNN inductive biases in VIT, makes this model difficult recond-generatinn of VTs
to train without using (very) large datasets. For this reason, very recently, has focused on hybrid architectures, in which convolutions are used jointly with long- range attention layers [61, 35, 58, 60, 59, 33, 28]. The common idea behind all these works is that the sequence of the individual token embeddings can be shaped/reshaped in a geometric grid, in which the position of each embedding vector corresponds a fixed location in the input image. Given this geometric layout of the embeddings, convolutional layers can be applied to neighboring embeddings, so encouraging the network focus on local properties of the image. The main difference among these works concerns where the convolutional operation is applied (e.g.. only in the initial representations [61] or in all the layers [35, 58, 60, 59, 33], in the token to query/key/value projections [58] or in the forward-layers [60, 33, 28]. etc.). In most of the experiments of this paper, we use three of VTs for which there is public implementation: T2T [61], Swin [35] and CvT [58]. For each of them, we select the model whose number of parameters is comparable with a ResNet-50 [24] (more details in Section 3 and Section 5). We do not modify the native architectures because the goal of this work is to propose a pretext task and a loss function which can be easily plugged in existing VTs.
the-art second-generateen
Similarly to the original Transformer [55], in ViT. an (absolute) positional embedding is added to the representation of the input tokens. In Transformer networks, positional embedding is used to provide information about the token order, since both the attention and the (individual token based) ferd-forward layers are permutation invariant. In [35, 591, relative positional embedding [49] is used, where the position of each token is represented relatively to the others. Generally speaking, positional embedding is a representation of the token position which is provided as imput to the network. Conversely, our relative localization loss exploits the relative positions (of the final VT embeddings) as a pretext task to extract additional information without manual supervision.
Self-supervised learning. Reviewing the vast self-supervised learning literature is out of the scope of this paper. However, we briefly mention that self-supervised learning was first successfully applied in NLP, as a means to get supervision from text by replacing costly manual annotations with pretext tasks [37, 38]. A typical NLP pretext task consists in masking a word in an input sentence and asking the network to guess which is the masked token [37, 38, 15, 45]. ELECTRA [12] is a cumple-efficien language model in which the masked-token -okkn pretext task is replaced by discriminative task defied over all the tokens of the input sentence. Our work is inspired by this method, since we propose a pretext task which can be efficiently computed by densely sampling the final VT embeddings. However, while the densely supervised ELECTRA A task is obtained by randomly replacing (word) tokens and using a pre-trained BERT model to generate plausible replacements, we do not need pre-trained model and we do not replace input tokens, being task based on predicting the
3

---

inter-token geometric distances. In fact, in NLP tasks, tokens are discrete and limited (e.g., the set of words of apecific-language dictionary), while image patches are "continuous" and highly variable, hence replacement-basee based task is hard to use in a vision scenario.
In computer vision, common pretext tasks with still images are based on extracting two different views from the same image (c.g., two different crops) and then considering these as a pair of positive images. likely sharing the same semantic content [9]. Most current self-supervised computer vision approaches can be categorised in contrastive learning [54, 25, 9, 23, 52, 56, 18), clustering methods [3. 67, 29, 5, 20, 6, 7], asymmetric networks [22, 10] and feature decuureaaccoooltiionn methods [19, 63, 2, 27]. While the aforementioned approaches are all based on ResNets, very recently, both
[11] and [7] have empirically tested some of these methods with ViT architecture [17].
One important difference of our proposal with respect to previous work, is that we do not propose a -auly-self-supprviieed method, but we rather self-supervision jointly with standard supervision (i.e., image labels) in order to regularize VT training, hence our framework a multi-task learning approach [13]. Moreover, our dense relative localization loss is not based on positive pairs, and we do not use multiple views of the same image in the current batch, thus our method can be used with standard (supervised) data-augmentation techniques. Specifically, our pretext task is based on predicting the relative positions of pairs of tokens extracted from the same image.
Previous work using localization for self- sulf-supervision is based predicting the input image rotation [21] or the relative position of adjacent patches extracted from the same image [16, 42, 43, 39]. For instance, in [42], the network should predict the correct permutation of grid of patches (in NLP, a similar, permutation based pretext task, is deshuffling [46]). In contrast, we do not need to extract multiple patches from the same input image, since
efficiently the final token embeddings (thus, we need a single forward and backward pass per image). Moreover, differently from previous work based on localization pretext tasks, our loss is densely computed between many random pairs of (non necessarily adjacent) token embeddings. For instance, a trivial extension of the ordering task proposed [42] using a grid of
patches would lead to 49! possible permutations, which becomes intractable if modeled as a classification task. Finally, in [14], the position of random query patch used for the self self-supervised training of a transformer-B based object detector [4]. However. the localization loss used in [14] is specific for the final task (object localization) and the specific DETR architecture [4], while our loss generic and can be plugged in any VT.
a)
b)
MLP
Bid
fall
Pooling/CIs token
Localization MLP (f)
Visual Transformer
d
k
MLP
d
Input image
Figure 1: A schematic representation of the VT architecture. (a) A typical second- gener-tgnettionn VT. (b)
Our localization MLP which takes as input (concatenated) pairs of final token embeddings.
3 Preliminaries
typical VT network takes as input an image split in a grid of (possibly overlapping) K K patches. Each patch is projected in the input embedding space, obtaining a set of K x K input tokens. A VT is based on the Transformer multi-attention layers [55], which model pairwise relations over the token intermediate representations. Differently from pure Transformer [55], the hybrid architectures mentioned in Section 1-2 usually shape or reshape the sequence of these token embeddings spatial grid, which makes it possible to apply convolutional operations over small set of neighboring token embeddings. Using convolutions with a stride greater than and/or pooling operations, the resolution of the initial K x K token grid can possibly be reduced, thus simulating the hierarchical structure of CNN. We assume that the final embedding grid has a resolution of k x (where, usually, k < K), see Fig. 1 (a).

---

The final k x k grid of embeddings represents the imput image and it is used for the discriminative task. For instance, some methods include an additional "class token" which collects contextual information over the whole grid [17, 61, 58, 60, 59, 33], while others [35] apply an average global pooling over the final grid to get a compact representation of the whole image. Finally, a standard, small MLP head takes as input the whole image representation and it outputs a posterior distribution over the set of the target classes (Fig. (a)). The VT is trained using a standard cross-entropy loss (Coe), computed using these posteriors and the image ground-truth labels.
When we plug our relative localization loss (Section 4) in an existing VT. we always use the native VT architecture of each tested method, without any change apart from the dedicated localization MLP (see Section 4). For instance, we use the class token when available, or the average pooling layer when it is not, and on top of these we use the cross- enosopy loss. We also keep the positional embedding (Section 2) for those VTs which use (see Section 4.1 for a discussion about this choice). The only architectural change we do is to downsample the final embedding grid of T2T [61] and CvT [58] to make them of the same size as that used in Swin [6]. Specifically, in Swin, the final grid has a resolution of 7 7 (k- = 7). while, in T2T and in CvT, is 14 x 14. Thus, in T2T and in CvT, we use a 2 average pooling (without learnable parameters) and we get a final 7 X 7 grid for all the three tested architectures. This pooling operation is motivated in Section 4.1, and it is used only with our localization task (it does not affect the posterior computed by the classification MLP). Finally, note that T2T uses convolutional operations only in the input stage, and it outputs a sequence of 14 x 14- = 196 embeddings, corresponding to its 14 x 14 input grid. In this case, we first reshape the sequence and then use pooling. In the Supplementary Material, we show additional experiments with a ViT architecture [17], in which we adopt the same reshaping and pooling strategy.
4 Dense relative localization task
The goal of our regularization task is encourage the VT to learn spatial information without using additional manual annotations. We achieve this by densely sampling multiple embedding pairs for each image and asking the network to guess their relative distances. In more detail, given an image I, we denote its corresponding k x k grid of final embeddings (Section 3), as G2 = { J Jisiskk. where ei E Rdd and d is the dimension of the embedding space. For each Ga, we randomly sample multiple pairs of embeddings and, for each pair (e.,). we compute the 2D normalized target translation offset (Iu tu)", where:
[i hl k
L
(L,, E [0,1]²
(1)
The selected embedding vectors eig and 6p.h are concatenated and input to a small MLP (/), with two hidden layers and two output neurons, one per spatial dimension (Fig. 1 (b)), which predicts the relative distance hetween position (i.j) and position (p.h) h) on the grid. Let (du, d.)"
f(ea,) ep, Given a mini-batch B of images, our dense relative localization loss is:
Lárloc =
Σ B(en, II(tu,to) (du, d FEB
(2)
In Eq. 2, for each image x, the expectation is computed by sampling uniformly at random m pairs (ei.) kn,h) in Gz, and averaging the L, loss between the corresponding (lu, 1. and (du, d=)
Lactoc is added to the standard cross-entropy loss (Co.) of each native VT (Section 3). The final loss is: Loot Lee + XLrrtoc- We use X = 0.1 in all the experiments with both T2T and CvT, and = 0.5 in case of Swin. Note that the same pairwise localization task can be associated with slightly different loss formulations. In the Supplementary Material we present some of these variants and compare them empirically with each other.
4.1 Discussion
Intuitively, Larioe transforms the relative positional embedding (Section 2), used, for instance, in Swin [35], in a pretext task, asking the network to guess which is the relative distance of a random subset of all the possible token pairs. Thus question arises: is the relative positional embedding used in some VTs sufficient for the localization MI.P (/) to solve the localization task? The experiments
presented in Section 5.2-5.3 show that, when we plug Lartoe CvT, in which kind of positional
5

---

embedding is used [58], the relative accuracy boost is usually smaller than in case of Swin, confirming that the relative positional embedding, used in the latter, is not sufficient to make our task trivial. We further analyze this point in the Supplementary Material.
In Section 3, mentioned that, in case of T2T and CvT, we average-pool the final grid and obtain a 7 X 7 grid Gz. In fact, in preliminary experiments with both T2T and CvT at their original 14 14 resolution, we observed a very slow convergence of Cárloc- We presume this is due the fact that, with finer grid, the localization task is harder. This slows down the convergence of f, and it likely generates noisy gradients which are backpropagated through the whole VT (see also the Supplementary Material). We leave this for future investigation and, in the rest of this article, we always assume that our pretext task is computed with 7 grid Gg.
Table 1: The size of the datasets used in our empirical analysis.
Dataset
Train size
Test size
Classes
ImageNet-IIe [48]
1,28 1,167
100,000
1000
ImageNet-1000 [52]
126,689
5,000
100
CIFAR-10 [31]
50,000
10,000
10
CIFAR- [3]]
50,000
10,000
100
Oxford Flowers100 [41]
2,040
6,149
102
SVHN [40]
73,257
26,032
10
ClipArt
33,525
14,604
DomainNet
Infograph
36,023
15,582
Painting
50,416
21,850
345
Quickdraw
120,750
51,750
Real
120,906
52,041
Sketch
48,212
20,916
Experiments
All the experiments presented in this section are based on image classification tasks, while in the Supplementary Material we also show object detection, instance segmentation and semantic segmentation tasks. In this section we use 11 different datasets: ImageNet- 100 (IN-100) [52, 56), which is subset of 100 classes of ImageNet- IK [48]; CIFAR-10 and CIFAR- 100 [31]. Oxford Flowers 102 [41] and SVHN [40], which are four widely used computer vision datasets; and the six datasets of DomainNet [44]. a benchmark commonly used for domain adaptation tasks. We chose the latter because of the large domain-shift between some of its datasets and ImageNet-1K. which makes the fine-tuning experiments non-trivial. Tab. 1 shows the size of each dataset.
We used, when available, the official VT code (for T2T [61] and Swin [35]) and publicly available implementation of CvT [. In the fine-tuning experiments (Section 5.3), we use only T2T and Swin because of the lack of publicly available ImageNet pre-trained CvT networks. For each of the three baselines, we chose a model of comparable size to ResNet-50 (25M parameters): see Tab. 3 for more details. In the Supplementary Material, we show additional results obtained with larger models (ViT-B [17]), larger datasets (e.g., ImageNet-]1]] and more training epochs. When we plug our loss on one of the adopted baselines, we follow Section 4, keeping unchanged the VT architecture apart from our localization MIP (f). Moreover, in all the experiments, we train the baselines, both with and without our localization loss, using the same aatteaugmentttionn protocol for all the models, and configuration suggested by the authors of each VT. We do
we use the VTT-pecifiic not tune the V'Tspecifie hyperparameters when we use our loss and we keep fixed the values of m and X (Section 5.1) in all the experiments. We train each model using 8 V 100 32GB GPUs.
5.1 Ablation study
In Tab. (a) we analyze the impact on the accuracy of different values of m (the total number of embedding pairs used per image, see Section 4). Since we use the same grid resolution for all the VTs (i.e., 7 7, Section 3), also the maximum number of possible embeddings per image the same for all the VTs (k22 49). Using the results of Tab. 2 (a) (based on CIFAR- 100 and Swin), we chose
Pattps://githur http://////wddddaa///itttttteaa

---

m = 64 for all the VTs and all the datasets. Moreover, Tab. 2 (b) shows the influence of the loss weight X (Section 4) for each of the three baselines, which motivates our choice of using 0.1 for both CvT and T2T and À = 0.5 for Swin.
These values of and À kept fixed in all the other experiments of this paper, independently of the dataset, the main task (e.g., classification, detection, segmentation, etc.), and the training protocol (from scratch or fine-tuning). This done to emphasise the ease of of our loss. Finally, in the Supplementary Material, we analyze the influence of the size of the localization MLP (f).
Table 2: CIFAR-100, 100 training epochs: (a) the influence the accuracy of the number of pair samples (m) in Ldrloc using Swin, and (b) the influence of the À value using all the 3 VT baselines.
(a)
(b)
Model
Top-1 Acc.
Model
x=0.0
A=
A=0.5
A=1.0
A: Swin-T
53.28
CvT-13
73.50
74.51
74.07
72.84
B: A + Lartoc, m=32
63.70
Swin-T
53.28
58.15
66.23
64.28
C: A Lartoes m=64
66.23
T2T-ViT-14
65.16
68.03
67.03
66.53
D: + Ldrios: m=128
65.16
E: A Cartoes m=256
64.87
Table 3:
Top-1 accuracy on IN-100 using either 100 300 epochs. In the former case, show the average and the standard deviation values obtained by repeating each single experiment times with 5 different random seeds.
Params
ImageNett1100
Model
(M)
100 epochs
300 epochs
CvT-13
20
85.62 + 0.05
90.16
CvT
CVT-13+Gartor
20
86.09 + 0.12 (+0.47)
90.28 (+0.12)
Swin-T
82.66 0.10
89.68
Swin
Toin-T+Cdrioo
29
83.95 + 0.05 (+1.29)
90.32 (+0.64)
T2T-ViT-14
22
82.67 1 0.01
87.76
T2T
12T-ViT-14+Lare
83.74 + 0.08 (+1.07)
88.16 (+0.40)
5.2 Training from scratch
In this section, we analyze the performance of both the VT baselines and our regularization loss using small-medium datasets and different number of training epochs, simulating scenario with limited computational resources and/or limited training data. In fact, while fine-tuning a model pre-trained ImageNet-IK, IK is the most common protocol when dealing with small training datasets, this is not possible when, c.g., the network input is not an RGB image (c.g., in case of 3D point cloud data [65]) or when using task-specific backbone architecture [32. 34]. In these cases, the network needs to be trained from scratch on the target dataset, thus, investigating the robustness of the VTs when trained from scratch with relatively small datasets, is useful for those application domains in which fine-tuning protocol cannot be adopted.
We start by analyzing the impact on the accuracy of the number of training epochs on IN-100. Tab. 3 shows that, using Lartoes all the tested VTs show an accuracy improvement, and this boost is larger with fewer epochs. As expected, our loss acts as a regularizer, whose effects are more pronounced in a shorter training regime. We believe this result is particularly significant considering the larger computational times which are necessary to train typical VTs with respect to ResNets.
In Tab. we use all the other datasets and we train from scratch with 100 epochs (see the Supple- mentary Material for longer training protocols). First, we note that the accuracy of the VT baselines varies a lot depending on the dataset (which is expected). but also depending on the specific VT architecture. This is largely in contrast with the ImageNet- IK results, where the difference between the three baselines is much smaller. As a reference, when these VTs are trained on ImageNet- K (for 300 epochs), the differences of their respective top-1 accuracy is much smaller: Swin-T, 81.3 [35]; TiT-VVT-14, 81.5 [61]; CvT-13, 81.6 [58]. Conversely, Tab. shows that, for instance, the accuracy difference between CvT and Swin is about 45-46 points in Quickdraw and Sketch, 30 points
7

---

on CIFAR-10, 10, and about 20 points on many other datasets. Analogously, the difference between CvT and T2T is between 20 and 25 points in Sketch, Painting and Flowers102, and quite significant in the other datasets. This comparison shows that CvT is usually much more robust in a small training set regime with respect to the other two VTs, a behaviour which is completely hidden when the training/evaluation protocol is based on large datasets only.
In the same table, we also show the accuracy of these three VTs when training done using Carloc as regularizer. Similarly to the IN-100 results, also in this case our loss improves the accuracy of all the tested VTs in all the datasets. Most of the time, this improvement is quite significant (e.g., almost 4 points on SVHN with CvT), and sometimes dramatic (e.g., more than 45 points on Quickdraw with Swin). These results show that a supervied auxiliary task can provide a significant "signal" to the VT when the training set is limited, and, specifically, that our loss can be very effective in boosting the accuracy of VT trained from scratch in this scenario.
In Tab. we also report the results we obtained using a ResNet-50, trained with 100 epochs and the standard ResNet training protocol (e.g., using Mixup [64] and CutMix [62] data-anggenttations etc.). These results show that the best performing VT (CvT) is usually comparable with a same size ResNet, and demonstrate that VTs can potentially be trained from scratch with darasets smaller than InageNet-IK, KK Finally, in the last row of the same table, we train the ResNet-50 baseline jointly with our pretext task. In more detail, we replace the VT token embedding grid (G₂ in Eq. 2) with the last convolutional feature map of the ResNet, and we apply our loss (Eq. 2) on top of this map. comparison between the results of the last 2 rows of Tab. 4 shows that our loss is useful also when used with a ResNet (see the Supplementary Material for longer training protocols). When using ResNets, the improvement obtained with our loss is marginal, but it is consistent in 9 out of 10 datasets. The smaller improvement with respect to the analogous VT results may probably be explained by the fact that ResNets already embed local inductive biases in their architecture, thus a localization auxiliary task is less helpful (Section 1).
Table 4: Top-1 accuracy of VTs and ResNets, trained from scratch on different datasets (100 epochs).
CIFAR- 1000-1000
100
Flowers102
Quickdraw
CIFAR-10
Infograph
SVHN
ClipArt
Painting
Sketch
Real
CvT-13
89.02
73.50
54.29
91.47
60.34
19.39
54.79
70.10
76.33
56.98
CvT
CVT-13+Ldrioc
90.30
74.51
56.29
95.36
60.64
20.05
55.26
70.36
77.05
57.56
(+1.28)
(+1.01)
(+2.00)
(43.89)
(+0.30)
(+0.67)
(+0.47)
(-0.26)
(+0.68)
(+0.58)
Swin-T
59.47
53.28
34,51
71.60
38.05
8.20
35.92
24.08
73.47
11.97
Swin
Swin-T+Larlee
83.89
66.23
39.37
94.23
47.47
10.16
41.86
69.41
75.59
38.55
(424.42)
(+12.95)
(+4.86)
(+22.63)
(+9.42)
(+1.96)
(+5.94)
(+45.33)
(+2.12)
(+26.58)
T2T- VIT-14
84.19
65.16
31.73
95.36
43.55
6.89
34.24
69.83
73.93
31.51
T2T
T2T- Cdvioo
87.56
68.03
34.35
96.49
52.36
9,51
42.78
70.16
74.63
51.95
(+3.37)
(+2.87)
(+2.62)
(+1.13)
(+8.81)
(42.62)
(+8.54)
(40.33)
(+0.70)
(+20.44)
ResNet-50
91.78
72.80
46.92
96.45
63.73
19.81
53.22
71.38
75.28
60.08
ResNet
92.03
72.94
47.65
96.53
63.93
20.79
53.52
71.57
75.56
59.62
ResNet-50+LLaete
(+0.25)
(+0.14)
(+0.73)
(40.08)
(+0.20)
(40.98)
(+0.30)
(+0.19)
(+0.28)
(-0.46)
5.3 Fine-tuning
In this section, we analyze a typical fine-tuning scenario, in which a model is pre-trained on big dataset (c.g., ImageNet), and then fine-tuned on the target domain. Specifically, in all the experiments, we use VT models pre-trained by the corresponding VT authors on ImageNet- 1K without our localization loss. The difference between the baselines and ours concerns only the fine-tuning stage, which is done in the standard way for the former and using our Ldrioe regularizer for the latter. Starting from standard pre-trained models and using our loss only in the fine-tuning stage, emphasises the easy to use of our proposal in practical scenarios, in which fine-tuning can be done without re-training the model on ImageNet. As mentioned Section 5, in this analysis we do not include CvT because of the lack of publicly available ImageNet- IK trained models for this architecture
8

---

Table 5: Pre-training ImageNet-I IK and then fine-tuning the target dataset (top-1 accuracy, 100 fine-tuning epochs).
CIFAR-IH 100
Plowers102
Quickdraw
CIFAR-10
Infograph
SVHN
ClipArt
Painting
Sketch
Real
Swin-T
97.95
88.22
98.03
96.10
73.51
41.07
72.99
75.81
85.48
72.37
Swin Swin-T+Lare
98.37
88.40
98.21
97.87
79.51
46.10
73.28
76.01
85.61
72.86
(+0.42)
(+0.18)
(+0.18)
(+1.77)
(+6.00)
(+5.03)
(+0.29)
(+0,20)
(+0.13)
(+0.49)
T2T-ViT-14
98.37
87.33
97.98
97.03
74.59
38.53
72.29
74.16
84.56
72.18
T2T
T2T-ViT-144
98.52
87.65
98.08
98.20
78.22
45.69
72.42
74.27
84.57
72.29
(+0.15)
(+0.32)
(+0.10)
(+1.17)
(+3.63)
(+7.16)
(+0.13)
(+0.11)
(+0.01)
(+0.11)
ResNct-50
97.65
85.44
96.59
96.60
75.22
44.30
66.58
72.12
80.40
67.77
ResNet
ResNet-50+.Carte
97.74
85.65
96.72
96.71
75.51
44.39
69.03
72.21
80.54
68.14
(+0.09)
(+0.21)
(+0.13)
(+0.11)
(+0.29)
(+0.09)
(+2.45)
(+0.09)
(+0.14)
(+0.37)
The results are presented in Tab. Differently from the results shown in Section 5.2, the accuracy difference between the T2T and Swin baselines is much less pronounced, and the latter outperforms the former in most of the datasets. Moreover, analogously to all the other experiments, using Lriloc leads to an accuracy improvement with all the tested VTs and in all the datasets. For instance, on Infograph, Swin with Larion improves of more than points, and T2T more than 7 points.
In the last two rows of Tab. 5, we show the ResNet based results. The comparison between ResNet and the VT baselines shows that the latter very competitive in this fine-tuning scenario, even more than with a training-froon loss) are outperformed by ResNet only in 2 out of 10 datasets. This confirms that VTs are likely be widely adopted in computer vision applications in the near future, independently of the training set size. Finally, analogously the experiments in Section 5.2, Tab. shows that our loss is (marginally) helpful also in ResNet fine-tuning.
protocol (Tab. 4). For instance, the two VT baselines (without our
6 Conclusion
In this paper, we have empirically analyzed different VTs, showing that their performance largely varies when trained with small-medium datasets, and that CvT is usually much more effective in generalizing with less data. Moreover, we proposed a self-supervised auxiliary task to regularize VT training. Our localization task, inspired by [12], is densely defined for a random subset of final token embedding pairs, and it encourages the VT to learn spatial information.
In our extensive empirical analysis, with 11 datasets, different training scenarios and three VTs, our dense localization loss has always improved the corresponding baseline accuracy, usually by significant margin, and sometimes dramatically (up to +45 points). We believe that this shows that our proposal is an easy-to- reproduce yet very effective tool to boost the performance of VTs, especially training regimes with a limited amount of data/training time. It also paves the way to learning which are specific for VT's, and can
investigating other forms of wwwprruisedmulltiiiiiiiie help VT training without resorting to the use of huge annotated datasets.
Limitations. A deeper analysis on why fine-grained embedding grids harm our auxiliary task (Section 4. was left future work. Moreover, while we show in the Supplementary Material few experiments with ViT-B, which confirm the usefulness of Lartoc when used with bigger VT models, in our analysis we mainly focused on VTs of approximately the same size as a ResNet-50. The goal of this paper is to investigate the VT behaviour with medium-smm small datasets, thus, high- high-cap models most likely are not the best choice training scenario with scarcity of data.
Acknowledgements
This work was partially supported by the EU H2020 A14Media No. 951911 project and by the EUREGIO project OLIVER.

---

References
[1] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020.
[2] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Yariance-iivarrnnce--vvaaaaa regular- ization for self-supervised learning. arXiv:2 105.00405449 2021.
[3] Miguel A Bautista, Artsiom Sanakoyeu, Ekaterina Tikhoncheva, and Bjorn Ommer. CliqueCNN: deep unsupervised exemplar learning. In NeurIPS, 2016.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[5] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
[6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervieed vision transformers. arXiv:2104.14294, 2021.
[8] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CVPR, 2021.
[11] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. ICCV, 2021.
[12] Kevin Clark, Minh-Thang Pre-training text encoders as discriminators rather than generators. ICLR, 2020.
Thang Luong, Quoe V. Le, and Christopher D. Manning. ELECTRA:
[13] Michael Crawshaw. Multi-task learning with deep neural networks: survey, arX1v:2009.0979. 2020.
[14] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. UP-DETR: unsupervised pre-training for object detection with transformers. In CVPR, 2021.
[15] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[16] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[18) Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisser- man. With little help from my friends: Nearest-neighbor contrastive learning of visual representations. In ICCV, 2021.
[19] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. ICML, 2021.
[20] Wouter Van Gansheke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and
Lue Van Gool. SCAN: learning to classify images without labels. In ECCV, 2020.
10

---

[21] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ICLR, 2018.
[22] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach self-supervised learning. In NeurIPS, 2020.
[23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[25] R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
[26] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In ICCV, 2019.
[27] Tianyu Hua, Wenxiao Wang. Zihui Xue, Yue Wang, Sucheng Ren, and Hang Zhao. On feature decorrelation self-supervised learning. arXiv:2105.00470 2021.
[28] Drew A. Hudson and C. Lawrence Zitnick. Generative Adversarial Transformers. In JCML, 2021.
[29] Xu Ji, João F. Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, 2019.
[30] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. TransGAN: Two transformers can make one strong GAN. arXiv:2/02.07074, 2021.
[31] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[32] Hei Law and Jia Deng. Cornernet: Detecting objects as paired keypoints. Int. Comput. Vis., 128(3):642-655) 656, 2020.
[33] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. LicalViT: Bringing locality to vision transformers. arXiv:2104.0570,, 2021.
[34] Zeming Li, Chao Peng, Gang Yu, Xiangyu Zhang, Yangdong Deng, and Jian Sun. Detnet: Design backbone for object detection. In ECCV, 2018
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 2rXiv:2103.1400 14030. 2021.
[36] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixé, and Christoph Feichtenhofer. Track- Former: Multi-object tracking with transformers. arXiv:2 02702, 2021.
[37] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arX1v:1301.3781, 2013.
[38] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In NeurIPS, 2013.
[39] Ishan Misra and I Laurens van der Maaten. Self-supervised learning of pretext-invariant repre- sentations. In CVPR, 2020.
[40] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
11

---

[41] Maria-Elens Nilsback and Andrew Zisserman. Automated flower classification over large number of classes, In Indian Conference on Computer Vision, Graphies & Image Processing. 2008.
[42] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
[43] Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. In ICCV, 2017.
[44] Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In CVPR, 2019.
[45] Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018.
[46] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research,
2020.
[47] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy, Do vision transformers see like convolutional neural networks? arXiv:2108.088 088100 2021.
[48] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-253 2015.
[49] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. In NAACL, 2018
[50] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021.
[51] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBERT: A joint model for video and language representation learning. In ICCV, 2019.
[52] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
[53] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv:2012. 12877, 2020.
[54] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. 1rXiv:1807.03748, 2018
[55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeuriPS, 2017.
[56] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.
[57] Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
(58) Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. CvT: Introducing convolutions to vision transformers. 203. 2021.
[59] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. 2044:2200.00639, 2021.
[60] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. 113 2021.
12

---

[61] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi. Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token ViT: Training vision transformers from scratch on ImageNet. In ICCV, 2021.
[62] Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. CutMix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[63] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In ICML, 2021.
[64] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[65] Hengshuang Zhao, Jiang, Jiaya Jia, Philip S. Torr, and Vladlen Koltun. Point transformer. arXiv:20/2009644, 2020.
[66] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable transformers for end-to-end object detection. In ICLR, 2021.
[67] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In ICCV, 2019.
13


Reviews:
[{"review_id": "rJm5S46CsX4", "reviewer": "Reviewer_Tk2H", "summary": "The authors compare the robustness of three existing visual transformers (VTs) in a small training-set regime and propose a self-supervised pretext task with a corresponding loss function to extract additional information from the images. Specifically, the authors find that VTs are more data-hungry than common CNNs due to the lack of inductive bias and different VTs have largely different behaviors when training data are scarce. To solve this, the authors propose to learn spatial relations within an image by predicting the relative positions of pairs of tokens extracted from the same image. Experimental results on 13 datasets demonstrate the effectiveness of the proposed method. However, some details are unclear and the experiments can be further improved. Please see my comments below for details.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Contributions**:\n1.\tThe authors quantitatively show that different VTs have largely different performances when trained from scratch with small datasets or few training epochs.\n\n2.\tThis paper presents a new self-supervised relative localization loss and multiple variants, which is able to increase the generalization ability of trained VTs.\n\n3.\tExperimental results show that the proposed localization task is able to improve the performance and training speed of the VTs on small datasets.\n\n**Questions and points needed to be improved**:\n1.\tSome important detail is missing. In Eq. (6), what does $\\alpha$ denote?\n\n2.\tSAM [1] also focuses on improving the generalization of VTs. It would be better for the authors to provide more discussions on the difference between the proposed method and SAM. \n\n[1] Sharpness-aware Minimization for Efficiently Improving Generalization, ICLR, 2021.\n\n3.\tThe coefficient $\\lambda$ of relative localization loss in the final loss of three VTs is different. How to determine these hyper-parameters? More experiments or explanations are required.\n\n4.\tThe authors state that the proposed method focuses on small datasets. However, the generalization of the proposed method on large-scale datasets is also important. It would be better for the authors to provide more results on the large-scale dataset.\n\n5.\tThe authors state that the bad performance of the last loss variant is due to the finer resolution grids of intermediate blocks in Swin. Can we use the pooling operation to solve this issue?\n\n6.\tThe authors propose to impose a Gaussian prior on $P_u$ and $P_v$ as the third variant to alleviate the problem in the second loss variant that prediction errors are independent of the “distance”. However, in Tab. 1(b), the performance of the proposed method with a Gaussian prior (row E) is worse than that without a Gaussian prior (row D). Please give more explanations.\n\n7.\tIn Tab. 2(a), it is unclear why the performance of Swin-T goes worse with the increase of $m$? Please give more analyses.\n\n8.\tExperiments are not sufficient. The results in the experiment can not show that the proposed method is able to speed up the training. To demonstrate this, it would be better for the authors to show the training curve of different methods in terms of the loss or classification accuracy.\n\n9.\tThere are some typos in the paper.\n(1)\tIn Eq. (5), the closing parenthesis of the log function is missing.\n(2)\tIn Line 250 of Page 6, “… imposes …” should be “… to impose …”.\n(3)\tIn Line 272 of Page 7, “… mekes …” should be “… makes …”.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20"}, "extra_text": {}}, {"review_id": "dtdiEt3mYt", "reviewer": "Reviewer_ndSd", "summary": "In this paper, the authors propose a self-supervised loss to improve ViT-based models' accuracy on small datasets. The motivation is that ViTs are more flexible than CNNs, and thus overfit more on small datasets; The authors thus aim to provide additional supervision to help these ViTs to work better. Concretely, what the authors study is a relative localization task that trains the network to predict the relative spatial relationship between two feature vectors located at different spatial locations. On multiple small datasets, the authors demonstrate good improvement over the original networks that don't use the self-supervised loss.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n**[originality: medium]**\n\nWhile the high-level idea of using self-supervised loss to improvement performance when supervision is scare is not new (e.g., semi-supervised learning has a related idea), the relative location task is interesting and novel. \n\n**[quality: low-medium]**\n\n - I find the ablation study limited. For example, there have been many other self-supervised tasks proposed (e.g., [15, 41, 42, 38] to name a few), but the authors didn't compare to any of them. It's unclear what the pros and cons of the proposed task is.\n - While this paper focuses on small dataset, how the effectiveness of the proposed method changes with dataset size is also an important thing to understand (i.e., when does the proposed method work?). \n\n**[clarity: medium-high]**\n\nI found the paper clearly written and easy to follow. The method and implementation details are described with a satisfactory level of details.\n\n**[significance: low]**\n\nThe main concerns I have are as follows\n - The main message from the experiments presented is that using the proposed additional self-supervision helps on small datasets. I find this expected and hard to draw new knowledge from. (For example, in the extreme case of having only 1 example in a dataset, the supervised learning will fail completely, while adding self-supervised training signals is expected to help.)\n - The authors argue that ViTs require very large datasets to train. This is not entirely true; prior work (e.g., [51]) shows that on the small dataset of ImageNet-1K, ViTs can be successfully trained with proper training procedures. I wonder why the authors not include ImageNet-1K, but only even smaller datasets which typically are not trained from scratch. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "My main concerns are regarding the lack of comparison to prior work with the proposed self-supervised task and the usefulness of the small-dataset trained-from-scratch settings.\n\nAs of societal impact, I don't see anything particularly concerning.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours"}, "extra_text": {}}, {"review_id": "ZzL_NYN97g", "reviewer": "Reviewer_44YH", "summary": "This paper proposes a new self-supervised pretext task for vision transformer models to improve training on small-medium size datasets. The paper also conducted extensive experiments to show this method can improve the performance on various datasets with different vision transformer architectures.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is clearly-written and well-organized. The method is clearly formulated in mathematical terms and its differences from previous methods are carefully explained and well-cited. The main contribution of this paper is the design of the new pretext task used as a regularizer with any kind of vision transformer models. In fact, the major difference lies in the way token embeddings are sampled. The novelty of this design is minor. \n\nHowever, there are also a few aspects of this paper that might require further discussions. \nFirstly, it seems that all the loss variants proposed in the paper do not outperform the “vanilla” loss version $\\mathcal{L}_{drloc}$ proposed at first. Most of the experiments are also done with vanilla version except for those in the first part of the ablation studies. I think it is a bit unnecessary for the paper to spend much space talking about those variants that do not actually perform well. Besides, the paper also did not provide detailed analysis on the performance of those loss variants, making introduce loss variants at beginning less meaningful.\n\nSecondly, here are some thoughts on the experimental design.\nIt is not found in the paper how many different random seeds are used to train each model. Therefore, we cannot tell if the improvement is statistically significant, especially in those cases where the improvement is only around 0.1. \nMoreover, the paper uses the hyper-parameter configuration suggested by the authors of each VT. However, the hyper-parameters are chosen based on the specific datasets and tasks in the original paper, which might not be appropriate for all the datasets used in this paper. The fact that the top-1 accuracy of models trained on certain datasets is extremely low (e.g. Infograph) also suggests that. Knowing that it is quite computationally expensive and unnecessary to tune every single model, I would suggest to lightly tune once for each dataset to make sure the performance does not largely influenced by wrongly chosen hyper-parameters.\n\nThirdly, the reason why the paper uses DomainNet datasets is not very clear to me. Hence, it is also not so clear that the comparisons of experimental results between training from scratch and fine-tuning is necessary. The paper gives its reason for using DomainNet datasets: \"We chose the latter because of the large domain-shift between some of its datasets and ImageNet, which makes the fine-tuning experiments non-trivial,\" but it is not articulated how would this results support the two major claims of the paper in the introduction. If the paper is trying to show that certain architecture is more robust in training small-medium sized datasets, then it is quite indirect to use domainNet datasets for this purpose. \n\nOverall, the method proposed by this paper seems effective and can be used in the future by other researchers. However, it still needs more careful experimental design to support and solidify this method's effectiveness. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper has a section dedicated to the limitations of this paper. \nThere are two major contributions the paper proposes to make. With regard to the first one, more extensive experiments are needed and the result section can provide better comparisons to prove its point (e.g. quantifying the differences in accuracies among different VT architectures in the table). For the second contribution, the method itself proposed by the paper is not very different from ELECTRA but seems to work well. The paper needs to provide more convincing results to show that this method is truly effective. ", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical issues with this paper as far as I concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "3QVW3EqSKVH", "reviewer": "Reviewer_jUVG", "summary": "The paper tackles the issue of low performance of recently proposed Vision Transformers in small data regime, by adding an extra self-supervision loss in addition to classification loss during training. The proposed loss regresses offsets between randomly selected embeddings, and is supposed to act as a regularizer. The approach is validated experimentally on a number of datasets, with an ablation study on the type of self-supervision loss and the number of training epochs. The approach provides a significant boost to ViTs of various architectures in small data regime evaluations.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles the issue of low performance of recently proposed Vision Transformers in small data regime. Improving the performance of Vision Transformers in this setting would enable researchers with smaller computational budgets to experiment with Transformers and  potentially faster research iteration speed. Additionally, applying Transformers in setting where fine-tuning is not possible and data is scarce, yet global context reasoning of Transformers is desired, might be unlocked.\n\nThe authors propose to add self-supervision task in addition to main classification head of ViT. The self-supervision task is to regress relative offsets between pairs of chosen random number of embeddings. They experimentally show that the proposed extra training loss improves performance of various recent ViT architectures on a number of small datasets. They also ablate the choice of the extra training loss (regression, classification, regression with dense prior).\n\nPros:\n- the motivation behind adding the extra self-supervision loss is clear, and has been explored in other Transformer applications before\n- the paper is easy to follow and clearly written\n- the approach is simple and should be straightforward to implement, and also has minimal computational cost\n\nCons:\n- the approach adds multiple hyperparameters (number of loss locations, weighting of the loss wrt the classification loss) to training\n- the experimental evaluation lacks analysis of statistical significance of the results. In small data regime the final classification accuracy often has high variance, so it is possible that conclusions drawn on a single experiment might not actually support experiment hypothesis. In particular, in table 1b the authors conclude that the best choice for their supervision loss type based on a single run for each type, and used it for all experiments. Moreover, test set is used instead of a separate validation set for the type tuning. I would suggest to create a separate validation set, and train multiple models with different random seed, or to do e.g. 5-fold validation, and report mean and std on the validation set for each loss type, to reinforce the hypothesis.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper uses a common image classification setting, I do not see potential negative societal impacts from their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "13EV8uKLUs", "reviewer": "Reviewer_8uyZ", "summary": "This paper presents:\n- a comparison of some recent transformer architectures (CvT, Swin and T2T) on small(er) datasets with limited computational ressources\n- a simple self-supervised loss (predicting relative position with L1 loss) that can be directly added to regularize and improve performances. \nIt includes results on 5 standard datasets, and a study of several relevant alternative for the self-supervised loss (using the same pretext task)", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Not being an expert in recent vision transformers, I enjoyed reading this paper and felt that despite it's very limited novelty it was useful. It was in any case well written and easy to read, with meaningful comparison and many experiments. Thus, despite some limitations that need to be addressed (most importantly 1 and 2 bellow) I fell the paper can be accepted\n\nlimitations:\n1. I would like to know how much the proposed training have converged: Table 2 hints 100 epochs (used in all the rest of the paper) is really far from convergence, and the results might not hold after convergence. I feel this is a bit hidden/not outlined enough in the paper, and I think a clear analysis on this is necessary\n2. Since the paper is using standard dataset, I would like to see a clear comparison (e.g. in Tab 3) with State of the Art and standard (ResNet) results on these datasets. It seems to me the reported results are quite far (which is ok but should be clearly stated and visible in the tables)\n3. I would have liked to see a detailed study of the influence of the size of the localization MLP: as hinted by the authors, it is probably crucial to the quality of the results\n4. I am not sure some of the results are significative, in particular:\n* the boost of the selected L1 loss over over the CE alternative loss in Table 1. I would actually like to see this table for all three architectures.\n* the boost after 300 training epochs for CvT in Table 2\n5. I am missing an intuition on why an absolute value in the relative localization helps\n\nmisc., l. 114, grammar issue", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think some limitations and results compared to SoA should be discussed in more details (see above)\nI see no societal impact issue.", "review_text": "", "extra_scores": {"ethical_concerns": "I see no ethical concerns", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I enjoyed reading this paper and I find it generally solid. I find that the most important element of this paper is a new, presumably novel, self-supervised learning task that regularizes the vision transformer. Although the results presented in the paper were reasonably strong, the reviewers found several issues in the experiments. The authors provided answers which unfortunately did not satisfy the reviewers enough to increase their scores. Although this paper is not very far from meeting the bar for acceptance, I think it is still slightly below the expected level. Some things that in my opinion could tip the balance in the next version: (1) improved experiments, (2) a better justification for the proposed self-supervised task.
Decision: Accept (Poster)