Title: Efficient Training of Visual Transformers with Small Datasets
Authors: Yahui Liu
Enver Sangineto
Wei Bi
Nicu Sebe
Bruno Lepri
Marco De Nadai
Venue: NeurIPS 2021
Keywords: Vision Transformers
Transformers
Computer vision
Self-supervision

Abstract: Visual Transformers (VTs) are emerging as an architectural paradigm alternative to Convolutional networks (CNNs). Differently from CNNs, VTs can capture global relations between image elements and they potentially have a larger representation capacity. However, the lack of the typical convolutional inductive bias makes these models more data hungry than common CNNs. In fact, some local properties of the visual domain which are embedded in the CNN architectural design, in VTs should be learned from samples. In this paper, we empirically analyse different VTs, comparing their robustness in a small training set regime, and we show that, despite having a comparable accuracy when trained on ImageNet, their performance on smaller datasets can be largely different. Moreover, we propose an auxiliary self-supervised task which can extract additional information from images with only a negligible computational overhead. This task encourages the VTs to learn  spatial relations within an image and makes the VT training much more robust when training data is scarce. Our task is used jointly with the standard (supervised) training and it does not depend on specific architectural choices, thus it can be easily plugged in the existing VTs. Using an extensive evaluation with different VTs and datasets, we show that our method can improve (sometimes dramatically) the final accuracy of the VTs. Our code is available at: https://github.com/yhlleo/VTs-Drloc.


Review discussion:
Overall, the paper is nicely organized and well written. The improvements over baselines are also significant on small datasets. I do have several comments that might need the authors’ attention.
1. The self-supervised task proposed in this paper shares a lot of similarities with the Jigsaw puzzle paper. It is discussed in the related work, but I think mentioning it in the introduction and discussing the similarity and differences would be nicer.
2. The authors propose to use the self-supervised task to encourage the models to learn local features. The second-generation visual transformers were also motivated by it. I would suggest the authors have more discussions about why they couldn’t address the locality issue properly and how the proposed method complements that.
3. The proposed method adds a self-supervision task to supervised learning. Does this method show improvements mostly on transformer-based networks or on any types of neural networks in general for small datasets? I think having some results for other types of networks -- maybe in the supplementary material -- would give the readers a better big picture.
4. The self-supervised task might learn a shortcut from the positional embedding. The authors discuss this issue in 4.1 but only use the performance of different models with and without positional embedding to show that the self-supervised task is non-trivial. Performance improvements have many factors, which might not be related to the presence of positional embeddings. I would suggest the authors having more discussions on this issue.
5. The self-supervised task requires a low resolution for predicting the pairwise spatial relationship. Will this affect the model pre-training on dense prediction tasks that require high-resolution inputs? The authors show instance segmentation and semantic segmentation results but didn’t show self-supervised pertaining on them.
6. The authors didn’t show results on large-scale datasets e.g. ImageNet-1K. Some ablation studies might be missing, for example, on $\lambda$.
**EDIT: Post-rebuttal**
Thanks for preparing a detailed response to the initial comments. The authors have addressed the concerns adequately. I would encourage the authors to revise the draft accordingly, and I will keep my original positive rating.
No
**1.Q. The Jigsaw puzzle should be mentioned also in the introduction with similarity and differences. A.** We agree that the Jigsaw puzzle [41] task is similar to our pretext task, and we will mention it also in the Introduction, as you suggested.
Besides the differences highlighted in l. 152-160, another important difference w.r.t. the Jigsaw puzzle is that we do not ask the network to predict the correct permutation of a set of (neighbouring) patches, but our task is based on a pairwise (and not necessarily adjacent) patch distance prediction.
In other words, it is not an “ordering” task (transformed in a classification task in [41]) but a distance regression task. Note also that a plain implementation of the Jigsaw puzzle with our 7X7 grid would lead to an intractable number of possible combination (49!) and that our goal is to define a task which is densely sampled on the whole grid.
**2.Q. Why the second-generation VT couldn’t address the locality issue properly and how the proposed method complements that. A.** We believe that the second-generation VTs [59, 34, 56, 58, 57, 33] can successfully embed a local prior in their design (using a convolution-like approach), as witnessed by their results on ImageNet-1k (a dataset in which ViT-like architectures usually struggle to be trained).
However, this is probably not enough, as shown by our experiments on the 11 datasets whose size is smaller than ImageNet-1k. We believe that our regularization loss can complement these inductive biases because of the following reasons.
First, convolution-like operations are mainly useful to aggregate neighboring information.
Conversely, asking the VT to regress the spatial distance between two token embeddings, it likely encourages the network to differentiate the information represented in these tokens.
In other words, without location-specific information, embeddings representing different input image patches cannot be distinguished the one from the others (l. 62-63).
On the other hand, our task likely encourages the VT to also (better) represent contextual information (which is aggregated using the attention layers), which is necessary to solve ambiguities due to similar input patch pairs (l. 63-64). We agree that this is an interesting question, and we will include a deeper discussion on this point in the final version.
**3.Q. Does this method show improvements mostly on transformer-based networks or on any types of neural networks?. A.** Please, see Answer 1 to Reviewer K16j, in which we show that our pretext task and our loss can improve also the results of a standard ResNet-50.
**4.Q. Model pre-training on dense prediction tasks that require high-resolution inputs. The authors show instance segmentation and semantic segmentation results but didn’t show self-supervised pretraining on them. A.** Please, note that in our segmentation experiments we actually use our regularization method during pretraining (see the Supplementary Material, l. 7-17).
**5.Q. Positional embedding. A.** Please, see our Answer 1 to Reviewer Z3Px.
**6.Q. ImageNet-1K experiments. A.** Please, see our Answer 2 to Reviewer K16j.
**7.Q. Ablation on the lambda hyper-parameter. A.** Please, see our Answer 5 to Reviewer K16j.
Strengths
The paper proposes a new self-supervised task that makes Visual Transformers more effective for problems with small data sizes. Transformers are emerging as a key architecture for many computer vision tasks, as such making them more effective in the regime of smaller datasets is of great interest to the research and practitioner communities.
The proposed method is clearly introduced and the paper presents a thorough empirical study of the proposed method that clearly indicates the effectiveness of the proposed method.
Weaknesses
One shortcoming of the proposed method is that it focuses only on a fixed size of model and pre-training data. It would be of great interest to the reader if the authors could provide an ablation for how the effectiveness of the proposed method varies for models of different size or models pre-trained on larger datasets.
Minor suggestions:
Regarding the title, small is already defined as size that is less than usual.
As such, the ‘-size’ is redundant.
Regarding writing, the paper uses a lot of unnecessary hyphenation that makes the paper slightly harder to read and parse. Examples include: line 5 ‘data-hungry’, line 8 ‘training-set’, line 24 ‘well-known’. Grammatical points: line 14 ‘training data is scarce’.
No
**1.Q. Ablation for models of different size or models pre-trained on larger datasets. A.** Following your suggestion, we used ViT-B/16, which has 86.4 M parameters (about 4X the number of parameters of the other tested VTs). The table below shows that our loss can significantly improve the results of VT models bigger than those used in the paper. We will add this experiment in the final version.
||CIFAR-10|CIFAR-100|Infograph|
|:----|:----:|:----:|:----:|
|ViT-B/16|71.70|59.67|11.79|
|ViT-B/16+L_{drloc}|73.91|61.42|12.22|
Table: ViT-B/16 experiments.
Please, see Answer 2 to Reviewer K16j for additional experiments with ImageNet-1k.
Finally, concerning experiments with datasets larger ImageNet-1k, we unfortunately did not have time for testing.
However, please note that the goal of our paper is to provide a regularization approach which helps with small-scale datasets and reduced computational resources.
**2.Q. Minor suggestions. A.** Thank you for your suggestions, we will correct all the typos.
Dear authors,
Thank you very much for your detailed replies and providing the additional results.
_orginality_ \
The proposed loss based on self-supervision is interesting also because it does not require to perform a dedicated pre-training.
Instead it can be used as regularization during a normal training.
_quality_ \
The proposed approach seems quite straight forward and the pretext task of estimating distances between patches seems well adapted transformers, without the need of changing much the models architecture (only a pooling layer at the end for certain models).
However, it is not clear why the positional encoding by-itself cannot solve the task. In sec. 4.1 the authors show that improvements of the proposed approach are even higher with a method with positional encoding.
However this does not give a clear justification of what is actually happening.
_clarity_ \
Apart from some typos, the overall paper is clear and well written.
_significance_ \
The improvements of the proposed algorithm are considerable only when the performance of the original model is poor.
Thus, in practice I do not see a great use of the proposed approach.
No
**1.Q. Why the positional encoding by-itself cannot solve the task. A.** The relative positional encoding (called “B” in [34]) used in Swin, is added to the Query/Key product before the softmax operation (please, see Eq. 4 of [34]). The result of this softmax is then used to weight the importance of each Value, and the new embedding representation of each Query (“e_i,j”, in our terminology) is given by this weighted sum of Values.
Thus, the content of B is not directly represented in e_i,j, but only used to weight the Values forming e_i,j (note that there is also a skip connection).
For this reason, B may be useful for the task for which it is designed, i.e., computing the importance (attention) of each Key w.r.t. the current Query.
However, in order to solve our task (i.e., to compute t_u and t_v in Eq. 1 of our paper), the VT should be able to recover and extract from a given embedding pair (e_i,j , e_p,h) the specific offset information originally contained in B_{(i,j), (p,h)} and then blended in the Value weights. Probably this is a task (much) harder than exploiting appearance information contained in (e_i,j , e_p,h). This is somehow in line with different previous work showing the marginal importance of positional encoding in VTs.
For instance, in Naseer et al., “Intriguing Properties of Vision Transformers”, the authors show that the (absolute) positional encoding used in ViT is NOT necessary for the transformer to solve very challenging occlusion or patch permutation tasks, and they conclude that these tasks are solved by ViT thank to its “dynamic receptive field” (i.e., the context represented in each individual token embedding). We will add this discussion in the Supplementary Material.
**2.Q. The improvements are considerable only when the performance of the original model is poor. A.** Please, note that we have improvements also with the strongest of the tested models, CvT, and these improvements are **observed in all the experiments and in all the 11 datasets**.
For instance, in Tab. 3, on CIFAR-10, CIFAR-100, Flowers and SVHN, we get improvements which are between 1 and almost 4 points, which are not that marginal. Note also that the other two tested VTs (T2T and Swin) are considered state-of-the-art architectures, used as comparison baselines or as backbones in many other works. Presently, it is not clear which specific architecture will “dominate” the VT field in the next future. We believe that our empirical analysis may be useful in showing that not all the VTs are equally performing in a scarce training-data regime (a phenomenon which is completely hidden in the common ImageNet-1k/ImageNet-22K evaluation protocols used in the VT literature, l.
310-312), and in showing that architectures like CvT better exploit the local inductive bias. Generally speaking, it is not clear a priori when a model performs poorly, and, as you also emphasize in your Review, our regularization method behaves similarly to many other regularization approaches, which are particularly useful in enforcing the model’s generalization capabilities in difficult scenarios (e. g. , weak inductive biases, scarce training data, limited training resources, etc. ).
Hence, we believe that the importance of the proposed method lies in regularizing a VT when trained in these difficult scenarios, alleviating the risk of overfitting and poor performance.
Finally, please, refer to Answer 1 to Reviewer K16j, in which we show that our pretext task is useful also when applied to standard ResNets.
I read the other reviewers comments and authors answers.
While I found the paper well presented and interesting, I still believe that the proposed contribution is not enough for a NeurIPS paper.
As in my original comment, I see that when the performance of a method gets close to the state of the art, the improvement becomes very limited.
Furthermore, I fully agree with K16j that the original ViT models were proposed on large datasets.
When trained on small datasets, their performance is still not competitive even when using the proposed loss, and this seems to be the most important contribution of the paper, as the paper's title suggests.
For instance, on CIFAR10 and CIFAR100, the best results reported for the ViT models with the proposed loss are 90.3% and 74.5%, while the best CNN based models obtained 96.1% and 80.6% respectively (as reported by the authors experiments).
Thus, a very large gap still.
Thank you very much for your answer.
However, please note that there is a crucial mismatch in your comment when you mention the comparison between the best VT model (CvT) and ResNet-50, because the values you report **refer to different training epochs** (respectively, 100 for the VTs, and 200 for the ResNets). This difference is crucial, as shown by the difference between the ResNets results with 100 epochs (e.g., 72.80 on CIFAR-100) and the results of the same ResNets when trained with 200 epochs (80.45 	on CIFAR-100, about +8 points).
Thus, in order to have a fair comparison ResNets-VTs, they should be compared with the same training budget.
Below we report part of the experiments shown in our answer to Reviewer K16j, selecting only the best VT model (CvT) and the two datasets you mention (CIFAR-10/100). The following table is based on 100 epochs training for both architectures (VTs and ResNets), and we are currently running additional experiments using CvT with 200 epochs (we will show them as soon as ready, in the hope they will be ready before the conclusion of the discussion period):
|Model|CIFAR-10|CIFAR-100|
|:----|:----:|:----:|
|CvT-13|89.02|73.50|
|CvT-13+ L_{drloc}|90.30|74.51|
|ResNet-50|91.78|72.80|
|ResNet-50+L_{drloc}|92.03|72.94|
*Table: A comparison between the best VT model (CvT) and ResNet-50 using 100 training epochs.*
As you can see, CvT and ResNets results are comparable to each other, and, in CIFAR-100, CvT outperforms ResNet-50, even without using our loss.
When using our loss, the results **always** improve (including when our method is used on top of a ResNet, instead of a VT).
Generally speaking, you are right that our auxiliary task is most useful when the performance of the base model is poor.
But this is the goal of any regularization method, and it is the declared goal of our paper, i.e., to boost the performance of VTs when trained in a scarce data/reduced training budged regime. Tab. 3 and 4 of our paper show that our method is very effective in this goal, sometimes drastically boosting the native VT performance (e.g., Swin +45.33 on Quickdraw in Tab. 3 and T2T +7.16 on Infograph in Tab. 4).
As mentioned in the previous comment, we repeated the comparison between CvT and ResNet-50, this time using 200 epochs (training from scratch). We show below the results.
|Model|CIFAR-10|CIFAR-100|
|:----|:----:|:-----:|
|CvT-13|94.80|81.23|
|CvT-13+L_{drloc}|95.45|81.77|
|ResNet-50|95.91|80.45|
|ResNet-50+L_{drloc}|96.10|80.61|
*Table: A comparison between the best VT model (CvT) and ResNet-50 using 200 training epochs.*
The above table confirms the results obtained with 100 epochs.
Specifically, we believe that this ResNet vs VT comparisons show two important things. (1) VTs are comparable or even better than ResNets, even when used with small datasets (e.g., on CIFAR-100, 200 epoch training, CvT outperforms ResNet-50, even without our loss). This confirms that VTs are likely to be widely adopted in vision applications in the near future, independently of the training set size, motivating our regularization approach. (2) Our auxiliary task can improve the results of both VTs and ResNets, with both 100 and 200 epochs, and both when training is “from-scratch” and when we use fine-tuning.
The paper is very well written and has a nice flow to it that makes it easy to understand, intuitive and enjoyable. The literature review and introduction show excellent understanding and awareness of the literature and they were very nice to read.
The method is, in my opinion, interesting, correct and original, but not groundbreaking and of a somewhat narrow applicability.
The experimental results are well-laid and positive, bringing home the main claims.
My main concern is that I am not very excited about the "training transformers from scratch on small datasets" goal. The main reasons are:
1) we know (have some strong evidence) that the strength of transformers lay in the dataset size (for training or pre-training). We have other models that have traditionally worked well (or better) within this setting - traditional convnets. From the practitioner's point of view, it would make sense to simply choose the right architecture for the problem at hand based on the very clear different behavior of both families of models. Now, this would change if it was shown in the experimental results that in the end, with the proposed changes, the transformer architectures did become competitive respect to a resnet50 model - then this paper would modify the choice of the practitioner when tackling a low data count setting.
But no comparison is given with more traditional architectures so it is unclear we transformers stand in comparison, even with the proposed changes.
2) in my opinion, imagenet + finetunning is by far the most interesting setting when dealing with small datasets, and for those datasets so far off in terms of domain that it might not help, there probably is some other large-scale dataset to pre-train on.
However, the authors also tackle the aforementioned scenario, even if not the main focus, and the results are still interesting, finding good stabilization of the results across architectures (though not a merit of their method), consistent (if sometimes very minor) improvements, and very large improvements on others (the fewer).
Thus, overall the "from scratch" focus is a bit of a weak point in the argumentation, but it is in the end is alleviated by section 5.3 and that the numbers are still on the positive side.
Other aspect to cover is the experiments: 1) I'm very curious why no experiments on ImageNet are reported. Is there a cut-off point in terms of size where the proposed loss hurts? if yes, we should know about it and, if not, then it would add to the paper. 2) training beyond 100 epochs (so until convergence) would also be nice. It is quite different to manage to train a model faster and training it to a better accuracy when training to optimality (the second is a much stronger result in my opinion). 3) There are some very big differences in terms of performance, especially for the finetunning experiments, almost forming two clusters, very large improvement or marginal improvement. Is there any intuition or analysis on why is this the case? data count? domain gap?...
Overall, I like the paper (partly because it was so accessible and informed, plus I really liked the "limitations" section), but the issues outlined above are still substantial in my opinion. I'm keen to hear the author's opinion about them and I'm of course very open to raise the mark provided the authors give convincing arguments.
Other comments that do not need addressed (but authors are welcome if they feel like it):
- I really liked the good awareness of the literature shown by the authors in Sec. 2, but my two cents: I would not rate asymmetric networks as a class of its own at all, I think that's mostly a design quirk than a conceptually deep difference.
- In sec. 4, it would be nice to hear what's the typical sensitivity of the method to the hyperparameter lambda and how you figure out what the well-performing values were.
- Isn't the fact that the 14x14 setting does not work as well as the 7x7 suggesting some sort of multi-scale solution? Maybe the 7x7 helps early on and the 14x14 is a harder task that is more beneficial later on?
- At the beginning of Sec. 5: I've never understood why so many people focus on num. of parameters...
in practice we would most typically care for latency (FLOPs as proxy) and accuracy.
- In the "very dense loss" setting, I'm surprised about the design. I'd rather use 2 prediction points (end and intermediate), a-la-inception. No wonder that the setting used does not work!
- Limitations section is nice. Very honest and open, as research should be.
- Please check the "deltas" on Table 4 (and just in case, also rest of the tables) as I spotted two that are wrong - one VERY wrong and against the author's interest.
- l272 "mekes"
No
**1.Q. Finetuning vs training from scratch. A.** We agree that, with small datasets, finetuning is the most common strategy in computer vision tasks. We believe that our experiments in Tab. 4 show the effectiveness of our regularization loss in this scenario, which gives a boost in **all** the 10 tested datasets (which may be more than 5/7 points in some cases).
However, we also believe that training-from-scratch is important in some scenarios in which the domain shift w.r.t. ImageNet is too large for the finetuning procedure to be effective (l. 317-322).
For instance, if the VT architecture should be changed to be adapted to a specific task and/or to deal with sensory data different from RGB images, then all or some of the layers need to be trained from scratch on the new domain. Other application examples regard specific domains such as medical image analysis, industrial-defect inspection, etc., where the images are not “natural” scenes as those contained in ImageNet, so pretraining may be less effective than training from scratch (note that, in these medical/industrial image analysis domains, the training datasets are usually quite small).
Moreover, following your suggestion, we trained a typical ResNet-50 on CIFAR-10, CIFAR-100 and Infograph. The table below compares the results reported in Tab. 3 with those we obtained using the ResNet-50, and shows that (as you supposed) the CNN performs much better than both T2T and Swin in all the 3 datasets.
However, the results are comparable with CvT.
Moreover, CvT plus our loss improves the ResNet-50 results in 2 out of 3 datasets (in the final version we will include all the other datasets, since we did not have enough time to run all of them in the short rebuttal period). We believe that these results show that in the next future VTs may completely replace CNNs also for small-scale datasets, leading to a uniform architecture which will be used for all the computer vision tasks. This further motivates the usefulness of our proposal.
Finally, the last row of the same table shows an additional experiment, in which we used the ResNet-50 baseline jointly with our pretext task. In more detail, we replaced the VT token embedding Grid (G_x in Eq. 2) with the last convolutional layer of the ResNet, and we applied our loss (Eq. 2) on top of this convolutional feature map. A comparison between the results of the last 2 rows of that table shows that **our loss is useful also when used with a ResNet.**
||CIFAR-10|CIFAR-100|Infograph|
|:----|:----:|:----:|:----:|
|CvT-13|89.02|73.50|19.39|
|CvT-13+ L_drloc|90.30|74.51|20.05|
|Swin-T|59.47|53.28|8.20|
|Swin-T+ L_drloc|83.89|66.23|10.16|
|T2T-ViT-14|84.19|65.16|6.89|
|T2T-ViT-14+ L_drloc|87.56|68.03|9.51|
|ResNet-50|91.78|72.80|19.81|
|ResNet-50+L_{drloc}|92.03|72.94|20.79|
Table: ResNet-50 results (with and without our loss).
**2.Q. ImageNet-1k experiments. A.** Following your suggestion, we used ImageNet-1k to train our models for 300 epochs. We report the results in the table below, where the column “Original” corresponds to our train/testing of the publicly available code of the corresponding VT baseline.
|Model|Original|+ L_{drloc}|
|:----|:----:|:----:|
|Swin-T|81.2|81.33|
|T2T-ViT-14|80.7|80.85|
Table: ImageNet-1k experiments.
We are currently training CvT and we hope the results will be ready by the end of the NeurIPS discussion phase (anyway they will be reported in the final version of the paper). These ImageNet-1k results show that our loss, despite being designed to regularize small-scale datasets and few training epochs, it is not decremental and it can still help to (slightly) improve the results when used with datasets of the size of ImageNet-1k.
**3.Q. Training beyond 100 epochs. A.** Following your suggestion, we used Infograph and we trained (from scratch) all the models for 300 epochs. The table below shows that the improvements observed with 100 epochs are persistent even with a longer training. For Swin and T2T, the gain is even larger (in the latter case, much larger: more than 8 points vs. 2.62 points obtained with 100 epochs).
|Method| Acc@1|
|:----|:----:|
|CvT-13|29.76|
|CvT-13 + L_{drloc}|30.31|
|Swin-T|17.17|
|Swin-T + L_{drloc}|20.72|
|T2T-ViT-14|12.62|
|T2T-ViT-14 + L_{drloc}|20.68|
Table: Infograph with 300 training epochs.
**4.Q. Differences in terms of performance in the finetuning experiments. A.** We also noticed this gap. Basically, we observe that the largest improvements are obtained in datasets with non-natural images (e.g., digits in SVHN, drawings in Clipart, drawings + text in Infograph), i.e., where there is a large domain shift w.r.t. ImageNet (as expected).
However, the improvement in both Quickdraw and Sketch (where the domain-shift is also quite drastic) is much more marginal. We presume this is due to the fact that the latter datasets are big enough (see Tab. 1 (a)) to cover the domain-shift during the finetuning stage.
**5.Q. Ablation on the lambda hyper-parameter. A.** we agree on the importance of showing an empirical ablation on the lambda value. We report below the study we performed in the beginning of our work (using CIFAR-100), which motivates our choice of the lambda values as reported in l. 214-215 (we will include this table in the final version).
|Model|0.1|0.5|1.0|
|:----|:----:|:----:|:----:|
|CvT-13|68.03|67.03|66.53|
|Swin-T|58.15|66.23|64.28|
|T2T-ViT-13|74.74|74.07|72.84|
Table: Ablation study on lambda.
**6.Q. multi-scale solution. A.** Thank you for your suggestion.
Indeed, using a multi-scale (or “curriculum-learning”) solution, in which the task becomes harder as training proceeds, is an interesting idea. We leave this as a future investigation.
However, the advantage of having a single scale and a single task, is that the overall approach is easier to be reproduced.
**7.Q. Latency. A.** We computed the overhead of the L_{drloc} at training time. The results below refer to seconds per batch (the overall batch size is 1024), without v.s. with our loss:
- CvT-13: 0.6037 +- 0.0040 v.s. 0.6184 +- 0.0070    (+2.43%)
- Swin-T: 0.6684 +- 0.0031 v.s. 0.6842 +- 0.0033    (+2.36%)
- T2T-ViT-14: 0.5941 +- 0.0053 v.s. 0.6046 +- 0.0058 (+1.77%)
Overall, the overhead due to our additional pretext task is negligible w.r.t. the whole training.
**8.Q. “Very dense loss" with 2 prediction points. A.** Thank you for your suggestion. We will  include this experiment in the final version.
**9.Q. Check the "deltas" in Tab. 4. A.** Again thank you for your suggestion and sorry for the typos.
Indeed, the improvement of our loss w.r.t. T2T on Clipart is + 3.63 (and not + 0.37 as wrongly reported). The second typo is in Flowers (it should be +0.1 and not +0.2). We confirm that all the other numbers in all the other tables are correct.
**Finetuning vs training from scratch:** Thanks for the detailed answer:
- (arguing for the scenario) Some of these scenarios might indeed exist...
although my argument was that even in those cases where ImageNet pre-training is too far off, there is likely to exist another dataset to pre-train on. I can be wrong of course, but there are no specific examples given either in the reply.
- (competitiveness over ResNet50) On the experiments with ResNet50, I see a couple of problems at the moment: 1) the numbers are really low. It is easy to get to 75+ with a ResNet18 on CIFAR-100, and if you use some sort of extra regularization (mixup, cutout, whatever...) you get to close to 80 (note that I'm talking the standard ResNet variants, not the CIFAR-100 ones). This makes me think, is the proposed technique some sort of regularizer that plays a similar role to, say, mixup? Would it stand when used together with cutout/mixup? What is shown is that the transformers trained are competitive...
when compared to a not-very-thoroughly optimizer ResNet (now, maybe the transformers would benefit from similar improved training regime, but that's to be proven). Unfortunately, I see these low CIFAR-100 performances as baseline all the time in published papers, maybe it is a bit unfair to be too harsh with the current submission.  2) I see from the experiments over 300 epochs (A3) that the optimization is likely poor throughout. This experiment of increasing the number of epochs is only done on Infograph, but results go from 6.89 -> 12.62 when shifting from 100 to 300 epochs with T2T, and 19.39 -> 29.76 for CvT-13. In this experiment, the gains of the propose method are maintained (or even increased I believe?) but the relative performances during early training, or in low-regularization regimes, do not need to extrapolate to settings with proper optimizations. This is probably even more true when comparing transformer-based archs to convolutional ones.
**2.Q. ImageNet-1k experiments:** Thanks for these experiments (ImageNet1k takes quite a few resources). It seems that there are diminishing returns but it is still not damaging.
**3.Q. Training beyond 100 epochs:** Please refer to R1 .
**4.Q. Differences in terms of performance in the finetuning experiments:** It would be good to have a discussion in the paper about this, and maybe indicate size of datasets these correlations can be spotted? (just a suggestion, you can decide either way)
Q5-Q9 did not need reply (as stated in my review), but thanks anyway for it. The latency comment though was probably not clear in my original review: I am pretty sure the latency introduced by your loss is negligible. What I meant is that in Sec. 5 you choose competing transformers with similar number of parameters, not similar latency/FLOPs, which would be more natural to me.
Edit: re-reading my reply, I focused on the negative side too much. The ImageNet experiments are on the positive side I would say (no gain but no degradation), the CIFAR-100 are somewhat positive too though inconclusive. The training for 300 epochs experiment is similarly positive, though it also results in some extra uncertainty over the rest of the results.
So overall, I would rate the rebuttal as somewhat positive but high uncertainty.
Thank you for your kind answer and for your constructive and interesting feedback, which we have appreciated. We answer below to the main points.
**Q. Training-from-scratch example scenarios. A.** Some interesting example domains in which fine-tuning is not easy/possible are those in which the input is not an RGB image.
For instance, 3D point cloud data are usually provided as an unordered, sparse set of 3D point coordinates, thus fine-tuning from ImageNet is not possible (e.g., see Zhao et al., “Point Transformer”). Note that available 3D point cloud datasets usually contain much less scene samples than ImageNet-1k (e.g., the LiDAR  data of the KITTI dataset), hence pre-training using a big dataset in these cases is not possible.
**Q. ResNet50 experiments. A.** Please, note that the ResNet experimental results shown in our previous answer do already contain standard data-augmentation techniques, since we used both mixup and cutmix (both with and without our loss). You are right that the final CIFAR absolute accuracy values are low, but this is due to the number of epochs used for training.
In fact, in those experiments we used 100 epochs, and this choice was made to make the ResNet results directly comparable with the VT ones (same training budget). We skipped these (important) details in our previous answer for brevity.
We agree that much higher results can be obtained on CIFAR (e.g., in “Manifold Mixup”, ResNets-50 are trained on CIFAR-10/100 for 600 epochs). As a comparison, we repeated all the previous ResNet-based experiments (with and without our loss) using 200 epochs and we show in the table below the results:
||CIFAR-10|CIFAR-100|Infograph|
|:----|:-----:|:-----:|:----:|
|ResNet-50|95.91|80.45|30.20|
|ResNet-50+L_{drloc}|96.10|80.61|30.54|
Table: Training from scratch (200 epochs).
The absolute accuracy values are now much higher, however, the relative improvement of our loss is still (slightly) positive in all the three tested datasets. For completeness, we also used the fine-tuning scenario of Tab. 4 of our paper with the same ResNet-50 (same mixup + cutmix data-augmentation, 100 epochs). In this case, the results are averaged over 5 different runs and we report below the average accuracy and the standard deviation:
||CIFAR-10|CIFAR-100|Infograph|
|:----|:-----:|:-----:|:-----:|
|ResNet-50|97.65 (+- 0.03) |85.44 (+- 0.09) |44.30 (+- 0.04) |
|ResNet-50+L_{drloc}|97.74 (+- 0.05) |85.65 (+- 0.04) |44.39 (+- 0.07) |
Table: Fine-tuning (100 epochs) based on an ImageNet-1K pretrained model.
Also in the fine-tuning scenario we observe an improvement using our loss. The improvement is marginal but consistent in all the experiments. Note, however, that the goal of our work is to propose a regularization technique for VTs, while CNNs already include local biases in their design, and this is likely the reason for which they benefit less from our method.
**Q. Experiments with 300 epochs. A.**  We are not sure that we fully understood this comment. We try to do our best to answer. The relative gain of our pretext task on Infograph (using VTs) has indeed generally increased when using 300 epochs w.r.t. 100 epochs.
However, we do not think we can draw a conclusion from this, since in most of the other experiments, when we increase the number of epochs, we usually get a lower gain (e.g., comparing the 300 vs. 100 epochs training on ImageNet-100, Tab. 2 (b), or looking at the ImageNet-1k experiments with 300 epochs). Somehow this was expected, as usually regularization techniques are mostly useful with scarce data/shorter training regimes, which is indeed the scope of our paper.

Meta-review:
role: PC
data: comment: The reviewers were in agreement that the proposed self-supervision method for vision transformers is novel and should be of general interest to the community. Some points were brought up in the discussion on additional experiments to show effectiveness on larger datasets/models and effect on CNNs, which were addressed by the authors in the rebuttal. I encourage the authors to add these new results to the paper main text.

Decision: Accept (Poster)