Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a new self-supervised pretext task for vision transformer models to improve training on small-medium size datasets. The paper also conducted extensive experiments to show this method can improve the performance on various datasets with different vision transformer architectures.

main_review: This paper is clearly-written and well-organized. The method is clearly formulated in mathematical terms and its differences from previous methods are carefully explained and well-cited. The main contribution of this paper is the design of the new pretext task used as a regularizer with any kind of vision transformer models. In fact, the major difference lies in the way token embeddings are sampled. The novelty of this design is minor. 

However, there are also a few aspects of this paper that might require further discussions. 
Firstly, it seems that all the loss variants proposed in the paper do not outperform the “vanilla” loss version $\mathcal{L}_{drloc}$ proposed at first. Most of the experiments are also done with vanilla version except for those in the first part of the ablation studies. I think it is a bit unnecessary for the paper to spend much space talking about those variants that do not actually perform well. Besides, the paper also did not provide detailed analysis on the performance of those loss variants, making introduce loss variants at beginning less meaningful.

Secondly, here are some thoughts on the experimental design.
It is not found in the paper how many different random seeds are used to train each model. Therefore, we cannot tell if the improvement is statistically significant, especially in those cases where the improvement is only around 0.1. 
Moreover, the paper uses the hyper-parameter configuration suggested by the authors of each VT. However, the hyper-parameters are chosen based on the specific datasets and tasks in the original paper, which might not be appropriate for all the datasets used in this paper. The fact that the top-1 accuracy of models trained on certain datasets is extremely low (e.g. Infograph) also suggests that. Knowing that it is quite computationally expensive and unnecessary to tune every single model, I would suggest to lightly tune once for each dataset to make sure the performance does not largely influenced by wrongly chosen hyper-parameters.

Thirdly, the reason why the paper uses DomainNet datasets is not very clear to me. Hence, it is also not so clear that the comparisons of experimental results between training from scratch and fine-tuning is necessary. The paper gives its reason for using DomainNet datasets: "We chose the latter because of the large domain-shift between some of its datasets and ImageNet, which makes the fine-tuning experiments non-trivial," but it is not articulated how would this results support the two major claims of the paper in the introduction. If the paper is trying to show that certain architecture is more robust in training small-medium sized datasets, then it is quite indirect to use domainNet datasets for this purpose. 

Overall, the method proposed by this paper seems effective and can be used in the future by other researchers. However, it still needs more careful experimental design to support and solidify this method's effectiveness. 


limitations_and_societal_impact: The paper has a section dedicated to the limitations of this paper. 
There are two major contributions the paper proposes to make. With regard to the first one, more extensive experiments are needed and the result section can provide better comparisons to prove its point (e.g. quantifying the differences in accuracies among different VT architectures in the table). For the second contribution, the method itself proposed by the paper is not very different from ELECTRA but seems to work well. The paper needs to provide more convincing results to show that this method is truly effective.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: In this paper, the authors propose a self-supervised loss to improve ViT-based models' accuracy on small datasets. The motivation is that ViTs are more flexible than CNNs, and thus overfit more on small datasets; The authors thus aim to provide additional supervision to help these ViTs to work better. Concretely, what the authors study is a relative localization task that trains the network to predict the relative spatial relationship between two feature vectors located at different spatial locations. On multiple small datasets, the authors demonstrate good improvement over the original networks that don't use the self-supervised loss.

main_review: 
**[originality: medium]**

While the high-level idea of using self-supervised loss to improvement performance when supervision is scare is not new (e.g., semi-supervised learning has a related idea), the relative location task is interesting and novel. 

**[quality: low-medium]**

 - I find the ablation study limited. For example, there have been many other self-supervised tasks proposed (e.g., [15, 41, 42, 38] to name a few), but the authors didn't compare to any of them. It's unclear what the pros and cons of the proposed task is.
 - While this paper focuses on small dataset, how the effectiveness of the proposed method changes with dataset size is also an important thing to understand (i.e., when does the proposed method work?). 

**[clarity: medium-high]**

I found the paper clearly written and easy to follow. The method and implementation details are described with a satisfactory level of details.

**[significance: low]**

The main concerns I have are as follows
 - The main message from the experiments presented is that using the proposed additional self-supervision helps on small datasets. I find this expected and hard to draw new knowledge from. (For example, in the extreme case of having only 1 example in a dataset, the supervised learning will fail completely, while adding self-supervised training signals is expected to help.)
 - The authors argue that ViTs require very large datasets to train. This is not entirely true; prior work (e.g., [51]) shows that on the small dataset of ImageNet-1K, ViTs can be successfully trained with proper training procedures. I wonder why the authors not include ImageNet-1K, but only even smaller datasets which typically are not trained from scratch. 
limitations_and_societal_impact: My main concerns are regarding the lack of comparison to prior work with the proposed self-supervised task and the usefulness of the small-dataset trained-from-scratch settings.

As of societal impact, I don't see anything particularly concerning.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper tackles the issue of low performance of recently proposed Vision Transformers in small data regime, by adding an extra self-supervision loss in addition to classification loss during training. The proposed loss regresses offsets between randomly selected embeddings, and is supposed to act as a regularizer. The approach is validated experimentally on a number of datasets, with an ablation study on the type of self-supervision loss and the number of training epochs. The approach provides a significant boost to ViTs of various architectures in small data regime evaluations.
main_review: The paper tackles the issue of low performance of recently proposed Vision Transformers in small data regime. Improving the performance of Vision Transformers in this setting would enable researchers with smaller computational budgets to experiment with Transformers and  potentially faster research iteration speed. Additionally, applying Transformers in setting where fine-tuning is not possible and data is scarce, yet global context reasoning of Transformers is desired, might be unlocked.

The authors propose to add self-supervision task in addition to main classification head of ViT. The self-supervision task is to regress relative offsets between pairs of chosen random number of embeddings. They experimentally show that the proposed extra training loss improves performance of various recent ViT architectures on a number of small datasets. They also ablate the choice of the extra training loss (regression, classification, regression with dense prior).

Pros:
- the motivation behind adding the extra self-supervision loss is clear, and has been explored in other Transformer applications before
- the paper is easy to follow and clearly written
- the approach is simple and should be straightforward to implement, and also has minimal computational cost

Cons:
- the approach adds multiple hyperparameters (number of loss locations, weighting of the loss wrt the classification loss) to training
- the experimental evaluation lacks analysis of statistical significance of the results. In small data regime the final classification accuracy often has high variance, so it is possible that conclusions drawn on a single experiment might not actually support experiment hypothesis. In particular, in table 1b the authors conclude that the best choice for their supervision loss type based on a single run for each type, and used it for all experiments. Moreover, test set is used instead of a separate validation set for the type tuning. I would suggest to create a separate validation set, and train multiple models with different random seed, or to do e.g. 5-fold validation, and report mean and std on the validation set for each loss type, to reinforce the hypothesis.
limitations_and_societal_impact: The paper uses a common image classification setting, I do not see potential negative societal impacts from their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents:
- a comparison of some recent transformer architectures (CvT, Swin and T2T) on small(er) datasets with limited computational ressources
- a simple self-supervised loss (predicting relative position with L1 loss) that can be directly added to regularize and improve performances. 
It includes results on 5 standard datasets, and a study of several relevant alternative for the self-supervised loss (using the same pretext task)
main_review: Not being an expert in recent vision transformers, I enjoyed reading this paper and felt that despite it's very limited novelty it was useful. It was in any case well written and easy to read, with meaningful comparison and many experiments. Thus, despite some limitations that need to be addressed (most importantly 1 and 2 bellow) I fell the paper can be accepted

limitations:
1. I would like to know how much the proposed training have converged: Table 2 hints 100 epochs (used in all the rest of the paper) is really far from convergence, and the results might not hold after convergence. I feel this is a bit hidden/not outlined enough in the paper, and I think a clear analysis on this is necessary
2. Since the paper is using standard dataset, I would like to see a clear comparison (e.g. in Tab 3) with State of the Art and standard (ResNet) results on these datasets. It seems to me the reported results are quite far (which is ok but should be clearly stated and visible in the tables)
3. I would have liked to see a detailed study of the influence of the size of the localization MLP: as hinted by the authors, it is probably crucial to the quality of the results
4. I am not sure some of the results are significative, in particular:
* the boost of the selected L1 loss over over the CE alternative loss in Table 1. I would actually like to see this table for all three architectures.
* the boost after 300 training epochs for CvT in Table 2
5. I am missing an intuition on why an absolute value in the relative localization helps

misc., l. 114, grammar issue
limitations_and_societal_impact: I think some limitations and results compared to SoA should be discussed in more details (see above)
I see no societal impact issue.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors compare the robustness of three existing visual transformers (VTs) in a small training-set regime and propose a self-supervised pretext task with a corresponding loss function to extract additional information from the images. Specifically, the authors find that VTs are more data-hungry than common CNNs due to the lack of inductive bias and different VTs have largely different behaviors when training data are scarce. To solve this, the authors propose to learn spatial relations within an image by predicting the relative positions of pairs of tokens extracted from the same image. Experimental results on 13 datasets demonstrate the effectiveness of the proposed method. However, some details are unclear and the experiments can be further improved. Please see my comments below for details.
main_review: **Contributions**:
1.	The authors quantitatively show that different VTs have largely different performances when trained from scratch with small datasets or few training epochs.

2.	This paper presents a new self-supervised relative localization loss and multiple variants, which is able to increase the generalization ability of trained VTs.

3.	Experimental results show that the proposed localization task is able to improve the performance and training speed of the VTs on small datasets.

**Questions and points needed to be improved**:
1.	Some important detail is missing. In Eq. (6), what does $\alpha$ denote?

2.	SAM [1] also focuses on improving the generalization of VTs. It would be better for the authors to provide more discussions on the difference between the proposed method and SAM. 

[1] Sharpness-aware Minimization for Efficiently Improving Generalization, ICLR, 2021.

3.	The coefficient $\lambda$ of relative localization loss in the final loss of three VTs is different. How to determine these hyper-parameters? More experiments or explanations are required.

4.	The authors state that the proposed method focuses on small datasets. However, the generalization of the proposed method on large-scale datasets is also important. It would be better for the authors to provide more results on the large-scale dataset.

5.	The authors state that the bad performance of the last loss variant is due to the finer resolution grids of intermediate blocks in Swin. Can we use the pooling operation to solve this issue?

6.	The authors propose to impose a Gaussian prior on $P_u$ and $P_v$ as the third variant to alleviate the problem in the second loss variant that prediction errors are independent of the “distance”. However, in Tab. 1(b), the performance of the proposed method with a Gaussian prior (row E) is worse than that without a Gaussian prior (row D). Please give more explanations.

7.	In Tab. 2(a), it is unclear why the performance of Swin-T goes worse with the increase of $m$? Please give more analyses.

8.	Experiments are not sufficient. The results in the experiment can not show that the proposed method is able to speed up the training. To demonstrate this, it would be better for the authors to show the training curve of different methods in terms of the loss or classification accuracy.

9.	There are some typos in the paper.
(1)	In Eq. (5), the closing parenthesis of the log function is missing.
(2)	In Line 250 of Page 6, “… imposes …” should be “… to impose …”.
(3)	In Line 272 of Page 7, “… mekes …” should be “… makes …”.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: I enjoyed reading this paper and I find it generally solid. I find that the most important element of this paper is a new, presumably novel, self-supervised learning task that regularizes the vision transformer. Although the results presented in the paper were reasonably strong, the reviewers found several issues in the experiments. The authors provided answers which unfortunately did not satisfy the reviewers enough to increase their scores. Although this paper is not very far from meeting the bar for acceptance, I think it is still slightly below the expected level. Some things that in my opinion could tip the balance in the next version: (1) improved experiments, (2) a better justification for the proposed self-supervised task.
Decision: Accept (Poster)