Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work introduces a kind of convolution path in the transformer architecture. The main goal is to incorporate in the vision transformer architecture the intrinsic inductive bias (for instance, for modeling local visual structures) from the convolution. 
main_review: 
1. The approach is limited in novelty as this path of introducing convolutions in transformers was already explored in the previous works (for instance, in Conformer [46] and LocalViT [32]).

2. The results are unconvincing. In Table 2 the work claims superior performance over the other approaches but there are some question marks. For instance, the authors report for T2T-ViT-14 only the results with 224x224 crop size but they did not report also the results with 384x384 crop size, where T2T-ViT-14↑384 obtain an accuracy of 83.3% (see Table 2 in T2T work https://arxiv.org/pdf/2101.11986.pdf). This is curious because T2T-ViT-14↑384 outperforms the proposed approach (which obtains for ViTAE-S↑384 an accuracy of 83.0%), importantly T2T-ViT-14↑384 requires lower number of parameters and FLOPS than ViTAE-S↑384. This is especially concerning as the proposed approach is built on top of T2T, thus in this comparison the proposed approach shows even worse results than its baseline. Furthermore, this work did not report the results of T2T with both cases: transformer and performer in T2T module (see, for instance, Table 3 in T2T work https://arxiv.org/pdf/2101.11986.pdf). This may show an unfair/misleading comparison. 

3. At line 246 the authors highlighted: “It is exciting to see that our ViTAE-T with fewer parameters even outperforms the distilled model DeiT⚗, demonstrating the efficacy of introducing intrinsic IBs in transformers by design.” This is a quite unfair direct comparison, as the proposed approach is built on top of T2T, thus it accumulates the benefits from T2T and DeiT did not benefit from T2T improvement. 

4. Unclear results. In Table 3, the pre- and post-fusion have quite similar results 69.1% vs. 69%, thus there is no clear choice. However, when adding the BN, the pre-fusion outperforms significantly the post-fusion, which looks quite questionable. What is the reason that BN influences so critically the fusion type?

5. It is important to have reported also in Table 3 the number of parameters and FLOPs for each configuration. It is not very clear that these multiple dilations really help. What are the results of using just a convolution with a single dilation of 1? What are the results of using PCM and not using multiple dilations (just using the convolution with a single dilation of 1)?

6. Besides ImageNet, the work does not train from scratch on any other dataset (just performs the fine-tuning on other datasets). As the authors claim data efficiency and training efficiency, it is important to see how the approach performs when training from scratch on other datasets. Particularly, it can be important to see how it performs on smaller datasets (for instance, CIFAR10/100) where in general the vision transformer architectures did not perform that well.  
limitations_and_societal_impact: -
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposed two types of basic cells to modify the vanilla vision transformer structure: reduction cell (RC) and normal cell (NC). RCs are used to down-sample and embed the input images into tokens and NCs are to jointly model locality and global dependencies in the token sequence. Specifically, they used paralleled attention module and convolutional layers followed by a feed-forward network (FFN) to construct these two cells. Experiments on ImageNet and downstream tasks verified the superiority of the proposed network design.
main_review: Strengths:

1. The paper is well-presented.

2. Consider the intrinsic inductive bias in modeling local visual structures and scale variance in transformer for vision modality is reasonable.

Weaknesses:

1. This paper proposed to use an additional conv branch together with the self-attention branch to construct the new network architecture, it is obvious that the extra conv layers will help to improve the performance of the network. However, more conv layers will definitely involve more computational cost. My main concern of this paper is that the proposed network modification looks a little bit incremental and not very interesting to me.

2. For the experimental part, the performance is also not competitive if comparing to many recently proposed vision transformer works. I understand it may be too fastidious to consider these newly released papers, but I think using the additional branch of conv layers combining with the self-attention module, the improvement is expected, meanwhile, more FLOPs will be involved.

3. There are no results on the downstream object detection and segmentation tasks, since this paper aims to introduce the inductive bias on the visual structure. These downstream tasks are more useful in real-world applications so it will be better to see whether the proposed structure modification can consistently help on these tasks.

limitations_and_societal_impact: The network modification with more convolutional layers in this paper looks a little bit incremental and less interesting. I feel the contribution is not significant but my final rating will consider the other reviewers’ comments.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper propose to inject inductive bias (IB) for vision tasks to boost the performance of vision transformers (ViT). The main contributions of the paper are two-fold: a reduction cell that aggregate the features from different scales by controlling the dilation of a  few of parallel convolution branches and a parallel convolution branch with the multi-head self-attention (MHSA) block. The results on ImageNet is promising compared with the selected baselines but I found it is hard to get a conclusive result on downstream classification tasks.

main_review: Strong aspects:
1. The idea of injecting multi-scale features is interesting and promising.
2. The paper is well written and easy to follow.

Weakness:

1. Novelty: The main contribution of the paper is the multi-scale token aggregation of the image tokens. This is interesting as the token downsampling has been used by some recent papers and shown to be beneficial to tradeoff between the model accuracy and the resource consumption. Introduce a multi-scale reduction cell is reasonable and could protentional mitigate the information loss. However, as the experiments in the paper are mainly based on the single object classification datasets, it is not clear how many benefits come from the hand-crafted multi-scale feature aggregation. Results on multi-scale datasets could be more convincing.

2. Ablation experiments. As the contribution of the paper is two fold, it would be clearer to have an ablation of the impacts of each component. For an example, what if only reduction cell is used with following block the same as the original MHSA blocks?  

3. As the original MHSA block structure also learns the global relationship, the multi-scale features could have already been learned. Why using hand-crafted multi-scale convolution branch could improve the final accuracy?

4. Concerns on the model latency and memory consumption. In the reduction blocks and normal blocks, multi-branch convolutions are used. The proposed method is mainly verified on small input images. Thus, I am a little bit concerned about its memory consumption and running speed when applied on large images (as segmentation or detection typically uses large image resolutions).

I would be happy to increase my score if the authors could address my concerns sufficiently.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper augments vision transformers by using spatial pyramid downsamples to create multiple scales for learning, thereby introducing scale invariance as an inductive bias. Using this approach, the authors show improved performance on ImageNet and several downstream tasks across a number of metrics, including training and data efficiency. Reviewers rated the paper as borderline, with concerns regarding the novelty with respect to other hybrid model approaches and proper evaluation of baselines. The authors provided a substantial response to the reviewer concerns, but unfortunately, several of the reviewers did not respond to the author rebuttal. In my reading, the authors were able to address a number of concerns, but some points remain unclear, particularly with respect to comparison to prior models. As such, this paper would likely benefit from further work before publication.
Decision: Accept (Poster)