Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes Dynamic Hybrid Vision Transformer (DHVT) for visual recognition. They argue that spatial relevance and diverse channel representation are two important inductive biases for visual recognition and they address the issue from two perspectives.
1. On the spatial aspect, they adopt a hybrid structure, they integrate convolution into patch embedding and multi-layer perceptron (MLP) module, forcing the model to capture the token features and their neighbouring features. 
2. On the channel aspect, they introduce a dynamic feature aggregation module in MLP and head tokens in multi-head self-attention module to help re-calibrate channel representation and make different channel group representation interacts with each other.
The experiments are conducted on CIFAR-100, ClipArt, Sketch, Painting, and ImageNet. All of these datasets are for classification.
strengths_and_weaknesses: Pros:
- Well-organized and easy to read.
- Motivation is clear from both the spatial and channel sides.
- Good results on many small- and medium-size datasets.
- Nice ablation study is shown in Tab. 5.
- LImitation is stated.

Cons:
- Some unclear captions should be more concrete. E.g., DAFF in Figure. 1.
- Limited novelties. The depth-wise convolution is well investigated in previous works, like CVT, CPVT, PVT v2, etc. And the use of both channel-wise and spatial-wise representation is explored in DaVIT, CoaT, etc.
- What is the intuitive explanation of the HI-MHSA? The component is a bit complicated.
- Why is the work suitable for small datasets? Why does the author put the experiment on ImageNet in Appendix? Which part is specifically designed for small datasets?
- No downstream tasks were evaluated. It would be better if the authors could provide some results on ADE20K or COCO.
- The use of batchnorm, how does it influence the performance? VIT-based architectures typically use layernorm so that the batch size will not hinder the performance.
questions: See the Strengths And Weaknesses part.
limitations: See the Strengths And Weaknesses part.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new Vision Transformer architecture that merges CNNs and ViT to have the best of the two worlds. The paper focuses on small datasets and Vision Transformers, which have to be somehow regularised or helped to learn the inductive biases that are naturally enforced with CNNs.

The paper proposes a new architecture that modifies the FFN with a Dynamic Aggregation Feed Forward, and proposes a "Head Token" so that each head of the MHA can attend to all others somehow. This supposedly helps learning better features.

The authors experimented the proposals with various small datasets and observed improvements in all tests.
strengths_and_weaknesses: PROS:
- the problem is very important: Vision Transformers focus a lot on big datasets but small datasets are nowadays almost forgotten
- the paper is clear

CONS:
- lack of comparison with some state of the art
- lack of clarity on the motivations for some choices

DETAILS:
To me some parts of the paper are unclear. E.g.:
- the choice of batch norm
- why you did not compare with CvT, very famous architecture joining CNNs with Transformers
- why you did not compare with [54], published one year ago in the same conf
- why is the Head Token beneficial? I mean, the multi-head attention was created to have somehow the same approach CNNs have with multiple filters: learning different aspects of the same input. I am surprised your modification helps

I did not also like the lack of clarity of the experiments. Why is the resolution of CIFAR-10 changed? What happens if you do not do the changes explained in L244? Are all baselines trained with the original augmentations, params, resolutions?

Then, [54] reports 75.22 on ResNet-50 on ClipAart, 66.58 on Paintings etc. Your results are lower both for the baseline and your method. Why?

L288 why 4 heads? it is not standard at all.

L265 DomainNet is much bigger. Why did you select only a few datasets from DomainNet? Unfortunatedly, this raises the question on me whether you cherrypicked the results. Moreover, depending on the Table (2, 3, 4) the baselines change. The reason is a bit unclear.


OTHER:
- L144: needs a citation, e.g. 54. However, 54 shows that the networks are competitive with ResNet-50
- L55-79: it is not clear why it is important to solve everything in the spatial dimension.
- L79 SE is not defined
- L150: non-hierarchical here is not clear. Try to explain it better.
- L180: this is a result I suppose, on the methods. If you want to state this, show the results.
- L181: more solution is not english
- 
questions: - The Transformer is a good architecture also because it does not depend too much on the batch size. However, your architecture now has Batch Normalization. Why? Couldn't you use Layer Norm? If not, why?

- Why didn't you compare with [54]?

- What is the impact of the batch size on your results?

- what if I use this architecture with a big dataset? Is this an architecture that is general or only for small datasets? what is your opinion here

- What about downstream tasks? Does your architecture help there?

- answer the other questions raised in the previous section plz!
limitations: No limitations
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a new Vision Transformer structure named DHVT targeting to improve the performance of ViTs on the image classification task with small datasets. Specifically, DHVT is designed to improve the spatial relevance among the patches with convolutional inductive bias and enhance the channel representation by re-calibrating the channel representations. To validate the effectiveness of the design choices, the authors conducted experiments on small datasets represented by CIFAR-100 and ImageNet-1k. The CIFAR-100 results look good, and the proposed method outperforms the other baseline methods by large margins.
strengths_and_weaknesses: Strengths:
- It’s meaningful to study why and how ViTs are inferior to CNNs when training on small datasets from scratch.
- The quantitative results in Table 2 look good and surpass the other methods. 

Weakness:
- It’s hard to understand the motivation for the proposed detailed design choices and why they are better than other simple designs. For instance, why re-calibrating only the class token with SE operation [25], considering localViT [18] proposed re-scaling channels for all tokens in FFN blocks? A proper discussion with the SE module in localViT is needed. Also, the head tokens are proposed to enhance the channel groups for each head by modeling head correlations, which can easily be implemented by an MLP layer. Can you explain the benefit of modeling the head correlations with MSA instead of a simple MLP layer?

- The experiments are insufficient to support the authors’ claim. Tables 3 and 4 have very few baseline methods and it’s hard to see the gain of DHVT over the SOTA methods.

- The novelty for modeling spatial relevance is limited, since introducing early convolutions has been shown effective in previous works, e.g. [a]. 

[a] Xiao, Tete, et al. "Early convolutions help transformers see better." Advances in Neural Information Processing Systems 34 (2021): 30392-30400.

questions: 
Please address the issues pointed out in the weakness.
limitations: In general, yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focuses on generalizing vision transformers to the domain of small datasets through injecting spatial-wise and channel-wise inductive biases into model architecture design. Specifically, the authors point out that it is hard for the vanilla vision transformers to model spatial relevance and diverse channel representation under data-constrained scenarios. The weaknesses for vision transformers compared to their CNN counterparts finally lead to worse performances on small datasets, *e.g.,* cifar. To address the drawbacks, the authors introduce two augmented modules SOPE and DAFF, corresponding to the patch embedding and feedforward operators. By using additional convolution operations to gather the local feature, spatial relevance within patches is emphasized, and is forced to learn better relations even with limited data. On the other hand, *head tokens* are adopted on top of the widely used classification token and serve as the global token for each group of channels (*i.e.,* a head) in the multi-head self-attention module. In this way, the interaction across different heads is facilitated and shows better representation capability under a low data regime. Regarding the experiments, the authors validate the effectiveness of their module on various datasets, including small datasets like cifar100 and DomainNet, and larger datasets like ImageNet-1K. A good trade-off between model parameters and test accuracy is shown compared to several state-of-the-art baselines, and ablation studies are also conducted to show the contribution of each part.
strengths_and_weaknesses: + Strengths
  + This paper is organized in an easy-to-follow pattern. Related works on data-efficient vision transformers are thoroughly investigated and properly introduced. Each component of the proposed model is described in detail respectively. The experiments are substantial and some results and experiment details in the supplementary material also help to understand the paper.
  + The practical value of this paper is promising. Although vision transformer models show better performance and scalability on large-scale datasets, the potential application on data constraint scenarios is still under-explored. The gap between vision transformer and cnns on the small dataset is a topic that is worth deep investigation. This paper attempts to address this limitation from the spatial and channel perspective, and the experiments show improvements on widely used benchmarks, *e.g.,* cifar100. I think this shows a new direction to practically using vision transformer models on datasets with low-scale / low-resolution data.
  + The experiments are sufficient in this paper. The parameter-accuracy trade-off on three datasets, ablative studies on each proposed component, visualization on attention maps, and the flops-accuracy trade-off in the supplementary material are detailedly described.
  + Finally, I really approve that the authors are willing to share the code of each module. This really helps to understand the implementation detail.

+ Weaknesses
  + The motivation of this paper can be addressed more clearly. The authors point out that spatial relevance and diverse channel representation are two essential aspects that influence the performance of vision transformers under a low data regime. Nevertheless, I am still concerned about the practical connection between **lack of two inductive biases** with **low data regime**. To my knowledge, the lack of inductive biases is also a question for large-scale datasets like ImageNet-1k. Previous research [1] have already explored strengthening the spatial relationship between patches, and the diversity of features among different heads has also been investigated [2]. As a result, the fact that the lack of two inductive biases leads to inferior performances is not so surprising. As the authors are motivated from these two perspectives to address the limitation of the vision transformer, I think it would be better if they can **emphasize the unique influence of the inductive biases for small datasets**, and explain how the situation differs from large-scale datasets.

  + Technical contribution. It is an interesting idea to improve the performance of the vision transformer from spatial and channel perspectives. However, it must be pointed out that the proposed modules share some resemblance with several previous works. The convolutional patch embedding is widely adopted in several works [1][3]. Using depthwise convolution in the feedforward network is also a common trick [4]. The dynamic aggregation feedforward module also seems like a trivial implementation of SENet, and similar insight has been explored in [5]. It would be better if the authors can focus on addressing the differences with previous works.

  + I also have concerns about some experimental settings. For the main results on cifar100 shown in table 2, I wonder if the comparisons are fair. According to the authors, DeiT augmentations are adopted for proposed models. Do previous cnn models use the same augmentation? Also, the timm implementation can practically improve the model performances of cnn models. Have the authors reimplement the baselines on timm and see the differences with the original paper? I understand it is impractical to ask the authors to re-implement all the baseline approaches, I think a simple comparison with a widely used model like ResNeXt would also do the trick and makes the result more convincing. In section 4.2, the authors state that the results are the best out of 5 runs. This is clearly different from the routine in previous works where an average of 10 runs are usually used for cifar datasets. Considering the large variance for cifar100, I think some part of the gain may come from the randomness.



[1] Xiao, Tete, et al. "Early convolutions help transformers see better." Advances in Neural Information Processing Systems 34 (2021): 30392-30400.

[2] Raghu, Maithra, et al. "Do vision transformers see like convolutional neural networks?." Advances in Neural Information Processing Systems 34 (2021): 12116-12128.

[3] Yuan, Li, et al. "Volo: Vision outlooker for visual recognition." arXiv preprint arXiv:2106.13112 (2021).

[4] Guo, Jianyuan, et al. "Cmt: Convolutional neural networks meet vision transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[5] Yuan, Li, et al. "Tokens-to-token vit: Training vision transformers from scratch on imagenet." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
questions: Most of my concerns are addressed in the **Strengths And Weaknesses** section. The followings are my detailed questions regarding the weaknesses.

+ I understand that the authors have stated several times in the paper like in the abstract '... the lack of data hinders ViTs to attend the spatial relevance ...' or '... the scarce data can not enable ViTs to learn strong enough representation ...', nevertheless, from my perspective, these statements are more like intuitive claims that short of technique supports. For example, it would be better if some qualitative results can show that spatial relevance is more critical for small datasets. The previous work [1] actually shows that lower attention layers do not learn to attend locally with less training data. Nevertheless, the *less training data* in [1] refers to ImageNet-1k without ImageNet-22k pretraining, which still belongs to large-scale data in the scope of this paper. I wonder if this claim is also correct with small datasets like cifar.

+ Do the setups in table 2 correct? I have checked the wide-resnet paper and found that WRN28-10 is trained for 200 epochs instead of 300 epochs. Also, as I have mentioned before, reporting the best results of 5 runs may enjoy the gain from randomness, especially on small datasets.

+ A comparison on flops-acc trade-off for cifar100 would be better. According to supplementary material, the DHVT-S model that shows the highest performance (85.68) has 6.3G flops, which requires high computation for small image size. The trade-off for computation cost would be more valuable than training parameters.

+ A Typo in line 181: require->required.

[1] Raghu, Maithra, et al. "Do vision transformers see like convolutional neural networks?." Advances in Neural Information Processing Systems 34 (2021): 12116-12128.
limitations: The authors have addressed the limitations. I feel that inference speed is also a question that is worth further investigation. Although the authors provide the model throughput in the supplementary, the detailed comparison with baseline models and efficient cnns is still lacking.
review_title: null

Meta-review: metareview: Authors introduce 3 modifications to ViT architecture to introduce additional inductive biases to improve performance in low-data scenarios:
- SOPE: Sequential Overlapping Patch Embedding -- essentially convolutions before partitioning the image into patches.
- DAFF: Dynamic Aggregation Feed Forward -- a DWCONV operation is applied to tokens after a FC layer increases the channel dimension. The new tokens are average pooled, input to additional FC layers, and then are used to scale the CLS token.
- HI-MHSA: Head-Interacted Multi-Head Self-Attention -- this approach's name is confusing. This does not change the heads in MHSA. Rather a new mechanism is introduced prior to MHSA to introduce new tokens where each new token is derived from a different partition of the original channel dimensions.  AC recommends authors use a different name. For example, "Intra-Channel Modeling (ICM) MHSA" or something would be more clear.  
	
Performance is evaluated on CIFAR-100, DomainNet subsets, and ImageNet 1K

Pros:
- [R/AC] The topic is important to the community.
- [R/AC] The paper is well written and clear. 
- [R/AC] The authors present improved performance versus other recent SOTA hybrid model designs (during rebuttal phase, though missing from original work -- should be added to paper). 

Cons:
- [R/AC] The evaluation could be significantly improved. For example, more training experiments on undersampled version of more datasets, with comparisons to other SOTA methods. 
- [R/AC] The design is complicated and the motivation isn't always clear. 
- [R] Novelty of the components implemented is low. 
- [R] Concerns over use of BN as opposed to LN. Authors have provided ablation experiments to demonstrate that BN improves performance of their model over LN. These ablations should be included in the manuscript.
- [R/AC] Concerns over lack of comparison to other SOTA methods that mix convolutions with transformers, such as CvT. Authors have provided additional experiment tables that compare against CvT. These tables should be included in the manuscript in a consistent manner (showing number of parameters and FLOPS).
- [R/AC] Authors do not include FLOPS in their experiment tables. Please ensure all tables report number of parameters and FLOPS for all models explored. There are python packages to help with computing this, such as "flopth". 
- [AC] Some spelling and grammatical mistakes. Please spell check the manuscript.

Overall Recommendation: Reviews lean toward acceptance, but marginally so. Given that the authors have provided more comparisons against recent relevant SOTA methods, and that the reviewers (including expert in the field) lean toward accept, the AC opinion is that this manuscript can be published and provides some valuable knowledge to the community. There are ways in which the paper can still be improved before publication, such as inclusion of additional evaluation datasets.

AC Rating: Borderline Accept

confidence: Less certain
Decision: Accept