Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets
Authors: Zhiying Lu
Hongtao Xie
Chuanbin Liu
Yongdong Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: vision transformer

Abstract: There still remains an extreme performance gap between Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) when training from scratch on small datasets, which is concluded to the lack of inductive bias. In this paper, we further consider this problem and point out two weaknesses of ViTs in inductive biases, that is, the spatial relevance and diverse channel representation. First, on spatial aspect, objects are locally compact and relevant, thus fine-grained feature needs to be extracted from a token and its neighbors. While the lack of data hinders ViTs to attend the spatial relevance. Second, on channel aspect, representation exhibits diversity on different channels. But the scarce data can not enable ViTs to learn strong enough representation for accurate recognition. To this end, we propose Dynamic Hybrid Vision Transformer (DHVT) as the solution to enhance the two inductive biases. On spatial aspect, we adopt a hybrid structure, in which convolution is integrated into patch embedding and multi-layer perceptron module, forcing the model to capture the token features as well as their neighboring features. On channel aspect, we introduce a dynamic feature aggregation module in MLP and a brand new "head token" design in multi-head self-attention module to help re-calibrate channel representation and make different channel group representation interacts with each other. The fusion of weak channel representation forms a strong enough representation for classification. With this design, we successfully eliminate the performance gap between CNNs and ViTs, and our DHVT achieves a series of state-of-the-art performance with a lightweight model, 85.68% on CIFAR-100 with 22.8M parameters, 82.3% on ImageNet-1K with 24.0M parameters. Code is available at https://github.com/ArieSeirack/DHVT.

Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets
Zhiying Lu, Hongtao Xie*, Chuanbin Liu", Yongdong Zhang
University of Science and Technology of China, Hefei, China
arieseirackCmail ustc. edu. en, (htxie, www.c9i,ccliiiii 2uw.csss,,,yy73)auuuustttt edu. en
Abstract
There still remains an extreme performance gap between Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) when training from scratch on small datasets, which concluded to the lack of inductive bias. In this paper, we further consider this problem and point out two weaknesses of ViTs in inductive biases, that is, the spatial relevance and diverse channel representation. First, on spatial aspect, objects are locally compact and relevant, thus fine- ine-grained feature needs to extracted from token and its neighbors. While the lack of data hinders VITs to attend the spatial relevance. Second, on channel aspect, representation exhibits diversity on different channels. But the scarce data can not enable ViTs to learn strong enough representation for accurate recognition. To this end, we propose Dynamic Hybrid Vision Transformer (DHVT) as the solution to enhance the two inductive biases. On spatial aspect, we adopt hybrid structure, in which convolution is integrated into patch embedding and multi-layer perceptron module, forcing the model to capture the token features well as their neighboring features. On channel aspect, we introduce a dynamic feature aggregation module in MLP and brand new "head token" design in multi-head self-attention module to help re-calibrate channel representation and make different channel group representation interacts with each other. The fusion of weak channel representation forms strong enough representation for classification. With this design, we successfully eliminate the performance gap between CNNs and ViTs, and our DHVT achieves series of state-of-the-ar performance with a lightweight model, 85.68% on CIFAR-100 with 22.8M parameters, 82.3% ImageNet- with 24.0M parameters. Code is available at https::/ //g thub. com/ Ar ArieSeirack/DNV
Introduction
Convolutional Neural Networks (CNNs) have dominated in Computer Vision (CV) field as the
backbone for various tasks like classification [1, 2, 3, 4, 5, 6, 7], object detection [8, 9, 10] and segmentation [11, 12, 13]. These years have witnessed the rapid growth of another promising alternative architecture paradigm, Vision Transformers (ViTs). They have already exhibited great performance in common tasks, such classification [14, 15, 16, 17, 18, 19], object detection [20, 21, 22] and segmentation [23,2] 24].
VIT [14] is the pioneering model that brings Transformer architecture [25] from Natural Language Processing (NLP) into CV. It has a higher performance upper bound than standard CNNs, while is at the cost of expensive computation and extremely huge amount of training data. The vanilla ViT needs to be firstly pre-trained on the huge dataset JFT-300M [14] and then fine-tuned on the common dataset ImageNet-1K [26). Under this experimental setting, it shows higher performance than standard CNNs. However, when training from seratch on ImageNet- IK only, the accuracy
Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

much lower. From the practical perspective, most of the datasets are even smaller than ImageNet- IK, and not all the researchers can hold the burden of pre-training their own model on large datasets and then fine-tuning on the target small datasets. Thus, an effective architecture for training ViTs from scratch on small datasets is demanded.
Recent works [27. 28, 29] explore the reasons for the difference in data efficiency between ViT and CNNs, and draw a conclusion to the lack of inductive bias. In [27], it points out that with not enough data, VIT does not learn to attend locally in earlier layers. And in [28], it says that the stronger the inductive biases, the stronger the representations. Large datasets tend to help VIT learn strong representations. Locality constraints improve the performance of ViT. Meanwhile, in recent work [29]. it demonstrates that convolutional constraints can enable strongly sample -effciin training in the small-data regime. The insufficient training data makes ViT hard to derive the inductive bias of attending locality, thus many recent works strive to introduce local inductive bias by integrating convolution into ViT's [18, 15, 30, 31, 32] and modify it to hierarchical structure [33, 34, 16, 17,35], making ViTs more like traditional CNNs. This style of hybrid structure shows comparable performance with strong CNNs when training from scratch on medium dataset ImageNet- IK only. But the performance gap on much smaller datasets still remains.
Here, we consider that the scarce training data weakens the inductive biases in ViTs. Two kinds of inductive bias need to be enhanced and better exploited to improve the data efficiency, that is, the spatial relevance and diverse channel representation. On spatial aspect, tokens are relevant and objects are locally compact. The important fine- -eeeeerried low-level feature needs to be extracted from the token and its neighbors at the earlier layers. Rethinking the feature extraction framework in ViTs, the module for feature representation is the multi-layer perceptron (MIP) and its receptive field can be seen as only itself. So ViTss depend on the multi-head self-attention (MHSA) module to model and capture the relation between tokens. As is pointed out work [27], with less training data, lower attention layers do not learn to attend locally. In other words, they do not focus on neighboring tokens and aggregate local information in the early stage. As is known, capturing local features in lower layers facilitates the whole representation pipeline. The deep layers sequentially process the low-level texture feature into high-level semantic features for final recognition. Thus ViTs have an extreme performance gap compared with CNNs when training from scratch on small datasets. On channel aspect, feature representation exhibits diversity in different channels. And ViT has its own inductive bias that different channel group encodes different feature representation of the object, and the whole token vector forms the representation of the object. As is pointed out in work (28), large datasets tend to help ViT learn strong representation. The insufficient data can not enable ViTs to learn strong enough representation, thus the whole representation is poor for accurate classification.
In this paper, we solve the performance gap of training from scratch small datasets between CNNs and ViTs and provide a hybrid architecture called Dynamic Hybrid Vision Transformer (DHV'T as a substitute. We first introduce hybrid model to address the issue spatial aspect. The proposed hybrid model integrates a sequence of convolution layers in the patch embedding stage to eliminate the non-overiapping problem. preserving fine- mnee-graind low-level features, and it involves depth-wise convolution [36] in MLP for local feature extraction. In addition, we design two modules for making feature representation stronger solve the problem on channel view. To be specific, in MIP, depth-wise convolution is adopted for the patch tokens, and the class token is identically passed through without any computation. We then leverage the output patch tokens to produce channel weight like Squeeze-Excitation (SE) [4] for the class token. This operation helps re-calibrate each channel for the class token to reinforce its feature representation Moreover, in order enhance interaction among different semantic representations of different channel groups and owing to the variable length of the token sequence in vision transformer structure, we devise a brand new token mechanism called "head token". The number of head tokens is the same as the number of attention heads in MHSA. Head tokens are generated by segmenting and projecting input tokens along the channel. The head tokens will be concatenated with all other tokens to pass through the MHSA. Each channel group in the corresponding attention head in the MHSA now is able to interact with others. Though maybe the representation in each channel and channel group is poor for classification on account of insufficient training data, the head tokens help re-calibrate each learned feature pattern and enable a stronger integral representation of the object, which is beneficial to final recognition.
We conduct experiments of training from scratch on various small datasets, the common dataset CIFAR-100 and small domain datasets Clipart, Painting and Sketch from DomainNet [37] to examine the performance of our model. On CIFAR- 100, our proposed models show a significant performance
2

---

margin with strong CNNs like ResNeXt, DenseNet and Res2Net. The Tiny model achieves 83.54% with only 5.8M parameters. and our Small model reaches the state-of-the-art 85.68% accuracy with only 22.8M 2M parameters, outperforming series of strong CNNs. Therefore, we eliminate the gap between CNNs and ViTs, providing an alternative architecture that can train from scratch on small datasets. We also evaluate the performance of DHVT when training from scratch on ImageNet- I K. Our proposed DHVT-S achieves competitive 82.3% accuracy with only 24.0M parameters, which is the state-of-the-art art non-hierarchical vision transformer structure as far as we know, demonstrating the effectiveness of our model on larger datasets. In summary, our main contributions are:
1. We conclude that the data efficiency on small datasets can be addressed by strengthening two inductive biases in ViTs, which are spatial relevance and diverse channel representation.
2. On spatial aspect, we adopt a hybrid model integrated with convolution, preserving fine-grained low-level features at the earlier stage and forcing the model to extract tokens feature and corresponding neighbor feature.
3. On channel aspect, we leverage the output patch tokens to re-calibrate class producing better feature representation. We further introduce "head token", a novel design that helps fuse diverse feature representation encoded in different channel groups into a stronger integral representation.
token channel-wise
2 Related Work
Vision Transformers. Convolutional Neural Networks [38, 39, 1, 40, 2, 41] dominated the computer vision fields in the past decade, with its intrinsic inductive biases designed for image recognition. The past two years witnessed the rise of Vision Transformer models in various vision tasks [42, 43, 23, 20, 44, 45]. Although there exist previous works introducing attention mechanism into CNNs [4, 46, 47], the pioneering full transformer architecture in computer vision are iGPT [48] and ViT [14]. ViT is widely adopted as the architecture paradigm for vision tasks especially image recognition. It processes the image as token sequence and exploits relations among tokens. It uses "class token" like BERT [49] to exchange information at every layer and for final classification. It performs well when pre-trained on huge datasets. But when training from scratch on ImageNet- 1K only, underperforms ResNets, demonstrating data-hungry problem.
Data-efficient viTs. Many of the subsequent modifications ViT strive for more data-efficient architecture that can perform well without pre-training on larger datasets. The methods can be divided into different groups. [42, 50] use knowledge distillation strategy and stronger augeenttgmtntiionn methods to enable training from scratch. [51] points out that using convolution in the patch embedding stage greatly benefits ViTs training. [52, 15, 18, 53, 54] leverage convolution for patch embedding to eliminate the discontinuity brought by non-overlapping patch embedding in vanilla ViT, and such design becomes paradigm in subsequent works. To further introduce inductive bias into ViT. [15, 30, 34, 55, 56] integrate depth-wise convolution into feed forward network, resulting in hybrid architecture combining the self-attention and convolution. To make ViTs more similar to standard CNNs, [16, 54, 17, 34, 33, 35, 57, 32] re-design the spatial and channel dimension of vanilla ViT. producing a series of hierarchical style vision transformer. [31, 58, 59] design another parallel convolution branch and enable the interaction with the self-attentio branch, making the two branch complements each other. The above architectures introduce strong inductive bias and become data-efficient when training from scratch on ImageNet-1K K. In addition, works like [60, 61, 62] investigate channel-wise representation through conducting self-attention channel-wise, while we enhance channel representation by dynamically aggregating patch token features to enhance class token channel wise and compatibly involve channel group-wise head tokens into vanilla self self-attention. Finally, works like [63, 64, 65], suggesting that the number of tokens can be variable.
VITs for small datasets. There exists several works on solving the training from scratch problem on small datasets. Though the above modified vision transformers perform well when trained on ImageNet- 1K they fail to compete with standard CNNs when training on much smaller datasets like CIFAR-100. Work [66] introduces self-supervised style training strategy and a loss function to help train ViTs on small datasets. CCT [67] adopts a convolutional tokenization module and replaces the class token with the final sequence pooling operation. SL-VIT [68] adopts shifted patch tokenization module and modifies self-attention to make focus more locally. Though the previous works reduce the performance gap between standard CNNs ResNets[[], they fail to be sub-optimal when compared
3

---

with strong CNNs. Our proposed method leverages local constraints and enhances representation interaction, successfully bridging the performance gap on small datasets.
3x3 Cons,
LayerNorm
BatchNorm
Sequential Overlapping Patch Embedding
BI-MHSA
GELU
A Affine
Encoder Layer
Reshape
Lay LoyerNom
Encoder Layer
DAFF
token
FC Head
patch token
Classification Result
Figure 1: Overview of the proposed Dynamic Hybrid Vision Transformer (DHVT). DHV'T follows non-hierarchical structure, where each encoder layer contains two pre-norm and shorteut, Head- Interacted Multi-Head Self-Attention (HI-MHSA) and a Dynamic Aggregation Feed Forward (DAFF).
3 Methods
3.1 Overview of DHVT
As shown in Fig. 1, the framework of our proposed DHVT is similar vanilla ViT. We choose non-hierarchical structure, where every encoder block shares the same parameter setting, processing the same shape of features. Under this structure, we can deal with variable length of token sequence. We keep the design of using class token to interact with all the patch tokens and for final prediction. In the patch embedding module, the input image will be split into patches first. Given the input image with resolution H x W and the target patch size P. the resulting length of the patch token HW/P² Our modified patch embedding is called Sequential Overlapping
sequence will be Patch Embedding (SOPE). which contains several successive convolution layers of 3 x 3 convolution with stride 2, Batch Normalization and GELU [69] activation. The relation between the number of convolution layers and the patch size is P 2 SOPE is able to eliminate the discontinuity brought by the vanilla patch embedding module, preserving important low-level features. It is able to provide position information to some extent. We also adopt two affine transformations before and after the series of convolution layers. This operation rescales and shifts the input feature, and acts like normalization, making the training performance more stable small datasets. The whole process of SOPE can be formulated as follows.
Aff(x) = Diag(a)x
G((x) = GELU I,...,
(2)
SOPE(x) = Reshhap(AAf[(CL. C0
(3)
In Eq. and (3 are learnable parameters, and initialized and respectively. After the sequence of convolution layers, the feature maps are then reshaped as patch tokens and concatenated with class token. Then the sequence of tokens will be fed into encoder layers. After SOPE, the token sequence will pass through layers of encoder, where each encoder contains Layer Normalization [70], multi-head self-attention and feed forward network. Here we modified the MHSA as (lead-Interacte (HI-MHSA) and feed forward network as Dynamic Aggregation Feed
Multi-Heed Self-Attentic Forward (DAFF). We will introduce them the following sections. After the final encoder layer, the output class token will be fed into the linear head for final prediction.

---

3.2
Dynamic Aggregation Feed Forward
FC
DIVCONV
GELU
PC
AveiPool
+--
FC
GELU
PC
ds token
patch tokan
concat
spiit
Figure 2: The structure of Dynamic Aggregation Feed Forward (DAFF).
The vanilla feed forward network (FFN) in ViT is formed by two fully-connected layers and GELU activation. All the tokens, either patch tokens or class token, will be processed by FFN. Here integrate depth-wise convolution [36] (DWCONV)
FFN and resulting in a hybrid model. Such hybrid model is similar standard CNNs because can be seen as using convolution do feature representation. With the inductive bias brought by depth-wise convolution, the model is forced to capture neighboring features, solving the problem on spatial view. It greatly reduces the performance gap when training from scratch on small datasets, and converges faster than standard CNNs. However, such a structure still performs worse than stronger CNNs. More investigations are required solve the problem on channel aspect.
We propose methods that make the whole model more dynamic and learn stronger feature representation under insufficient data. The first proposed module Dynamic Aggregation Feed Forward (DAFF). We aggregate the feature of patch tokens into the class token in channel attention way, similar to the Squecze-Excitation operation in SENet [4], as is shown in Fig. 2. Class token split before the projection layers. Then the patch tokens will go through depth-wise integrated multi-layer perceptron with a shortcut inside. The output patch tokens will then be averaged into a weight vector W. After the squeeze-excitation operation, the output weight vector will be multiplied with class token channel-wise. Then the re-calibrated class token will be concatenated with output patch tokens to restore the token sequence. We use Xe, Xp denote class token and patch tokens respectively. The process can be formulated as:
W Tinear(GETT((lnnear)
(X))))
X. = X
(5)
3.3 Head Token
The second design enhance feature representation "head token". which a brand new mechanism as far as we know. There are two reasons why we introduce head token here. First, in the original MHSA module, each attention head has not interacted with others, which means each head only focuses on itself to calculate attention. Second, channel groups in different heads are responsible for different feature representations, which is the inductive bias of ViTs. And as we pointed out above, the lack of training data can not enable models to learn strong representation. Under this cireumstance, the representation in each channel group too weak for recognition. After introducing head tokens into attention calculation, the channel group in each head are able to interact with those the object.
in other heads, and different representation can be fused into an integral representation Representation learned by insufficient data may be poor in each channel, but their combination will produce a strong enough representation. The structure of vision transformer also guarantees this mechanism because the length of input tokens is variable, except for the hierarchical structure vision transformer with window attention such 35].
The process of generating head tokens is shown in Fig. 3 (a). We denote the number of patch tokens as N, so the length of the input sequence N + According to the pre-defined number of heads each D-dimensional token, including class token, will be reshaped into h parts. Each part contains d channels, where D totally h tokens and each one is d-dimensional. All such intermediate tokens will be projected into
We average all the separated tokens in their own parts. Thus we get
5

---

D
toput tokens
token
petch token
N+1
N+1+h
head, token
Reshape
head embedding
split
D = dxc
concat
Multt-
N+1
head tokens
Self ttention
Average
h
D
d
d
h
GELU
A Avergge
Reshape
PC
N+1
output tokens
(a) Generate Head Tokens
(b) Multi-Hedk Self-Attention
Figure Pipeline Head-Intedd Interacted Multi-Hear Self-Attention (HI-MHSA).
D-dimension again, resulting in head tokens in total. The head tokens will be added with head embedding, which provides positional information for head tokens. Head embedding is group of learnable parameters, just like positional embedding. Finally, they are concatenated with patch tokens and class token, forming the token sequence for standard MHSA, as Eq. 7, in which X denotes head tokens. We do not change the attention calculation in MHSA. Head tokens will also linearly projected into query, key and value, and they will interact with all other tokens. After MHSA, the head tokens will be averaged and added class token, just as Fig. (b) shows. Head tokens can be derived as Eq. 6 shows. We use Epead to denote head embedding.
X# GELU (Aaeniiieeeeeeeerootssssaaaaal)
Exend
(6)
X
Xx; Xp; Xa)
[X: X ...... x5 ; X X....., X
4 Experiments
All the experiments presented in our paper are based on image classification. We do not conduct experiments downstream tasks. We first introduce the training datasets and experimental settings in Section 4.1. 1. The performance comparisons are shown in Section 4.2. We also show the result of the ablation study in Section 4.3, And finally, we present an example of visualization in 4.4.
4.1 Datasets and Experimental Settings
Datasets. Our main focus is training from scratch on small datasets. There are two factors to consider whether dataset is small: the total number of training data in the dataset and the average number of training data for each class. Some datasets are small on the first factor, but large on the second. The example is CIFAR-10 [71], with 50000 training data in total for 10 classes, has average of 5000 instances in each class. Considering this, we do not choose CIFAR-10 10 as our target dataset here. We choose 5 different datasets here. The main performance comparisons
on CIFAR- 100 [71]. And we choose three datasets from DomainNet [37], a benchmark commonly for domain adaptation tasks. They have large domain-shin shift from common medium dataset ImageNet- K. [26], making the fine-tuning experiments non-tric triviall as pointed in [66]. Finally, we also choose ImageNet-I 1K to test the performance of our proposed model. The details of the datasets shown in Table 1.
Model Variants. propose two architecture variants. Detailed information on model variants can be seen in Supplementary Materials.

---

Table 1: The details of training datasets. We report the train and test size of each dataset, including the number of classes. We also show the average images per class in the training set.
Dataset
Train size
Test size
Classes
Average images per class
CIFAR-100 [71]
50000
10000
100
500
ClipArt [37]
33525
14604
345
97
Sketch [37]
48212
20916
345
140
Painting [37]
50416
21850
345
146
ImageNet- I [26]
1281167
100000
1000
1281
DHVT-T: 12 encoder layers, embedding dimension of 192, MLP ratios of 4, attention heads of 4 on CIFAR-100 and DomainNet, and 3 on ImageNet-1KK
DHVT-S: 12 encoder layers, embedding dimension of 384, MLP ratios of 4, attention heads of 8 CIFAR-100, 6 on DomainNet and ImageNet-IK.
Implementation Details, When training our DHVT, we keep the image size in CIFAR-100 as its original resolution 32 x 32, and the patch size is set to 4 or 2. For ImageNet-IK, ClipArt, Painting and Sketch, we adopt resolution 224 x 224, and the patch size comes to 16. All the data augmentations the same as those in DeiT [42]. We do not tune data-augmentation hyperparameters for better performance. On all of the datasets, we train our network from random initialization with the Adam W [72] optimizer with a cosine decay learning-raid scheduler. We set the batch size of 512 and 256 for DHVT-T and DHVT-S when training on CIFAR- 100, with an initial learning rate of 0.001, and weight decay of 0.05, warm-up epoch of When on ClipArt, Sketch and Painting, we use a batch size of 256 and 128 respectively for DHVT-T and DHVT-S, with an initial learning rate of 0.001, a warm-up epoch of 20 and weight decay of 0.05. For ImageNet- IK, we use the batch size of 512 for both models with and initial learning rate of 0.0005 and weight decay of 0.05, a warm-up epoch of 10. The training epochs on all datasets are 300, except that WRN28- 10 [73] is trained for 200 epochs on CIFAR. All of the training devices are Nvidia 3090 GPUs. We use Pytorch tools and our code is modified from timm'.
4.2 Performance Comparisons
Table 2: Results on DomainNet
Table 3: Results on ImageNet- IK
Method
#Params
ClipArt
Painting
Sketch
Method
#Params
ImageNet-IK
Re
24.2M
71.90
64.36
67.45
DHVT-T
6.2M
76.5
DHVT-T
6.1M
71.73
63.34
66.60
DHVT-S
24.0M
82.3
DHVT-S
23.8M
73.89
66.08
68.72
Results DomainNet. We also conduct experiments other small datasets. Here we choose three datasets from DomainNet as our target. We use the implementation of ResNet-50 in Pytorch official code for performance comparison. All of the anttentugmett,ion such as Mixup [74] and CutMix [75] and Automugment [76], are also adopted for training ResNet-50 from scratch on these datasets. All of the results reported are the best out of four runs. As is shown in Table 2, our model shows better results than standard ResNet-50, demonstrating its performance across different small datasets. The whole comparison of all of the DomainNet datasets with more baseline models is shown in Supplementary Materials.
Results on Image 1K To test the train-from-scratee performance of our model on the common medium-size dataset ImageNet- IK, we also conduct experiments on it. We follow the same ex- perimental settings in DeiT [42]. The results are shown in Table 3. Surprisingly, reaches 76.47 accuracy and our DHVT-S reaches 82.3 accuracy. As far as we know, this is the best performance under such non-hicnnn non-hierarchical vision transformer structure with class token. And our model outperforms many of the state-of-the-ie methods with comparable parameters. This experiment shows that our model not only behaves well on small datasets but also exhibits powerful performance on larger datasets. We will show the performance comparison with other methods that train from scratch on I in the Supplementary Materials.
DHVT-T
'https://gi http:/gtttth....iiiiieeeeeeeotee wwwptwwwgttaa//p/eeeeeeeooeeeeee
7

---

Table 4: Performance comparison of different methods the CIFAR-100 dataset. All models
trained from random initialization. "*" denotes that we re-implement the method under the same training scheme. The other results are cited from the corresponding works.
Type
Method
Patch Size
#Params
GFLOPs
Acc (%)
WRN28-10 [73]
36.5M
5.2
80.75
SENet-29 [4]
35,0M
5.4
82.22
ResNeXt-29, 8x64d [40]
I
34.4M
5.4
82.23
CNN
SKNet-29 [41]
27.7M
4.2
82.67
DenseNet-BC (k 40) [2]
25.6M
9.3
82.82
Res2NeXt-29, 6cx24wx6s-SE [77]
36.9M
5.9
83.44
DeïT-T [42]*
4
5.4M
0.4
67.59
DeiT-S [42]
21.4M
1.4
66.55
DeïT-T [42]*
2
5.4M
1.4
65.86
VIT
DeiT-S [42]*
21.4M
5.5
63.77
PVT-T [35]
15.8M
0.6
69.62
PVT-S [35]
27.0M
1.2
69.79
Swin-T
27.5M
1.4
78.07
NesT-T [35]
6.2M
1.7
78.69
NesT-S [35]
23,4M
6.6
81.70
NesT-B [35]
90.1M
26.5
82.56
CCT-7/3 1 [67]
4
3.7M
1.0
80.92
CvT-13 [18]*
19.6M
1.1
79.24
Hybrid
CvT-13 [18]*
2
19.6M
4.5
81.81
DHVT-T (Ours)
6.0M
0.4
80.93
DHVT-S (Ours)
4
23.4M
1.5
82.91
DHVT-T (Ours)
5.8M
1.4
83.54
DHVT-S (Ours)
2
22.8M
5.6
85.68
Results CIFAR-100. We mainly compare the performance of proposed model on CIFAR-I00. Patch size set to means taking raw pixel input. For comparison with other methods, we directly cite the results reported in the corresponding paper. The results of our model are the best out of five runs with different random seeds. As is shown in Table 4, CNN models occasionally have more parameters and conduct fewer computations, while ViT models have much fewer parameters and conduct much higher computations. Our model DHVT-T reaches 83.54 with 5.8M parameters. And DHVT-S reaches 85.68 with only 22.8M parameters. With much fewer parameters, our model achieves much higher performance against other ViT-based models and strong CNNs ResNeXt, SENet, SKNet, DenseNet and Res2Net. And compared with other ViT and Hybrid models, we exhibit significant performance improvement under reasonable parameters and computational burdens. We not only bridge the performance gap between CNNs and ViTs but also push the state-of-the-ee result to higher level. Moreover, scaling up and smaller patch size benefit our method. Both DeiT and PVT fail to achieve higher performance when scaling up. And when the patch size gets smaller, the performance of DeiT even drops. These results reasonable because insufficient data is hard to train a large model from scratch. And smaller patch size further intensifies the non-overlapping problem of vanilla VIT and thus decreases the performance. More experiment results such training from scratch on 224x224 resolution can be seen in Supplementary Materials.
4.3 Ablation Studies
All the results in the ablation study are the average over four runs with different random seeds.
The model for the ablation study is DHVT-T, with patch size of 4 and training from scratch on CIFAR-10M with the same data augmentation as in Section 4.2. Here DHVT-T is trained with learning rate of 0.001, warm-up epoch of 10 and batch size of 512, and total epoch of 300. The baseline is D with 4 heads and the patch size set to 4. The results are shown in the following tables.

---

The importance of positional information. We have a baseline performance of 67.59 from DeiT-T with heads, training from scratch with 300 epochs. When removing absolute positional embedding. the performance drops drastically to 58.72, demonstrating the importance of position information in vision transformers. SOPE is able to provide positional information some extent because such absolute positional information can be derived from zero padding. As is shown in Table 5, when adopting SOPE and removing absolute position embedding, the performance does not drop so drastically. But only depending on SOPE to provide position information is not enough.
Table 5: Ablation study SOPE and DAFF
Table 6: Ablation study on head token
Abs. PE
SOPE
DAFF
Acc (%)
SOPE
Abs. PE
Head Token
Acc (%)
X
X
67.59 (+0.00)
DAFF
X
X
X
58.72 (-8.87)
67.59
X
X
X
73.68 (+6.09)
(+0.00)
X
X
69.65 (+2.06)
X
V
69.10
XX
79.47 (+11.88)
(+1.51)
X
X
79.75 (+12.16)
X
80.85
(+13.26)
80.17 (+12.58)
X
80.35 (+12.76)
The role of DAFE. When adopting DAFF, the performance gain increases greatly 79.47, because DAFF solves the problem on both spatial and channel aspects, introducing strong local constraints and re-calibrating channel feature representation. It is sensible embedding can increase performance. The positional information has been encoded into tokens through the depth-wise convolution in DAFF, and the absolute position embedding will break translation invariance. When both SOPE and DAFF are adopted, the positional information will be encoded comprehensively, and SOPE will also help address the non-overlapping problem here, preserving fine-grained low-level features in the early stage.
that removing absolute position
The role of head tokens. From Table 6, we can also the stable performance gain brought by head tokens across different model structures. When introducing head tokens into DeiT-T, the performance gets +1.51 gain, demonstrating its effectiveness. As said before, head tokens guarantee the interaction among different channel groups, better fusing the diverse representation. The resulting integral representation now strong enough for classification. When adopting all three modifications, we get a +1 +13.26 accuracy gain, successfully bridging the performance gap with CNNs.
head token 1
head token
head token
head token
Figure 4: Visualization of the attention map head tokens to patch tokens on low layer

---

4.4 Visualization
We visualize the attention maps of head tokens to patch tokens Fig. 4. Each row represents one image. The results are samples in the second encoder layer. We can see that different head token activates on different patch tokens, exhibiting their diverse representations. On such low layers, low-level fine-grained features are able to be captured in our model. More visualization results are shown in the Supplementary Materials.
Limitation
Though achieve much higher performance than existing methods, such performance gain comes at the expense of computation. The performance when patch size set to 2 boosts higher than using patch size of 4. But the computation expense rises quadratically. In practical usage, suggest choose a good patch size for better trade-off between performance and computation.
Conclusion
In this paper, present alternative vision transformer architecture DHVT, which can train from scratch on small datasets and reach state-of-the-art performance on series of datasets. The weak inductive biases spatial relevance and diverse channel representation brought by insufficient training data are strengthened in our model. The highlighted head token design is able to transfer to variants of VIT model enable better feature representation.
Acknowledgements
This work is supported by the National Nature Science Foundation of China (62121002, 62022076. 62232006, U1936210, 62272436), the Youth Innovation Promotion Association Chinese Academy of Sciences (Y2021 (22),, Anhui Provincial Natural Science Foundation (2208085QF190) the China Postdoctoral Science Foundation 202IM703081.
10

---

References
[1] He, K., X. Zhang, Ren, et al. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognifion, pages 770-778. 2016.
[2] Huang, G., Z. Liu, L. Van Der Maaten, et al. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708 2017.
[3] Liu, C.,, H. Xie, Z. Zha, et al. Bidirectional Autention-recogniinn model for fine Line-grained object classification. IEEE Transactions on Multimedia, 22(7):1785-1995 2019.
[4] Hu, J., L. Shen, G. Sun. Squeeze-and-excitttion networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141. 2018.
[5] Min, S., H. Yao, H. Xie, et al. Domain-orrented semantic embedding for zero-shot learning. IEEE Transactions on Multimedia, 23:3919-3930, 2020.
[6] Fu, J., H. Zheng. Mei. Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4438-4446. 2017.
[7] Min, S., H. Yao, H. Xie, et al. Multi-objective matrix normalization for fine- tne-grained visual recognition. IEEE Transactions on Image Processing, 29:4996-5009, 2020.
[8] Ren, S., K. He, R. Girshick, et al. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[9] Lin, T.-Y., Goyal, R. Girshick, et al. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988. 2017.
[10] Tian, Z., C. Shen, H. Chen, al. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9627-9636. 2019.
[11] He, K., Gkioxari, Dollár, et Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-299000 2017.
[12] Chen, L.-C., G. Papandreou, I. Kokkinos, al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
[13] Ronneberger, O.,, P. Fischer, T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234 241. Springer, 2015.
[14] Dosovitskiy, A., L. Beyer, A. Kolesnikov, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[15] Yuan, K., S. Guo, 7. Liu, et al. Incorporating convolution designs into visual transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 579-588, 2021.
[16] Wang, W., Xie, X. Li, et al. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 568-578. 2021.
[17] Liu, Z., Y. Lin, Y. Cao, et al. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 10012-10022. 2021.
[18] Wu, H., B. Xiao, N. Codella, et al. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEFF/CVE International Conference on Computer Vision, pages 22-31. 2021.

---

[19] He, J...N Chen, S. Liu, et al. Transfg: A transformer architecture for fine imeeggriined recogmition. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pages 852-860. 2022.
[20] Carion, N., F. Massa, G. Synnaeve, et End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[21] Dai, Z., B. Cai, Y. Lin, et al. Up-detr: Unsupervised pre-training for object detection with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1601-1610. 2021.
[22] Zhu, X.,, W. Su, L. Lu, al. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
[23] Strudel, R., R. Garcia, I. Laptev, et al. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Viston, pages 7262-7272. 2021.
[24] Guo, R., D. Niu, L. Qu, et al. Sotr: Segmenting objects with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7157-7166. 2021.
[25] Vaswani, A., N. Shazeer, N. Parmar, et al. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[26] Russakovsky, O., J. Deng, H. Su, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115())211-252, 2015.
[27] Raghu, M., T. Unterthiner, Kornblith, et al. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34, 2021.
[28] Park, N., S. Kim. How do vision transformers work? arXiv preprint arXiv:2202.06709, 2022.
[29] d' d'Ascoli, S., H. Touvron, M. L. Leavitt, et al. Convit: Improving vision transformers with soft convolutional inductive biases. International Conference on Machine Learning, pages 2286 2296. PMLR, 2021.
[30] Li, Y., K. Zhang, J. Cao, et al. Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2104.055 055077 2021.
[31] Peng, Z., W. Huang, S. Gu, et al. Conformer: Local features coupling global representations for visual recognition. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 367-376. 2021.
[32] Chen, 2., L. Xie, J. Niu, et al. Visformer: The vision-friendly transformer. 122/2310411533, 2021.
CoRR,
[33] Zhao, Y., G. Wang, C. Tangg al. A battle of network structures: An empirical study of enn, transformer, and mlp. arXiv preprint arXiv:2108. 13002, 2021.
[34] Heo, B., S. Yun, D. Han, et al. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 11936- 11945. 2021.
[35] Zhang, Z., H. Zhang, L. Zhao, et al. Nested hierarchical transformer: Towards accurate, data- efficient and interpretable visual understanding. In AAAI Conference on Artificial Intelligence (AAAI), 2022. 2022.
[36] Howard, A. G., M. Zhu, B. Chen, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1704 2017..
[37] Peng, X., Q. Bai, X. Xia, al. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVE international conference on computer vision, pages 1406-1415. 2019.
[38] Krizhevsky, A., I. Sutskever, G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
12

---

[39] Szegedy, C., W. Liu, Y. Jia, et al. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9. 2015.
[40] Xie, S., R. Girshick, P. Dollár, et al. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1492-1500. 2017.
[41] Li, X., W. Wang, X. Hu, et al. Selective kernel networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 510-519. 2019.
[42] Touvron, H., M. Cord, M. Douze, et al. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[43] Jiang, Y., S. Chang, Z. Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances in Neural Information Processing Systems, 34, 2021.
[44] Arnab, A., M. Dehghani, G. Heigold, et al. Vivit: video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6836-6846. 2021.
[45] Chen, X., S. Xie, K. He. An empirical study of training self-supervised vision transformers, In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9640-9649. 2021.
[46] Wang, X., R. Girshick, A. Gupta, et al. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pages 7794-7803. 2018.
[47] Hu, H., Z. Zhang, Z. Xie, et al. Local relation networks for image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3464-3473 3373. 2019.
[48] Chen, M., A. Radford, R. Child, et al. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[49] Devlin, J., M.-W. Chang, K. Lee, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint /
2018.
[50] Jiang, Z.-H., Q Hou, L. Yuan, et al. All tokens matter: Token labeling for training better vision transformers. Advances in Neural Information Processing Systems, 34, 2021.
[51] Xiao, T., M. Singh, E. Mintun, et al. Early convolutions help transformers see better. Advances in Neural Information Processing Systems, 34:30392-30400 2021.
[52] Yuan, L., Y. Chen, Wang, et Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 558-567. 2021.
[53] Chu, X., Z. Tian, B. Zhang, et al. Conditional positional encodings for vision transformers. arXiv preprint arXiv: 10882, 2021.
[54] Wang, W., E. Xie, X. Li, et al. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, pages 1-10, 2022.
[55] Ren, S., D. Zhou, S. He, et al. Shunted self-attention via multi-scale token aggregation, 2021.
[56] Guo, J., K. Han, H. Wu, et al. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 22175-12185. 2022.
[57] Mao, X., G. Qi, Y. Chen, et al. Towards robust vision transformer. arXiv:2105.0792 2021.
arXiv preprint
[58] Chen, Q. Q. Wu, Wang, al. Mixformer: Mixing features across windows and dimensions. arXiv preprint arXiv:2204.025577 2022.
13

---

[59] Xu, Y., Q. Zhang. J. Zhang, et al. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances Neural Information Processing Systems, 34, 2021.
[60] Ali, A., H. Touvron, M. Caron, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34, 2021.
[61] Ding, M., B. Xiao, N. Codella, et al. Davit: Dual attention vision transformer. arXiv preprini arXiv:2204.03645 2022.
[62] Xu, W., Y. Xu, T. Chang, et al. Co-scale conv-attentional image transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9981-9990. 2021.
[63] Ryoo, M., A. Piergiovanni, A. Arna,, et al. Tokenlearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems, 34, 2021.
[64] Fang, J., L. Xie, X. Wang, et al. Msg-transformer: Exchanging local spatial information by manipulating messenger tokens. In CVPR. 2022.
[65] Liang, Y., C. Ge, Z. Tong, et al. Not all patches are what you need: Expediting vision trans- formers via token reorganizations, In International Conference on Learning Representations. 2022.
[66] Liu, Y., E. Sangineto, W. Bi, et al. Efficient training of visual transformers with small datasets. Advances Neurul Information Processing Systems, 34, 2021.
[67] Hassani, A., S. Walton, N. Shah, et al. Escaping the big data paradigm with compact transform- ers. arXiv preprint arXiv:104.005004, 2021.
[68] Lee, S. H., S. Lee, B. C. Song. Vision transformer for small-size datasets, 2021.
[69] Hendrycks, D., K. Gimpel. Gaussian error linear units (gelus), 2016.
[70] Ba, L., J. R. Kiros, G. E. Hinton. Layer normalization. arXiv preprint arXiv: 2016.
[71] Krizhevsky, A., G. Hinton. Learning multiple layers of features from tiny images. Master's s thesis, Department of Computer Science, University Toronto, 2009.
[72] Loshchilov, I. F. Hutter. arXiv1171.055001, 2017.
Decoupled weight decay regularization. arXiv preprint
[73] Zagoruyko, S., N. Komodakis. Wide residual networks. arXiv preprint wrXiv:605.00//1 2016.
[74] Zhang, H., M. Cisse, N. Dauphin, et al. mixup: Beyond empirical risk minimization. In International Conference Learning Representations. 2018.
[75] Yun, S., D. Han, Oh, et Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEEE/CVE international conference computer vision, pages 6023-6032. 2019.
[76] Cubuk, H. D., B. Zoph, D. Mane, et al. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 113-123. 2019.
[77] Gao, S.-H., M.-M. Cheng, K. Zhao, et al. Res2net: A new multi-scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence, 43(2):652 662, 2019.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] We do not think there are direct negative societal impacts from our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material or URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] We use standard datasets and there are no personally identifiable information or offensive content the best of the community knowledge.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "claapWcVb-e", "reviewer": "Reviewer_u2nu", "summary": "The paper proposes a new Vision Transformer architecture that merges CNNs and ViT to have the best of the two worlds. The paper focuses on small datasets and Vision Transformers, which have to be somehow regularised or helped to learn the inductive biases that are naturally enforced with CNNs.\n\nThe paper proposes a new architecture that modifies the FFN with a Dynamic Aggregation Feed Forward, and proposes a \"Head Token\" so that each head of the MHA can attend to all others somehow. This supposedly helps learning better features.\n\nThe authors experimented the proposals with various small datasets and observed improvements in all tests.", "questions": "- The Transformer is a good architecture also because it does not depend too much on the batch size. However, your architecture now has Batch Normalization. Why? Couldn't you use Layer Norm? If not, why?\n\n- Why didn't you compare with [54]?\n\n- What is the impact of the batch size on your results?\n\n- what if I use this architecture with a big dataset? Is this an architecture that is general or only for small datasets? what is your opinion here\n\n- What about downstream tasks? Does your architecture help there?\n\n- answer the other questions raised in the previous section plz!", "limitations": "No limitations", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "PROS:\n- the problem is very important: Vision Transformers focus a lot on big datasets but small datasets are nowadays almost forgotten\n- the paper is clear\n\nCONS:\n- lack of comparison with some state of the art\n- lack of clarity on the motivations for some choices\n\nDETAILS:\nTo me some parts of the paper are unclear. E.g.:\n- the choice of batch norm\n- why you did not compare with CvT, very famous architecture joining CNNs with Transformers\n- why you did not compare with [54], published one year ago in the same conf\n- why is the Head Token beneficial? I mean, the multi-head attention was created to have somehow the same approach CNNs have with multiple filters: learning different aspects of the same input. I am surprised your modification helps\n\nI did not also like the lack of clarity of the experiments. Why is the resolution of CIFAR-10 changed? What happens if you do not do the changes explained in L244? Are all baselines trained with the original augmentations, params, resolutions?\n\nThen, [54] reports 75.22 on ResNet-50 on ClipAart, 66.58 on Paintings etc. Your results are lower both for the baseline and your method. Why?\n\nL288 why 4 heads? it is not standard at all.\n\nL265 DomainNet is much bigger. Why did you select only a few datasets from DomainNet? Unfortunatedly, this raises the question on me whether you cherrypicked the results. Moreover, depending on the Table (2, 3, 4) the baselines change. The reason is a bit unclear.\n\n\nOTHER:\n- L144: needs a citation, e.g. 54. However, 54 shows that the networks are competitive with ResNet-50\n- L55-79: it is not clear why it is important to solve everything in the spatial dimension.\n- L79 SE is not defined\n- L150: non-hierarchical here is not clear. Try to explain it better.\n- L180: this is a result I suppose, on the methods. If you want to state this, show the results.\n- L181: more solution is not english\n- ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bY4zzDmf_WJ", "reviewer": "Reviewer_SHR5", "summary": "This paper proposes a new Vision Transformer structure named DHVT targeting to improve the performance of ViTs on the image classification task with small datasets. Specifically, DHVT is designed to improve the spatial relevance among the patches with convolutional inductive bias and enhance the channel representation by re-calibrating the channel representations. To validate the effectiveness of the design choices, the authors conducted experiments on small datasets represented by CIFAR-100 and ImageNet-1k. The CIFAR-100 results look good, and the proposed method outperforms the other baseline methods by large margins.", "questions": "\nPlease address the issues pointed out in the weakness.", "limitations": "In general, yes.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- It’s meaningful to study why and how ViTs are inferior to CNNs when training on small datasets from scratch.\n- The quantitative results in Table 2 look good and surpass the other methods. \n\nWeakness:\n- It’s hard to understand the motivation for the proposed detailed design choices and why they are better than other simple designs. For instance, why re-calibrating only the class token with SE operation [25], considering localViT [18] proposed re-scaling channels for all tokens in FFN blocks? A proper discussion with the SE module in localViT is needed. Also, the head tokens are proposed to enhance the channel groups for each head by modeling head correlations, which can easily be implemented by an MLP layer. Can you explain the benefit of modeling the head correlations with MSA instead of a simple MLP layer?\n\n- The experiments are insufficient to support the authors’ claim. Tables 3 and 4 have very few baseline methods and it’s hard to see the gain of DHVT over the SOTA methods.\n\n- The novelty for modeling spatial relevance is limited, since introducing early convolutions has been shown effective in previous works, e.g. [a]. \n\n[a] Xiao, Tete, et al. \"Early convolutions help transformers see better.\" Advances in Neural Information Processing Systems 34 (2021): 30392-30400.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "99wi6Lnfb6", "reviewer": "Reviewer_b3Vm", "summary": "The paper proposes Dynamic Hybrid Vision Transformer (DHVT) for visual recognition. They argue that spatial relevance and diverse channel representation are two important inductive biases for visual recognition and they address the issue from two perspectives.\n1. On the spatial aspect, they adopt a hybrid structure, they integrate convolution into patch embedding and multi-layer perceptron (MLP) module, forcing the model to capture the token features and their neighbouring features. \n2. On the channel aspect, they introduce a dynamic feature aggregation module in MLP and head tokens in multi-head self-attention module to help re-calibrate channel representation and make different channel group representation interacts with each other.\nThe experiments are conducted on CIFAR-100, ClipArt, Sketch, Painting, and ImageNet. All of these datasets are for classification.", "questions": "See the Strengths And Weaknesses part.", "limitations": "See the Strengths And Weaknesses part.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- Well-organized and easy to read.\n- Motivation is clear from both the spatial and channel sides.\n- Good results on many small- and medium-size datasets.\n- Nice ablation study is shown in Tab. 5.\n- LImitation is stated.\n\nCons:\n- Some unclear captions should be more concrete. E.g., DAFF in Figure. 1.\n- Limited novelties. The depth-wise convolution is well investigated in previous works, like CVT, CPVT, PVT v2, etc. And the use of both channel-wise and spatial-wise representation is explored in DaVIT, CoaT, etc.\n- What is the intuitive explanation of the HI-MHSA? The component is a bit complicated.\n- Why is the work suitable for small datasets? Why does the author put the experiment on ImageNet in Appendix? Which part is specifically designed for small datasets?\n- No downstream tasks were evaluated. It would be better if the authors could provide some results on ADE20K or COCO.\n- The use of batchnorm, how does it influence the performance? VIT-based architectures typically use layernorm so that the batch size will not hinder the performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7c0-so-dySO", "reviewer": "Reviewer_4fHM", "summary": "This paper focuses on generalizing vision transformers to the domain of small datasets through injecting spatial-wise and channel-wise inductive biases into model architecture design. Specifically, the authors point out that it is hard for the vanilla vision transformers to model spatial relevance and diverse channel representation under data-constrained scenarios. The weaknesses for vision transformers compared to their CNN counterparts finally lead to worse performances on small datasets, *e.g.,* cifar. To address the drawbacks, the authors introduce two augmented modules SOPE and DAFF, corresponding to the patch embedding and feedforward operators. By using additional convolution operations to gather the local feature, spatial relevance within patches is emphasized, and is forced to learn better relations even with limited data. On the other hand, *head tokens* are adopted on top of the widely used classification token and serve as the global token for each group of channels (*i.e.,* a head) in the multi-head self-attention module. In this way, the interaction across different heads is facilitated and shows better representation capability under a low data regime. Regarding the experiments, the authors validate the effectiveness of their module on various datasets, including small datasets like cifar100 and DomainNet, and larger datasets like ImageNet-1K. A good trade-off between model parameters and test accuracy is shown compared to several state-of-the-art baselines, and ablation studies are also conducted to show the contribution of each part.", "questions": "Most of my concerns are addressed in the **Strengths And Weaknesses** section. The followings are my detailed questions regarding the weaknesses.\n\n+ I understand that the authors have stated several times in the paper like in the abstract '... the lack of data hinders ViTs to attend the spatial relevance ...' or '... the scarce data can not enable ViTs to learn strong enough representation ...', nevertheless, from my perspective, these statements are more like intuitive claims that short of technique supports. For example, it would be better if some qualitative results can show that spatial relevance is more critical for small datasets. The previous work [1] actually shows that lower attention layers do not learn to attend locally with less training data. Nevertheless, the *less training data* in [1] refers to ImageNet-1k without ImageNet-22k pretraining, which still belongs to large-scale data in the scope of this paper. I wonder if this claim is also correct with small datasets like cifar.\n\n+ Do the setups in table 2 correct? I have checked the wide-resnet paper and found that WRN28-10 is trained for 200 epochs instead of 300 epochs. Also, as I have mentioned before, reporting the best results of 5 runs may enjoy the gain from randomness, especially on small datasets.\n\n+ A comparison on flops-acc trade-off for cifar100 would be better. According to supplementary material, the DHVT-S model that shows the highest performance (85.68) has 6.3G flops, which requires high computation for small image size. The trade-off for computation cost would be more valuable than training parameters.\n\n+ A Typo in line 181: require->required.\n\n[1] Raghu, Maithra, et al. \"Do vision transformers see like convolutional neural networks?.\" Advances in Neural Information Processing Systems 34 (2021): 12116-12128.", "limitations": "The authors have addressed the limitations. I feel that inference speed is also a question that is worth further investigation. Although the authors provide the model throughput in the supplementary, the detailed comparison with baseline models and efficient cnns is still lacking.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Strengths\n  + This paper is organized in an easy-to-follow pattern. Related works on data-efficient vision transformers are thoroughly investigated and properly introduced. Each component of the proposed model is described in detail respectively. The experiments are substantial and some results and experiment details in the supplementary material also help to understand the paper.\n  + The practical value of this paper is promising. Although vision transformer models show better performance and scalability on large-scale datasets, the potential application on data constraint scenarios is still under-explored. The gap between vision transformer and cnns on the small dataset is a topic that is worth deep investigation. This paper attempts to address this limitation from the spatial and channel perspective, and the experiments show improvements on widely used benchmarks, *e.g.,* cifar100. I think this shows a new direction to practically using vision transformer models on datasets with low-scale / low-resolution data.\n  + The experiments are sufficient in this paper. The parameter-accuracy trade-off on three datasets, ablative studies on each proposed component, visualization on attention maps, and the flops-accuracy trade-off in the supplementary material are detailedly described.\n  + Finally, I really approve that the authors are willing to share the code of each module. This really helps to understand the implementation detail.\n\n+ Weaknesses\n  + The motivation of this paper can be addressed more clearly. The authors point out that spatial relevance and diverse channel representation are two essential aspects that influence the performance of vision transformers under a low data regime. Nevertheless, I am still concerned about the practical connection between **lack of two inductive biases** with **low data regime**. To my knowledge, the lack of inductive biases is also a question for large-scale datasets like ImageNet-1k. Previous research [1] have already explored strengthening the spatial relationship between patches, and the diversity of features among different heads has also been investigated [2]. As a result, the fact that the lack of two inductive biases leads to inferior performances is not so surprising. As the authors are motivated from these two perspectives to address the limitation of the vision transformer, I think it would be better if they can **emphasize the unique influence of the inductive biases for small datasets**, and explain how the situation differs from large-scale datasets.\n\n  + Technical contribution. It is an interesting idea to improve the performance of the vision transformer from spatial and channel perspectives. However, it must be pointed out that the proposed modules share some resemblance with several previous works. The convolutional patch embedding is widely adopted in several works [1][3]. Using depthwise convolution in the feedforward network is also a common trick [4]. The dynamic aggregation feedforward module also seems like a trivial implementation of SENet, and similar insight has been explored in [5]. It would be better if the authors can focus on addressing the differences with previous works.\n\n  + I also have concerns about some experimental settings. For the main results on cifar100 shown in table 2, I wonder if the comparisons are fair. According to the authors, DeiT augmentations are adopted for proposed models. Do previous cnn models use the same augmentation? Also, the timm implementation can practically improve the model performances of cnn models. Have the authors reimplement the baselines on timm and see the differences with the original paper? I understand it is impractical to ask the authors to re-implement all the baseline approaches, I think a simple comparison with a widely used model like ResNeXt would also do the trick and makes the result more convincing. In section 4.2, the authors state that the results are the best out of 5 runs. This is clearly different from the routine in previous works where an average of 10 runs are usually used for cifar datasets. Considering the large variance for cifar100, I think some part of the gain may come from the randomness.\n\n\n\n[1] Xiao, Tete, et al. \"Early convolutions help transformers see better.\" Advances in Neural Information Processing Systems 34 (2021): 30392-30400.\n\n[2] Raghu, Maithra, et al. \"Do vision transformers see like convolutional neural networks?.\" Advances in Neural Information Processing Systems 34 (2021): 12116-12128.\n\n[3] Yuan, Li, et al. \"Volo: Vision outlooker for visual recognition.\" arXiv preprint arXiv:2106.13112 (2021).\n\n[4] Guo, Jianyuan, et al. \"Cmt: Convolutional neural networks meet vision transformers.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[5] Yuan, Li, et al. \"Tokens-to-token vit: Training vision transformers from scratch on imagenet.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept