Transformer in Transformer
Authors: Kai Han
An Xiao
Enhua Wu
Jianyuan Guo
Chunjing Xu
Yunhe Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transformer
computer vision
ImageNet

Abstract: Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (\eg, 16$\times$16) as “visual sentences” and present to further divide them into smaller patches (\eg, 4$\times$4) as “visual words”. The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, \eg, we achieve an 81.5\% top-1 accuracy on the ImageNet, which is about 1.7\% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at \url{https://github.com/huawei-noah/CV-Backbones}, and the MindSpore code is available at \url{https://gitee.com/mindspore/models/tree/master/research/cv/TNT}.

Transformer in Transformer
Kai Han
An Xiao
Enhua Wu Jianyuan Guo Chunjing Xu² Yunhe Wang?*
State Key Lab of Computer Science, ISCAS & UCAS Huawei Noah's Ark Lab University of Macau
(hankai, Mhankai,wehh@ieee wehh cn, yunhe. wangohuavei.com
Abstract
Transformer is new kind of neural architecture which encodes the input data powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, point out that the attention inside these local patches also essential for building visual transformers with high performance and we explore new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16x 16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4x4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be ag- several benchmarks
gregated to enhance the representation ability. Experiments demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% 7% higher than that of the state-of-the-a visual transformer with similar computational cost. The PyTorch code is available at
com http:///waaeeeeeaao/////aaaaaaaa the MindSpore code available at httpp::////ttt //tt:e///eeee cwt.///iaoodoooee/oooeeee... brre/mmateer/ressarc///////
and
Introduction
In the past decade. the mainstream deep neural architectures used in the computer vision (CV) are mainly established on convolutional neural networks (CNNs) [17] 12 [T]. Differently, transformer a type neural network mainly based on self-attention mechanism [35], which can provide the relationships between different features. Transformer is widely used in the field of natural language processing (NLP), e.g., the famous BERT 8 and GPT-3 2 models. The power of these transformer
models inspires the whole community investigate the use of transformer for visual tasks.
To utilize the transformer architectures for conducting visual tasks, number researchers have explored for representing the sequence information from different data. For example, Wang el al. ex- plore self-attention mechanism in non-local networks [37] for capturing long-range dependencies in video and image recognition. Carion al. present DETR [3). which treats object detection as a direct set prediction problem and solve it using a transformer encoder-decoder architecture. Chen et al. propose the iGPT [5). which is the pioneering work applying pure transformer model (i.e., without convolution) on image recognition by self-supervised pre-training.
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

I. x
Outer Transformer Block
Sentence position encoding
Output
Inner Transformer Block
Inner Transformer Book
Inner Transformer Block
Sentence embediding
Linear Projection of Visual Sentences Words
Word pasition encoding
Word embedding
Figure 1: Illustration of the proposed Transformer-iN-Trasfiie (TNT) framework. The inner transformer block is shared in the same layer. The word position encodings are shared across visual sentences.
Different from the data in NLP tasks, there exists a semantic gap between input images and the ground-truth labels in CV tasks. To this end, Dosovitskiy al. develop the VIT [9, which paves the way for transferring the success of transformer based NLP models. Concretely, ViT divides the given image into several local patches between any two image patches for generating effective feature representations for the recognition task. Subsequently, Touvron al. explore the data-efficient training and distillation to enhance the performance of ViT on the ImageNet benchmark and obtain an about 81 81.8% ImageNet top-1 accuracy, which is comparable that of the sfate-of-the-art convolutional networks. Chen el al. further treat the image processing tasks (e.g., denosing and super-resolution) as series of translations and develop the IPT model for handling multiple low-level computer vision problems 4 Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image 40 29]. object detection 46), and segmentation 38].
visual sequence. Then, the attention be naturally calculated
recognition
Although the aforementioned visual transformers have made great efforts to boost the models' performances, most of existing works follow the conventional representation scheme used in ViT, i.e. dividing the input images into patches. Such a exquisite paradigm can effectively capture the visual sequential information and estimate the attention between different image patches. However, the diversity of natural ímages in modern benchmarks is very high, egg., there are over 120 M images with 1000 different categories in the ImageNet dataset (26). As shown Figure [ representing the given image into local patches can help us to find the relationship and similarity between them. However, there are also some sub-patches inside them with high similarity. Therefore, we are motivated to explore more exquisite visual image dividing method for generating visual sequences and improve the performance.
In this paper, we propose novel Transformer-iN. --rassffrree (TNT) architecture for visual recog- nition shown in Figure [ To enhance the feature representation ability of visual transformers, we first divide the input images into several patches as "visual sentences" and then further divide "visual words". Besides the conventional transformer blocks for extracting
them into sub-patches features and attentions of visual sentences, we further embed a sub-transformer into the architecture for excavating the features and details of smaller visual words. Specifically, features and attentions between visual words in each visual sentence are calculated independently using a shared network so that the increased amount of parameters and FLOPs (floating-poin) operations) negligible. Then, features of words will be aggregated into the corresponding visual sentence. The class token is also used for the subsequent visual recognition task via fully-connected head. Through the proposed TNT model, we can extract visual information with fine granularity and provide features with more details. We then conduct a series of experiments on the ImageNet benchmark and downstream tasks to demonstrate its superiority and thoroughly analyze the impact of the size for dividing visual words. The results show that our TNT can achieve better accuracy and FLOPss trade-off over the fate-of-the-as transformer networks.

---

2 Approach
In this section, we describe the proposed transformer-iin-taasffrrnee architecture and analyze the computation and parameter complexity in details.
2.1 Preliminaries
We first briefty describe the basic components in transformer [35 including MSA (Multi-head Self-Attention), MLP (Multi-Layer Perceptron) and LN (Layer Normalization).
MSA. In the self-attention module, the inputs X E Raxd are linearly transformed to three parts, i.e., queries Q E R.xxd keys K E < and values V Raxd, where is the sequence length, dx, dy are the dimensions of inputs, queries (keys) and values, respectively, The scaled dot-product attention applied on Q, K, V:
QKT VTa
Attention(Q. K, V) softmax(
VV.
Finally, linear layer used to produce the output. Multi-head self-attention splits the queries, keys and values to h parts and perform the attention function in parallel, and then the output values of each head are concatenated and linearly projected to form the final output.
MLP. The MLP is applied between self-attention layers for feature transformation and non-linearity
MLP(XX) FC(((FC( FC(X) XW + b,
(2)
where W and b the weight and bias term of fully-connected layer respectively, and o-) is the activation function such as GELU 13
LN. Layer normalization 1 is key part in transformer for stable training and faster convergence.
LN is applied over each sample x E IRa as follows:
LN(x) = 2 - =
8
-
where E R, 5 E IR are the mean and standard deviation of the feature respectively, o is the element-wise dot, and E Rd, 3 e Idd are learnable affine transform parameters.
2.2 Transformer in Transformer
Given 2D image, we uniformly split it into n patches X' [x, [."] E B=xpxpx3, where pp is the resolution of each image patch. ViT 99] just utilizes standard transformer process the sequence of patches which corrupts the local structure of a patch, as shown in Fig. [a). Instead, we propose
Cnnimmerr---Taaasoonnn (TNT) architecture to learn both global and local information in an image. In TNT, we view the patches as visual sentences that represent the image. Each patch is further divided into m sub-patches, iee. visual sentence is composed of sequence of visual words:
X ا
f.2
(4)
where rnd E 3 is the j-th visual word of the i-th visual sentence, (s, s) is the spatial size of sub-patches, and j 1, -... ... m. With a linear projection, we transform the visual words into sequence of word embeddings:
Y
[y
" ليd = FC(Vec(x>)))
(5)
where nار E R the j-th word embedding, the dimension of word embedding, and Vec(.) the vectorization operation.
In TNT, we have two data flows in which one flow operates across the visual sentences and the other processes the visual words inside each sentence. For the word embeddings, we utilize a transformer block explore the relation between visual words:
Y = YL_ + MSA(LN(Y
(6)
Y = -Y=Y;; MLP(LN(Y
(7)
3

---

where = 1, is the index of the 1-th block, and L is the total number of stacked blocks. The input of the first block Y is just Y in Eq. 5 All word embeddings in the image after transformation are 3 [ Y?: y This can be viewed as an inner transformer block, denoted as Tin- This process builds the relationships among visual words by computing interactions between any two visual words. For example, in patch of human face, a word corresponding to the eye is more related to other words of eyes while interacts less with forehead part.
For the sentence level, we create the sentence embedding memories to store the sequence of sentence- level representations: Zn- = Zccasss Z Z5. ... Zö E [R(++))dd where Zelass is the class token similar to ViT [9], and all of them initialized as zero. In each layer, the sequence of word embeddings are transformed into the domain of sentence embedding by linear projection and added into the sentence embedding:
- + FC(V)c(Y()).
(8)
where Zi_, E Rd and the fully-connected layer FC makes the dimension match for addition. With the above addition operation, the representation of sentence embedding is augmented by the word-level features. We use the standard transformer block for transforming the sentence embeddings:
3', 2 (IS((LN(2.-)))
(9)
2 = Z', MLP(LM(P i)).
(10)
This outer transformer block Tout is used for modeling relationships among sentence embeddings. In summary, the inputs and outputs of the TNT block include the visual word embeddings and sentence embeddings as shown in Fig. bb), so the TNT can be formulated as
V4. Zi = INT(M--1, Z(-))
(1I)
In our TNT block, the inner transformer block is used model the relationship between visual words for local feature extraction, and the outer transformer block captures the intrinsic information from the sequence of sentences. By stacking the TNT blocks for L times, we build the transformer- in-transformer network. Finally, the classification token serves as the image representation and fully-connected layer is applied for classification.
Position encoding. Spatial information is an important factor in image recognition. For sentence embeddings and word embeddings, we both add the corresponding position encodings retain spatial information as shown in Fig. []] The standard learnable ID position encodings are utilized here. Specifically, each sentence is assigned with a position encodings:
Zo Zo + seentence
(12)
where Esentence E R (n+1)xx are the sentence position encodings. As for the visual words a sentence, word position encoding is added to each word embedding:
Yi
Yd Emord, i 1,2.
(13)
where Eword Rmxe the word position encodings which are shared across sentences. In this way, sentence position encoding can maintain the global spatial information, while word position encoding is used for preserving the local relative position.
2.3 Complexity Analysis
standard transformer block includes two parts, i.e., the multi-head self-attention and multi-layer perceptron. The FLOPs of MSA 2nd(d, d,) n n22 + do), and the FLOPs of MLP 2nd,rd, where r is the dimension expansion ratio hidden layer in MLP. Overall, the FLOPs of a standard transformer block are
FLOPsT = 2nd(d; + do)) + n² (d2 + do) + 2nddr.
(14)
Since r is usually set as 4, and the dimensions of input, key (query) and value are usually set as the same, the FLOPs calculation be simplified
FLOPst 2nd(66 n)).
(15)

---

The number of parameters can be obtained
Paramsy 12dd.
(16)
Our TNT block consists of three parts: inner transformer block Tin, outer transformer block Tout and a linear layer. The computation complexity Ti and Tout are 2nme(6e + m) and 2nd(6d + n) respectively. The linear layer has FLOPs of nmed. In total, the FLOPs of TNT block are
FLOPsTNT = 2nmc(6c + m) + nmed + 2nd(6d + n²).
(17)
Similarly, the parameter complexity of TNT block is calculated as
ParamsT NT = 12cc - mod + 12dd.
(18)
Although we add two more components in our TNT block, the increase of FLOPs small since < d and (m) = (n) in practice. For example, in the DeiT-S configuration, we have d 384 and n 196. We set c = 24 and m = in our structure of TNT-S correspondingly. From Eq. [5] and Eq. 17 we can obtain that FLOPsT 376M and FLOPsTNT
429M The FLOPs ratio of TNT block over standard transformer block about 1.14x. Similarly, the parameters ratio is about 1.08x. With small increase of computation and memory cost, our TNT block can efficiently model the local structure information and achieve a much better trade-off between accuracy and complexity demonstrated in the experiments.
2.4 Network Architecture
We build our TNT architectures by following the basic configuration of ViT 9 and DeiT (31] The patch size is set as 16x 16. The number of sub-patches is set as m 4 4 16 by default. Other size values
evaluated in the ablation studies. As shown in Table T there are three variants of TNT networks with different model sizes, namely, TNT-Ti, TNT-S and TNT-B. They consist of I.M. 23, 33.8M and 65.6M parameters respectively. The corresponding FLOPs for processing 224x224 image are 1.4B, 5.2B and IB respectively.
Table 1: Variants of our TNT architecture. "Ti' means tiny, 'S' means small, and 'B' means base.
The FLOPs are calculated for images at resolution 224x224,
Inner transformer
Outer transformer
Params (M)
FLOPs
Model
Depth
dim
#heads
MLP
dim d #heads
MLP r
(B)
TNT-Ti
12
4
6.1
TNT-S
12
24
4
4
384
4
23.8
5.2
TNT-B
12
40
4
4
640
10
4
65.6
14.1
Experiments
In this section, we conduct extensive experiments on visual benchmarks to evaluate the effectiveness of the proposed TNT architecture.
Table 2: Details of used visual datasets.
Dataset
Type
Train size
Val size
# #Classes
ImageNct
26
Pretrain
1,281,167
50,000
1,000
Oxford 102 Flowers 22
2,040
6,149
102
Oxford-IIIT Pcts 23
3,680
3,669
37
iNaturalist 2019
34
Classification
265,240
3,003
1,010
CIFAR-10 [6
50,000
10,000
CIFAR-100
16
50,000
10,000
COC02017
19
Detection
118,287
5,000
ADE20K
+5
Segmentation 20,210
2,000
150
5

---

3.1 Datasets and Experimental Settings
Datasets. ImageNet ILSVRC 2012 26 an image classification benchmark consisting of 1.2M training images belonging to 1000 classes, and 50K validation images with 50 images per class. We adopt the same data augmentation strategy as that in DeiT [T] including random crop, random clip. Rand-Augment 77. Random Frasing [44 Mixup 42 and CutMix [41]. For the license of ImageNet dataset, please refer to http:://wwwt image- net org/download
In addition to ImageNet, we also test on the downstream tasks with transfer learning to evaluate the generalization ability of TNT. The details of used visual datasets are listed in Table The data augmentation strategy of image classification datasets are the same as that of ImageNet. For COCO and ADE20K, the data augmentation strategy follows that in PVT 36]. For the licenses of these datasets, please refer to the original papers.
Implementation Details. We utilize the training strategy provided in DeiT 31 The main ad- vanced technologies apart from common settings [12] include AdamW [20]. label smoothing (7] DropPath [18]. and repeated augmentation [1]. We list the
in Table 3 for better understanding. All the models are implemented with Py Torch (24] and MindSpore [I5] and trained on NVIDIA Tesla V100 GPUs. The potential negative societal impacts may include energy consumption and carbon dioxide emissions of GPU computation.
Table 3: Default training
used in our method, unless stated otherwise.
Batch
Learning rate
LR dccay
Weight decay
Warmup epochs
Label smooth
Drop path
Repeated Aug
Epochs
Optimizer
size
300
AdamW
1024
le-3
cosine
0.05
5
0.1
0.1
V
TNT on ImageNet
We train our TNT models with the same training settings as that of DeiT [31] The recent transformer- based models like ViT g and DeiT [31] are compared. To have a better understanding of current progress of visual transformers, we also include the representative CNN-based models such as ResNet [2), RegNet 25 and EfficientNet [28]. The results are shown in Table We can see that transformer-basee model, i.e., TNT outperforms all other visual transformer models. In particular, TNT-S achieves 81.5% top-1 accuracy which is 1.7% higher than the baseline model DeiT- S, indicating the benefit of the introduced TNT framework to preserve local structure information inside the patch. Compared to CNNs, TNT can outperform the widely-used ResNet and RegNet. models still inferior to EfficientNet which utilizes special
Note that all the transformer-basee depth-wise convolutions, so it is yet a challenge of how to beat EfficientNet using pure transformer.
Table 4: Results of TNT and other networks on ImageNet.
Model
Resolution
Params (M)
FLOPs (B)
Top-1
Top-5
CNN-based
ResNet-50 12
224
25.6
4.1
76.2
92.9
ResNet-152 12
224x224
60.2
11.5
78.3
94.1
RegNetY-8GF 25
224x224
39.2
8.0
79.9
RegNetY-16GF 25
224x224
83.6
15.9
80.4
EfficientNet-BB 28
300x300
12.0
1.8
81.6
94.9
EfficientNer- B4
380
19.0
4.2
82,9
96.4
Transformer-based
DeiT-Ti 31
224x224
5.7
1.3
72.2
TNT-Ti
224x224
6.1
1.4
73.9
91.9
DeiT-S 3]
224x224
22.1
4.6
79.8
PVT-Small
36
224x224
24.5
3.8
79.8
T2T-ViT 1-14
224x224
21.5
80.7
TNT-S
224x224
23.8
5.2
$1.5
95.7
VIT-B/16 9
384x384
86.4
55.5
77.9
DeiT-B 31
224x224
86.4
17.6
81.8
T2T-ViT
40
224x224
63.9
13.2
82.2
TNT-B
224x224
65.6
14.1
82.9
96.3

---

82
82-
Accurary (Ng
80
Accurary (80)
80
Redive
75
75
(M)
FLOPS
(a) Acc v.s. Params
(b) Ace v.s. FLOPs
Figure 2: Performance comparison the representative visual backbone networks on ImageNet.
also plot the accuracy-parameters and accuracy-FLOO line charts in Fig. to have an intuitive comparison of these models. Our TNT models consistently outperform other transformer-based models by significant margin.
Inference speed. Deployment of transformer models on devices is important for practical applica- tions, so we test the inference speed of our TNT model. Following [3]], the throughput is measured on an NVIDIA V100 GPU and PyTorch, with 224x 224 input size. Since the resolution and content inside the patch smaller than that the whole image, we may need fewer blocks to learn its representation. Thus, we can reduce the used TNT blocks and replace some with vanilla transformer blocks. From the results in Table we can see that our TNT is more efficient than DeiT and PVT by achieving higher accuracy with similar inference speed.
Table 5: GPU throughput comparison of vision transformer models.
Model
Indices of TNT blocks
FLOPs (B)
Throughput (images/s)
Top-1
DeiT-S
31
907
79.8
DeïT-B
31
17.6
292
81.8
PVT-Small 336
3.8
820
79.8
PVT-Medium
6.7
526
81.2
TNT-S
11,2, 3,
10, 11, 12)
5.2
428
81.5
TNT-S-I
[1,4, 8, 12]
4.8
668
81.4
TNT-S-2
[1, 6, 12] [1. 6] [1]
4.7
704
TNT-S-3
4.7
757
81.1
TNT-S-4
4.6
822
80.8
3.3
Ablation Studies
Effect of position encodings. Position infor- mation important for image recognition. In TNT structure, sentence position encoding is for maintaining global spatial information, and word position encoding is used to preserve lo- cally relative position. We verify their effect by removing them separately. As shown in Table 6 we can see that TNT-S with both patch position encoding and word position encoding performs the best by achieving 81.5% top-1 accuracy. Removing sentence/word position encoding results in 0.8% 7% accuracy drop respectively, and removing all position encodings heavily decrease the accuracy by 1.0%.
Table 6: Effeec of position encoding.
Position encoding
Model
Sentence-level
Word-level
Top-1
80.5
TNT-S
80.8
80.7
81.5
Number of heads. The effect of #heads in standard transformer has been investigated in multiple works ( 35 and head width 64 is recommended for visual tasks 31. We adopt the head width of 64 in outer transformer block in our model. The number of heads in inner transformer block

---

is another for investigation. We evaluate the effect of #heads in inner transformer
block (Table 7. We can see that proper number of heads (e.g., 2 or 4) achieve the best performance.
Table 7: Effect #heads inner transformer block in TNT-S.
#heads
2
4
6
8
Top-1 81.0
81.4
81.5
81.3
|
81.1
Number of visual words. In TNT, the input image is split into a number of 16x 16 patches and each patch is further split into m sub-patches (visual words) of size (s, s) for computational efficiency. Here we the ef- fect of hyper-parameter m on TNT-S architecture. When we change m, the embedding dimension also changes correspondingly to control the FLOPs. As shown in Table
Table 8 Effect #words m.
m
c
Params
FLOPs
Top-1
64
23.8M
5.1B
81.0
16
24
23.8M
5.2B
81.5
4
96
25.1M
6.0B
81.1
we can see that the value of m has slight influence on the performance, and we use m 16 by default for its efficiency, unless stated otherwise.
Deil
DeiT
Block-1
Block-6
Biock-12
TNT
(a) Feature maps in Block-1/6/11
(b) T-SNE of Block 12.
Figure 3: Visualization of the features of DeiT-S and TNT-S.
3.4 Visualization
Visualization of Feature Maps. visualize the learned features of DeiT and TNT to further understand the effect of the proposed method. For better visualization, the input image is resized to 1024x 1024. The feature maps are formed by reshaping the patch embeddings according to their spatial positions. The feature maps in the 1-st, 6-th and 12-th blocks are shown in Fig. ([a) where 12 feature maps are randomly sampled for these blocks each. In TNT, the local information are better preserved compared Dei'.. We also visualize all the 384 feature maps in the 1 12-th block using t-SNE 33 (Fig. ((b)). We can see that the features of TNT are more diverse and contain richer information than those of DeiT.. These benefits owe the introduction of inner transformer block for modeling local features.
In addition to the patch-level features, also visualize the pixel-level embeddings of TNT in Fig, For each patch, we reshape the word embeddings according to their spatial positions to form the feature maps and then average these feature maps by the channel dimension. The averaged feature maps corresponding to the 14x patches are shown in Fig. We can see that the local information is well preserved in the shallow layers, and the representations become more abstract gradually the network goes deeper.
Visualization of Attention Maps. There are two self-attention layers in our TNT block, i.e., an inner self-attention and an outer self-attention for modeling relationship among visual words and sentences respectively. We show the attention maps of different queries in the inner transformer Figure 5 For given query visual word, attention values of visual words with similar appearance are higher, indicating their features will be interacted more relevantly with the query. These interactions are missed VIT and DeiT, etc. The attention maps in the outer transformer can be found in the supplemental material.

---

Blook-1
Block-6
Block-12
Figure 4: Visualization of the averaged word emheddings of TNT-S.
Figure 5: Attention maps of different queries in the inner transformer. Red cross symbol denotes the query location.
3.5 Transfer Learning
To demonstrate the strong generalization ability of TNT, transfer TNT-S, TNT-B models trained on ImageNet to the downstream tasks.
Pure Transformer Image Classification. Following DeiT [3]] we evaluate our models on image classification datasets with training set size ranging from 2,040 to 50,000 images. These datasets include superordinate-level object classification (CIFAR-10 [16), CIFAR-100 (16]) and fine-grained object classification (Oxford-IIIT Pets 23 Oxford 102 Flowers 22 and iNaturalist
2019 (34) ), shown in Table 2 All models are fine-tuned with an image resolution of 384x384. We adopt the same training settings those the pre-training stage by preserving all data augmentation strategies. order to fine-tune in a different resolution, we also interpolate the position embeddings of new patches. For CIFAR- 10 and CIFAR- 100, we fine-tune the models for 64 epochs, and for fine-grained datasets, we fine-tune the models for 300 epochs. Table A compares the transfer learning results of TNT to those of ViT, DeiT' and other convolutional networks. We find that TNT outperforms DeiT in most datasets with less parameters, which shows the superiority of modeling pixel-level relations get better feature representation.
Table Results on downstream image classification tasks with ImageNet pre-training. 384 denotes fine-tuning with x84x3384 resolution.
Model
Params (M)
ImageNet
CIFARIO
CIFARI00
Flowers
Pets
iNat-19
CNN-based
Grafit ResNet-50 32
25.6
79.6
75.9
Grafit RegNetY-SGF
39.2
99.1
80.0
EfficientNet-B5 28
30
83.6
98.7
91.1
98.5
Transformer-based
ViT-B/16;3s1 9
86.4
77.9
98.1
87.1
89.5
93.8
DeiT-B 1384
81
86.4
83.1
99.1
90.8
98.4
TNT-Sra84
23.8
83.1
94.7
81.4
TNT-B
65.6
83.9
99.1
91.1
99.0
95.0
83.2
We construct a pure transformer object detection pipeline by combining TNT and DETR B]. For fair comparison, in PVT [36] and add a 2 2 average pooling to make the output size of TNT backbone the same as that of PVT and ResNet. All the compared models are trained using AdamW [20] with batch size of 16 for 50 epochs. The training images are randomly resized to have a shorter side in the range of [640,800] and a longer side within 1333 pixels. For testing, the shorter side is 800 pixels. The results on COCO v are shown in Table TO Under the same setting, DETR with TNT-S backbone outperforms the representative pure transformer detector DETR+PVT-Small by 3.5 AP with similar parameters.
Pure Transformer Object Detection.
adopt the training and testing settings

---

Table 10: Results of object detection on COC02017 val set with ImageNet pre-training "Results from our implementation.
Backbone
Params
Epochs
AP
AP50
AP75
APs
M
APr.
ResNet-50 33
41M
50
32.3
53.9
32.3
10.7
33.8
53.0
DeiT-St 31
38M
50
33.9
54.7
34.3
11.0
35.4
56.6
PVT-Small
36
40M
50
34.7
55.7
35.4
12.0
36.4
56.7
PVT-Medium
36
57M
50
36.4
57.9
37.2
13.0
38.7
59.1
TNT-S
39M
38.2
58.9
39.4
15.5
58.8
Pure Transformer Semantic Segmentation. We adopt the segmentation framework of Trans2Seg 38 to build the pure transformer
Table 11: Results of semantic segmentation on ADE20K val set with ImageNet pre-training.
Results from our implementation.
semantic segmentation based on TNT backbone. We follow the training and testing configuration in PVT (36] for fair comparison. All the com- pared models are trained by Adam W optimizer with initial learning rate of le-4 and polynomial decay schedule. We apply random resize and crop of 512x512 during training. The ADE20K results with single scale testing are shown in Table IT With similar parameters, Trans2Seg with TNT-S backbone achieves 43.6% mloU, which is 1.0% higher than that of PVT-small backbone and 2.8% higher than that of DeiT-S backbone.
Backhone
Params
FLOPPs
Steps
mloU
ResNet-50
38
56.1M
79.3G
40k
39.7
DeiT-S' [3
30.3M
27.2G
40k
40.5
PVT-Small
43
32.1M
31.6G
40k
42.6
TNT-S
32.1M
30.4G
40k
43.6
Conclusion
In this paper, propose novel Transformer-iN- Trrasfformer (TNT) network architecture for visual recognition. In particular, we uniformly split the image into a sequence patches (visual sentences) and view each patch as a sequence of sub-patches (visual words). We introduce TNT block in which an outer transformer block is utilized for processing the sentence embeddings and an inner transformer block is used to model the relation among word embeddings. The information of visual word embeddings is added to the visual sentence embedding after the projection of a linear layer. We build our TNT architecture by stacking the TNT blocks. Compared to the conventional vision transformers (ViT) which corrupts the local structure of the patch, our TNT can better preserve and model the local information for visual recognition. Extensive experiments on ImageNet and downstream tasks have demonstrate the effectiveness of the proposed TNT architecture.
Acknowledgement
This work supported by NSFC (62072449, 61632003), Research Grant (2020B1515130004 and Macao FDCT (0018/209//AKP, 0015/2019/KK)),
Joint
References
[1] Jimmy Lei Jamie Ryan Kiros, and Geoffrey Hinton. arXiv:1607.0m
Layer normalization.
arXiv preprint
2016.
[2] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
Language models are few-shot learners.
In NeurlPS, 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-ent object detection with transformers. In ECCV, 2020.
[4] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. ICML, 2020.
[6] Xiangxiang Chu, Bo Zhang. Zhi Tian. Xiaolin Wei, and Huaxia Xia. Do we really need explicit position encodings for vision transformers? arXiv preprint 21X::1002210888 2021.
10

---

[7] Ekin D Cubuk. Barret Zoph. Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
[8] Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- reetrainiin of deep bidirec- tional transformers for language understanding. In NAACL- 2019.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth
16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[10] Kai Han, Yunhe Wang. Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang. An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. arXiv preprint arXiv:2012. 12556,, 2020.
[11] Kai Han. Yunhe Wang. Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features
from cheap operations. In CVPR, 2020.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[13] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08.. 2016.
[14] Elad Hoffer, Tal Ben-Nun, Iay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In CVPR, 2020.
[15] Huawei. Mindspore. https://ww/ mi mindooore... cn/| 2020.
[16] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, pages 100,-1105, 2012.
[18] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint 1od5::6600007660 2016.
[19] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755, 2014.
[20] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05100 2017.
arXiv preprint
[21] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In NeurlPS, 2019.
[22] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
[23] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, pages 3498-3505. IEEE, 2012.
[24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 2019.
[25] Iliaa Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. CVPR, 2020.
[26] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 11(())211-252, 2015.
[27] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecttre for computer vision. In CVPR, 2016.
[28] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
[29] Yehui Tang, Kai Han, Chang Xu, An Xiao, Yiping Deng. Chao Xu, and Yunhe Wang. Augmented shortcuts for vision transformers. arXiv preprint arXiv:2 1594441 2021.
[30] Yehui Tang, Yunhe Wang, Yixing Xu, Yiping Deng, Chao Xu, Dacheng Tao, and Chang Xu. Manifold regularized dynamic network pruning. In CVPR, pages 5018-5028 2021.
[31] Hugo Touvron, Matthiuu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé
Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
[32] Hugo Touvron, Alexandre Sablayrolles, Mathijs Douze, Matthieu Cord, and Hervé Jégou. Grafit: Learning ine-grained image representations with coarse labels. arXiv preprint arXiv:201
2020.
[33] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)). 2008
11

---

[34] Grant Van Horn, Oisin Mac Aodha, Yang Song. Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, pages 8769-8778, 2018.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
[36]
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[37] Xiaolong Wang. Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794-7803, 2018.
[38] Enze Xie, Wenjia Wang, Wenhai Wang, Peize Sun, Hang Xu, Ding Liang, and Ping Luo. Segmenting transparent object the wild with transformer. IJCAI, 2021.
[39] Yixing Xu, Yunhe Wang. Kai Han, Ychui Tang, Shangling Jui, Chunjing Xu, and Chang Xu. Renas: Relativistic evaluation of neural architecture search. In CVPR, pages 4411-4420, 2021.
[40] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arX1v:2101.11986, 2021.
[41] Sangdoo Yun. Dongyoon Han, Seong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[42] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint vrXiv:1710.09412 2017.
[43] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang. Philip Torr, Rethinking semantic segmentation from sooguence-to-eqqeeee perspective with transformers. In CVPR, 2021.
[44]
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmentation. In AAAI, volume 34, pages 13001-13008, 2020.
[45] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[46] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiangang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
12


Reviews:
[{"review_id": "dUqTXIPh2nB", "reviewer": "Reviewer_Gg7D", "summary": "The work proposes a nested vision Transformer variant (TNT), which adds an additional inner Transformer to capture within-patch information. By setting the hidden dimension of the inner Transformer much smaller, the FLOPs and #Params of the proposed TNT won't be overly large. Empirically, authors show that the proposed model performs better than ViT/DeiT on ImageNet classification, COCO object detection and ADE20K semantic segmentation.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using a nested Transformer for visual processing is intuitive and reasonable. From the perspective of FLOPs and #Params, the complexity analysis \"partially\" address the concern of why adding another Transformer which processes the input at a much fine-grained level won't be too expensive.\n\nHowever, since the inner Transformer has to process the image at the \"word\" level, the number FFN projections is dominated by `n x m` (`n` is the number of patches and `m` is the number of sub-patches in each patch), which is much larger than the number of FFN projections in ViT. While using a small inner hidden dim (`c`) seems to resolve the problem in complexity, I was wondering how much this actually slows down the running time (training & inference), since many smaller matrix multiplications is usually slower than few larger matrix multiplications of the same FLOPs under modern hardwares (e.g. GPU with tensor cores and TPU). This information is not mentioned or discussed in the current version.\n\nAnother question I have is whether one needs such a complicated inner network (Transformer) to capture the local granularity (16x16 pixels), especially if we are concerned with the ultimate efficiency. Experimenting with some simpler architectures may give more insights for this question.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The propose method should bring in any additional limitation of negative societal impact when compared to ViT.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "ZPq-W05ADC9", "reviewer": "Reviewer_6Dax", "summary": "This paper proposes a novel vision transformer (TNT) to better model the fine-grained relationship within local image patches. \nSpecifically, based on the patch splitting scheme in ViT, the authors further split the local patches (\"visual sentence\") into smaller ones (\"visual words\"). Inner transformers are adopted to model relationships between words and improve sentence representation.\nExperiments on ImageNet classification and several downstream tasks validate the effectiveness of the proposed method.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The motivation is clear. The idea of modeling fine-grained local features is also explored in concurrent works, e.g., Swin Transformer[1] and Twins[2].\n2. Experiments are carried out for ImageNet classification and several downstream tasks. \n3. The writing is good and easy to follow.\n\n\nWeakness:\n1. With the inner transformers, the number of transformer layers in TNT is doubled compared to the DeiT baseline. It's hard to judge whether the performance improvement owes to better intra-visual sentence relationship modeling, or simply improved model depth. An empirical study is recommended to figure out the reason.\n\n2. Compared with concurrent works, the proposed method only brings moderate performance gain to the DeiT baseline. Moreover, only two variants of TNT are instantiated, making it hard to understand the scaling property of TNT or compare with other methods in a systematic way. Considering the difficulty of training deep ViTs [3, 4] and the doubled model depth in TNT, will the model be more prone to overfitting when the model scales? Besides, the performance of lightweight backbone architecture is also of great importance, especially for applications with limited computation resources. Therefore, smaller and larger models, e.g., ~6M (Tiny), ~100M (Large), or bigger ones, should be trained and evaluated for a comprehensive comparison.\n\n3. Modern object detectors [5, 6] usually exploit multi-scale features, where small objects are detected from high-resolution feature maps. However, there is no design in TNT to make it adaptable to high-resolution inputs or features. This will inevitably result in high memory consumption. In the experiments, TNT-based object detection and segmentation can only be performed on single-scale image features. The reviewer wonders that is it adaptable to methods that can significantly benefit from multi-scale features, e.g., Deformable DETR and Faster RCNN with FPN, considering that the TNT is designed as a general backbone network for different downstream tasks?\n\n4. Since TNT specifically focuses on modeling the \"visual words\" by splitting the local patches (\"visual sentence\") into smaller ones, it should provide better local feature representations, which could be useful for fine-grained visual understanding tasks, e.g., fine-grained image classification. Although the authors have carried experiments on the Flowers and Pets benchmarks, the performance on these two benchmarks has been saturated. Experiments on more challenging datasets such as iNaturalist19 can be provided to further evaluate the effectiveness of the proposed method.\n\nReferences:\n1. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows\n2. Twins: Revisiting the Design of Spatial Attention in Vision Transformers\n3. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\n4. Going deeper with Image Transformers\n5. Deformable DETR: Deformable Transformers for End-to-End Object Detection\n6. Feature Pyramid Networks for Object Detection", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have mentioned potential negative impacts such as energy consumption and carbon dioxide emissions of GPU computation. The limitations are also mentioned in the Experiments section.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "WlGbtMZGQZi", "reviewer": "Reviewer_u1uh", "summary": "This paper proposes an improved ViT network that applies self-attention layers to a hierarchical patches: a 16x16 patch on the original image (the authors called them \"sentence-level attention), and a 4x4 sub patch for each of the original path (the authors called them \"word-level attention). Overall, the idea is very simple and the results are promising: 1.7% better top-1 accuracy on ImageNet than DeiT.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "----------- Strengthes -----------------\n\n(1) Very simple but effective idea: the 16x16 patch on the original ViT paper seems like an obvious problem, as it brutally combines a large image patch into a single \"word\" in order to feed to Transformer.  Therefore, this paper proposes to apply intra-word attention to solve this issue. It is a quite nice idea to me.\n(2) Comprehensive analysis: it provides detailed analysis on the additional FLOPs/latency, and provides various analysis on visualization and transfer learning.\n(3) Promising results: compared to DeiT, the improvements are promising.\n\n\n--------- Weaknesses ------------------\n\n(1) Missing latency results: while the FLOPs/Parameters overhead seem to be small, it is unclear whether such intra-work attention is slow or not on real hardwares like GPUs/TPUs. It would be helpful to compare the training/inference speed.\n(2) Fair comparison: ViT is generally sensitive to training hyperparamters (e.g., DeiT has much better accuracy than ViT for the same network). Although the authors report 1.7% better accuracy, but it is not clear whether such improvements are partially from training settings. I know the authors use similar settings as DeiT, but it is better to report your reproduced numbers of ViT/DeiT using EXACTLY the same settings as your TNT.\n\n\n-------  Additional questions ----------\n\nWhen the input image is small, using two-levels of attention is probably good enough. However, if the input image is large (like 640 for detection or 1024 for segmentation), would it be reasonable to use more levels of attention?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "7QOR78FvrI", "reviewer": "Reviewer_xZPU", "summary": "This paper presents a new Transformer-based framework \"Transformer-iN-Transformer\" (TNT) for visual recognition. The proposed TNT module can better preserve and model the local information for visual recognition compared to conventional VIT. Extensional experiments prove its effectiveness in image classification and object detection.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n\nThis idea is novel, especially because the proposed framework is a leading work that views each patch as a sequence of sub-patches (visual words).\n\n---\n\nQuality: \n\nThe quality of this paper is good in general. Everything looks fair and reasonable.\n\n---\n\nClarity: \n\nThe paper is well-written and easy to follow. The figures and tables in this paper are neat and clear.\n\n---\n\nSignificance: \n\nThe paper will be impactful as it is one of the leading works that explore the locality of vision Transformer.\n\n---\n\nOverall, I agree with the contributions claimed in this paper and I suggest this paper be accepted.\n\n\n---\n\nSummary: This paper presents a new Transformer-based framework \"Transformer-iN-Transformer\" (TNT) for visual recognition. The proposed TNT module can better preserve and model the local information for visual recognition compared to conventional VIT. Extensional experiments prove its effectiveness in image classification and object detection.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a simple but effective ViT network architecture, which introduces hierarchical structure in transformers. The architecture is motivated well and the paper is well-organized. 

Although in the submission, the empirical study is not sufficient to the advantage of the proposed model, during the rebuttal period, the authors did a great job to provide more evidences. It is a clear acceptance for this paper. Please include all the additional empirical experiments results into the final version, especially the running time in training & inference stage and the necessity of inner transformer.  
Decision: Accept (Poster)