Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new Transformer-based framework "Transformer-iN-Transformer" (TNT) for visual recognition. The proposed TNT module can better preserve and model the local information for visual recognition compared to conventional VIT. Extensional experiments prove its effectiveness in image classification and object detection.
main_review: Originality: 

This idea is novel, especially because the proposed framework is a leading work that views each patch as a sequence of sub-patches (visual words).

---

Quality: 

The quality of this paper is good in general. Everything looks fair and reasonable.

---

Clarity: 

The paper is well-written and easy to follow. The figures and tables in this paper are neat and clear.

---

Significance: 

The paper will be impactful as it is one of the leading works that explore the locality of vision Transformer.

---

Overall, I agree with the contributions claimed in this paper and I suggest this paper be accepted.


---

Summary: This paper presents a new Transformer-based framework "Transformer-iN-Transformer" (TNT) for visual recognition. The proposed TNT module can better preserve and model the local information for visual recognition compared to conventional VIT. Extensional experiments prove its effectiveness in image classification and object detection.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The work proposes a nested vision Transformer variant (TNT), which adds an additional inner Transformer to capture within-patch information. By setting the hidden dimension of the inner Transformer much smaller, the FLOPs and #Params of the proposed TNT won't be overly large. Empirically, authors show that the proposed model performs better than ViT/DeiT on ImageNet classification, COCO object detection and ADE20K semantic segmentation.
main_review: The idea of using a nested Transformer for visual processing is intuitive and reasonable. From the perspective of FLOPs and #Params, the complexity analysis "partially" address the concern of why adding another Transformer which processes the input at a much fine-grained level won't be too expensive.

However, since the inner Transformer has to process the image at the "word" level, the number FFN projections is dominated by `n x m` (`n` is the number of patches and `m` is the number of sub-patches in each patch), which is much larger than the number of FFN projections in ViT. While using a small inner hidden dim (`c`) seems to resolve the problem in complexity, I was wondering how much this actually slows down the running time (training & inference), since many smaller matrix multiplications is usually slower than few larger matrix multiplications of the same FLOPs under modern hardwares (e.g. GPU with tensor cores and TPU). This information is not mentioned or discussed in the current version.

Another question I have is whether one needs such a complicated inner network (Transformer) to capture the local granularity (16x16 pixels), especially if we are concerned with the ultimate efficiency. Experimenting with some simpler architectures may give more insights for this question.
limitations_and_societal_impact: The propose method should bring in any additional limitation of negative societal impact when compared to ViT.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a novel vision transformer (TNT) to better model the fine-grained relationship within local image patches. 
Specifically, based on the patch splitting scheme in ViT, the authors further split the local patches ("visual sentence") into smaller ones ("visual words"). Inner transformers are adopted to model relationships between words and improve sentence representation.
Experiments on ImageNet classification and several downstream tasks validate the effectiveness of the proposed method.  
main_review: Strength:
1. The motivation is clear. The idea of modeling fine-grained local features is also explored in concurrent works, e.g., Swin Transformer[1] and Twins[2].
2. Experiments are carried out for ImageNet classification and several downstream tasks. 
3. The writing is good and easy to follow.


Weakness:
1. With the inner transformers, the number of transformer layers in TNT is doubled compared to the DeiT baseline. It's hard to judge whether the performance improvement owes to better intra-visual sentence relationship modeling, or simply improved model depth. An empirical study is recommended to figure out the reason.

2. Compared with concurrent works, the proposed method only brings moderate performance gain to the DeiT baseline. Moreover, only two variants of TNT are instantiated, making it hard to understand the scaling property of TNT or compare with other methods in a systematic way. Considering the difficulty of training deep ViTs [3, 4] and the doubled model depth in TNT, will the model be more prone to overfitting when the model scales? Besides, the performance of lightweight backbone architecture is also of great importance, especially for applications with limited computation resources. Therefore, smaller and larger models, e.g., ~6M (Tiny), ~100M (Large), or bigger ones, should be trained and evaluated for a comprehensive comparison.

3. Modern object detectors [5, 6] usually exploit multi-scale features, where small objects are detected from high-resolution feature maps. However, there is no design in TNT to make it adaptable to high-resolution inputs or features. This will inevitably result in high memory consumption. In the experiments, TNT-based object detection and segmentation can only be performed on single-scale image features. The reviewer wonders that is it adaptable to methods that can significantly benefit from multi-scale features, e.g., Deformable DETR and Faster RCNN with FPN, considering that the TNT is designed as a general backbone network for different downstream tasks?

4. Since TNT specifically focuses on modeling the "visual words" by splitting the local patches ("visual sentence") into smaller ones, it should provide better local feature representations, which could be useful for fine-grained visual understanding tasks, e.g., fine-grained image classification. Although the authors have carried experiments on the Flowers and Pets benchmarks, the performance on these two benchmarks has been saturated. Experiments on more challenging datasets such as iNaturalist19 can be provided to further evaluate the effectiveness of the proposed method.

References:
1. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
2. Twins: Revisiting the Design of Spatial Attention in Vision Transformers
3. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
4. Going deeper with Image Transformers
5. Deformable DETR: Deformable Transformers for End-to-End Object Detection
6. Feature Pyramid Networks for Object Detection
limitations_and_societal_impact: The authors have mentioned potential negative impacts such as energy consumption and carbon dioxide emissions of GPU computation. The limitations are also mentioned in the Experiments section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an improved ViT network that applies self-attention layers to a hierarchical patches: a 16x16 patch on the original image (the authors called them "sentence-level attention), and a 4x4 sub patch for each of the original path (the authors called them "word-level attention). Overall, the idea is very simple and the results are promising: 1.7% better top-1 accuracy on ImageNet than DeiT.
main_review: ----------- Strengthes -----------------

(1) Very simple but effective idea: the 16x16 patch on the original ViT paper seems like an obvious problem, as it brutally combines a large image patch into a single "word" in order to feed to Transformer.  Therefore, this paper proposes to apply intra-word attention to solve this issue. It is a quite nice idea to me.
(2) Comprehensive analysis: it provides detailed analysis on the additional FLOPs/latency, and provides various analysis on visualization and transfer learning.
(3) Promising results: compared to DeiT, the improvements are promising.


--------- Weaknesses ------------------

(1) Missing latency results: while the FLOPs/Parameters overhead seem to be small, it is unclear whether such intra-work attention is slow or not on real hardwares like GPUs/TPUs. It would be helpful to compare the training/inference speed.
(2) Fair comparison: ViT is generally sensitive to training hyperparamters (e.g., DeiT has much better accuracy than ViT for the same network). Although the authors report 1.7% better accuracy, but it is not clear whether such improvements are partially from training settings. I know the authors use similar settings as DeiT, but it is better to report your reproduced numbers of ViT/DeiT using EXACTLY the same settings as your TNT.


-------  Additional questions ----------

When the input image is small, using two-levels of attention is probably good enough. However, if the input image is large (like 640 for detection or 1024 for segmentation), would it be reasonable to use more levels of attention?
limitations_and_societal_impact: N/A
review_title: null

Meta-review: In this paper, the authors propose a simple but effective ViT network architecture, which introduces hierarchical structure in transformers. The architecture is motivated well and the paper is well-organized. 

Although in the submission, the empirical study is not sufficient to the advantage of the proposed model, during the rebuttal period, the authors did a great job to provide more evidences. It is a clear acceptance for this paper. Please include all the additional empirical experiments results into the final version, especially the running time in training & inference stage and the necessity of inner transformer.  
Decision: Accept (Poster)