Question: For Figure 1. Did authors include all recent transformer models?
Evidence: Transformer, which was initially proposed for natural language processing (NLP) and is a type of deep neural networks (DNNs) mainly based on the self-attention mechanism, has achieved significant breakthroughs in NLP tasks. Thanks to its strong representation capabilities, many works have developed ways to apply Transformer to computer vision (CV) tasks, such as image classification, object detection, semantic segmentation , and video classification. Among them, Vision Transformer (ViT) stands out and demonstrates that a pure Transformer applied directly to sequences of image patches can perform very well on image classification tasks, e.g., achieving a comparable ImageNet top-1 accuracy as ResNet. Motivated by ViT's promising performance, a fast growing number of works follow it to explore pure Transformer architectures in order to push forward its accuracy-efficiency trade-off and deployment into real-world applications, achieving an even better performance than EfficientNetV1, a widely used efficient convolutional neural network (CNN).
The success of recent ViT works suggests that the model architecture is critical to ViT's achievable performance. Therefore, in this work we explore ViT architectures from a new perspective, aiming to analyze and understand ViT's architecture design space and real hardware-cost across different devices. Despite the recent excitement towards ViT models and the success of model scaling for CNNs, little effort has been made into exploring ViT's model scaling strategies or hardware-cost.
DeiT-Scaled (Ours) PiT-Scaled (Ours) Our scaled ViT models achieve comparable or better accuracy-efficiency trade-off as compared to some recent dedicatedly designed ViT variants.
Note that directly applying the scaling strategies for CNNs or Transformer on NLP tasks  will lead to sub-optimality, as discussed in Section 3.2. Furthermore, scaling strategies targeting one device/task might not be transferable to another device/task. Interestingly, we find that simply scaled ViT models can achieve comparable or even better accuracy-efficiency trade-off than dedicatedly designed ViT variants, as shown in Figure. Motivated by this, we further explore the transferability of our scaling strategies across different real hardware devices and (2) to different ViT variants and tasks. In particular, we make the following contributions:
• We are the first to show that simply scaled vanilla ViT models can achieve comparable or even better accuracy-efficiency trade-off as compared to dedicatedly designed ViT variants, as illustrated in Figure. Specifically, as compared to DeiT-Tiny, our scaled model achieves a ↑ 1.9% higher ImageNet top-1 accuracy under the same FLOPs and a ↑ 3.7% better ImageNet top-1 accuracy under the same latency on an NVIDIA Edge GPU TX2. • We study the transferability of the scaled ViT models across different devices and show that the transferability effectiveness depends on the underlying devices and deployment tools. For example, scaling strategies targeting FLOPs or the throughput on V100 GPU (NVIDIA LLC.) can be transferred to the Pixel3 (Google LLC., a) device with little or even no performance loss, but those targeting the latency on TX2 (NVIDIA Inc., c) may not be transferred to other devices due to the obvious performance loss. Additionally, we provide ViT models' cost breakdown and rank correlation between their hardware-cost on different devices for better understanding it. • We show that our scaling strategies can also be effectively transferred to different ViT variants and recognition tasks to further boost the achieved accuracy-efficiency trade-off, e.g., achieving a ↑ 2.1% higher accuracy under a similar FLOPs when being transferred to the PiT model and ↑ 0.7% higher average precision under a similar inference throughput when being transferred to an object detection task.
Figure 1: Our scaled ViT models achieve comparable or better accuracy-efficiency trade-off as compared to some recent dedicatedly designed ViT variants.
Answer: Thank you for your suggestions! As we all recognize, the field of ViTs is evolving very fast now, and thus we were not able to include all the recent transformer models. Meanwhile, we hope to humbly remind that ICLR doesn’t require submissions to compare with works published on or after June 5, 2021 as mentioned in the Reviewer Guide, so we did not include all recent transformer models, e.g., LeViT [1] published in ICCV’21 which released the decision to authors on July 22, 2021. Following your suggestion, we have modified the corresponding claim about Figure 1 in our updated manuscript.