Question: Existing calling methods for ViT. in section 3.2 authors first talk about CNN scaling strategies and conclude that those applied to ViTs can lead to ambiguity and sub-optimal performance. Where can we find supporting evidence for the statement?
Evidence: CNN and ViT scaling factors do not match. Scaling strategies dedicated to CNNs mostly come with CNN-specific scaling factor definitions (e.g., the number of channels in convolution layers represents the model width), which cannot be directly transferred to ViT. For example, doubling (2×) the width in CNNs can be achieved via various combinations of the number of heads (h) and embedding dimension for each head (e) in ViT. Furthermore, there are extra scaling factors for ViT, e.g., the linear projection ratio (r) and the patch size (p), which do not exist in the scaling factors for CNNs but are important for ViT as shown in Appendix D, thus directly transferring the scaling strategies from CNNs to ViTs can lead to ambiguity and sub-optimal performance.
Transformer scaling strategies for NLP is sub-optimal on ViT.  noted that for NLP, model performance (i.e., accuracy or training loss) depends "strongly on the model scale (i.e., the number of parameters), but weakly on the model shape". However, when scaling ViT along the factors summarized in Figure, our observations suggest that this is not true for ViT. As shown in Figure, when performing an extensive search on top of DeiT-Small following, we observe that a model's shape has a great impact on the performance. Specifically, if we change the aspect ratio, i.e., the ratio between the embedding dimension (e × h) and the number of layers (d), while keeping the model parameters to be the same, the accuracy drifts as much as 18.61%. This set of experiments motivates exploring scaling strategies dedicated to ViT. Starting from a relatively small model defined in Table, we adopt a simple iterative greedy search to perform the ViT scaling step by step, similar to the previous algorithms for exploring CNN design spaces and feature selections. Specifically, based on the starting point model or the optimal one from the previous scaling step, we scale up the model along each standalone scaling factor introduced in Section 3.1 to match the target hardware-cost (e.g., FLOPs, or latency on a specific hardware device), and select the one with the best accuracy-efficiency trade-off to be the starting point model of the next step. As analyzed in, unlike scaling strategies extracted from a specific model, scaling based on such an iterative greedy search can avoid the unscalability of the resulting scaling strategies on a specific model. Our experiments in Section 4.1 also verify that such a scaling method is simple yet effective for scaling ViT models, and only requires training a few models during each search step.
Furthermore, there are extra scaling fac_x0002_tors for ViT, e.g., the linear projection ratio (r) and the patch size (p), which do not exist in the scaling factors for CNNs but are important for ViT as shown in Appendix D, thus directly transferring the scaling strategies from CNNs to ViTs can lead to ambiguity and sub-optimal performance.
Answer: One example is the extracted scaling strategies for TX2, as shown in Appendix D. Specifically, we can observe that the scaled ViTs in TX2 tend to enlarge more on the linear projection ratio (r), while the linear projection ratio does not exist in the scaling factors for CNNs. So applying the CNN scaling strategies to ViT while ignoring the linear projection ratio will lead to a sub-optimal performance in terms of accuracy vs. latency on TX2 trade-off. To make it more clear, we have also modified the corresponding description in Section 3.2 in the updated manuscript.