Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies different hyper-parameters in the ViT design space (e.g., the number of layers, the dimensions per head, the number of head, the MLP ratio, the input image size). It starts with a very small, basic model, and then scales it up based on an iterative, greedy search. Each step it will select the best model (in terms of speed-accuracy trade-off) as the starting point of the next step. The "speed" aspect is firstly measured by FLOPs, and then later replaced by empirical measurements like latency on some devices. Overall, the resulting architectures show better speed-accuracy trade-offs than previous DEiT architectures, and shows consistent gains when transferred to other architectures (PiT) and tasks (e.g., object detection).
main_review: + The best part of the paper is about its "usefulness" and practical friendliness. The paper just studies the different factors that contributes to the model size/speed and accuracy (usually larger models on higher resolutions lead to better accuracy but slow down inference speed). An impressive part of the paper is that it shows measurements of speed beyond FLOPs -- it actually runs on various devices and measure the latency there. I think this practical mind is great.
+ It also shows good generalization of the finding. E.g. it can be transferred to more recent models (PiT), other tasks (object detection, video). This suggests solid improvement.
+ Overall the paper is well-written. The idea is clear, the organization is clean, and the figures are very illustrative. Overall it is even a good learning experience for me (e.g. knowing the latency break down of different devices) who is not so familiar with the hardwares.

- Biggest concern about this paper is novelty. It has some good findings, it shows good numbers, and it also teaches us something. However, the method (for search) is based on existing work, the application to other devices is quite straightforward (though surely requires a lot of hard work), and to me, it is more like a technical report about something is well executed, rather than an interesting "finding" that has a good deal of scientific merit.
- One more concern is about the word "scale". I don't think the study presented in this paper is about "scaling". To me the scaling is more about methods that scales *up* to larger models -- when people say about doing things "at scale", it indicates at "a large" scale. I come with the expectation that this paper can show how to scale our existing model to the next generation of model size, but only to find this paper is more about "speed-accuracy" trade-offs. So I would recommend replacing the term "scale" with "speed-accuracy trade-off" to be precise.
summary_of_the_review: I think this paper is more on the "technical report" side, with quite limited novelty. I also find it gives me the wrong impression that the investigation is more about "scaling-up", but instead it just does search in the "small-to-base" scale regime. It definitely teaches me something and presents some interesting data points, but to me the scientific merit is not enough to warrant a publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper is focused on scaling width, input image resolution and other configurations of ViT model (6 in total) to make it hardware friendly and reach better accuracy when trained from the scratch. Results are verified on the edge devices - Nvidia TX2, Pixel3. Models optimized for different hardware result in higher performance on the targeted device. 

main_review: Strength:
- Approach is tested on multiple devices under different constrains (FLOPs, latency etc). It is clearly shown that optimization for a specific hardware results in a better corresponding model. 
- Scaling factors for models are clearly indicated in the tables for reproducibility.
- Improvements are shown over the baseline Deit models and are up to 1.9% better.

Weakness:
- For Figure 1. Did authors include all recent transformer models? For example LeViT is not there, it demonstrates higher accuracy and lower number of FLOPs (LeViT is 1.2 and 2.4x faster than PiT). If it holds then authors should be careful when comparing to hand designed ViT and claiming comparable or better accuracy-efficiency trade-off. Will be interesting to look at the updated figure with recent models. 
- Scaling method based on an iterative greedy search is not presented well. Going over the section 3.3 multiple times, it is still not clear how exactly the search was performed. Additionally, it is not obvious how different target constraints are reached (FLOPs or latency).  

Minor:
- In the introduction when comparing to CNNs, author mention EfficientNetV1, however, a much stronger EfficientNetV2 has been published. 
- Include top performing CNN models into Figure 1.
- Existing calling methods for ViT. in section 3.2 authors first talk about CNN scaling strategies and conclude that those applied to ViTs can lead to ambiguity and sub-optimal performance. Where can we find supporting evidence for the statement? 

summary_of_the_review: Paper contains an interesting empirical study on how iterative greedy search can be applied to the design parameters of ViT. Resulting models show speed-latency improvements over hand designed ViT models. The method of the search is not well explained (it seems to be the previously proposed method). Resulting models seem to be less generic and more engineered, not clear how presented results will scale beyond Tiny/Small/Base models. Presented study will be interesting for people designing ViT models, however its limitation is in the conclusions/findings specific only to ViT models (even with transfer to PiT).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a model scaling strategy to change the model size. With a greedy search strategy, the author proposes a hardware-aware scaling method. Compared with the existing ViT variants, the proposed method achieves better performance.
main_review: Strength:

-This paper analyzes different scaling strategies of vision transformers and proposes a hard-ware scaling strategy for vision transformers. The authors conduct extensive experiments and bring some insights.

-The proposed model achieves a better trade-off between FLOPs and accuracies than the existing models. It is interesting to see that only adjusting the model size can achieve better performance than SOTA transformer models.

Weakness:
- [1] has discussed how to scale vision transformers. A more detailed comparison with [1] is required.

- Hardware-Aware architecture searching has been well investigated. The proposed method is not novel.

[1] Scaling Vision Transformers, arxiv 2106



summary_of_the_review: The proposed method is not novel, but the insights brought by the authors are interesting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a study for exploring hardware-aware ViT scaling and demonstrates that a simply scaled vanilla ViT model can achieve a comparable accuracy-efficiency trade-off as compared to dedicatedly designed ViT variants. Furthermore, they discuss the transferability of the scaling strategies on different hardware devices, ViT variants, and computer vision tasks. 
main_review: Strengths:
1.This work provides a new perspective to explore ViT architectures, aiming to analyze and understander ViT’s architecture design space and real hardware-cost across different devices.
2. This work summarizes how to determine if a model searched on one hardware device applies to another.

Weakness:
1.Some typos such as “TRAFE-OFFS” in the title of section 4.1.
2.The 24 different structures generated by random premutation in section 4.1 should be explained in more detail.
3.The penultimate sentence of Section 3.3 states that "iterative greedy search can avoid the suboptimality of the resulting scaling strategy on a particular model", which is not a serious statement because the results of the iterative greedy search are also suboptimal solutions. 
4.The conclusion of "Cost breakdown can indicate the transferability effectiveness" in Figure 7 is not sufficient. We cannot extend the conclusion obtained from a few specific experiments to any different hardware devices or different architectures.
5.Why not use the same cost for different devices instead of flops, latency, and 1/FPS for different hardware?
6.The result comparison of "Iteratively greedy Search" versus "random search" on the model structure should be supplemented.

summary_of_the_review: Please provide a short summary justifying your recommendation of the paper
Although some parts of the paper are not clearly explained and the method is not very innovative, the analysis of the transferability of different scaling strategies on different hardware devices in this work is very positive and provides great inspiration to relevant fields. Therefore, it is recommended to accept this paper.
review_title: null

Meta-review: This paper explores strategies for scaling vision transformers that can be transferable across hardware devices and ViT variants. While it presents some interesting observations as well as a useful practical guide, multiple reviewers expressed major concerns over the novelty and significance of the methods and findings. Besides novelty and significance, there are also some concerns about comparison with existing work as well as clarity of the presentation.
Decision: Reject